morgan
(Morgan McGuire)
August 22, 2020, 12:17am
1
@ilovescience highlighted in Discord that Andrej Karpathy recently released a nicely minimalist implementation of GPT, with example notebooks too
@muellerzr said I should take a look. And so here is a fastai version!
Fastai Version
Fastai version of his Play_Char notebook, training it on Shakespeare and then generating new dialogue. Happy to hear any and all feedback!
(Its not working with mixed precision yet due to a small bug in model’s forward)
minGPT-Fastai_Play_Char.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# A Quick Demo of Andrej Karpathy's minGPT Play Char Demo \n",
"- You can find the Play Char demo in the minGPT repo here: https://github.com/karpathy/minGPT\n",
"- Goal: Generate Shakespere\n",
"\n",
This file has been truncated. show original
11 Likes
nchukaobah
(nkiruka chuka-obah)
August 22, 2020, 9:26am
2
This is very nice. Thank you.
1 Like
morgan
(Morgan McGuire)
August 24, 2020, 7:25pm
3
Updated version here, more fastai-like and also working with mixed precision! (Note that I had to use to_native_fp16
as to_fp16
wasn’t training well at all (maybe due to LayerNorm?)
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# A Quick Demo of Andrej Karpathy's minGPT Play Char Demo \n",
"\n",
"**Our Goal**: Generate Shakesperean text\n",
"- You can find the original Play Char demo in the minGPT repo here: https://github.com/karpathy/minGPT\n",
"- This notebook is partially based on the fastai Transformers tutorial: http://docs.fast.ai/tutorial.transformers"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# For Colab\n",
This file has been truncated. show original
3 Likes