tinystories

Llama2 inference in one TypeScript file

🎈 A series of lightweight GPT models featuring TinyGPT Base (~51M params) and TinyGPT2 (~95M params). Fast, creative text generation trained on whimsical stories.

Codebase for Linguistic Collapse: Neural Collapse in (Large) Language Models [NeurIPS 2024] [arXiv:2405.17767]

Dataset Generation Code for SimpleStories

Implementation of a small GPT-style transformer from scratch in PyTorch. Learn how Large Language Models work by building, training, generating text, and visualizing attention.

Train and run a small Llama 2 model from scratch on the TinyStories dataset.

Train GPT style model on tinystories dataset

Code implementation for our paper "BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-training" as part of the 2024 BabyLM Challenge

Small transformer trained from scratch

A ~30M GPT small language model trained from scratch on TinyStories — GPT-2-compatible so the ember inference server runs it.

This project fine-tunes GPT-2, a popular pre-trained transformer model, to generate short stories using the TinyStories dataset. The goal is to teach GPT-2 to produce creative and coherent stories based on prompts.

Specialized agentic coding LLM — 3B, pre-trained from scratch, runs locally on Apple Silicon. No API key, no data leaving your machine, zero cost per call.

A 30.72M parameter LLM running dense on an $8 ESP32-S3. Every stored parameter is multiplied on every token. 0.95 tok/s, no cloud, no SD card.

A complete LLM inference engine from scratch in ~1400 lines of dependency-free C — transformer, BPE tokenizer, sampler, hand-rolled thread pool, int8 quantization. No OpenMP, no BLAS, no API keys.

A port of karpathy/llama2.c to MS-DOS

A 27.8M-parameter GPT trained from scratch on TinyStories — transformer, tokenizer pipeline and training loop written by hand in PyTorch. Trained weights included.

LLM Architecure, Open Source

A PyTorch implementation of a Bigram Language Model using Transformer architecture for character-level text generation.

StoryByte — a ~1M-parameter GPT trained from scratch on TinyStories. The model behind the Let's Data Science 'Build a Tiny LLM — From Tokens to Text' course; its whole forward pass runs in ~40 lines of NumPy in a browser.

Built a GPT-2 architecture from scratch in PyTorch and trained it on the TinyStories dataset. The result is a lightweight model that generates coherent, short form children's stories.