What it does

Just a quick overview of what the dashboard does:

• I have picked 20 Indian metropolitan cities and displayed them on an interactive map, with live temperature, humidity and AQI refreshed every 15 minutes
• When you click a marker or a leaderboard entry, you get 24h / 7d / 30d charts for the temperature, humidity and AQI
• The map is based on Mapbox, locked to India’s bounds
• The site is static, so fetching runs server-side on a cron and the page just reads JSON

All the numbers are fetched from Open-Meteo and WAQI:

Why these two options:

• I wanted free APIs with no client-exposed key which rules the paid tiers out like AccuWeather and OpenWeatherMap (even Google Maps I guess).
• Open-Meteo is free for non-commercial use and runs on the same ECMWF + GFS models behind most weather apps.
• WAQI is the more interesting pick. Its Indian feed comes directly from the CPCB ground stations (the same network behind airquality.cpcb.gov.in) and republishes on the natural CPCB cadence of about 15 minutes, so it is effectively the official live AQI reading with a free developer API in front.
• Open-Meteo’s air-quality stream is different in kind. It is a global chemistry model on a coarse grid which is good for the 24h/7d/30d historical trends but tends to under-read Indian pollution episodes compared to CPCB stations.

I built this in three loose stages, with stages 2 and 3 overlapping in practice. Below is roughly how each went.

Stage 1: brainstorming and building a working v1

When I am building something new my first instinct is to brainstorm with the coding assistant (Claude Code in my case, codex or whichever one you use) before even asking it to write a single line of code. The shape of the prompt is roughly:

here is what I want to build, here are the options I see, here are the constraints, help me decide.

For this dashboard the biggest constraint was easy to state: since the site is my personal portfolio, anything I pulled in had to fit inside a free tier with enough headroom for the kind of traffic a personal site sees.

The one line that ALWAYS helps me is appending this at the end of the prompt:

Interview me until you have 95% confidence about what I actually want, not what I think I should want.

It tells the model to keep questioning your assumptions instead of taking them at face value which is exactly what you want when you are scoping a new or large feature. Whenever I am about to integrate something I have not built before, this single line saves me a lot of time in fast tracking the development and arriving at a plan that actually fits.

For this dashboard the back-and-forth helped me take the important decisions early.

• I had said Google Maps, Claude pointed out that on a static site any key would leak in client JS and steered me to Mapbox’s free tier.
• I had said AccuWeather, it asked whether brand mattered more than accuracy for India, and pushed me to Open-Meteo + WAQI, both free and key-optional.
• It also laid out the cadence logic for me: a GitHub Actions cron every 15 minutes that writes a single JSON to a data branch on the same repo which the page then reads on load.
• By the end of the session I had a page that was live with 8 metros, a marker plus card UI and a working live tile and the data was being fetched every 15 minutes.

Full Claude Code conversation for this stage: stage-1-brainstorm-and-v1.txt.

Stage 2: improving the dashboard with history charts and UI fixes

The next thing that I wanted to do was to add history charts under the map. I wanted to be able to view the history of the temperature, humidity and AQI for each city over certain time periods. This is the first version that I made conversationally with Claude:

Obviously, this was not the best way to view the values. The 7d and 30d views looked terrible with the day-night cycle turning every chart into a sawtooth. Next, I sent Claude the screenshot and asked one question:

Currently, this is how the values look over the 7d and 30d period, I dont think that is the best way to view the values. What is the standard way to show these values?

Claude pointed me at the weather.com pattern where weather apps aggregate the values by day, show a min/max bands with a daily mean line for temp and humidity, and color the AQI bars by EPA category. A second daily cron rebuilds the 30-day window from Open-Meteo each run which looked much nicer:

Finally, I improved the UI by fixing a few small issues. Some of them you can see below:

• Better labeling for the cities that sit close together, better placement of the labels and leader lines
• Locking the map to India’s bounds
• Adding a reset-view control to the map
• A few alignment and styling tweaks

Full Claude Code conversations for this stage: history charts, daily aggregates, UI polish.

Stage 3: moving the cron from GitHub Actions to Cloudflare

After the dashboard was deployed I noticed the live tile timestamps were drifting. The label said “every 15 minutes” but the GitHub Actions free-tier cron was firing erratically, sometimes 40 mins or even >1 hour late. This is usually the case with the free-tier cron jobs on GitHub Actions.

I decided to move the cron to Cloudflare Workers which has a free tier and fires within seconds of schedule.

The interesting thing for me here is that I had never used Cloudflare workers for cron jobs before. Given how good these assistants are at writing code, it wrote the code, walked me through setup steps and the live cadence has been working well since. Even unfamiliar backend infra is something you can prompt your way through (obviously you should have the fundamental knowledge to nudge it in the right direction).

Full Claude Code conversation for this stage: Cloudflare Worker migration.

What I would carry forward

A few things from this build that I will keep doing:

• Brainstorm before you build: Asking the model to interview you back, with a 95% confidence target, saves a lot of rework downstream. This has always worked for me!
• Let it disagree with you: Mapbox over Google Maps, daily bands over raw sawtooth points, Cloudflare over GitHub Actions. None of those calls were mine. However, make sure your fundamental knowledge is strong enough to nudge it in the right direction.
• Iterate from screenshots: A picture of what looks wrong plus one question is usually enough to move forward. I also use playwright mcp to let Claude test the UI (though they at times consume a lot of tokens).

Models

Below are the models and their quantization that I used for benchmarking:

• All four models were run with thinking mode enabled and on April 3rd, 2026
• I used Unsloth GGUFs model versions on llama.cpp.

Standard Benchmarks with llama-bench

llama.cpp has a llama-bench utility that runs standard prefill and generation (decode) benchmarks. It is a quick way to get raw throughput numbers..

This is the command I used to run the benchmarks:

./llama.cpp/llama-bench \
  -m $MODEL_PATH \
  -ctk q8_0 -ctv q8_0 -fa 1 -b 2048 -ub 512 \
  -p 512,2048,4096,8192,16384,32768,65336 \
  -n 128 -r 3 -o md

Agentic Coding: One-Prompt Test

The llama-bench numbers tell you how fast tokens move (sort of the max limit we can expect) but they say nothing about how a model actually performs in reasoning, tool calls, writing code and actual speed with coding assistants.

To test it, I ran a simple practical test: give the model one prompt and see if it can figure out the rest on its own. There will be no hand-holding and multi-turn guidance. The idea is to see how the model performs in such a scenario.

This is not a formal test. It is two prompts at different complexity levels to see how well the model handles mult-step workflows. This is usually the case with most of us, we describe what we want in a single prompt and let the model do its thing.

use context7 to look up the httpx library docs. then write me a python script
that fetches the post from https://jsonplaceholder.typicode.com/posts/1 and
prints the title. also write a pytest test for it, no mocks, hit the real API.
use uv run to run everything so we don't install anything in the current
environment. run the test and make sure it passes.

use context7 to search for the latest google gemini image generation API docs.
I want you to write a python script that uses the google-genai SDK to generate
images using the gemini-3.1-flash-preview model (nano banana). use TDD
red-green methodology, write failing tests first then make them pass. do not
use any mock tests. use uv run to run everything so we don't install anything
in the current environment. test the script and if it works fine and generates
an image, then use this script to run image generation on the five prompts
given in prompts.json. save the images to an images folder, make sure the
folder exists, if it doesn't then create it.

Comparing the Performance

Appendix

llama-server \
    --model $MODEL_PATH \
    --jinja \
    --host 100.80.101.103 \
    --port 8001 \
    --parallel 1 \
    --batch-size 2048 \
    --ubatch-size 512 \
    --cache-type-k q8_0 --cache-type-v q8_0 \
    --flash-attn on \
    --context-shift \
    --metrics

--ctx-size 256000     # MoE (26B-A4B)
--ctx-size 65336      # Dense (31B) - reduced due to VRAM constraints
--temp 1.0
--top-p 0.95
--top-k 64
--min-p 0.00

--ctx-size 200000     # MoE (35B-A3B)
--ctx-size 130672     # Dense (27B)
--temp 0.6
--top-k 20
--chat-template-file $TEMPLATES_DIR/qwen35-chat-template-corrected.jinja
--chat-template-kwargs '{"enable_thinking":true}'

sudo apt-get update
sudo apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev libssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && git pull origin master && cd ..
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF \
    -DGGML_CUDA=ON \
    -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j24 --clean-first \
    --target llama-cli llama-mtmd-cli llama-server llama-gguf-split llama-bench
cp llama.cpp/build/bin/llama-* llama.cpp

export const TodoFixPlugin = async (ctx) => {
  return {
    "tool.execute.before": async (input, output) => {
      if (input.tool === "todowrite" && typeof output.args.todos === "string") {
        output.args.todos = JSON.parse(output.args.todos)
      }
    }
  }
}

My Setup

• RTX 4090 GPU server running Ubuntu with CUDA installed
• Tailscale set up on the server and all my devices (phone, laptop)
• gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf from the Unsloth HuggingFace repo. If you also want vision/image support, use mmproj-BF16.gguf from the same repo.
• The Q4_K_XL quant fits on a 4090 even with full 256K context at approximately 20.5 GB VRAM

1. Build llama.cpp

sudo apt-get update
sudo apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev libssl-dev -y

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

cmake . -B build \
    -DBUILD_SHARED_LIBS=OFF \
    -DGGML_CUDA=ON \
    -DLLAMA_CURL=ON

cmake --build build --config Release -j$(nproc) --clean-first \
    --target llama-server

Verify OpenSSL is linked:

ldd build/bin/llama-server | grep -i ssl
# Should show: libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3

Note: OpenSSL is needed because the MCP proxy makes HTTPS calls to external servers (like Exa for web search). Without it, you’ll get a 500 error about CPPHTTPLIB_OPENSSL_SUPPORT not being defined.

2. Create the MCP Config

Before launching the server, create a config file that sets up web search and a system prompt with today’s date.

The systemMessage is important. I found that without it, the model usually won’t initiate a web search on its own when you ask for current information or facts. It just responds with its training data. The system prompt with today’s date nudges it to actually use the search tools.

Create ~/MODELS/templates/llamacpp-webui-chat-template.json:

{
  "systemMessage": "You are a helpful assistant. Today's date is {{DATE}}. When the user asks for current or recent information, use the available search tools to find up-to-date answers rather than relying on your training data.",
  "mcpServers": [
    {
      "url": "https://mcp.exa.ai/mcp?exaApiKey={{EXA_API_KEY}}",
      "name": "exa",
      "useProxy": true,
      "enabled": true
    }
  ]
}

I use Exa because they give you 1,000 free searches per month (no credit card required). You can get your API key at dashboard.exa.ai. Other web search MCP options:

• Brave Search MCP
• Tavily

3. Create the Launch Script

I use a wrapper script that injects today’s date and the API key into the config, then starts the server. This way the date stays fresh on every restart.

#!/bin/bash

# export EXA_API_KEY="" or source from bashrc/zshrc
TEMPLATE_FILE=~/MODELS/templates/llamacpp-webui-chat-template.json
CONFIG_FILE=~/MODELS/templates/temp-$(basename $TEMPLATE_FILE)
HOSTNAME="<YOUR_TAILSCALE_IP>"
PORT=8001
CONTEXT_SIZE=65536

sed -e "s/{{DATE}}/$(date +%Y-%m-%d)/" \
    -e "s/{{EXA_API_KEY}}/$EXA_API_KEY/" \
    "$TEMPLATE_FILE" > "$CONFIG_FILE"

# start the server
MODEL_PATH=~/MODELS/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf
MMPROJ_PATH=~/MODELS/unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf

./llama.cpp/build/bin/llama-server \
    --model $MODEL_PATH \
    --mmproj $MMPROJ_PATH \
    --jinja \
    --host $HOSTNAME \
    --port $PORT \
    --ctx-size $CONTEXT_SIZE \
    --parallel 1 \
    -ngl 999 \
    --batch-size 2048 \
    --ubatch-size 512 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64 \
    --cache-type-k q8_0 --cache-type-v q8_0 \
    --flash-attn on \
    --context-shift \
    --metrics \
    --webui-mcp-proxy \
    --webui-config-file "$CONFIG_FILE"

This assumes Tailscale is already set up on your system. Replace <YOUR_TAILSCALE_IP> with your server’s Tailscale IP (find it with tailscale ip -4).

What the key flags do:

Start the server:

bash ~/start-server.sh

Note: The full 256K context size works on a 4090 with Q4_K_XL, but I don’t think it’s needed for chat. I usually run with 64K or 128K.

4. Connect from Your Devices

Open a browser on your phone or laptop and go to:

http://<YOUR_TAILSCALE_IP>:8001

5. Verify Web Search Is Working

The MCP config should be loaded automatically, but it’s worth verifying:

1. Open the web UI and go to MCP server settings
2. You should see the Exa entry already configured and enabled
3. Send a message like “What happened in tech news today?”
4. The model should trigger a search tool call and cite results

To confirm tools are being sent, open browser DevTools and go to Network tab, send a message and click the completions request. Check the payload for a tools array.

Note: If the model says “I can’t search the web” or “my knowledge cutoff is January 2025”, the MCP toggle may have auto-disabled itself. Edit the MCP entry in settings and flip the toggle back ON.

6. Enable Vision (Optional)

The launch script in Section 3 already includes the --mmproj flag, so vision is enabled by default. If you don’t need it, remove the --mmproj $MMPROJ_PATH line from the script.

The web UI will automatically show an image upload button when vision is enabled.

Note: The BF16 projector is ~800MB-1GB on GPU. If VRAM is tight, add --no-mmproj-offload to keep it on CPU (slightly slower image processing but saves VRAM).

Troubleshooting

Step 1: Build llama.cpp on your GPU machine

sudo apt-get update
sudo apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j4 \
    --clean-first \
    --target llama-cli llama-mtmd-cli llama-server llama-gguf-split

Use -j4/-j2/-j8 instead of -j to limit parallel jobs and avoid OOM errors during compilation.

Step 2: Install Tailscale on both machines

If you are running everything on the same machine, you can skip this and just use 127.0.0.1.

# Install Tailscale
curl -fsSL https://tailscale.com/install.sh | sh

# Start with SSH enabled and authenticate
sudo tailscale up --ssh

# Enable on boot
sudo systemctl enable tailscaled

# Check your IP and hostname
tailscale status

# Install via Homebrew (or get it from the Mac App Store)
brew install --cask tailscale

# Open the app, log in from the menu bar icon, then:
sudo tailscale up --ssh

Step 3: Download the Qwen3.5-27B GGUF model

mkdir -p ~/MODELS
cd ~/MODELS
uv run --with huggingface_hub[cli] hf download unsloth/Qwen3.5-27B-GGUF \
    --local-dir unsloth/Qwen3.5-27B-GGUF \
    --include "*mmproj-F16*" \
    --include "*UD-Q4_K_XL*"
cd -

uv run ensures you don’t need to install huggingface_hub[cli] into your venv separately.

Step 4: Test the llama.cpp server locally

Start the server on localhost first to make sure everything works:

QWEN35_27B_MODEL_PATH=~/MODELS/unsloth/Qwen3.5-27B-GGUF
./llama.cpp/build/bin/llama-server \
    --model $QWEN35_27B_MODEL_PATH/Qwen3.5-27B-UD-Q4_K_XL.gguf \
    --mmproj $QWEN35_27B_MODEL_PATH/mmproj-F16.gguf \
    --host 127.0.0.1 \
    --port 8001 \
    --ctx-size 16384 \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00

Test it from another terminal:

curl http://127.0.0.1:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-no-key-required" \
  -d '{
    "model": "Qwen3.5-27B",
    "messages": [
      {"role": "user", "content": "What is 2+2?"}
    ]
  }' | python3 -m json.tool

Step 5: Start the server on the Tailscale IP

Now start the server bound to your Tailscale IP so it is accessible from your MacBook:

QWEN35_27B_MODEL_PATH=~/MODELS/unsloth/Qwen3.5-27B-GGUF
TEMPLATES_DIR=~/MODELS/templates
./llama.cpp/build/bin/llama-server \
    --model $QWEN35_27B_MODEL_PATH/Qwen3.5-27B-UD-Q4_K_XL.gguf \
    --jinja \
    --chat-template-file $TEMPLATES_DIR/qwen35-chat-template-corrected.jinja \
    --host <YOUR_GPU_SERVER_IP> \
    --port 8001 \
    --ctx-size 65536 \
    --parallel 1 \
    --batch-size 2048 \
    --ubatch-size 512 \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --cache-type-k bf16 --cache-type-v bf16 \
    --flash-attn on \
    --context-shift \
    --metrics \
    --chat-template-kwargs '{"enable_thinking":true}'

• Replace <YOUR_GPU_SERVER_IP> with your GPU server’s IP (Tailscale IP if remote or 127.0.0.1 if local). Check with tailscale status.
• I would recommend starting with a smaller --ctx-size (eg 16384) first to verify everything works. The server starts faster with less KV cache allocation so you can catch misconfigurations quickly. Once confirmed, restart with your target context size.
• The sampling parameters (--temp 0.6, --top-p 0.95, --top-k 20) are the recommended values from Qwen3.5 for thinking mode with precise coding tasks.
• I chose --ctx-size 65536 because at this context length the total VRAM usage sits around 22 GB (includig model) on a 24 GB card. I could probably go higher by 10k but this leaves enough breathing room to avoid OOM on longer prompts or during prefill spikes.

curl http://<YOUR_GPU_SERVER_HOSTNAME>:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-no-key-required" \
  -d '{
    "model": "Qwen3.5-27B",
    "messages": [
      {"role": "user", "content": "What is 2+2?"}
    ]
  }' | python3 -m json.tool

Step 6: Add the provider in OpenCode

Update ~/.config/opencode/opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "llama-local": {
      "name": "Llama.cpp (RTX4090)",
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "http://<YOUR_GPU_SERVER_IP>/v1"
      },
      "models": {
        "unsloth/Qwen3.5-27B-GGUF": {
          "name": "Qwen3.5-27B Q4_K_XL"
        }
      }
    }
  }
}

If everything worked, you should see the local model available in OpenCode’s model selector:

Trying it out

I ran through a few prompts in OpenCode using the Qwen3.5-27B model to see how well it handles agentic coding tasks, tool calls and skills:

1. I ask it to write a Python script for Gemini image generation, pointing it at context7 to fetch the latest docs
2. The model initially uses Gemini 2.5, so I tell it to switch to the 3.1 image generation model and it updates the script
3. I run the script with uv run to generate an image of a cat on a window sill
4. I use /explain-code (a custom skill) to have the model explain the generated script
5. Finally, I ask it to save the explanation as a readme

The model handles all of this well. It picks up the tool calls, uses the skills correctly, follows up on corrections and produces working code. Honestly, for a 27B model running quantized on a single 4090, the quality is surprisingly good.

For reference, here are the speeds I am getting across some of the sessions:

Using it with Codex

This setup also works with Codex. Add this to your ~/.codex/config.toml (refer to this thread for more details):

[model_providers.llama_cpp]
name = "llama_cpp API"
base_url = "http://<YOUR_GPU_SERVER_IP>:8001/v1"
wire_api = "responses"
stream_idle_timeout_ms = 10000000

[profiles.gpt-oss]
model = "gpt-oss"
model_provider = "llama_cpp"
web_search = "disabled"

Then start Codex with:

codex -p gpt-oss

Reasoning and things I learned along the way

Some of the choices I made and what I picked up in the process.

• Run inference on a separate machine if you can. You don’t need two machines for this but if you have a personal GPU workstation or a Mac Mini, I would recommend running the model there instead of on your daily use machine. Running inference on your laptop eats into your available RAM, it drains your battery fast and the laptop starts heating up bad.
• Why llama.cpp? I started with the Unsloth guide which uses llama.cpp with the GGUF format. Since I am setting this up locally for myself, llama.cpp felt like an easier choice than vLLM.
• Why Qwen3.5-27B over 35B-A3B? The MoE variant is 3-5x faster (~60-100 tok/s) because only 3B parameters are active per token but the 27B has all 27B parameters active and consistently scores higher across benchmarks. For coding tasks, I preferred quality.
• Why UD-Q4_K_XL quantization? Unsloth’s Dynamic 2.0 quantization selectively upcasts important layers to 8 or 16-bit precision, so you get better quality without paying the full VRAM cost of a higher quant. Benjamin Marie’s benchmarks show UD-Q4_K_XL stays within a 1-point accuracy drop of the original while being ~8GB smaller than comparable quants.
• Hybrid architecture and KV cache. Qwen3.5 uses a Gated DeltaNet + Gated Attention hybrid architecture. Only every 4th layer has standard attention (16 out of 64 for 27B) and the rest use DeltaNet which maintains a fixed-size state regardless of context length. This makes the KV cache dramatically smaller than a pure transformer of the same size which is why 64K context fits on a 24 GB card at all.
• KV cache type. Qwen3.5 is trained in bfloat16, so bf16 is a better choice than llama.cpp’s default f16 given it has a better dynamic range. This r/LocalLLaMA discussion mentions that q8_0 doesn’t seem to hurt quality too much but I haven’t tested it myself and decided to go with the safe option of bf16.
• Start with a small context size. Begin with --ctx-size 16384 to verify everything works (correct IP, template path, model loading) before committing more VRAM. The server starts faster with a smaller KV cache, so you can iterate quickly on configuration issues.
• Use -j4 instead of -j when building llama.cpp. The -j flag without a number spawns as many parallel compiler processes as it can. This can lead to an OOM kill (Error 137). Limiting to -j4/2/8 depending on your available RAM avoids this.
• Use uv run for one-off CLI tools. uv run --with huggingface_hub[cli] lets you run hf download without installing the package into your venv. It keeps your environment clean.
• The chat template fix is critical for OpenCode/Codex. The default Qwen3.5 template throws a 500 error when OpenCode or Codex sends messages where the system message isn’t strictly first. The corrected template removes this restriction. Without it, the server will reject most agentic tool-calling prompts.
• Use Context7 with local models. Smaller models due to the size are more likely to hallucinate APIs or use outdated syntax. They also rely much more heavily on the context you give them. Using Context7 to inject up-to-date documentation into the prompt makes a noticeable difference in code quality.

The GPU Memory Hierarchy

It is helpful to have a basic understanding of GPU memory hierarchy before diving into FlashAttention. A GPU has two levels of memory that matter here:

• HBM (High Bandwidth Memory): It is the GPU’s main (slow) memory which you see when you run nvidia-smi. It sits off-compute chip. An A100 has about 80 GB of HBM with a bandwidth of ~2 TB/s.

• SRAM (Static RAM) is on-chip memory. A A100 has about ~20 MB total (spread across 108 SMs) with ~19 TB/s bandwidth. This is roughly 10x the bandwidth of HBM but nearly 4000x smaller in capacity.

These numbers scale with each generation: an H100 SXM has 80 GB HBM3 at 3.35 TB/s, a B200 pushes to 192 GB HBM3e at 8 TB/s but the SRAM-to-HBM bandwidth gap persists across all of them. The principles apply regardless of which GPU you are on.

Every GPU kernel (operation) must load its inputs from HBM into SRAM, do the computation and write results back to HBM. The key intuition: SRAM is where compute happens. HBM is where data lives.

Standard Attention and Its IO Cost

Now that we have the memory hierarchy picture, we can revisit the standard attention and forward and backward pass implementation.

What FlashAttention Does

FlashAttention is an IO-aware, exact attention algorithm. It computes the same output as standard attention but it never materializes the attention matrix in HBM. The total HBM IO drops from to , where is the SRAM size per SM. For typical values of and , this is dramatically less than .

The algorithm achieves this through five interdependent fixes:

1. Operator fusion: run the entire attention computation (matmul, softmax, matmul) in a single kernel so intermediates stay in SRAM.
2. Tiling: partition , , into blocks that fit in SRAM, processing one block at a time.
3. Recomputation: do not store the probability matrix . Recompute it from , , and a tiny normalization constant during the backward pass.
4. Online softmax: compute softmax incrementally across tiles using running statistics so that tiling produces the exact result without ever seeing the full row.
5. The logsumexp value : a single scalar per query row that encodes everything needed to recover during the backward pass.

These fixes are not independent. Fusion needs tiling (intermediates are too large for SRAM otherwise). Tiling needs online softmax (softmax is not directly associative). Recomputation needs (the backward pass must recover without storing it). I will walk through each fix in turn, building up the complete algorithm.

Left: GPU memory hierarchy. Center: FlashAttention tiled computation with Q, K, V blocks loaded into SRAM. Right: FlashAttention fuses all attention ops into a single kernel, eliminating separate HBM round-trips. (Source: Dao et al., 2022)

Wrapping Up

The core insight behind FlashAttention is not about clever matmul tricks. It is about bytes and minimizing the data movement. Standard attention is slow because it moves too much data through the memory bus: the attention matrix gets written to and read from HBM multiple times across three separate kernel launches.

The five fixes of FlashAttention addresses that bottleneck systematically:

• Operator fusion eliminates HBM round-trips by running the entire attention computation in a single kernel.
• Tiling breaks the computation into SRAM-sized blocks so that fusion is actually possible.
• Recomputation removes the need to store the probability matrix , trading cheap extra compute for a massive reduction in activation memory.
• Online softmax makes tiling exact by maintaining running statistics that converge to the correct answer in a single pass.
• The logsumexp value bridges the forward and backward passes, compressing everything needed to recover into one scalar per query row.

The result is an attention algorithm with the same numerical output, dramatically less HBM IO and less memory. It is faster despite doing slightly more work because the bottleneck is memory, not compute.

References

The Tools

Here is what is there so far, grouped by category. All tools support drag-and-drop for file input and run entirely in the browser (nothing leaves your machine).

Image Tools

• Image Format Converter: Convert between PNG, JPEG, and WebP with transparency handling.
• Image Resizer: Resize imagesby by dimensions, aspect ratio and percentage.
• Image Cropper: Interactive crop with draggable selection and aspect ratio presets.
• Binary Mask Creator: Paint binary black and white masks for AI inpainting and fill models.
• Image Adjust & Transform: Flip, rotate, invert, grayscale, brightness, and contrast adjustments.
• SVG Viewer & Converter: Preview SVGs, inspect metadata, export to raster formats.

I find the Binary Mask Creator pretty handy, especially when working with inpainting models like FLUX Fill or Ideogram v3. Creating clean masks is usually a bit of a pain using some clunky web app. The SVG Viewer is another one I reach for often to preview and convert SVGs on the go (saves me time).

Binary Mask Creator and SVG Viewer & Converter

How I Built Them

Nathan Lambert wrote a good piece on how Claude Code hits different and Sergey Karayev put it well when he said Claude Code with Opus is “moving software creation from an artisanal, craftsman activity to a true industrial process”. I think that captures it. The bottleneck is no longer writing the code, it is planning, designing and validating.

When I was building these tools, most of my time went into two things. First, planning out what tools to build with focus on being simple, standalone HTML files that serve my purpose. I would plan everything out, write a detailed plan of tools description and then fire multiple sub-agents in Claude Code to build them in parallel. Second, doing that last ~10% of the work where you polish things according to your needs, making sure there are no missing libraries, no edge cases falling through, testing each tool to make sure it actually works as expected.

I did not sit down and write HTML or JavaScript. I spent my time figuring out what these tools should do and then verifying that they do it correctly. Without Claude Code I would not have attempted this not because the tools are complex but because HTML and JavaScript are not what I work in daily. With Claude Code, I could just focus on the requirements and planning part, ensure a correct test path, and let it handle the implementation.

Try Them Out

All tools are at aayushgarg.dev/tools. If you have ideas for useful tools, feel free to reach out.

Quick Recap on GRPO

GRPO (Group Relative Policy Optimization) is a RL algorithm that eliminates the need for a separate critic/value model by using group-relative advantages. It generates multiple candidate outputs per prompt, scores them and normalizes rewards within the group to get advantages. It also uses ratio clipping (similar to PPO) to prevent the policy from drifting too far from the reference distribution. Instead of imitating expert reasoning traces (SFT), it lets the model discover its own strategies by generating multiple candidate solutions per problem, scoring them and reinforcing the better ones.

You can read more about it in the GRPO derivation blog post I wrote a few weeks back and in this blog post.

Building the Training Loop

I followed the same approach as I mentioned in the assignment whiere I wrote and tested the helper functions first and made sure each piece works in isolation. Finally, wired all the helper functions together into the full training loop.

The GRPO algorithm has two nested loops:

• Outer loop: sample a batch of prompts, generate G rollouts per prompt via vLLM, compute rewards, normalize advantages within each group
• Inner loop: policy gradient updates over the rollout batch (using gradient accumulation to fit on the GPU)

Note, I have made sure to reuse the functions, data pipelines etc. from the SFT code wherever possible.

For the training data, I used the MATH dataset (problems only, no reasoning traces) sourced from this CS336 MATH dataset repo.

The core GRPO functions (group-normalized rewards, four loss variants, microbatch train step) live in utils/grpo.py and the main training script that wires the outer and inner loops together is train_grpo.py. You can find all the key files and details in the GRPO README.

A few important notes:

• I used the same vLLM colocate setup and workarounds as I did for SFT (see the SFT blog post for the debugging details). This allowed me to run the training loop and intermediate evaluations and rollout generations on a single GPU.
• All training configs are managed via OmegaConf structured configs and yaml files. This is especially useful for ablation studies where each experiment config is a minimal diff from the defaults, making it easy to see exactly what changed and importantly to reproduce any run.
• As usual, I added intermediate evaluations, model checkpointing at configurable intervals, wandb logging, other eval and timging metrics for better observability.

To keep costs manageable, all intermediate evaluations use a 1024 examples subset of the validation set rather than the full ~5K.

GPU Memory Optimization (Fitting on 24 GB)

I wrote the training code and tested it on my personal RTX 4090 with 24 GB of VRAM. With the default parameters suggested in the assignment, I immediately ran into OOM errors and had to do some memory optimizations to fit the training loop.

• Peak memory tracking: Not an optimization in itself but an essential step to keep track of the peak memory usage. I made sure to log the peak memory at important junctures in the training loop.
• Gradient checkpointing: The simplest one is to enable gradient checkpointing which recomputes activations during backward pass instead of storing them. You can trade up to ~30% memory savings for speed.
• vLLM sleep mode: This is quite a nice trick to offload vLLM KV cache and weights to CPU during the training phase (when vLLM is not generating). This frees GPU memory for the backward pass and prevents the vLLM cache from competing with training activations.
• 8-bit AdamW: I added an option to use bitsandbytes AdamW8bit optimizer instead of the default AdamW optimizer. This reduces the optimizer state memory by almost half.

After all the above optimization tricks, I was able to successfully train with rollout_batch_size=256, group_size=8, gradient_accumulation_steps=256 (microbatch size = 1) on 24 GB.

You should be able to run most of the ablation studies on your local RTX 4090/3090 GPU with the default configs and optimization flags.

Scaling to Modal (H100s)

The local training script runs fine on a 24 GB GPU (e.g. RTX 4090) but there are two practical limitations that made me want to scale to Modal:

1. Speed: Even with all memory optimizations, a single experiment on the 4090 takes a few hours. Some of the ablation studies and hyperparameter searches become impractical. It always helps to run the experiments on a larger GPU for faster iterations.
2. Parallelism: I wanted to run a lot of ablation experiments. On a single local GPU that means running them one at a time which would take weeks. I needed a way to fire off multiple experiments in parallel and compare them side-by-side in wandb. At the same time, I am not spending my full time on this and I work on it whenever I get time. Thus, I did not want to deal with spinning up and shutting down GPU instances each time.

Modal lets you define GPU workloads purely in Python (no Docker/Kubernetes), with pay-per-second billing and containers that spin up in seconds. I could fire off multiple H100 runs in parallel and everything scales back to zero when done.

Ablation Studies

I ran a series of ablation studies as per the assignment to understand what matters in GRPO training. Each ablation isolates one design choice while keeping everything else fixed.

You will notice the experiments vary in length: some run for 200 GRPO steps, some for 100/50 or some were terminated mid-way. This is intentional and done to keep the total cost manageable while still getting the information I needed.

Summary and Key Takeaways

Best configuration:

• on-policy (epochs_per_rollout_batch=1)
• lr=3e-5
• loss_type=grpo_clip
• use_std_normalization=True
• r1_zero structured prompt
• base model (no SFT initialization)

Best performance: ~0.75 on MATH validation (up from ~3% base model accuracy).

Key lessons:

• Eval reward accuracy is the north star metric. It directly measures what we care about. Especially in the case of reinforcement learning with verifiable rewards.
• However, gradient norm and mean_response_length are the two other important metrics to watch. They are the early warning signals for instability and reward collapse.
• Binary math reward is robust to some design choices (length normalization) but sensitive to others (baseline subtraction, learning rate).
• On-policy training wins in this case, reusing rollouts introduces policy drift that is not worth the compute savings.
• The R1-zero structured prompt matters. A dedicated reasoning scratchpad produces sharper final policies and higher accuracy.
• SFT initialization does not help in this case.

Ideally, I should now try to push the accuracy further by training for longer, using curriculum strategies or modifying the GRPO loss itself. But that’s for another time! I think it is enough learning and compute expense for now.

Resources

Tracking Your AI Usage

If you are using Claude Code you are prone to hitting the session limit more often than not. Thus, knowing how much you are consuming and session limits comes in quite handy. This visibility across claude code and different providers is quite useful.

Managing Your AI Context

One of the biggest challenges with AI assistants is getting the right context in and keeping the context window healthy.

Prefer Skills Over MCP Servers

MCP server tool descriptions consume tokens upfront and often hundreds or even thousands of tokens regardless of whether you actually use them in that session. Skills on the other hand use progressive loading where Claude sees only the name and description (~30-100 tokens) at startup and loads the full instructions only when relevant.

For example, here is the difference in context usage between Playwright as an MCP server versus Playwright as a skill:

Difference in context usage between Playwright as a skill vs Playwright as an MCP server

My recommendation is to use skills whenever possible. For example, Hugging Face Skills and Playwright CLI Skills instead of corresponding mcp servers.

The llm CLI Tool

This is the tool I use a lot in terminal outside of Claude Code itself. Simon Willison’s llm is a command-line tool for interacting with LLMs directly from the terminal. It supports multiple providers and models, stores conversation logs and is endlessly composable with other CLI tools.

I still prefer working in iTerm2. I do not use agentic terminals like Warp. Cursor or claude code are my standard coding assistants but not everything needs a full agent session. For example while I am terminal, you sometimes need to quickly answer a question, run a bash or single command, explain/review a file. These are the tasks where llm comes in really handy. It is fast, intuitive to use and does not consume my Claude Code session limits.

One way I use llm in my day-to-day work is by wrapping it as shell functions in my .zshrc. Each function has a specific system prompt tuned for its task, giving me dedicated LLM-based commands I can use for without thinking. Here are some I use most often:

cmd <query>                       Convert natural language to a shell command
explain <question> [file]         Answer a question, optionally using a file as context
image_qa <question> <image>       Ask a question about an image (vision)
pycode [-x|--exec] <task>         Generate a Python script from a task description
                                    -x  also execute the script via uv run

The pattern I follow for each function is the same: define a model, write a system prompt, wrap llm in a function. For example, here is the cmd function:

CMD_LLM="gpt-5-mini"
CMD_SYSTEM_PROMPT="You are inside a macOS terminal. Output only the raw shell command(s). \
No formatting, no code blocks, no explanations, no extra text."

cmd() {
    if [[ "$1" == "--help" || -z "$1" ]]; then
        echo "Usage: cmd <natural language shell command>"
        echo "  Converts natural language to a raw shell command using LLM."
        echo "  Example: cmd 'find all png files larger than 1MB'"
        return 0
    fi
    llm -m $CMD_LLM -s "$CMD_SYSTEM_PROMPT" "$1"
}

$ cmd "find all png files larger than 1MB"
find . -name "*.png" -size +1M

Upgrading Your Terminal Basics

The tools below are not AI-specific but make my terminal experience much better.

alias lla='eza -alh --git --sort=modified --icons' # list view with hidden files
alias ll='eza -lh --git --sort=modified --icons'    # list view sorted by timestamp
alias lt='eza -lh --git --sort=modified --tree --level=2 --icons' # tree view

Wrapping Up

None of these tools and tips are complex or elaborate and most of them are not even AI-specific. But together they have quietly improved how I work with better visibility into usage, healthier context windows, quicker LLM access from the terminal and a slightly nicer shell experience.

What is Expert Iteration?

Expert Iteration was first introduced by Anthony et. al. (2017) in the context of game-playing AI. The core idea is to alternate between using a slow but powerful expert to find high-quality solutions and training a fast apprentice to imitate those solutions. As the apprentice improves, it provides a better starting point for the expert. Thus, creating a virtuous cycle of improvement.

The Self-taught Reasoner (STaR) uses Expert Iteration idea for improving reasoning capabilities. It works as follows: at each iteration, we prompt the model to generate rationales for many problems, filter to keep only those that lead to correct answers, finetune on the filtered set and repeat.

Here, the filtering step acts as the “expert”. It identifies which of the model own generations are correct. The model then learns to imitate its own best behaviors.

I see Expert Iteration for LLMs as SFT on self-generated, filtered data, repeated iteratively as shown in this loop:

Note: STaR paper retrains from the base pretrained model at each iteration to avoid overfitting and introduces rationalization for failed problems. The approach used here (following the CS336 assignment) does not uses rationalization and continues training from the previous iteration checkpoint.

A key advantage of expert iteration in comparison to SFT is we do not need human or other LLMs generated reasoning traces. The model generates its own reasoning trace and correctness filtering ensures quality. This makes Expert Iteration particularly attractive for domains with verifiable rewards like math or code.

Experimental Setup

I use the same model and evaluation setup as my previous SFT experiments. Looking at the Expert Iteration diagram above, you can see how straightforward it is to extend an SFT to Expert Iteration codebase. You basically just wrap the training loop with generation and filtering steps with other minor changes.

Key Hyperparameters

The Expert Iteration loop has three main key params:

Parameter	Symbol	Description
batch_per_ei		number of questions sampled per iteration
num_rollouts		number of outputs generated per question
num_ei		number of expert iteration steps (fixed at 5)

Each iteration samples questions, generates candidate solutions per question (giving total rollouts), filters for correct answers and finetunes on the filtered set.

I also use an adaptive learning rate and batch size scheme that scales based on the number of filtered examples per iteration. However, more on that in the next section. You can find the full hyperparameters configuration details in the train_exp_iter.py script.

Tuning the Training Setup

Exploring D (Sample Size) and R (Rollouts)

The assignment suggests to vary the batch size and rollout count to understand what configurations work best. I ran a grid of experiments to find the best configuration and see if I could make some sense of the results:

• (batch_per_ei):
• (num_rollouts):

Looking at the results, some observations stand out:

• The best configuration is achieving 41.7% accuracy.

• Increasing does not always guarantee better performance, both configurations underperform their counterparts.

Beyond a certain batch size, sampling more questions with limited rollouts means pulling in harder problems without correct solutions (contributing nothing to training) and a filtered dataset skewed toward easy problems the model can already solve. This results in less diversity per question and degrade performance.

• Increasing from 2 to 4 improves accuracy across all values of , hinting that rollouts matter more than batch size.

More rollouts means higher probability of solving each question, more diverse reasoning traces when you do solve (which we saw helps in the multi-trace experiment) and better coverage of harder problems that rarely get solved with fewer attempts.

This aligns with an observation from the STaR paper where the performance stalled when the model received no direct training signal for problems it failed to solve. The large rollouts helped with training signal from each sampled question.

Pushing Accuracy with Multi-Epoch Training

So far, all my experiments used just 1 epoch of SFT per expert iteration step. I wanted to see if I could push accuracy even higher and the simplest way to check this is by training for more epochs per iteration. I ran additional experiments with 2 epochs per iteration for two batch sizes: and (both with ).

Training for 2 epochs per iteration consistently outperforms 1 epoch across both configurations. My understanding is that with a large rollout count the filtered dataset is already diverse enough that training for multiple epochs extracts more signal without memorizing. This improved learning leads to better filter rates and accuracy in subsequent iterations.

Comparison to SFT Experiments

In my previous SFT experiments, I finetuned the same Qwen2.5-Math-1.5B model using the same train and eval data, with train data being GPT-oss-120B generated reasoning traces. Below is the comparison of SFT and Expert Iteration best experiments:

Despite extensive tuning across batch sizes, rollout counts and learning rate schedules Expert Iteration reached 47.1% accuracy which is around 6% below SFT’s 53.4%.

In my understanding, the gap comes down to two factors: data quality and rationalization:

• SFT trains on reasoning traces from GPT-OSS-120B, a much more capable model. These traces aren’t just correct but also demonstrate sophisticated problem-solving strategies. Expert Iteration by contrast trains on the model own correct outputs which are inherently less refined. The model can only learn from its own best behaviors.
• Moroever, the STaR paper shows that rationalization (providing the answer as a hint to generate rationales for failed problems) provides crucial training signal that accelerates learning. Without it, the model receives no direct signal for problems it cannot solve.

There are still experiments one could run to close this gap like larger rollout counts (8, 12), adaptive sampling strategies to limit correct rollouts per question and prevent overfitting etc. However, I didn’t find it compelling enough to run more experiments.

Also, I have also not come across many experiments/blogs/papers on Expert Iteration replacing SFT and given my experiments, I think there are a few reasons:

1. Expert Iteration requires a model that can already solve some problems correctly. A weaker base model would struggle to provide any training signal.

2. The method is sensitive to learning rate, batch size, and rollout count. You need to find the right balance otherwise you risk overfitting. My grid search showed final accuracy ranging from 21.9% to 47.1% depending on configuration.

3. Each iteration requires generating rollouts before any training happens. With and , that is 4,096 generations per iteration. You end up spending as much or more compute on generation as on training.

Despite all the above, Expert Iteration has one compelling advantage: it does not require expensive annotated data and is well-suited for domains with verifiable rewards like math and code.

It is also worth noting that the core mechanism in expert iterations, generating many outputs and filtering for correct ones is linked to rejection sampling mentioned and used in DeepSeek-Math and DeepSeek-R1 papers. You can think of Expert Iteration as rejection sampling applied iteratively: sample, filter, finetune, repeat.

Resources

What Are Agent Skills?

Simply put, Skills are organized folders that package expertise into discoverable capabilities. Each skill contains a SKILL.md, a markdown file with some YAML metadata, with instructions that Claude reads when relevant. This is along with optional supporting files like scripts and templates.

As Barry described in his AI Engineer talk, think of them as “expertise packages” that Claude can discover and load dynamically.

That is really all there is to skills. There are no complex protocols, no server infrastructure and no elaborate configuration. Just text files that describe how to do something optionally paired with scripts that make the task more reliable. Simon Wilson has rightly mentioned it in his blog post: “Skills feel a lot closer to the spirit of LLMs - throw in some text and let the model figure it out.”

Since Anthropic released Agent Skills as an open standard in December 2025, skills are rapidly becoming available across different coding agents: GitHub Copilot, Codex CLI, Cursor and more.

It is also important to distinguish skills from other customization options like claude.md, MCP servers, and subagents.

This is how I differential them:

• Claude.md gives Claude context about where it’s working
• MCP gives Claude access to data and services
  
• Skills give Claude expertise on how to do things well
• Subagents let Claude delegate work to specialized assistants

How Skills are Organized

The basic structure is surprisingly simple. Every skill follows the same pattern:

my-skill/
├── SKILL.md          # Required: the brain of the skill
├── scripts/          # Optional: utility scripts
│   └── helper.py
│.....                # Other optional files

The only required file is SKILL.md. Everything else is optional and loaded only when needed. It has two parts: YAML frontmatter for metadata and the markdown content for instructions.

For example, here is the frontmatter from my basic image editing skill:

---
name: basic-image-editing
description: Image manipulation tool for resizing, rotation, flipping, cropping, padding, format conversion (JPEG/PNG/WebP/TIFF/HEIC), transparency operations (remove/replace/extract/blend), grayscale conversion, auto-cropping borders, and file size optimization. Use when users need to transform, convert, or optimize images.
---

This is what gets loaded in Claude’s context at startup. You can see the full SKILL.md on GitHub.

After the frontmatter comes the actual instruction content. Your SKILL.md content usually include:

• A brief definition of what the skill does
• Clear instructions on how to use it (commands, workflows, steps)
• Examples showing common use cases with actual command snippets. I find that providing examples in the SKILL.md file really helps Claude understand the correct way to call the scripts.
• Edge cases or constraints if any exist
• References to supporting scripts or templates if the skill uses them

Building Your Own Skill

There are multiple ways to create skills:

1. Write everything from scratch: You have full control. Create the folder structure manually, write the SKILL.md yourself and add your own scripts. > I would not recommend this approach. It is better to use Claude to generate the first version and iterate over it.

2. Use Claude Code with the Skills Creator skill — If you are already working in Claude Code, you can describe what you want and let Claude scaffold the skill for you. See Eleanor Berger’s videos where she walks through an example of building invoice and reports generation skills in Claude Code.

3. Use the Claude web app: This is my preferred approach, especially for personal use. Everything happens in a UI-based interface. You describe what you want, have a conversation with Claude, test it immediately and iterate until it works.

Conclusion

Skills are simple yet powerful. They are simply a folder with markdown file and optionally some scripts that provides domain expertise to your agents without blowing up the context window. If you work with AI agents regularly, building your own skills is worth the investment. Start with something you do repeatedly. Create the simplest version that works and improve it iteratively.

References

• Simon Willison’s Blog Post: A good overview of Claude Skills.
• Barry’s AI Engineer Talk: I highly recommend watching this talk if you are interested in skills.
• Agent Skills Documentation
• Claude Skills Cookbook
• Using Skills with the API
• Agent Skills Open Standard

Key Insights

The Iterative Data Curation Pipeline

As I mentioned earlier, one of the key contributions of this paper is their data curation pipeline that extracts high-quality mathematical content from Common Crawl.

The reason I am going into detail here is that you can draw parallels from this approach to create any other domain-specific dataset from publicly available data like Common Crawl. The pipeline is iterative and uses a fastText classifier at its core. Crudely, this is how the pipeline works:

Conclusion

DeepSeekMath is worth reading for a few reasons.

• First, it shows that a well-curated dataset matters more than model size for math reasoning.
• Second, the data curation pipeline is explained in enough detail that you can adapt it for other domains.
• And thirdly, the discussed ablation studies and experiments are genuinely useful.

I: The PPO Objective and the Critic Problem

Let’s briefly recap the key relevant elements of PPO. For the full derivation and PPO details, see my previous blog on PPO.

PPO optimizes an LLM by maximizing a clipped surrogate objective (constrained using KL regularization):

where is the probability ratio between current and old policies.

The critical component here is the advantage estimate (). The advantage measures how much better (or worse) a specific action is compared to what we expected:

To compute it, PPO uses a value function (baseline) also called the critic) that predicts expected future rewards from any state. The critic is trained alongside the policy and PPO uses Generalized Advantage Estimation (GAE) to compute advantages from per-token value predictions.

II: Replacing the Critic with Group Sampling

As mentioned earlier, in PPO the value function acts as a baseline for advantage estimation: Subtracting this baseline reduces the variance of the policy gradient estimator which in turn stabilizes training.

Key insight: the value function is just one possible choice of baseline. In principle, any function that depends only on the state and not on the action can be used without introducing bias into the gradient estimates.

Common baseline choices are:

• Constant baseline: The average reward across samples. This is the simplest option and is used in vanilla REINFORCE.
• Learned value function: trained alongside the policy as in PPO.
• Monte Carlo estimate: An empirical average of returns computed from multiple samples starting from the same state.

GRPO adopts the third approach. Instead of learning a value function, it directly estimates the expected return using multiple samples.

III: The GRPO Objective

We now have all the pieces needed to construct the full GRPO objective. The construction follows three key modifications:

1. Start with PPO’s clipped surrogate. Recall from Section I that PPO optimizes:

Here is and this clipping mechanism provides a soft trust region that prevents destructively large policy updates.

2. Replace GAE advantage with group-relative advantage. Instead of computing using a learned critic and GAE, we substitute the group-relative advantage from Section II:

This is the key simplification. > We no longer need per-token value predictions. Instead we estimate the baseline directly from sampled completions.

3. Move KL penalty from reward to loss. In PPO, the KL penalty is typically subtracted from the reward signal () before computing advantages:

GRPO takes a different approach by adding the KL divergence directly as a penalty term in the loss function. This is a design choice that simplifies advantage computation since we dont need to consider KL penalties in the baseline estimation.

From DeepSeekMath: “Also note that, instead of adding KL penalty in the reward, GRPO regularizes by directly adding the KL divergence between the trained policy and the reference policy to the loss, avoiding complicating the calculation of .”

IV: KL Divergence in GRPO

The KL Divergence is a measure of the difference between two probability distributions. It is defined as:

It can simply be estimated as:

However this can be negative for individual samples (when ) even though KL divergence is always non-negative. This may lead to high variance in gradient estimates.

GRPO uses an alternative estimator that is both unbiased and guaranteed non-negative:

This estimator can be understood as measuring the gap between and its tangent line at . Since is concave. This gap is always non-negative ensuring the KL penalty never incorrectly suggests that diverging from the reference decreases the penalty.

For a detailed derivation of why this estimator is unbiased and non-negative, see John Schulman’s excellent blog post on approximating KL divergence.

V: Outcome vs. Process Supervision

The GRPO formulation presented so far assumes outcome supervision which provides a single reward at the end of each completion with the same advantage assigned to every token. However for complex reasoning tasks knowing only the final answer reward might not be sufficient.

From DeepSeekMath Paper: “Outcome supervision only provides a reward at the end of each output, which may not be sufficient and efficient to supervise the policy in complex mathematical tasks.”

Process supervision addresses this by providing rewards at the end of each reasoning step. Given a completion with reasoning steps, a process reward model (PRM) assigns rewards at step boundaries where is the end token index of the -th step.

GRPO extends to process supervision with two modifications:

1. Normalize across all step rewards in the group:

where contains all step rewards across all completions.

2. Compute advantages as cumulative future rewards:

This mirrors return-to-go in traditional RL where earlier tokens accumulate rewards from all subsequent steps, while tokens near the end see only remaining rewards.

The DeepSeekMath experiments found process supervision can accelerate learning, though the gap narrows with iterative training. For domains with reliable verifiers (code execution, math answer checking), outcome supervision with RLVR has become dominant. DeepSeek-R1 uses only outcome-level verification.

VI: Connection to REINFORCE Leave-One-Out (RLOO)

GRPO is not the only critic-free algorithm leveraging group sampling. REINFORCE Leave-One-Out (RLOO) takes a similar approach but computes the baseline as the mean reward over all other completions, excluding the current sample:

This “leave-one-out” baseline avoids a subtle correlation that exists when the baseline includes the sample being evaluated.

The two algorithms are conceptually very similar. However, there are some key differences:

GRPO can be understood as inheriting PPO’s clipping mechanism for stability while adopting RLOO-style group sampling to eliminate the critic.

Conclusion

GRPO ingenuity comes from recognizing that PPO value function is fundamentally just a baseline for advantage computation and that an stimate obtained via group sampling can serve the same role. By sampling multiple completions per prompt and using their mean reward as the baseline, GRPO achieves the stability provided by PPO clipped surrogate objective without the memory overhead or training complexity of training a separate critic model.

This design choice is what makes GRPO a preferred approach for RLVR training of LLMs focused on reasoning capabilities.

References

Papers:

• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models: The original GRPO paper
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning: DeepSeek’s reasoning model trained with GRPO
• Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs: The RLOO paper comparing critic-free methods

Blogs:

• John Schulman’s post on approximating KL divergence: Derivation of the unbiased non-negative KL estimator
• RLHF Book by Nathan Lambert: Comprehensive resource on RLHF algorithms including GRPO implementation details
• Understanding GRPO by Cameron R. Wolfe: Deep dive into GRPO mechanics and implementation

I: The RLHF Objective

Let’s recall the RLHF objective from the PPO blog. The goal of RLHF is to find a policy that maximizes expected reward while staying close to a reference model :

The first term encourages the model to generate high-reward responses. The second term (KL penalty) prevents the model from drifting too far from the reference which helps avoid reward hacking and maintains language quality.

As we saw in the PPO blog, we can’t optimize this objective directly with gradient descent because the expectation requires sampling from the policy and sampling is non-differentiable. This is why we needed reinforcement learning algorithms like REINFORCE and PPO. They provide ways to estimate policy gradients without differentiating through the sampling process.

What if we could reformulate the problem so that we don’t need to sample from the policy during training? This is exactly what DPO will achieve.

II: The Bradley-Terry Model for Preference Learning

We also need to understand the Bradley-Terry model for reward model training in a bit more detail with focus on why it works the way it does.

Training a reward model requires human-labeled preference data that compares pairs of responses.

where: - is the prompt - is the preferred (winning) response - is the dispreferred (losing) response

III: Optimal Policy in Closed Form

Here, we will find the exact analytical optimal policy solution to the optimization problem. We want to find the policy that maximizes expected reward while keeping the KL divergence from the reference policy bounded:

Note, I am writing instead of to emphasize that we are looking for the optimal policy in general not just the parameterized version.

Expanding the KL divergence:

So our objective becomes:

For a fixed prompt , we want to find the distribution that maximizes:

This is a constrained optimization problem over probability distributions. We can solve it using the method of Lagrange multipliers, enforcing that sums to 1. For discrete :

Taking the derivative with respect to and setting it to zero (stationary point):

Now, solving for :

The term is a constant (with respect to ) that ensures normalization. To find its value, we enforce that must be a valid probability distribution and sum to 1:

Substituting our expression for :

Since doesn’t depend on , we can factor it out of the sum:

Solving for the constant:

We define this normalizing sum as the partition function :

Substituting back, we get the optimal policy:

We have an exact closed-form expression for the optimal policy. However, we cannot compute it directly because is intractable. To compute it, we need to sum over all possible responses which not possible.

IV: The Reparameterization Trick

The key insight of DPO is to flip the relationship between reward and policy. Now, we frame the problem as: “given an optimal policy what reward function does it correspond to?”

Starting from the optimal policy equation (III.III):

We solve for the reward by first taking the log of both sides:

Now rearrange to get on left-side:

This can be written more compactly as:

The reward is expressed as: - A term involving the log-ratio of the optimal policy to the reference policy - which depends only on (not on )

V: Deriving the DPO Loss

Finally, we have all the pieces to derive the DPO loss. From Section II, the Bradley-Terry preference model is:

From Section IV, assuming we have access to an optimal policy , the reward can be written as:

Substituting this into Bradley-Terry:

Simplifying the expression inside the sigmoid:

The terms cancel:

Now recall the critical insight from Section II where we mentioned that the Bradley-Terry model depends only on reward differences. Thus, when we compute the intractable partition function cancels out. This is what makes DPO possible.

We can write this more cleanly by defining the implicit reward in terms of the optimal policy:

Thus:

We dont actually have access to the optimal policy . But we can parameterize a policy and optimize it to maximize the likelihood of the observed preferences. This is exactly what the reward model loss (II.III) does except now our reward is implicitly defined by the policy itself.

The DPO loss is the negative log-likelihood:

for implicit reward notation, it can be written as:

Some key insights from the above DPO loss:

• The policy implicitly defines its own reward via the log-ratio with the reference. There is no separate reward model.
• This is just a supervised classification loss on preference pairs with no RL.
• DPO uses the fixed preference dataset . Thus, no sampling during training.
• No value function needed since we’re not doing policy gradients
• DPO still optimizes the KL-constrained reward maximization objective but in a different way

VI: Building Intuition for DPO

Now that we have the DPO loss we can build some intuition around the implicit reward model and its gradient updates.

VII: Computing Log Probabilities in Practice

This section is fully adapted from Umar Jamil’s video. I think it is essential to understand how log probabilities are computed in practice.

The DPO loss requires computing , the log probability of a complete response given a prompt . Let’s see how this works in practice with LLMs.

Language models are autoregressive: they generate text one token at a time, conditioning on all previous tokens. For a response , the probability factorizes as:

Taking the logarithm:

The log probability of the full response is the sum of log probabilities at each position.

Here’s how to compute :

1. Prepare input: Concatenate the prompt and response into a single sequence

   

2. Forward pass: Run the transformer to get hidden states at each position

3. Project to logits: Apply the language model head (typically a linear layer) to get vocabulary logits at each position

4. Log softmax: Convert logits to log probabilities over the vocabulary using logsoftmax

5. Gather relevant log probs: For each position in the response extract the log probability of the actual next token (since we know the output)

6. Sum with masking: Sum the log probabilities but only for response tokens (not prompt tokens)

   

This gives us for one response. We do this for both the preferred response and the dispreferred response and we also do it for both the policy model and the frozen reference model . With these four log probabilities in hand, we can compute the DPO loss.

Conclusion

Once you derive the DPO loss, you start appreciating the simplicity and elegance of the solution especially when compared to PPO. The derivation hinges on one observation that the Bradley-Terry model only cares about reward differences and this causes the intractable partition function from analytical solution to cancel out completely. In turn, what remains is a straightforward classification loss.

References

• Papers:
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model: The original DPO paper
  • Training language models to follow instructions with human feedback: The InstructGPT paper that established PPO-based RLHF
• Videos:
  • Umar Jamil’s video on DPO: Excellent walkthrough of the DPO derivation

I: Reinforcement Learning: Core Definitions

Concept	General RL Definition	LLM Context (RLHF)
Reinforcement Learning	A learning setup where an agent learns to act in an environment to maximize expected cumulative reward.	Fine-tuning a language model to generate responses that better match human preferences using reward-based feedback.
Environment	Everything outside the agent that it interacts with and that produces observations and rewards.	The prompt distribution and interaction loop and the reward signal from a reward model evaluating generated responses.
Agent	The learner/decision-maker that observes states, takes actions, and receives rewards.	The language model generating text token by token.
Action ()	A choice made by the agent, usually conditioned on the state .	Picking the next token at each step of generation.
State ()	The information available to the agent at a given time step.	The prompt plus the response generated so far (the current token context).
Reward ()	A scalar signal telling the agent how good or bad an outcome was.	A score from the reward model (trained on preference data) that judges how good or bad a response is.
Policy ()	A stochastic mapping from states to a distribution over actions.	The model’s probability distribution over the next token given the context.
Goal	Find an optimal policy that maximizes expected cumulative reward over time.	Update (align) the model so it tends to generate responses with higher reward-model scores.

II: Reward Model in RLHF for LLMs

A Reward Model (RM) is a neural network that takes a prompt and a response as input and outputs a scalar reward indicating how “good” or “aligned” that response is according to human preferences.

Policy-gradient methods (including PPO) require a scalar objective to update the policy parameters. In standard RL, the environment provides this signal. However for language generation, there is no natural environment giving us rewards for “good” responses. Having humans rate every output is impractical and for gradient-based optimization, we need a differentiable scalar signal to backpropagate through. Thus, we require a cheap, differentiable proxy for human preferences during RL training. A learned RM provides exactly this.

III: Trajectories and Returns

IV: Policy Gradient Optimization and REINFORCE Algorithm

The goal of reinforcement learning is to find a policy that maximizes the expected return over all possible trajectories:

This is our objective function and we want to find parameters such that:

To maximize using gradient-based methods, we need to compute and perform gradient ascent:

This policy gradient looks simple in equation form but it is intractable to compute. The expectation is over trajectories sampled from , which itself depends on . We can’t simply enumerate all possible trajectories. This is computationally intractable for any reasonably sized state-action space (and certainly not possible for LLMs!).

Thus, as a next step we need to derive some sort of reasonable and tractable approximation for . We do this by using the log-derivative trick.

This expectation can be written as an integral:

Bringing the gradient inside the integral:

Now we apply the log-derivative trick:

Rearranging: and substituting back, we get:

which can also be written as the following expectation:

Note, here the gradient is now the expectation of the gradient of the log-probability of the trajectory. This can further be simplified by using the trajectory probability expression (III.I):

Taking the log:

When we take , only the policy term depends on :

The initial state distribution and transition dynamics are independent of , so their gradients vanish. Substituting back, we obtain the policy gradient theorem:

This is a remarkable result. We can compute the gradient of our objective without differentiating through the environment dynamics and only need gradients of the log-probabilities of our policy.

Since we cannot compute the expectation exactly, we approximate it with a sample mean by sampling trajectories:

This gives us the REINFORCE algorithm:

1. Initialize: Start with a pretrained or supervised fine-tuned (SFT) language model

2. Sample prompts: Draw a batch of prompts from a dataset

3. Generate trajectories: For each prompt , generate a response by sampling tokens from the policy . Each trajectory is the sequence of states (prompt + generated tokens so far) and actions (selected tokens).

4. Compute log-probabilities: For each trajectory, compute the log-probability of each generated token given its context:

5. Compute rewards: Score each complete (prompt, response) pair using the reward model:

6. Estimate policy gradient: Compute the gradient estimate using (IV.V):

7. Update policy: Perform a gradient ascent step:

8. Repeat: Go back to Step 2 and iterate until convergence

While REINFORCE provides an unbiased gradient estimate, it suffers from two critical issues that make it impractical for LLM training:

1. High Variance: The gradient estimate suffers from high variance depending on the sampled trajectories. This variance can be large and can lead to noisy gradients and unstable training. > If you look again at (IV.V), the gradient estimate for each action is weighted by the return of the entire trajectory . This means that even if an action was good, it might receive a negative gradient update simply because other actions in the trajectory led to poor outcomes (or vice versa). Over many samples, the noise introduced by this coupling can be substantial, leading to high variance

2. On-Policy Constraint (Sample Inefficiency): REINFORCE requires trajectories sampled from the current policy . Thus after every gradient update, previously collected trajectories must be discarded and new ones need to be sampled from the updated policy. For LLMs, where each trajectory requires a full forward pass through a billion(s)-parameter model, this is prohibitively expensive especially when we need many small gradient steps to train effectively.

V: Reducing Variance and the Advantage Function

The REINFORCE algorithm provides an unbiased gradient estimate (IV.V). However while unbiased, this estimator suffers from high variance.

VI: Importance Sampling and Off-Policy Policy Gradients

Note: In RL literature, “off-policy” typically refers to methods where the behavior policy (generating data) is arbitrarily quite different from the target policy (being optimized) say where transitions from policies thousands of updates old are reused. In this section, what we will call “off-policy” should more precisely be called “local off-policy”.

The advantage-weighted policy gradient (V.IV) requires trajectories sampled from the current policy . … The advantage-weighted policy gradient (V.IV) requires trajectories sampled from the current policy . This creates a fundamental inefficiency i.e., after each gradient update all previously collected trajectories become “stale” and we must discard these trajectories and sample new ones from the updated policy.

For LLMs, where each trajectory requires a full forward pass through billion(s)-parameter model, this is prohibitively expensive especially when we need many small gradient steps to train effectively.

We need a way to reuse the same trajectories for multiple gradient updates. Importance sampling provides the mathematical machinery to do exactly this!

VII: Trust Region Policy Optimization (TRPO)

The CPI objective is attractive because it lets us reuse data via importance ratios, but unconstrained optimization is unstable. When drifts far from , the probability ratios become extreme and the advantage estimates become stale and can be exploited by the optimizer.

The key insight of Trust Region Policy Optimization (TRPO) is that the surrogate objective is only a valid approximation to the true objective within a local neighborhood of . TRPO paper formalized this by proving policy performance is guaranteed to improve as long as the KL divergence between consecutive policies remains bounded. This theoretical result motivates constraining the policy update to stay within a “trust region” where the surrogate objective remains reliable. See the TRPO paper for the formal proof.

TRPO converts this insight into a constrained optimization problem that ensures the policy update stays within a “trust region” where the surrogate objective remains reliable.

The hyperparameter defines the trust region size, the maximum allowed divergence between consecutive policies. This constraint ensures that remains close to 1, keeping our importance-weighted estimates reliable.

Solving (VII.I) requires second-order optimization. TRPO approximates the objective linearly and the KL constraint quadratically (using the Fisher Information Matrix) and then solves the resulting problem via the conjugate gradient algorithm followed by a line search to ensure constraints are satisfied.

For large-scale LLM training, this approach is impractical:

• Computational overhead: Each policy update requires multiple conjugate gradient iterations and line search steps, significantly more expensive than standard gradient descent.
• Memory requirements: Computing Fisher-vector products adds substantial memory overhead for billion(s)-parameter models

The theory behind TRPO also suggests using a KL penalty rather than a hard constraint. It is easier to implement and more computationally efficient.

However, choosing a penalty coefficient that works across different problems or even across different training stages is notoriously difficult. This motivates Proximal Policy Optimization (PPO): a first-order method that achieves TRPO’s stability through a clipped surrogate objective rather than explicit constraints.

VIII: Proximal Policy Optimization (PPO)

Proximal Policy Optimization (PPO) achieves TRPO’s stability guarantees using only first-order optimization. Instead of explicitly constraining the KL divergence, PPO modifies the objective function itself to discourage large policy updates through a clipping mechanism. It implicitly limits how far the policy can move, providing a “soft” trust region using only standard gradient descent.

IX: Complete PPO Objective with KL Penalty

When fine-tuning an LLM with “vanilla” PPO, the policy learns to maximize rewards from the reward model. However, the reward model is an imperfect proxy for human preferences. It is a neural network trained on limited data that can be exploited. Without constraints, the policy may discover adversarial outputs that achieve high reward scores while producing text that:

• Degenerates into repetitive or nonsensical patterns that “fool” the reward model
• Drifts far from natural language, losing fluency and coherence
• Exploits spurious correlations learned by the reward model

This phenomenon is called reward hacking. The policy finds a way to “game” the reward model rather than genuinely improving response quality.

To prevent reward hacking, the InstructGPT paper adds a KL divergence penalty that regularizes the policy to stay close to a reference model (typically the SFT model before RL fine-tuning).

From Section VIII, the PPO objective (to be maximized via gradient ascent) consists of three terms:

Now, we don’t use raw reward model scores directly. Instead, we define a KL-penalized reward that regularizes the policy to stay close to a reference model :

where: - is the reward signal at timestep - is the KL penalty coefficient - is the frozen reference model

At each token position, the KL divergence simplifies to:

In practice we estimate this expectation with the sampled token , yielding:

Note that the reward model produces a single scalar for the complete response . This score is assigned only at the final token , while the KL penalty applies at every token.

The KL penalty serves two purposes: 1. Prevents reward hacking: The policy cannot drift arbitrarily far from natural language 2. Maintains fluency: Outputs remain similar in distribution to the well-trained SFT model

It modifies the advantage estimates used in PPO through the modified per-token rewards. However, it is mathematically equivalent (and more efficient in implementation) to add the KL term directly to the objective. The PPO objective with KL penalty is:

The first term is exactly what vanilla PPO optimizes using the clipped surrogate. The KL penalty term appears as a separate additive component that penalizes divergence from the reference model. Substituting the PPO clipped surrogate for the first term:

Combining all components, the complete PPO objective with KL penalty (to be maximized) is:

Here, each term serves a distinct purpose:

Term	Role
Policy Objective	Improves the policy while preventing destructive updates via clipping
Value Loss	Trains the critic for accurate advantage estimation (subtracted to minimize)
Entropy Bonus	Encourages exploration, prevents premature convergence
KL Penalty	Prevents reward hacking, maintains language quality (subtracted to penalize drift)

It is important to distinguish the two KL-related mechanisms in the complete loss. The PPO clipping mechanism acts as a short-term anchor that constrains how much the policy can change in a single update, while the KL penalty is a long-term anchor that constrains how far the policy can drift from its starting point across all of training.

Finally done…

And that’s the full derivation! What I find satisfying is that every term in the final loss has a specific purpose. Each one exists because we ran into a specific problem along the way and needed to fix it. I will admit it was not easy to understand all the math and concepts behind the loss. I still do not fully understand every detail but I understand it far better than I did a few days ago.

I hope this was useful. If you spot any errors in derivation (which I’m sure there are) or have suggestions, feel free to reach out.

References

• Video:
  • Umar Jamil’s video on RLHF and PPO: A comprehensive and must-watch video covering RLHF and PPO concepts.
• Papers:
  • Proximal Policy Optimization Algorithms: The foundational PPO paper introducing the clipped surrogate objective.
  • Training language models to follow instructions with human feedback: The InstructGPT paper demonstrating PPO with KL penalty to mitigate reward hacking in LLM fine-tuning.
  • Trust Region Policy Optimization: The TRPO paper that motivates the trust region constraints used in PPO.
  • High-Dimensional Continuous Control Using Generalized Advantage Estimation: GAE paper introducing the exponentially-weighted advantage estimator for variance reduction in policy gradients.

What I Built

I loosely followed Stanford’s CS336 Assignment 5 as a guide, wrote all the SFT core components, and ran two sets of experiments:

1. Reasoning SFT: Fine-tuned Qwen2.5-Math-1.5B on math reasoning traces to improve step-by-step problem solving capabilities.

Best: 53.4% reward accuracy (up from 2.9% baseline) with 99.3% format accuracy

Best: GSM8K 16->33%, Safety 62->78%, AlpacaEval 1.6->5.3%, MMLU ~58%

All experiment code, training scripts, and detailed notes are available in my building-from-scratch repo.

Part 1: Reasoning SFT with Qwen2.5-Math-1.5B

The idea behind reasoning SFT is simple. You take a base model that barely outputs correct answers, show it high-quality examples of how to solve problems step-by-step, and train it to replicate/mimic that reasoning process. The model learns to think in a structured format with first generating reasoning inside <think> tags, then outputting the final answer in <answer> tags.

My starting point was Qwen2.5-Math-1.5B, which had quite poor baseline accuracies on the math validation set: ~2.9% for answers and ~14% for format.

The Training Loop: Per-Token vs. Sequence Loss

The original assignment uses sequence level loss normalization where you sum the loss over all tokens in a sequence and normalize by a constant, not by the variable number of tokens.

While running the initial experiments, I noticed the gradient norms were really large values, and training felt unstable. Even though the loss seemed to be going in the right direction, something didn’t feel right. After some investigation, I realized the issue: with variable-length sequences (my training examples ranged from short to quite long), longer sequences contribute more to the gradient than shorter ones. This creates high variance in gradient updates.

Left: Sequence-level loss (high variance gradients) | Right: Per-token loss (stable gradients)

Thus, I added a per_token_loss flag to my training step which when enabled normalizes the loss by the actual number of response tokens in each sequence. The difference was noticeable with subtle improved accuracy. More importantly, the gradients became much more stable with per-token normalization.

Run	Loss Normalization	Reward Accuracy
run_filtered	Per-token	0.5204
run_filtered-res-len	Sequence-level	0.5106

AttributeError: 'LLMEngine' object has no attribute 'model_executor'

ValueError: There is no module or parameter named '_orig_mod' in Qwen2ForCausalLM

Part 2: Instruction SFT with Llama-3.1-8B

With the reasoning SFT working, I moved on to the second part: instruction fine-tuning. This loosely follows the CS336 Supplementary Assignment 5, where the goal is to build a model that can follow diverse instructions and refuse harmful requests.

Unlike reasoning SFT, instruction fine-tuning uses conversational instruction-response pairs. The training data combines UltraChat-200K (diverse multi-turn conversations) and SafetyLlama (safety-focused examples) totaling around 200K examples, formatted using the Alpaca prompt template.

For evaluation, I used four benchmarks as specified in the assignment: - GSM8K: Grade-school math problems (tests math reasoning) - MMLU: Multiple-choice questions across 57 subjects (tests factual knowledge) - AlpacaEval: Open-ended instructions judged by LLM-as-judge (tests instruction-following quality)
- Simple Safety Tests (SST): Harmful prompts to test refusal behavior (tests safety)

# Conservative fix: drop last prompt token to avoid boundary issues
prompt_length = len(prompt_tokens) - 1
labels[:prompt_length] = -100

Conclusion

While writing the SFT code from scratch, I ran into a lot of debugging challenges. It was at times painstaking and frustrating but was also a valuable learning experience. By debugging, I learned a lot about how things work under the hood, and the whole experience prepares you for how to go about debugging code/projects in the future.

I leave you with some of the debugging tips I came across:

• vLLM OOM: Tune max_model_len, max_num_seqs, and gpu_memory_utilization and start conservative.
• Per-token loss: Normalize by response token count to prevent long sequences from dominating gradients.
• torch.compile + vLLM: Access weights via model._orig_mod when loading into vLLM.
• BPE boundaries: Drop last prompt token before masking to avoid tokenization edge cases.
• Data quality matters: Filtering incorrect traces gave me a 10% accuracy boost.
• vLLM version issues: Set VLLM_ENABLE_V1_MULTIPROCESSING=0 if model_executor is missing.

Resources

I have made all the code, datasets, and model checkpoints publicly accessible.

• Code: building-from-scratch/sft
• Datasets: garg-aayush/sft-cs336-assign5-datasets
• Checkpoints:
  • Reasoning:
    • run_all: qwen-2.5-math-sft-all-2epoch
    • run_filtered: qwen-2.5-math-sft-filtered-2epoch
    • run_filtered-res-len: qwen-2.5-math-sft-filtered-res-len
    • run_filtered-2epoch: qwen-2.5-math-sft-filtered-2epoch
  • Instruction d:
    • run_mask: llama31-8b-sft-mask
    • run_nomask: llama31-8b-sft-nomask
• Training logs: wandb/sft and wandb/sft_instruct

Svg2Raster

ComfyUI does not natively support vector graphics like SVGs. I often work with them and needed lightweight nodes to load (SVG->JPEGs/PNGs) and manipulate SVGs in ComfyUI.

Thus, I built Svg2Raster, a small custom node package that makes it easy to use SVGs with other nodes.

Writing your Custom ComfyUI Node

So here I am assuming that you are fairly comfortable using ComfyUI and you already have ComfyUI installed locally on your system/cloud instance.

# Simple SVG read and conversion check
import cairosvg
from PIL import Image, ImageOps
import io

# Read SVG file
with open('logo.svg', 'r', encoding='utf-8') as f:
    svg_text = f.read()

# Basic conversion to PNG
img_bytes = cairosvg.svg2png(bytestring=svg_text.encode('utf-8'), 
                              output_width=600)
img = Image.open(io.BytesIO(img_bytes)).convert('RGBA')
print(f"Image size: {img.size}")

class LoadSVG:
    @classmethod
    def INPUT_TYPES(cls):
        return {
            "required": {
                "svg_file": ("STRING", {"default": "file.svg"}),
            }
        }
    
    RETURN_TYPES = ("IMAGE", "STRING")
    RETURN_NAMES = ("image", "svg_text")
    FUNCTION = "load_svg"
    CATEGORY = "Svg2Raster"
    
    def load_svg(self, svg_file):
        # Your actual logic here
        return (image_tensor, svg_text)

cd ComfyUI/custom_nodes
mkdir svg2raster
cd svg2raster

from .svg2raster_node import *

__all__ = [ "NODE_CLASS_MAPPINGS",
            "NODE_DISPLAY_NAME_MAPPINGS"]

class LoadSVGImage:
    @classmethod
    def INPUT_TYPES(cls):
        """Define what inputs this node accepts"""
        # Use `folder_paths` to access ComfyUI's input directory
        input_dir = folder_paths.get_input_directory()
        files = [f for f in os.listdir(input_dir) if os.path.isfile(os.path.join(input_dir, f)) and f.lower().endswith('.svg')]
        return {
            "required": {
                "svg": (sorted(files), {"image_upload": True}),
            }
        }
    
    # Output configuration
    RETURN_TYPES = ("STRING", "IMAGE")
    RETURN_NAMES = ("svg_text", "preview_image")
    FUNCTION = "load_svg"  # Method name to execute
    CATEGORY = "FromSVG/Tools"  # Menu location
    
    def load_svg(self, svg, background="#FFFFFF"):
        """Main execution method - does the actual work"""
        # Your logic here
        return (svg_text, image_tensor)
    
    @classmethod
    def IS_CHANGED(cls, svg):
        # Returns file hash or modification time
        pass
    
    @classmethod
    def VALIDATE_INPUTS(cls, svg):
        # Check if file exists, return error string if invalid
        return True

def _pil_to_tensor(pil_img: Image.Image):
    """Convert PIL image to ComfyUI IMAGE tensor: (B, H, W, C) in [0,1]"""
    # conversion logic

NODE_CLASS_MAPPINGS = {
    "LoadSVGImage": LoadSVGImage,
}
NODE_DISPLAY_NAME_MAPPINGS = {
    "LoadSVGImage": "Load SVG Image",
}

Baseline: Bigram Model (e0b5864)

Karpathy starts with the simplest possible language model: a bigram model. It predicts the next character based only on the current character. No context at all. The tokens aren’t talking to each other.

This still works somewhat because some characters naturally follow others (the letter ‘q’ is almost always followed by ‘u’). But the output is complete gibberish because the model has no way to look at what came before.

Result: ~2.49 validation loss.

Update 1: Self-Attention (7b0e03a)

We want tokens to communicate with each other and predictions to consider context from previous tokens, not just the current one. A token at position 5 should be able to look at tokens 1-4 and gather information from them. But at the same time, it can’t look at tokens 6, 7, 8 because those are the future we’re trying to predict.

Self-attention solves this. Every token is represented by 3 vectors: - Query: “What am I looking for?” - Key: “What do I contain?”
- Value: “If you find me interesting, here’s what I’ll tell you.”

The query dot-products with all the keys. High dot product means high affinity: “I find you interesting.” The values of interesting tokens get aggregated via weighted sum.

Note: Attention is really a communication mechanism. You can think of it as nodes in a directed graph where every node aggregates information from nodes that point to it. In our case, token 5 can receive information from tokens 1-4 (and itself), but not from tokens 6-8. The triangular mask creates this directed structure and is what makes this a “decoder” block.

One subtle but important point: attention has no notion of space. The tokens don’t inherently know where they are in the sequence. That’s why we add positional embeddings. Each position gets its own learned embedding that’s added to the token embedding, giving the model spatial information.

Result: ~2.4 validation loss. Tokens can now see context.

Update 2: Multi-Head Attention (9d2a7b5)

Tokens have a lot to talk about. One head might look for consonants, another for vowels, another for word boundaries, another for patterns at specific positions. Having multiple independent communication channels lets the model gather diverse types of data in parallel.

Note: This is similar to grouped convolutions. Instead of one large convolution, you do it in groups. With 4 heads of 8 dimensions each, we get the same total dimensionality (32) but with 4 separate communication channels. Each head can specialize in different patterns.

Result: ~2.28 validation loss.

Update 3: Feed-Forward Network (c4c46ff)

The FFN layer addresses a key problem. Until now, “the tokens looked at each other but didn’t have enough time to think about what they found.”

Self-attention is the communication phase. Tokens gather data from each other. But then they need to compute on that data individually. That’s what the feed-forward network does. It operates on a per-token level. All the tokens process their gathered information independently.

So the Transformer block becomes: communicate (attention) → compute (feed-forward). This pattern repeats for every layer.

Result: ~2.27 validation loss. The architecture now has both communication and computation.

Update 4: Residual Connections (0239c07)

This is one of two optimizations that make deep networks actually trainable. Without it, stacking many layers leads to vanishing gradients and optimization difficulties.

Karpathy visualizes it nicely: imagine a residual pathway running from top to bottom. You can “fork off” from this pathway, do some computation, and project back via addition. The path from inputs to outputs is just a series of additions.

Note: Why does this help? During backpropagation, addition distributes gradients equally to both branches. The gradients “hop” through every addition node directly to the input. This creates a “gradient superhighway” from supervision to input, unimpeded. The residual blocks are initialized to contribute very little at first, then “come online” over time during optimization.

Result: ~2.09 validation loss. Now we can stack layers without vanishing gradients.

Update 5 & 6: Layer Normalization (63ef5f8, 4f5bef8)

Batch normalization normalizes columns (across examples in a batch). Layer normalization normalizes rows (across features for each example). The implementation is almost identical, you just change which dimension you normalize over.

Layer norm has advantages for Transformers: - No dependency on batch size (works even with batch size 1) - No running buffers to maintain - No distinction between training and test time

The original Transformer paper used post-layer norm (normalize after attention/FFN). Modern implementations use pre-layer norm (normalize before). Pre-layer norm creates a cleaner residual pathway since the transformation happens on normalized inputs, leading to more stable training.

Result: ~2.076 validation loss.

Update 7: Scaling Up (d4141d7)

With all the architectural pieces in place, Karpathy scales up the architecture:

Dropout is added for regularization. It randomly shuts off neurons during training, effectively training an ensemble of sub-networks. At test time, everything is enabled and the sub-networks merge.

Result: ~1.48 validation loss. The generated text now looks like Shakespeare (structure, dialogue formatting, character names) even though it’s nonsensical when you actually read it.

How This Compares to GPT-3

The architecture we built is essentially the same as GPT-3. The difference is pure scale: 17,500x more parameters trained on 1 million times more data. By today’s standards, even GPT-3’s 300B tokens is considered modest. Current models train on 1T+ tokens.

This is what makes the Transformer architecture so remarkable. The same fundamental design (attention for communication, feed-forward for computation, residual connections, layer norm) scales from a 10M parameter Shakespeare generator to a 175B parameter model!

Resources

• Code: building-from-scratch/basic-gpt
• Video: Let’s Build GPT from Scratch by Andrej Karpathy

Key Takeaways

Task and Dataset

Similar to my previous blogs, the custom task is to predict the structured functional representation from the given text video game opinions of the ViGGO validation dataset.

To make this exercise interesting and challenging for future experiments, I will use a maximum of 1000 examples for fine-tuning any LLM model, instead of the full ~5K train dataset.

Below is an example from the randomly selected 1K train dataset:

Text                      : I remember you saying that you loved The Room. Do you tend to enjoy PC games from 2012?
functional_representation : verify_attribute(name[The Room], release_year[2012], rating[excellent], platforms[PC])

As shown in the graph below, the selected 1K dataset is a fairly representative sample of the full ViGGO train dataset.

Upload the dataset to Predibase

Predibase requires you to upload the instruction fine-tuning dataset in particular format. This is from Predibase docs:

For instruction fine-tuning, your dataset must contain two columns named prompt and completion: - prompt: Your input prompt. It serves as the starting point or the guiding information for the model. - completion: The expected response that corresponds to the input provided in the “prompt” column. - split (optional): Should be either train or evaluation. To learn more, check out this section.

Make sure to add the prompt template to the examples and convert them to the correct format. For this exercise, I use the following prompt template:

prompt_template = """Given a target sentence convert it structured functional representation.

### Target sentence: {text}

### Output Functional representation:
"""

You can connect your dataset to Predibase via the UI or Python SDK. Here, I will upload the dataset using SDK.

# Initialize Predibase client
pb = Predibase(api_token=os.environ["PREDIBASE_API_TOKEN"])

# Upload the dataset
dataset = pb.datasets.from_file("viggo_train_val_dataset_1K.csv", 
                                name="viggo_train_val_dataset_1K")

Once uploade, you can check the uploaded dataset on the Predibase UI.

For detailed steps on uploading the dataset to Predibase, please refer to the companion blog notebook.

Setup and Finetune

Once you have uploaded the dataset, running the fine-tuning process is refreshingly simple. For this example, I fine-tune the base llama-3-8b model with the following parameters: epochs=3, rank=16, and learning_rate=2e-4.

# Create an adapter repository
repo = pb.repos.create(name="viggo-finetune-1K", 
                description="Llama-3-8b adapter repository for viggo 1K examples"
                )

# Create and run the fine-tuning job
adapter = pb.adapters.create(
   config=FinetuningConfig(
       base_model="llama-3-8b",
       epochs=3,
       rank=16,
       learning_rate=0.0002,
   ),
   dataset=dataset,
   repo=repo,
   description="baseline-llama-3-8b",
)

That’s all you need to do to submit a job! Once completed, it will be available on the Predibase platform.

You can always tweak multiple hyperparameters (see Finetuning Config) and run the fine-tune job again. All your fine-tune jobs will be available on the Predibase platform.

Evaluate the Fine-tuned Model

Predibase provides both popular Serverless endpoints and Dedicated deployments options for opens-source LLMs and their fine-tuned LORA checkpoints. I will create serverless endpoint for this case.

Note, atleast for now, serverless deployments are available for free.

# Initialize the Predibase deployment client
lorax_client = pb.deployments.client("llama-3-8b")

# Load the validation dataset
viggo_dataset = load_dataset("GEM/viggo")
val_dataset = viggo_dataset['validation']

# finetuned adapter id
adapter_id = "viggo-finetune-1K/2" 

responses_dict = {}
for idx in range(len(val_dataset)):
    if idx % 50 == 0: print(f"Processing {idx}/{len(val_dataset)}")
    output = lorax_client.generate(prompt_template.format(text=val_dataset["target"][idx]), adapter_id="viggo-finetune-1K/2", max_new_tokens=150).generated_text
    ground_truth = val_dataset["meaning_representation"][idx]
    text = val_dataset["target"][idx]
    responses_dict[idx] = {"output": output, "ground_truth": ground_truth, "text": text}

Improved Performance with Updated Prompt Template

A simple yet effective way to enhance the model’s performance is by refining the prompt template. By providing clearer instructions that convey the structure of the functional representation, we can guide the model to produce more accurate outputs.

I updated the prompt template as follows:

prompt_template = """Given a target sentence construct the underlying meaningful functional representation of the input sentence as a single function with attributes and attribute values.

### Target sentence: {text}

### Output Functional representation:
"""

After uploading this new dataset, finetuning the model, and evaluating it with the new adapter viggo-finetune-1K/3, there is significantly improved evaluation metrics. Notably, the model now surpasses GPT-4o’s scores for exact_match and function_name_match.

This improvement highlights the importance of clear and specific instructions in prompt engineering, even when working with finetuned models.

Conclusions..

• First of all, My overall experience with Predibase has been positive, particularly in terms of rapid finetuning of models. While there are some limitations such as restricted hyperparameter tuning, standardized dataset format, and inability to download adapters in the developer tier, it offers a user-friendly platform for fine-tuning (LORA) large language models. I was able to quickly upload, setut, finetune and infer the llm models.

• I achieve out-of-the-box performance using only random 1K examples. Although the fine-tuned llama-3-8b model doesn’t match the performance of GPT-4 and Sonnet 3.5 on all metrics. This demonstrates the potential of fine-tuning with limited data, highlighting the efficiency of the approach for task-specific model adaptation.

Next steps…

• My next goal is to further enhance the model’s performance on evaluation metrics while maintaining a limit of 1,000 training examples. LIMA: Less Is More for Alignment paper has demonstrated in the past that even 1,000 well-curated examples can lead to strong finetuning performance.
• Careful curated selection of examples and hyerparameters will definitely improve the performance benchmarks on evaluation metrics.
• In addition to it, I will deep dive into one of my favorite LLM fine-tuning tools, Axolotl.

References

• Part II blog post of series
• Notebook I: Analyze_Viggo_Dataset.ipynb
• Notebook II: Finetune_llama-3-8b_Predibase.ipynb
• Predibase Platform
• LORA blog I: Finetuning Large Language Models (LLMs)
• LORA blog II: LLM Research Insights: Instruction Tuning & Training Paradigms
• Llama 3
• LIMA: Less Is More for Alignment
• Axolotl: A Framework for Fine-tuning LLMs

Thanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.

Dataset and Prompt Template

For consistency and fair comparison, I used the same ViGGO validation dataset and the prompt template as in the previous blog post. The prompt template and the few-shot examples are designed to guide the models in generating structured functional representations from the given text input:

PROMPT_TEMPLATE = """
Given a target sentence construct the underlying meaning representation of the input sentence as a single function with attributes and attribute values. 
This function should describe the target string accurately and the function must be one of the following ['inform', 'request', 'give_opinion', 'confirm', 'verify_attribute', 'suggest', 'request_explanation', 'recommend', 'request_attribute'].

The attributes must be one of the following: ['name', 'exp_release_date', 'release_year', 'developer', 'esrb', 'rating', 'genres', 'player_perspective', 'has_multiplayer', 'platforms', 'available_on_steam', 'has_linux_release', 'has_mac_release', 'specifier']. The order your list the attributes within the function must follow the order listed above. For example the 'name' attribute must always come before the 'exp_release_date' attribute, and so forth.

For each attribute, fill in the corresponding value of the attribute within brackets. A couple of examples are below. Note: you are to output the string after "Output: ". Do not include "Output: " in your answer.

Example 1)
Sentence: Dirt: Showdown from 2012 is a sport racing game for the PlayStation, Xbox, PC rated E 10+ (for Everyone 10 and Older). It's not available on Steam, Linux, or Mac.
Output: inform(name[Dirt: Showdown], release_year[2012], esrb[E 10+ (for Everyone 10 and Older)], genres[driving/racing, sport], platforms[PlayStation, Xbox, PC], available_on_steam[no], has_linux_release[no], has_mac_release[no])

Example 2) 
Sentence: Were there even any terrible games in 2014?
Output: request(release_year[2014], specifier[terrible])

Example 3)
Sentence: Adventure games that combine platforming and puzzles  can be frustrating to play, but the side view perspective is perfect for them. That's why I enjoyed playing Little Nightmares.
Output: give_opinion(name[Little Nightmares], rating[good], genres[adventure, platformer, puzzle], player_perspective[side view])

Example 4)
Sentence: Since we're on the subject of games developed by Telltale Games, I'm wondering, have you played The Wolf Among Us?
Output: recommend(name[The Wolf Among Us], developer[Telltale Games])

Example 5) 
Sentence: Layers of Fear, the indie first person point-and-click adventure game?
Output: confirm(name[Layers of Fear], genres[adventure, indie, point-and-click], player_perspective[first person])  

Example 6) 
Sentence: I bet you like it when you can play games on Steam, like Worms: Reloaded, right?  
Output: suggest(name[Worms: Reloaded], available_on_steam[yes])

Example 7)
Sentence: I recall you saying that you really enjoyed The Legend of Zelda: Ocarina of Time. Are you typically a big fan of games on Nintendo rated E (for Everyone)?    
Output: verify_attribute(name[The Legend of Zelda: Ocarina of Time], esrb[E (for Everyone)], rating[excellent], platforms[Nintendo])

Example 8)
Sentence: So what is it about the games that were released in 2005 that you find so excellent?  
Output: request_explanation(release_year[2005], rating[excellent])

Example 9)
Sentence: Do you think Mac is a better gaming platform than others?
Output: request_attribute(has_mac_release[])

Give the output for the following sentence:
{input}
"""

Generating the responses

I used the respective official API calls for the closed models (GPT-4o, Gemini-1.5-flash, Claude-1.5-Sonnet) and the Replicate API client for open-source models (Llama-3-70B, Llama-3-8B, Mistral-8x7B).

For detailed information on API endpoints and the process of generating responses for all models, please refer to the Generate_responses_all_llms.ipynb notebook.

Generating responses for this experiment had the following associated costs:

_*for limited usage_

Evaluation Strategy

To assess the models’ performance, I used the same evaluation criteria as in the previous post:

1. Function Name Match: The function name must match the ground truth function name.
2. Function and Attributes Match: The generated function name and attributes must match the ground truth function attributes. However, the order of the attributes does not matter.
3. Function, Attributes, and Values Match: The generated function name, attributes, and values must match the ground truth function attributes and values. The order of the attributes and values does not matter.
4. Exact Match: The generated function must exactly match the ground truth function.

Note: I implemented custom Python functions using regex and string manipulation to calculate these metrics, rather than relying on another LLM for evaluation. This approach helps avoid potential biases that might be introduced by using an LLM in the evaluation process.

For the complete evaluation code and functions used, please refer to the Compare_models_performances.ipynb notebook.

Comparing the models performance

Based on the evaluation metric plot:

1. Claude Sonnet-3.5 and GPT-4o consistently outperform other models across all metrics.
2. Claude Sonnet-3.5 performs better than GPT-4o, aligning with all the twitter chats about its superior performance.
3. Gemini-1.5-Flash, despite its optimization for speed, maintains competitive performance for less stringent metrics but struggles significantly with exact matches.
4. The performance gap between the top-performing models and others widens sharply for more stringent metrics.
5. As expected, the smaller Llama-3-8B model shows the lowest performance, highlighting the evident size advantage of larger models.
6. Mistral-8x7B's performance is lower than anticipated, suggesting lower instruction-following capabilities.

Some key observations

Based on the quickly eyeballing the generated responses:

1. Almost all models consistently captures straightforward attributes such as player perspective and multiplayer status.
2. For queries involving multiple attributes or conditions, the model sometimes misses or misinterprets parts of the input.
3. The models struggles with capturing subtle distinctions in opinions and inferring information that is implied but not explicitly stated in the text. For example, models struggled to differentiate between inform, give_opinion and suggest.

Conclusions

This comparison provides valuable insights into the capabilities of various LLMs in functional representation extraction. As expected, the proprietary large models like Claude Sonnet-3.5 and GPT-4o perform best out of the box, with Claude Sonnet-3.5 being the best.

One can argue that these results may not fully represent the models’ maximum capabilities. Different model-specific prompt engineering approaches or dynamic few-shot examples could potentially improve performance further. However, my aim is just to assess how well a model perform without fancy RAG/function calling/complicated prompt engineering approaches.

Next steps….

• I plan to fine-tune the base Llama-3-8B and other smaller models (like Phi and Gemma) on the ViGGO dataset to assess whether a fine-tuned smaller model can compete with or surpass the performance of Claude Sonnet-3.5 and GPT-4o.

• At the same time, investigate the trade offs like inference speed, accuracy, and latency associated with fine-tuned smaller models and propreitory models api calls.

References

• Part I blog post of series
• ViGGO Dataset
• Notebook I: Generate_responses_all_llms.ipynb
• Notebook II: Compare_models_performances.ipynb

Thanks for reading! If you have any questions or feedback, please let me know on Twitter or LinkedIn.