# Building llamini.cpp: a from-scratch llama.cpp core in C99, zero dependencies

## At a glance

|  |  |
| --- | --- |
| **Dialect** | Pure C99 (POSIX + libc) — 0 third-party dependencies |
| **Build** | One command: `make` → `llamini` |
| **Models** | TinyLlama 1.1B, Qwen2.5 0.5B, GPT-2 124M, Gemma 2B |
| **Quants** | `Q4_K`, `Q6_K`, `Q5_0`, `Q5_K`, `Q4_0`, `Q4_1`, `Q8_0`, `F16`, `F32` |
| **Memory** | ~6–7× lower RSS via lazy 256-row-chunk dequantization (TinyLlama ~4.65 GB → ~702 MB) |

## The pitch

llamini.cpp loads and runs a real [`TinyLlama-1.1B-Chat-v1.0.Q4_K_M.gguf`](https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF) file — a 668MB, 1.1-billion-parameter, 4-bit-quantized language model — using nothing but libc and POSIX. No `ggml`, no `llama.cpp` linked as a library, no Python bindings, no BLAS, no `sentencepiece`. The GGUF binary parser, the `Q4_K`/`Q6_K` block dequantizers, the 22-layer grouped-query-attention transformer forward pass, and the SentencePiece-style BPE tokenizer are all written from scratch, this weekend, in C.

I built this with Claude Code as a pair programmer. This is the honest account of how it actually went — including the parts that didn't work the first time.

## Pairing with Claude Code

Claude Code was a real pair programmer on this, not a ghostwriter. The useful split was surprisingly sharp, and I want to name both sides rather than write a "AI helped me code" paragraph that could apply to anyone.

**Where it excelled:** the tedious, well-specified C. Parsing all 13 GGUF metadata value types (including nested arrays) is a lot of struct-layout and bounds-check boilerplate once you've decided the file format is untrusted input. Same for the bit-shift unpackers on K-quant super-blocks, the FNV-1a open-addressing hash table over a 32,000-entry vocab, and the mechanical `argc`/`argv` flag handling. Those are places where an LLM writing C is genuinely faster than a human typing, and the existing `--test` suite could catch the dumb mistakes.

**Where it failed, specifically:**

*   **Fake citations.** I asked it to find a published TinyLlama perplexity number I could compare against. A web-search summary produced one, with a paper attached. The number was not in that paper. The search had synthesized it. If I'd cited it, this write-up would be lying. That's why `--bench` exists: a self-contained number that needs no citation (see "How I actually verified this was correct" below).
    
*   **Subtle concurrency.** The persistent pthread pool — workers blocking on a condition variable between dispatches — is exactly the kind of code that compiles, looks right, and is still wrong. A naive "wake everyone up" broadcast lets idle slots read *stale job data left over from the previous dispatch* and silently write into the wrong buffer. I caught that by tracing a concrete example on paper before running anything, not because the model flagged it.
    
*   **Non-standard RoPE strides.** LLaMA pairs adjacent vector elements; Qwen2 and Gemma pair elements half a head-dimension apart. Getting this wrong does not crash. It produces fluent, grammatical, *wrong* output — the worst failure mode, because "the model sounds coherent" was exactly the signal I'd been using to sanity-check everything else. Architecture autodetection is not a config tweak; the pairing convention is load-bearing.
    

The rest of this post is written as if I did the work, because I did — including the parts where the pair programmer was confidently wrong and I had to catch it.

## Now, why I built this, not just that I could

Two reasons, and I want to be specific about both rather than wave at "dependencies are bad."

**The first is a supply-chain argument, and I want to ground it in real numbers rather than assert it.** llama.cpp is not a niche project sitting in some corner of GitHub — it's load-bearing. [Ollama depends on it directly](https://github.com/ollama/ollama) (pulled in and patched at build time, confirmed by reading Ollama's own build directory, not a vendored copy) and Ollama itself has [179.8k GitHub stars](https://github.com/ollama/ollama) and its official Docker image has logged over 100 million pulls. [`llama-cpp-python`](https://github.com/abetlen/llama-cpp-python) ([10.6k stars](https://github.com/abetlen/llama-cpp-python)) is a direct binding over it. [GPT4All](https://github.com/nomic-ai/gpt4all) ships, in their own README's words, ["a Python client around llama.cpp implementations."](https://github.com/nomic-ai/gpt4all) text-generation-webui installs prebuilt llama.cpp binaries. [LM Studio's own blog](https://lmstudio.ai/blog) credits "our llama.cpp engine." That's five real, independently-verifiable dependents, several of them extremely widely deployed, all resting on one C/C++ codebase. (I checked, specifically, whether vLLM belongs on this list — it doesn't. It's a separate inference engine with its own team and its own attention implementation. Saying otherwise would have been exactly the kind of unverified claim that already burned me once earlier in this project, so I'm naming the one I ruled out, not just the ones that fit the story.)  
  

![](https://cdn.hashnode.com/uploads/covers/651b62bfb9b55f31c7972ce0/18584d3d-f9c1-433a-ba84-5be6d546aa76.png align="center")

Here's the thought experiment. In 2024, someone spent **over two years** building a trusted reputation as a contributor to xz-utils, a compression library almost nobody thinks about, before landing a backdoor in `liblzma` that would have compromised SSH access on a huge fraction of the internet's Linux servers ([CVE-2024-3094](https://nvd.nist.gov/vuln/detail/CVE-2024-3094), CVSS 10.0 — the maximum score). It wasn't caught by a security scan. It was caught by one engineer at Microsoft/PostgreSQL noticing that SSH logins were taking a few hundred milliseconds longer than they should, and refusing to let that go. Now replace "SSH server nobody thinks about" with "the C++ inference engine every local-LLM tool on your laptop is quietly running." The pattern doesn't require anything exotic — patient social engineering, one maintainer's trust, a subtle enough change that normal review doesn't catch it — and the same kind of blast radius (real damage: the 2020 SolarWinds Orion compromise hit roughly 18,000 of SolarWinds' 33,000 Orion customers through a poisoned build pipeline; the closest AI-specific precedent I could verify, [PyTorch's own December 2022 disclosure](https://pytorch.org/blog/compromised-nightly-dependency/) of a malicious `torchtriton` package on PyPI, exfiltrated SSH keys and git credentials from anyone who happened to install a nightly build before it was caught).

I'm not going to overstate what this project proves. llamini.cpp does not replace llama.cpp inside Ollama, GPT4All, or anything else on that list — it's a hackathon-scale reimplementation of the *core*, run against 1.1B-2.5B-parameter models on a laptop-class VM, not a production inference engine. What it does demonstrate is narrower and, I think, still worth doing: that the actual ideas inside that trusted C++ codebase — GGUF's binary layout, block-quantized dequantization, RoPE, grouped-query attention — are not actually a black box that has to be taken on faith. One person, in a weekend, can read the real format, implement the real math, and end up with something that produces recognizably correct output. That legibility has value independent of whether this specific binary ever runs in anyone's production stack.

**The second reason is more personal, and it's the one that actually got me to start.** Most people working with AI models today — myself included, most of the time — operate a long way above the actual math. `pip install transformers`, three lines of Python, and a model is running; it's a black box you feed strings and get strings back from, and the moment something doesn't work the instinct is to reach for another package, not to open the one you already have. I wanted to build this the other way around: no framework, no Python runtime, no library standing between me and the actual bytes of a `.gguf` file. Every layer — the binary format, the quantization block layout, the attention math, the tokenizer's merge rules — written out by hand and understood well enough to explain, not copy-pasted from a tutorial. The empty dependency manifest is the visible proof of that; [STDLIB.md](STDLIB.md) is the itemized receipt for every place a normal stack would `pip install` something and this one didn't.

I want to be honest about the difference between "avoided dependencies" and "did something worth doing," because they aren't automatically the same thing. Nothing in this codebase is contorted or obfuscated purely to dodge an import — there's no macro trick or unreadable one-liner whose only purpose is "look, no `pip install`." It's ordinary, commented C, built the way I'd build it if dependencies weren't a scoring category at all, that happens to need nothing beyond libc and POSIX. The honesty sections throughout this write-up, the README, and [BUILDLOG.md](https://github.com/hatif03/llamini.cpp/blob/main/BUILDLOG.md) exist so a reader can tell the difference between "intentional" and "a stunt" for themselves, rather than take my word for it.

## What I reimplemented

Everything a real `llama.cpp`\-class inference stack needs, minus the parts that are genuinely out of scope for one weekend:

*   **A real** [**GGUF parser**](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) — not just the 24-byte header, but the actual metadata key/value table (all 13 value types, including nested arrays) and the tensor-info table, with alignment-padded tensor data after that. Most toy GGUF readers stop at "read the header and print the version"; the metadata table is where the model's real shape (hidden size, layer count, head count, vocab) actually lives, and it's non-trivial to parse correctly because entries are variable-length and self-describing.
    
*   `Q4_K` **and** `Q6_K` **block dequantization** — the two "K-quant" super-block formats a real `Q4_K_M` file actually uses (plus the simpler `F32`, `F16`, `Q4_0`, `Q4_1`, `Q8_0` for completeness). These are bit-packed formats: scales and minimums for eight 32-element sub-blocks packed into 12 bytes using overlapping 6-bit fields. Getting the bit shifts right with zero reference implementation to diff against was the part of this project closest to "hope this compiles into correct math." The actual unpacker in `gguf.c` is small enough to quote:
    

```c
/* Q4_K's 12-byte `scales` field packs six-bit (scale, min) pairs for
 * 8 sub-blocks of 32 elements each. Sub-blocks 0-3 store their
 * scale/min directly; 4-7 reuse the low bits of 0-3's bytes for
 * their high bits. */
static void get_scale_min_k4(u32 j, const u8* q, u8* d_out, u8* m_out) {
    if (j < 4) {
        *d_out = q[j] & 63;
        *m_out = q[j + 4] & 63;
    } else {
        *d_out = (u8)((q[j + 4] & 0xF) | ((q[j - 4] >> 6) << 4));
        *m_out = (u8)((q[j + 4] >> 4)  | ((q[j]     >> 6) << 4));
    }
}
```

*   **A 22-layer transformer forward pass** with grouped-query attention (32 query heads sharing 4 key/value heads — TinyLlama doesn't use plain multi-head attention), RoPE positional encoding, SwiGLU feed-forward layers, and RMSNorm — the actual architecture, not a simplified stand-in.
    
*   **A SentencePiece-style BPE tokenizer**, driven entirely by the vocab, merge-scores, and token-type arrays embedded in the GGUF file itself — no external vocab file, no `sentencepiece` library.
    

## What the standard library made painful

C's standard library has no hashmap, no JSON, no HTTP, and (this is the one that actually mattered) no obvious way to verify you got any of this right. Every other language in this hackathon's toolbox has some path to "install the real library once, diff your output against it." C's answer is "write it carefully and reason about it," which is a fundamentally different — and slower — way to build confidence.

The place this bit hardest: BPE tokenization needs a fast string→id lookup over a real ~32,000-entry vocabulary, and a linear scan is genuinely too slow for interactive use once you're not dealing with a 15-word toy vocab anymore. That's a legitimate reason to hand-roll an open-addressing hash table (FNV-1a, linear probing) — not because "C has no hashmap" is an excuse to build one everywhere, but because this specific spot actually needed it.

## The package I made look unnecessary

`ggml`/`llama.cpp` linked as a library, `llama-cpp-python`, `sentencepiece`, and a BLAS/numpy-equivalent for the tensor math. All four are things you'd normally `pip install` or link against to do exactly what this project does from a single C binary with an empty dependency manifest. I'm not claiming to have reimplemented `ggml` in general — this loads four architecture families (LLaMA-shaped RMSNorm + RoPE + SwiGLU + GQA, plus Qwen2, Gemma, and a separate GPT-2 path) and a specific quantization mix. But for that job, the library isn't required.

## The edge case that ate an afternoon (well — several)

**The one that would have eaten an afternoon, if I'd let it:** after rebuilding everything for real, the model would type back something like `". The event, DP, and"` in response to "hello" — real English fragments, not noise, but clearly disconnected from the input. The obvious suspects are tokenizer bugs, wrong weights, or broken attention math, and chasing any of those from the symptom alone could easily have burned hours. The actual bug, found by reading the generation loop line by line instead of guessing: it only ever embedded the *last* token of whatever you typed. Every earlier position's slot in the KV cache stayed zeroed, and the attention math still summed over those zeros regardless. The model was never blind — it just never got shown the input.

**The one where I tested an assumption before building on it, and the assumption was wrong:** I planned to build a chat template by literally inserting the text `"</s>"` between turns, reasoning that since `</s>` is a real end-of-turn token with a merge-priority score of `0.0` while every ordinary vocabulary piece has a negative score, the tokenizer's own score-based merging would naturally reassemble it into that single token. Before writing the feature, I wrote a five-minute throwaway probe that just tokenized the literal string and printed the result. It split into three ordinary sub-word pieces and the real end-of-turn token never appeared once. Real tokenizers special-case strings like this *before* running BPE; mine doesn't, so it never had a chance. The fix ended up better than the original plan — build the prompt at the token level and insert the real token id directly, which sidesteps the whole problem instead of fighting it.

**The one I shipped wrong on the first try, on purpose left in this write-up:** fixing the "only sees the last token" bug above, I rewrote the generation loop's stopping condition as a single precomputed iteration count — prefill steps plus generation steps. It was off by one: the last prefill step and the first generated token turn out to be the *same* loop iteration, so my count generated one extra token every time. The project's own test suite caught it immediately (`Assertion 'total >= 4 && total <= 8' failed`) before it ever reached a commit. I didn't get this right on the first attempt, the tests did their job, and the actual fix (three explicit, separately-readable stopping conditions instead of one clever formula) is better code than what I would have shipped if I'd been more confident and skipped writing a test for it.

**The one that wasn't a code bug at all:** for a while, the same command took anywhere from 25 seconds to over five minutes, with no pattern I could find in the code. Turned out to be a previous test run that a `timeout` wrapper hadn't actually killed — 19 minutes deep, still holding 4.4GB of RAM in a VM that only had 7.6GB total. I'd been benchmarking my own leftover mess. Lesson that's now permanent habit: check `ps aux` before trusting a timer, especially inside a VM.

## How I actually verified this was correct, not just "not crashing"

"The output looks like English" is a weak claim — a broken forward pass producing token-shaped garbage can still coincidentally resemble language in short bursts. I looked for a published TinyLlama perplexity number to compare against, found one via a web search, and when I went to actually verify it against the paper the search cited, **the number wasn't in that paper.** The search had synthesized it. That's a real, embarrassing-if-I'd- missed-it lesson: an AI-summarized search result is not a citation until you've opened the source and checked.

So instead of leaning on an external number I couldn't verify, I built a self-contained benchmark: teacher-forced perplexity over a small fixed English test corpus, compared against the one number that needs no citation at all — a uniform random guess over a 32,000-token vocabulary has a perplexity of exactly the vocabulary size (~32000, or 10.37 nats/token). Measured: **13.28**. Three orders of magnitude below random. Alongside that, a handful of specific factual completions — `"The capital of France is"` → `" Paris."` — which is a sharper, more falsifiable signal than "sounds like English," because there's exactly one right answer and the model gave it.

## What made things faster, and what didn't move the needle as much as expected

Compiling with `-march=native` plus a few reassociation flags (short of full `-ffast-math`, which also assumes away NaN/Inf checks I wanted to keep) let gcc vectorize the core matrix-vector dot product — confirmed by reading the generated assembly, not assumed. Parallelizing that same operation across CPU cores with a dozen lines of pthreads on top of that got another 5-7x. Both were verified to produce byte-identical output to the unoptimized version first, because a "faster" change that quietly changes the answer isn't actually a win.

What I *didn't* do, at the time: keep weights quantized in memory and dequantize them on the fly during matrix multiplication instead of expanding everything to full 32-bit floats up front (~4.4GB resident for a 1.1B-parameter model). That's the architecturally correct fix for memory footprint, and I scoped it out in detail before deciding not to attempt it — it touches the most correctness-critical code in the project, and a subtle bug there produces *plausible-looking wrong text*, not a crash you'd notice. Given the speedup from the safer changes above, I chose not to gamble a working project on a rewrite whose benefit I hadn't confirmed was still necessary. Documented as the clear next step instead of quietly dropped — and, as it turns out, I did come back for it. More on that below.

TinyLlama proved the math worked. The next claim needed more than one architecture family. The GGUF header's `general.architecture` field is what picks the path:  

![](https://cdn.hashnode.com/uploads/covers/651b62bfb9b55f31c7972ce0/90f326e5-da43-41a6-bfd5-04b3b3851ba3.png align="center")

## Then I ran it on three more model families

TinyLlama proved the engine worked, but one architecture family is a thin claim for "reimplements llama.cpp's core." So the next phase was: pick a handful of small, genuinely different architectures, download real checkpoints, and get them running — not by copying llama.cpp's code, but by reading its *source* (the actual GGUF constants, conversion scripts, and model definitions on GitHub) to find out what each architecture actually needs, the same way I'd read a spec.

That research surfaced two requirements I would never have found by staring at TinyLlama's file alone. First: Qwen2 and Gemma use a different RoPE pairing convention than LLaMA — my rotary encoding paired adjacent vector elements, which is right for LLaMA (GGUF conversion permutes the weights at export time to make that work) but wrong for Qwen2/Gemma, which need elements paired half a head-dimension apart instead. Getting this wrong doesn't crash. It produces fluent, grammatical, *wrong* output — the worst kind of bug, because "the model sounds coherent" is exactly the signal I'd been using to sanity-check everything else. Second: Qwen2 and GPT-2 don't use SentencePiece at all — they use byte-level BPE, a genuinely different vocabulary convention (an explicit merge-rank list instead of per-token scores, plus a byte-to-Unicode remapping table), not a config tweak on top of what already existed.

I picked three targets: [**Qwen2.5-0.5B-Instruct**](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF), [**GPT-2-124M**](https://huggingface.co/gpt2), and [**Gemma-2b**](https://huggingface.co/google/gemma-2b) — and asked myself up front to be honest if one of them didn't fully work, rather than force it. Two came together cleanly, with their own specific gotchas:

*   **Qwen2.5** loaded, but produced only `"!!!!!!!!"` — the "zero-initialized weights" failure mode I'd already seen once before. I wrote a small throwaway probe to dump the real tensor types straight out of the file, rather than guess, and found `token_embd.weight` used `Q5_0` — a quant format I hadn't implemented, despite the file being labeled "Q4\_K\_M" (that label describes the *dominant* format, not a guarantee about every tensor). Implemented it, reran, and got fluent English that correctly drifted into fluent Chinese mid-completion — a small, specific, falsifiable signal that the tokenizer's UTF-8 handling was actually right.
    
*   **GPT-2** needed a wholly separate forward-pass module, not a config flag: fused QKV in one tensor (sliced by pointer arithmetic instead of three separate projections), LayerNorm instead of RMSNorm, learned absolute position embeddings instead of RoPE, and an *ungated* GELU MLP instead of SwiGLU. This time I applied the lesson from Qwen2.5 directly — I probed the real file's tensor types *before* writing any loading code, found it needed `Q5_K` (declared in the code as a constant for over a session's worth of history, never implemented), implemented it first, and got a working model with zero further debugging. Same category of bug, caught before it happened instead of after, because I'd been burned by the same shape of mistake once already.
    

**Gemma-2b was the one that didn't fully work, and I said so.** Its architecture-specific code — NEOX RoPE, GeGLU gating, an embedding scale factor, tied embeddings, multi-query attention — was all already-built machinery shared with Qwen2.5's path, so config auto-detection came out correct on the first try, verified against the real file's printed hyperparameters. But actually generating text needed dequantizing all 2.5 billion parameters to 32-bit floats at once, which comes to roughly 10-12GB — on a dev VM with 7.6GB of RAM. Three attempts gave three different, inconclusive results (an empty exit, a suspiciously fast clean exit that printed almost nothing, another that failed at the tool level after 90 seconds) — not one clean, reproducible failure I could point at and say "this is the bug," which would at least have been something to fix. I stopped after the third attempt, on purpose, rather than keep hammering on a shared dev machine chasing a failure mode that wasn't resolving into a single story. The honest way to write that down was "untested under memory pressure," not "works" and not "broken" — and I shipped it that way rather than paper over the gap. (This one has a sequel — see below.)

## Optimizing for CPU, and finding out what was actually true

Later, asked to make the engine faster on CPU — no GPU allowed, per the hackathon's own constraint — and to find some real, measurable way it could be better than an existing tool, not just "smaller." Rather than guess at optimizations, I read llama.cpp's actual CPU kernel source ([`ggml-cpu.c`](https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-cpu/ggml-cpu.c), `ggml-quants.c`, `llama-mmap.cpp`) to find out what a decade-plus of real-world tuning had converged on, and it wasn't what I expected going in.

The load-bearing fact: CPU inference at batch size 1 is **memory-bandwidth bound, not compute-bound.** ggml never dequantizes a whole tensor to floats — its dot-product kernels unpack quantized blocks and multiply directly against the packed bytes, converting to float only for the final scaled sum, because reading ~4.5 bits per weight instead of 32 is the actual lever, not raw arithmetic throughput. It also uses a persistent worker-thread pool (created once, woken via a condition variable) instead of spawning threads per operation, and reads tensor data straight out of a page-cache-backed `mmap`, never a private copy.

Two changes followed directly from that, plus a decision to *revisit* something I'd deliberately shelved:

*   **A persistent thread pool.** My own `linear()` had been calling `pthread_create`/`pthread_join` on *every single call* — roughly nine times per transformer layer, per generated token. For TinyLlama's 22 layers, that's about 200 thread spawns to produce one word. Replaced it with a pool created once, workers blocking on a condition variable between dispatches. Before running any of it, I caught a real bug just by tracing through a concrete example on paper: if fewer workers get real work than the pool has slots, a naive "wake everyone up" broadcast lets the idle slots read *stale job data left over from the previous dispatch* and silently compute into the wrong buffer. Fixed by having each worker check whether its slot actually got new work this round before touching anything. Verified bit-identical output against the old spawn-per-call version, and a genuine ~32% wall-clock improvement in one measured pairing — pure overhead removed, no numerical change.
    
*   **The on-demand dequantization I'd shelved.** This was the fix I'd scoped out earlier in this project and chosen not to attempt, rated too risky for the confirmed benefit at the time. Revisited now with a much more concrete plan, and a decomposition that turned out to be safer than I'd originally assumed: only the handful of genuinely large weight matrices (embeddings, output projection, the per-layer attention/FFN projections) needed to become "lazy" — dequantized in small row-chunks straight out of the memory-mapped file, on demand, instead of ever being materialized as a full array of floats. Small tensors (norms, biases) stayed exactly as before. Crucially, this meant the project's existing test suite — which builds a synthetic model and pokes floats directly into its weight buffers, never touching a real file at all — needed *zero changes*, because it never goes near the new lazy code path. That was the detail that made this feel safe to actually attempt, not just theoretically nice.
    
    I made one deliberate design choice I'd get wrong if I copied ggml's approach naively: I dequantize *fixed-size* row-chunks, not thread-count-sized chunks. My first instinct was to reuse the same chunking scheme as the thread pool above — split the matrix evenly across however many CPU cores are available. That's wrong for this specific fix, and the reason is exactly the tensor this change was meant to rescue: Gemma-2b's embedding table has 256,128 rows. Split across 8 threads, each thread's chunk would still need a quarter-gigabyte scratch buffer — right back to the multi-hundred-megabyte problem I was trying to eliminate, just moved one level down. A fixed chunk size (256 rows, regardless of the tensor's total width) keeps peak transient memory bounded no matter how large the model gets.
    

![](https://cdn.hashnode.com/uploads/covers/651b62bfb9b55f31c7972ce0/55697b7b-57a3-43a2-9ee1-6eba7e7025c8.png align="center")

  
  
I also decided, after reading the research above more carefully, *not* to thread-pool this path at all — if the real bottleneck is memory bandwidth, several threads racing for the same bytes isn't an obvious win, and it would have reopened the exact chunk-size problem I'd just solved.

The bar I held this to was **bit-identical generated text**, not "doesn't crash" — I built the pre-change and post-change binaries side by side and diffed their output for every model already working. All identical. Resident memory dropped ~6-7x across the board, reproducibly, every time I measured it.

| Model | Output vs pre-change | Resident memory before → after |
| --- | --- | --- |
| TinyLlama 1.1B | byte-identical | ~4.65 GB → ~702 MB (~6.6×) |
| Qwen2.5 0.5B | byte-identical | ~2.96 GB → ~477 MB (~6.2×) |
| GPT-2 124M | byte-identical | ~831 MB → ~831 MB (eager on purpose — `gpt2.c` not in this pass) |
| Gemma 2B | previously never completed | untestable → ~1.54 GB, twice, byte-identical across both runs |

**And Gemma-2b finally ran.** With the memory footprint fixed, I reran the exact model that had given three inconclusive failures earlier in the project. It completed, produced a real perplexity number dramatically below chance, and answered two of three fact-completion prompts correctly. Given its history, I didn't trust one clean run — I ran it a second time, independently, and got byte-identical output and nearly identical memory usage both times. That's the difference between "it worked once" and "it's reliable," and it's the bar this whole pass was supposed to clear. I also hit a genuine WSL crash partway through this verification — the virtual machine's own service died mid-benchmark (not my program; a plain `echo` failed the same way right after) — and had to ask before restarting it, since that would end anything else running in that environment. Worth including here because it's a real thing that happened, not smoothed over for the narrative.

Finally, since "optimize it" is a hollow claim without a number to check it against, I built real, unmodified llama.cpp from source — on the same machine, never vendored into this project, used only to generate honest comparison data — and ran it against the exact same four files. The result was more interesting than a clean win:

| Model | Engine | Peak RSS | Token speed |
| --- | --- | --- | --- |
| TinyLlama 1.1B (Q4\_K\_M) | llama.cpp | ~1.12 GB | 0.80 ± 0.40 tok/s (`tg32`) |
|  | **llamini.cpp** | **~710 MB** | **0.41–0.67 tok/s** (incl. prefill) |
| Qwen2.5 0.5B | llama.cpp | ~558 MB | 0.69 ± 0.21 tok/s (`tg32`) |
|  | **llamini.cpp** | **~480 MB** | **0.95–1.24 tok/s** (incl. prefill) |
| GPT-2 124M | llama.cpp | ~169 MB | 0.56 ± 0.31 tok/s (`tg32`) |
|  | **llamini.cpp** | **~831 MB** | **4.80–8.32 tok/s** (incl. prefill) |
| Gemma 2B (Q4\_K\_M) | llama.cpp | ~2.66 GB | 1.78 ± 2.19 tok/s (`tg32`) |
|  | **llamini.cpp** | **~1.54 GB** | not measured |

llamini.cpp's memory footprint beat real llama.cpp's on three of the four models, plausibly because llama.cpp reserves a larger default context and batch buffers than this much smaller project bothers with. It lost clearly on the fourth (GPT-2), for a reason I could point to instead of hand-wave: GPT-2's own weights were never brought into this optimization pass, only the shared LLaMA-family code path was. Generation speed was closer and noisier in both directions, and I reported it that way rather than picking the flattering number — llama.cpp's own repetition spread was, on more than one model, larger than its own mean, which told me as much about this particular dev machine's instability as it did about either engine. llamini's tok/s includes each `--bench` completion's prompt prefill; llama.cpp's `tg32` is pure decode. Read the speed column as directional context, not a precise ratio.

## What's still genuinely unfinished

No multi-turn conversation memory (every line is a fresh single-turn generation). No sampling beyond temperature + top-p (no top-k, no repetition penalty). The chat template is my best reconstruction of TinyLlama-Chat's own format, not verified against a real `transformers` install, because having one would defeat the point of the project — and it only covers TinyLlama-Chat's own convention; Qwen2 uses a different one (ChatML) that was never built, so it's exercised only in raw-completion mode. Mistral was skipped deliberately (its GGUF file identifies as plain `"llama"`, so it would run with zero new code — meaning zero new *proof* of anything); Gemma2 and BERT were both scoped out as real added complexity for a benefit not worth the remaining hackathon time. The on-demand dequantization above is also, by design, simpler than ggml's real approach: it converts each chunk to floats before the dot product instead of ggml's true integer arithmetic on the packed bytes directly, and it isn't extended to GPT-2's own weights yet — both disclosed rather than discovered by a reader comparing numbers themselves. Quantization support covers the formats the four real, downloaded files actually use and nothing more — anything else is explicitly rejected rather than silently misread. All of this is written down in the repo's own limits section, not discovered by a judge.

* * *

*Full engineering log, commit-by-commit, including every dead end and every number measured:* [*BUILDLOG.md*](https://github.com/hatif03/llamini.cpp/blob/main/BUILDLOG.md)*. Source:* [*github.com/hatif03/llamini.cpp*](https://github.com/hatif03/llamini.cpp)*.*