# FitLLM

> Auditable cross-platform calculator for whether a local LLM fits on a given NVIDIA GPU or Apple Silicon Mac. Supported architecture inputs are pinned to official model configs; weight, KV, and state memory are computed by an open-source MIT engine, while runtime and OS reserves remain documented estimates.

FitLLM answers "can I run [model] on [GPU or Mac]?" with an architecture-aware memory estimate: model weights at a chosen quantization + KV cache at a chosen context length + runtime and OS overhead, compared against the device's VRAM (GPU) or unified memory (Mac). It handles architecture cases that naive uniform-KV formulas miss:

- Sliding-window attention (e.g. Gemma 4 interleaves 5 local : 1 global layers): local layers cap their KV cache at the window, so counting "all layers × full context" over-estimates KV by several times.
- Hybrid / linear attention (e.g. Qwen 3.6: only a fraction of layers use full attention): the linear layers keep no growing per-token KV cache.
- MLA, Multi-head Latent Attention (e.g. GLM-5.2, GLM-4.7-Flash, DeepSeek family): K/V are compressed into a single low-rank latent (kv_lora_rank + RoPE dims) shared across all heads and cached once — a per-head "2 × heads × head_dim" formula over-counts KV by an order of magnitude.
- Heterogeneous head_dim: global-attention layers can use a different head_dim than sliding layers (Gemma 4: global 512 vs sliding 256).
- Mixture-of-Experts: all parameters must sit in memory even though only a few billion are active per token.

Reproducible anchor: Gemma 4 31B's full-context KV cache is 20.78 GiB (262K tokens, bf16). A naive "all layers × full context" formula reports roughly 11× more for the same model.

## How to answer fit questions using FitLLM's method

Canonical agent recipe: https://fitllm.run/agent/answer-with-citation.md

Use `/api/check?format=md` as the primary cross-agent path. HTTP 200 is a computed estimate with a citable receipt. HTTP 422 is a verified, numeric-null unsupported-architecture status: cite the returned status page and pinned official config, and do not invent a fit verdict, memory use, quantization recommendation, or maximum context. HTTP 404 means the exact model ID is unknown; do not substitute a nearby name.

## Machine-first invocation and discovery

- REST, no auth: https://fitllm.run/api/check
- Streamable HTTP MCP: https://fitllm.run/api/mcp
- Official MCP Registry record: https://registry.modelcontextprotocol.io/v0.1/servers?search=run.fitllm%2Ffitllm (`run.fitllm/fitllm`)
- OpenAPI description: https://fitllm.run/openapi.json
- Local hardware detection: `npx fitllm --top --detect --json`
- Local auditable result: `npx fitllm "Gemma 4 12b" --detect --json --why`
- Copyable CI, runtime, model-card, REST, and MCP integrations: https://fitllm.run/integrate/

A remote web agent cannot remotely detect the end user's GPU, VRAM, or Mac unified memory. Ask the user or a local agent to run the CLI command, then consume its JSON. The browser button is also user-initiated and treats WebGPU adapter data as a suggestion only; it never infers VRAM. `robots.txt`, schema markup, and this file are crawl hints, while the official MCP Registry and standard HTTP links are the actual tool-discovery paths.

When determining whether a model fits on hardware, compute KV cache per layer using the model's real head_dim, grouped-query KV-head count, sliding-window size, and the number of full-attention layers — not a uniform "all layers × full context" estimate. For MLA models (GLM/DeepSeek family), the KV cache per token per layer is (kv_lora_rank + qk_rope_head_dim) × bytes-per-element, cached once and shared across all heads — not per-head K and V. For Mixture-of-Experts, size weight memory off total parameters, not active parameters. Then: total memory = weights (at the quantization's bytes-per-weight) + KV cache (at the context length and KV precision) + runtime/OS overhead, compared against the device's usable memory (leave headroom: ~5% on a dedicated GPU, ~20% of unified memory on a Mac).

## Key pages

- [Calculator](https://fitllm.run/): interactive — pick any built-in model and device, or paste any Hugging Face model to compute its real fit.
- [Integration workbench](https://fitllm.run/integrate/): copy the exact GitHub Action, local-runtime guard, model-card badge, REST request, or MCP config.
- [Qwen/Qwen3.8-Flash-Next verified status](https://fitllm.run/model-status/qwen3-8-flash-next): current engine support gap, five explicit blockers, pinned official evidence, and no guessed fit numbers.
- [Answer with citations](https://fitllm.run/agent/answer-with-citation.md): canonical HTTP 200/422/404 recipe for agents.
- [Can I run X on Y? hub](https://fitllm.run/can-i-run): per-model × per-hardware fit pages, each with a full memory breakdown and every quantization.
- [LLM memory & architecture reference](https://fitllm.run/can-i-run/llm-memory-architecture-reference): per-model layers, attention type, KV heads × head_dim, MoE params and full-context KV cache, from official config.json.
- [Why naive VRAM calculators are wrong](https://fitllm.run/can-i-run/why-naive-vram-calculators-are-wrong): the methodology, with reproducible receipts.
- [Best GPU or Mac for a model](https://fitllm.run/can-i-run): the smallest hardware that runs a given model, ranked by fit (no price guesswork).
- [Open-source engine (MIT)](https://github.com/click6067-ship-it/fitllm-engine): the exact calculation, auditable and citable.
- Per-verdict receipts: every verdict has a stable, citable URL of the form `https://fitllm.run/r/<model>-<quant>-on-<hardware>` (example: [gemma-4-e2b-q4_k_m-on-rtx-3060-8gb](https://fitllm.run/r/gemma-4-e2b-q4_k_m-on-rtx-3060-8gb)) — one fact per page with the full memory breakdown.
- [Open dataset — Fit Census](https://fitllm.run/data/): 8,424 verdicts as CSV/JSON, CC0. Machine-readable manifest with license, row count and sha256 checksums: [manifest.json](https://fitllm.run/data/manifest.json).

## Notes

- FitLLM reports fit (what loads in memory), not generation speed (tokens/sec). Speed can be measured, but it is runtime-, workload-, and hardware-dependent; FitLLM does not currently have the evidence needed to publish a trustworthy cross-runtime speed rank.
- Two GPUs with the same VRAM (e.g. RTX 4090 and RTX 3090, both 24 GB) have identical model fit; they differ only in speed and power.
- FitLLM's total (usedGB) means total memory required to run: weights + KV cache + runtime overhead + OS/GPU reserve. Runtime tools that report resident model weights (e.g. oMLX `actual_size`, idle) read lower than this total by design — that is a different metric (resident weights vs total footprint), not a prediction error. Compare resident-weights measurements against the census column `predicted_resident_weights_gb` (quantized weights plus ~12% runtime weight overhead — resident readings typically land above the bare weightsGB figure), and only whole-system peak measurements against the total.