AIToolLabHub.org

GGUF is the file format llama.cpp uses, and the letters after a model name tell you how much precision was thrown away to make it smaller. Q4_K_M, Q5_K_M, Q8_0 and the rest are not brand names: they describe a quantisation method and a target size. This guide explains what the labels mean, what the trade-off is, and how to pick one that fits your machine. It is not a benchmark. The numbers below come from llama.cpp's own quantisation documentation, which lists bits per weight and file sizes for Llama 3.1 8B. They were checked on 29 October 2026. Your model and your tasks may behave differently. What quantisation does A model is a large set of weights, normally stored at 16-bit or 32-bit precision. Quantisation stores those weights with fewer bits: 8, 6, 5, 4, 3 or even 2. The file gets smaller and usually needs less memory, which is how a model that would not fit on your GPU can suddenly run on it. The cost is accuracy. llama.cpp's documentation says quantisation ...

Ollama is a command-line program, not a chat app. You ask it for a model, it downloads and runs it, and the same process exposes a local API. These are the commands worth learning first; everything below was checked against Ollama's CLI, API and Modelfile documentation on 29 October 2026. The commands at a glance Command What it does Example ollama run <model> Downloads the model if it is missing and starts a chat in the terminal. ollama run gemma4 ollama pull <model> Downloads or updates a model without starting it. ollama pull gemma4 ollama ls Lists the models stored on this machine. ollama ls ollama ps Shows the models that are currently loaded or running. ollama ps ollama stop <model> Stops a running model. ollama stop gemma4 ollama rm <model> Deletes a model from disk. ollama rm gemma4 ollama serve Starts the local server and its API. ollama serve ollama create <name> -f <file> Build...

Ollama, LM Studio and llama.cpp all run open models on your own machine, but they are not the same kind of tool. One is a command-line daemon with an API, one is a desktop app with a chat window, and one is the C/C++ engine the other two can sit on. Choosing well is mostly about how you want to work, not about which one is "best". This is a feature comparison, not a speed test. Inference speed depends on your GPU, the model and the quantisation, so any "fastest" claim without those details is marketing. The facts below come from each project's own documentation and were checked on 25 October 2026. The short answer Tool What it is Best for Ollama Open-source (MIT) command-line runner with a local REST API and an OpenAI-compatible endpoint. Scripts, servers, repeatable setups and anyone who prefers the terminal. LM Studio Desktop app with a chat interface, a model browser and a programmable local API; uses llama.cpp and MLX under the...

Short answer: an 8-billion-parameter model at 4-bit quantisation needs roughly 6 GB of VRAM at 8,192 tokens of context. The same model at 16-bit needs about 18 GB . The number people usually quote ("you need 16 GB for local AI") is the worst case with no context at all. Rather than a list of models we would have to keep re-checking, here is the arithmetic. It is the same formula our VRAM calculator uses, so you can reproduce every row below. Three things use your VRAM Weights. The model itself: parameters × bits per weight ÷ 8. KV cache. The memory for the attention keys and values of the conversation so far. It grows with the context window you allow and with the size of the model. Runtime overhead. The engine, the GPU context, working buffers. We budget 0.8 GB. weights_GB = params_B × bits ÷ 8 kv_GB ≈ (context ÷ 8192) × params_B × 0.15 total ≈ weights + kv + 0.8 The KV line is an approximation on purpose: the exact figure depends on the a...