LLM Quantization: What fp8 and fp4 Do to Quality and Price

When you call an open-weight model through an API, several providers may serve it at different precisions, such as 16-bit bf16, 8-bit fp8 or 4-bit fp4. The precision can affect both the answers and the price. LLM quantization creates these versions by storing a model’s weights in fewer bits, so the model needs less memory and hardware, at some cost in accuracy.

This post explains the formats and reports what we measured on OpenRouter on October 20, 2026, across 1,455 provider endpoints and six endpoints serving Qwen3.8 27B. On our test, we didn’t detect an accuracy difference between fp8, fp4 and fp16, so the precision label alone didn’t predict accuracy. Lower precision was usually cheaper, but not for every model.

TL;DR

• We didn’t detect an accuracy difference between fp8, fp4 and fp16 on our 240 scored tasks. All six endpoints scored between 98.3% and 99.2%.

• On 40 more multiplication tasks, one fp4 endpoint scored lowest, by a gap too small to rule out chance.

• 8-bit was cheaper than 16-bit for 14 of the 24 models served at both, and more expensive for 7. 4-bit was cheaper than 8-bit for 26 of 33.

• provider.quantizations pins the precision you get. List every precision you’d accept, so the router has more than one endpoint to fall back to.

What is LLM quantization?

LLM quantization means storing a large language model’s numbers with fewer bits than it was trained with. Most LLMs today use FP16 or BF16, which are 16-bit formats. fp8 halves the memory each weight takes, and fp4 or int4 quarters it, so the same GPU holds a bigger model or serves more requests. The cost is accuracy, because each weight is rounded to the nearest value the smaller format can store.

Format	Bits per weight	Weights for a 27B model	Where you see it
bf16 / fp16	16	54 GB	The precision most models are released in
fp8	8	27 GB	The most common labeled precision on OpenRouter
fp4, nvfp4, mxfp4, int4	4	13.5 GB	Cheaper endpoints, local runs, some lab releases

These figures cover weights only. A running model also needs memory for its KV cache, the stored attention state for each token in the context.

There are two main types of quantization. Post-training quantization converts a finished model’s weights. It can be dynamic, computing the range of each activation at run time, or static, computing those ranges in advance by passing sample data through the model. Quantization-aware training simulates the rounding during training so the weights learn to tolerate it. The Hugging Face Optimum concept guide explains both, and the Transformers quantization overview covers the quantization methods you can run yourself.

fp8 vs bf16: what’s the difference?

Floating-point formats split their bits between a sign, an exponent and a mantissa. The exponent sets the range of values, and the mantissa sets how finely values within that range are resolved.

bf16 has 1 sign bit, 8 exponent bits and 7 mantissa bits, so it has the same range as 32-bit fp32 with less precision. fp16 splits its 16 bits as 5 exponent and 10 mantissa bits. The OCP 8-bit floating point spec defines two fp8 encodings, E4M3 (4 exponent bits, 3 mantissa bits) and E5M2 (5 exponent bits, 2 mantissa bits). fp8 gives up both range and precision, so quantized models store a scale factor per tensor or per block of weights. MXFP4 shares one scale across each block of 32 values, and NVFP4 uses one fp8 scale per 16 values.

For an API user, fp8 vs bf16 shows up as a different label and price on the same model. In our test, we didn’t detect a difference between the three fp8 endpoints and the fp16 endpoint.

How much quality is lost with fp8 and fp4?

We didn’t detect a quality loss at fp8 or fp4 in our test. All six endpoints scored between 98.3% and 99.2%, and a test this size can’t rule out a gap of a point or two.

We ran 280 tasks with exact answers through qwen/qwen3.8-27b, pinned to six endpoints. Cerebras serves it at fp16. Parasail, CoreWeave and AkashML serve it at fp8, and Darkbloom and ModelRun serve it at fp4. Each request set provider.only, quantizations and allow_fallbacks: false, so it could only reach the endpoint we named, at temperature 0 with reasoning off. The tasks were multi-step arithmetic with working, Python programs to trace, lists of 20 to 30 numbers to sort, a key to find among 240 or 1,200 entries (about 4,000 and 20,000 tokens), and a four-step lookup chain through about 9,000 tokens. We generated every task, so each has an exact answer and no judge is needed. Another 40 tasks asked for the product of two four-digit numbers with no working, scored separately.

Task family	Tasks	fp16 Cerebras	fp8 Parasail	fp8 CoreWeave	fp8 AkashML	fp4 Darkbloom	fp4 ModelRun
Arithmetic with working	50	48	49	48	49	49	50
Short code traces	50	50	50	50	50	50	50
Long code traces	30	30	29	29	29	29	28
Sorting 20 to 30 numbers	30	30	29	29	29	29	30
Recall at 4,000 tokens	20	20	20	20	20	20	20
Recall at 20,000 tokens	20	20	20	20	20	20	20
Four-hop recall	40	39	40	40	40	39	40
Total	240	98.8%	98.8%	98.3%	98.8%	98.3%	99.2%
Multiplication, no working	40	30	33	31	32	26	30

Source: OpenRouter, Qwen3.8 27B, October 20, 2026. The 95% confidence intervals on the totals run from 95.8% to 99.8%, and all six overlap.

No task family separated the precisions. Each endpoint missed at most two tasks in any family, and the misses were ordinary slips, such as a number dropped from a sorted list. The fp8 endpoints disagreed with each other on 2 to 5 of the 240 tasks, and the two fp4 endpoints on 4. Darkbloom disagreed with the fp16 and fp8 endpoints most often, on 7 or 8 tasks, yet its total was within half a point of fp16.

The one place a gap showed was multiplication with no working. Darkbloom at fp4 got 26 of 40 products exactly right, against 30 to 33 on the other five endpoints, including 30 on ModelRun at fp4. A two-proportion z-test against the best endpoint (33), a comparison we chose after seeing the results, puts the chance of a gap that large arising with no real difference at about 7.5%, so we can’t call it a precision effect. Wrong products were close at every precision, with the median miss off by less than 0.01%.

We ran all 280 tasks twice on Cerebras, and both runs returned identical text on every task. The small differences between endpoints can come from precision, from each provider’s hardware and inference software, or from run-to-run variation on the other endpoints, and at this size we can’t separate them. Cerebras is the only fp16 endpoint, so fp16 has no second endpoint to compare it with.

This test covers one model, 280 tasks and six endpoints on one day. It doesn’t show that 4-bit is safe for every model or workload, so test your own prompts before you move traffic to a 4-bit endpoint.

Which precision do API providers serve?

fp8 is the most common labeled precision, and most endpoints don’t report a precision at all. We pulled every model’s endpoints from our endpoints API, which returned 1,455 endpoints across 470 models.

Reported precision	Endpoints	Share of all	Share of labeled
unknown	865	59.5%
fp8	343	23.6%	58.1%
fp4	123	8.5%	20.8%
bf16	70	4.8%	11.9%
int4	20	1.4%	3.4%
nvfp4	13	0.9%	2.2%
mxfp4	12	0.8%	2.0%
fp16	8	0.5%	1.4%
fp32	1	0.1%	0.2%

Source: OpenRouter (openrouter.ai/api/v1/models/{author}/{slug}/endpoints), as of October 20, 2026.

610 of the 865 unknown endpoints belong to closed-weight models with no labeled endpoint at all, including every Claude and Gemini model and every OpenAI model except gpt-oss. Of the other 255, 253 are on open-weight models whose providers don’t label precision. Only 54 of the 163 models with a labeled endpoint have a 16-bit one, so for most open-weight models, 8-bit or 4-bit is the only labeled option.

Is a quantized endpoint cheaper?

Usually, but not always. For each model, we compared the cheapest endpoint at each precision, at a blended price of three input tokens to one output token.

Comparison	Models	Lower precision cheaper	Within 2%	Lower precision more expensive	Median price ratio
8-bit vs 16-bit	24	14	3	7	0.94
4-bit vs 8-bit	33	26	3	4	0.76
4-bit vs 16-bit	11	9	0	2	0.73

Source: OpenRouter endpoints API, list prices including provider discounts, as of October 20, 2026.

Here’s the cheapest endpoint at each precision for five models, in dollars per million blended tokens. Two are exceptions. gpt-oss-120b costs more at 8-bit than at 16-bit, and MiniMax M3 costs more at 4-bit than at 8-bit.

Model	16-bit	8-bit	4-bit
Qwen3.8 27B	$0.581	$0.342	$0.140
Kimi K2.6	$1.400	$1.079	$0.825
Gemma 4 31B	$0.205	$0.200	$0.160
gpt-oss-120b	$0.067	$0.100	$0.060
MiniMax M3	None	$0.388	$0.412

Source: OpenRouter endpoints API, as of October 20, 2026.

The provider affects the price more than the precision does. Where one model has several endpoints at the same precision, the most expensive was a median 1.72 times the cheapest. For Qwen3.8 27B, the ten fp8 endpoints ranged from $0.342 to $1.137 per million blended tokens. To send a request to the cheapest endpoint that passes your precision filter, add :floor to the model slug, as our lowest-cost inference guide explains. With quantizations: [“fp8”], qwen/qwen3.8-27b:floor went to Chutes, the cheapest fp8 endpoint, on both of our tries.

In our test, the cheapest full 280-task run was at fp4 and the most expensive at fp16, but the order in between didn’t follow precision. One run cost $0.58 on Darkbloom (fp4), $0.72 on AkashML (fp8), $0.77 on ModelRun (fp4), $0.81 on Parasail (fp8), $1.19 on CoreWeave (fp8) and $1.24 on Cerebras (fp16). Speed didn’t follow precision either. Cerebras (fp16) and ModelRun (fp4) answered in a median 1.3 and 2.4 seconds, against 8.4 to 15.0 seconds for the other four. Across all endpoints, median 24-hour uptime was 99.8% or higher at every labeled precision except fp32, which has one endpoint.

How do you choose the precision you get?

Check what a model’s endpoints serve, then set provider.quantizations on your requests. A GET to https://openrouter.ai/api/v1/models/qwen/qwen3.8-27b/endpoints returns one entry per endpoint with its quantization, price, context length and uptime. It answered without an API key when we called it, and the model’s providers tab shows the same list. For Qwen3.8 27B it listed 21 endpoints, with one bf16, one fp16, ten fp8, three fp4 and six unknown. Check the count for the precision you want before you pin it, because that count is how many endpoints can take over when one fails.

To keep requests at 8 bits or higher, list the precisions you accept.

import os
import requests

response = requests.post(
    "https://openrouter.ai/api/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
    json={
        "model": "qwen/qwen3.8-27b",
        "messages": [{"role": "user", "content": "What is the capital of Australia? One word."}],
        "reasoning": {"enabled": False},
        "provider": {"quantizations": ["fp8", "bf16", "fp16"]},
    },
    timeout=120,
)
data = response.json()
if "error" in data:
    raise RuntimeError(data["error"])
print(data["provider"], data["choices"][0]["message"]["content"], f'${data["usage"]["cost"]:.6f}')

We ran it three times, and each run printed the same line.

Reka Canberra $0.000007

The provider routing quantization docs list every accepted value and explain how quantizations combines with only, order and sort.

quantizations filters the list of endpoints. We drop every endpoint whose precision isn’t in the list, then load-balance across the rest, weighted toward the lowest price, and fall back to the next endpoint if a provider fails. Nothing in the filter guarantees the same provider each time, so log the response’s provider field if you track quality. The docs also note that fp4 includes mxfp4 and nvfp4, and fp8 includes mxfp8.

Common problems with precision filters and how to fix them

The filter runs before any provider is contacted, so an empty candidate list fails fast. Asking Qwen3.8 27B for int8, which no endpoint serves, returned a 404 in 0.4 seconds.

{"error":{"message":"No endpoints found for the request with quantization: int8. To learn more about provider routing, visit: https://openrouter.ai/docs/guides/routing/provider-selection","code":404,"metadata":{"routing_funnel":[{"step":"Initial Endpoints","endpoint_count":21}],"failed_routing_step":"Filter by Quantization"}}}

The routing_funnel field shows how many endpoints survived each step, and failed_routing_step names the step that left none. {“quantizations”: [“fp8”], “only”: [“cerebras”]} returned the same 404 with quantization: fp8, because Cerebras serves fp16, so the funnel went from 21 endpoints to 1 to none.

The slower failure happens when only one endpoint matches. Cerebras is the only fp16 endpoint for Qwen3.8 27B. During our test, which pinned Cerebras with allow_fallbacks: false, 93 of the first 280 requests failed three times in a row with this error (abridged).

{"message":"Provider returned error","code":429,"metadata":{"raw":"qwen/qwen3.8-27b is temporarily rate-limited upstream. Please retry shortly, or add your own key to accumulate your rate limits: https://openrouter.ai/settings/integrations","provider_name":"Cerebras","limit_source":"upstream_provider_shared_pool","retry_after_seconds":59}}

We retried them until all 280 completed. quantizations: [“fp16”] on its own leaves the same single candidate, so a rate limit there has nowhere else to go. Widening the list to [“fp16”, “bf16”] gave the router a second candidate, and the same request went to DeepInfra.

A provider failure can also arrive with an HTTP 200 status, because we send the status as soon as the provider accepts the request. Check the body for error, not only the status code.

Sort endpoints by quality signals with Exacto

Quantized quality depends “on the method used”, and each provider runs its own inference software, so two endpoints with the same label aren’t guaranteed to match. For requests with tools, Auto Exacto already moves endpoints with lower throughput, tool-calling success rates and benchmark scores to the back of the list. To apply that ordering to requests without tools, add :exacto to the model slug. Exacto ranks endpoints on tool-calling signals, not on accuracy at your task, so run your own prompts against the endpoints you’re considering. How to evaluate LLM provider performance covers how to compare their latency, throughput and uptime.

Frequently asked questions

Is higher or lower quantization better?

More bits keep more of the original model’s precision and take more memory. In our test of Qwen3.8 27B on October 20, 2026, we didn’t detect an accuracy difference between fp8, fp4 and fp16 endpoints on 240 tasks with exact answers. That covers one model on one day, so test 4-bit on your own workload before you commit to it.

Why doesn’t quantization destroy an LLM’s accuracy?

Not all weights matter equally. AWQ found that protecting about 1% of the most important weights greatly reduces quantization error, and LLM.int8() uses mixed precision, keeping outlier values at 16-bit while multiplying more than 99.9% of values in 8-bit. Our test of Qwen3.8 27B can’t show which methods the providers use, but the fp16 endpoint missed 3 of 240 tasks and the fp8 and fp4 endpoints missed 2 to 4.

When should you quantize a model?

Quantize when the model doesn’t fit your hardware at 16-bit, or when you need more throughput from the same GPUs. In our test we didn’t detect a loss at fp8, and fp8 rounds each weight less than 4-bit does, so it’s the smaller step. Test either one on your own tasks first. Through an API you don’t quantize the model yourself. You choose an endpoint on OpenRouter that serves the precision you want, with provider.quantizations.

What does “unknown” quantization mean on a provider?

It means the provider hasn’t reported a precision for that endpoint. On OpenRouter, 865 of 1,455 endpoints were unknown on October 20, 2026, and 610 of those belong to closed-weight models such as Claude and Gemini. unknown is also an accepted value in provider.quantizations, so you can include or exclude those endpoints.