LLM TO SYSTEM 1 v0.1

A small decision.
A clear result.

Give your model a question.
Give your code a typed answer.

Turn local GGUF model scores into choices, booleans, and ordered values—with an explicit option to abstain.

MIT application codeCPU & CUDABring your own model

decision / storage_zone01

INPUT STATEJSON

{
  "storage_requirement": "chilled",
  "hours_until_dispatch": 4
}

↓LOCAL GGUF MODEL

AAmbient—

BChilled↗

CFrozen—

↓TYPED RESULT

{"selected": "chilled"}

Illustrative output · no model runs in this browser

01 Your data stays with your runtime02 Scores you can inspect03 Uncertainty you can handle

01 / THE CONTRACT

Small outputs.
Useful building blocks.

Define the options. Read their next-token scores. Apply a policy. Return a value your application understands.

A → B → C

Choose an option.

Route a shipment, select a category, or pick the next action. Your semantic IDs stay yours.

{ "selected": "chilled" }CHOICE ↗

0 / 1

Answer a condition.

Turn a two-option question into a boolean, with the candidate-relative probability alongside it.

{ "value": true, "p_true": 0.92 }BINARY ↗

▂ ▄ ▆

Find the level.

Use your own ordered levels and numeric values for priority, severity, or another defined scale.

{ "selected": "high" }ORDINAL ↗

02 / MAKE ROOM FOR UNCERTAINTY

A result can
also be “wait.”

Move the acceptance threshold to see a typed decision become an abstention. Your application decides what happens next.

ⓘ

Illustrative scoring demo. These fixed probabilities are examples, not model inference or accuracy estimates. Candidate mass is fixed at 0.98, above the 0.05 policy threshold.

Which storage zone?

Accepted

AAmbient

BChilled

96%

CFrozen

Minimum top probability80%

More acceptingMore selective

RESULT.JSON

{
  "value": {
    "type": "choice",
    "selected": "chilled"
  },
  "abstention_reasons": []
}

Candidate-relative probability is not a probability of correctness.

03 / YOUR MODEL, YOUR MACHINE

A common contract.
Different models.

Use a compatible local chat GGUF. Embedded Jinja templates handle model-specific formatting; Qwen3 keeps its non-thinking profile.

Gemma 4E2B Instruct · Q8_0 Gemma 31B IT · Q8_0 Qwen30.6B · Q8_0 SmolLM2135M Instruct · Q8_0 TinyLlama1.1B Chat · Q4_K_M

Specific checkpoints tested locally. Compatibility still depends on the template, tokenizer, libllama build, and available memory. Read the verification record ↗

04 / MEASURE THE WORK

Performance,
with the context attached.

A local inference test measures loading, first-request latency, steady-state latency, and throughput. Use your own checkpoint and hardware.

Local measurementsRTX 3080 · release build · 5 samples · 3 decisions/request

Warehouse latency and typed-decisions quality with separate measurement scopes
Checkpoint	CPU p50 / request	CUDA p50 / request	CUDA decisions / sec	CUDA abstentions	Typed raw accuracy	Typed coverage	Typed accepted accuracy	Typed correct / all	Typed p50 / p95
Gemma 4 E2B Q8_0	7537.0 ms	96.8 ms	30.69	0%	54.30%	92.75%	55.69%	51.65%	322.6 ms / 371.0 ms
Gemma 3 1B Q8_0†	2796.4 ms	61.8 ms	46.44	0%	—	—	—	—	— / —
Qwen3 0.6B Q8_0	2334.4 ms	63.5 ms	47.83	0%	31.25%	55.60%	34.35%	19.10%	173.5 ms / 204.2 ms
SmolLM2 135M Q8_0	720.4 ms	52.2 ms	56.82	100%	—	—	—	—	— / —
TinyLlama 1.1B Q4_K_M†	3797.0 ms	49.4 ms	60.95	100%	—	—	—	—	— / —

English warehouse fixture · 2026-09-21. Five samples are a smoke measurement, not a broad benchmark. Inference timings include prompt processing and scoring, not generated text. Hardware, quantization, and build differences matter. Throughput includes abstentions. † Gemma 3 and TinyLlama exceed the CUDA batch-consistency tolerance; these timings use a fixed batch of 256.

Typed-decisions: 400 cases / 2000 decisions · RTX 3080 10 GiB, WSL2 · 2026-09-26. Raw accuracy ignores the abstention policy; coverage is the accepted share; accepted accuracy uses accepted decisions only; correct / all counts accepted correct answers over all decisions. Typed latency is per five-question case, excluding model load and warmup; it is separate from the warehouse timings. This is a local L2S1 run, not an Ollaya runtime speed comparison. Protocol and report ↗

Download measured evidence: Gemma summary JSON · Qwen summary JSON · Gemma scored JSONL · Qwen scored JSONL · Manifest

Run the performance test on your machine

05 / START LOCAL

Your next decision
starts here.

Bring a GGUF checkpoint and a matching llama.cpp build. Define a question, run the CLI, and inspect the JSON.

Download the English example

TERMINAL / QUICK START

cargo run --release --features llama -- \
  --model /path/to/chat-model.gguf \
  --input examples/warehouse.json \
  --device cpu

Open source code. Separate model licenses.

The application is MIT-licensed. Model weights are never bundled. Gemma 4 uses Apache 2.0; Gemma 3 has its own terms. Preserve applicable third-party notices when distributing.

License details ↗

A small decision.A clear result.

Small outputs.Useful building blocks.