It is a dedicated machine that serves open-weight models to my own tools over an OpenAI-compatible API. I patch the inference engine myself, watch tok/s while a job runs, and pick a quantization level instead of picking an invoice.
Running it here means a prompt never leaves the room and nothing gets rate-limited mid-debug. The failures are mine too: when something crawls I profile it and keep the patch, which is how the batching work on this engine happened at all.
- Local inference
- EXL3
- exllamav3
- OpenAI-compatible API
- 256k context
- Machine
- GX10, a dedicated inference node on my home network
- Model
- Qwen3.8-Flash-Next quantized to 3.05 bpw EXL3, with its MTP draft head
- Engine
- exllamav3, a local fork carrying two patches of mine
- Context
- 262,144-token window
- Measured
- about 1k tok/s cold prefill, 50 to 60 tok/s decoding
- Reachable from
- this network, and nothing past it