When streaming responses from OpenAI, users frequently close their client connections (or refresh tabs) before the response completes. If your gateway does not actively monitor this, the upstream connection remains open, and you continue to pay for tokens that are never read. Setting up an openai streaming ghost billing fix is critical.
For detailed pipeline patterns, review The Complete Architecture of Cost-Efficient LLM Pipelines.
The Cost of Ghost Billing
If a user requests a 4,000-token summary but closes the connection after reading 100 tokens, the remaining 3,900 tokens are still generated and billed by OpenAI. Under load, this can waste substantial budget:
$$\text{Ghost Cost} = (\text{Total Tokens} - \text{Read Tokens}) \times \text{Output Cost per Token}$$
To fix this, you must deploy an active reverse proxy in front of your OpenAI connections. To learn how to configure this as a circuit breaker, read llm api circuit breaker reverse proxy.
---
Active Downstream Socket Audit
Our Rust gateway uses non-blocking I/O checks to monitor downstream connections. If a client terminates their socket, the gateway drops the connection to OpenAI instantly.
This is implemented using tokio's channel cancellation checks:
// Drop upstream request if client closes socket
let mut client_disconnected = req.body_mut().take_data_stream();---
Run the Ghost Billing Guard
Run the guard gateway using Docker:
docker run -d \
--name kilovolt-ghost-guard \
-p 8080:8080 \
-e KILOVOLT_PORT=8080 \
-e KILOVOLT_DEFAULT_BUDGET=1.50 \
yodsarun/kilovolt-proxy:latestAll incoming calls routed through this gateway will have automatic connection abortion active.