Models, Context, and Reliability

The Railway deployment delegates chat, vision, and embedding inference to the RapidDraft Local AI Server. LiteLLM exposes authenticated OpenAI-compatible model aliases; separate llama.cpp services execute the models.

Active model routes

Verified on 2026-07-21.

Alias	Model	Use	llama.cpp context	Parallel slots
local/qwen-coder	Qwen3-Coder-Next Q4_K_M	Chat, Agent reasoning, titles, MCP tool selection	32,768	1
local/qwen-vision-fast	Qwen3-VL-8B-Instruct Q4_K_M	Image/vision requests	8,192	1
local/embed-engineering	Qwen3-Embedding-0.6B Q8_0	RAG and Bauer vectors, 1,024 dimensions	8,192	2

The server is a Fedora host with an AMD Ryzen AI Max+ 395, Radeon 8060S, 16 cores/32 threads, and 128 GB of unified physical memory. Firmware reserves 96 GiB of that shared memory for the GPU, leaving Linux approximately 30 GiB as ordinary CPU-visible RAM; these are two views of one physical memory pool, not separate 96 GB VRAM and 30 GB system-RAM modules. The host also has 8 GiB swap.

At the 2026-07-21 measurement, the GPU reported approximately 55.3 GiB used out of its 96 GiB reservation, while Linux reported approximately 25 GiB of the CPU-visible region available. The coder model directory occupies approximately 45 GB on disk; disk size, process resident memory, GPU allocation, and context capacity are different measurements.

Context budget

Layer	Limit
Physical Qwen coder context	32,768 tokens
LibreChat Agent working context	24,000 tokens
Output allowance	2,048 tokens
Individual tool result	6,000 characters

The 24,000-token Agent setting intentionally leaves headroom below the physical model window. Context tokens measure text inside one chat request; they do not measure memory capacity. The stock Bauer filename inventory alone consumed 13,270 Qwen tokens. The deployed batch-search overlay removes that inventory from model-visible instructions and leaves only a compact corpus count.

The reported request (34149 tokens) exceeds the available context size (32768 tokens) failure occurred before these safeguards were applied. The working-context and output limits reduce general multi-turn growth; the batch-search overlay separately removes the large filename-instruction cost.

Current concurrency

LibreChat permits at most two concurrent messages, but local/qwen-coder has one inference slot. Requests therefore queue behind the same large local model. RAG embedding has two slots and is not the primary chat bottleneck.

The live LiteLLM configuration currently has a 600-second request timeout. A burst test also observed a 429 response associated with a 200,000-token rate budget and poor recovery of the queued Bauer request. The practical weakness is therefore not answer quality; it is admission control and recovery when several long Agent prompts reach a single model slot.

Local-only hardening

Apply these changes in order and benchmark after each one:

1. Keep the deployed compact filename and batch-retrieval path under regression test.
2. Reduce the LibreChat message concurrency limit to match the single Qwen coder slot, or place a bounded FIFO queue in front of LiteLLM.
3. Set a shorter request timeout for interactive traffic and fail clearly instead of occupying the queue for ten minutes.
4. Configure limited retries only for transient connection failures. Do not retry context errors or sustained 429 responses blindly.
5. Add a short cooldown after a local-model failure and expose queue depth and inference latency.
6. Use separate scoped keys and budgets for interactive UI, automated tests, and background ingestion.
7. Benchmark a smaller chat model for fast document questions while retaining Qwen3-Coder-Next for complex tool use.
8. Test --parallel 2 only after measuring unified-memory allocation, prompt throughput, time to first token, and answer quality.

Cloud and hybrid options

Design	Behaviour	Advantages	Trade-off
Local only	All chat and embeddings stay on the Fedora host	Maximum data control and predictable provider cost	One host and one main inference slot
Cloud only	LibreChat calls a managed model provider	Elastic concurrency and simpler recovery	Data leaves the local inference boundary; variable cost
Hybrid	LiteLLM uses local primary and an approved cloud fallback	Keeps normal traffic local while protecting demonstrations from outages and queue saturation	Requires explicit data policy, fallback observability, and provider credentials

The recommended demonstration architecture is hybrid after legal and data-location approval:

• local model as primary;
• EU-hosted Azure OpenAI or an EU AWS Bedrock region as fallback;
• fallback only for timeout, unavailable, or controlled overload conditions;
• no fallback for authorization, malformed request, or context-limit failures;
• visible logging of which provider answered each request;
• a per-Agent policy to disable cloud fallback for restricted customer data.

No cloud fallback is currently configured.

Relationship to RAG V2

Model-serving tuning and retrieval tuning solve different problems. The controls on this page reduce queueing, context overflow, and local-model failure. The private RAG V2 deployment improves which evidence reaches the model through exact, lexical, vector, and table-aware retrieval.

RAG V2 keeps its interactive cost bounded: candidate searches run in parallel, a bounded deterministic reranker examines only the best results, and deterministic validation avoids a second large-model call on ordinary questions. The measured private-development p95 is 1.519 seconds. No remote reranker is currently configured, so a separate Local AI model benchmark is still required. See RAG V2 Target Architecture.

Monitoring targets

Record at least:

• request count by model and caller class;
• queue depth and queue wait;
• time to first token and total latency;
• provider input/output tokens;
• 429, timeout, context, and tool-call error counts;
• fallback attempts and outcomes;
• model process memory and restart count.

Open Questions

• What maximum interactive queue time is acceptable during a customer demonstration?
• May Bauer demonstration content use an EU cloud fallback?
• Which smaller local model meets the required German/English retrieval and tool-calling quality?

Sources

• Local AI systemd service definitions and host status captured on 2026-07-21
• Local llama.cpp memory, queue, and token metrics captured on 2026-07-21
• D:\02_Code\LibreChat_Setup\librechat.yaml
• D:\02_Code\LibreChat_Setup\services\librechat-custom\fileSearch.js
• D:\02_Code\LibreChat_Setup\README.md
• Local AI model catalog
• D:\02_Code\00_Project_Management_n_skills\01_tracks\rapiddraft-studio\plans\260720_librechat-rapiddraft-integration\README.md