Skip to content

Operations Runbook

Routine health checks

Run the public checks without credentials:

Invoke-WebRequest 'https://chat.rapiddraft.ai/health' -UseBasicParsing
Invoke-RestMethod 'https://rag-api-testing.up.railway.app/health'
Invoke-RestMethod 'https://bauer-twin-api-testing.up.railway.app/health'

Expected results:

Endpoint Expected response
LibreChat HTTP 200 and OK
RAG API HTTP 200 and status: UP
Bauer Twin HTTP 200, status: ok, database_mode: postgresql, 12 projects, 75 parts, 8 documents, 74 terminology aliases

Check all Railway services and persistent volumes:

railway service list --json
railway volume list --json
railway bucket info --bucket librechat-files --environment testing --json

Do not run railway variable list --json in a shared terminal or paste its output into tickets. It includes raw secret values.

Administrator credential

The testing administrator is [email protected]. The automation credential is stored locally as a Windows DPAPI-encrypted PowerShell credential XML file outside the Git repositories.

Load it without printing the password:

$credential = Import-Clixml -LiteralPath '<secure-path>\testing-admin.credential.xml'

The XML file is:

  • encrypted for the Windows user and machine context that created it;
  • an input to the local PowerShell automation;
  • not a Railway variable;
  • not uploaded to LibreChat;
  • not portable to another Windows account or machine.

Use the repository default secure path or pass an explicit path:

.\scripts\bootstrap-librechat-admin.ps1 `
  -CredentialPath '<secure-path>\testing-admin.credential.xml'

The bootstrap script verifies /health, creates the administrator only during the controlled registration window, verifies ADMIN login, exports the encrypted credential, and restricts its Windows ACL. Registration must be returned to false after the first bootstrap.

Knowledge-base validation

From the LibreChat configuration repository:

.\scripts\provision-knowledge-bases.ps1 -SkipUploads -RunQueryTests

Expected checks:

  • administrator login succeeds;
  • Agents are private and shareable only through groups;
  • Test Archive has 41 embedded files;
  • Bauer has 373 embedded files;
  • no file ID is shared;
  • Test Archive returns an EPLAN-grounded response;
  • Bauer returns a Bauer-grounded response.

The 2026-07-21 batch-search release completed this acceptance run in 25.6 seconds. RAG logs showed one successful /query_multiple request for each of the two Agent queries, and the two chat answers consumed 5,338 prompt tokens in total on the local Qwen service. Treat these measurements as regression references rather than guaranteed service-level objectives.

Use the full command only when synchronizing corpus content:

.\scripts\provision-knowledge-bases.ps1 -RunQueryTests

The operation is resume-safe, but corpus preparation, OCR, embedding, and the local model can make a complete import long-running.

Bauer regression

The bearer token is also stored as an encrypted local credential. Load it into memory and pass it to the benchmark without printing it:

$credential = Import-Clixml -LiteralPath '<secure-path>\bauer-twin-api-testing.credential.xml'
$token = $credential.GetNetworkCredential().Password
.\.venv\Scripts\python.exe .\scripts\run-bauer-benchmark.py `
  --url 'https://bauer-twin-api-testing.up.railway.app' `
  --token $token
$token = $null

The expected result is 19/19, zero hard-filter violations, zero status violations, and German/English result consistency.

The regression set includes:

  • existing exact, similarity, part, document, and bilingual searches;
  • English and German helium safe-no-match cases;
  • an unknown-medium rejection;
  • a qualified synthetic nitrogen medium;
  • hard-pressure, medium, topology, family, and category checks.

After an MCP schema change, redeploy or restart LibreChat so its in-memory tool definition is refreshed. Run B12 and B14 from fresh Agent conversations. B12 must return SYN-BK-N2-420-500 in both languages. B14 must make one structured call, preserve helium/420 bar/booster constraints, and return no_compatible_match without a nitrogen or air substitute.

Deployment diagnostics

LibreChat or RAG is unhealthy

  1. Check railway service list --json for deployment and replica state.
  2. Inspect recent service logs in Railway or with railway service logs.
  3. Confirm MongoDB, Meilisearch, and VectorDB remain running.
  4. Check that the commit-pinned CONFIG_PATH is reachable.
  5. Confirm the last deploy used the intended pinned image digest and passed the upstream source checks.
  6. Roll back the deployment or pinned configuration commit if the failure follows a release.

Login is rejected or the account is banned

  1. Stop automated retries.
  2. Confirm the client is using a normal, stable User-Agent and is not repeatedly sending invalid credentials.
  3. Check LibreChat logs for login, non-browser, concurrent-message, or message-rate violations.
  4. Wait for the configured ban interval or remove the specific ban through an authorized administrative procedure.
  5. Correct the caller before resuming automation.

Do not disable violation protection globally as the first response.

File search fails

  1. Verify the selected Agent.
  2. Confirm the Agent still has file_search and the expected file count.
  3. Check the RAG health endpoint.
  4. Confirm the RAG OpenAPI QueryMultipleBody includes entity_id.
  5. Inspect RAG logs for /query_multiple; one normal Agent search should produce one request, not one request per file.
  6. Check Local AI embedding health and latency.
  7. Inspect the file's embedded state rather than re-uploading immediately.
  8. Resume provisioning by checksum if an upload was interrupted.

Private Bauer RAG V2 validation and rollback

Current private-evaluation identifiers:

V2 Agent             agent_pmPMcA25UXS7vznUaz-DU
V2 index run         7e23f6e1-d876-4da0-a677-9a3cdb37db36
RAG deployment       4c779152-def4-4c9a-a245-3e205a603547
LibreChat deployment 1142dfd1-fc50-477a-8cc4-7f226cc212c2
RAG rollback         5d434f9f-1111-4c64-85d6-4cb0f0ae8d1b
LibreChat rollback   154edb4c-54f1-496f-ab4d-60aac87caef8

Validate public health first, then authenticate as an approved administrator and check /v2/status plus the index-run resource. Confirm that:

  • the V2 schema is ready and only one namespace is allow-listed;
  • the index run is active, expects 373 files, and uses bauer-rag-v2-2026-07;
  • the normal and private Bauer Agents each have the same 373-file SHA-256 set;
  • the normal Bauer and Test Archive namespaces receive HTTP 403 from /query_v2;
  • a Test Archive file ID cannot return evidence from the private Bauer V2 namespace;
  • a V1 /query_multiple query and a V2 exact-document query both return HTTP 200.

Fastest rollback:

  1. Remove only the private V2 Agent ID from RAG_V2_AGENT_IDS.
  2. Redeploy LibreChat and confirm that the private Agent returns to /query_multiple.
  3. If the RAG overlay is unhealthy, restore the recorded RAG deployment above.
  4. Verify normal Bauer V1 retrieval and Test Archive isolation.
  5. Leave bauer_rag_v2 in place for diagnosis; dropping it is a separate destructive change.

Batch-search rollback

  1. Record the failing LibreChat and RAG deployment IDs and relevant logs.
  2. Roll back LibreChat and RAG together to their recorded compatible deployments.
  3. Verify both health endpoints.
  4. Run the no-upload knowledge-base acceptance command.
  5. Do not delete or re-ingest files; the overlay release has no data migration.

Known pre-overlay rollback deployments:

LibreChat  0dc80ad2-09fc-4a5d-b0cf-59f32a762a81
RAG API    deb0f4c3-abe7-45a4-adb6-baa85fdbfee4

Bauer terminology rollback

The pre-Option-B baseline is:

Git commit          65504fefc524ab6878d30a6c9d86fc9b35ae3ac1
Railway deployment  c8cdd391-63d7-412e-b982-65d536f81c68
Local source copy   D:\02_Code\LibreChat_Setup-baseline-65504fef.zip
Archive SHA-256     A80686D522ACFC5E87CC47D919F6C79EBB101A43EFC7E1E32CCD376F41C7041E
  1. Record the failing prompt, tool arguments, result status, service commit, and deployment ID.
  2. Reproduce through the protected HTTP endpoint and MCP before attributing the failure to the Agent.
  3. Revert the responsible Git commits normally and push the existing deployment branch; do not force-push.
  4. Verify health, run the 19-query benchmark, and repeat B12/B14 through LibreChat.
  5. Leave the additive terminology_aliases table in place unless a separate reviewed database migration removes it.

Context-size error

  1. Start a new conversation.
  2. Ask one focused question.
  3. Avoid broad repeated file searches.
  4. Confirm Agent context is 24,000 and output allowance is 2,048.
  5. Inspect the provider-reported input tokens.
  6. Treat repeated fresh-chat failures as a configuration defect; do not simply increase the context limit beyond the physical 32,768-token model window.

Timeout or 429 from the local model

  1. Stop parallel benchmark or ingestion traffic.
  2. Check the Qwen coder queue and LiteLLM logs.
  3. Wait for the active request to finish or fail before retrying once.
  4. Restart a service only when health or logs show it is stuck; a long queued request alone is not proof of a hung process.
  5. Record the prompt size, caller, queue time, model latency, and recovery result.

Backup recovery order

No complete restore drill has been recorded. A planned recovery should use this order:

  1. Restore PostgreSQL and MongoDB to a consistent recovery point.
  2. Restore S3 objects.
  3. Restore or rebuild Meilisearch.
  4. Deploy the matching Git-pinned LibreChat configuration.
  5. Verify Agent/group/file relationships.
  6. Run isolation and grounded-query tests.
  7. Run the Bauer structured benchmark.

Change checklist

  • [ ] Change is committed in the configuration repository.
  • [ ] No secret is present in Git diff or terminal capture.
  • [ ] Tests pass locally.
  • [ ] Deployment references the intended commit or image digest.
  • [ ] Health checks pass.
  • [ ] Both Agent file counts and isolation checks pass.
  • [ ] RAG logs show one /query_multiple request per normal Agent search.
  • [ ] If V2 is touched, /v2/status, active index, namespace denial, and foreign-file safe refusal pass.
  • [ ] The normal Bauer Agent is still V1 unless a verified formal report explicitly authorizes promotion.
  • [ ] Representative grounded queries pass.
  • [ ] Bauer benchmark passes when the change touches MCP, models, embeddings, or PostgreSQL.
  • [ ] MCP schema accepts open business vocabulary and preserves strict actions/numerical fields.
  • [ ] B12 bilingual match and B14 safe no-match pass through the production Agent.
  • [ ] Rollback target is recorded.

Sources

  • D:\02_Code\LibreChat_Setup\scripts\bootstrap-librechat-admin.ps1
  • D:\02_Code\LibreChat_Setup\scripts\provision-knowledge-bases.ps1
  • D:\02_Code\LibreChat_Setup\scripts\run-bauer-benchmark.py
  • D:\02_Code\LibreChat_Setup\README.md
  • Railway health, storage, MCP, and Agent checks captured on 2026-07-22