Skip to content

RAG V2 Engineering Plan

Status: Private implementation deployed; formal evaluation and promotion pending Implementation state: Phases 0-4 and private rollout complete; phase 5 gated by gold review Operational source: D:\02_Code\00_Project_Management_n_skills\01_tracks\rapiddraft-studio\plans\260723_bauer-rag-v2\README.md

RAG V2 has been built alongside the deployed V1 retrieval path. The endpoint, additive index, private V2 Agent, executable evaluation harness, live isolation checks, and rollback anchors are in place. The work is not complete for promotion: the gold set remains provisional and the formal five-run end-to-end and holdout evaluation has not begun.

Delivery principles

  • Keep /query_multiple and the current Bauer Agent working throughout the project.
  • Add no new Railway service.
  • Reuse the existing source objects, file IDs, authorization model, RAG API, and PostgreSQL service.
  • Build derived V2 indexes under a new version; do not rewrite V1 data.
  • Use the same local Qwen answer model for V1 and V2 comparisons.
  • Make the benchmark and gold evidence executable and version-controlled.
  • Promote V2 from measured evidence, not from a small number of attractive examples.

Phase 0 - freeze the baseline

Deliverables:

  • record the 373-file Bauer corpus manifest and source checksums;
  • record the deployed Git commit, Railway deployment IDs, Agent IDs, model aliases, and settings;
  • preserve the existing Bauer benchmark as the V1 baseline;
  • convert the benchmark cases and verified evidence into machine-readable development and holdout sets;
  • capture V1 retrieval and end-to-end results before tuning V2.

Exit gate:

  • another engineer can reproduce the V1 benchmark from a clean conversation and identify the exact corpus, code, model, and configuration used.

Phase 1 - V2 schema and indexing

Deliverables:

  • additive V2 PostgreSQL schema and migration;
  • metadata field specification;
  • normal prose and table-aware chunk formats;
  • deterministic metadata extraction;
  • index-run, checksum, and version tracking;
  • incremental indexing and failed-run recovery;
  • unit tests using representative German, English, table, certificate, and identifier examples.

Exit gate:

  • the complete Bauer corpus can be indexed without altering V1, and sampled records retain correct file, page, section, table, unit, and authorization provenance.

Phase 2 - hybrid retrieval

Deliverables:

  • exact metadata search;
  • lexical full-text and fuzzy literal search;
  • vector search;
  • parallel candidate generation;
  • fusion, deduplication, and bounded result counts;
  • /query_v2 with the same authorization boundary as /query_multiple;
  • retrieval debug output for the evaluation runner.

Exit gate:

  • retrieval-only development tests show improved Recall@5 on exact and table questions with zero cross-knowledge-base results.

Phase 3 - reranking and evidence validation

Deliverables:

  • benchmarked multilingual reranker;
  • bounded top-candidate reranking;
  • compact evidence package;
  • deterministic checks for citations, quotations, identifiers, numerical claims, source type, and mandatory constraints;
  • documented fusion-only fallback if the reranker is unavailable;
  • selective model-based validation experiment, disabled by default until justified.

Exit gate:

  • the reranker materially improves development-set accuracy without exceeding the latency budget, and the deterministic validator catches the seeded unsupported-claim cases.

Phase 4 - LibreChat V2 Agent

Deliverables:

  • private Bauer Kompressoren - RAG v2 Test Agent;
  • server-side routing to /query_v2;
  • same Bauer files, group permissions, model, context settings, instructions, and Bauer Twin MCP tool as V1;
  • admin-only visibility during evaluation;
  • V1 Agent left unchanged.

Exit gate:

  • fresh V1 and V2 conversations can run the same prompt against the same corpus, and logs prove which retrieval route handled each request.

Phase 5 - formal evaluation

Run:

  1. retrieval-only V1 versus V2;
  2. end-to-end V1 versus V2 versus Codex;
  3. frozen-evidence Qwen versus Codex to separate retrieval and answer-model effects;
  4. five runs per prompt and system;
  5. development tuning followed by a blind holdout run.

The protocol and promotion gates are defined in RAG V2 Evaluation Protocol.

Exit gate:

  • raw runs, aggregate metrics, per-case failures, latency, and safety-gate results are committed and published without changing the gold answers after the holdout run starts.

Phase 6 - promotion and observation

If V2 passes:

  • route the normal Bauer Agent to V2;
  • keep V1 available as a hidden rollback Agent;
  • record the deployment and index version;
  • run health, isolation, benchmark, and rollback checks;
  • monitor retrieval latency, answer latency, failure rate, and validator rejections;
  • update the current-state, data, model, demonstration, and operations pages.

If V2 fails:

  • leave the normal Agent on V1;
  • publish the failed gate and affected categories;
  • continue development against the development set;
  • do not weaken safety gates to produce a passing score.

Promotion requirements

V2 must satisfy all hard safety gates and the following starting targets:

Measure Target
Exact identifier/document lookup At least 95%
End-to-end exact-answer accuracy At least 90%
Citation correctness At least 95%
Safe-refusal repeated runs 100%
V2 wins or ties V1 At least 80% of cases
Retrieval p95 Under 2 seconds
End-to-end p95 Under 45 seconds

In addition:

  • Recall@5 must improve materially over V1;
  • no evaluated category may become materially worse;
  • no fabricated project, part, product, document, or certificate identifier is allowed;
  • no synthetic demo record may be represented as confirmed Bauer data;
  • no incompatible medium, pressure, topology, or family result may be presented as compatible;
  • no cross-company knowledge-base leakage is allowed.

Targets may be revised before the blind holdout begins, but every revision must be recorded with a reason. They must not be relaxed after holdout results are visible.

Rollback

V1 remains the rollback path. A V2 release must be reversible by:

  1. disabling or hiding the V2 Agent;
  2. routing the normal Bauer Agent back to /query_multiple;
  3. rolling back the RAG API to its recorded compatible deployment if necessary;
  4. leaving V2 additive tables in place unless a separate reviewed migration removes them;
  5. running V1 health, isolation, and grounded-query acceptance tests.

No rollback step should require deleting source objects, changing file IDs, or re-uploading the Bauer corpus.

Definition of done

  • [x] V2 schema, indexer, hybrid retrieval, bounded reranker/fallback, and validator are implemented.
  • [x] V1 remains functional and the selector-based rollback path is verified.
  • [x] The V2 Agent uses the same authorized Bauer corpus and cannot return Test Archive evidence.
  • [x] Machine-readable gold cases, runners, scorers, raw runs, and reports are committed.
  • [ ] Development and holdout evaluations are complete.
  • [ ] All promotion and safety gates pass.
  • [ ] Production deployment and observation checks pass.
  • [x] Wiki pages describe the private deployed state and current limitations.
  • [x] The rollout record contains exact commits, deployments, index versions, open risks, and recovery steps.

Open Questions

  • Who is the Bauer adjudicator for the disputed evidence locations?
  • Which remote multilingual reranker should be hosted on the existing Local AI Server and tested against the deterministic fallback?
  • What duration and query count define the post-promotion observation period?

Sources