Skip to content

RAG V2 Evaluation Protocol

Status: Development retrieval smoke executed; formal comparison awaits gold adjudication Systems: LibreChat V1, LibreChat V2, and Codex reference Primary judge: Verified gold answers and evidence, not any compared model

This protocol determines whether RAG V2 improves Bauer document retrieval without sacrificing latency, safety, citations, or consistency.

The 2026-07-23 one-repetition development smoke validates the harness and private deployment but is not the formal promotion run. It must not unlock holdout, end-to-end comparison, or normal-Agent promotion while gold_status remains provisional.

Codex is a reference competitor. It can show whether a failure is caused by retrieval or by the local answer model, but Codex does not grade itself and is not the source of truth.

Systems under comparison

System Retrieval Answer generation Role
LibreChat V1 Current authorized /query_multiple vector path local/qwen-coder Deployed baseline
LibreChat V2 Exact, lexical, vector, fusion, and reranking path Same local/qwen-coder Candidate
Codex reference Same frozen corpus and allowed tools; no web or outside knowledge Codex Reference competitor

All structured project searches continue to use the same Bauer Twin MCP/API. The document-RAG comparison must not treat a change in synthetic Bauer Twin data as a retrieval improvement.

Evaluation layers

Layer A - retrieval only

Compare V1 and V2 before answer generation.

For each prompt, record:

  • whether the gold document, page, section, and table row were retrieved;
  • the rank of the first relevant item;
  • irrelevant and duplicate results;
  • retrieval latency and errors.

This layer identifies indexing and ranking quality without chat-model variation.

Layer B - end to end

Run the same prompt through fresh V1 and V2 LibreChat conversations and through the Codex reference runner. Score the final answer, evidence, citations, constraints, refusals, failures, and total latency.

Layer C - frozen evidence

Give the same V2 evidence package to the local Qwen answer model and Codex. Neither system may search again.

This layer isolates answer-model and prompting quality:

  • both answer correctly: retrieval and local generation are sufficient;
  • Codex succeeds but Qwen fails: evidence is available and the local answer step needs work;
  • both fail: evidence, gold expectations, or question design requires investigation;
  • either system uses unsupported information: count it as an answer failure.

Benchmark set

Expand the existing 14 demonstration prompts to approximately 40 cases:

Category Examples
Exact model and part lookup Product model, alphanumeric part, exact identifier
Document and certificate lookup Title, document number, certificate, revision
Technical tables Maximum pressure, capacity, units, qualified row values
Semantic product questions Requirements described differently from source wording
Cross-document reconciliation Conflicting year, revision, or product-family statements
Similar projects Mandatory medium, pressure, capacity, family, and topology
Project comparison Deterministic differences and affected synthetic records
Combined evidence Structured synthetic project plus public document evidence
Safe no-answer Missing fact, incompatible medium, unsupported engineering conclusion
German and English Equivalent intent in both languages

Split the set before tuning:

  • approximately 30 development cases;
  • approximately 10 blind holdout cases.

The holdout prompts and gold evidence must not be used to tune retrieval thresholds, fusion weights, reranker settings, or Agent instructions.

Gold case format

Each case must define:

id: BXX
category: exact_table
prompt_en: "..."
prompt_de: "..."
expected_answer:
  required_claims: []
  optional_claims: []
  forbidden_claims: []
required_evidence:
  - file_id: "..."
    filename: "..."
    page: null
    section: "..."
    table: null
mandatory_constraints: {}
safe_refusal_expected: false
source_type: public_document

Gold answers should be written from inspected Bauer sources. Every exact numerical answer must name the supporting document location and qualification. Ambiguous or contradictory sources must be represented as such rather than resolved by guesswork.

Repeated runs

Run every prompt five times per system from fresh conversations:

40 prompts x 3 systems x 5 runs = 600 end-to-end runs

Retrieval-only runs may be repeated separately because they are cheaper and expose ranking variance.

For each case and system, report:

  • correct runs out of five;
  • answer and retrieved-document consistency;
  • timeout, tool, and retrieval error rate;
  • repeated or unnecessary search calls;
  • p50 and p95 latency;
  • input, evidence, and output size when available.

Because the current local Qwen service has one inference slot, the runners must not execute competing systems in parallel. Alternate V1 and V2 order across cases so server warming and load do not consistently favour one system.

Retrieval metrics

Metric Meaning
Recall@1, @3, @5, @10 Whether required evidence appears within the first results
Mean reciprocal rank How early the first required result appears
Exact metadata success Correct result for exact model, part, document, certificate, or revision
Table integrity Required header, row, unit, and footnote remain together
Duplicate rate Fraction of returned evidence that repeats the same source/location
p50 and p95 latency Typical and slow-tail retrieval time

Answer metrics

Metric Meaning
Exact-answer accuracy Required answer is present and correct
Claim-level correctness Supported correct claims divided by all scored claims
Grounded-claim rate Claims supported by returned evidence
Citation correctness Citation supports the adjacent claim
Citation completeness Important claims have citations
Source-type discipline Public evidence and synthetic records are labelled correctly
Constraint compliance Mandatory engineering filters remain intact
Safe-refusal accuracy Correctly refuses absent or incompatible requests without over-refusing
Fabrication count Unsupported identifiers, numbers, documents, or relationships

Quality score

Use the weighted score only after hard safety gates:

Area Weight
Answer correctness 30%
Retrieval quality 25%
Citation and grounding 20%
Completeness 10%
Refusal and constraint handling 10%
Latency and reliability 5%

A high weighted score cannot compensate for a hard safety failure.

Hard safety gates

The candidate fails promotion if any evaluated run:

  • fabricates a project, part, product, certificate, or document identifier;
  • presents an incompatible medium, pressure, topology, or compressor family as compatible;
  • presents synthetic demo data as confirmed Bauer data;
  • gives a high-severity unsupported numerical claim;
  • leaks or cites a file outside the selected knowledge base;
  • fails a required safe-refusal case;
  • silently weakens an explicit constraint to obtain a result.

Promotion gates

Recommended starting gates:

  • exact identifier/document lookup at least 95%;
  • end-to-end exact-answer accuracy at least 90%;
  • citation correctness at least 95%;
  • all safe-refusal cases pass all repeated runs;
  • V2 wins or ties V1 on at least 80% of cases;
  • Recall@5 improves materially over V1;
  • no category becomes materially worse;
  • retrieval p95 below 2 seconds;
  • end-to-end p95 below 45 seconds.

Record any gate revision before the holdout run. Do not change gates after holdout results are visible.

Fair-run controls

  • Use the same frozen 373-file Bauer corpus and file-ID allow-list.
  • Use the same Bauer Twin data and service version.
  • Use the same answer model, Agent context, output, citation, and recursion limits for V1 and V2.
  • Start every end-to-end case in a new conversation.
  • Disable external web access and outside knowledge for the Codex reference.
  • Record Git commit, deployment ID, Agent ID, index version, embedding model, reranker, and prompt version.
  • Do not run background ingestion, title generation, or acceptance tests during timed runs.
  • Preserve raw responses and retrieval results before scoring.

Failure diagnosis

Observation Most likely layer
V1 fails, V2 and Codex pass Retrieval improvement
V1 and V2 miss the gold evidence Corpus, parsing, metadata, or indexing
V2 retrieves correct evidence but V2 Agent answers incorrectly Local model or Agent instructions
Codex finds evidence but V2 retrieval does not Query analysis or retrieval routing
Answer is correct but citation is wrong Evidence mapping or citation generation
Correct result varies across five runs Ranking, orchestration, or model instability
Both systems safely refuse Confirm whether evidence is genuinely absent
Codex adds unsupported facts Codex failure; gold evidence remains the judge

Evaluation artifacts

The executable source should live in the LibreChat configuration repository:

evals/bauer-rag-v2/
|-- cases/
|   |-- development.yaml
|   `-- holdout.yaml
|-- gold/
|   |-- answers.yaml
|   `-- evidence.yaml
|-- runners/
|-- scorers/
|-- raw-runs/
`-- reports/

The wiki publishes the protocol and human-readable results. Raw artifacts remain versioned with the code and are referenced by commit and report checksum.

Open Questions

  • Who performs the final human adjudication of disputed gold answers?
  • Which cases require a Bauer engineer before they can become production-quality gold evidence?
  • Should exact German and English translations share one case ID or be scored as separate cases?
  • What cost and time budget is acceptable for the 600-run end-to-end suite?

Sources