Skip to content

RAG V2 Indexing and Retrieval

Status: Additive V2 index deployed for private evaluation; not promoted

RAG V2 preserves the existing source files while building a second, versioned search index. The index is designed for engineering documents where an exact identifier, table row, unit, or certificate number can be more important than general semantic similarity.

Current implementation

The active bauer-rag-v2-2026-07 index contains all 373 authorized Bauer files in the private V2 Agent namespace. Corpus inspection produced 6,111 chunks, including 1,643 table rows, and 30,899 deterministically extracted entities. The extractor is bauer-deterministic-v2.1; the embedding version is local/embed-engineering-1024.

Exact metadata, PostgreSQL lexical, and pgvector semantic searches run concurrently. Results use reciprocal-rank fusion, source/location deduplication, a bounded deterministic multilingual reranker, and final file-ID filtering. The remote reranker interface exists but is not configured or claimed as evaluated. V1 tables and vectors remain unchanged.

Flattened Markdown does not always associate a page heading with a later table row. This currently affects the provisional section labels for cases B07, B18, and B24 and must be adjudicated before formal evaluation is locked.

What V2 indexes

Each V2 record must retain an authorization and provenance envelope:

Field group Required examples
Access knowledge-base ID, Agent/entity namespace, file ID
Source identity checksum, filename, document title, source type, language
Location page, section path, table title, row identifier
Product metadata model name, product family, part number, component category
Document metadata document number, certificate, revision, publication date
Organization metadata company, address, location
Engineering metadata pressure, capacity, medium, units, standards
Search content normalized lexical text, original passage, embedding vector
Index lifecycle extractor version, embedding version, index version, indexed time

Not every document contains every field. Missing metadata must remain missing rather than being invented from nearby documents.

Normal text chunks

Normal prose should be split along document structure rather than only by character count. Each chunk should include:

  • the document title;
  • the heading path;
  • the page or source location;
  • enough neighbouring text to preserve meaning;
  • extracted identifiers and engineering values;
  • a stable chunk ID derived from the source checksum and location.

Chunk overlap should be measured during evaluation. Excessive overlap increases storage and can fill the top results with near-duplicates; too little overlap can separate a claim from its qualifier.

Table-aware chunks

Technical tables must not be flattened into unrelated text fragments. A table-aware record should preserve:

  • document and table title;
  • column headers;
  • row label and complete row values;
  • header-level units;
  • page and section;
  • footnotes and qualification marks that apply to the row;
  • repeated headers needed to understand a row independently.

Example logical record:

Document: Compressors for Industry EN
Section: Technical data / K 25 - K 28
Table: Operating range
Row: K 28
Columns: Medium=Air; Minimum pressure=...; Maximum pressure=525 bar
Footnote: ...
Page: ...

The stored form may be structured JSON plus a searchable rendering. The original row and source location must remain available for citation.

Metadata extraction

Use deterministic extraction first:

  • regular expressions and boundary-aware tokenization for model names, document numbers, part numbers, revisions, pressure values, capacities, dates, and standards;
  • document headings and PDF/HTML structure for titles, sections, and tables;
  • controlled dictionaries for known Bauer product families and certificate vocabulary;
  • language-aware normalization for German and English variants.

If model-assisted extraction is later added, it must:

  • emit structured fields with the supporting source span;
  • use a versioned extractor prompt and model;
  • keep confidence and review state;
  • never overwrite the raw extracted text;
  • be re-runnable without uploading the file again.

Identifier normalization

Exact search requires both an original and a normalized representation:

  • preserve punctuation and capitalization for display;
  • store a case-folded comparison form;
  • normalize harmless separators where appropriate;
  • do not merge identifiers merely because they look similar;
  • keep units separate from numeric values;
  • retain document language and revision.

For example, B-DETECTION PLUS, B DETECTION PLUS, and a known localized spelling may be searchable variants of one product name. K 25 and K 28 must remain distinct models.

Retrieval pipeline

1. Query analysis

Parse explicit identifiers, numbers, units, quoted phrases, language, and likely field targets. This should be deterministic where possible and must not weaken an explicit constraint.

2. Candidate generation

Run three bounded searches:

  • exact and normalized metadata lookup;
  • PostgreSQL lexical/full-text and trigram search;
  • pgvector semantic search.

Each search returns source IDs, locations, raw scores, and channel information. All searches apply the same authorized file-ID and knowledge-base namespace filter.

3. Fusion and deduplication

Combine the channels without assuming that their raw scores are directly comparable. Reciprocal rank fusion is the recommended starting point because it operates on rank rather than incompatible score scales.

Exact identifiers should receive a controlled priority boost, not an unconditional bypass of authorization or evidence requirements.

Near-duplicate chunks from the same document and location should collapse into one candidate while retaining the reasons it matched.

4. Reranking

Send only the best approximately 20 to 30 candidates to a small reranker. The reranker should return the best approximately five to eight evidence items.

The selected model must be benchmarked for:

  • German and English queries;
  • alphanumeric model names;
  • numerical and unit-sensitive questions;
  • table rows;
  • latency and memory use on the Local AI Server.

The large Qwen chat model should not be used as the default reranker.

5. Evidence package

Each result returned to LibreChat should contain:

  • source file and title;
  • page, section, and table/row location when available;
  • original evidence text;
  • relevant structured metadata;
  • retrieval channels and final rank;
  • authorization-safe file ID;
  • citation identifier;
  • source type, including the distinction between public evidence and synthetic demo data.

The model-visible result must be compact. Debug scores and internal metadata should be logged or available to the evaluation runner without consuming the answer context.

Database layout

V2 should use new versioned tables or a clearly versioned namespace in the existing Railway PostgreSQL service. Final physical table names should be chosen during the schema-design phase, but the logical separation must cover:

  • document metadata;
  • chunks and table rows;
  • extracted entities;
  • lexical indexes;
  • embedding vectors;
  • index runs and extractor versions.

V1 tables remain untouched. V2 migrations must be additive and reversible without deleting the source S3 objects or MongoDB file associations.

Incremental indexing

An index run should compare source checksums:

  • unchanged file: keep the existing V2 records;
  • changed file: create the new version, validate it, then retire the previous active version;
  • deleted Agent association: make the content unavailable to that Agent without relying only on later filtering;
  • extractor or embedding change: create a new index version and switch only after validation.

A failed index run must not leave a partially active corpus.

Performance controls

Recommended starting limits:

Control Starting point
Exact candidates 20
Lexical candidates 30
Vector candidates 30
Fused candidates sent to reranker 20-30
Evidence items returned to Agent 5-8
Citations in final answer Maximum 8
Citations per file Maximum 2

These are experiment starting points, not production facts. Final values must come from the evaluation protocol.

Exact, lexical, and vector candidate generation should run in parallel. Query embeddings should be cached only with a bounded, non-sensitive key and invalidated when the embedding model changes.

Failure behaviour

  • If one search channel fails, log it and return a degraded result only when the remaining evidence is sufficient and the response clearly records the degraded state.
  • If authorization cannot be resolved, fail closed.
  • If the reranker fails, use a documented fusion-only fallback rather than arbitrary ordering.
  • If the validator rejects a high-risk claim, qualify or refuse the answer.
  • If no evidence passes the minimum threshold, report that the answer was not established in the indexed Bauer sources.

Open Questions

  • Which PDF/table parser best preserves the current Bauer corpus without introducing a new operational service?
  • Should lexical retrieval use PostgreSQL full-text search alone or combine full-text and trigram indexes?
  • Which multilingual reranker fits the Local AI memory and latency budget?
  • How should conflicting document revisions be ranked and displayed?
  • Which metadata fields require Bauer-approved dictionaries before production use?

Sources