Filter RAG passages with Jev: 4 Nouls between retrieval and generation

Retrieval ranks passages by wording similarity – and hands the top few to your answering model, contradictions, prompt injections and all. On one demo query the planted forum injection ranks first by similarity (0.584) while the passage refuting the query's false premise sits seventh (0.509), in a 0.584–0.455 spread too narrow to separate them. TypeSafe's classifying-RAG-passages cookbook adds a second stage between retrieval and generation: four Noul questions per passage, one request each, then plain threshold logic in code decides what reaches the prompt – as evidence, as flagged conflict, or not at all. This guide summarises the recipe.

The four questions

State carries the query plus one passage (id, title, text, source type), so every question is about the pair, not the passage alone. Same four questions for every query; only the state changes:

• is_relevant – does this passage address the query's subject? (the relevance floor)
• contains_answer_evidence – does it state anything usable in a direct answer? (include, or drop)
• contradicts_query_premise – does it conflict with something the query takes for granted? (promotes to the conflict block)
• contains_prompt_injection – does it try to control the answering system? (excludes outright)

Note what is not asked: whether to include the passage. That decision lives in code, where changing policy means editing a number under review instead of rewording a question.

Route in code, first match wins

THRESHOLDS = {
    "injection_max": 0.70,   # above this the passage never reaches the prompt
    "contradicts_min": 0.70, # above this it disputes the query's premise
    "relevant_min": 0.45,    # below this it is not about the query at all
    "evidence_min": 0.55,    # above this it states something usable
}

def route(answers: dict, thresholds: dict = THRESHOLDS) -> str:
    if answers["contains_prompt_injection"] > thresholds["injection_max"]:
        return "exclude"
    if answers["contradicts_query_premise"] > thresholds["contradicts_min"]:
        return "conflicting_evidence"
    if answers["is_relevant"] < thresholds["relevant_min"]:
        return "exclude"
    if answers["contains_answer_evidence"] > thresholds["evidence_min"]:
        return "include"
    return "exclude"

Order matters: injection first because it is a security decision, not an evidence one; contradiction before evidence because a passage denying the query's premise usually states something usable too, and tested the other way round it would land in the accepted block instead of the conflict one. And re-routing is free – thresholds read stored answers, so tuning policy costs zero API calls.

What it catches

On the headline query ("Refresh tokens expire after 30 days – how do I extend that window?", a false premise), the routing table reads:

The injection clears the relevance floor (0.71) – relevance alone would have let it through – and only the 0.99 injection score drops it. The refuting passage scores 0.49 relevance / 0.51 evidence, which alone would have dropped it too; the 0.92 contradiction score rescues it into the conflict block. Nothing reaches the prompt as evidence – correct, for a question built on a false premise – and the generator answers "I don't have sufficient accepted evidence", names the conflict, and quotes the refuting passage instead of inventing a 30-day setting.

On an ordinary answered query ("How long should an access token live?") four passages reach evidence – three of them retrieved at ranks 8, 9 and 11, while similarity-top ranks 2–4 (the wrong kind of "lifetime", signing keys) all score ≤0.08 relevance and drop. The injection is excluded again at 0.99. Across six queries and 72 passages, at least two thirds of every query's dozen is excluded.

Two honest caveats

• The injection question is a filter, not a boundary: a passage under threshold still reaches the prompt, so the generator prompt must treat every passage as untrusted text regardless of score.
• Cost scales with k – one request per passage. Passages are never batched into one request, because each question is about one pair.

Accepted and conflicting evidence travel in separate prompt blocks with rules (untrusted text, cite passage IDs, report conflicts, say "insufficient" rather than guess). Merge them into one block and the generator cannot tell an answer from a denial.

Bottom line

Similarity retrieves; judgment filters; code routes; the generator only ever sees labelled evidence. Four Nouls and five comparisons buy you injection defence plus false-premise detection for the price of one request per passage. Full source: TypeSafe's classifying-RAG-passages cookbook; the "filter first, send only what the question needs" principle is failure mode 5 in the Jev 1.13 jaggedness guide.