RAG evidence coverage: 66.25% → 91.25%, with 87% less time after tuning

01 Background

Existing retrieval missed needed evidence and attached evidence even to unanswerable questions, leading to wrong answers. I believed accurate information is this project's core, so I checked each step: the correct sources were all in the initial candidates, but limiting checks to the first 8 pages dropped them for 76 of 90 questions. I needed a step that picks the right evidence and says when there is none.

02 Experiments and improvements

1. Routing comparison and the decision to try RAG

   Problem Some questions were misclassified as document edits, leading to the wrong action. Task types and additional conditions needed to be selected more accurately and quickly.

   Experiment and result Applied Jev, a choice-based decision model released on September 14, to routing questions, document edits, document creation and folder organization. Across 98 requests, correct task types and additional conditions totaled 77 for the existing JSON approach, 63 for GPT-5-nano given the same options, and 81 for Jev. Median decision time improved from 8.967s to 0.658s. I measured API cost on the same 100 questions: about $0.0408 for Jev against about $0.1027 for the GPT approach given the same options.

   Decision Jev showed promise on accuracy, speed and cost alike, so I decided to apply it to evidence selection, which the existing retrieval had not solved. I chose Jev for two reasons. First, having GPT pick the evidence cost about 2.5 times as much even with the same options and took longer, too heavy for a step that checks hundreds of candidates per question; re-ranking by tuning retrieval scores did not help either, as scores stayed similar across many hybrid weights and RRF settings. Second, embeddings sometimes failed to separate differences in Korean phrasing, so fitting one weight to questions from different fields would bring only small gains. Rather than polishing the ranking further, I had a model judge the needed evidence directly from the collected candidates.

2. Full-corpus inspection: evidence coverage on all 90 answerable questions

   Problem Using the original question for retrieval and increasing semantic-score weight raised top-1 correct retrieval from 50 to 57 out of 110 questions. However, the experiment had some bias, and reliable evidence selection on new questions remained unsolved. Delivering reliable answers to general questions is the core of the project, so this quality had to improve.

   Experiment and result Tested whether Jev could find evidence missed by retrieval. It inspected all 2,020 evidence units in a test Wiki without retrieval filtering and returned at most 8, matching the existing return limit. In a semantic evaluation allowing paraphrases, coverage on 90 answerable questions rose from 24/90 (26.7%) to 90/90 (100%). It returned no evidence for all 10 unanswerable questions. Median processing time was 1.511s for the original approach and 5.338s for Jev. Jev processed 26 batches of 80, with up to 4 in parallel. Full-corpus inspection took 2,600 calls for 100 questions, and the API reported roughly 124.5M input and 6.47M output tokens. I did not confirm the billed amount. To understand the size of the gap I traced where the existing retrieval dropped the answer. The page holding the reference sentence was in the initial candidate pool for all 90 questions, but 76 fell out at the step that limits inspection to the top 8 pages, 8 at the threshold of 85% of the best in-page score, and the remaining 2 at the final 8-evidence cap and the related-page lookup.

   Decision The existing retrieval was discarding the answer before selection even began, so the gap cannot be read as a difference in selection ability over the same candidates. Full-corpus access found the evidence, but inspecting everything per question would grow calls as documents grew. Rather than set the deployment scope from this 90-question wiki alone, I decided to narrow candidates through retrieval first and let Jev pick the final evidence.

3. Controlled comparison on 500 candidates: 87.50% coverage

   Problem Checking every piece of evidence was too costly to use in the service as-is. The evaluation questions were also concentrated in certain documents and included document titles. The original search judged from only some pages while Jev saw all evidence, so it was hard to tell whether the accuracy gap came from judgment or from access scope. I had to cut candidates and match the comparison conditions.

   Experiment and result Created 100 new questions from 9 papers (80 answerable, 20 unanswerable) and fixed them before the runs. Both methods received the same 500 candidates with returns capped at 8, reference answers were excluded from inputs, and GPT and Gemini graded without seeing method names. Coverage rose from 66.25% to 87.50%, but sequential candidate processing raised median time from the original 0.431s to 9.090s.

   Decision Jev selected better evidence under the same conditions, so I kept retrieval for candidate collection and Jev for final selection. Results on public papers and self-authored questions were not treated as general accuracy, and reducing the waiting added by sequential processing became the next task.

4. 300 candidates and a parallelism comparison to reduce latency

   Problem Answers start only after evidence selection finishes, so processing time is the user's waiting time. A 9.090s median for 500 candidates processed sequentially was far too long against the previous 0.431s to be usable. Still, the higher accuracy was worth keeping, so I set out to cut the time.

   Experiment and result Since candidate count drives cost and speed, I cut candidates from the top 500 to 300 and ran up to 4 batches in parallel. Time fell to 2.251s, while questions with all required evidence increased from 70 to 73. The parallel count had been set without evidence, so I re-ran the same 100 questions with only parallelism changed to 4, 6, 8 and 10. Median selection time was 2.066, 1.165, 0.945 and 0.886s, and blind grading of the 42 questions whose selections differed gave 37, 38, 36 and 37 fully supported, with no real difference.

   Decision Beyond 8 the gain was only 0.06s, and in production concurrent requests grow as workers × parallelism, so I chose 8. At 1.137s including candidate filtering it is still slower than the original 0.431s; I rejected cutting candidates further because some required evidence ranked below 250. I adopted Jev despite the added latency because accurate evidence is central to the service.

5. Comparing candidate generators to find the limits of the production setting

   Problem Jev can only choose from what reaches it, so evidence missing from the candidate pool is unreachable. I needed to know how far the production candidate generator preserved the answer-bearing evidence.

   Experiment and result Reusing stored scores, I regenerated rankings for four candidate generators and compared how well each preserved the 97 pre-registered citation groups across the 80 answerable questions: BGE-M3 alone, the production hybrid (0.75 embedding + 0.25 normalized BM25), RRF, and an alternating union. At 100 candidates, the questions that preserved every citation were 74/80 for BGE-M3 alone, 63/80 for the production hybrid, 63/80 for RRF and 65/80 for the alternating union. MRR@500 over each citation’s best rank was 0.1811 for BGE-M3 alone against 0.1245 for the production hybrid, with median ranks of 19 and 40. Beyond 300 candidates the four converged to 79/80 and 78/80.

   Decision The production hybrid weighting ranks answer-bearing evidence lower than plain BGE-M3. At the current 300 candidates the gap is small, so I left production unchanged and recorded it as the first condition to fix before cutting candidates further.

03 Improved processing flow

flowchart TD
 Q[User question] --> S[Collect semantic and keyword search candidates]
 S --> C[Top 300 candidates]
 C --> J[Jev selects evidence in up to 8 parallel batches]
 J --> E{Sufficient evidence?}
 E -->|Yes| A[Pass selected evidence to answer generation]
 E -->|No| N[No evidence]

flowchart TD
 Q[User question] --> S[Collect semantic and keyword search candidates]
 S --> C[Top 300 candidates]
 C --> J[Jev selects evidence in up to 8 parallel batches]
 J --> E{Sufficient evidence?}
 E -->|Yes| A[Pass selected evidence to answer generation]
 E -->|No| N[No evidence]

04 Results and lessons

Questions with sufficient evidence: 53/80 → 73/80; total median time 1.137s

On our own golden set, I raised the share of 80 answerable questions whose needed facts were fully supported from 66.25% to 91.25%. On a separate set of 20 questions, I cut evidence wrongly attached to documents with no answer from 20 to 0. It is still slower than the original selector, but I judged accuracy mattered more to the service and decided to adopt Jev. I also applied it to grouping expressions with the same meaning. Correct answers on 76 questions stayed similar (71 original, 72 Jev), and I confirmed median decision time could drop from 8.162s to 0.254s.