Retrieve first, then answer
RAG supplements generation with retrieved information. In a typical application, documents are split into passages and indexed. Relevant passages are retrieved for a user’s query and sent to the model alongside the question.
This lets an application use an evolving collection of documents without retraining the model whenever a document changes. However, answer quality depends on both retrieval and generation. If a relevant fact is missing or a source is out of date, RAG does not automatically solve the problem.
A practice scenario
Take ten short instruction documents for a fictional product. Write five questions that the documents answer and five more for which they provide no information.
- Save the passages retrieved for each question.
- Ask the model to answer only from those passages and identify its sources.
- Check whether the cited text actually supports the answer.
- Observe whether the system acknowledges missing information.
Two different kinds of error
If the relevant document never reaches the context, investigate retrieval. If it is found but its meaning is distorted, investigate generation and the instruction. Separating these stages helps identify what to change.
A link next to an answer does not establish that it is correct. The source should support the specific claim, rather than merely discuss the same topic.
Reference: Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020). The instruction-document scenario is an independent teaching exercise.