The idea in one minute. Retrieval supplies information at inference time; fine-tuning changes how a model uses information. RAFT trains a language model on questions accompanied by relevant and irrelevant documents, with evidence-aware target answers. It teaches a reader to cope with a retrieval context, while a separate retriever still determines which evidence reaches it.
Reading path. Read the pipeline and loss first, then the training-data mixture. The latter sections explain why citations, abstention, and held-out document evaluation are separate requirements. Prerequisites are conditional probabilities and the idea of supervised learning.
1. Two ways to adapt a language model
A language model represents a distribution over token sequences:
\[p_\theta(y\mid x)=\prod_{t=1}^{\vert y\vert }p_\theta(y_t\mid x,y_{<t}).\]Fine-tuning updates \(\theta\) using task examples. Retrieval instead forms a context \(D_q=\operatorname{Retrieve}(q,\mathcal C)\) from a document collection \(\mathcal C\). A common deployed pipeline is
\[q\longrightarrow D_q\longrightarrow p_\theta(y\mid q,D_q).\]This top-\(k\) context pipeline differs from the original RAG formulation that can marginalize over latent retrieved documents. Writing \(p_\theta(y\mid q,D_q)\) is an accurate abstraction of a concatenated-context reader, not a claim that it performs that latent marginalization.
Fine-tuning and retrieval address different variables. Retrieval can change evidence without changing weights. Fine-tuning can teach answer style, evidence selection, or domain behavior, but does not guarantee that weights contain a current copy of every document. Combining them is natural when the deployed reader receives retrieved context. RAG and RAFT.
2. What RAFT changes
RAFT constructs supervised examples with question \(q\), an oracle document \(d^*\) containing answer evidence, distractors \(d_1^-,\ldots,d_k^-\), and an evidence-aware target answer \(a^*\). Its target may include quoted evidence and reasoning text before the answer. Training is still autoregressive supervised learning; RAFT is not a new optimizer and does not by itself train the retriever.
A schematic example is:
Question: Which version introduced feature X?
Oracle: Release 2.4 introduced feature X.
Distractor: Release 2.3 improved startup time.
Target: The release note identifies version 2.4; answer: 2.4.
This is an invented teaching example. In a real dataset, preserve document IDs, versions, spans, provenance, and permissions. The point is to distinguish evidence-bearing content from plausible but irrelevant neighboring text.
The paper’s name is Retrieval Augmented Fine-Tuning. Its “retrieval-aware” behavior comes from the data and target construction, rather than an assumption that any fine-tuned model will use arbitrary retrieved text correctly.
3. The training objective
Let \(x_i=(q_i,D_i)\) and target tokens \(a_i=(a_{i,1},\ldots,a_{i,T_i})\). With target-only token supervision, the empirical loss is
\[\mathcal L_{\mathrm{SFT}}(\theta) =-\frac1n\sum_{i=1}^{n}\sum_{t=1}^{T_i} \log p_\theta(a_{i,t}\mid q_i,D_i,a_{i,<t}).\]One may normalize by total target tokens or average per example; those weight variable-length answers differently. Document and question tokens normally form conditioning context, with their loss masked if the goal is to optimize answer prediction. The exact masking and normalization belong in the experimental specification.
RAFT mixes oracle-present and oracle-absent contexts. Introduce \(B_i\sim\operatorname{Bernoulli}(P)\) and
\[D_i=\begin{cases} \{d_i^*\}\cup D_i^-,&B_i=1,\\ D_i^-,&B_i=0. \end{cases}\]The target remains an answer created from the oracle. This is a historical design choice in RAFT. It encourages some ability to answer without retrieved oracle evidence, but it is not an abstention rule. For a product that must answer only from supplied sources, oracle-absent examples may instead need an insufficient-evidence target. That is an application adaptation and should be evaluated as a changed training recipe.
4. Reader quality is bounded by evidence availability
Let \(H\) be the event that the retrieved context contains sufficient evidence, and let \(S\) be successful evidence-grounded answering. Then
\[\Pr(S)=\Pr(S\mid H)\Pr(H)+\Pr(S\mid\neg H)\Pr(\neg H).\]If success requires supplied evidence and \(\Pr(S\mid\neg H)=0\), then \(\Pr(S)\le\Pr(H)\). A reader cannot recover absent evidence through better attention alone. For multi-hop questions, retrieving one relevant document can be insufficient; define \(H\) in terms of the complete required evidence set.
This decomposition motivates separate measurements: evidence recall, reader correctness given sufficient context, distractor robustness, and behavior when evidence is missing. A single end-to-end answer score mixes all four.
5. Reasoning text and citation fidelity
Teacher-forcing a target that includes an explanation optimizes the likelihood of that explanation. It does not prove the model’s internal process follows the explanation, nor that every explanation step is correct. A well-written rationale can contain an unsupported claim.
A citation has at least three obligations: the cited document is present and accessible, the quoted span is reproduced correctly, and the span supports the claim. A document ID that exists meets only the first obligation. For synthetic training data, verify spans against source documents and reject fabricated or mismatched evidence before training.
For exact extractive evidence, a deterministic string or offset check is useful. For semantic support, use annotated evaluation or a separately validated entailment process. Generated reasoning and model-based judging can be correlated, so they do not substitute for independent gold labels on the key evaluation slice.
6. Data splits and distribution shift
Randomly splitting question–answer pairs can leak the same source document, paragraph, or template across training and test sets. A model might memorize an answer and appear retrieval-aware. Use held-out documents or versions when testing evidence use; group near duplicates; reserve a separate calibration set if using confidence thresholds.
Include controlled contexts:
| Context | What it tests |
|---|---|
| Oracle only | Reader capability when evidence is ideal |
| Oracle plus distractors | Robustness to irrelevant material |
| Real retrieved top-\(k\) | Deployed retrieval-and-reading behavior |
| Distractors only | Missing-evidence behavior |
| Conflicting dated sources | Version and source-priority handling |
| Correct source with changed answer | Reliance on supplied evidence rather than memorized text |
Keep model, retriever, chunking, corpus snapshot, context budget, training tokens, and answer format fixed for an ablation. If a new retriever supplies different noise or longer chunks, the trained reader’s distribution changes. “Works with a retrieval API” is interface compatibility, not invariant accuracy across retrievers.
7. Training and inference costs
Fine-tuning adds an upfront cost \(C_{\mathrm{train}}\). Let per-query deployed cost be \(c_1\) after tuning versus baseline \(c_0\). Ignoring quality differences and maintenance, a cost-only break-even exists when \(c_1<c_0\):
\[N_{\mathrm{break\text{-}even}}=\frac{C_{\mathrm{train}}}{c_0-c_1}.\]This arithmetic does not say fine-tuning is beneficial; it clarifies the assumptions. Add data preparation, source updates, periodic retraining, evaluation, retrieval, and escalation costs for a real decision. If accuracy changes, compare cost per successful grounded answer rather than cost per request alone.
Use LoRA or another adaptation method if suitable, but that is orthogonal to the RAFT data strategy. A shorter generated answer can also lower inference cost without any retraining. Hold output requirements constant in a performance comparison.
8. What the historical results establish
The RAFT paper evaluates particular domain datasets, retrievers, and language models. Its results support evidence-aware training in those conditions. They do not imply RAFT consistently beats every later model, nor that a quoted percentage improvement transfers to a different denominator, baseline, or dataset.
This revision removes ambiguous “percent improvement” claims from the earlier note. Reproduce the original table with the exact metric and distinguish percentage points from relative percentage change before quoting a gain. Compare against a strong current untuned reader, domain SFT, and context-aware SFT at matched budgets on your own held-out corpus.
References and current scope
- Zhang et al., RAFT: Adapting Language Model to Domain Specific RAG.
- RAFT implementation.
- Lewis et al., Retrieval-Augmented Generation.
- Hu et al., LoRA.
RAFT is a training-data and reader-adaptation strategy. Retrieval, permission filtering, evidence verification, and abstention remain explicit components of a complete system.