When Not to RAG

Let me try to describe RAG, or the Retrieval Augmented Generation, in one sentence: before you ask the LLM a question, you go find some documents that might help and paste them into the prompt as supporting evidence.

When a student asks an LLM what a derivative is, the LLM gives a great response on its own. It has read more math and calculus books than any of us ever will. It doesn’t need the Wikipedia page on derivatives as supporting evidence. Now imagine the same student asking what’s on my MATH 101 midterm next week? The LLM has no clue. The MATH 101 syllabus was (probably) not in its training data. This is why RAG is popular with customer-service chatbots: they answer from the actual policy documents instead of from memory.

RAG is a good technique. It’s also easy to overuse, and in fact it is overused.

Context

Suppose you are at an edtech company as a hands-on managers of multiple teams. After yet another reorg, you’ve just inherited the AI team. Some months earlier, the team shipped an AI chat for students, which is also yours now.

This is how the system works:

1. A student sends a chat message.
2. A classifier model filters the message and detects intent.
3. Semantic search runs over the document library (RAG) to find relevant documents.
4. The best chunks are sent to the LLM along with the system prompt and the student’s message (RAG).
5. The LLM’s response goes through a judge (another LLM call).
6. The response is sent to the student.

That’s a lot of steps for a single response. Worse, none of them can be parallelized.

Let’s look at some rough numbers. The classifier is very fast and takes a few 100 milliseconds. The semantic search (Pinecone) adds another 300–500ms. The main LLM call, with 3–4k tokens of context, adds 6–8 seconds. Then comes the judge. An older, smaller model to keep costs down. It has a rubric in its system prompt and grades the response against query and the rubric, adding another 3–5s. Average response takes about 10 seconds. Your design team is asking whether they should build a pizza tracker while students wait for chat messages. ChatGPT starts showing you a response in less than 2 seconds. You know you are in trouble.

The team had done a good job. They indexed high-quality documents, which sounds easy but it isn’t: consider more than 50 million documents, sparsely tagged to courses and subjects, and need a sensible chunking strategy and a pass to strip out non-printable characters. They tested prompts. They built an excellent judge that caught harmful and low-quality responses. The brief they were handed was to differentiate from ChatGPT by grounding answers in the company’s documents, and that was a good approach for the models of early 2023. The models have since improved massively. But the design didn’t.

But the latency is horrendous, and the Amplitude charts show that students aren’t engaging with chat.

You ask your UX researcher to talk to students who have been active on chat. Ten interviews later, the signal is clear and consistent:

1. Responses take too long. Students switch to another tab and get the answer from ChatGPT, or in some cases Google.
2. The answers are no better than ChatGPT’s, and sometimes worse.

The second one is news to you.

Here’s a made-up example to illustrate. A student types “describe photosynthesis.” Your system describes photosynthesis, then takes a strange detour through whichever biology documents happened to score highest in search. You are paying extra to confuse both GPT and the student.

What you change

After a lot of debate, you land on the following decisions:

• Stop retrieving what the model already knows. Your analysis shows that nearly two-thirds of messages can be answered without any of your documents. They’re general questions, and the LLM handles them well on its own.

• Keep RAG when the question needs course context. You don’t drop RAG. You keep it mostly as is (with a boost for certain document types) and run it only when the question calls for it, not on every message.

• Remove the blocking judge. This is not an easy decision, and not one you can make alone. The judge is a safety net against a technology people don’t fully understand yet. There is a real fear that someone will make the chat say something horrible and post it on X or Reddit. But when you audit it, the judge is blocking 8% of answers, and a good share of those calls are questionable. It was built for 2023-era models, with their hallucinations and their habit of repeating themselves. You lobby, show with data that the same guardrails in the system prompt work just as well, and get the green light. Removing the judge doesn’t just save 3–5 seconds. It lets your team stream responses, and streaming is what users perceive as speed.

• Inject course-specific knowledge into the system prompt. An engineer on the team notices that for popular courses, the same chunks come back most of the time. When you read those chunks, you realize the LLM only needs 3–4% of what’s in them: the key details. So the team builds a pipeline that runs an LLM over each course’s documents, extracts those details, and injects them straight into the system prompt. Think 5–7 bullets per course.

• Run document search for every query, in parallel. You still search on every message, even the ones that don’t need RAG. But instead of feeding the results to the LLM, you show them under the response as “Related Materials.” The search runs alongside the LLM call instead of before it. Students get a fast answer and still see the documents they can dig into.

When to RAG

I’m not against RAG. It’s a useful technology and here’s when I’d use it:

• The information needed to answer a question is private: syllabus, company policy, your own files. The model doesn’t know it and no amount of intelligence will compensate for it (or may be it’ll one day 🤔)
• The information changes frequently: syllabus, schedules, calendars, etc.
• You need to show citations and actually ground answers: When your users actually want information to be grounded (e.g. in their own documents), and you need to show sources. E.g. a PhD candidate wants answers grounded in the best and recent papers. Note: if you are grounding answers in your source material, you must make sure that you have very high quality source material. In most cases, UGC (user generated content) doesn’t fit this criteria.

I’d still skip it when speed is the key feature and marginal gains from RAG are modest or when the system already knows the information well.

Lessons

• RAG is not free. Every chunk you retrieve has a cost: latency, price of the API and a chance that it could confuse the model and degrade answer quality.

• Guardrails and judges also have a cost too. A judge disables streaming and streaming is how users perceive “speed”.

• The AI world is improving at light-speed. What you knew or used last year may no longer be the optimal solution. Re-evaluate and check your assumptions regularly. The judge made sense for older models but wasn’t as effective with newer.

With these changes, your AI chat is faster and better. But will it outperform ChatGPT and convince your users to use it instead? Allow to me go on a tangent and discuss what’s happening in the industry.

Every Edtech company I know of is building the same AI chat: their flavor of RAG on top of GPT. It’s not secret that ChatGPT is eating their lunch so we need to compete and not let users switch tabs to go to ChatGPT. One stop shop. I strongly believe that competing with OpenAI head-on is a losing bet. Why are companies doing this? Well I think in some cases, they force themselves to believe that “we have all the study materials, so we will do a better job.” I think this is flawed. Your data is a moat only if:

• Students actually need it. A great document from 2011 isn’t relevant anymore. Neither are most of the others. Once you apply this and other filters, your pile of “all our study materials” shrinks to a very, very small set of useful documents.

• ChatGPT hasn’t seen it and can’t synthesize it. GPT has read millions of books (and very likely your documents as well if they were accessible for free on the web). You have an excellent set of proprietary notes for a large number of academic subjects and topics. ChatGPT will give you the same explanation without them. It doesn’t need your pristine notes to answer any question about Shakespeare. May be you argue and convince yourself and perhaps you do have higher quality documents. Here’s the next question: will a student pay $20 or $40 a month for marginally better? Abosultely not. You then bundle it with the main subscription and lose money on enagement, however, remove the feature tomorrow and no one would complain because they bought the bundle for another reason.

• You understand user intent and build for it. ChatGPT is a general-purpose tool for everyone. It’s both its strength and weakness. If you know what your most engaged users are actually trying to do with chat, you can build an experience they can’t get from a blank text box. The catch is that you have to go small and segment (which is easy for startups but not so easy for large or medium sized companies). You can’t target everyone this way. If students come to your site for specific reasons, build the best possible product and experience for them, instead of a general-purpose ChatGPT clone and keep sending cheques and data (they may not train model but I refuse to believe they are not extracting valuable signals. As a growth person, I absolutely would) to OpenAI which you are competing against.

Hope you enjoyed.

Note: I have changed a few key details but the lessons are all real.