Understanding LLM Limitations and the Promise of Retrieval Augmented Generation (RAG)

Large Language Models (LLMs) are remarkable technological achievements that have revolutionized how we interact with information. By compressing vast amounts of human knowledge - or at least most of the internet - into neural networks, they can retrieve and synthesize information with unprecedented ease, even when given imperfect prompts. These models have effectively digested much of humanity's publicly available digital knowledge.

However, this impressive capability comes with notable caveats. First, LLMs can only learn from publicly accessible data used in their training. This excludes vast amounts of valuable information, including data from the deep web, private company databases, internal documentation, proprietary research, secured government data, and content behind paywalls or authentication barriers.

The second major caveat is temporal: LLMs operate with a fixed knowledge cutoff date. Unlike humans who continuously learn and update their understanding, these models remain frozen in time until their next training iteration. This means they lack awareness of recent events, breakthrough research, or evolving situations that occurred after their last training.

These limitations create a significant risk: when asked about information outside their training scope – whether due to privacy restrictions or recency – LLMs might either decline to answer or, more problematically, generate plausible-sounding but potentially incorrect responses (a phenomenon known as hallucination).

One solution that's been developed to tackle the knowledge cutoff problem is connecting LLMs to live web search. This approach lets models access current information by pulling in search results as context for their responses. While this works well for getting recent information, it means relying heavily on how search engines like Google rank and present their results. It's essentially a form of RAG, but one where the retrieval process is outsourced to search engines rather than being tailored to specific needs.

RAG implements a straightforward principle: relevant information is retrieved from external sources and provided to the LLM as context during inference. This allows the model to search and process information within its immediate context, rather than relying solely on its training knowledge.

A well-designed RAG infrastructure addresses both the temporal and privacy limitations of LLMs. For the recency problem, RAG can be implemented through integration with search engines (as demonstrated by SearchGPT) or through dedicated pipelines that continuously update an organization's knowledge base. For the privacy challenge, RAG enables LLMs to access proprietary information by connecting them to internal databases during inference, allowing them to reference private documentation and sensitive data without requiring this information to be part of the model's training data.

Due to the sensitive nature of private data, we'll demonstrate these limitations and their solutions through a recency example. Using Claude by Anthropic, one of the most capable general-purpose LLMs with a knowledge cutoff of April 2024, when asked about Deutsche Bank's Q3 2024 financial results (a period beyond the model's cutoff date), the model refuses to answer and indicates that this period lies in the future, as shown in Figure 1.

Figure 1: Chatbot response without RAG, illustrating the limitations caused by knowledge cutoff.

To address this limitation, we implemented a news-ingestion pipeline that uploads recent financial reports to a vector database. This process transforms news article segments into semantic vectors – mathematical representations that capture the meaning of the text rather than just its literal content. This semantic encoding enables retrieval based on conceptual similarity rather than mere keyword matching, allowing for more intelligent and context-aware information retrieval.

Using similarity search, we retrieve the most relevant article segments related to our query and provide them as context to the LLM along with the original question. Instead of relying on the model's trained knowledge, we instruct it to find the answer within these provided texts. This transforms the challenge from retrieving compressed information in the model's weights into a more manageable "needle-in-the-haystack" search problem. Modern LLMs excel at this type of information extraction, consistently achieving perfect scores on relevant benchmarks.

Figure 2: Chatbot response with RAG activated, showcasing accurate and updated answers through semantic vector search.

As demonstrated in Figure 2, RAG successfully delivers accurate and comprehensive responses by synthesizing information from multiple ingested documents. The model can now confidently answer questions about Deutsche Bank's Q3 2024 results by drawing from these recent external sources.

The RAG framework thus addresses both the recency and privacy challenges, enabling organizations to leverage their proprietary information more effectively and securely. When implemented properly, RAG transforms workplace efficiency by providing employees with a knowledgeable AI assistant that has comprehensive access to organizational knowledge across all workflows.

If you're interested in implementing RAG for your organization, we offer specialized consulting services in this area. Our solutions include ready-to-deploy RAG systems, with infrastructure that can be quickly set up in your environment using Infrastructure-as-Code (IaC) deployments. These systems can be hosted either in a dedicated cloud environment or your own virtual private cloud (VPC), ensuring secure access to your data.

For a deeper dive into the components of a RAG pipeline, stay tuned for our upcoming blog posts that will explore each element in detail.