Understanding RAG (Retrieval-Augmented Generation) and LangChain's Features and Usage

The Popularization of LLMs

LLMs proved their value in the market through remarkable performance. To people working in AI, early ChatGPT was fascinating mainly because it showed just how capable autoregressive models had become and because it could build context through an ongoing conversation. At that point, OpenAI still looked like an extension of “academic AI,” and there was little sense of how dramatically this technology would win over the wider world.

Before long, however, ChatGPT acquired eyes and ears. New methods for improving performance, vast amounts of high-quality data acquired with substantial capital, and services built by connecting AI models into coordinated systems all suggested that AI was moving beyond merely “so-so” performance on the data it had been trained on.

• We can no longer read effectively without help from LLMs.
• It is increasingly difficult to work efficiently when coding without help from LLMs.

Using AI is no longer merely a matter of preference; it has rapidly begun to create a difference in competitiveness.

Limitations of LLMs

Even so, LLMs often behave differently from what we intend. One of their biggest problems is that they do not simply answer, “I don't know.”

Suppose, for example, that you are asked the following question:

Tell me how to make a Bluetooth shower!

If you know the meme about Bluetooth showers, or simply grew up with a basic education, you will know why a “Bluetooth shower” is impossible.

Even if you do not, the explanation is simple: a shower needs a channel that carries liquid water, not electromagnetic waves transmitted over Bluetooth. A physical pipe is unavoidable.

The example may be a little silly, but an LLM does not inherently possess the logical reasoning needed to determine why an impossible combination of technologies such as Bluetooth plus a shower cannot work, unless it has already learned the Bluetooth-shower meme. In any case, an LLM is trained and operates according to the following principle:

Based on the context constructed so far, compose the most probable text to come next.

It will therefore combine the semantic concepts of “Bluetooth” and “shower” and produce the most plausible-sounding nonsense it can about a Bluetooth shower.

The model may lie while trying to implement an “impossible technology.” The more surprising question is whether people do much the same thing.

An intelligent person can clearly distinguish what they know from what they do not. Yet even intelligent people often fail to do so. At a first job interview, an unexpected question can leave us rambling about something we barely understand and ruining the interview in the process.

Even outside an interview, we sometimes say, “I think I've heard of that somewhere,” and invent details about a subject we know nothing about. At other times, we distort a rumor through our own perspective and retell it in a strange form. We “remembered” something, but failed to reproduce it faithfully.

An LLM is a memory mechanism that has modeled token continuity statistically from vast numbers of documents; it is not a search model. When it retrieves a word buried somewhere in that memory, the many possible words associated with it pull out a next token, which pulls out another, and so on until a paragraph takes shape. A sentence generated in this way cannot easily stop and say “I don't know,” even if the original question was flawed. By the time the model could conclude, “It turns out I don't know this,” it has already completed an answer.

The Ability to Understand Context

LLMs therefore have clear fundamental limitations. If a model was trained on data only through time $N$, then 1) it can reason only from data available up to that point, and 2) even if it learned that data, there is no guarantee it can answer every kind of question correctly. After all, although we studied many subjects in middle and high school, we could not now score 100 on every exam covering them.

GPT likewise says its last general-knowledge update was in June 2024. It therefore cannot know what happened afterward, such as the current KRW-to-USD exchange rate.

To use ChatGPT with sufficient confidence, we need ways to make effective use of the statistical knowledge it contains. This is the perspective from which in-context learning (ICL) and RAG emerge.

We may be unable to sit an exam right now on every subject we once studied, but if we were given the relevant textbooks and reference materials for an open-book test, we would at least avoid the embarrassment of scoring zero. Likewise, we cannot be tested on something we have never learned, but with appropriate reference material—and enough time—we can still produce a reasonably reliable answer. The key idea here is the “open book.”

Models Optimized for Contextual Understanding

LLMs are highly capable of finding clues in context. Even if their training data contains no material from a particular specialty, they can infer meaning from the context provided. Let the full corpus (dataset) used for training be $\mathcal{C}$, consisting of $N$ documents (samples) $D^{(k)}$:

$$\mathcal{C} \;=\; \{\, D^{(k)} \,\}_{k=1}^{N}.$$

Represent each document $D^{(k)}$ as a sequence of tokenized or parsed units—tokens, sentences, chunks, and so on—denoted by $t^{(k)}_i$:

$$D^{(k)} \;=\; \bigl(t^{(k)}_{1},\, t^{(k)}_{2},\, \dots,\, t^{(k)}_{n_k}\bigr), \qquad n_k \;=\; |D^{(k)}|.$$

For a single document (for example, $D := D^{(1)}$),

$$D \;=\; (t_1, t_2, \dots, t_n), \qquad n \;=\; |D|.$$

Learning Continuity Across the Corpus

For each document $D^{(k)}=(t^{(k)}_1,\dots,t^{(k)}_{n_k})$, the model $p_\Theta$ learns by factoring the sequence probability as follows:

$$p_\Theta\!\bigl(D^{(k)}\bigr) \;=\; p_\Theta\!\bigl(t^{(k)}_{1:n_k}\bigr) \;=\; \prod_{i=1}^{n_k} p_\Theta\!\left(t^{(k)}_i \mid t^{(k)}_{1:i-1}\right).$$

The maximum-likelihood objective over the full corpus—equivalently, minimizing negative log-likelihood—is usually defined as follows:

$$\Theta^\ast= \arg\max_\Theta \sum_{k=1}^{N}\sum_{i=1}^{n_k} \log p_\Theta\!\left(t^{(k)}_i \mid t^{(k)}_{1:i-1}\right).$$

The LLM's Learned Behavior: Updating the Next-Token Distribution

At every step of autoregressive generation, the model updates the conditional distribution of the “next token” given the preceding “context,” $F_\Theta(t_{1:j}) := p_\Theta(\cdot \vert t_{1:j})$. Put simply, when the preceding context changes, so does the probability distribution over the words that may follow. As discussed above, the prior underlying this conditional distribution reflects the distribution of the entire corpus.

Getting the Desired Answer by Increasing the Relevant Prior Probability

If the answer (text) expected by the user is $y = (y_1,\dots,y_m)$, the conditional probability that the model generates that answer is

$$p_\Theta(y \mid x) = \prod_{\ell=1}^{m} p_\Theta\!\left(y_\ell \mid x, y_{1:\ell-1}\right)$$

This probability factors into token-level terms. Here, $x$ is the context—the prompt or prefix—$t_{1:j}$. The most direct way to increase the probability of obtaining the desired answer is therefore to increase the probability assigned to the correct token $y_\ell$ at every step. The lever that changes those probabilities is the condition, or context, $x$.

The Power of Guidance from Rich Context

Providing richer context $x=t_{1:j}$ narrows the space of possible intentions and situations the model must consider—such as the domain, terminology, and output format—so $p_\Theta(\cdot\mid x)$ shifts closer to the mode we want. Intuitively, this raises the conditional probability $p_\Theta(y_\ell\mid x,y_{1:\ell-1})$ assigned to the tokens that form the correct sequence $y$ at each step, while lowering the probability of taking an incorrect branch due to ambiguity.

In particular, context can provide 1) explicit terminology and assumptions, 2) a specified role and domain, and 3) output constraints such as format and rules. These narrow the possible output space and make the generation path more likely to converge on the correct path. We can therefore conclude that sufficiently rich context increases the probability of obtaining the answer we want.

Why RAG Is Necessary

This exposes a weakness in simply adding more context. That strategy assumes the prior is clearly related to the distribution from which we want an answer, but in reality that is not always true. No matter how clearly we specify terminology, conditions, and output requirements, hallucinations will still occur if the prompt lacks the accurate information needed to answer.

Consider asking about today's weather. If the LLM receives too little information, it may give a vague answer such as, “Well, Seoul is generally cold and dry in winter.” But even with context, it can still answer incorrectly when the supplied observation data is wrong—for example, if the KMA data was fetched incorrectly or the API cannot provide real-time information. To obtain accurate answers about specific facts, it is therefore essential to construct genuinely useful context instead of relying only on distributions embedded in the training data.

To unpack the meaning of Retrieval-Augmented Generation in more detail, “augmentation” in deep learning means enriching the data. Early deep-learning research often focused on relatively simple tasks and modestly sized datasets, such as image classification on ImageNet. A shortage of training data can weaken generalization regardless of model size, so various heuristic augmentation methods—vertical and horizontal flips, rotation, cropping, and so on—were proposed to avoid overfitting during training (image source).

As datasets grew and models generalized better, this form of augmentation gradually became less important. Yet large models such as LLMs, despite being trained on enormous amounts of data, were not free of problems.

In traditional computer vision, data was augmented to improve generalization because there was too little data from which to learn reliable feature statistics—for example, the characteristics of a cat. But even when abundant data exists, the information required for a particular inference often lies outside the dataset's scope.

A common example is wanting to analyze the stock market as of January 2026 when the training data contains market information only through 2024. The model may give a broad, generic answer, but it cannot reflect stocks newly listed in 2025–2026 or other developments during that period, so it is unlikely to provide the answer we actually need.

The simplest response might be to keep adding the data we need to the original dataset and retrain the model. But as time advances through 2026, 2027, and beyond, we would have to collect all preceding data and train on it again every time.

Moreover, because we effectively want an LLM to be universal, we would need to cover far more than stock data: newly introduced policies in 2026 and 2027, financial products such as savings accounts and bank comparisons, restaurants at travel destinations, and countless other domains in which continuously collecting data is impractical. As the dataset grows, training time also rises dramatically. From this perspective, we cannot design a “fully generalized” model. An LLM has merely learned the structures, grammatical rules, and contexts of language from an enormous body of linguistic data; no method has yet transcended those structural properties and capabilities altogether.

We therefore redefine “augmentation” for enormous models and datasets such as those behind LLMs. Because an LLM has already learned linguistic structure and grammar from ample language data and is highly capable of constructing context, we can simply retrieve the information we need and provide it as prose.

An LLM is a model of remembered patterns, so it does not know 2027 stock data or financial-product details. But if we organize that information into a document and ask it to summarize it, the model can do so reliably. Similarly, internal policies and ERP-system information unavailable outside an organization are not present in an open-source model's training data. Rather than retraining that model on premises, we can organize the information as documents and use the LLM's language-understanding ability with those documents as context.

Spelled out directly, this workflow is Retrieval → Augmentation → Generation.

LangChain

What, then, is LangChain? RAG concerns how to elicit a desired, accurate answer from one model without generating false information. Once we have considered context engineering for a single model, the next question is how to connect multiple AI models, databases, cloud services, and other components into a coordinated system.

The era of relying exclusively on ChatGPT is already over. OpenAI's models dominated early and once seemed impossible to catch, but numerous competitors soon challenged their lead, including Google's Gemini and xAI's Grok. Most such LLM services perform well across a wide variety of general tasks.

When designing a service, however, do we really need a model that performs well on every task? If resources are abundant, such a model may not be excessive, but abundant resources alone do not justify spending the capacity of a general-purpose model on a narrowly standardized task.

Suppose we build a pipeline for a service workflow. It may use several LLMs of different sizes and specialties, and may include retrieval steps that reference particular data. Retrieval itself might be semantic search; if it uses a vector database dependent on a particular AI encoder, the pipeline must incorporate yet another AI model beyond the LLMs.

With such a complex backend, one of the hardest problems is aligning the communication protocols of every pipeline component. Even after placing proxies around each module and building infrastructure around a common communication scheme, a model update or pipeline redesign can change the specification and force another round of normalization.

LangChain can be thought of as a kind of driver. When installing a printer, we install a driver so the printer works smoothly: it establishes an agreement between the printer and the laptop. Barring special circumstances, that agreement has the same form across similar devices. LangChain extends this idea by providing compatible interfaces for backend applications in which LLMs are components.

LangChain provides more than interfaces. It is a Swiss Army knife for conveniently working with the many components required by LLM applications: input and output, connections, document indexing, memory, agents, RAG, and more.

LLM Abstraction

Through abstraction, LangChain simplifies LLM programming—or, more precisely, the input and output interfaces of LLMs. The primary goal of abstraction is to improve usability by hiding unnecessary details. The UI-driven computers we use today work the same way: we create, delete, and move folders; extract archives; install applications; or quickly open a text editor to jot down a note.

Computers became so intuitive and convenient precisely because their low-level operations—file-system inodes, kernel process scheduling, device drivers, memory management, packet routing, and so on—are hidden behind the operating system and multiple layers of abstraction. Users can accomplish their goals through high-level concepts such as “save / run / copy / install”, while the computer translates those requests into the appropriate system calls and resource-management operations.

Working with LLMs is moving in the same direction. It begins simply: write a prompt, send it to a model, receive a string, and post-process it. In a real product or research system, however, complexity rises immediately.

• Inputs are structured as system, user, and tool messages rather than plain text.
• Outputs become multilayered: JSON, function calls, intermediate reasoning results, and source documents instead of plain answer strings.
• A model call is followed by retries, routing, caching, streaming, and cost or latency control rather than ending after one request.
• When external information is needed, the system must also handle RAG, document splitting, embeddings, indexing, and reranking.

In this setting, “using an LLM well” increasingly means “assembling an LLM-centered application reliably.” This is where LangChain's LLM abstraction becomes meaningful. Its abstraction does more than hide model calls; it standardizes the execution conventions around the model. In LangChain Expression Language (LCEL), a chain is an object that follows the Runnable protocol, so invocation is standardized through invoke, stream, and batch. This is the concrete implementation of “treating an LLM like a function.”

from langchain_openai import ChatOpenAI
from langchain_core.prompts import PromptTemplate
from langchain_core.output_parsers import StrOutputParser

model = ChatOpenAI(model="gpt-4o-mini", temperature=0)
prompt = PromptTemplate.from_template("Explain {topic} in three sentences.")
chain = prompt | model | StrOutputParser()

# 1) invoke: one request
print(chain.invoke({"topic": "multimodality"}))

# 2) stream: token-by-token streaming
for chunk in chain.stream({"topic": "RAG"}):
    print(chunk, end="", flush=True)

# 3) batch: process multiple inputs together (+ concurrency control)
results = chain.batch(
    [{"topic": "LangChain"}, {"topic": "Vector DB"}, {"topic": "Agent"}],
    config={"max_concurrency": 3},
)
print(results)

This code directly demonstrates LangChain's core idea: a Runnable pipeline. The prompt | model | parser form fixes input → transformation → execution → parsing into a single execution graph; it is more than attractive syntax.

• PromptTemplate.from_template(...)
  • Creates a prompt function with variables such as {topic}.
  • In other words, this stage transforms {"topic": "..."} into the final prompt string.
• ChatOpenAI(...)
  • Performs the actual LLM call.
  • Internally, the input is wrapped in a chat-message structure rather than passed as a bare string.
• StrOutputParser()
  • Model output is normally an object such as AIMessage; this parser extracts the final plain string.
  • In other words, this stage converts a model-output object into a string.
• chain.invoke(...) / chain.stream(...) / chain.batch(...)
  • The important point is that the chain interface stays the same even when the way the LLM is called changes.
  • invoke: sends one input through the entire pipeline and returns the final result.
  • stream: runs the same pipeline while streaming intermediate tokens or chunks.
  • batch: runs multiple inputs through the same graph in parallel, with concurrency control.

This presents the LLM through a proxy governed by a standardized contract, allowing it to be treated like a function instead of leaving the call as an isolated API request.

Prompts

A prompt is how we give an actual task to an LLM. Anyone who has used ChatGPT or another text-input AI has probably struggled to elicit the desired result, as in the following example:

Although not exact, prompts that reliably produce the desired kind of output tend to follow a “template.” That template may include a format or particular sentences. Finding one is conceptually simple, but reaching the desired result takes time and effort. LangChain provides a way to preserve and use this context. Treating prompts only as well-written sentences makes the process labor-intensive; LangChain turns them into reusable objects that make it easier to 1) manage input variables, 2) fix the format, and 3) compose chains. PromptTemplate is essentially a way to capture a successful template in code.

from langchain_core.prompts import PromptTemplate

template = """You are a metadata analyst for a broadcast scheduling and archive system.
Question: {question}
Constraints: {constraints}

Output format (JSON):
{{
  "period": "...",
  "programs": ["..."],
  "summary": "..."
}}
"""
prompt = PromptTemplate.from_template(template)

filled = prompt.format(
    question="Statistics on the types and concepts of programs featuring Yoo Jae-suk from September 2022 through the first half of 2025",
    constraints="The period must be reflected, duplicate programs removed, and each concept summarized in one sentence",
)
print(filled)

Here, the prompt is not a “sentence” but a template object with input variables. The logic is simple, but that simplicity becomes efficiency during operation and expansion.

• template = """ ... {question} ... {constraints} ... """
  • Creates variable slots inside the prompt.
  • Including the JSON format encourages the model to follow the desired output shape, acting as a weak schema.
• prompt = PromptTemplate.from_template(template)
  • Turns the string template into a prompt object that LangChain can handle.
  • Variable validation and substitution then happen consistently through format().
• prompt.format(...)
  • Fills {question} and {constraints} to create the final prompt string.
  • This is the structured-input → string-generation stage.

Prompts can also be downloaded from LangChain Hub instead of being stored locally every time. Much like publishing images to Docker Hub, this is useful when a team shares standardized prompt forms.

from langchain import hub

# Example: fetch a summary prompt from the Hub (for sharing and versioning)
prompt = hub.pull("teddynote/summary-stuff-documents-korean")
prompt.pretty_print()

• hub.pull(...)
  • This supports reusing prompts validated by a team or project.
  • Instead of copying and pasting locally, a versioned prompt can be fetched and connected directly to a chain, reducing operational mistakes.

Chains

A chain combines an LLM with other components. Suppose, for example, that a system must search an internal database and then use summarized metadata to answer a user's question. At my company, the query might look like this:

Based on broadcasts from September 2022 through the first half of 2025, compile statistics on the types and concepts of programs featuring Yoo Jae-suk.

The request requires more than querying a database within a date range. The system may need to organize episode metadata, collect unique program types as a set, and use another LLM to summarize program concepts. In this way, the output of one task feeds subsequent tasks or triggers additional requests, forming a sequential chain. The essence of a chain is not “calling an LLM several times,” but building a pipeline that passes intermediate artifacts into the next stage. LCEL makes that pipeline intuitive to assemble. Users can build a chain themselves or use a predefined structure.

(Example 1) Assembling a RAG chain with LCEL

This flow matters because each RAG component becomes a replaceable part, while the chain describes how those parts connect.

import bs4
from langchain import hub
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import WebBaseLoader
from langchain_community.vectorstores import FAISS
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

# 1) Load
url = "https://n.news.naver.com/article/437/0000378416"
loader = WebBaseLoader(
    web_paths=(url,),
    bs_kwargs=dict(
        parse_only=bs4.SoupStrainer(
            "div",
            attrs={"class": ["newsct_article _article_body", "media_end_head_title"]},
        )
    ),
)
docs = loader.load()

# 2) Split
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=50)
splits = splitter.split_documents(docs)

# 3) Index (Embed + VectorStore)
vectorstore = FAISS.from_documents(splits, OpenAIEmbeddings(model="text-embedding-3-small"))
retriever = vectorstore.as_retriever()

# 4) Generate
prompt = hub.pull("rlm/rag-prompt")
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

def format_docs(docs):
    return "\n\n".join(d.page_content for d in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

print(rag_chain.invoke("Summarize the article's key points in one paragraph."))

This code implements RAG as an executable pipeline. Step by step, it works as follows.

1. Load documents (WebBaseLoader(...).load())
   • Fetches HTML from a URL, parses only selected DOM regions, and creates a list of Document objects.
   • Here, Document.page_content holds the text, while metadata holds data such as the URL.
2. Split, or chunk (RecursiveCharacterTextSplitter(...).split_documents(docs))
   • Divides a long document into chunks that can be searched.
   • chunk_overlap is a safeguard that reduces context breaks by preserving information near boundaries.
3. Index with embeddings and a vector store (FAISS.from_documents(splits, OpenAIEmbeddings(...)))
   • Converts each chunk into an embedding vector and stores it in a FAISS index.
   • This is where text becomes vectors and then a searchable structure.
4. Create a retriever (vectorstore.as_retriever())
   • Provides a search interface that embeds an incoming question and finds the most similar chunks.
5. Assemble the RAG chain
   • Splits the input question into two branches.
     • context: question → retriever → relevant documents → joined string
     • question: preserves the original question as is
   • The result then flows through | prompt | llm | parser:
     • The context and question fill the prompt, the LLM produces an answer, and the parser returns it as a string.

(Example 2) Using a predefined chain structure for document summarization

Common document-summarization patterns include Stuff, Map-Reduce, and Refine, and LangChain provides them as chain functions.

from langchain import hub
from langchain_openai import ChatOpenAI
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_community.document_loaders import TextLoader

docs = TextLoader("data/news.txt").load()

prompt = hub.pull("teddynote/summary-stuff-documents-korean")
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

stuff_chain = create_stuff_documents_chain(llm, prompt)
print(stuff_chain.invoke({"context": docs}))

The Stuff chain is not a complex strategy. It is the simplest approach: place all documents in one prompt and summarize them in a single pass.

• docs = TextLoader(...).load()
  • Reads a local file and creates a list of Document objects.
• prompt = hub.pull(...)
  • Fetches a summarization prompt template whose rules already specify where document context belongs.
• create_stuff_documents_chain(llm, prompt)
  • Internally, it joins the input documents into one context string, fills the prompt with that context, calls the LLM once, and returns the result.

If the documents grow too long, however, they can easily exceed the context limit. At that point, the workflow can be extended by switching to a chain such as Map-Reduce or Refine.

Indexes

A service built for a particular task may depend on external sources absent from its training dataset. In the stock-market RAG example above, it might need to consult Bloomberg data and news articles. LangChain groups the mechanisms for working with such external data under indexes, which consist of the following elements:

• Document Loaders: retrieve documents from stored sources.
• Vector Databases: search efficiently indexed sources stored as vectors.
• Text Splitters: divide and manage text data in meaningful units.

In a RAG process, documents are loaded with a Document Loader, chunked into desired semantic units with Text Splitters, converted into a Vector Database, and then used for RAG. The stock-market example above therefore follows the process shown below. For the detailed logic, compare it with the chain code discussed earlier.

More precisely, an “index” from LangChain's perspective is the standardized pipeline for handling external knowledge itself: load → split → embed → store/retrieve. It covers the database and the steps needed to populate and use it.

(Example 1) Text Splitter (RecursiveCharacterTextSplitter)

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=250,
    chunk_overlap=50,
)
chunks = splitter.split_text("...long text...")
print(len(chunks), chunks[0][:80])

• RecursiveCharacterTextSplitter(chunk_size=..., chunk_overlap=...)
  • Divides text into segments of a given length while preferring natural boundaries whenever possible.
  • It is widely used because its default separators are tried recursively in the order \n\n → \n → space → character, which tends to preserve context.
• split_text(...)
  • Returns the input string as a list of chunks.
  • These chunks become the atomic units stored in the Vector DB.

(Example 2) Embeddings (OpenAIEmbeddings)

from langchain_openai import OpenAIEmbeddings

emb = OpenAIEmbeddings(model="text-embedding-3-small")
vec = emb.embed_query("broadcast schedule statistics analysis")
print(len(vec), vec[:5])

• OpenAIEmbeddings(...).embed_query("...")
  • Converts query text into a numeric vector.
  • The vector can be understood as a coordinate in semantic space and is later used to compute distance, or similarity, to document vectors.

(Example 3) VectorStore (Chroma / FAISS)

from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings

# Assume split_docs is the result of splitter.split_documents(...)
db = Chroma.from_documents(
    documents=split_docs,
    embedding=OpenAIEmbeddings(model="text-embedding-3-small"),
    collection_name="my_db",
    persist_directory="./chroma_db",  # Without this, use temporary in-memory storage
)

• Chroma.from_documents(documents=..., embedding=..., ...)
  • Internally, this embeds each document chunk and stores the vector, original text, and metadata in a collection.
• persist_directory
  • When provided, the index is stored on disk and survives restarts.
  • When omitted, it can operate in memory.

For retrieval, as_retriever() creates a standard interface, so a chain needs only to plug in the retriever. The database can then remain interchangeable.

Memory

Memory is not important for a single request. But if a user has several exchanges with an LLM and wants it to continue reasoning from prior conversation, the system must understand and store that conversational context. LangChain provides modular memory functionality for maintaining it.

Memory can mean “storing a conversation log,” but in practice it falls into two categories.

1. Memory that retains the conversation history itself, for conversational UX
2. Memory stored in a searchable form, for long-term memory, personalization, or history RAG

LangChain provides both as modules.

(Example 1) ConversationBufferMemory (the simplest conversational memory)

from langchain.memory import ConversationBufferMemory

memory = ConversationBufferMemory(
    memory_key="chat_history",
    return_messages=True,
)

# Save
memory.save_context({"input": "I'm searching a broadcast archive"}, {"output": "Great—what constraints do you need?"})

# Load
print(memory.load_memory_variables({})["chat_history"])

• ConversationBufferMemory(return_messages=True)
  • The simplest memory, accumulating conversation content verbatim.
  • With return_messages=True, it retains Message objects rather than strings.
• save_context({"input": ...}, {"output": ...})
  • Stores a user input and model output as one turn.
• load_memory_variables({})
  • Returns the accumulated history as {"chat_history": ...}.
  • Inserting chat_history into a later prompt lets the model respond with awareness of the preceding conversation.

(Example 2) Injecting memory into an LCEL chain (RunnablePassthrough.assign) RunnablePassthrough.assign(...) is a representative pattern showing how to “insert memory into a chain” in code.

from operator import itemgetter
from langchain_core.runnables import RunnableLambda, RunnablePassthrough

# Assume memory.load_memory_variables({}) returns {"chat_history": ...}
runnable = RunnablePassthrough.assign(
    chat_history=RunnableLambda(memory.load_memory_variables) | itemgetter("chat_history")
)

print(runnable.invoke({"input": "hi"}))

• RunnableLambda(memory.load_memory_variables)
  • Retrieves history from memory each time it is called.
• itemgetter("chat_history")
  • Extracts only chat_history from the memory result.
• RunnablePassthrough.assign(chat_history=...)
  • Adds the chat_history key to the original input dict.
  • Thus, the chain's final input expands from the original input to input + chat_history.

This pattern allows prompts and chains to be written under the assumption that chat_history will always be present.

(Example 3) VectorStoreRetrieverMemory (memory that “retrieves” summaries or history)

from langchain.memory import VectorStoreRetrieverMemory

# Assume retriever was created with vectorstore.as_retriever()
memory = VectorStoreRetrieverMemory(retriever=retriever, memory_key="history")

memory.save_context({"input": "I want statistics on Yoo Jae-suk's appearances by genre"}, {"output": "OK, I'll organize them by genre, program, and period"})
print(memory.load_memory_variables({"prompt": "Continue the Yoo Jae-suk statistics"})["history"])

Rather than appending the entire conversation log, this approach retrieves only relevant history when it is needed.

• VectorStoreRetrieverMemory(retriever=...)
  • Internally converts conversation content → summaries or sentences → vectors → storage, then retrieves similar past content when a question arrives.
• save_context(...)
  • Stores the turn in the vector index.
• load_memory_variables({"prompt": ...})
  • Finds earlier conversations similar to the current prompt and returns them as history.

This memory is therefore used to reduce context-length pressure in long-running conversations.

Agents

An agent refers to a model—such as OpenAI's GPT API or Claude—used to pursue a particular objective. Beyond declaring the model, it can be customized with a list of available tools, a reasoning process designed through the system prompt, and many other settings.

(Example 1) Defining a tool and injecting its schema with bind_tools

from langchain.agents import tool
from langchain_openai import ChatOpenAI

@tool
def get_word_length(word: str) -> int:
    """Returns the length of a word."""
    return len(word)

tools = [get_word_length]

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
llm_with_tools = llm.bind_tools(tools)

# The model output now contains tool_calls
print(llm_with_tools.invoke("What is the length of the word 'teddynote'?").tool_calls)

• The @tool decorator
  • Attaches tool metadata—its description and input schema—to a function so the LLM recognizes it as a callable API.
• llm.bind_tools(tools)
  • Tells the model which tools exist and how they can be called.
  • There is no execution loop yet; this step merely enables the model to generate tool calls.
• tool_calls in the result of .invoke(...)
  • The model returns a structured request saying which tool to call with which arguments.
  • The result is therefore an execution plan rather than a natural-language answer.

(Example 2) Creating an execution loop with create_tool_calling_agent + AgentExecutor

from langchain_core.prompts import ChatPromptTemplate
from langchain.agents import create_tool_calling_agent, AgentExecutor
from langchain_openai import ChatOpenAI

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a helpful assistant. Use tools when needed."),
        ("human", "{input}"),
        ("placeholder", "{agent_scratchpad}"),
    ]
)

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

agent = create_tool_calling_agent(llm, tools, prompt)

agent_executor = AgentExecutor(
    agent=agent,
    tools=tools,
    verbose=True,
    max_iterations=10,
    max_execution_time=10,
    handle_parsing_errors=True,
)

result = agent_executor.invoke({"input": "Tell me the length of the word teddynote"})
print(result["output"])

This is where it becomes a genuine “agent.” Its execution flow usually looks like this:

1. A user input arrives.
2. The LLM decides what to do next: answer immediately or call a tool.
3. If it chooses a tool call, the corresponding Python function executes.
4. The tool result is sent back to the LLM.
5. The process repeats until the objective is satisfied.

Looking at the code line by line:

• ChatPromptTemplate.from_messages([... , ("placeholder","{agent_scratchpad}")])
  • agent_scratchpad is where intermediate logs and results from tool use are placed.
  • This space lets the LLM see prior tool results and decide its next action.
• create_tool_calling_agent(llm, tools, prompt)
  • Combines a tool-capable LLM with a prompt contract to create an agent.
• AgentExecutor(...)
  • Manages the actual execution loop, including repeated execution, iteration limits, and error handling.
  • max_iterations and max_execution_time prevent infinite loops and runaway execution.
• .invoke({"input": ...})
  • A single invocation may trigger several internal LLM calls and tool executions.
  • The final output is the answer delivered to the user.

Chains vs. Agents: Which Is the Better Fit?

The same problem can be solved in different ways. We might design a complex chain that queries a database, organizes the retrieved content, passes it to an LLM agent for summarization, and then hands the summary to another agent. But the same workflow could be implemented by turning each capability into a function and giving those functions to one agent as tools. Which approach—Chain or Agent—should we take in LangChain? Both can suit a service architecture depending on the situation, but their strengths and weaknesses are distinct.

• When chains are better Choose a chain when the execution path is explicit and fixed, operational reproducibility matters because the same stages should always run, and you want to tune failure points, performance, and cost stage by stage. Chains commonly perform predefined batch jobs on a regular basis and make root-cause analysis easier when a problem occurs.
• When agents are better Choose an agent when user queries vary enough that each request requires different steps; hard-coding conditional branches for follow-up questions—such as narrowing the period or adding search filters—would be complex; and many tools for search, databases, statistics, reranking, and summarization must be combined in too many ways to enumerate. In that setting, it is faster to build a “toolbox” and let the LLM choose.