In general, Large Language Models are incredibly capable. They can answer questions, summarize text, write code, and perform many other tasks.

But they are still general-purpose models. They don't automatically have access to your private or application-specific data.

So, what if you want an LLM to answer questions about your data?

For example, imagine you have an internal company document containing:

"Employees can work remotely for up to 3 days per week."

Now you ask an LLM:

"How many days can I work remotely?"

A general-purpose LLM doesn't automatically have access to that document.

You could provide the document along with the question, but imagine doing this with thousands of documents and hundreds of questions. Sending all that information to the LLM every time would be expensive, inefficient, and eventually run into context limits.

Instead, we can give the LLM access to the information it needs when it needs it.

This is where RAG — Retrieval-Augmented Generation comes in.

The idea is fairly simple:

User Question → Retrieve relevant information → Give it to the LLM → Generate an answer

For the same question:

"How many days can I work remotely?"

The RAG system first searches through the available documents and retrieves the relevant information:

"Employees can work remotely for up to 3 days per week."

That information is then passed to the LLM along with the user's question.

The LLM doesn't need to know our company's remote-work policy beforehand. It just needs the relevant context to generate an answer.

And while the basic idea sounds simple, it introduces a few interesting problems.

Let's understand them one by one.

Key Concepts

Chunking

At this point, our biggest problem is that we have to send the whole document as context to the LLM for it to answer our query.

But in most cases, the answer we're looking for might exist in just a small part of that document.

For example, imagine we have a 100-page company policy document, and the user asks:

"How many days can I work remotely?"

The answer might be just one sentence somewhere in those 100 pages.

There is no reason to send all 100 pages to the LLM when we only need that small piece of information.

So the first question becomes:

How can we break a large document into smaller, meaningful pieces so that we can later work with only the relevant context?

This process is called chunking.

Instead of treating the entire document as one large piece of text, we split it into smaller sections called chunks.

For example:

Document
│
├── Chunk 1
├── Chunk 2
├── Chunk 3
├── Chunk 4
└── ...

Now, instead of eventually giving the LLM the entire document, we can aim to give it only the chunks that are relevant to the user's question.

We'll worry about how to find those relevant chunks in the retrieval section.

For now, the important idea is:

We don't need the entire document to answer every question. We need the right part of the document.

And chunking is the first step towards making that possible.

Embeddings

Now we have our document split into smaller chunks.

But we still have a problem.

Suppose we have these chunks:

Chunk 1

Employees can work remotely for up to 3 days per week.

Chunk 2

Employees are eligible for 20 days of annual leave.

Chunk 3

Health insurance covers immediate family members.

And the user asks:

"How many days can I work from home?"

How do we know that Chunk 1 is the relevant one?

A simple keyword search might look for words like work, home, or days. But the question and the chunk don't necessarily have to use the exact same words.

For example:

"How many days can I work from home?"

and

"Employees can work remotely for up to 3 days per week."

use different words, but they have a similar meaning.

This is where embeddings come in.

Essentially, we convert text into a numerical representation called an embedding.

Chunk 1 → [0.12, -0.45, 0.78, ...]
Chunk 2 → [0.81,  0.21, -0.32, ...]
Chunk 3 → [-0.14, 0.67, 0.19, ...]

We can do the same thing with the user's question:

"How many days can I work from home?"
                ↓
        [0.15, -0.42, 0.74, ...]

The important idea is that text with similar semantic meaning should have embeddings that are closer together.

So the embedding for our question should be closer to Chunk 1 than to Chunk 2 or Chunk 3.

This allows us to search based on semantic similarity, rather than relying only on exact keyword matches.

And this idea of converting text into numerical representations isn't unique to RAG.

At a high level, this is also part of how modern language models work: text is ultimately converted into numerical representations that models can process and operate on.

RAG uses a similar idea for a different purpose — we use embeddings to make our documents searchable by meaning.

Now we have:

Document → Chunks → Embeddings

But we still need to compare the user's question against these embeddings and decide which chunks are relevant.

That's where retrieval comes in.

Retrieval

Now we have our document split into chunks, and each chunk has an embedding.

We also have an embedding for the user's question.

But having these embeddings alone doesn't give us an answer.

We need to actually find the chunks that are most relevant to the question.

Let's take our previous example.

The user asks:

"How many days can I work from home?"

We convert the question into an embedding and compare it with the embeddings of our stored chunks.

Conceptually, we might get something like:

Question
   ↓
Question Embedding
   ↓
Compare with chunk embeddings
   ↓
┌─────────────────────────────────────┐
│ Chunk 1 → 0.91                       │
│ Chunk 2 → 0.32                       │
│ Chunk 3 → 0.18                       │
└─────────────────────────────────────┘

Here, the numbers represent how similar each chunk is to the question.

Chunk 1 has the highest similarity, so it is the most relevant chunk.

We can then retrieve the top relevant chunks and provide them as context to the LLM.

User Question
      ↓
   Embedding
      ↓
Semantic Search
      ↓
Relevant Chunks
      ↓
Question + Retrieved Context
      ↓
     LLM
      ↓
    Answer

This is the core retrieval step in RAG.

Instead of sending the entire document to the LLM, we first search through our stored knowledge and retrieve only the information that is relevant to the current question.

There are different ways to perform this similarity search. One common approach is cosine similarity, which measures how close two embedding vectors are. I implemented this calculation directly in Python instead of hiding it behind a vector database, so I could understand the retrieval process better.

In production systems, we usually don't calculate this against every chunk ourselves. A vector database can store the embeddings and perform these similarity searches efficiently.

So now we have most of the core pieces:

Document → Chunking → Embeddings → Retrieval → Relevant Context

But we still need the final step.

How do we take this retrieved context and actually get an answer from the LLM?

Generation

So, we now have the relevant context.

But how do we actually use it to get an answer from the LLM?

The answer is simple: we pass the retrieved context along with the user's question to the LLM.

For example, the user asks:

"How many days can I work from home?"

Our retrieval step found this relevant chunk:

"Employees can work remotely for up to 3 days per week."

We can now provide both pieces of information to the LLM:

Context:
Employees can work remotely for up to 3 days per week.

Question:
How many days can I work from home?

The LLM uses the provided context to generate the answer:

"You can work remotely for up to 3 days per week."

And that's the Generation part of RAG.

The LLM isn't responsible for searching through the entire document. The retrieval system has already found the relevant information.

The LLM's job is now to use that context and generate a natural-language response to the user's question.

So if we put everything together:

Document → Chunking → Embeddings → Retrieval → Context → LLM → Answer

And that's the basic RAG pipeline.

How I Implemented the RAG Pipeline

Now that we understand the basic pieces of RAG, let's look at how I used them in this project.

The implementation has two main flows:

1. Upload flow — turn a Markdown document into searchable chunks.

2. Query flow — take a user's question, retrieve useful context, and generate an answer.

Tech Stack

• Backend: Python and FastAPI

• Database: PostgreSQL with SQLAlchemy

• AI: OpenAI-compatible text and embedding APIs, with Ollama for local generation

• Retrieval: NumPy cosine similarity

• Tooling: Docker and uv

Upload Flow

The upload flow starts when a user uploads a Markdown file with a short description of what the document contains.

First, the document is parsed into a structured tree containing its title, sections, subsections, headings, and content. I use the main LLM for this parsing step because Markdown files can have different structures. If the request times out, encounters a network error, or returns invalid JSON, a simpler regex-based parser falls back to reading the Markdown headings.

Next, the section tree is converted into chunks. Instead of cutting the document at arbitrary positions, each section becomes a meaningful chunk and keeps a breadcrumb such as:

Troubleshooting > Files Not Syncing

If a section is larger than 2,000 characters, it is split into smaller windows with a 200-character overlap. The overlap prevents useful context near a boundary from being lost.

Each chunk is then sent to the configured embedding API. The returned embedding, chunk text, heading information, and breadcrumb are stored in PostgreSQL. Embeddings are currently stored as JSON arrays rather than using pgvector directly.

I chose this approach because it keeps the implementation easy to inspect. PostgreSQL stores all document data in one place, while calculating cosine similarity in Python makes the retrieval logic visible instead of hiding it behind a vector database. This works well for a learning project and a small number of documents, although pgvector would be a better option at a larger scale.

Query Flow

The query flow begins by understanding the question, then decides how much context is needed before searching the document.

Step 1 — Handle Follow-Up Questions

The system checks for phrases such as "tell me more", "continue", and "what about". A normal question continues unchanged. For a follow-up, the last five cached question-and-answer turns for the document are loaded.

The main model, with Ollama as fallback, then chooses one of three actions:

• Answer: the recent history already contains enough context, so return an answer immediately.

• Clarify: the reference is unclear, so ask the user what topic they meant.

• Search: rewrite the follow-up as a standalone question and continue through the pipeline.

If no history exists, the API asks the user to clarify rather than searching for an ambiguous phrase.

Step 2 — Classify the Question

The prepared question is classified using simple keyword rules:

• Definition: what is, define, explain, or what does

• How-to: how do I, how to, steps, install, or set up

• Comparison: vs, compared to, difference, or better

• Troubleshooting: why doesn't, error, problem, fix, or broken

• Factual: the default when no other rule matches

The type then chooses the retrieval mode:

• Quick: definition and factual — top 3 chunks, threshold 0.78

• Balanced: how-to — top 5 chunks, threshold 0.68

• Detailed: comparison and troubleshooting — top 10 chunks, threshold 0.58

Step 3 — Check the Cache in Two Ways

Before validation or retrieval, the system checks whether the question has already been answered:

1. Exact match: normalize the question to lowercase and collapse extra spaces.

2. Semantic match: embed the question and compare it with previous question embeddings. A score of 0.90 or higher is a cache hit.

On either hit, the stored answer is returned immediately with cached: true. Scores between 0.85 and 0.90 are logged as near misses but continue through the pipeline.

Step 4 — Validate the Question

On a cache miss, the main model checks that the question is both related to the document description and written as a clear question rather than disconnected keywords. Invalid questions return isValid: false with a short reason. If the validation call itself fails, retrieval is allowed to continue instead of blocking the user.

Step 5 — Retrieve Relevant Chunks

The question is converted into an embedding and compared with every stored chunk embedding using cosine similarity in Python. Chunks must:

• pass the selected mode's minimum threshold,

• remain within 0.12 of the best score,

• and fit within the mode's top_k limit.

If nothing passes the threshold but weak matches exist, those nearest chunks are given to the main model to rewrite the query. Ollama is used if the main model fails. Retrieval is then attempted one more time. If the retry also fails, answer generation receives no chunks and returns no match.

Step 6 — Generate the Answer

The retrieved chunks and prepared question are sent to an answer model. A deterministic keyword check chooses the model order:

• Simple question: Ollama first, then the main model as fallback.

• Other question: main model first, then Ollama as fallback.

The model returns structured JSON containing whether the chunks actually satisfy the question and the answer message. If the chunks do not support an answer, or both models fail, the API returns "No matching information was found for your question."

Step 7 — Save and Return

Only successful answers are saved to conversation_history, together with the normalized question and its embedding. This row is used by both future cache lookups and follow-up resolution.

The result is logged as pipeline.answer_found or pipeline.answer_not_found, and the final response includes the original question, the query actually used for retrieval, whether retrieval rewrote it, its question type, retrieval mode, and cache status.

The cache is important for cost and latency, while validation and answer checking are important for correctness. Query rewriting improves recall when the user's wording differs from the document. Together, these steps make the system more reliable than a pipeline that simply retrieves the closest chunk and always asks an LLM to answer.

Key Features Beyond Basic RAG

The core RAG flow is only part of the implementation. I added several features around it to make the system more useful and resilient.

Conversation History

Conversation history makes short follow-ups understandable. Instead of embedding "tell me more" directly, the system uses recent turns to determine what the user means.

Query Rewriting

Query rewriting gives retrieval one more chance when the original question does not find a sufficiently similar chunk. The nearest weak chunks provide useful vocabulary for producing a better standalone query.

Model Routing

Model routing uses a deterministic keyword check. Simple questions such as definitions can be handled by a smaller local Ollama model, while questions involving comparison, architecture, integration, or troubleshooting go to the main model.

Fallbacks

Fallbacks prevent one model failure from breaking the whole request. Answer generation, follow-up resolution, and query rewriting can fall back to Ollama if the main model fails. Simple answer generation can also fall back in the opposite direction.

Caching

Caching supports both exact and semantic matches. Found answers are stored with the normalized question and its embedding, so repeated and closely paraphrased questions can return immediately.

Metrics and Rate Limiting

The project also records query events for metrics and applies simple per-IP rate limits. These are not part of RAG itself, but they helped me understand the operational pieces that surround a real question-answering system.

Limitations

This is a learning and portfolio project focused on understanding how a RAG system works end to end, so some production concerns are intentionally out of scope.

• No frontend yet. The main goal was the RAG pipeline itself; the API is used through Swagger (/docs) or curl.

• Markdown only. Chunking relies on Markdown headings. Other formats such as PDF, DOCX, and HTML could be supported by converting them to Markdown before upload.

• pgvector is not used directly yet. Embeddings are stored as JSON arrays in PostgreSQL and scored with cosine similarity in Python. This is fine for a few documents, but pgvector with an index would move the search into PostgreSQL and scale much better.

• Embedding search only. There is no keyword or full-text search alongside it, so exact terms such as error codes and version numbers can be missed.

• One document per query. Questions cannot currently span multiple documents.

• Shared history per document. Follow-ups use the last five question-and-answer turns for the document, rather than maintaining separate history per user or session.

Conclusion

Building this project showed me that RAG is much more than placing a document inside an LLM prompt. The quality of the final answer depends on how the document is chunked, how context is retrieved, how weak queries are rewritten, and how unsupported answers are rejected.

The most useful lesson was that many reliability improvements happen around the LLM, not inside it. Caching reduces repeated work, validation avoids irrelevant searches, conversation history makes follow-ups meaningful, and model routing balances local generation with a stronger fallback.

This implementation is intentionally small and transparent, but it covers the complete path from document upload to a grounded answer.

You push code to GitHub and your CI pipeline starts running. Then it happens:

Lint fails.

You fix it and push again.

Tests fail.

You fix that, push again, and now the build fails.After a few iterations, you've spent more time waiting for CI than actually fixing the problem.

The frustrating part isn't that the pipeline failed. It's that every failure usually means starting the entire pipeline again, even when only one step was the problem.

Your first instinct is to automate it with a shell script:

#!/bin/bash

go lint ./...
go test ./...
go build ./...
./scripts/deploy.sh

It works. The commands run one after another, and the script exits as soon as something fails.

But shell scripts have limits:

• A failure means starting from the beginning.
• Retrying a single step usually means rerunning the script manually.
• If the machine crashes halfway through, the execution state is gone.
• There's no built-in way to pause for approvals or inspect progress while it's running.

This is the problem Conductor tries to solve.

Instead of writing imperative scripts, you describe your CI pipeline in a simple YAML file, and Temporal executes it as a durable workflow.

That changes how a pipeline behaves.

Tech Stack

Conductor is built around a few simple technologies:

• Go — The main language used to build Conductor.
• Temporal — Handles workflow orchestration, retries, state, and durable execution.
• Bubble Tea — Powers the terminal UI (TUI).
• Docker Compose — Used to run Temporal and its dependencies locally.
• PostgreSQL — Used by the local Temporal setup for persistence.

For the terminal UI, I'm using Bubble Tea with the Dracula theme to keep the interface simple and consistent.

The local development environment can be started with Docker Compose, which sets up the Temporal server, Temporal UI, and PostgreSQL.

So the overall stack looks like this:

                    Conductor
                       │
             ┌─────────┴─────────┐
             │                   │
        Bubble Tea           Temporal
          (TUI)             (Workflow Engine)
                                 │
                         ┌───────┴───────┐
                         │               │
                     PostgreSQL      Temporal UI
                         │
                    Docker Compose

What is Conductor CI

Conductor turns a YAML pipeline into a Temporal workflow that you can monitor and control from your terminal.

The idea is simple:

• YAML defines what the pipeline should do.
• Temporal executes and manages the workflow.
• Terminal UI lets you monitor and control it.

Instead of writing a shell script, you describe your pipeline in YAML:

steps:
  - name: lint
    run: go lint ./...

  - name: test
    run: go test ./...

  - name: build
    run: go build ./...

  - name: deploy
    run: ./scripts/deploy.sh

Conductor takes this definition and runs it as a Temporal workflow. This gives your pipeline features that a simple script doesn't have. Lets understand in details

User Journey

Once you have a workflow.yaml in your project, getting started is simple:

$ task run

or execute the go binary from your workflow.yaml location folder. Checkout Readme for more details

Conductor starts in the terminal and gives you two choices:

Option 1: Validate Workflow

Before running anything, you can check whether your workflow.yaml is valid.

Select validate-workflow-yaml and Conductor parses and validates the YAML without connecting to Temporal or starting a workflow.

The checks are grouped to make it easy to understand what went wrong.

Nothing has been started at this point. No Temporal server. No worker. No side effects. Just validation.

You fix your YAML and run the validation again.

Option 2: Start Workflow

Once you're ready to run the pipeline, select start-workflow.

Conductor first performs the same validation. This makes sure an invalid configuration never reaches Temporal. If validation passes, Conductor moves on to Temporal:

The TUI: Watch It Happen in Real-Time

Now TUI Takes Over .Instead of printing a continuous stream of logs, Conductor gives you a live view of the workflow:

Each task has a simple status:

✓ — Task passed ✗ — Task failed ⟳ — Task is running ⏳ — Task is queued

You can also see elapsed time, stage progress, and interact with the workflow when needed.

The goal is simple:

Instead of watching logs, you watch the workflow itself.

What You Can Do During a Workflow (Features)

Now the workflow is running. You're not passively watching. You can control it.

Automatic retries

Tasks can retry on their own when they fail. That count lives on the task:

tasks:
  - name: test
    script: go test ./...
    retries: 2

If test fails, Temporal retries it according to that count. You do not press anything. The engine does it.

Manual retry

If a task still fails — and you have fixed the problem — you can retry just that task from the TUI.

build
  ✗ build

Press r to retry:

build
  ⟳ build

build
  ✓ build

Completed tasks do not run again. The workflow continues from the failed task.

Approval gates

Some stages should not continue until a person says yes:

execution:
  - stage: production
    tasks: [deploy]
    requires_approval: true

The workflow pauses at that stage:

production
  ⏳ deploy (awaiting approval...)

Press a to approve. Pressing a does not run deploy by itself. It lets the workflow continue. The deploy activity then goes onto the queue, and a worker picks it up.

Cancel a workflow

If something goes wrong, you can cancel the run from the TUI.

Terminate workflow? (y/n): y

Workflow terminated by user

The workflow stops. Its execution history is preserved.

Parallel and sequential tasks

You control how tasks inside a stage execute:

execution:
  - stage: quality
    tasks: [lint, format]
    mode: parallel

  - stage: deploy
    tasks: [deploy-staging, health-check]
    mode: sequential

Stages themselves still run top to bottom. Parallel is inside a stage, not across the whole pipeline.

Lets Understand Temporal

Temporal is a durable execution engine. You describe a long-running process as a workflow — a function that says what should happen, in what order, and what to do when something fails.

Temporal records every step of that function as history. If a machine dies, a deploy happens, or the process restarts, Temporal replays that history and continues. The run does not start over. It resumes.

That is the whole pitch, independent of CI. Temporal is not a dashboard for jobs. It is infrastructure for processes that must not get lost.

How Conductor Actually use Them

These are all the pieces we use,

A CI pipeline is that pattern. Lint, test, build, wait for approval, deploy. The scripts are activities. The human at the keyboard is signals. The live status on screen is a query.

For us, that process starts as YAML in the repo — not as something you click together in Temporal:

name: PR Validation

tasks:
  - name: lint
    script: echo "Linting From Terminal..."
    retries: 0
  - name: test
    script: echo "Testing From Terminal..."
    retries: 2
  - name: build
    script: echo "Building From Terminal..."
    retries: 1
  - name: deploy
    script: echo "Deploying From Terminal..."
    retries: 0

execution:
  - stage: validation
    tasks: [lint, test]
    mode: sequential
  - stage: build
    tasks: [build]
    mode: sequential
  - stage: deploy
    tasks: [deploy]
    mode: sequential
    requires_approval: true

Temporal never sees this file. It sees a workflow that means this file: stages in order, each task as an activity, deploy paused until a human arrives.

The moving parts

Temporal is a server plus workers. The server is the source of truth. Workers are how work actually happens.

The server and the task queue

When a workflow wants something done — “run this lint script,” “charge this card” — it does not run that code itself. It schedules an activity. That activity is placed on a task queue: a named inbox on the Temporal server.

A task queue is not a thread. It is a mailbox.

task_queue: conductor-ci-queue

# The workflow says:
#   put this activity on conductor-ci-queue
# The server holds it until a worker is listening.

What a worker is

A worker is a process you run. It connects to the Temporal server, registers the workflow and activity functions it knows how to execute, and polls a task queue.

When a task appears on that queue, a worker picks it up and runs the matching function. When it finishes, the result goes back to the server, and the workflow continues.

If no worker is running, the workflow is still recorded. Nothing executes until a worker comes online and starts polling. That split is the point: the server remembers the process; the worker is replaceable.

Registration is how the worker announces itself:

worker:
  listens_on: conductor-ci-queue
  workflows:
    - prWorkflow          # the process
  activities:
    - runTaskActivity     # the real work (your scripts)

Until that happens, Temporal has history, but no hands.

How an activity actually runs

The path is always queue → worker → result.

flowchart TD
  A[Workflow decides: run lint] --> B[Activity lands on the task queue]
  B --> C[Worker polls and picks it up]
  C --> D[Activity runs the script]
  D --> E[Result returns to the workflow]
  E --> F[Next stage, retry, or wait]

1. The workflow reaches a step that needs the real world.
2. It schedules an activity onto the task queue.
3. A worker polling that queue picks the activity up.
4. The worker runs the activity — in our case, the task’s script, in the project directory.
5. Success or failure goes back to Temporal. The workflow decides what is next.

Activities are allowed to fail. That is why they exist. The workflow stays deterministic; the activity is allowed to touch the world.

Timeouts and retries live on the activity. In YAML, that is the retries field on a task:

- name: test
  script: echo "Testing From Terminal..."
  retries: 2    # Temporal tries the activity up to 3 times on its own

Temporal UI

The screen is the product. Temporal is the engine behind it. It gives you a visual view of your workflow execution, making it easy to see:

• Which workflows are currently running or completed
• The status of each workflow
• Individual activities and their execution state
• Workflow execution history
• Retries and failures
• Signals and other workflow events

For Conductor, this is especially useful because you can see the pipeline you defined in YAML being executed by Temporal in real time.

Signals

A signal is how the outside world writes into a running workflow. It has a name. It becomes part of history. If the worker restarts, Temporal still knows the signal happened — or that the workflow is still waiting for it.

This is not a sleep. It is not a prompt that vanishes when you close the terminal. The workflow blocks on a named event. Until that event arrives, the process is paused and durable.

In general, signals are how you inject a human or another system: payment received, customer cancelled, manager approved.

We use two. Both are born from the YAML, then sent from the UI:

# In the file: this stage must not start until a human says yes
- stage: deploy
  tasks: [deploy]
  mode: sequential
  requires_approval: true

Queries

A query is how the outside world reads a running workflow. It does not change history. It does not wait for a step to finish. It asks: what is your state right now?

In general, queries are how you build a UI on a process you do not own: order status, where this onboarding is, whether this is still waiting on approval.

We register one:

query: execution-state

# What the UI asks, on a short interval:
#   which stages are queued, running, passed, failed, or awaiting?
#   how long did each task take?

The UI is not inside the workflow. The workflow is running on the worker. So the terminal asks that query and paints whatever comes back. That is why the screen can show a task as running while the worker is still executing the activity, and why approval shows up as a pause instead of a frozen app.

Conclusion

Building Conductor started as a way to explore how Temporal could be used for something familiar: CI pipelines.

Conductor is still a learning project, but building it gave me a much better understanding of Temporal, durable execution, signals, queries, and workflow orchestration.

You can find the complete project here:

Conductor CI on GitHub

Thanks for reading! 🚀

Before Starting on LSM Trees, Let's Rewind a Bit

Imagine it's 1970-something, and you're building a simple database for a manga library.

You need to store manga records:

Your database lives on disk.If Someone asks:

"Give me the manga with ID 3."

You need to find that record as quickly as possible.So let's forget about fancy database data structures for a moment.

How would you store these records if you had to build this yourself?

The Simplest Thing You Could Do

The most obvious solution is to just write the records to disk one after another, in whatever order they arrive.

Disk:

Position 1 → Manga ID 5
Position 2 → Manga ID 2
Position 3 → Manga ID 8
Position 4 → Manga ID 1
Position 5 → Manga ID 7

Now someone asks:

"Find me Manga ID 8."

Well... we don't know where it is. So we start reading from the beginning:

Position 1 → Manga 5 → not it
Position 2 → Manga 2 → not it
Position 3 → Manga 8 → found it!

For five records, who cares? But what if we have a million manga?

Now finding one manga could mean reading hundreds of thousands of records before we get to the one we want. And there's a problem with that:

Disk reads are expensive.

We don't want our database spending most of its time reading records that have nothing to do with the query. So maybe we can organize the data a little better.

What If We Sort It?**

Instead of writing manga in whatever order they arrive, we could keep them sorted by manga_id.

Disk:

Position 1 → Manga ID 1
Position 2 → Manga ID 2
Position 3 → Manga ID 5
Position 4 → Manga ID 7
Position 5 → Manga ID 8

That's already better.

If we're looking for Manga ID 8, we know the IDs are ordered. Once we reach an ID greater than 8, we can stop.

But there's still a problem.

With a million records, we might still have to read a huge number of records before getting to the one we want. Sorting tells us how the data is ordered.

It doesn't tell us where to jump.

What If We Had an Index?

What if we kept a small amount of information separately that told us roughly where different ranges of IDs live?

Something like:

Index:

Manga IDs 1–250,000
        ↓
Disk Position 1

Manga IDs 250,001–500,000
        ↓
Disk Position 500,001

Manga IDs 500,001–750,000
        ↓
Disk Position 1,000,001

Now suppose we're looking for:

Manga ID 8

Instead of starting from the beginning and blindly scanning through records, we first check the index.

The index tells us:

Manga ID 8
    ↓
IDs 1–250,000
    ↓
Start around Disk Position 1

We have narrowed down the search before touching the actual records. And that's a pretty useful idea.

Instead of storing only the data, we also keep some information about where the data is.

The question now becomes:

How do we build an index that lets us find things quickly without the index itself becoming huge?

That's where B-Trees come in.

So, What Does A B-Tree Look Like?

We now have a basic idea of what an index can do.

Instead of searching the entire dataset, we keep some information that helps us quickly figure out where to look.

But there's a new problem.

If we have a million, or even a billion, records, we can't just keep a million-entry index and scan through that index too.

We need the index itself to be searchable.

This is where the B-Tree comes in.

A B-Tree is essentially a tree designed specifically for storing and searching data efficiently on disk. Let's build a very small one.

A Simple B-Tree

Suppose our manga IDs are:

1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12

A simplified B-Tree might look something like this:

                         [ 4 | 8 ]
                       /     |     \
                      /      |      \
             [1 | 2 | 3] [5 | 6 | 7] [9 | 10 | 11 | 12]

The important thing to notice is that the tree doesn't have just one value per node.

A node can contain multiple keys.

The keys divide the data into ranges.

For the root:

                 [ 4 | 8 ]

we can think of it as three ranges:

        < 4        4–8        > 8
         ↓          ↓           ↓
    left child   middle child  right child

So the root doesn't contain every manga.

It simply helps us decide which part of the tree we need to look at.

Searching for a Manga

Let's say someone asks:

"Find Manga ID 10."

We start at the root:

[ 4 | 8 ]

We compare 10 with the keys:

10 > 4
10 > 8

So we know the answer, if it exists, must be in the right child.

We follow that pointer:

                         [ 4 | 8 ]
                              |
                              ↓
                    [9 | 10 | 11 | 12]

Now we search inside that node:

[9 | 10 | 11 | 12]
       ↑
     found!

That's it. We didn't scan all 12 records. We followed the path that could actually contain ID 10. With a much larger tree, the same idea continues.

For example:

                         [ 40 | 80 ]
                       /      |      \
                      /       |       \
             [10|20|30] [50|60|70] [90|100|110]

If we're searching for 70:

[40 | 80]
     |
     ↓
[50 | 60 | 70]
          |
        found

The tree keeps reducing the amount of data we need to examine.

Why Does a B-Tree Have Multiple Keys Per Node?

This is an important detail.

Remember where our data lives:

On disk.

Reading from disk is expensive, so we don't want a tree where every tiny node requires another disk read.

Instead, a B-Tree packs many keys and pointers into a single node, typically sized to work well with disk pages.

Conceptually:

Disk Page
┌──────────────────────────────────────┐
│ Key │ Key │ Key │ Key │ Key │ ...    │
│     │     │     │     │     │        │
│ Pointer │ Pointer │ Pointer │ ...     │
└──────────────────────────────────────┘

One disk read can therefore give us a lot of information.

The tree can have a very large number of records while remaining relatively shallow.

That's one of the main reasons B-Trees work so well for disk-based databases.

B-Trees In Production

B-Tree and B+Tree variants are widely used in real database systems.

For example:

• PostgreSQL uses B-tree indexes as its default index type.
• MySQL/InnoDB uses B+Tree-style indexes for its primary and secondary indexes.
• SQLite uses B-trees for tables and indexes.

So when you create something like:

CREATE INDEX idx_manga_id
ON manga(manga_id);

the database isn't necessarily doing some magical lookup.

It's maintaining a data structure that lets it efficiently navigate to the records — commonly a B-tree variant.

So What's the Problem?

At this point, B-Trees sound pretty great.

They give us:

• Fast lookups
• Ordered data
• Efficient range queries
• An index that doesn't require scanning everything
• Good performance with data stored on disk

So why would we ever need anything else?

The answer becomes obvious when we stop reading and start writing.

Imagine our database already contains:

                         [ 4 | 8 ]
                       /     |     \
                      /      |      \
                 [1|2|3]  [5|6|7]  [9|10|11]

Now a new manga arrives:

Manga ID = 6

Where does it go?

It belongs inside:

[5 | 6 | 7]

That sounds easy.But real database pages have limited space. Suppose the node is already full:

[5 | 6 | 7 | 8]

and we need to insert:

6

The database can't simply keep adding data forever.It may need to split the node. Something like:

Before:

              [ 4 | 8 ]
             /
      [5 | 6 | 7 | 8]

After the split, the tree structure may need to change:

              [ 4 | 6 | 8 ]
             /     |     \
            /      |      \
       [5]        [7]     [...]

The exact details are more complicated in a real B-Tree/B+Tree, but the important part is this:

A write isn't necessarily just writing the new record.

The database may need to:

1. Find the correct page.
2. Read that page.
3. Modify it.
4. Potentially split the page.
5. Update parent nodes.
6. Potentially split those parents too.
7. Write multiple modified pages back to disk.

And remember:

Disk I/O is expensive.

Now Imagine Heavy Writes

Suppose we're building something where data is arriving constantly:

10,000 writes/sec
100,000 writes/sec
1,000,000 writes/sec

Every write potentially involves modifying existing structures that are already sitting on disk. The problem isn't that B-Trees are bad. They are actually extremely good at what they were designed to do.

The problem is that frequent random writes to disk can become expensive. And this gives us a new question:

What if we didn't immediately modify the existing data structure on disk every time a write arrived?

What if we could accept writes somewhere much faster first, and deal with organizing them later?

That question takes us in a completely different direction.

And that's where LSM Trees begin.

LSM Tree

We just saw the main limitation of a B-Tree:

Writes modify existing data on disk.

What if we avoided that?

Instead of immediately finding the right place on disk and modifying it, we can keep new writes in memory and write them to disk sequentially in batches.

That's the basic idea behind an LSM Tree (Log-Structured Merge Tree).

A write follows roughly this path:

User Write
    ↓
  WAL
    ↓
Memtable
    ↓
  Flush
    ↓
SSTable
    ↓
Compaction

Lets Understand All Components in Details

WAL (Write Ahead Logs)

The first problem we need to solve is durability. Our writes are going into the Memtable, which lives in memory. That's fast, but memory is not permanent.

Imagine we receive:

PUT manga:10 Berserk

If we put it directly into the Memtable:

Memtable

manga:10 → Berserk

and the process crashes, the write is gone. So we first record the operation on disk in a Write-Ahead Log (WAL) and then put it into the Memtable:

User Write
    ↓
   WAL
    ↓
Memtable

A WAL is a common durability mechanism used by many databases, including traditional B-Tree-based databases. The idea is simply to record the operation before modifying the actual database state.

The WAL is an append-only log:

PUT manga:10 Berserk
PUT manga:11 One Piece
DELETE manga:12

We don't modify old entries. New operations are appended to the end.

The order is important:

1. Write to WAL
2. Write to Memtable

If the process crashes after step 1, the operation is still in the WAL. When the database starts again, it can replay the WAL and rebuild the Memtable.

WAL
 │
 │ replay
 ▼
Memtable

In my go implementation , we follow the same order.

wal.AddOperation("PUT", key, &value)
memtable.Put(key, value)

Now that the write is safely recorded, we need somewhere to accumulate these writes in memory.

That's the job of the Memtable.

Memtable

Now that the write is safely recorded in the WAL, we need somewhere to keep it while it's still in memory.

That's the job of the Memtable.

A Memtable is an in-memory data structure that holds the latest writes before they are written to disk.

For example:

Memtable

manga:10 → Berserk manga:11 → One Piece manga:12 → Naruto

Because the data is in memory, writes are fast. We don't need to perform a disk write for every operation. The Memtable also keeps its keys sorted:

10 → Berserk 11 → One Piece 12 → Naruto 13 → Monster

This becomes useful when we eventually write the Memtable to disk.

What happens when it fills up? Memory is limited, so the Memtable can't keep growing forever. Once it reaches its size limit, we take its contents and flush them to disk as a new sorted file.

Memtable ↓ Flush ↓ SSTable

The important part is that we don't update an existing file. We create a new SSTable containing the Memtable's data. After the flush, the Memtable can be cleared and start accepting new writes.

Memtable ↓ SSTable 1

Memtable ↓ SSTable 2

Memtable ↓ SSTable 3

This is where the LSM approach starts to become interesting: writes keep creating new immutable files instead of modifying existing files.

So now we have another problem:

What exactly is an SSTable, and how do we store and read these files?

3. SSTable

When the Memtable fills up, we need to move its data to disk.

Instead of modifying an existing file, we write the entire sorted Memtable into a new file.

This file is called an SSTable, or Sorted String Table.

For example, our Memtable might contain:

Memtable

10 → Berserk
11 → One Piece
12 → Naruto
13 → Monster

When it is flushed:

Memtable
    ↓
  Flush
    ↓
SSTable

10 → Berserk
11 → One Piece
12 → Naruto
13 → Monster

The data is already sorted, so we can write it to disk in sorted order.

SSTables are immutable

Once an SSTable is written, we never modify it.

Suppose we later update:

10 → Berserk

to:

10 → Berserk Deluxe Edition

We don't open the old SSTable and change it.

The new value goes through the same process:

WAL
 ↓
Memtable
 ↓
New SSTable

So for some time, we might have:

SSTable 1

10 → Berserk


SSTable 2

10 → Berserk Deluxe Edition

When reading, we check the newest SSTables first. The newer value therefore takes precedence. This is one of the key differences from the B-Tree approach we saw earlier: we don't update old files in place; we keep writing new ones.

But there is a problem

Every Memtable flush creates another SSTable.

After enough writes, we could end up with:

SSTable 1
SSTable 2
SSTable 3
SSTable 4
SSTable 5
...

Now a read may have to check multiple files.

Some of those files may also contain old versions of the same keys.

So we need a way to merge these files and remove data that is no longer needed.

That's where compaction comes in.

4. Compaction

We now have multiple SSTables on disk:

SSTable 1
10 → Berserk
11 → One Piece

SSTable 2
10 → Berserk Deluxe Edition
12 → Naruto

SSTable 3
13 → Monster
14 → Death Note

There are two problems here:

1. A read may need to check multiple SSTables.
2. Older SSTables may contain values that have already been replaced.

We can solve both by merging SSTables together.

This process is called compaction.

SSTable 1 ──┐
SSTable 2 ──┼──→ Compaction → New SSTable
SSTable 3 ──┘

Since each SSTable is already sorted, we can efficiently merge them.

For duplicate keys, the newest value wins:

SSTable 1
10 → Berserk

SSTable 2
10 → Berserk Deluxe Edition

After compaction:

New SSTable

10 → Berserk Deluxe Edition

The older version can now be discarded.

The same applies to deletes. Instead of immediately removing a key from older SSTables, we record a tombstone indicating that the key was deleted. During compaction, that tombstone can be used to remove the older value.

So instead of:

SSTable 1
SSTable 2
SSTable 3
SSTable 4
SSTable 5

we periodically reduce them to fewer, larger files.

SSTable 1 ──┐
SSTable 2 ──┤
SSTable 3 ──┼──→ Compaction
SSTable 4 ──┤
SSTable 5 ──┘
                 ↓
             SSTable 6

The goal is simple:

Keep the number of SSTables manageable and remove old versions of data.

At this point, we have the core LSM Tree write path:

Write
  ↓
 WAL
  ↓
Memtable
  ↓
SSTable
  ↓
Compaction

The next question is now about the read path: if the same key can exist in the Memtable and multiple SSTables, how do we find the correct value efficiently?

Reading and Deleting Data

Now that we have all the components, let's look at what actually happens when we read or delete a key.

Read

A read starts with the newest data and moves towards older SSTables:

GET manga:10
      ↓
  Memtable
      ↓
Newest SSTable
      ↓
Older SSTable
      ↓
    ...

If the key is found, we return the value immediately.

But checking every SSTable can become expensive as the number of files grows.

So we use a Bloom Filter for each SSTable.

Before reading a file, the Bloom Filter tells us whether the key might exist in that file:

GET manga:10
      ↓
  Memtable
      ↓
 Bloom Filter
   /       \
  No       Maybe
  ↓          ↓
Skip      Read SSTable

A No means the key definitely isn't there, so we skip the disk read.

A Maybe means we have to check the SSTable. Bloom Filters can have false positives, but they never have false negatives.

This makes reads much cheaper when we have many SSTables.

Delete

Because SSTables are immutable, we can't simply remove a key from an old SSTable.

Instead, a delete is written as a tombstone:

DELETE manga:10

The tombstone is stored like any other write and eventually ends up in an SSTable.

During a read, if we encounter the tombstone:

GET manga:10
      ↓
SSTable
      ↓
10 → DELETE
      ↓
  Not Found

We stop searching. This prevents an older SSTable from returning a value that has already been deleted.

During compaction, the old value and its tombstone can eventually be removed together.

Closing Thoughts

Reading about LSM Trees is one thing, but implementing one makes the trade-offs much easier to understand.

I built a small LSM-based key-value store in Go from scratch to get a better feel for how the pieces actually fit together — from the WAL and Memtable to SSTables, compaction, tombstones, and Bloom Filters.

The implementation is intentionally simple and is meant for learning rather than production use.

You can find the complete project here:

LSM KV Store in Go

The biggest takeaway for me was that an LSM Tree isn't really about one complicated data structure. It's about making a series of trade-offs:

Make writes cheap by avoiding in-place updates, and pay the cost later through compaction.

Once you look at it this way, the different pieces of an LSM Tree start to make a lot more sense.