I have a small problem with "chat with your PDF" projects.
Not because they're bad projects. They're actually a great way to learn how retrieval-augmented generation works.
It's just that most of them stop at the exact point where things start getting interesting.
Upload a PDF. Split it into chunks. Create embeddings. Put them in a vector database. Ask a question. Send the closest chunks to a language model. Done.
And technically, yes, you built RAG.
But I kept getting stuck on one sentence in that whole process:
Find the relevant chunks.
Relevant according to what?
What if the question uses completely different words from the document? What if it contains an exact product code that embeddings don't particularly care about? How big should a chunk even be? How many chunks should I retrieve? And when I change any of this, how do I know I actually made the system better?
So I built my own version.
The product itself is intentionally simple. You create a workspace, upload documents, ask questions, and get answers with sources you can open and check.
The system underneath it is where I spent most of my time.
This isn't really a tutorial. It's more a walk through the decisions that made me stop, rethink something, delete something, or decide that the obvious choice was not actually the right one.
I didn't want to build another chat interface
The chat was almost the least interesting part.
There are already a thousand ways to build a nice message box, stream tokens into it, and make it look vaguely like ChatGPT.
What I cared about was what happens before the model ever receives the question.
When somebody uploads a document, the system has to:
- store the original file
- extract its text
- clean that text without destroying useful structure
- break it into pieces
- keep track of where every piece came from
- turn those pieces into embeddings
- index them
- and only then decide that the document is ready to search
So what looks like a simple upload is really a small data pipeline.
upload
↓
store
↓
extract
↓
clean
↓
chunk
↓
embed
↓
index
↓
ready
I liked that.
It made the project feel less like "an LLM app" and more like a real data system that happened to have an LLM at the end.
A 100-page PDF should not be an HTTP request
My first obvious decision was moving document processing into the background.
I could have made the upload endpoint accept the file, parse the whole thing, generate all the embeddings, write everything to the database, and finally return when it was done.
That would technically work.
It would also mean keeping an HTTP request open while a PDF goes through several slow and failure-prone operations.
Not ideal.
So the API does the smaller job.
It accepts the upload, stores the file, creates the document record, puts a job on a queue, and gets out of the way.
The worker handles the expensive part.
The user sees something much simpler:
Uploading → Reading document → Preparing content → Ready
Internally, there are more states than that. Extracting, chunking, embedding, indexing, failed, retrying.
The user doesn't need to know all of them.
I think technical products get heavy very quickly when the interface starts exposing the implementation instead of the information somebody actually needs.
Nobody needs to see:
embedding batch 7/12
They need to know whether their document is ready.
That difference sounds tiny. It isn't.
Chunking is annoyingly important
Before building this, chunking sounded like the boring step.
Take the text, split it every few hundred tokens, add a little overlap, move on with your life.
Then you actually try it.
Large chunks preserve a lot of context, which sounds great until one chunk contains four unrelated paragraphs and only one of them answers the question.
Small chunks give you more precise retrieval, except now you can retrieve a sentence that makes absolutely no sense because the heading or previous paragraph got cut away.
So the question stopped being:
What's the best chunk size?
and became:
How much context does this particular piece of text need to still make sense?
That's a much less convenient question.
There isn't one magic number.
So I tried to respect the document structure where I could. Headings stay with the section below them. Paragraph boundaries are preferred over arbitrary cuts. Page information survives because I need it later for citations.
When that structure isn't available, I fall back to token-based splitting with a small overlap.
Nothing revolutionary.
But it worked better than pretending every document is one giant string.
And more importantly, I kept those settings configurable so I could actually test them later instead of choosing 512 because I saw it in somebody else's tutorial and never thinking about it again.
Vector search was not enough
This was probably my favorite part of the project.
Embeddings are genuinely useful.
Imagine the source says:
Employee credentials must be revoked immediately after termination.
And somebody asks:
What happens to an employee's account when they leave the company?
Those sentences use different words, but they mean roughly the same thing.
Semantic search handles that beautifully.
Then you ask:
What does SEC-104 say?
And suddenly meaning is not the interesting part anymore.
SEC-104 is an exact identifier. I don't want the system to get creative about what is semantically similar to it. I want the thing called SEC-104.
Same problem with:
- product names
- plan names
- error codes
- acronyms
- policy numbers
- people's names
- exact technical terms
So I stopped treating vector search as the search system.
It's one search system.
The other one is plain lexical search.
The app runs both.
Semantic search asks:
What passages mean something similar to this question?
Keyword search asks:
Where do these actual words appear?
Then I combine the results.
Simple idea, big improvement.
The scores don't actually agree with each other
Once you have two search systems, you immediately get another problem.
Vector search gives you one kind of score.
Full-text search gives you another.
It is tempting to do something like:
final_score = vector_score + keyword_score
Lovely.
Except those scores don't mean the same thing.
A 0.84 from vector similarity and a 0.84 from a lexical ranking function are not two versions of the same measurement.
So instead of trying to invent a perfect formula for comparing them, I mostly care about rank.
Was this chunk first in semantic search?
Was it third in keyword search?
Did both systems independently think it was useful?
A rank-fusion approach lets me combine that information without pretending the underlying scores are directly comparable.
I like decisions like this because they're boring in a good way.
No machine learning.
No complicated tuning.
Just admitting that two numbers mean different things and not forcing them together because addition is convenient.
Retrieve broadly, then get picky
My first retrieval pipeline was basically:
question
↓
find nearest chunks
↓
send them to model
It works.
Until "nearest" and "actually useful for answering the question" stop being the same thing.
A passage can be very related to a topic without containing the answer.
So I changed the first search stage to optimize more for recall.
Its job is not to perfectly answer the question.
Its job is:
Please don't throw away the right passage too early.
I retrieve a slightly broader candidate set first.
Then those candidates go through a reranking step.
The reranker sees both the question and the candidate passage and gives me a better answer to:
Does this passage actually help with this question?
So the pipeline becomes:
question
↓
semantic search + keyword search
↓
merge candidates
↓
rerank
↓
select context
↓
generate answer
This ended up giving me a very simple mental model:
Retrieve broadly. Rank carefully. Generate last.
The trade-off is that reranking costs time.
There is no point retrieving fifty passages and spending forever reranking all of them because "more context must be better."
More context is often just more noise.
So I keep the candidate count controlled.
A better pipeline does not necessarily mean a bigger pipeline.
I kept the vector database inside Postgres
This is probably the choice people will argue with the most.
I used PostgreSQL with pgvector.
I did not add Pinecone, Qdrant, Weaviate, Milvus, or another dedicated vector database.
Not because those tools are bad.
Mostly because I already needed Postgres.
The application has users, workspaces, documents, conversations, messages, permissions, processing states, citation metadata, and a bunch of normal relational data.
Postgres was already sitting there doing all of that.
At the scale of this project, letting it store the embeddings too meant one less system to run, configure, deploy, authenticate against, monitor, and explain.
There's another nice side effect.
Retrieval is always scoped.
A user should only search documents in a workspace they can access.
Sometimes they should only search two selected files.
Those relationships already live in the relational database, so filtering vectors against them is very natural when everything is in the same place.
Would I still do this with hundreds of millions of vectors?
Maybe not.
At that point I'd actually benchmark the problem in front of me and probably look harder at dedicated vector infrastructure.
But I didn't want to build for imaginary scale just because the architecture diagram would look fancier.
One database was enough.
So I used one.
The citations became more useful than I expected
Originally, citations were a product feature.
You ask something.
You get:
Employees have 30 days to return company equipment [1].
And [1] opens the supporting source.
Nice.
Then I started debugging the system and realized the citations were doing something much more useful.
When an answer is wrong, there are at least two very different things that may have gone wrong.
Problem one: retrieval found the wrong passages.
Problem two: retrieval found the correct passages, and the model still produced a bad answer.
Without sources, both failures just look like:
The AI got it wrong.
With citations, I can immediately inspect the context that reached the model.
Wrong source? Retrieval problem.
Right source, wrong answer? Generation problem.
That sounds obvious once you say it, but it changed how I thought about citations.
They are not decoration.
They are part of the debugging system.
They just happen to also make the product more trustworthy for the person using it.
Sometimes the correct answer is "I don't know"
One thing I did not want was the system desperately trying to answer every question.
If you upload an employee handbook and ask:
What was Apple's revenue last year?
the answer should not suddenly come from whatever the base model remembers about Apple.
That's not what the product is for.
The model is told to answer from the retrieved sources and be clear when those sources are not enough.
So sometimes the best result is basically:
I couldn't find enough information in these documents to answer that confidently.
I like that.
Confidence is cheap when you're generating text.
Evidence is harder.
The awkward part is that this only works properly if retrieval is good.
If the information is in the documents but your retrieval system fails to find it, the model saying "I don't know" is technically well-behaved and still completely useless.
Which is how I ended up building the part I should have probably built earlier.
I got tired of evaluating the system by vibes
RAG evaluation can become very scientific very quickly.
I did not need that.
I just needed something better than:
Hmm, this version feels pretty good.
So I made a small evaluation dataset.
A question.
The document or passage that should answer it.
Run retrieval.
Check what came back.
That's enough to calculate a couple of useful things.
The first is Recall@K.
I mostly think of that as:
Did the right source show up somewhere in the first few results?
If it didn't, nothing later in the pipeline can rescue the answer.
The model cannot use a passage it never received.
The other metric I care about is MRR, or Mean Reciprocal Rank.
Terrible name if you're trying to make something sound approachable.
The idea is simple:
If the correct result was found, how close to the top was it?
Correct passage at position one: great.
Correct passage hiding at position twelve: technically found, practically less useful.
And then I track latency because there is always a wonderfully stupid way to improve quality by making everything ten times slower.
With those few measurements, I can actually change things like:
- chunk size
- overlap
- number of retrieved candidates
- hybrid search behavior
- reranking depth
and compare what happened.
It's not a research lab.
It's enough to stop guessing.
No, I didn't make it an agent
I could have.
Right now you can turn nearly anything into an agent if you put enough arrows on a diagram.
One agent could inspect the question.
Another could choose which documents to search.
Another could rewrite the query.
Another could judge whether retrieval was good.
Another could ask for more retrieval.
Another could write the final response.
And now we've got six models discussing a PDF.
I didn't really want that.
The core problem in this project is retrieval, and I wanted that pipeline to stay visible.
A question comes in.
I can follow it through search, ranking, context selection, generation, and citations.
If something goes wrong, I have a fairly small number of places to look.
I like that.
If I had a real product requirement that needed planning, tool selection, or multi-step actions, I'd happily reconsider agents.
But I wasn't going to introduce them just because this is an AI project and agents are currently the cool thing to put in one.
I also refused to make it microservices
Same problem, different part of the stack.
The application has several responsibilities.
Authentication.
Documents.
Processing.
Retrieval.
Conversations.
Evaluation.
Generation.
I could draw a box around each one, give it a port, deploy it independently, and spend a weekend learning a lot about network errors I didn't need to have.
Instead, I kept the backend modular and boring.
The API is one application.
The worker is a separate process because background processing actually has different operational needs.
Postgres is Postgres.
Redis handles the queue.
Object storage keeps the raw files.
That's plenty.
The modules are separated in code so I can reason about them independently without pretending they need to be independently deployable.
If ingestion eventually became ten times heavier than query traffic, maybe I would split things further.
If different teams owned different pieces, maybe I would split things further.
If one part needed completely different scaling, maybe I would split things further.
Those are reasons.
"Microservices are scalable" is not a reason by itself.
The frontend was not allowed to look like homework
I care about this more than I probably should.
A lot of technically good portfolio projects look terrible.
You open them and immediately know that the frontend was built at 1:40 in the morning after the "real engineering" was finished.
Gray card.
Blue button.
Five tables.
A spinner.
Done.
I did not want that here.
The whole point of the app is that something fairly complicated underneath should feel calm on top.
The document library has useful empty states.
Processing states are understandable instead of technical.
The chat does not throw retrieval scores at the user.
Citations open into a source view instead of dumping raw JSON.
Loading states are proper skeletons where they make sense.
The interface works on a smaller screen instead of politely collapsing into chaos.
I wanted the product to feel like one thing, not a FastAPI project with a React demo glued to it.
A few things I picked up
Retrieval quality is not model quality
This was probably the biggest one.
Changing the language model can improve the final answer, obviously.
But a very good model with bad context is still working from bad context.
Some of the most noticeable improvements came from things that never touched the generation model at all.
Better chunking.
Keeping headings.
Hybrid search.
Reranking.
Those are quieter improvements, but they affect what information the model gets to see in the first place.
Exact search is still very useful
Embeddings are clever.
Sometimes I don't need clever.
If the user asks about SEC-104, I would quite like the system to search for SEC-104.
There is something slightly funny about using a fairly sophisticated semantic retrieval system and then realizing:
Ah yes. Ctrl+F still has value.
It absolutely does.
Small architecture is easier to improve
Using Postgres for relational data and vectors is not the most exotic setup.
Keeping the backend as a modular application is not very exotic either.
That's sort of why I like both decisions.
There are fewer moving pieces, which means I can spend more of my time improving the actual behavior of the product.
Complexity should buy you something.
If it doesn't, it's just rent.
"I don't know" is a feature
I think we overvalue the ability of AI products to always respond with something.
For this product, refusing to answer when there is no supporting source is part of the contract.
I'd rather get a boring truthful response than a beautifully written paragraph the documents never said.
Things that are still imperfect
There are plenty.
Scanned PDFs are not the interesting case yet. Proper OCR would be the next step there.
Tables are awkward because flattening a table into plain text can destroy the relationships that made the table useful in the first place.
Documents with complex layouts would benefit from layout-aware parsing.
Retrieval evaluation is intentionally small. If this became a real product, I would want more continuous evaluation, more failure analysis, and datasets built from real usage.
And eventually, yes, there may be enough vectors that Postgres stops being the obvious choice.
That's fine.
I don't need the first architecture to be the final architecture.
I just need the decisions to make sense for the system that actually exists.
What I would change if this got much bigger
If this project suddenly had a lot more users, the first thing I would not do is immediately split everything into fifteen services.
I would look at where the actual pressure is.
If document ingestion becomes the bottleneck, I can scale workers separately.
If embedding calls become expensive, batching and caching become more important.
If retrieval gets slow because the corpus becomes huge, then the vector storage decision deserves another look.
If teams start sharing workspaces, permissions become much more interesting than they are right now.
If people upload scanned documents, OCR stops being an optional feature.
If they upload financial reports full of tables, plain text extraction is no longer good enough.
If they start asking questions across thousands of documents, query rewriting and better filtering probably become worth testing.
That's the kind of scaling I find useful to think about.
Not:
What architecture would survive one billion users?
More:
What part would actually break first, and what would I change when it does?
The part I like most is still the boring part
There is something funny about building an AI project and ending up most interested in the parts that do not look very "AI."
The worker queue.
The document states.
The chunk metadata.
The rank fusion.
The source references.
The evaluation dataset.
None of those things are impressive in isolation.
Together, they are what make the model useful.
The model is very good at writing an answer.
The harder part is making sure it gets the right information before it starts writing.
So, that's the project
The visible part is still very simple.
Upload something.
Ask a question.
Get an answer.
Check the source.
But I think the useful lesson for me was that the answer is almost the end of the story.
Before that answer appears, somebody has to decide how a document becomes data, how that data gets searched, how several kinds of relevance get combined, how much context is enough, which passages deserve to reach the model, and how you know any of it is actually improving.
That's the part I wanted to understand.
The chat just gave me somewhere nice to put it.
With that being said, see you in the next one.
Much love,
Yassine Erradouani