How does AI convert documents into embeddings?
From messy PDF to clean numbers — the ingestion pipeline.
Picture it first
It's like making a smoothie for a robot. First you peel the fruit (extract the text), throw away the stickers and stems (headers, page numbers), cut it into small pieces (chunks), label each cup with what's inside (metadata), and then the blender (embedding model) turns each piece into a special number-juice the robot can taste and compare.
Now the same idea, in simple QA words
Before documents can be searched, they must be prepared. The system takes the text out of the files, cleans away junk (page numbers, headers), cuts it into pieces, adds labels (which file, which page, which date), and finally turns each piece into numbers (embeddings) and stores them. For QA: if text is read wrongly here — for example a scanned '₹5,00,000' becomes 'R5,0O,OOO' — the AI will later give wrong answers, and no prompt can fix it.
How the story connects to the real thing
Real-world example
An insurance company indexes 20,000 policy PDFs. Many are scanned images, so OCR is needed. One OCR setting reads '₹5,00,000' as 'R5,0O,OOO'. The embedding happily stores the wrong text, and later the bot quotes the wrong sum insured. The bug was in ingestion — no amount of prompt tuning will fix it.
Step by step, in plain words
- Read the text out of each file (use OCR for scanned files).
- Clean it.
- Cut it into pieces.
- Label each piece (source, page, date).
- Turn each piece into numbers.
- Save everything in the database.
How a tester tests it
- Spot-check extracted text against the source PDF — especially numbers, tables, and non-English text.
- Assert metadata is present and correct for every chunk (page number, document version, access level).
- Count: number of chunks in DB vs expected; detect silently skipped files.
- Test re-ingestion: updated docs replace old vectors; deleted docs disappear from search.
Try it yourself
1) Run a free PDF-to-text tool on a scanned document and compare the output line by line with the original. 2) Note every number that was misread.
Interview angle
“A RAG bot gives wrong amounts, but retrieval looks correct. Where do you look?”
At the ingestion pipeline. If the right chunk is retrieved but its text is wrong, the defect happened at extraction or OCR. I compare the stored chunk text against the source document, especially numbers and tables, and add ingestion-level assertions to the test suite.