How it works

~200 arXiv papers on RAG, agents, transformers, embeddings, prompting, and evaluation — downloaded as PDFs with their titles, authors, and abstracts.

A paper is too big to search as one unit — so each is cut into index-card-sized chunks of ~800 tokens, split along the paper’s own section boundaries, with a 150-token overlap between neighbors so no sentence is ever lost on a cut line. One chunk ≈ one idea — that focus is what makes a chunk findable later.

You can’t search pixels. So every extracted figure is shown to a vision model that writes a text description of it, and every table becomes markdown plus a summary line. The description is what gets searched — the original figure or table is what gets returned. Search the label, deliver the source.

Each chunk — text, figure description, table description — is converted into an embedding: a list of 1,536 numbers that encodes its meaning. Chunks about similar ideas end up with similar numbers, even when they share no keywords. That’s the property everything else is built on.

All chunks + their embeddings land in PostgreSQL with the pgvector extension — three indexes: paper text, figures, tables, all in one shared embedding space, so a single question can search all three at once. Ingestion is incremental: re-running it only embeds what’s new.

The search query is embedded with the same model as the library — meaning can now be compared to meaning, as geometry: similar meaning = nearby points.

pgvector computes the cosine similarity between the question’s fingerprint and every chunk’s, then returns the closest matches: top 8 text chunks, top 2 figures, top 2 tables. This is classic nearest-neighbor search — the ranking is the geometry.

Top-k has a trap: it always returns k results, relevant or not. So there’s a similarity floor of 0.30 — chunks scoring below it are treated as “no real match.” When the writing agent checks a specific claim and nothing clears the floor, it must drop or soften the claim rather than invent support. Honest “not found” beats confident nonsense.

The Researcher doesn’t search your topic verbatim — it writes its own focused queries (often several), runs them against all three indexes, and distills the findings into working notes.

The Drafter writes the article — and before stating any specific fact, it calls a verify_claim tool that re-searches the library for support (with that same 0.30 floor). Supported → written with a citation. Unsupported → omitted or generalized. Figures are referenced by ID, so image links can’t be invented.

A Critic reviews the draft against the sources and returns a structured accept-or-rewrite verdict. A rewrite carries the critique back to the Drafter — at most 3 times — then the best version ships. A per-run budget cap is checked before every model call.