Zovda
Every long-document pipeline starts by cutting. The cut is where the document stops being a document and becomes a set of samples that happen to share a file name.
Models are then evaluated on questions that span those cuts, which is why long-context quality often looks worse than the training data suggested.
Chunking deletes the difficulty
A window of a few hundred tokens contains local coherence almost by construction. What it does not contain is the pronoun whose antecedent sits three sections back, the table introduced ten pages earlier, or the exception stated in a footnote.
Those are the cases that separate a model that reads from a model that predicts the next plausible sentence.
What a long record has to keep
- The document id, stable across versions
- The section path, so a passage knows where it sits
- The position, in the original, not in the window
- The references inside the passage, resolved to the section they point at
- The question, written against the whole document
The fourth field is the one that turns a pile of windows into something a model can learn structure from.
Where models actually fail
Three failures repeat. Answering from the section that mentions the topic rather than the one that decides it. Missing a negation that applies to a whole page. Treating a summary table as the source of a number that the prose contradicts.
All three require the model to hold two distant places at once, which no window teaches.
Build the question from the document
Write tasks by reading the whole thing, then noting what a careful reader had to connect. If the connection needs more than one section, that is the task.
Zovda builds these collections through Datuva, where each passage keeps its document id and section path, so a question can be traced to the pages it depends on. The question bank collection shows how these rows are structured.
Tell us the documents your model has to read. We reply within one business day.