Field notes on training data, evaluation, and social data — what we learn building datasets.
Models converge on the same architectures and the same tricks. What still differs is the data, and the discipline around it.
Data quality is argued with adjectives and bought with numbers. The only number that matters is the change it makes to a run you care about.
The first fifty rows teach you the schema, the guideline, and the failure mode. The next fifty thousand only confirm them.
Good data takes a guideline, a pilot, and one revision. A schedule that starts in launch week buys the first draft.
Buyers compare a price and assume a quality. Selling the same collection in tiers is honest only if the tiers differ in something checkable.
A data budget is usually a number of rows. The number that decides whether it works is hours of review per hundred rows.
If you cannot rebuild the exact dataset, you cannot explain the exact result. Most teams discover this during an audit.
Evaluation is usually run in the language the team writes in. The product is used in four others, and nobody scores them.
A safety set built from known failures tests memory. The rows you need are the ones nobody has thought of yet.
A model asked to play a user will ask what the model expects. The gaps you needed to find are exactly what it smooths over.
When the person who built a dataset leaves, the knowledge leaves too, unless it was written down next to the rows.
A cleaning script that runs on a sample takes a week on the full set. Throughput is a design decision made at the start.
Average performance is a promise about the middle. The complaints come from the tail, and the tail is not in the sample.
A smaller model trained on a larger one's outputs inherits its tone, its hedging, and its mistakes, at the same confidence level.
Every ticket is a real question, a real constraint, and a resolution someone verified. Most companies delete them quarterly.
A small set of near-identical examples teaches a tone and calls it a capability. The model gets more confident, not more able.
Evaluation built from the same clean pool as training measures the pool. The requests users actually send are elsewhere.
Every tool call can fail. A model trained only on successful sequences has never seen the most common situation in production.
When a grounded system is wrong, the passage was usually missing. Improving the writing prompt fixes almost none of it.
A dashboard is designed for a person who will look at it. An agent needs records, bounds, and a way to ask a follow-up question.
Attribution built only on clicks measures the minority who clicked and then did something. The rest is inference wearing a number.
Audience age and gender are usually modeled, not counted. The decimal places suggest a precision that the method does not have.
Early signal is real, and it is shaped by who was awake. Read the first day for direction, not for size.
Cross-posting inflates volume, splits engagement, and makes one campaign look like a movement.
A dashboard with nine flags in the filter usually has real volume in two of them. The rest is a trend line made of twelve posts.
A post is one person's claim. The replies are the audience's verdict, and they are usually the first thing a pipeline drops.
Short video metadata usually starts at the caption. Everything that decides retention happens before a word is read.
Listed, discounted, and actually paid are three different values. A price series that stores one of them answers a question nobody asked.
Stars hide when the reviews were written. A product that improved last spring still carries last year's complaints in one number.
Two sellers list the same item in two categories. Every comparison you build on top inherits the disagreement.
Refreshing weekly is a ritual, refreshing never is decay. The right cadence follows how fast the answer to your question changes.
The first afternoon decides whether a team trusts a dataset. Read twenty rows by hand before you write a loader.
Most data pilots test nothing: a small clean batch, a vague goal, and a decision that was made before it started.
The salary is the visible part. Guidelines, review, retraining and idle time are the part that decides the bill.
A demo shows clean data and a chart. These five questions decide whether it will survive your first month.
Every dataset has a slice the seller does not ship. Knowing which one tells you more than any sample they send.
A supplier paid per row will deliver rows. Whether those rows contain a hard example is a separate question with a separate answer.
Two quotes a thousand rows apart often buy the same amount of usable data. The difference is what happens after delivery.
An item edited, re-published, or migrated comes back with a new id. The history you were tracking becomes two strangers.
The vendor renamed a column, the loader filled it with nulls, and the model trained on a slightly different world.
A large instruction set often contains one task in many costumes. The model learns the format and none of the skill.
Throwing every page into an index gives a system that finds something for every query and the right thing for few.
Facts change weekly. Training runs quarterly. The gap is a data pipeline problem that no amount of prompting closes.
A character sheet describes a person. A conversation requires one. The gap shows up as soon as the user pushes back.
A conversation is not a list of turns. It has a goal, a state, and a cost for asking the same thing twice.
Agent training needs an environment that answers back the same way twice. Most sandboxes drift, and the errors hide in the reward.
The question, the query, and the number someone checked by hand already exist. Nobody stores them as a dataset.
Every code review is a judgment: this is wrong, this is fine, this should be different. Most teams throw the labels away.
A filled form is a sentence with the grammar removed. Key-value layout is the only thing that tells a model which number is which.
Chart QA data is usually an image and a sentence. Without the underlying table, the model learns to guess from pixels.
Bot filtering is a threshold, and the threshold decides which humans you delete. Show the cut, not only the filtered total.
Data pulled through a ranking returns what the ranking already likes. Every conclusion downstream inherits that filter.
History is the first thing buyers assume and the last thing vendors have. Ask what date the backfill really reaches.
Sorting creators by followers selects for accounts that grew in a different era, for a different audience, on a different platform.
A creator's price is a function of last month's performance. A card older than a quarter describes a different seller.
Trend detection is a lag problem, not a chart problem. The useful signal is the change in slope, not the height of the peak.
A brand is a set of spellings, abbreviations, misspellings and product names. Query one of them and you monitor a fraction.
Two dashboards report different shares for the same brand because they count different universes. The post count is fine; the base is not.
A sentiment tag from last year was applied by a model and a guideline that have both since moved. The post is unchanged; the label is not.
Joining two vendors looks like field mapping until a name means two things and a date means two time zones.
A benchmark built from clean examples measures the easy half of the job. The failures live in the half nobody collected.
A multilingual set is usually one large language and a set of small ones. The small ones set the floor for everyone outside the majority.
If longer answers win in your preference set, the reward model learns length before quality. The fix is in the data, not the loss.
Preference data collected from obvious wins teaches a model to prefer longer, more confident, more list-shaped output.
A rubric that requires discussion has already failed. The goal is a decision two people reach alone, from the same page.
Refusal rows are usually a list of banned topics. What a model needs is the near miss: the request that is almost fine.
A negative sampled at random teaches almost nothing. The useful one is the case a careful reader has to think about.
Entity lists are cheap to produce. The relations between them took judgment, and they are the first thing cut when time runs out.
Reading order, merged cells and captions carry the meaning in a document image. Flat text throws all three away.
Long-document data is often a pile of fixed-size windows. The references that make a document hard disappear exactly at the cut.
The new month arrives with a renamed field and a dropped category. Nothing crashes. The model just learns something else.
A data card should answer what a buyer asks after the demo: what is in here, what is missing, and how was it cut.
A gold set built last spring grades this year's data against last spring's judgment. Refresh the rows, not only the model.
If two annotators read the same rule and label differently, the rule is the bug, not the annotator.
A corpus with a tenth of near-duplicates is not slightly larger. It is a different dataset with a different weighting.
Generated rows inherit the bias of the seed set and the habits of the model that wrote them. Neither shows up in a training log.
Word error rate hides what a speech corpus is missing: overlap, distance, noise, and speakers the model never hears.
A click is data only when it is tied to the element it hit, the state before it, and whether the screen changed after it.
Models learn tool use from the calls they get wrong: a missing field, a wrong type, a call that should never have happened.
Two files that differ by an import line are the same training example. Most code datasets never run that check.
Chapters, examples, exercises and answers are four different row types. Flattened into paragraphs, a textbook loses the structure that made it teach.
Most math data failures are not hard problems. They are inconsistent symbols, missing units, and answers that cannot be parsed the same way twice.
Adding paragraphs to a solution teaches verbosity, not inference. What transfers is the chain of decisions, including the ones that were rejected.
A conversation flattened into one block loses who said what, when a topic changed, and which reply answered which question.
Chinese has no spaces, so every later step inherits the segmentation decision. Get boundaries wrong and deduplication, search and evaluation all drift.
Copying an English set into nine languages produces nine copies of one culture. Real multilingual data differs in idiom, length and what counts as polite.
Different length, different pacing, different labels. Treating short clips as a scaled-down long-video set is why the model misses the first second.
Text extraction keeps the paragraphs and drops the structure. Tables, headers and footnotes are where most document reasoning actually lives.
A caption that says what is visible teaches recognition. A pair that states the task teaches behaviour. Most corpora only contain the first kind.
Resolution is the least interesting number in an image corpus. Eight metadata fields decide whether a collection can be filtered, joined and reused.
A listening tool optimises for decisions inside the tool. A data interface gives you the records. Three questions decide which one you are actually buying.
Public conversation is edited and removed continuously. Coverage grids, stable ids and record versioning decide what a longitudinal study can support.
Contamination arrives as quoted paragraphs, solved examples and discussion posts. Per-item overlap checks and time splits are the only defence that holds.
Every ratio is a stack of claims about value per token. Deduplicate first, cap synthetic share, and tune against an eval instead of the training loss.
Ties and disagreements are usually deleted from preference data. They are the rows that teach a model when to hesitate, and how a rubric actually behaves.
Polished instruction sets often teach a voice rather than a task. Intent-level deduplication, length balance and held-out prompts from real usage fix it.
Channel layout, loudness, natural segment boundaries and word-level timestamps decide whether an audio corpus is usable or just a folder of recordings.
A commit is the answer, not the path. Agent training needs per-step observations, actions and results, plus the failed branches most teams throw away.
A question bank can score 92% on a held-out split and still fail on rephrased questions. Answer leakage hides in stems, options and solution text.
Video is priced by the hour and learned by the second. Annotation depth, time ranges and a filterable manifest decide whether footage is signal or storage.
How teams run social listening on one schema: volume, sentiment, alerts, and competitor share of voice. 6 platforms, 100+ unified fields.
How consumer electronics teams read a launch with social data: reception by market, share of voice, creator response. 6 platforms, 100+ fields.
Order, inventory, review and price signals sit apart. One unified layer normalizes them and adds sentiment and trend for pricing and selection decisions.
One data layer for social platforms: a single schema, 100+ unified fields, and 6 platforms covered. See how teams replace many integrations with one.
FMCG pricing has moved to the marketplace shelf. Minute-level price tracking and sentiment turn price moves into decisions. 6 platforms, 100+ fields.
Zovda