Substitution detection
Records where the asset no longer matches the product it claims to be — the failure that looks most like correct data and is not.
AI Catalog Data Quality Auditing
Find the bad data before the model learns it
What this covers
Training data degrades in ways that are invisible in aggregate statistics. A supplier substitutes a similar product and the image no longer matches the record. A label class gets interpreted differently by a second cohort of labellers. A scraping job quietly starts returning the wrong page.
None of this shows up as missing data. It shows up as a model that underperforms for reasons nobody can locate.
StaffCloud audits datasets against their own guideline and against your source of truth: sampled label review, substitution detection, drift measurement across batches and cohorts, and a defect list by type rather than a quality score.
What the team does
Label accuracy review
Sampled re-review against the guideline by a reviewer who did not produce the original label, with disagreements adjudicated rather than averaged.
Drift measurement
Consistency compared across batches, time periods and labeller cohorts, which is where mid-project interpretation shifts become visible.
Class balance and coverage
Distribution across classes and edge conditions reported, since a dataset can be entirely accurate and still teach the wrong priors.
Guideline conformance
Whether the data actually follows the guideline it was produced under — frequently the most uncomfortable finding.
Defect list by type
Findings as an actionable list by defect type, not a score. A score tells you there is a problem; a list tells you what to fix.
How the engagement runs
-
01
Scope and sampling plan
What is being audited, at what sample rate and against which reference — the guideline, the source catalogue, or both.
-
02
Blind review
Sampled records re-reviewed without sight of the original label, so agreement means something.
-
03
Analysis
Defects categorised, drift compared across cohorts and batches, and the causes identified where they can be.
-
04
Report and remediate
A defect list with causes, and re-labelling of the affected slices where you want it.
Why audit before training, not after
A model trained on quietly wrong data does not fail loudly. It just underperforms, and the investigation costs more than the audit would have.
- Substitutions found before they become learned associations
- Drift measured across cohorts rather than suspected
- A defect list you can act on instead of a quality percentage
Platforms and tools we work in
The pod works inside your stack. These are the systems this team uses day to day — if yours is not listed, we will learn it.
- PythonSampling, comparison and drift analysis
- Label StudioBlind re-review workflows
- CVATVisual re-review
- SQLRecord matching against source catalogues
- Excel / Google SheetsDefect logs and reporting
Questions buyers ask
The things worth knowing before the first call.
Not sure this is the right scope?
Describe the workload in a sentence and we will point you at the right team — or tell you honestly if it is not something we should take on.
Ask usIt depends on the size of your dataset and how much confidence you need in the result. StaffCloud proposes a rate with the statistical reasoning behind it.
That matters because a round number picked for convenience can leave you with an estimate too loose to act on. Seeing the reasoning lets you trade cost against confidence deliberately rather than discovering the limitation afterwards.
Yes, and it is one of the most common reasons clients get in touch.
The review is carried out against the guideline that data was produced under. Where no written guideline exists, that is itself the headline finding — without one there is no agreed standard the work can be measured against, and consistency was always going to be accidental.
By matching each asset against the source record it claims to represent, which is the same discipline StaffCloud applies to catalogue product matching.
Where your own source of truth is unreliable, that has to be established first, because an audit against a flawed reference produces findings you cannot act on. We will tell you if that is what we are seeing rather than reporting mismatches as errors.
If you want us to. The audit and the remediation are deliberately kept as separate decisions.
Sometimes the right answer is to discard an affected slice rather than repair it, and that is a cost comparison worth making with the audit findings in front of you. Bundling the two commits you to repair before anyone knows whether repair is the cheaper path.
The same discipline covers ecommerce catalogue auditing, and that has its own page under Product Data.
This page is aimed at training sets specifically, because the consequence of a bad record is different. A wrong product attribute costs you a sale; a wrong label teaches a model something incorrect that then shows up everywhere the model is used.
An error is a wrong label. Drift is a label that is correct under a different reading of the guideline — which is the more damaging of the two.
An error is scattered and detectable. Drift is consistent: every record in the affected cohort is wrong in the same direction, so it passes a spot check and teaches the model the drifted interpretation. StaffCloud looks for it specifically.