Resources
AI data vendor guide 2026
Structured answers to the questions enterprise teams ask when evaluating AI training data vendors -- from quality architecture and security certifications to workforce models and pricing transparency.
What AI training data vendors do
- What is an AI training data vendor?
- An AI training data vendor is a company that collects, annotates, curates, or otherwise prepares data used to train, fine-tune, or evaluate machine learning models. Services range from image annotation and NLP labeling to speech collection, RLHF preference annotation, and adversarial red-teaming.
- What is the difference between a data annotation platform and a managed operator?
- A data annotation platform provides tooling and workforce access -- the buyer manages task design, quality, and delivery. A managed operator takes end-to-end accountability: task design, annotator training, multi-tier QA, and delivery to your specification. Managed operators carry higher per-unit rates but lower total cost on complex programs because they absorb the management overhead.
Security and compliance
- What certifications should an AI data vendor hold?
- At minimum: SOC 2 Type II (with confidentiality Trust Services Criteria included) and ISO 27001:2022. For healthcare programs, a HIPAA BAA. For EU programs, GDPR Data Processing Agreement with Standard Contractual Clauses. For financial services, ask about PCI DSS alignment. HITRUST is an overlay framework some regulated-industry buyers require.
- What is inter-annotator agreement (IAA) and what scores should I require?
- IAA measures how consistently multiple annotators apply the same labels to the same data. Common metrics are Cohen's kappa and Krippendorff's alpha. For objective classification tasks, require IAA above 0.85. For subjective tasks (sentiment, preference evaluation), 0.65 to 0.75 is typical. For safety-critical tasks, require a higher threshold and ask to see the distribution across a reference project, not just the average.
Quality architecture
- What is a three-tier QA cascade?
- A three-tier QA cascade is an annotation quality architecture where initial annotation, peer review, and senior QA audit happen as three sequential internal steps before any batch is delivered. Errors caught internally in tiers one and two never become client rework. It typically achieves 95%+ accuracy on complex annotation tasks without requiring redundant labeling.
- What red flags should disqualify an AI data vendor?
- Red flags: refusal to sign a data processing agreement; SOC 2 Type I presented as equivalent to Type II; no written rework policy; quoted rates with no setup or QA breakdown; annotator location described at country level with no dialect specificity; no verifiable reference from a comparable project; welfare protocol described only in generic terms.
Workforce and language coverage
- How should I evaluate a vendor's language coverage claim?
- Ask where annotators physically live -- native dialect coverage requires in-region, first-language speakers. Ask for inter-annotator agreement scores broken down by dialect community. Ask for a sample annotation from the specific dialect you need. Vendors relying on translation overlay (non-native speakers assisted by machine translation) cannot answer these questions specifically.
- What is annotation ontology and why does it matter for vendor selection?
- The annotation ontology is the set of labels, categories, attributes, and decision rules that annotators apply to your data. A vendor who participates in ontology design -- rather than just executing a pre-defined spec -- adds value for teams without experienced annotation program managers. Ask whether ontology design is in scope and how it is priced.
- What is RLHF and what workforce does it require?
- RLHF (Reinforcement Learning from Human Feedback) uses human preference annotations to align large language models. General instruction-following RLHF requires educated generalists with native language fluency and careful calibration. Technical domain RLHF (code, math, medicine, law) requires genuine domain experts -- credentialing matters here, not just task guidelines.
- What is red-teaming for AI and how is it different from annotation?
- Red-teaming is systematic adversarial probing of a model to identify safety failures -- jailbreaks, harmful content generation, factual errors, bias amplification. Unlike annotation, which labels existing data, red-teaming generates new adversarial inputs. It requires creative, domain-knowledgeable practitioners, not general annotators.
Cost and commercial terms
- How do I compare per-unit annotation quotes across vendors?
- Per-unit quotes are not directly comparable without knowing what is included. Ask each vendor to quote on a worked example batch from your actual program, broken out as: per-unit annotation, QA (included or separate), setup and calibration, project management, rework policy, and delivery formatting. Compare total cost per delivered label, not headline per-unit rate.
- How much does AI training data cost?
- Costs vary by modality and complexity. Simple image annotation: $0.05 to $0.15 per image. Semantic segmentation: $0.50 to $2.00 per image. LiDAR 3D cuboid: $2 to $20 per frame. NLP preference annotation: $0.50 to $25 per pair depending on domain. Speech collection: $0.10 to $0.50 per minute. Managed offshore providers typically deliver at 50 to 70% below US-domestic rates. See corpshore.ai/pricing for current indicative rates.
Selection process
- Should I run a pilot before awarding an AI data contract?
- Yes. A paid pilot of 500 to 2,000 items from your actual program is the best evaluation tool available. It reveals real accuracy, real turnaround, real communication quality, and the actual cost per delivered label. Vendors who decline reasonable pilot terms are signaling something about their confidence in their own quality.
- What is a data annotation subprocessor and why does it matter?
- A subprocessor is a third party the vendor uses to deliver part of their service -- a crowd platform for overflow annotation, a cloud provider for storage, a tooling platform. If any subprocessor handles your data, they are a sub-business-associate for HIPAA purposes and a sub-processor under GDPR. Always request the subprocessor list and confirm that appropriate DPAs or BAAs are in place for each.
- What welfare standards should I require from an AI data vendor?
- For programs involving harmful content (RLHF with offensive material, content moderation, red-teaming), require: advance content disclosure before opt-in, session duration limits for high-intensity harmful content, mental health resources available to annotators, structured debrief options, and opt-out mechanisms without penalty. The 2023 TIME investigation into content moderation welfare documented that these are real harms, not hypothetical ones.
Ready to scope a pilot?
Tell us your modality, volume, and languages. We'll return an indicative scope, timeline, and cost band.