Resources
Language and dialect coverage guide
Answers to the questions enterprises and model teams ask when evaluating multilingual annotation coverage -- what dialect depth means, how to verify it, and which languages require specialist annotators.
- Why does dialect coverage matter for AI training data?
- Language models trained on data from one dialect systematically underperform for speakers of other dialects. A voice assistant trained on Modern Standard Arabic annotation will mishandle Egyptian, Moroccan, and Gulf Arabic spoken queries. A Spanish NLP model calibrated on Castilian data will miss Mexican and Rioplatense idiom. Dialect specificity in training data directly determines model performance for the communities the model serves.
- What is the difference between a language and a dialect in annotation context?
- A language is a broad category (Arabic, Spanish, Chinese). A dialect is the specific variety spoken by a community in a particular region -- Gulf Arabic, Mexican Spanish, Cantonese. For annotation purposes, what matters is whether the annotators are native speakers of the specific dialect, because vocabulary, idiom, phonology, formality norms, and pragmatic conventions differ significantly across dialects of the same language.
- What is translation overlay annotation and why is it a problem?
- Translation overlay annotation uses non-native speakers who work in a language with machine translation assistance. The resulting labels reflect English linguistic structure and idiom imposed onto the target language's surface form. For NLP tasks -- intent classification, sentiment, preference evaluation -- translation overlay produces labels that may be grammatically acceptable but culturally and pragmatically wrong, creating systematic model bias toward English linguistic patterns.
- How do I verify that a vendor's annotators are genuine native speakers?
- Ask where annotators physically live -- not what language they speak. Native dialect coverage requires in-region, first-language speakers. Ask for inter-annotator agreement scores broken down by dialect community. Ask for a real annotation sample from the specific dialect and ask who annotated it and their language background. A vendor with genuine native coverage can answer all three quickly.
- Which Arabic dialects does Corpshore AI cover?
- Corpshore AI covers all six major Arabic dialect groups: Gulf (Saudi Arabia, UAE, Kuwait, Qatar, Bahrain, Oman), Egyptian (Egypt, Sudan), Levantine (Syria, Lebanon, Jordan, Palestine), Iraqi, North African/Maghrebi (Morocco, Algeria, Tunisia, Libya), and Modern Standard Arabic. Each dialect group is covered by annotators from the corresponding geographic communities.
- How many Spanish dialects does Corpshore AI cover?
- Corpshore AI covers Mexican Spanish, Rioplatense Spanish (Argentina, Uruguay), Caribbean Spanish (Dominican Republic, Puerto Rico, Cuba), Castilian Spanish, Andean Spanish, and Central American Spanish. Coverage is through in-region annotators from each community, not through Castilian-trained generalists.
- Does Corpshore AI cover Chinese languages including Cantonese?
- Yes. Corpshore AI covers Mandarin (Putonghua), Cantonese, Hokkien/Taiwanese, and Shanghainese. Mandarin and Cantonese coverage is through in-region native speaker annotators. For written Chinese, traditional (Taiwan/Hong Kong) and simplified (mainland China) variants are handled separately.
- What low-resource languages does Corpshore AI cover?
- Corpshore AI covers Uzbek (three dialects: Tashkent, Samarkand, Fergana Valley), Swahili (Coastal, Standard, Congo), Twi (Ghana), Luganda (Uganda), Amharic (Ethiopia), Hausa (Northern Nigeria, Niger), Yoruba (Southwestern Nigeria), Igbo (Southeastern Nigeria), and Afrikaans (South Africa). These are covered by in-region, first-language annotators from the specific geographic communities.
- What Southeast Asian languages does Corpshore AI cover?
- Coverage includes Filipino/Tagalog, Vietnamese (Northern, Southern, Central dialects), Thai, Indonesian (Bahasa Indonesia), Malay (Malaysian and Brunei variants), Burmese, and Khmer. Each language is covered by in-region native speakers.
- Does dialect coverage affect pricing?
- Yes. Rare language and dialect coverage typically commands a 50 to 150% premium over equivalent common-language rates, reflecting the limited supply of qualified native-dialect annotators and the additional calibration required. However, the cost of dialect-authentic data is typically lower than the cost of retraining a model that systematically underperforms for a specific dialect community.
- What is the right way to structure a multilingual annotation program?
- Structure annotation by dialect community, not by language. Use separate annotation tracks for each dialect, with calibration on dialect-specific gold sets and IAA measured within each dialect community. Aggregate data across dialects only after dialect-specific quality gates are passed. Project management should include a linguist who can interpret disagreements between dialect communities.
- How do tonal languages like Mandarin and Yoruba require different annotator qualifications?
- Tonal languages encode meaning through pitch contour as well as phoneme sequence. For audio annotation (ASR, TTS, phoneme labeling), annotators must be native speakers because pitch contour is cognitively inaccessible to non-native listeners regardless of second-language proficiency. Written annotation of tonal language text requires native speakers because tonal ambiguity in unvocalized text is resolved by native pragmatic knowledge.
- Does Corpshore AI cover speech and audio for non-English languages?
- Yes. Corpshore AI collects spontaneous and scripted speech for non-English languages including Arabic, Uzbek, Swahili, Filipino, Vietnamese, Turkish, Spanish, Portuguese, German, French, Dutch, Polish, Afrikaans, Zulu, and others. Speech data collection includes structured capture logs, speaker demographic metadata, and quality review for acoustic conditions.
- What is the difference between ASR training data and TTS training data?
- ASR (automatic speech recognition) training data is annotated recordings of real speech, used to train models that transcribe audio to text. TTS (text-to-speech) training data is scripted, studio-quality recordings of a small number of speakers, used to train models that generate synthetic speech. ASR data needs diversity (many speakers, many conditions); TTS data needs consistency and acoustic quality (few speakers, controlled conditions).
- How many languages does Corpshore AI support in total?
- Corpshore AI supports annotation in 35+ languages with full dialect-level coverage for major languages. Full language documentation is at corpshore.ai/languages. For rare language or dialect requirements not listed, contact the team -- specialized coverage can often be scoped through Corpshore's network of regional annotation hubs.
Explore languages
Browse Corpshore AI's full language portfolio, including annotator hub locations, dialect breakdown, and example use cases.
View all languagesReady to scope a pilot?
Tell us your modality, volume, and languages. We'll return an indicative scope, timeline, and cost band.