Inside Ramsa’s Research on Emirati Arabic Speech

In Rania Al-Sabbagh’s 2026 Ramsa paper, an automated transcription system returned a word error rate of 13.8% on one Emirati television source and 72.7% on another. Both were part of the same research evaluation. The harder source, a cooking programme, featured fast exchanges and people talking over each other. That contrast is a starting point for anyone building voice software for the UAE. The language label can stay the same while the conditions under which a system has to understand it change completely. Ramsa paper, Table 6

Ramsa Lab describes a broader effort to document spoken Emirati dialects, build a dictionary and develop speech and language technology. Mohamed bin Zayed University of Artificial Intelligence lists natural language processing researcher Hanan Aldarmaki among the people associated with that initiative. Al-Sabbagh is the author of the published corpus paper. The lab’s public description and the paper describe different levels of the work. One sets out the wider research goals. The other tells readers which recordings went into a particular corpus, how it was tested and who can access its underlying material. Ramsa Lab · MBZUAI project record · LREC proceedings

The work begins with choosing what to record

The paper describes a developing corpus of about 41 hours, drawn from structured interviews and Emirati television. It reports 157 speakers and includes both people speaking alone and conversations. Topics range from heritage to daily routines and work. The university’s wider project asks members of the public to contribute recordings. That invitation and the paper’s corpus description should each be read on their own terms. The paper identifies the sources behind its benchmark. A reader cannot assume the benchmark includes everything the ongoing national project collects. Ramsa methods and corpus description · Ramsa Lab participation page

The two kinds of recordings let the researcher observe different challenges. The structured interview recordings were made in a quiet setting and captured the participant’s voice. Television supplied other formats, including conversations. That made it possible to see how recognition changed across settings. It also made it harder to attribute the gap to any one factor. Recording quality, turn-taking, topic and the speakers themselves can differ between two shows. The result identifies where systems struggled in this evaluation. A controlled experiment would be needed to isolate the effect of one variable. Ramsa paper, sources and evaluation

A collection can include several ways of speaking and still have gaps. Al-Sabbagh describes a dataset weighted towards Urban Emirati speech, with fewer Bedouin examples and still fewer Mountain or Shihhi speakers. The paper also notes uneven representation by gender. Roughly 10% of the audio had been manually transcribed at the time of the reported evaluation. The figures are a snapshot of a developing dataset. For a product team reading the paper, the relevant question is which users and environments are represented in the test portion, where the model scores come from. Ramsa paper, limitations

What the score counts

Word error rate compares an automated transcript with a reference. It counts words that were inserted, omitted or substituted and divides that total by the number of reference words. If a fictional transcript of 100 reference words has five substitutions, three omissions and two insertions, the error rate is 10%. The errors may cluster in one part of the recording or appear across the whole transcript. A wrong booking reference and a missed hesitation can each count as one edit even though the first matters more to the person using the result.

In the Ramsa paper’s recognition table, the best average reported word error rate among the systems tested was 26.8%. On the cooking programme, the systems returned rates of roughly 73% to 79%. The benchmarks were run without fine-tuning for the dataset. These are baseline scores from the transcribed subset, with the paper’s normalisation rules, rather than measured error rates for Emirati speech generally. The evaluation also reported character error rates, which give a second view when word boundaries or spelling choices affect the comparison. Ramsa paper, section 6.1 and Table 6

For a buyer comparing voice products, ask how the provider wrote the reference transcript. Two reviewers may agree on what a speaker said and write a dialectal expression differently. A team can document accepted spelling variants and how it handles fillers, interruptions and words spoken in English. Keep a second score for information the application must get right, such as a branch name, date or order reference. A model with a lower general error rate can still make more of the errors that matter to a particular service.

Recognition and synthesis are different jobs

The paper also examined text-to-speech, which starts with text and produces audio. The researcher gave the systems reference text, generated audio and used a recogniser to turn the result back into text. Comparing that output with the reference supplies an indirect measure of how much of the wording could be recovered. It can also reflect weaknesses in the recogniser used for the back-transcription. A team selecting a voice for a UAE product would still need listeners to assess whether the speech is natural, easy to follow and appropriate for the service. Ramsa paper, section 6.2

The research ecosystem includes model development as well as corpus work. In October 2026, MBZUAI and Inception42 announced the Jais 3 family and its Jais Speech model. The university describes the family as built on K2 Horizon and discusses speech that switches between Arabic and other languages. That announcement and the Ramsa paper establish separate pieces of work. The model release can tell a team what the developers designed for. A benchmark can tell it how specific systems handled specific audio under a stated protocol. A buyer still has to check the exact model and version it expects to use. MBZUAI’s Jais 3 announcement

MBZUAI’s September 2026 Instagram reel introduces the separate K2 Horizon model family. It provides context for the foundation model release cited by the Jais 3 announcement.

Check what the research lets you reuse

A published paper can make a method visible even when the original recordings have restricted access. Al-Sabbagh states that the interview audio and transcripts are available by request to qualified researchers for approved non-commercial scholarly use under an institutional data use agreement. The paper also says the broadcast audio, video and full transcripts are withheld from distribution because of copyright and licensing constraints. The author describes which supporting material may be requested, including public source links and methodological documentation. Ramsa paper, ethical statement and availability

For a UAE product team, make access part of the evaluation plan. Record the purpose for which each test sample was gathered, which people can listen to it and which service is allowed to process it. Keep a clear distinction between evaluating a model and allowing a vendor to retain or use the audio for further training. Where the team has the appropriate permissions, it can build a small evaluation set from the actual setting it needs to serve. Use fictional names and reference numbers in test scenarios when real identifiers add no value. That lets reviewers examine errors without turning the test into a collection of unnecessary customer information.

Decide how to write a fair reference

For a team making its own test, the reference transcript deserves the same attention as the model output. Ask a reviewer who understands the speech to mark places where a word is unclear or more than one spelling is defensible. Decide how to treat English terms inside Arabic sentences and when a number written as digits should match the same number written in words. A transcript can disagree with the reference at the character level and still preserve an instruction. Conversely, a small spelling change can turn one proper name into another. Keep examples of both in the review guide.

Assign a second reviewer to a portion of the recordings. If the reviewers repeatedly disagree about what was said, examine whether the reference instructions need improvement or whether the audio is genuinely ambiguous. Mark ambiguous cases so an apparent model error can be interpreted in context. For a trial involving several products, use the same scoring rules and the same set of recordings for each. When you revise a rule, apply it to earlier results before comparing the scores.

Hold back examples from speakers the team did not use to tune the system. That gives the evaluation a chance to reveal mistakes on new voices. Record the conditions of each sample alongside the result, including whether the speech was spontaneous, whether the speakers overlapped and what kind of device carried the conversation. The next review can then show whether a change helped the intended workflow or only the easiest recordings.

Build a trial around the conversation the product will hear

Imagine a UAE business deciding whether to use Emirati Arabic transcription to help agents write call notes. Start with a representative set of tasks. One might be a clear request to move an appointment. Another might contain an interruption, an English product name and a new time agreed near the end. Have reviewers establish what the note must preserve before they see the model output. They can then count material errors and record how long it takes to repair the note. Compare that with the time and errors in the existing workflow.

The first trial can keep the transcript as a draft and let agents approve it before it becomes part of the customer record. Give reviewers a way to distinguish ordinary wording changes from changes to what the customer wanted. If the system often loses an instruction when people overlap, the team can test narrower conditions or route those notes for more careful review. If it handles clear calls well but cannot preserve mixed-language product names, collect more examples of that pattern and check whether a configuration change helps. Keep the original comparison set so the next test can show whether the fix held.

Ramsa’s most useful result for that decision is the spread across the recording sources. It shows why asking for a single Emirati Arabic accuracy score leaves a team with too little information. Ask instead where the model works, which speakers and recording conditions were evaluated, and which errors would change a customer’s outcome. Use the answers to define the first workflow and the conditions under which a person should take over.