Arabic AI Performance: How to Test Before Buying

A UAE property maintenance team wants software that can turn resident messages into work orders. A customer writes in Arabic, includes an English equipment name and corrects the apartment number in a follow-up. The AI replies in fluent Arabic and opens a ticket. It uses the first apartment number. A reviewer who only checks the reply’s grammar might give the tool full marks. A technician sent to the wrong door would have a different verdict.

For a UAE buyer, Arabic support is a starting point for a trial. The purchase decision depends on whether the proposed service preserves meaning, handles the channel and completes the task safely. Compare the exact deployment you would buy against the way your team works today. Then specify where the service can run, where a person has to review it and what would cause you to stop using it.

Use research to choose what to test

In October 2026, the Institute of Foundation Models at Mohamed bin Zayed University of Artificial Intelligence and Inception42 announced Jais 3, a family of Arabic-focused models that includes a speech recognition component. Their announcement distinguishes speech and text and discusses conversations that switch between languages. A supplier should likewise identify which capabilities are included in the edition and configuration it is quoting. A successful text chat gives the buyer little evidence about the same workflow over a phone line. MBZUAI’s Jais 3 announcement

ArabicMMLU uses school-exam questions to assess language understanding. MBZUAI’s EDRAC dataset instead includes free-form questions about transcripts of naturally occurring speech, including Emirati Arabic. The first can inform a shortlist and the second can inspire tests of dialect comprehension. Your company’s handoff, interface, policies and error costs still need their own evidence. A model that understands a transcript might have received an inaccurate transcript from the product’s speech recognizer. ArabicMMLU paper and MBZUAI EDRAC dataset

Define success at the point of handoff

Write down what the tool receives, what it may change and what the next person should see. The maintenance team might require a draft ticket with the building, apartment, problem and urgency. If the unit is ambiguous, the system should ask for confirmation. If a message raises a safety concern, the workflow should follow the company’s escalation procedure. Set out whether the AI can send replies or change records. One vendor may quote a draft assistant and another an automated agent. Put both through the same review and approval boundary before comparing them.

Use the channel customers actually use. If requests arrive over voice, run the audio through the proposed production route and compare the original recording, transcript and ticket. If requests arrive through a mobile chat, test short messages sent in bursts, corrections and screenshots. If the job is document intake, inspect the extracted fields and the final screen. A value may appear correctly in a data export but be difficult for a reviewer to read in a mixed right-to-left layout.

Decide which kinds of mistake invalidate a case. An awkward greeting might warrant an edit. A change from AED 1,500 to AED 15,000, an invented promise about service timing or a disclosure to the wrong recipient should carry a different consequence. Agree severity before running the trial so vendors and internal reviewers face the same standard.

Assemble a set that includes routine work and hard cases

Start with approved examples and map them to the language, channel and task your staff actually see. Include formal written Arabic where it belongs and everyday speech or chat where it belongs. Avoid using a translator to create every example from an English script. Have reviewers familiar with the intended UAE audience identify plausible phrasing and where a reasonable response would ask for clarification. Where the team uses synthetic examples, label them and check that they resemble the structure and ambiguity of the real work.

Keep one portion representative enough to compare with ordinary operations. Use a separate stress set for rare cases with expensive consequences. Mixing the two hides whether an error rate describes daily work or an intentionally difficult sample. Check that each critical scenario appears often enough to tell you something useful. If you only test one noisy call, report the result as one call. Expand the set before drawing a conclusion about speech.

Keep the scoring set separate from the examples used to configure the tools. Group messages from the same conversation into a single case so a correction is tested with its original context. Use approved data and the permitted environment. A synthetic case can test whether the tool handles a confusing building name without sending a resident’s address and phone number to a supplier.

Grade meaning, accuracy and reviewer effort separately

Give reviewers the original input, approved policy and expected disposition. For a ticket, the expected disposition could be draft, clarify or escalate. Identify fields that must match exactly and where two answers could be equally acceptable. A polite reply can use different wording and still be suitable. An account number must survive intact. If a request genuinely has two plausible interpretations, put the clarification in the expected answer rather than forcing reviewers to guess the customer’s intent.

Review fluency and tone after checking the case outcome. Capture whether the response uses a register appropriate for the audience and whether the customer can understand the next step. Have bilingual reviewers score a common batch independently and resolve disagreements against the rubric. Where they differ over local wording, write down what made one interpretation reasonable. This makes the next batch easier to judge and gives a future reviewer an example to work from.

The UAE Government’s bilingual UAE Skills launch post gives local training context for the people who will oversee the workflow. For the trial, give those reviewers a short practice set with worked examples of the differences between harmless wording, factual corrections and critical errors. Time the task only after they understand the scoring rules.

Consider an illustrative test with 100 cases for each candidate, including 20 mixed-language cases. Tool A completes 94 overall and 16 of the mixed-language cases. Tool B completes 90 overall and 19 mixed-language cases. The headline favours A by four percentage points. The mixed-language slice favours B by three cases out of 20. Review the failures individually. If A lost an equipment ID while B’s missed cases were mainly routine routing problems, the buyer has a reason to collect more evidence on mixed-language work. The 20-case slice still leaves uncertainty about what will happen in production.

Time the work through to an approved outcome, including rewrites, corrections and escalation. Suppose the current process takes six minutes per case and the assisted process takes four and a half, with review included. At 400 comparable cases that is ten hours of monthly capacity under the test assumptions. Check that assisted work still meets the same standard. Record the time spent on training and setup separately. A stalled case that a supervisor eventually closes belongs in the total, even if the tool’s interface reports a response in seconds. Track how often staff have to open the original message to correct an otherwise plausible draft.

Make the result traceable to the configuration

Ask the vendor to record the model, speech engine if applicable, application version, policy settings, prompts and reference material. Test the proposed version through the actual interface with the integration and permissions you would use. Keep a case log with the input or approved reference, observed output, expected outcome, elapsed review time and reason for failure. Use matching input and output rules for competing tools. If one has access to your catalogue and the other does not, describe the difference and determine whether both were tested as they would be deployed.

For voice, distinguish what the system heard from what it decided. A transcript with an imperfect rendering of a greeting could still yield a correct ticket. A transcript that reverses the unit number could produce an otherwise sensible ticket that fails the job. For document intake, inspect number formats and the order of values in mixed-script output before moving to a live pilot. For chat, test whether the tool carries later corrections through to the downstream record. The cheapest fix may be at the transcription, extraction or interface layer.

Repeat a small set of consequential cases and inspect any changes in outcome. Record when the vendor alters the system to address a failure. Rerun the failed case and test similar unseen cases to find out whether the change generalizes. Protect the original results so the team can see what it bought and what still depends on a workaround.

Turn the test into an acceptance decision

Define the approved scope in terms a supervisor could enforce. For the maintenance example, an initial approval might cover draft tickets from written requests, with mandatory human confirmation of unit and urgency. Voice could stay in a staff queue while the team gathers a larger sample. Automatic customer messaging could require a separate trial. Tie each restriction to an observed limitation and decide what evidence would support lifting it.

Set a few conditions for acceptance. A buyer might require zero observed critical errors on a named stress set, a minimum correct completion rate on representative cases and review times that improve on the current workflow. Zero observed failures should trigger a review of what the team actually tested and which cases are absent. Put results beside sample sizes, confirm how supervisors will monitor real cases and agree what causes the team to fall back to the manual process.

Finally, ask when the service will change and whether the buyer can retest before that change affects production. Keep a small suite of Arabic cases for every material release, including any previous failures. Add cases when the company introduces a new channel, business line or group of customers. A purchase can then be justified with a specific claim about the work the company tested and the controls it verified.