Enterprise AI Pilots: What to Measure Before Scaling

A UAE facilities management company tests an AI assistant that drafts work orders from incoming requests. The pilot team sees faster drafts and asks to roll the tool out to every site. The operations director wants to know whether the drafts are correct after review, whether the dispatch team has more work and how the service behaves when it reaches sites with different equipment and approval rules. Those questions can lead to a different decision from the one the demonstration suggests.

An enterprise pilot should give the person approving expansion a basis for choosing the next group, the operating budget and the conditions for stopping. Before the trial begins, define the business outcome and the limits of the authorization. During the pilot, track completed work and failures through to the point where someone can act. Before scale, show that the next team can reproduce the result with the people and support it will actually have.

Write the expansion decision before the test

Specify what would be added if the pilot succeeds. Expansion could mean moving from one site to four, extending to a new language or shifting from staff-reviewed drafts to automated dispatch. Each change exposes a different assumption. A pilot designed to test reviewed drafts at an office site provides a narrow basis for opening another comparable site. Automated dispatch or a specialist site with a different risk profile needs its own evidence.

Document the work covered, the people allowed to use the system and the actions that still need approval. Set the review date, name the person with authority to pause the pilot and decide who signs off on expansion. Ask what would cause each decision maker to withhold approval. An operations lead may care most about missed urgent work. Finance may need to see a capacity constraint ease. Security may require the team to demonstrate that one site cannot see another site's records. Include those questions in the same decision record.

The UAE Artificial Intelligence Office provides a self-assessment for developers and operators based on its AI ethics principles and guidelines. Use it to prompt questions about responsibility and oversight while the trial is still being designed. Attach the evidence your team uses to answer those questions. An assessment response can identify who is supposed to intervene. A pilot exercise shows whether that person can intervene during a real shift. UAE AI Ethics Self Assessment

Build a baseline you can compare with the pilot

Start with all eligible requests, including the ones that end up being transferred, reopened or handled manually. Record case type, shift, site and the reason anything is excluded. Keep eligible work that the tool declined visible in the assisted cohort. Removing those cases makes the tool look faster while the business still has to process them.

Choose a period that captures the work you expect the next sites to receive. Record disruptions such as a planned maintenance shutdown, a backlog clearance or a change in staffing. Compare like cases across the pilot and the existing process, ideally at the same time where practical. One option is to assign eligible cases by a predetermined rule while preserving urgent routing. Another is to stagger activation across comparable sites and keep the same measurements in both groups. If assignment follows supervisor discretion, document how the supervisor chose cases. A team that sends easier work through the pilot will otherwise make the assisted process look artificially strong.

Collect the baseline with the same definitions the pilot will use. Agree whether a case is complete when a draft is accepted, a work order is approved or the job reaches the right technician. For this example, measure the approved work order reaching dispatch with the correct site, equipment, priority and instructions. Use a fixed follow-up window to capture corrections after dispatch. This avoids counting an early draft as finished work or dropping later rework from the comparison.

Digital Dubai's survey participation carousel shows one step in collecting evidence. For an enterprise pilot, the equivalent discipline is to define who is represented in the records and which requests were missed before interpreting the results.

Measure the workflow and the operating controls

Track effort, quality, reliability and the guardrails for the approved use. Put a denominator, owner and source next to each measure. A queue system can report how many orders reached dispatch. A reviewer may need to check whether those orders were correct. If reviewers inspect a sample, show how it was selected and label the result as a sample.

Decide where to count time when work moves between teams. If dispatch approves an order faster but the technician has to call the office to confirm the location, log the call against the same case. Compare elapsed time as well as labour. A queue can get longer even when each person spends less time handling a request, especially if approvals are restricted to a few supervisors.

Include an independent review of some accepted outputs. A case that nobody corrected could have been right, or the reviewer might have missed a fault. Give the auditor access to the original request and the approved operating procedure. Log each severe event separately with its consequence, who was notified and what the team did. A pilot can run without a severe event and still have too little exposure to assess rare failures. Test those failures deliberately in a safe environment and record how the system and staff respond.

NIST's AI Risk Management Framework Playbook discusses comparing production metrics with pre-deployment measurements and monitoring performance as the environment changes. Keep the pilot's definitions and configuration record so a future site can compare its first month against the same evidence. Record model and application changes with dates, and label results from different configurations separately if they could change the outcome. NIST AI RMF Playbook, Measure

Price the result after review and rework

Consider a hypothetical trial with 600 comparable requests under each process. The manual cohort takes 100 employee hours to handle and produces 570 correct work orders. The assisted cohort takes 82 hours, including review and rework, and produces 582 correct orders. Correct completion rises from 95% to 97%. The team records 18 fewer hours of work on the assisted cohort. Both results depend on comparable case mix and consistent scoring.

At an illustrative loaded rate of AED 120 per hour, the time difference is worth AED 2,160 in capacity. If incremental tooling and operating costs for those 600 cases are AED 1,800, the implied capacity value exceeds those costs by AED 360. That calculation excludes setup investment and assumes the 18 hours can be used productively. Finance should separate this value from a reduction in spending. The business may still pay the same wages and gain faster turnaround or room to take on more jobs. Document where those hours would be used and whether the next sites have enough similar work to use them.

Inspect the economics by site and shift before applying the blended result to a larger group. Suppose manual review at a new site adds two minutes to each of 600 orders. That is 20 extra employee hours, enough to erase the 18-hour gain if all other assumptions hold. Ask what creates the added review, such as a different asset catalogue or a separate safety approval. Test that part at the next site before using the pilot to justify a wider purchase.

Prove the team can operate it without the pilot specialists

List what the pilot team did behind the scenes. An engineer may have manually retried failed jobs. A supervisor may have answered questions about ambiguous requests during each shift. A vendor may have corrected mappings the same day. Turn those interventions into named work, estimate their frequency and price the time needed to do them at scale. A support arrangement that worked with one engineer in the room may need a funded rota and an escalation route across multiple sites.

Rehearse an outage with the next group. Have them move new requests to the manual path, reconcile cases already in flight and verify that nothing is dispatched twice. Confirm who makes the switch and how staff know which queue to use. An application that can be disabled in a settings screen can still leave partially processed orders behind. Give the receiving supervisor a way to check which orders made it downstream and which need manual attention.

Check that the next site can supply the same data, enforce the same permissions and support the same quality review. Test role boundaries with people who have ordinary permissions, including staff with access to only one site. Set a separate test for changes to data or operating scope. The UAE Charter for the Development and Use of Artificial Intelligence describes principles for AI development and use. For a deployment decision, record the local controls and evidence that the responsible teams will use to oversee this particular workflow. UAE AI Charter

Approve the next cohort and set a review date

Put the outcome on a page with a link to the evidence. Show the cohort, dates, exclusions, baseline and configuration. Present quality, employee effort, total operating costs and material failures alongside the recommendation. Identify the biggest uncertainty that would change the decision. For the facilities team, it could be review time at the second site or whether the dispatch integration can recover without an engineer present.

Approve the next cohort if it resembles the tested work, the safeguards passed and the support team can absorb it. Extend the trial if the likely benefit depends on an unresolved difference between sites. Pause the affected workflow when a critical safeguard fails and determine how staff will finish outstanding cases. Assign one owner to each condition and specify how the team will recognize the trigger in its own records.

Choose a date to review the next cohort and define the evidence required to proceed again. Use the original measures and add a breakdown where the new group differs. A decision to scale then rests on correct work reaching its destination, an operating cost the company can fund and controls the next team can exercise in practice.