Enterprise Data Quality: The Work Before Analytics and AI

A Dubai distributor wants a dashboard showing how often orders ship in full. Sales records an order in cases, the warehouse scans individual units and finance books replacements under a different code. The dashboard can join the three systems and produce a percentage. If nobody agrees which quantity belongs in the calculation, the number may change when a spreadsheet is corrected rather than when the company ships more reliably. An AI assistant built on the same records could explain the number fluently without making it more trustworthy.

Start with the decision the business wants to make. Identify the fields that could change that decision, then agree what they mean, who owns them and what happens when they fail a check. A bounded project can make one useful data product reliable while revealing problems that also affect later analytics or AI work.

Specify what a record represents

Ask people who use the data to explain a real case. An order may contain several lines, each with different delivery dates or quantities. A partial shipment can produce two dispatch records. A replacement may have a new order reference. Before counting orders shipped in full, agree whether the measure applies to the order, line or shipment and which events can change its status. Document the handling of cancellations, substitutions and agreed changes.

Write the definition in terms the source owner and report reader can both use. For example, the distributor might count an eligible order line as fulfilled when the cumulative accepted dispatch quantity matches the agreed quantity in the promised window, after approved exclusions. Store the version and effective date of that rule. If the company changes how it handles partial shipments, decide whether to recalculate history or show a break in the series.

Assign an owner to the definition and another to each source where responsibilities differ. Sales may own the agreed quantity, warehouse operations may own dispatch scans and finance may own adjustments. The analyst can document how the fields are combined. When two records disagree, the analyst needs a defined route for resolving the conflict with the people responsible for the process.

Digital Dubai announced an updated Dubai Data Manual in July 2026 as a governance and management framework for Dubai government entities. Its institutional context helps explain why shared definitions and responsibilities deserve attention before data is used for new services. A private distributor should specify the controls that fit its own operations and applicable requirements. Digital Dubai’s updated Dubai Data Manual announcement

Trace the fields back to where they are created

Follow an order from the point of entry to the report. Record the order and line identifiers, item identifier, quantity, unit, promised date and actual dispatch. Note every transformation, including where one system changes a unit or combines multiple events. Check how delayed or corrected records get into the dataset. Ask what happens when someone changes a source record after the daily report has already been published.

Inspect identifiers before relying on a join. The same display name can refer to two products, and the same product can be known by different codes at different branches. Keep the raw values and approved mappings, with effective dates where the mapping can change. GS1's GTIN Management Standard illustrates why identifying a base item and identifying higher packaging levels are distinct tasks. The distributor should verify its own catalogue and conversion rules for the packages it sells and ships. GS1 GTIN Management Standard

The Abu Dhabi Department of Economic Development’s business reporting carousel provides local context for keeping company records organized. For analytics, follow the record further and show how the published metric was produced from the source fields. Keep enough information to reconstruct the transformation without exposing records to people who should not see them.

Test missing values as well as incorrect ones. A blank shipment date may mean the order is still open, the integration failed or the field does not apply to a cancelled order. Those states have different consequences. Give them explicit meanings in the source or transformation rather than filling every blank with a plausible date. Preserve the reason where the business needs to tell a known exception from an unknown value.

Measure the defects that affect the decision

Profile a period with ordinary work and predictable exceptions. Look for missing records, duplicate identifiers, inconsistent units and updates that arrive late. Inspect both records that failed a rule and a sample of records that passed. A format check can establish that a date looks valid. It cannot establish that the date reflects when the parcel left the warehouse.

Microsoft Purview documents examples of rules for uniqueness, blanks, freshness and lookup across tables. A tool can help run checks repeatedly, but the team still has to decide when they apply and what a failure means. For example, a uniqueness check on an order number is useful for an order table and unsuitable for a table with multiple dispatch events per order. Check the product's coverage and the behavior of nulls before treating its score as a business result. Microsoft Learn on data quality rules

Suppose the distributor has 10,000 eligible order lines and 400 are missing a verified unit conversion. Of the 9,600 lines it can evaluate, 8,832 shipped in full under its definition. That is 92% of evaluable lines, with 96% coverage. If every unresolved line eventually qualifies, the full result is 92.32%. If none qualifies, it is 88.32%. Those bounds show how much the unresolved group could move the result. Investigate whether the missing conversions cluster around a supplier, branch or product family before treating the evaluable result as representative.

Compare the same checks across refreshes. An integration can deliver a file on time that contains only half the expected branches. Check counts by source and compare them with the source system or the feed manifest. When a file arrives twice, preserve enough information about its batch and revision to distinguish a retry from a corrected submission.

Fix the source and keep the correction visible

Prioritize defects by how often they occur and what they do to the decision. One missing unit conversion for a high-volume product may matter more than hundreds of unused blanks. Trace each material issue to the step that produced it. If the problem starts with a new product created without a required unit, change that setup process. If a nightly feed duplicates rows after a retry, fix how the import identifies work it has already processed.

Make an explicit decision about historical records. A source fix can prevent new defects without changing the reports people have already seen. If the business backfills earlier records, record the affected period, which rules changed and who approved the correction. Publish the revised figure with enough context for someone to reconcile it with the earlier version. Retain the source and correction record under the applicable access and retention arrangements.

Put material failures in a queue with an owner and a time for review. A rejected record can disappear from the dashboard while still representing an order a customer is waiting for. Reconcile what entered the process, what passed and what was held. Give report users a visible coverage figure and a way to tell when data were last refreshed. If an issue is likely to change a decision, pause that use or narrow it to the part of the data the business can verify.

Check whether the fix changed downstream behavior. Adding a stricter field requirement may stop bad values at entry and also block urgent orders that staff used to process manually. Exercise a real exception and agree the approved path for handling it. Monitor the correction rate for the next few reporting cycles. A lower error count has limited value if people shift the same work into an untracked file.

Test the data for the specific AI use

An analytics measure can be useful at a level of aggregation that does not support an AI assistant answering questions about individual orders. If the assistant is expected to explain a delay, test whether it sees the latest status, the right order and the records the user is allowed to open. Require it to distinguish an unknown status from an order still awaiting dispatch. A table that works for monthly analysis may have been refreshed too early for a customer-facing response.

If the data will be used to train or evaluate a model, define the answer the business wants the model to learn. A closed support ticket may represent a resolved issue or an employee ending the case after the customer stopped replying. Check how historical labels were assigned and whether the definition changed. Keep events from the same order or customer case together when building development and evaluation sets, where separating them would let the model see near copies in both.

Test the periods and groups that matter to the business. New suppliers, a recently opened branch or orders entered with mixed Arabic and English descriptions may have different defect patterns. Review them separately where the sample supports it. If one group lacks reliable data, limit the AI workflow for that group until the necessary records or oversight are in place.

Release a usable dataset with an owner

For the distributor, the first useful release could be a verified fulfilment dataset for one branch and the products with approved unit conversions. Publish its definition, exclusions, coverage and refresh schedule alongside the measure. Name the people who can correct an item mapping or investigate a delayed dispatch event. Publish the last verified source update with the reporting date. Set thresholds that trigger review and document whether the report should be flagged or paused when those thresholds are crossed.

Keep a small set of known cases and run them after changes to sources or transformations. Include partial fulfilment, a cancellation, a duplicate import and a revised order. Check that the metric changes only when the documented conditions change. If the same records are later reused by an AI service, give its owner the quality evidence and have them test the new use separately. The goal is to make each result traceable to the records and decisions the business has agreed to rely on.