Retrieval-Augmented Generation: When Company Documents Help an AI System

A Dubai distributor wants sales staff to ask a single question about warranty coverage. Its shared drive contains an approved UAE warranty guide, a superseded guide and supplier agreements with different terms. An AI assistant could search those files and write a neat answer in seconds. If it picks up the old guide or quotes terms from the wrong agreement, the apparent improvement creates a new problem. A sales employee may now give the customer a confident answer that nobody can substantiate.

Retrieval-augmented generation, or RAG, is worth testing when people need to interpret scattered, maintained information and can check the evidence behind an answer. Start with a question people already ask, identify the authoritative documents and decide when the system must say it cannot answer. Compare the result with ordinary search before committing to a new way of working.

Understand what retrieval adds

A RAG application searches an external collection for material relevant to a question and supplies selected passages to a language model as context for its response. The foundational paper by Patrick Lewis and colleagues combines a language model with an external passage index for knowledge-intensive tasks. A company implementation might instead search its approved warranty documents and return a short explanation with links to the relevant sections. The search and the written answer are separate parts of the job, and either can fail. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

For example, a staff member asks whether a replacement component inherits the original product's warranty. The search stage needs to identify the product family, the applicable UAE policy and the replacement clause. The model then needs to explain the clause, including any exception, and point to the source. If the right document never appears in the results, rewriting the instruction to the model may do little. If the correct passage appears but the assistant omits the exception, investigate how it used the evidence.

There is a separate question about whether the answer needs prose. A lookup for a product code or a claim status belongs close to the structured system that owns it. A staff member who knows the name of a form may simply need a link. When several sections of a manual must be read together, an explanation with verifiable citations can save work. Give the application a narrow job and test the simplest way staff could already do it.

Prepare the collection before connecting the model

List the documents you would let an employee rely on. Record the owner, version, effective date, intended audience and applicable product, customer or UAE entity. Decide how an expired rule should be treated. Staff checking a historical claim may need a past policy, while staff answering a current sales question should see the active one. Keep the effective dates searchable and give the application a clear way to distinguish those tasks.

Check what the search service actually indexes. A scanned PDF may have a poor text layer. A table may put the product family in one column and the exclusion in another. Breaking pages into small passages can leave the exception separated from the heading that defines its scope. Inspect a few indexed passages against their originals before judging answer quality. Include mixed Arabic and English names and exact product codes in the trial. An internal nickname and an official model number may refer to the same item, while two similar codes could refer to different warranty rules.

The Abu Dhabi Department of Economic Development’s business reporting post offers local context for the role of organized records. In a company knowledge system, that discipline extends to keeping track of which source was approved, who may read it and when it should stop informing current answers.

Agree how changes reach the index. Have a document owner publish a replacement during testing, then measure the time until the new version appears in search. Check whether the old version remains available for historical queries and whether current answers still quote it by mistake. Repeat the exercise for a withdrawal. A document deletion from a shared drive is only one event in a workflow that may also have indexed text, cached answers and records of earlier requests.

Keep access rules attached to the evidence

Before retrieving a passage, determine what the person asking the question is allowed to see. A general warranty guide might be available to all sales staff. A negotiated agreement may be restricted to the team assigned to the customer. Test the same question with two users who have different entitlements. The user without access should receive neither the passage nor an answer, citation or snippet that exposes its contents. Make sure the application uses the user's permissions where needed even if a service account can read the whole collection.

Microsoft's Azure AI Search documentation describes several document-level access approaches, including query-time restrictions. The supported method depends on the source and configuration. Treat the product documentation as a design reference, then test the actual connector and identity flow. Remove a tester's access and verify what the application returns after the change. Repeat the check across search results, generated answers and any conversation history the user can still open. Microsoft Learn on document-level access control

Review what information leaves the source system. A model request, a saved chat, a log entry and an evaluation dataset can each hold copies or excerpts. Decide which data can appear there and who can read it. Exclude files the workflow has no need to process. Give the privacy and security owners the actual path of the documents and extracted text so they can assess the design for the UAE entity and the data involved.

Treat retrieved material as evidence the model can read. A supplier file could contain a line telling the assistant to send another document or ignore its usual checks. OWASP describes this as indirect prompt injection when instructions are embedded in an external source. Test how the system behaves with a controlled example. Keep the first workflow read-only if its job is to answer questions, and enforce permissions outside the model for any operation the business later decides to allow. OWASP on prompt injection

Score retrieval, answer quality and restraint

Make a test set that a staff member can score from the source collection. For each question, record the correct passage, the conditions a useful answer should convey and the expected behavior if the source is missing or restricted. Include ordinary questions, conflicting versions, exact identifiers, questions that span two sections and questions that have no authorized answer. Hold back some cases until the scored run. Keep a copy of the document versions used in the evaluation so later reviewers can distinguish a model change from a policy update.

Inspect the retrieved passages before scoring the answer. Check whether the relevant material appeared among the passages provided to the model. Then check whether the answer is supported, preserves the qualification and cites the right location. Ask the reviewer to open the link and read the cited section. A citation to an approved policy says little if the section does not support the claim or if the link opens a different version.

Suppose a hypothetical pilot has 80 questions with authorized source material and 20 designed to require refusal or clarification. The correct passage reaches the model for 72 of the 80 answerable questions. Reviewers accept 60 of the 80 final answers. If all 60 accepted answers came from cases with the right passage, there are eight retrieval misses and 12 failures after successful retrieval. The two groups suggest different fixes. In the 20 cases without an authorized answer, suppose the assistant handles 18 correctly and invents an answer in two. Report both failures separately from the answerable cases and assess what someone could do with the false information.

Add a version test that makes the expected answer change. For example, replace a fictional 30-day reporting window with a 45-day window in the approved guide. Check the answer for a new claim and a historical claim covered by the earlier version. Then check whether a saved answer shows when it was generated. That distinction matters if a staff member reopens an old conversation and copies its advice into a new case.

Use the same cases to compare with ordinary search. Give a staff member a search result and see how long it takes to locate and explain the applicable clause. Then measure an assistant answer plus the time to check the cited evidence and fix any errors. Include questions where the generated response takes longer to verify than a search result takes to read. If a new interface adds value only for questions that span documents, deploy it for those questions and keep direct lookup available.

Assign ownership before opening the service

The business owner should be able to explain which documents answer which questions. The search owner should know how to diagnose missing or stale passages. The access owner needs a way to test what happens when permissions change. Set up a route for staff to flag a wrong or unsupported answer and include the question, the cited version and the issue in the report. The person who receives it should be able to correct the collection, the retrieval configuration or the workflow, depending on where the failure occurred.

Estimate the ongoing work. In addition to model and search charges, allow for scanning and cleanup, permissions testing, review, document changes and investigation of bad answers. Run an update without the implementation team present. If nobody in operations can replace a policy and confirm that the replacement has reached search, the service depends on support the business may not have funded.

Start with a bounded release, such as approved UAE warranty documents for one product line, read-only answers and a referral to the policy owner for exceptions. Expand when staff can verify the answers, restricted material stays out of reach and updates arrive in time for the decisions the service supports. If the underlying records disagree or nobody can maintain them, resolve that ownership first. The assistant can make a reliable source easier to use once the business has one.