AI in workflow softwareAI · Document intelligence

Ask a process a question and get an answer with its sources

An assistant inside an existing workflow system that reads the documents, images and data of a process, answers questions with the sources it used, and keeps conversations people can return to.

Overview

People who run a process in a workflow platform deal with its data and a stack of documents together. We added an AI assistant to the process screen itself. A user asks a question in plain language, and the assistant works out which documents, images and step data matter, reads them, and answers with the sources it used, without the user opening each file in turn.

What we built

  • Ask about a processQuestions are asked in plain language about one process, with room for a longer description of what the user is after.
  • Reads many file typesText is extracted from Word, PDF and PowerPoint files. Images, including pictures embedded in PDFs, are read by a vision model so their content can be asked about too.
  • Picks what is relevantA quick first step chooses which documents and images relate to the question, using only their names, so only those are read in full.
  • Reads like a person wouldThe assistant is given tools to search a document for a phrase, open a particular page or section, look at an image, read the fields of a step and check who has approved it, and it chooses which to use for each question.
  • Search by meaning and keywordDocuments are split into passages and indexed once. Searches combine matching by meaning with matching by keyword, which finds the right passage even when the wording differs.
  • Long documentsVery large documents are summarised in parts and then combined, and each document gets a fair share of what the assistant can read, so one large file does not crowd out the rest.
  • Sources shownEvery answer lists the documents and pages it used, and the documents are highlighted in the reply.
  • Live answersAnswers stream as they are written, with progress messages such as searching or reading a file, and can be stopped at any point.
  • Quicker repeat answersExtracted text and search indexes are stored against a fingerprint of each file, so unchanged files are not read again, and old indexes are cleared away automatically.
  • Saved conversationsFollow-up questions keep their context. Conversations are saved against the process, reopened, deleted or cleared, with a warning before unsaved work is lost.
  • Clear failuresIf the AI service is slow or unavailable the user gets a plain message and the assistant can fall back to a simpler answer, rather than an error screen.
  • Part of the existing screensThe assistant sits inside the screens people already use, rather than being a separate tool to open.

Under the hood

How it works

Two ways to answer

  • A small set of documents is read once and combined within a fixed text budget, shared fairly between the files, and the answer is written in a single pass.
  • A larger set of documents, or a question that reaches into other steps of the workflow, is handled by the assistant working step by step with tools, reading only what it needs.
  • Requests for a summary follow their own route, and their results are prepared in the background so they appear quickly.

Retrieval by meaning and by keyword

  • Each document is cut into passages of about 1,500 characters with a 200-character overlap, so a sentence is never lost at a boundary. Passages are turned into embeddings in batches, several batches at a time.
  • A question is matched against the passages twice: by vector similarity and by BM25 keyword ranking. The two rankings are merged with reciprocal rank fusion, which finds the right passage whether the user used the document's wording or their own.
  • The best passages come back with their page numbers, which is how an answer can point to where it came from.
  • Each index is stored against a SHA-256 fingerprint of the file. It is looked up in memory first, then in the database, and only built if neither has it. Simultaneous requests for the same file share one build, and indexes unused for 30 days are removed.

Reading documents and images

  • Separate readers extract text from Word, PDF and PowerPoint files, with page markers kept in the text so answers can cite pages.
  • Pictures embedded in PDFs are filtered by size, ranked, limited to the most useful ones, resized and described by a vision model at zero temperature, a few at a time and under their own time limits. Tiny decorative images are skipped.
  • Extracted text is cached for 24 hours by file fingerprint, and identical image jobs running at the same moment are shared rather than repeated.

An assistant that reads with limits

  • A first, inexpensive call chooses which documents and images are relevant, using only their names, so nothing irrelevant is read in full.
  • The assistant then has five tools: search a document, read a page or section, describe an image, read the fields of a step, and check a step's approvers. It decides which to call and in what order.
  • Every question is bounded. There is a maximum number of tool calls, a cap on the size of each tool result and a cap on the total text read. When a limit is reached the tool tells the assistant to write its final answer with what it already has.
  • If a tool fails, the assistant is told to carry on with what it has, so one broken file does not end the conversation.

Streaming and reliability

  • The API streams events to the browser as they happen: progress status, pieces of the answer, a reset when the assistant changes approach, and a final event carrying the sources and whether the cache was used.
  • Calls to the AI service and the embedding service retry a limited number of times, and each stage has its own time limit inside an overall time limit for the whole answer.
  • The user gets a plain message when something is slow or unavailable, stopping a question cancels the work on the server, and the browser can fall back to a simple non-streaming request if streaming is not possible.

Conversations and storage

  • Each saved conversation is stored against its process in the database, and can be listed, reopened and deleted.
  • Earlier turns are sent with each follow-up, and the relevance step looks at the most recent turns, so a short follow-up such as "and the second one?" still finds the right document.

See it

Diagram of how the assistant answers a question: ask, choose sources, read with tools, answer, save
How the assistant answers a question, from the question to the saved conversation. Click the image to open it full size.

Built with

  • AI language-model integration with tool use
  • Retrieval: embeddings plus BM25 keyword ranking
  • Document and image extraction (Word, PDF, PowerPoint, vision model)
  • Streaming responses
  • Caching by file fingerprint
  • Saved conversation history
  • .NET API with MongoDB
  • Angular front end

What it shows

Adding AI to an existing business application in a practical way: it works from the data and documents the user is already looking at, chooses what to read, shows its sources and keeps conversations people can return to.

Facing something similar?

Tell us what you are trying to do and we will tell you honestly how we could help.

Talk to us

Ready to talk?

Whether it is an AI idea, a project or extra people, tell us what you are after and we will tell you honestly how we could help.

Discuss Your Project