Intelligent Document Analysis

Your Challenges with Current Analytical Methods

Industries with a lot of customer contact usually have large collections of text and multimedia documents in the form of messages received and sent or conversation logs. In their totality and in each individual document, they contain valuable information: What does your customer need? What are the current trends? What might be the demand that you are not satisfying? We’ll show you how to answer these and many other questions precisely.

The insurance industry is a prime example of a market in which these questions are particularly significant for the ability to remain competitive. Historically low interest, increasing digitization, and greater storm damages due to climate change are forcing providers to look for innovations both in terms of customer focus and the reduction of marginal costs through process automation (black-box processing without manual process steps). AI-supported document analysis can make a valuable contribution in both areas.

Unstructured Data is a Problem for Black-Box Processing

While processes with structured data, such as data from forms, are easily automated, this unfortunately does not apply to “free” forms of communication such as letters, emails, pictures,and voice messages. This unstructured data does not follow any pattern and relevant information can be found anywhere in such a document. In synchronous communication, such as a telephone call or a consultation, a human participant is involved, entering the conversation in the existing processes in a structured manner. Asynchronous communication by means of letters, email, etc., is often recorded afterwards and with significant personnel resources. This situation is aggravated by the fact that feedback or inquiries for the customer – especially outside normal working hours – are delayed until an employee is able to process the corresponding documents. Even when the level of digitization increases thanks to products like chatbots – and with it also the portion of structured data – free communication is still preferred by many customers because they want to “get rid of” their concerns quickly.

Understanding Markets and Customers by Searching for Patterns

Similarly, unstructured documents represent both a great potential and a hurdle to overcome in understanding customers and markets. The potential compared to other, structured channels is the openness to new findings. For example, sales figures can certainly indicate which products are well received at the time, but they cannot show which products or product features are lacking in the product range, or how visible they are to the customer. This requires the evaluation of conversation logs, customer requests, and customer feedback according to the question. Customer satisfaction is just as difficult to determine from simple figures, but is easy to determine from the tone of the last email.

As long as manual processing of customer interaction processes by staff is taken for granted, the cost-benefit consideration of a manual analysis of a document archive is guaranteed to hinder any innovation project. This leaves much potential unused, and knowledge about time trends, white spots in the portfolio, or customer satisfaction disappears in the records.

Use Cases for Intelligent Document Analysis.
Source: Novatec internal

In the illustration, the magnifying glasses mark the spots where intelligent processing is required. For example, this includes extracting information, summarizing content, establishing connections, and looking up facts. At these points, artificial intelligence opens up new possibilities or considerable potential for automation and support.

Mass Data Processing

The objective of this area is to create a basis for realtime processing on the one hand, and to gain insights into the market and customers from the documents collected, on the other. This includes different activities and automated processes that are activated either as needed or at regular intervals respectively.

The basis for these activities is an archive in which all digital documents from realtime processing as well as intermediate results from the technical workflow are collected over time. The combination of both types of data allows the training of AI models for technical tasks using supervised learning methods. For example, a model can learn whether a letter is about a claim for damages or a complaint, if it has access to old letters and the appropriate correct references from the workflow data in the archive. Even if the workflow is already automated, the continuous gathering of human feedback in the archive is important, in the case of wrong decisions that are discovered or particularly uncertain AI responses, for example. This allows the models to be retrained regularly and thus improved over time. Models produced in mass data processing are used later in realtime processing.

But apart from supporting realtime processing, the use of AI in mass data processing is especially advisable for reporting. Because the models are improving continuously, regular reevaluation of the documents in the archive is also recommended in order to keep the database up to date for ad hoc analyses. Some learning processes, especially the so-called “unsupervised learning” processes, can only be used meaningfully here: Topic analyses summarize the contents of the archive semantically and allow you to detect changes in customer requirements or accumulations of certain damage claim details early on. Anomaly detection helps to maintain the high quality of the database or to inspect particularly unusual documents separately.

Realtime processing

The goal of realtime processing is to make unstructured document contents useful to better support the workflow processes. The result is a realization of all the aforementioned advantages, such as customer satisfaction, through faster answers, cost saving due to more black-box processing, and greater customization of customer communication. The AI building blocks created during mass data processing are used for this purpose. Hence, the processing follows the data flow from the incoming document to the result of the technical process, and is modeled as a pipeline.

Based on a concrete data source, non-digital contents such as letters or non-text contents such as audio recordings must be converted to digital documents by the appropriate preprocessing steps. Also in this process step, we have AI solutions available for OCR, intelligent word recognition (IWR), or transcription. But because these steps are usually not yet customer specific, it is also possible to use external providers such as Azure, Amazon, or Google. They have already trained their models with large volumes of data, even from other use cases and industries as well. We will gladly examine for you whether this is advisable for your solution, and to what extent.

Various AI building blocks automatically add more structured information to the document in the pipeline:

• A sentiment analysis describes the “mood” in the document in the form of a numerical value.
• A text classification assigns the matter to different process categories to start the right workflow next.
• An information extraction marks and extracts important contents such a customer numbers, names, places, types of damage, and other details.
• An image analysis can derive even more contents from attachments, such as the brand of vehicles involved in an accident, for example.

All these building blocks are based on machine learning methods, especially deep learning, and document how sure they are of their results with a confidence value.

At the end of this pipeline, your existing workflow engine takes over the process and can request additional human evaluation (manual processing) based on the confidence level, or process the request by automation (black-box processing). Because of the newly acquired information, you have a good basis for further optimization and automation of your existing processes.

Integration and Infrastructure

Because of their different focuses, both areas also differ in the type of integration and infrastructure required. Realtime processing can be parallelized well and, thanks to container virtualization, can be operated in a failsafe and redundant manner as a collection of microservices both in a local environment and in a cloud, for example. Here, integration in existing upstream or downstream processes has the highest priority to which the aspects must be subject. Because mass data processing is associated with the realtime area only through the export of models and the import of data, integration is not crucial here. Instead, in this area the focus is on the high-performing and efficient cooperation of the components among each other. If a specific infrastructure provider has already been determined by your company guidelines for an individual aspect such as data storage, it might be advisable to use the remaining components, such as the training environment for models or reporting services, from the same provider.