Your Challenges with Current Analytical Methods

Industries with a lot of customer contact usually have large collections of text and multimedia documents in the form of messages received and sent or conversation logs. In their totality and in each individual document, they contain valuable information: What does your customer need? What are the current trends? What might be the demand that you are not satisfying? We’ll show you how to answer these and many other questions precisely.

The insurance industry is a prime example of a market in which these questions are particularly significant for the ability to remain competitive. Historically low interest, increasing digitization, and greater storm damages due to climate change are forcing providers to look for innovations both in terms of customer focus and the reduction of marginal costs through process automation (black-box processing without manual process steps). AI-supported document analysis can make a valuable contribution in both areas.

Unstructured Data is a Problem for Black-Box Processing

While processes with structured data, such as data from forms, are easily automated, this unfortunately does not apply to “free” forms of communication such as letters, emails, pictures,and voice messages. This unstructured data does not follow any pattern and relevant information can be found anywhere in such a document. In synchronous communication, such as a telephone call or a consultation, a human participant is involved, entering the conversation in the existing processes in a structured manner. Asynchronous communication by means of letters, email, etc., is often recorded afterwards and with significant personnel resources. This situation is aggravated by the fact that feedback or inquiries for the customer – especially outside normal working hours – are delayed until an employee is able to process the corresponding documents. Even when the level of digitization increases thanks to products like chatbots – and with it also the portion of structured data – free communication is still preferred by many customers because they want to “get rid of” their concerns quickly.

Understanding Markets and Customers by Searching for Patterns

Similarly, unstructured documents represent both a great potential and a hurdle to overcome in understanding customers and markets. The potential compared to other, structured channels is the openness to new findings. For example, sales figures can certainly indicate which products are well received at the time, but they cannot show which products or product features are lacking in the product range, or how visible they are to the customer. This requires the evaluation of conversation logs, customer requests, and customer feedback according to the question. Customer satisfaction is just as difficult to determine from simple figures, but is easy to determine from the tone of the last email.

As long as manual processing of customer interaction processes by staff is taken for granted, the cost-benefit consideration of a manual analysis of a document archive is guaranteed to hinder any innovation project. This leaves much potential unused, and knowledge about time trends, white spots in the portfolio, or customer satisfaction disappears in the records.

Use Cases for Intelligent Document Analysis.
Source: Novatec internal

In the illustration, the magnifying glasses mark the spots where intelligent processing is required. For example, this includes extracting information, summarizing content, establishing connections, and looking up facts. At these points, artificial intelligence opens up new possibilities or considerable potential for automation and support.

The Advantages of Using Intelligent Document Analysis

How you benefit from AI-supported document analysis

More and more companies are benefiting from automated document analysis by speeding up their processes, for example, realizing faster response times, greater customer satisfaction, and higher customer throughput at lower costs. In addition, a combination with other technologies such as chatbots becomes much easier and creates more added value. (Partially) automated processing also frees up the capacity of employees for more important activities, which in turn increases employee satisfaction. This makes AI-supported document analysis a core element of your digitization strategy.

The advantages of using intelligent document analysis are:

  • Customer satisfaction increased as a result of faster processes
  • Improved quality of services through greater know-how
  • (Product) improvements through pattern and trend recognition
  • Targeted, personalized, and sustainable approach to customers through data-driven marketing

If your “data treasure” is already in digital form, and if it is maintained, it will provide many new possibilities to create added value. Discover previously hidden connections in your data and extract valuable findings about your processes or customer behavior. Improve your quality by connecting different sources of knowledge and expertise. Identify potentials for improvement and enhance your competitiveness. Design your approach to customers in a way that is more sustainable and more targeted by data-driven marketing.

What is analyzed?

Document analysis is a very broad term. It basically refers to everything that can be recorded and stored in the form of coherent analog or digital data. For example, in the context of insurance, it can be different channels for customer communication that are related: forms, letters, email correspondence, recorded telephone conversations (as an audio file or transcribed), just to mention a few possibilities.

Here, conversion from different sources to formats that can easily be processed by machine is of great relevance. In addition to classic text recognition (OCR), there are new processes that use artificial intelligence to significantly improve the recognition rate of characters, words, and sentences. In addition, it is also possible to extract topics (topic modeling) and moods (sentiment analysis) from texts. For example, this would allow you to prefilter your input channels and sort them according to subjects and categories such as automobile insurance cancellations, complaints concerning homeowners insurance, or general inquiries. Internet sites can also be considered as documents. They already exist in digital form, although the sheer volume may make it difficult to extract useful information under certain circumstances. With text mining methods, data can still be accumulated and rendered usable. Imagine forum postings in consumer portals where products and services are discussed. Here it is possible to search specifically for opinions and moods concerning your products and conduct a kind of consumer research without direct surveys.

The scenarios mentioned are intended only as examples. The possible uses are incredibly diverse, so there is certainly a valuable opportunity for you as well to make good use of AI-supported document analysis.

How Intelligent Document Analysis Works

The implementation of automatic document analysis can be broken down technically into two areas: Realtime processing applies AI models, while mass data processing creates the models and supports technical analyses. They differ in terms of their objectives and the chronological sequence of the activities.

Functionality of Intelligent Document Analysis.
Source: Novatec internal

Mass Data Processing

The objective of this area is to create a basis for realtime processing on the one hand, and to gain insights into the market and customers from the documents collected, on the other. This includes different activities and automated processes that are activated either as needed or at regular intervals respectively.

The basis for these activities is an archive in which all digital documents from realtime processing as well as intermediate results from the technical workflow are collected over time. The combination of both types of data allows the training of AI models for technical tasks using supervised learning methods. For example, a model can learn whether a letter is about a claim for damages or a complaint, if it has access to old letters and the appropriate correct references from the workflow data in the archive. Even if the workflow is already automated, the continuous gathering of human feedback in the archive is important, in the case of wrong decisions that are discovered or particularly uncertain AI responses, for example. This allows the models to be retrained regularly and thus improved over time. Models produced in mass data processing are used later in realtime processing.

But apart from supporting realtime processing, the use of AI in mass data processing is especially advisable for reporting. Because the models are improving continuously, regular reevaluation of the documents in the archive is also recommended in order to keep the database up to date for ad hoc analyses. Some learning processes, especially the so-called “unsupervised learning” processes, can only be used meaningfully here: Topic analyses summarize the contents of the archive semantically and allow you to detect changes in customer requirements or accumulations of certain damage claim details early on. Anomaly detection helps to maintain the high quality of the database or to inspect particularly unusual documents separately.

Realtime processing

The goal of realtime processing is to make unstructured document contents useful to better support the workflow processes. The result is a realization of all the aforementioned advantages, such as customer satisfaction, through faster answers, cost saving due to more black-box processing, and greater customization of customer communication. The AI building blocks created during mass data processing are used for this purpose. Hence, the processing follows the data flow from the incoming document to the result of the technical process, and is modeled as a pipeline.

Based on a concrete data source, non-digital contents such as letters or non-text contents such as audio recordings must be converted to digital documents by the appropriate preprocessing steps. Also in this process step, we have AI solutions available for OCR, intelligent word recognition (IWR), or transcription. But because these steps are usually not yet customer specific, it is also possible to use external providers such as Azure, Amazon, or Google. They have already trained their models with large volumes of data, even from other use cases and industries as well. We will gladly examine for you whether this is advisable for your solution, and to what extent.

Various AI building blocks automatically add more structured information to the document in the pipeline:

  • A sentiment analysis describes the “mood” in the document in the form of a numerical value.
  • A text classification assigns the matter to different process categories to start the right workflow next.
  • An information extraction marks and extracts important contents such a customer numbers, names, places, types of damage, and other details.
  • An image analysis can derive even more contents from attachments, such as the brand of vehicles involved in an accident, for example.

All these building blocks are based on machine learning methods, especially deep learning, and document how sure they are of their results with a confidence value.

At the end of this pipeline, your existing workflow engine takes over the process and can request additional human evaluation (manual processing) based on the confidence level, or process the request by automation (black-box processing). Because of the newly acquired information, you have a good basis for further optimization and automation of your existing processes.

Integration and Infrastructure

Because of their different focuses, both areas also differ in the type of integration and infrastructure required. Realtime processing can be parallelized well and, thanks to container virtualization, can be operated in a failsafe and redundant manner as a collection of microservices both in a local environment and in a cloud, for example. Here, integration in existing upstream or downstream processes has the highest priority to which the aspects must be subject. Because mass data processing is associated with the realtime area only through the export of models and the import of data, integration is not crucial here. Instead, in this area the focus is on the high-performing and efficient cooperation of the components among each other. If a specific infrastructure provider has already been determined by your company guidelines for an individual aspect such as data storage, it might be advisable to use the remaining components, such as the training environment for models or reporting services, from the same provider.

Our Intelligent Document Analysis Service

Document analysis is a very broad field and can be very complex under certain circumstances. Therefore it is important for us to get an accurate picture of your business case and understand your requirements of such a system very well. This is possible only with a common exchange of ideas and takes place in the form of a discussion or workshop with all the relevant participants on your side together with our team that will implement the solution. We help you in the choice of technology and in the implementation of a pipeline for document analysis and integration in your infrastructure:

  • In an initial discussion or workshop, together we will explore your current situation and constraints, discuss your business goals, and examine whether all the requirements for using possible solutions are met. Together we will work out a roadmap.
  • In the next step, based on our experience, we can choose the appropriate technologies for your requirements precisely and plan the implementation with you in detail. Together with you, we will also determine whether you would rather use existing systems or services to be integrated into your plan, or whether your specifications require a customized development, which we will happily assume for you and with you.
  • If necessary, or if you wish, we will demonstrate whether the solution can reasonably be implemented according to the objectives determined in a feasibility study.
    An agile approach in the implementation of the project is important for us. In close cooperation with you, we create the solution step by step and integrate it into your structures on an ongoing basis. This way we can react to changes flexibly and ensure maximum transparency. Our core idea is that each step and each intermediate product already create real added value for you.

Your direct contact

Dr. Harald Bosch

Managing Consultant
Table of contents
Dr. Harald Bosch Managing Consultant