Monitor processes in camunda BPM with the help of a Camunda cockpit plugin - part 1: overview
It has been about one year since Eberhard posted about our cockpit statistics community contribution to the open source bpm framework Camunda BPM.
It has been about one year since Eberhard posted about our cockpit statistics community contribution to the open source bpm framework Camunda BPM. In this post, Ingo, Kerstin and I want to write about what has happened since then in terms of features and what will be the next steps. In the further parts of this post, Ingo and Kerstin will go into detail with some of the Camunda cockpit plugin’s features, while I will present a general overview and my personal improvements. Similar to our first post I want to start with a short motivation on why we built the plugin.
If you enter “business process management” into your favorite search engine, you will quickly find some definition of that term on Wikipedia with a life-cycle overview on the different disciplines including modeling, execution, monitoring and optimization. If you take a closer look at, for instance, the Camunda BPM lifecycle, you will find the phases analysis, conception, implementation and controlling where the latter should be repeated until an improvement is needed. Both life-cycles include disciplines that focus on monitoring, controlling or optimization. But how do you optimize and monitor your process implementations and apply control mechanisms? How can you gain any insights on how your processes are really executed within your process applications? Does the execution of processes fit to the conditions and expectations you specified in the beginning? To answer these questions, you have to drill deeper and phrase more explicit questions:
Do process steps exist that need more time or human interaction than expected?
Do process participants skip or delegate certain steps more often than expected?
Do they make decisions that lead to congestion in some process branches that were considered as not overly important?
Do my service calls perform as expected?
I think that this list of questions can easily be continued especially in the context of your organization. In my opinion, the process monitoring and statistics plugin we have created helps to at least get a general idea of the current situation in your engine to phrase some of these questions from an engine point of view. Maybe, some of these questions can already be answered at least partially. As a result you can get into touch with your business organization and provide ideas and approaches on how to improve processes. It might help to analyze already existing problems, too.
The related context of the plugin is sometimes called ‘process monitoring’ or ‘business activity monitoring’ and can include many more features like report generation or alerting.
So what happened since the last post about the plugin? Due to the Camunda community and clients, we were able to fix several issues and got feedback on how to improve our Camunda cockpit plugin. As a result, we now have only one tab with pie charts showing information about your running process instances, completed process instances and process instances with incidents ordered by process definition key. If you want to get further information about one process definition key, you can easily ‘drill-in’ by a click on the respective piece of pie. If you click on a piece on the pie chart showing the ended instances, you get aggregated information about the minimal, maximal and mean running time of the included activities. If you click on the pie chart showing the running instances, you get the number of currently running user tasks and how many of them are assigned.
Overview on running and ended process instances and process instances with incidents
Another feature that was requested is the possibility to filter the process definitions considered for the pie charts. Next to the headline I positioned the link ‘Settings’ that opens a modal where you can select the considered process definitions for every pie chart. In addition, you can switch off the ‘auto-reload’ which currently happens after you refresh the cockpit page. If switched off, you have to reload the plots manually and can thus prevent the plugin from querying data again and again if you really like the ‘F5’-key on your keyboard 😉 .
In the settings modal you can filter the considered process definition keys
If you imagine a high number of process definitions, it would be very inconvenient if you have to filter the keys every time you want to use the plugin. Because of that, your settings are saved to your browser’s local storage (In case it is supported 😉 ) based on the user name of your cockpit account. That means that per browser each user of the cockpit can have different settings. If you want to export and import your settings, this can currently be achieved by a direct access to your browser’s local storage and exporting it to a text file. At the destination browser, you have to import it and store it using the same key as in the origin browser’s local storage. Please see the following issue on Github if you have any comments.
But there is more. Ingo did a heavy refactoring related to his analytics plots that he will describe in the second part of this post. Kerstin will introduce a completely new set of features in her part of the post covering process diagram overlays and new ways to get insights on your activity’s histories.
Backend Improvements & Refactoring
Apart from these visual improvements, we refactored our queries to the process engine database and shifted as many queries as possible to the REST API of Camunda BPM. By that, we can use a supported interface that is maintained in respect to the new versions of the engine. Most of the aggregations are thus done on the client side within your browser. Nevertheless, there still exist some custom queries that were improved as well since Camunda changed their schema a little lately. As a result, you will now find, for example, the process definition key in the history tables like ACT_HI_ACTINST or ACT_HI_PROCINST. In earlier versions of Camunda, an expensive ‘join’ was needed to get the process definition key in addition to the other historic details.
Further improvements were realized related to the responsiveness of the plugin interface in relation to different screen sizes. As there are still projectors with resolutions down to 1024x768px, I enhanced the plugin to inform every component about changing dimensions. As a result, the pie chart’s heights change as well as some of the plots in the “Analytics”-Tab. If possible, we used the provided css classes from the customized twitter bootstrap which is shipped with camundaBPM to further improve automatic responsiveness.
Many potential users ask questions on the performance of our Camunda cockpit plugin and if there may be any issues if they deploy it to their test or production environments. Although we do not have any extensive performance tests or proof-of-concepts yet, we can make the following statements:
Regarding our custom queries, all information is queried from the history tables (ACT_HI_*) or repository tables (ACT_RE_*) and no information is queried from the runtime tables. In addition to all completed activities or process instances, started and not yet completed activities (like, for instance, user tasks that were just created) can be found in the history tables, too. As stated in the user guide of Camunda BPM, the history table is not needed for the runtime to work.
Most of the queries are done via the official Camunda BPM REST API. This interface is supported and maintained by Camunda. There exists one custom query including a join of the historic activity instance table and the process definition repository table to align activity’s process definition ids to version and name of the respective process instance.
Most of the aggregations are done on the client side of the plugin which is your browser. There are some custom queries, where database aggregations like ‘sum’, ‘min’ or ‘max’ are used. If there are performance issues only your browser should crash in the worst case.
If you are interested in using the plugin and have performance concerns please contact me. I would really like to challenge the plugin with a fully-filled history table ;).
Future of the Plugin
So what are the plans for the future? First of all we want to get as much feedback as possible to further improve the Camunda cockpit plugin. Regarding new features we have many ideas including a generic dashboard and filtering mechanism consolidating our different approaches, the visualization of business data and of course information about rules executions. As presented on the bpmcon2015 Camunda BPM will support the executions of decision tables as part of the DMN notation from the upcoming version 7.4. I am really curious about that and possible new features since I already found history tables for decision executions and decision statements in their last alpha-2 snapshot 🙂 .
Furthermore, we would like to discuss new requirements or questions that you or your organization has in the context of process monitoring in general and specifically in terms of the plugin. So if you have any feedback, questions or requirements please do not hesitate to contact us.
As all community contributions are open-source, feel free to contribute!
Now I want to give the floor to Ingo in the next part.