Ocelot meets Wavefront - Enhanced Tracing with Wavefront

This is the third part of our blog series “Ocelot meets friends”. The introductory blog post explains how advanced data collection can enhance features provided by modern observability and APM platforms. In this post, we will take a closer look at how Wavefront’s analytics capabilities can benefit from advanced data collection through inspectIT Ocelot, our open source Java agent.

About Wavefront

Being part of VMware since 2017, Wavefront is a high-performance observability platform that allows you to monitor, visualize and analyze your distributed application environment. While supporting a variety of data sources, Wavefront manages to scale with high data ingestion rates and provides a powerful query language. Through this query language, you can turn data from metrics and traces into valuable insights by creating charts, dashboards and alerts. Wavefront is a true SaaS platform. To get started, simply send collected data to a Wavefront proxy forwards your data to Wavefront for further analysis.

Demo Setup

For illustration, we will use the standard example introduced in our initial blog post. We recommend to have a short look at the introduction. The demo application represents a pet clinic where pet owners may schedule visits to the vets. With this demo, we will aim for the following observability aspects:

1. Automated Data Collection
2. Flexible Customization of Data Collection
3. Increased Trace Transparency
4. Powerful Context Propagation
5. Flexible Collection of Business Data

To achieve all of this, we will integrate inspectIT Ocelot (short: Ocelot) with Wavefront using the following monitoring setup (created using openapm.io):

As Ocelot uses OpenCensus (and in the future OpenTelemetry) internally, we just use a Wavefront Proxy to report traces to the Wavefront platform. Metrics are published via Telegraf to the same Wavefront Proxy. This setup is more than easy.

You can easily give this a try yourself! Just check out this demo scenario and follow the corresponding instructions in the readme.

Configuring Ocelot

For simplicity, we will show just one configuration example to demonstrate the flexibility and power Ocelot’s configuration mechanism. The remaining configuration for this demo is available on GitHub.

Whenever a pet owner visits a vet in our demo application, we track the pet type so we can take it into account when analyzing the application behavior. As mentioned in our observability goals, we want to collect the data without the need to change the application’s source code. Let’s see an example of how this can be done with Ocelot. The code snippet below configures Ocelot to extract the pet type from the application code and add it to the traces captured. A more detailed description of this Ocelot configuration is available in Ocelot meets Bits – Enhanced Observability for Datadog

In addition to the configuration from our previous posts in this series, where the pet type tag only added to traces when measuring the number of visits, we’re now also capturing the pet type tag when measuring the services’ response time as well. This metric is already defined and measured automatically with Ocelot’s default configuration. In order to capture the pet type take, we overwrite the metric configuration and add the pet type tag using the instrumentation below:

Analysis in Wavefront

Now that we can collect data form our application using Ocelot, we can start analyzing data in Wavefront. The picture below shows the trace analysis view of Wavefront. In the lower right corner you can see details about the captured spans (traces). Wavefront recognizes our custom pet type tag and displays it in the trace details. This automatic detection allows us to reference our custom tag when using Wavefronts query interface to search through traces. In the example, we filtered all traces to show only those which include a pet type tag with the value “dog”.

Remember how we adjusted the Ocelot configuration for the default HTTP response metrics? In the first graph below we can see the result of this change. While usually the response time be displayed as overall average, we can nowuse Wavefronts query language to dive deeper and group it by pet type. This way, we can clearly identify the much higher response times for requests that have pet type “dog”.

Conclusion

Ingesting data with Ocelot was fairly easy and apart from a few fixed name tags required by Wavefront most of the standard OpenCensus tags were recognized correctly. This applies to our custom pet type tag in particular, which was properly processed by Wavefront at first attempt. To analyze metrics, Wavefront offers its own query language which provides many features and offers a lot of customization. This allows for many more possibilities than what we used in our example. Combined with Ocelot’s instrumentation capabilities, including customizable tags and metrics, Wavefront can offer highly customized information retrieval. Nonetheless, as it is the case for most interfaces, one has to learn Wavefront’s query language first. As this was our first step towards Wavefront as well, we too had to take a look at the documentation to implement this demo. Thankfully, it is written well.

Posts in this Series

• Introductory Post: Ocelot meets Friends – Enhancing Modern Observability Platforms
• Part 1 – Ocelot meets Bits – Enhanced Observability for Datadog 
• Part 2 – Ocelot meets Lightstep – Enhanced Tracing with Lightstep
• Part 4 – Ocelot meets Elastic – Better Java Instrumentation for Elastic APM via Jaeger

Coming Soon:

• Part 5 – Ocelot meets SignalFX
• Part 6 – Ocelot meets Instana
• Part 7 – Ocelot meets NewRelic