user-icon Nadja Hagen & Uwe Eisele
30. September 2021
timer-icon 8 min

AsyncAPI and Apicurio for Asynchronous APIs

The OpenAPI specification has already established itself in many project and application areas. It can be applied to describe HTTP APIs in a standardized way, understandable for humans and computers. While OpenAPI is mainly intended for synchronous interfaces, asynchronous communication places new demands on the interface definition.

Asynchronous architectures can build on different types of protocols, e.g., Kafka, AMQP, or MQTT. Additionally, communication often includes a messaging broker, which maintains several topics or channels. Another difference is the communication style: OpenAPI only allows specifying one-to-one connection links. In contrast to that, asynchronous communication often involves multiple communication partners. Therefore, common patterns like publish/subscribe require a new approach to defining APIs.
AsyncAPI was developed as an extension of OpenAPI to meet these new requirements of asynchronous communication and interfaces. This article highlights the differences between OpenAPI and AsyncAPI in more detail.

As another aspect, large projects usually include multiple teams working and developing together. Therefore, standardized development of APIs is a central aspect of cross-team application development. At the same time, the API definitions need to be available and easy to access for all teams. These demands are also addressed by Apicurio Registry which supports registering different document formats, e.g., OpenAPI, AsyncAPI, GraphQL, Apache Avro, or Protobuf.

Since AsyncAPI is developing quite fast, we want to have a look at what the specification and corresponding tools currently cover. Additionally, we will shortly examine how Apicurio and AsyncAPI can work together at the moment. Nevertheless, there are many new tools and features for AsyncAPI on the roadmap, so this article can only be a snapshot. Stay tuned until the end of this post to get an outlook on what is next to come!

Elements of an AsyncAPI Document

Coming to the basics of AsyncAPI: How does an AsyncAPI definition look? AsyncAPI allows using YAML or JSON for document definition. A document consists of the following elements:

Structure of an AsyncAPI document

In practice, the definition can become quite long. Consider the below example from AsyncAPI’s Github to get an impression of how such a definition can look:

If you want to look at it in a more readable way, you can copy and paste the definition to the AsyncAPI playground. More examples are available on Github.

I do not want to dive too deep into the exact details of the specification because the documentation is already helpful enough for this. But I want to highlight some parts of the specification that could be the icing on the cake when considering it.

Besides some general information about the API, like the title and the description, the “Info” object can contain a “Contact” object. Especially when thinking about multiple groups working together, the responsible team and how to reach them can be linked at this place. In the example from above, this would look like:

The fixed structure defined by the specification might not be sufficient for all use cases. Therefore, it is possible to link external documentation for nearly every object in the AsyncAPI definition. Additionally, user-defined properties can be set to extend the specification by simply prepending an “x-” at the field name. For example, we could add the internal team or project name:

For long and detailed API definitions, the document can become messy fast. As an improvement for this, it is possible to include references to internal and external resources (using the $ref field). They enable to reuse objects and to add schemas and definitions from external sources.
As another use case, let’s assume there are already event-driven applications using an Avro schema for their messages. For example, the schema is registered at Confluent Schema Registry or Apicurio Registry. Using $ref, it is possible to reference this Avro schema within a message object in the AsyncAPI definition:

AsyncAPI also offers an Avro schema parser which collects local Avro references or remote files from Confluent Schema Registry and inserts them into the definition.

As an additional remark: You probably already noted that the channel name in the example is smartylighting.streetlights.1.0.event.{streetlightId}.lighting.measured. In this case, {streetlightId} is a parameter in the channel name. For example, there are channels for each streetlight, and they only differ in the channel name. Then it is possible to define all those channels with the help of parameters.

Kafka, AMQP, MQTT or HTTP: Protocol-Specific Properties

As already mentioned at the beginning of this post, asynchronous communication can involve different types of protocols. As this is a fundamental aspect of your API definitions, AsyncAPI supports adding protocol-specific properties.
These protocol-specific properties can be necessary on different levels: server level, channel level, operation level, and message level. For these levels, AsyncAPI allows defining protocol-specific objects, so-called bindings. This repository provides a complete list of all bindings and their specification details.
For example, it is possible to add a key for Kafka messages:

Unfortunately, for many protocols, there are a lot of binding objects which are not specified yet. E.g., for Kafka, there are no specifications for the Server and Channel Binding objects. In the case of Kafka, relevant but missing properties could be the number of partitions or the delivery guarantee. These properties would be a good use case for specification extensions that currently cannot be used in binding objects.
In addition, some conventions for the specification of bindings are on the roadmap currently. As protocol-specific bindings are still in the alpha version, this feature will become mature with continuous development.

AsyncAPI and Apicurio: A Match?

In contrast to the Confluent Schema Registry, Apicurio is not limited to message schemas but also supports API specifications. Apicurio could therefore also replace the Confluent Schema Registry.
However, this is probably only true if the architecture does not mainly base on a Confluent ecosystem. But depending on the use case, Apicurio can be a helpful addition to the Schema Registry due to the additionally supported formats.

Apicurio Registry allows registering AsyncAPI definitions, which can then be searched and browsed. At the moment, unfortunately, AsyncAPI documents are not rendered to make them more readable. External references (like in the example above) are currently not resolved automatically either. Orientation for this could be the AsyncAPI playground, where this is already possible. However, Apicurio Studio already offers this functionality for OpenAPI, and support for AsyncAPI in Apicurio Studio is already planned.

At the same time, Apicurio also offers the Data Models Library, which allows reading, writing, and modifying AsyncAPI and OpenAPI documents. The Dereferencer-class resolves external references to other documents and inserts them into the definition. It is also possible to add your implementations of the IReferenceResolver interface for custom resolution of references.
When testing the library, I noticed that it cannot resolve all references in a document yet: The AsyncAPI specification also allows references in the payload object of a message (as shown in the example above). The library only checks objects for links that also implement the IReferenceNode interface. However, the payload object does not yet implement this interface, and therefore, references in this object remain unresolved.
There already is a Github issue which will hopefully resolve this soon.

For Avro schemas, Apicurio Registry supports schema validation and a compatibility check. However, this seems to be still missing for AsyncAPI, even when the user interface suggests the opposite (see screenshot below). I tested this feature with syntactically invalid AsyncAPI definitions (which were classified as invalid by AsyncAPI studio), which were not recognized as wrong by Apicurio Registry. Also, definition updates including breaking changes were not recognized as so.

Apicurio Registry: Details of an AsyncAPI document

The Data Models Library of Apicurio and Apicurio Registry give a first idea of what can be done with AsyncAPI. Nevertheless, the functionalities and use cases are still limited. When I tested the Data Models Library, I hoped for more features that can be used at runtime. In fact, it is currently cumbersome to extract message schemas from the AsyncAPI definition which is why I would not recommend the library for this use case.
However, one possibility would be to use Confluent Schema Registry or Apicurio Registry for Avro schemas. Messages could be serialized and deserialized with the schemas from there. The AsyncAPI definition can then reference these schemas and Apicurio Registry could be used as an API catalog. Developers could then use this catalog to work on new services or to mock applications.

Besides these aspects, you should consider that Apicurio has little to no competition. At least I could not find another established open-source registry on the market that supports both schemas and APIs and so many formats. Feel free to leave a comment if you know more about this!

As an alternative, it is also possible to build your own AsyncAPI catalog to support missing features. E.g., you could use existing AsyncAPI libraries and modules like the AsyncAPI React component to render definitions. If you want to know how this could look, have a look at this repository where I implemented this myself. In this prototype, I used the React component with Angular but other frameworks like Vue or NextJS are supported as well.

If you want to test how Apicurio works, check out this small quick start guide which starts an Apicurio container and deploys some AsyncAPI example files.

The Roadmap: What is Next to Come

If you take a closer look at the Github repositories of AsyncAPI, you will quickly notice that a lot is happening here right now. There are many exciting and promising ideas, and the community seems to grow steadily. For example, an event gateway is currently being developed that will make it possible to validate and modify messages even before they arrive at the broker. Another idea is the Glee framework, which ensures that AsyncAPI definition and code match. At first glance, an unexpected project is the AsyncAPI Chatbot: It helps to create an AsyncAPI document without having to know the specification yourself.

Besides these innovative ideas, there are also tools to enable validation and compatibility checks. The CLI tool validates AsyncAPI files, and the Diff tool points out breaking changes to ensure backward compatibility. A combination of these projects would be helpful, as it is already requested on Github. These two tools could also be integrated into Apicurio Registry to add missing features.

The AsyncAPI website also lists tools developed by the community. Many of these and several tools by the core team mainly support JavaScript – so ideally, you should not have a problem with that. Nevertheless, it is to be hoped that other programming languages will be supported in the future, which would make it easier for the community to access the tools.
Many other official projects of the AsyncAPI team are still in the alpha or beta phase and therefore, most of them cannot yet be used to their full potential. But since companies like eBay and Slack already use AsyncAPI, we can expect a lot more to come.
For some projects, it is difficult to find out which functionalities are already supported and which are only intended. Sometimes the documentation in the repositories is not yet very detailed, which makes testing somewhat difficult. However, this is probably because many tools are currently only intended for developers and not for users.

Have you already used AsyncAPI in projects? If yes, how and with which tools, and what were your experiences with them? Feel free to leave a comment!

Comment article