Skip to main content

Use Enterprise Events with Schema Registry

Tags:
🎯context

For your asychronous communications within and between services, you want to establish well defined "contracts" of the event payload data for everyone to use by means of using the new schema registry supported feature in IBM DevOps Solution Workbench.

Description​

This How-To will cover how you can configure the schema registry that holds the definitions of the possible payload structures, use enterprise events in your service and migrate old events to the new events 2.0.

ℹ️note

Please note that the links to the workbench tools in this tutorial only apply to the IBM Education Environment we provide. If you are using a different environment, e.g. your own installation, you will need to navigate directly to the required tools.

💡tip

The Workbench uses Apicurio as a schema Registry. The schema is in format of JSON Schema 2019-09.

How to configure​

In order to setup the Schema Registry integration, you will need to add the following secrets to the deployment target

k5-schema-registry-binding

  1. URL; Schema Registry URL.
  2. securityEnabled; Boolean indicating whether security is enabled for the Schema Registry. If yes, more data for authentication will have to be provided.
ℹ️note

You will need the following secret only if k5-schema-registry-binding.securityEnabled is TRUE.

k5-schema-registry-oauth-secret

  1. baseurl; Base url of Schema Registry authentication server.
  2. realm; Schema Registry authentication realm.
  3. client.id; Schema Registry client id.
  4. client.secret; Schema Registry client secret.

How to migrate old events to events 2.0​

If you have used events in your services, it is possible to migrate your existing events to the new events using schemas. To migrate events in your Domain Service projects to use schemas instead of entities, you need to follow some simple steps.

Java Pre-Migration steps​

For Java language as implementation language, there are some preparatory steps that should be executed before migrating old events to events 2.0.

  1. Clone (k5 clone) or update (k5 pull) your Domain Service service project.
  2. Open your Domain Service service project.
  3. Navigate to pom.xml file.
  4. Search for the parent attribute and change it to match the new "cp-framework-sdk" version, see the following snippet example:
<parent>
    <groupId>de.knowis.cp.sdk</groupId>
    <artifactId>cp-framework-managed-sdk-parent</artifactId>
    <version>4.0.173</version> <!-- Update this to the new version -->
    <relativePath>
        ./framework/repo/de/knowis/cp/sdk/cp-framework-managed-sdk-parent/4.0.173/cp-framework-managed-sdk-parent
    </relativePath>
</parent>
  1. Push your adjustment and run pipeline to make sure projects deploy successfully.

Migration steps​

Create New Schema​

  1. Open the Schema page in the Solution Designer.

  2. Create a new schema containing all the properties needed for the event.


Adjust event in producing service​
  1. Create a new event for the use case you want to migrate.
  2. Choose a new topic for it (one topic per event is suggested and recommended).
  3. Add your schema as payload of the event.
SchemaRegistry from Solution Designer
  1. Keep the old event to still support the old behavior.
  2. Commit and push changes in Solution Designer.
  3. Clone/Pull your project.
  4. Implement new publishing logic.
  5. Deploy/Release your service by triggering a pipeline. After these steps, your service will now publish the old and the new event.
Adjust agent in the consuming service​

You can adjust your agent in two different ways, either adjusting an existing service or creating a new event.

The following steps will show you how to adjust the existing event

  1. Remove the entity payload from the event.
  2. Adjust topic binding of the event to use the same one as in the consuming service.
  3. Select the new schema as payload for the event.
  4. Commit and push changes in Solution Designer.
  5. Clone/Pull your project.
  6. Adjust agent implementation to deal with the new event payload.
  7. Deploy/Release your service by triggering a pipeline.

Alternatively, you can create a new event and replace the old one

  1. Create a new event using the same topic binding as in the consuming service.
  2. Select the new schema as payload for the event.
  3. Create a new agent using the event as trigger event.
  4. Delete old event and agent.
  5. Commit and push changes in Solution Designer.
  6. Clone/Pull your project.
  7. Implement agent implementation of the new agent.
  8. Cleanup agent implementation of the old agent.
  9. Deploy/Release your service by triggering a pipeline.

Either ways, following the previous steps, your service will now use the new event and no longer the old event.

💡tip

After you ensured, that all consuming services use the new event, you can delete the old event and deploy your service again.

How to use events in your service​

Define event payload​

An event can carry a payload with it which can hold additional information around what happened, e. g. the orderId of the order that has been created or the new name of a customer after it changed.

To ensure governance when dealing with events, schemas from a shared schema registry are used to describe the payload of an event. They act as a reliable contract between event producers and consumers.

Within an event, you can define your schema via the Select schema from schema registry capability. IBM DevOps Solution Workbench will offer you all schemas from the connected schema registry. As a schema can evolve over time, multiple versions of the schema can exist. Therefore, the version has to be chosen in the second step. There, you also get an overview of all the properties that are defined for the schema.

ℹ️note

When a schema has been used as event payload in the event producer, the same schema is expected to be chosen in the service that consumes the event.

Update schema version of event payload​

As requirements change over time, new versions of schemas get created and usually the events want to use the new structure as well.

To update the event to use a new schema version, please use the Change schema version capability on the event page. Select your version of choice in the dropdown and save the changes for updating the event.

It is recommended to not update your event to a schema that has breaking changes to the current one. Otherwise event consumers will fail when they are processing the events. If you have major changes that you want to implement, it is recommended to create a new event publishing on a new topic. After all consumers are adjusted to use the new event, it is safe to delete the "old" one.

ℹ️note

It is also possible that the producer service updates to a new version of the schema, while the consumer services still keeps the older version. The consuming of the event will keep working as long as no breaking change in the new schema version.

🌟Congratulations!

Now, you understand all about using events with Schema Registry.

Please find more information about Schema Registry: