The Kumuluz Blog

KumuluzEE Security 1.3.0 released

Fri, 09 Jun 2023 00:00:00 +0000

We are excited to announce a new minor version of KumuluzEE Security - 1.3.0.

This release adds support for new authentication provider - Firebase Authentication.

Features

Validate ID tokens generated by Firebase.
Check roles provided in custom claims and deny access if user lacks required roles.
Validate session cookie instead of ID token.

Check out the release on GitHub: KumuluzEE Security v1.3.0

KumuluzEE REST 2.0.1 released

Tue, 28 Mar 2023 00:00:00 +0000

We are excited to announce a new major version of KumuluzEE REST - 2.0.1.

This release brings support for Java 17, along with the support for LocalDateTime, LocalDate, LocalTime and OffsetTime temporal fields and a couple of new filters - the BETWEEN and NBETWEEN filters.

The minimum supported Java version is now Java 11. If you are still using older versions of Java, please use the 1.x.x version of KumuluzEE REST instead.

Features:

Support for Java 17.
Added BETWEEN and NBETWEEN filters
Added support for LocalDateTime, LocalDate, LocalTime and OffsetTime fields

Enhancements:

Upped dependency versions.

Check out the release on GitHub: KumuluzEE REST v2.0.1

KumuluzEE JCache 2.0.0 released

Sat, 03 Dec 2022 00:00:00 +0000

We are excited to announce the next major version of KumuluzEE JCache - 2.0.0. This release adds support for the newest KumuluzEE version 4 and adds support for Java 17 LTS and Java 18.

This release also uses the new configuration prefix kumuluzee.jcache.caffeine.caches instead of kumuluzee.jcache.caffeine.caffeine.jcache and adds support for the entire configuration framework instead only yaml files, including overriding properties from ENV variables.

An example of migration to the new configuration prefix:

Old:

kumuluzee:
  jcache:
    caffeine:
      caffeine.jcache:
        default:
          policy:
            maximum:
              size: 10000
        exampleCache:
          policy:
            eager-expiration:
              after-write: "1s"

New:

kumuluzee:
  jcache:
    caffeine:
      caches:
        default:
          policy:
            maximum:
              size: 10000
        exampleCache:
          policy:
            eager-expiration:
              after-write: "1s"

Since the support for Java 1.8 is dropped by KumuluzEE 4.0.0, the KumuluzEE JCache 2.0.0 also drops support for Java 1.8.

Features

Added support for KumuluzEE 4.0.0 and above
Added support for Java 17 LTS and Java 18

Bugs

Fixed initialization from configuration - do not rely on a bug in core, instead use getMapKeys/getListSize
Fixed initialization from configuration - use the whole configuration framework instead of only yaml files

Note: This release of KumuluzEE Config MP requires the KumuluzEE 4.0 or higher in order to work correctly. Consequently, Java 11 or higher is also required.

Check out the release on GitHub: KumuluzEE JCache v2.0.0

KumuluzEE CORS 2.0.0 released

Fri, 02 Dec 2022 00:00:00 +0000

We are excited to announce the next major version of KumuluzEE CORS - 2.0.0. This release adds support for the newest KumuluzEE version 4 and adds support for Java 17 LTS and Java 18. Since the support for Java 1.8 is dropped by KumuluzEE 4.x, the KumuluzEE CORS 2.0.0 also drops support for Java 1.8.

Features

Added support for KumuluzEE 4.0.0 and above
Added support for Java 17 LTS and Java 18

Note: This release of KumuluzEE Config MP requires the KumuluzEE 4.0 or higher in order to work correctly. Consequently, Java 11 or higher is also required.

Check out the release on GitHub: KumuluzEE CORS v2.0.0

KumuluzEE Rest Client 2.0.0 released

Thu, 01 Dec 2022 00:00:00 +0000

We are excited to announce the next major version of KumuluzEE REST Client - 2.0.0. This release adds support for the newest KumuluzEE version 4 and adds support for Java 17 LTS and Java 18. Since the support for Java 1.8 is dropped by KumuluzEE 4.x, the KumuluzEE REST Client 2.0.0 also drops support for Java 1.8.

Features

Added support for KumuluzEE 4.0.0 and above
Added support for Java 17 LTS and Java 18

Bugs

Fixed issue with @FormParam processing and content-type header

Note: This release of KumuluzEE Config MP requires the KumuluzEE 4.0 or higher in order to work correctly. Consequently, Java 11 or higher is also required.

Check out the release on GitHub: KumuluzEE Rest Client v2.0.0

KumuluzEE Health 2.4.4 released

Wed, 30 Nov 2022 00:00:00 +0000

We have released a minor release of KumuluzEE Health 2.4.4 with the following: This patch release fixes a initialization bug, where built-in health checks were not initialized from configuration. Bugs:

Fixed built-in health check configuration detection - use getMapKeys and getListSize instead of yaml-specific Map/List detection, that was removed in KumuluzEE 4.1.0

Note: This release of KumuluzEE Config MP requires the KumuluzEE 4.1.0 or higher in order to work correctly. Consequently, Java 11 or higher is also required.

Check out the release on GitHub: KumuluzEE Health v2.4.4

KumuluzEE MicroProfile Config 2.0.0 released

Thu, 14 Jul 2022 00:00:00 +0000

We are excited to announce the next major release of KumuluzEE Config MP. KumuluzeEE Config MP 2.0.0 implements the MicroProfile Config 2.0 specification, which includes many improvements, among others the support for the @ConfigProperties annotation which extracts bulk config properties into a POJO, the API which allows developers to determine the winning source for a configuration value, and more.

Note: This release of KumuluzEE Config MP requires the KumuluzEE 4.1.0 or higher in order to work correctly. Consequently, Java 11 or higher is also required.

Some of the bigger changes in the MicroProfile Config 2.0 specification like property expressions (configuration value interpolation) and configuration profiles were added to the core KumuluzEE and can be used with or without KumuluzEE Config MP extension. To read more about these changes see the KumuluzEE 4.1.0 release notes.

The @ConfigProperties annotation allows bulk extraction of config properties into a CDI bean and works similarly to the KumuluzEE native @ConfigBundle annotation. It works in the MicroProfile Config’s environment and supports its features (e.g. converters, custom ConfigSource implementations, …). For example, take a look at the following class:

@ConfigProperties(prefix = "integrations.customer-service")
public class CustomerServiceConfig {
    @ConfigProperty(name = "host", defaultValue = "example.com")
    private String hostname;
    private int port = 8080; // 8080 is the default value
    private Optional<String> authorization;

    // getters
}

The Config API has also received some improvements. It includes a method to acquire a ConfigValue instance, exposing metadata about the configuration value such as the configuration source that provided the value, its ordinal and the raw value before property expression evaluation. Additionally, the converter mechanism has been exposed, allowing manual conversion of arbitrary values through the Config API.

For all changes included in this release, see the MicroProfile Config 2.0 specification.

Features:

Implemented MicroProfile Config 2.0 specification
Migrated to KumuluzEE 4.x.x and dropped support for Java 1.8

Bugs:

Fixed default value not being used when injecting Optional with the default value supplied by @ConfigProperty annotation

Check out the release on GitHub: KumuluzEE Config MicroProfile v2.0.0

Announcing KumuluzEE Database Schema Migrations 1.0.0 release

Mon, 04 Jul 2022 00:00:00 +0000

We are pleased to announce the first stable release of the KumuluzEE Database Schema Migration extension for the KumuluzEE microservice framework.

The extension provides an easy way to migrate database schemas with Liquibase. It allows migrations at application startup or, if necessary, when the application is already running. Therefore, the extension can speed up the development process by ensuring that database schemas are up to date when the application starts.

The extension can be configured in the common KumuluzEE configuration file. The only requirement is a configured data source where the schema migrations can be performed. Since the extension is based on the Liquibase, all data sources supported by the Liquibase can be used in conjunction with this extension.

The main advantages of the extension are:

Ensures that the database schema is up to date
Eliminates the need to perform migrations manually
Allows database schema migrations to be managed in the runtime
Enables database schema migrations for multiple databases
Manages database schema migrations on a per-application basis
Optionally starts each application run with fresh database state

To get started quickly, configure the desired data source and link the migration to it.

kumuluzee:
  datasources:
    - jndi-name: jdbc/ExampleDS
      connection-url: jdbc:postgresql://localhost:5432/postgres
      username: postgres
      password: postgres
      pool:
        max-size: 20
  database-schema-migrations:
    liquibase:
      changelogs:
        - jndi-name: jdbc/ExampleDS
          startup:
            drop-all: false
            update: true

For step-by-step guide (example) see KumuluzEE migrations with Liquibase.

Announcing KumuluzEE 4.1.0 release - with improvements to the configuration framework!

Tue, 21 Jun 2022 00:00:00 +0000

We are announcing the availability of KumuluzEE version 4.1.0.

This release brings improvements to the configuration framework, such as configuration profiles, improved variable interpolation, and more!

Configuration profiles can be used to define plug-in configuration fragments, that can be turned on or off at the deployment time. The fragments can define new configuration values or override existing ones in config.yml file. Let’s look at an example defining the profile live. Start by defining the config.yml file as usual:

config.yml

integrations:
  customer-service:
    url: http://localhost:8081

Now let’s define the live profile fragment:

config-live.yml

integrations:
  customer-service:
    url: http://customer-service.example.com:8081

When deploying the service, the profile can be activated by setting the configuration value kumuluzee.config.profile to live, for example add the environment variable:

KUMULUZEE_CONFIG_PROFILE=live

Multiple configuration profiles can be activated by setting the configuration value kumuluzee.config.profile as CSV. When overriding values, the last defined profile has the largest priority. For example: KUMULUZEE_CONFIG_PROFILE=docker,live.

The configuration profiles mechanism is generic and can be used to alter the behavior of your custom ConfigurationSource implementations by implementing the initProfile(String profileName) function of the interface. The interface now also defines the default implementations of converter functions, which makes implementing custom configuration sources easier and avoids duplication of implementations when the backing configuration source returns only String values.

Another improvement added in this release is the support for nested value interpolation and support for default values when using value interpolation. To understand these changes better, take a look at the next configuration:

integrations:
  active-tenant: t2
  tenant-t1:
    url: t1.example.com
  tenant-t2:
    url: t2.example.com
  
  customer-service:
    url: ${integrations.tenant-${integrations.active-tenant:t1}.url}/customer-service

In this case the integrations.customer-service.url configuration would be t2.example.com, since integrations.active-tenant is t2. If active tenant configuration key would not be defined, the default value (t1, defined after the :) would be used instead.

See the full changelog below.

Features:

Added support for configuration profiles
FileConfigurationSource loads additional files when configuration profile is activated

Enhancements:

Improved configuration value interpolation by adding nested interpolation and default value support
Implemented ConfigurationSource converter functions as default interface functions
Upped jandex version

Bugs:

Fixed return value of yaml configuration source: return Optional.empty() if found config value is a Map or a List
Defined default implementation for the deprecated getConfigurationSource() method in order to be able to remove it in the next major version

Check out the release on GitHub: KumuluzEE v4.1.0

KumuluzEE 4.0.0 released, with Java 17 LTS support, Jetty 10 and more!

Fri, 27 May 2022 00:00:00 +0000

We are happy to announce the next major release of KumuluzEE 4.0.0. This release includes a major Jetty upgrade, support for the latest released versions of Java, including the Java 17 LTS version and the use of JakartaEE 8 artifacts replacing the JavaEE 8 artifacts.

One of the major changes of this release is definitely the added support for the Java version 17. Java 17 is the latest LTS version, which are the most suitable versions for production use. Compared to the previous LST version (Java 11), Java 17 brings exciting new features such as records, text blocks, enhanced switch expressions, pattern matching of the instanceof operator, teeing collector for the Stream API and more.

Another exciting change in the new release of KumuluzEE is the transition from Jetty 9 to Jetty 10. The updated version of Jetty brings many important bugfixes and improvements. The Servlet API 4.0 is now fully implemented allowing better HTTP/2 support within internal components and general Servlet 4.0 usage according to the specification. The WebSocket implementation has been refactored, also allowing HTTP/2 usage. The Jetty HTTP client (used by internal components and our extensions, e.g. JAX-RS HTTP client, kumuluzee-rest-client’s backing HTTP client) has also been improved, with the major feature being the support for dynamic protocol upgrade to HTTP/2.

Another change in relation to Java versions is the end of support for the Java 1.8 runtime. The support for Java 1.8 was dropped in Jetty 10 and in turn we are also dropping support in KumuluzEE 4. If you are still running on Java 1.8, this is the perfect opportunity to upgrade, as the newer Java versions bring many exciting features.

Another transition we have made (also related to the Jetty 10 upgrade) is the transition from JavaEE 8 to JakartaEE 8 artifacts. This transition should not affect the actual code, only the locations and names of Maven artifacts. If you are referencing any JavaEE artifacts in your code, make sure to update the references there too.

The full list of breaking changes in this release:

KumuluzEE no longer supports Java 1.8 runtime, since the support for Java 1.8 has been dropped in Jetty 10.
Replaced JavaEE 8 artifacts with the JakartaEE 8 artifacts. If you are using JavaEE artifacts in your code, you should be able to find equivalent artifacts in the JakartaEE namespace.
Removed excludedAgentPatterns from gzip configuration since it was removed from Jetty.

See the full changelog below.

Features:

Upgraded from Jetty 9 to Jetty 10
Added support for Java 16, 17 and 18 and dropped support for Java 1.8
Use JakartaEE artifacts instead of JavaEE

Enhancements:

Upped dependency versions
Removed excludedAgentPatterns from gzip configuration since it was removed from Jetty

Bugs:

Updated jar loading to add support for multi-release jars when running in UberJar
Added missing spaces in dependency validation log messages
More robust exception handling when loading configuration file
Updated JAX-WS annotation processor in order to support some edge cases

Check out the release on GitHub: KumuluzEE v4.0.0

Announcing the first prerelease of KumuluzEE 4.0.0

Mon, 11 Apr 2022 00:00:00 +0000

We are happy to announce the first Beta of KumuluzEE 4.0.0. This is the first prerelease of the major release, which brings a major Jetty upgrade, support for the latest released versions of Java, including the Java 17 LTS version and the use of JakartaEE 8 artifacts replacing the JavaEE 8 artifacts.

The breaking changes in this release are:

KumuluzEE no longer supports Java 1.8 runtime, since the support for Java 1.8 has been dropped in Jetty 10.
Replaced JavaEE 8 artifacts with the JakartaEE 8 artifacts. If you are using JavaEE artifacts in your code, you should be able to find equivalent artifacts in the JakartaEE namespace.
Removed excludedAgentPatterns from gzip configuration since it was removed from Jetty.

See the full changelog below.

Features:

Upgraded from Jetty 9 to Jetty 10
Added support for Java 16, 17 and 18 and dropped support for Java 1.8
Use JakartaEE artifacts instead of JavaEE

Enhancements:

Upped dependency versions
Removed excludedAgentPatterns from gzip configuration since it was removed from Jetty

Bugs:

Updated jar loading to add support for multi-release jars when running in UberJar
Added missing spaces in dependency validation log messages
More robust exception handling when loading configuration file
Updated JAX-WS annotation procesor in order to support some edge cases

Check out the release on GitHub: KumuluzEE v4.0.0.beta.1

KumuluzEE Database Schema Migrations - Liquibase extension

Sat, 26 Feb 2022 00:00:00 +0000

In the rapidly growing world of automation, there is a constant need for application updates. Every change made to the application is nowadays version controlled, making it easy to automate deployment and, if necessary, roll back changes. Today, it is not uncommon to have multiple versions of an application deployed on the same day. Following this pattern, it quickly becomes clear that database schemas need to be managed in the same way.

KumuluzEE is pleased to announce the Liquibase extension for database schema migration. It allows you to migrate your schemas right at the application startup, so that you get the desired database state every time you run the application. Moreover, it also offers a possibility to drop the database before migration, making it a perfect choice for development and testing purposes. Not to be confused, Liquibase migrations via the Maven plugin are still infallible, especially in environments where we want to have full control.

To get started with the Liquibase extension, all you need to do is add the Liquibase database schema migration extension to your existing pom file.

<dependency>
    <groupId>com.kumuluz.ee.database-schema-migrations</groupId>
    <artifactId>kumuluzee-database-schema-migrations-liquibase</artifactId>
    <version>${kumuluzee-database-schema-migrations.version}</version>
</dependency>

Furthermore, the known KumuluzEE configuration file has to be adjusted.

kumuluzee:
  ...
  database-schema-migrations:
    liquibase:
      changelogs:
        - jndi-name: jdbc/example-db
          file: db/changelog-master.xml
          startup:
            drop-all: false           # default: false  
            update: true              # default: false

The above configuration instructs the Liquibase extension to update the current database schemas. The update is performed when the application is started. This involves making changes to a pre-configured data source with the appropriate JNDI name, which in our case is jdbc/example-db. What changes will take place is defined in a Liquibase changelog file (db/changelog-master.xml).

After we have everything set up, we can launch our application and see how it manages the migrations for us.

For more information and detailed walkthrough, we recommend you take a look at the KumuluzEE database schema migrations documentation on GitHub.

If you are not yet familiar with Liquibase, we also recommend you to give it a try.

KumuluzEE GraphQL 1.1.0 released!

Tue, 09 Feb 2021 00:00:00 +0000

We are excited to announce a new minor release - KumuluzEE GraphQL 1.1.0. This release includes a new MicroProfile compatible implementation, we have updated the GraphiQL (GraphQL UI) to support better schema exploration and fixed some bugs in our existing implementation.

To use the new MP-compatible implementation use the following dependency:

<dependency>
    <groupId>com.kumuluz.ee.graphql</groupId>
    <artifactId>kumuluzee-graphql-mp</artifactId>
    <version>${graphql.version}</version>
</dependency>

Note that the existing implementation (artifact kumuluzee-graphql) is still being updated. However, we advise new users to start with the MP implementation since this implementation is more likely to receive new features in the future.

GraphiQL now includes the explorer plugin, which enables you to interactively construct a query or mutation. This makes schema exploration and API usage even easier.

Integration with KumuluzEE REST has also been revamped to be more intuitive and future-proof. Sort and filter arguments are now string-based and limit/offset are now top level arguments. This makes queries more readable and easier to process. The QueryParameters object can be constructed and used with any KumuluzEE REST implementation (currently StreamUtils and JPAUtils).

The samples for MP-compatible implementation are already available:

Features:

Added MP-compatible implementation by SmallRye
Added GraphiQL Explorer plugin

Enhancements:

Converted GraphQL manual kumuluzee-rest processing to StreamUtils
Upped dependency versions

Bugs:

Fixed GraphiQL servlet mapping
Fixed pagination limit bug

Check out the release on GitHub: KumuluzEE GraphQL v1.1.0

Announcing a new minor release - KumuluzEE 3.12.0!

Wed, 13 Jan 2021 00:00:00 +0000

We are excited to announce a new KumuluzEE minor version - KumuluzEE 3.12.0! The most significant feature in this release is the support for Java 15. Dependencies have been updated to their latest minor versions, and a collection of tough bugs have been cracked. Check out the full changelog below:

Features:

Support for Java 15

Enhancements:

Use the new API for Jetty SSL configuration
Added GraphQL as an extension group
Upped dependency versions

Bugs:

Fixed integration between Agroal and Narayana for XA transactions
Fixed incorrect parsing of transaction-isolation configuration value
Fixed incorrect jUnit scope
Resolved dependency conflict in MP 3.3

Check out the release on GitHub: KumuluzEE v3.12.0

KumuluzEE OpenAPI MP and KumuluzEE Health new versions released

Fri, 13 Nov 2020 00:00:00 +0000

New versions of KumuluzEE OpenAPI MP and KumuluzEE Health were released today.

Kumuluz OpenAPI MP received an update to the Maven plugin which now works much faster and includes better integration with other KumuluzEE extensions.

KumuluzEE Health received update of the KafkaHealthCheck and now allows you to define any Kafka client configuration property in the KumuluzEE configuration framework. This allows you to fully control how the health check communicates with the cluster. Integration with the KumuluzEE OpenAPI MP has also been updated and now allows integration with OpenAPI Maven Plugin.

Check out the release notes for more information:

New releases of KumuluzEE OpenAPI MP and Health

Mon, 02 Nov 2020 00:00:00 +0000

We have just released new versions of KumuluzEE OpenAPI MP and KumuluzEE Health. New features include a Maven plugin which generates OpenAPI schema at compile-time, a new health check for Kafka clusters and integration between the two extensions.

KumuluzEE OpenAPI MicroProfile 1.3.0

This version of KumuluzEE OpenAPI MicroProfile brings a new Maven plugin which generates OpenAPI schema at compile-time and a new SPI interface which can be used to extend the generated schema.

To use the new Maven plugin add it to pom.xml:

<plugin>
    <artifactId>kumuluzee-openapi-mp-maven-plugin</artifactId>
    <groupId>com.kumuluz.ee.openapi</groupId>
    <version>${kumuluzee-openapi-mp.version}</version>
    <executions>
        <execution>
            <goals>
                <goal>generate</goal>
            </goals>
        </execution>
    </executions>
</plugin>

On next build the schema will be available in target/generated/openapi.yaml and target/generated/openapi.json. For more information about the maven plugin, see README.

The new SPI interface allows extension developers to extend the generated schema. To use it implement the com.kumuluz.ee.openapi.mp.spi.OasFilterProvider interface and register it with a service file.

KumuluzEE Health 2.3.0

This release of KumuluzEE Health brings a new health check for Kafka clusters, integration with KumuluzEE OpenAPI MP and a minor bugfix.

The new health check allows you to monitor the health status of your Kafka cluster. To use it add the kafka-clients dependency (already included in kumuluzee-streaming) and activate it in the configuration framework. For example in config.yml:

kumuluzee:
  health:
    checks:
      kafka-health-check:
        bootstrap-servers: localhost:9095,localhost:9096,localhost:9097
        minimum-available-nodes: 2
        request-timeout-ms: 1000

For more information see README.

We have integrated KumuluzEE Health with KumuluzEE OpenAPI MP extension. You can now add /health, /health/ready and /health/live endpoints to OpenAPI schema by activating the integration in the configuration framework. For example in config.yml:

kumuluzee:
  health:
    openapi-mp:
      enabled: true

We are excited to bring these two new releases into your hands!

KumuluzEE Kafka Streaming with Schema Registry

Mon, 15 Jun 2020 00:00:00 +0000

Apache Kafka is an excellent tool enabling asynchronous architecture in the modern microservice world. While Kafka offers very high-level abstractions in the form of Producer/Consumer APIs, Streams API, and Processor APIs, there is no out of the box support for managing the schema of records. Confluent Schema Registry is a tool solving this problem. This guide will show how you can use Schema Registry with kumuluzee-streaming and suggest a typical development flow for this setup.

In a way, the problem of defining and evolving a schema of your Kafka records is very similar to REST API and JSON schema evolution. Once you deploy a REST API to production, the clients have a basic expectation of JSON payloads not changing between minor API versions. Adding optional fields is usually considered a non-breaking change while renaming or removing required fields can break the clients and requires a major API version release (which implies a completely new API URL with eventual client migration).

In REST, you would usually use OpenAPI specification and diff tools to maintain compatibility between updates and use it as a contract between client and server. Similarly, if Kafka producer starts inserting incompatible records, these can break existing consumers which are not yet updated to use the new schema. One way to keep the record schemas compatible is by relying on developers to stick to a predefined set of rules of what changes are allowed in minor updates but since everyone makes mistakes, this might not be the most reliable option in the long term. The better way is using a Schema Registry, a tool that stores all our record schemas and does compatibility checks automatically every time we update a schema. This way we can always be sure that our schema changes are compatible. run This post assumes you are already familiar with regular Kafka tooling. For regular Kafka usage with kumuluzee-streaming extension, check out kumuluzee-streaming-kafka sample on GitHub. Full code sample used in this blog post can be found at kumuluzee-streaming-kafka-registry.

Additional tooling and dependencies

In order to use Schema Registry, we need additional maven dependencies and Docker containers in our projects.

Docker

You can find a sample docker-compose.yaml in the repository. It uses:

confluentinc/cp-zookeeper Official Confluent Zookeeper container.
confluentinc/cp-kafka Official Confluent Kafka container.
confluentinc/cp-schema-registry Official Confluent Schema Registry container for managing our schemas.
landoop/schema-registry-ui 3rd party UI to view (and potentially manage) the Schema Registry.
obsidiandynamics/kafdrop Kafdrop is one of the best Kafka UI tools and one of the only ones allowing us to view actual records in the UI.

The #4 and #5 are optional but good for visualization of this tutorial.

To start all the containers, simply run docker-compose up -d in the root folder.

Maven Dependencies

In addition to kumuluzee-streaming extension dependency, we need to include Avro for the handling of serialization and deserialization from POJO to Avro data format. Avro is the primary format used by Schema Registry to define schemas, other available formats are protobufs and JSON Schema.

<dependency>
    <groupId>org.apache.avro</groupId>
    <artifactId>avro</artifactId>
</dependency>

We also need Kafka Avro Serializers/Deserializers to plug into Kafka Producers/Consumers:

<dependency>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-avro-serializer</artifactId>
</dependency>

For Streams, we need Avro Serde dependency:

<dependency>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-streams-avro-serde</artifactId>
</dependency>

To generate Java POJOs from our Avro schema files, we need avro-maven-plugin:

<plugin>
    <groupId>org.apache.avro</groupId>
    <artifactId>avro-maven-plugin</artifactId>
    <executions>
        <execution>
            <phase>generate-sources</phase>
            <goals>
                <goal>schema</goal>
            </goals>
            <configuration>
                <sourceDirectory>...</sourceDirectory>
                <outputDirectory>...</outputDirectory>
            </configuration>
        </execution>
    </executions>
</plugin>

Finally, to register and update local schemas from and to registry, we need confluent-schema-registry-maven-plugin:

<plugin>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-schema-registry-maven-plugin</artifactId>
    <configuration>
        <schemaTypes>

        </schemaTypes>
        <subjects>
        </subjects>
        <outputDirectory>...</outputDirectory>
    </configuration>
    <goals>
        <goal>test-compatibility</goal>
    </goals>
</plugin>

Additional Maven Repositories

Confluent artefacts are only available from their own repository so we need to add it either in m2 configuration or root pom.xml.

<pluginRepositories>
    <pluginRepository>
        <id>confluent</id>
        <name>Confluent</name>
        <url>http://packages.confluent.io/maven/</url>
    </pluginRepository>
</pluginRepositories>

<repositories>
    <repository>
        <id>confluent</id>
        <name>Confluent</name>
        <url>http://packages.confluent.io/maven/</url>
    </repository>
</repositories>

Producers

Producers are the services inserting records into the topics so we start with them. Typical Producer development cycle consists of:

Adding the required maven dependencies to our project.
Writing a schema for our producing record in a .avsc file using the Avro Schema syntax.
Generating a Java POJO from the schema using the Avro Maven Plugin. This is done at compile-time with mvn compile.
Uploading the schema to the schema registry with Confluent Schema Registry Maven Plugin.

Writing our record schema

First we create a new schema file in src/main/resources/schemas/avro/v1.0.0/pricing.avsc. The exact path you decide to store your schemas is up to your personal preferences. We recommend using versioning in the file path to preserve the full history of schema evolution when new versions are added. You should configure your Avro and Confluent Schema Registry plugin paths accordingly.

An example of schema definition:

{
  "namespace": "com.kumuluz.ee.samples.kafka.registry.avro.lib",
  "type": "record",
  "name": "Pricing",
  "fields": [
    {"name": "price", "type": "string"},
    {"name": "priceExTax", "type": "string"},
    {"name": "basePrice", "type": "string"},
    {"name": "taxAmount", "type": "string"},
    {"name": "description", "type": "string"}
  ]
}

Now that we have our schema, a mvn compile will generate the Java class in src/generated/java using the namespace from the schema as package name. Since these classes are generated at compile time, we use src/generated output folder as opposed to src/main for hand-written code. The sample project has properly configured maven-compiler-plugin so IDEs such as IntelliJ properly mark the folder as generated sources folder.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
        <generatedSourcesDirectory>src/generated/java</generatedSourcesDirectory>
    </configuration>
</plugin>

Another option would be to generate the classes in target/generated-sources.

Producer code

We can use the generated Pricing class in our Kafka producer interface. For this example, we’ve put the producer in a JAX-RS resource to produce a record with an API call.

@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
@Path("/produce")
@RequestScoped
public class ProducerResource {

  @Inject
  @StreamProducer(config = "producer-avro")
  private Producer producerAvro;

  @POST
  public Response produceMessage(Input msg) {

      final Pricing pricing = new Pricing(msg.getPrice(), msg.getPriceExTax(), msg.getPrice(),
          msg.getTaxAmount(), msg.getDescription());
      final ProducerRecord<String, Pricing> record =
          new ProducerRecord<>("pricing-avro", UUID.randomUUID().toString(), pricing);

      producerAvro.send(record);

      return Response.ok().build();
  }
}

The code is essentially the same as if we did not use Schema Registry at all. The difference is in Producer configuration where we specify the Avro serializer and URL to the Schema Registry:

kumuluzee:
  streaming:
    kafka:
      producer-avro:
        bootstrap-servers: localhost:29092
        key-serializer: org.apache.kafka.common.serialization.StringSerializer
        value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
        schema-registry-url: http://localhost:8081
        auto-register-schemas: false

We also set the property auto-register-schemas to false, so the schemas are not published automatically with code deployment. Instead, we register schemas manually with the maven plugin. In the producer module run:

mvn schema-registry:register

We can verify that schema is registered by opening the Schema Registry UI at http://localhost:8000.

Producing a record

Run the producer service and produce a record with a POST request:

curl --header "Content-Type: application/json" \
  --request POST \
  --data '{"price":"0.00","priceExTax":"0.00","taxAmount":"0.00","basePrice":"0.00","priceAmount":"0.00","description":"Desc"}' \
  http://localhost:8080/produce

We can verify that a record was produced according to our schema by checking the pricing-avro topic in Kafdrop UI at http://localhost:9000.

Kafdrop has Schema Registry support so it is capable of showing us human-readable record data even though the actual data exists in Avro binary format. Regular Kafka CLI tooling does not have this capability, we would need to use the extended Confluent CLI tools to view human-readable record data from the CLI.

Evolving a schema

There are different types of compatibilities we can set in the Schema Registry depending on our needs. By default, BACKWARDS rule is used. You can see all the available types in this compatibility documentation. After we write an evolved schema and try to register it, Schema Registry will automatically reject the update if the changes do not satisfy the compatibility rule. We also have a dedicated command to check the compatibility without registration:

mvn schema-registry:test-compatibility

Consumer

Typical developer flow for the consumer is:

Download the schema from Schema Registry to a local .avsc file.
Generate the Java POJO.
Use the class in the consumer interface.

By downloading the schema from the registry, we can use the registry as a source of truth and a central repository of our schemas.

Consumer code

Download the schema with mvn schema-registry:download followed by POJO generation with mvn compile inside the consumer module. Now we can use the generated class in our consumer interface:

@StreamListener(topics = {"pricing-avro"}, config = "consumer-avro")
public void onMessage(ConsumerRecord<String, Pricing> record) {

    log.info(String.format("Consumed message: offset = %d, key = %s, value = %s%n", record.offset(), record.key()
            , record.value().toString()));

    messages.add(record.value().toString());
}

Again, the code is essentially the same as if we didn’t use Schema Registry, the difference is in configuration:

kumuluzee:
  streaming:
    kafka:
      consumer-avro:
        bootstrap-servers: localhost:29092
        group-id: group1
        enable-auto-commit: true
        auto-commit-interval-ms: 1000
        auto-offset-reset: latest
        key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
        value-deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
        schema-registry-url: http://localhost:8081
        specific-avro-reader: true

We must specify the Avro deserializer, Schema Registry URL and set specific-avro-reader to true. The default behaviour of Avro deserializer is to deserialize to generic records instead of the specified class which is not what we want.

Consuming the record

Run the consumer service (on a different port than producer) and produce another record with the producer service. You should see the record being received and logged on the consumer side.

Evolving the schema

Let’s assume we use a FORWARD strategy to evolve our schemas. Typical evolved deployment would then look like this:

Update the schema on the producer side and deploy it to production.
Pull down the new schema on the consumer side and rebuild the consumer, optionally develop new features that use the added fields.
Deploy consumer to production.

For a period of time, we have new producer(s) and old consumer(s) deployed at the same time. Other than consumers not taking advantage of the new schema, consumers run normally and don’t break when reading new records.

En example of adding a “v2.0.0” schema is shown in the full example.

Streams

Streams API allows us to consume, transform, and produce a record at the same time, perform aggregations, and much more.

In the following example, we will calculate the sum of all records in our pricing-avro queue. While it does not make any sense from logical perspective, just imagine these are order amounts and we are calculating a total of all orders instead. The total is sent into sum-avro topic each time it is recalculated.

Streams schema

Since our streams application consumes from one topic and produces to another topic, we need to download and compile the schema for the consumer side and write the schema for the producer side (sum-avro topic).

Getting the Pricing.java class:

mvn schema-registry:download
mvn compile

Our Sum schema has a single field:

{
  "namespace": "com.kumuluz.ee.samples.kafka.registry.avro.lib",
  "type": "record",
  "name": "Sum",
  "fields": [
    {"name": "sum", "type": "string"}
  ]
}

Streams code

@Inject
private GenericConfig config;

@StreamProcessor(id="price-sum", autoStart=false, config="streams-avro")
public StreamsBuilder sum() {

    StreamsBuilder builder = new StreamsBuilder();

    final Serde<String> keySerde = Serdes.String(); //string Serde for our UUID record key
    final Serde<BigDecimal> bigDecimalSerde = new BigDecimalSerde(); //BigDecimal Serde for price sum, custom implementation since it is not provided by Kafka out of the box
    final Serde<Pricing> consumeValueSerde = new SpecificAvroSerde<>(); //consumer of Avro format
    final Serde<Sum> produceValueSerde = new SpecificAvroSerde<>(); //producer of Avro format

    // config.yml only configures the default SerDes. All non-default SerDes need to be
    // configured explicitely here.
    final Map<String, String> serdeConfig = Collections.singletonMap("schema.registry.url",
        config.getSchemaRegistryUrl());
    consumeValueSerde.configure(serdeConfig, false);
    produceValueSerde.configure(serdeConfig, false);

    final String inputTopic = "pricing-avro";
    final String outputTopic = "sum-avro";

    builder.stream(inputTopic, Consumed.with(keySerde, consumeValueSerde)) //consume with Avro Serde<Pricing>
        .map((k, v) -> new KeyValue<>("total-sum", new BigDecimal(v.getPrice().toString()))) //map Pricing object to single BigDecimal value
        .groupByKey(Grouped.with(keySerde, bigDecimalSerde)) //group by static key and reduce
        .reduce(BigDecimal::add)
        .toStream()
        .map((k, v) -> new KeyValue<>(k, new Sum(v.toPlainString()))) //map to our Sum schema
        .to(outputTopic, Produced.with(keySerde, produceValueSerde)); //produce to sum-avro

    return builder;
}

Streams configuration:

kumuluzee:
  streaming:
    kafka:
      streams-avro:
        bootstrap-servers: localhost:29092
        application-id: sample-price-sum
        default-key-serde: org.apache.kafka.common.serialization.Serdes$StringSerde
        default-value-serde: org.apache.kafka.common.serialization.Serdes$StringSerde
        commit-interval-ms: 500
        schema-registry-url: http://localhost:8081
        specific-avro-reader: true
        auto-register-schemas: false

Since Serde for streams is dynamically configured in code, we can leave the defaults as StringSerde. Other properties have already been explained in the Consumer and Producer chapters.

Testing the Streams application

After running the Streams application, produce a value with price attribute > 0. For example:

curl --header "Content-Type: application/json" \
  --request POST \
  --data '{"price":"15.00","priceExTax":"0.00","taxAmount":"0.00","basePrice":"0.00","priceAmount":"0.00","description":"Desc"}' \
  http://localhost:8080/produce

In Kafdrop, you should see a new total sum being produced in the sum-avro topic each time a new record is created in the pricing-avro.

Future work - JSON Schema

Confluent has added support for JSON Schema as an alternative to Avro format in early 2020. Performance considerations aside, we see some benefits of using JSON Schema over Avro:

A lot of developers are already familiar with writing schemas for OpenAPI. JSON Schema 2019-09 and OpenAPI have converged and now have a very similar syntax and set of features so the developer only needs to learn one syntax for both REST and Kafka worlds.
The data format is JSON, no need for custom consumer tools for readable records.

When testing JSON Schema support, we determined that the feature is not quite ready yet. There is no support for draft 2019-09 yet. Generated POJOs need additional annotations to link the class with the source schema. This fix is not yet released at the time of writing. There is also no support for writing JSON Schema in yaml format (which is an often-used approach in OpenAPI world). While not a deal breaker, it needs some extra tooling in order to do the conversions on the fly.

We are eagerly monitoring new releases of Schema Registry to present this guide using JSON Schema in the future.

Conclusion

Schema Registry brings a lot of new, relatively complex tooling and new workflow in developing Kafka applications for the benefits of better schema management and a compatibility safety net.

Introducing new KumuluzEE releases

Thu, 11 Jun 2020 00:00:00 +0000

We are announcing new releases in the KumuluzEE family. The new release of the core microservice framework KumuluzEE 3.10.0 brings support for Java 14 and a set of new features and enhancements. Supporting projects, including Config, Security, Rest Client, CORS, REST and JWT Auth, have also been updated.

The details about each individual release can be found in the corresponding release notes:

Introducing KumuluzEE 3.9.0 with MicroProfile 3.3

Wed, 08 Apr 2020 00:00:00 +0000

We are happy to announce the release of KumuluzEE 3.9.0, which brings support for MicroProfile 3.3.

The new MicroProfile version brings updates to the following modules:

Check out the release notes of individual modules for the description of new features.

KumuluzEE Logs 1.4.0 released

Tue, 31 Mar 2020 00:00:00 +0000

We’re announcing the availability of KumuluzEE Logs 1.4.0, which introduces two big features; support for Fluentd logging provider and an audit logging module.

Features

Fluentd is now supported as a logging provider.
An audit logging module provides features for logging the audit trail using the supported logging frameworks.
Logging context can now be added to logs by implementing the RequestContext interface.