If you're brand new to Kafka, be sure to check out this <a id=0 href='https://www.stackchief.com/blog/Kafka%20Architecture%20Explained'>intro to Kafka architecture</a> first.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter-parent</artifactId>

<description>Demo project for Spring Boot</description>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter</artifactId>

<groupId>org.springframework.kafka</groupId>

<groupId>com.fasterxml.jackson.core</groupId>

<artifactId>jackson-databind</artifactId>

<artifactId>spring-boot-starter-test</artifactId>

<artifactId>spring-kafka-test</artifactId>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-maven-plugin</artifactId>

The pom.xml defines the dependencies we need for our Maven build. Notice how we include the dependencies needed for Spring Boot along with those needed for running and testing Kafka.

Note that the jackson-databind dependency is included to account for bugs in older versions of the Spring Kafka library. You may not need to include this depending on which version of Kafka you're using..

	public static void main(String[] args) {

		SpringApplication.run(DemoApplication.class, args);

The DemoApplication.java class runs our Spring Boot application.

Spring Boot automatically configures Kafka via @SpringBootApplication. With this annotation, Spring provides/configures a KafkaAutoConfiguration class for you.

import org.springframework.kafka.annotation.KafkaListener;

import org.springframework.stereotype.Component;

    public void processMessage(String content){

        System.out.println("Message received: " + content);

The Consumer.java class is what actually listens for messages. Notice how we specify the topics to listen to within the @KafkaListener annotation.

By simply using the annotations provided by the Spring Boot Kafka library, we are able to easily set up a consumer for a given topic.

spring.kafka.bootstrap-servers=localhost:9092

Thanks to auto-configuration, you only need to define a few things in application.properties to configure the consumer. Notice how our app is configured to use Kafka locally.

This example is configured to consume from a local Kafka topic.  You'll need to have Kafka installed and running locally to see messages being consumed from the Spring Boot app.

To start Kafka locally, you need to start both the Zookeeper and Kafka servers locally...

./zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties

This will start the zookeeper server with the default configurations.  Please note the path to your local Kafka install may vary.

./kafka-server-start /usr/local/etc/kafka/server.properties 

This will start the Kafka server with the default configurations.  Please note the path to your local Kafka install may vary.

To test the application consumer, you'll need to produce some messages for the test topic.  To start a Kafka console producer:

./kafka-console-producer --bootstrap-server=localhost:9092 --topic test

This starts a Kafka producer for the test topic. Notice how both the bootstrap-server and topic arguments match what we've defined in our application.properties file.

Once the producer is up and running, you can send a message to the test topic like this:

If everything is working correctly, you should see the message being logged through the Spring application like this:

Spring Boot Kafka Consumer example including consumer offsets and multiple consumer examples.

Spring Boot Kafka Consumer

In a world of low code frameworks, open source AI tools, and an obsession with automation, what the future demand of software engineering looks like in 2023.

ChatGPT and the Future of Software Engineers

Is Kafka a programming language?

How much do software engineers make?

How do you list indexes in MongoDb?

jstat and jconsole will help you monitor performance of GC. with jstat i know you can see things like total cycles run, survivor space capacity and usage, eden space capacity and usage, old gen capacity and usage etc.

How can you monitor garbage collection in Java

How can you monitor garbage collection in Java?

Understanding Java garbage collection begins with the heap and the stack. If you aren't familiar with <a id=1 href='https://www.stackchief.com/blog/Java%20Heap%20vs%20Stack'>Java Heap vs Stack</a> be sure to read <a id=1 href='https://www.stackchief.com/blog/Java%20Heap%20vs%20Stack'>this</a> first.

Garbage collection is the automatic process for handling memory management.  Memory management is the dynamic process of allocating/deallocating memory to/from your application.

Memory management is not unique to Java.  Every programming language must provide a means of requesting memory from the underlying OS and releasing memory when it's no longer needed. This allows programs to run without consuming too much memory or causing applications to crash.

Some languages like C and C++ require a manual process for allocating/deallocating memory.  Programmers must write constructor and destructor functions to allocate memory when it's needed and release it when objects are no longer being referenced by the application.

While this manual process gives developers more control over memory management, it's ultimately more tedious, repetitive, and vulnerable to memory leaks and other errors.  For these reasons, programming languages like Java implement an automatic garbage collection mechanism for managing memory.  This mechanism operates independently from the application code allowing developers to write code without worrying about memory management.

Why is garbage collection important in Java?

Garbage collection (GC) is important because it automatically handles memory management in Java.  This makes Java memory-efficient without the developer's intervention. As a developer, you can write Java code without worrying about memory allocation.  GC handles all of this for you.

While GC "magically" handles memory management for the developer, it's still important to understand what's going on under the hood. If you've ever experienced a OutOfMemory exception then you know GC won't save you every time. Furthermore, there are different implementations of GC and various configurations for optimizing this process based on the nature of your application.

Having a good grasp on the GC mechanism will help you write better code and avoid performance issues related to memory management.

Java applications run inside the JVM. The OS running your Java program allocates a configurable amount of <a href='https://www.stackchief.com/blog/Java%20Heap%20vs%20Stack'>heap space</a> memory to the JVM for running your applications.

The above are two sample arguments for configuring min and max <a href='https://www.stackchief.com/blog/Java%20Heap%20vs%20Stack'>heap space</a> for the JVM.

Since the JVM has a predetermined allotment of memory for running applications, it doesn't need to synchronize with the underlying OS for memory resources.  Instead, it can utilize this same memory space and free up unused memory without deleting or requesting more from the OS.  This is fundamental to what makes Java memory-efficient as a programming language.

Application execution: Java Heap vs Stack

When your application starts up, classes and methods are loaded into the <a href='https://www.stackchief.com/blog/Java%20Heap%20vs%20Stack'>heap space</a>. As your program executes, objects are created and referenced in the <a href='https://www.stackchief.com/blog/Java%20Heap%20vs%20Stack'>heap space</a> by the <a href='https://www.stackchief.com/blog/Java%20Heap%20vs%20Stack'>stack</a>

While the <a href='https://www.stackchief.com/blog/Java%20Heap%20vs%20Stack'>heap</a> represents a global space of memory that exists throughout the duration of your application, the <a href='https://www.stackchief.com/blog/Java%20Heap%20vs%20Stack'>stack</a> only exists during method or thread execution. Once a method finishes execution, the memory used is released from the stack via a relatively simple FIFO approach.

This simple approach is sufficient for automatically handling memory living in the stack (local variables and references to objects stored in the heap) but what about actual objects referenced in the heap?

Since the heap space is much larger and more permanent, it must be managed through more sophisticated means.  This is where GC comes into play.  GC automatically handles the memory used by the heap space.

Confused? I highly recommend checking out <a href='https://www.stackchief.com/blog/Java%20Heap%20vs%20Stack'>Java Heap vs Stack</a> before continuing any further...

With the understanding that memory management for the heap is automatically handled via GC, it's time to understand exactly how GC works in Java.

GC works by identifying or "marking" unused objects by the application.  An unused object is anything that isn't being referenced by a GC root.  A GC root is either:

1) Objects referenced in the <a href='https://www.stackchief.com/blog/Java%20Heap%20vs%20Stack'>stack</a>

2) Objects referenced by class static attributes

These special "root" objects give GC a starting point to search for "living" objects.  Any object that isn't referenced by one of these root objects is considered "unused".  Take the following code snippet as an example:

    public static final MyClass myC = MyClass("final");

While setting c = null will dereference c, it's static member myC won't be garbage collected after testMyClass() executes.

Since the instance of MyClass c is created within the <a href='https://www.stackchief.com/blog/Java%20Heap%20vs%20Stack'>VM stack</a>, it will be garbage collected.

A combination of the following GC algorithms are used by modern GC collectors in Java:

GC algorithms follow a high-level "mark-sweep" phase concept.  In the "mark phase", GC identifies objects that are no longer being referenced by the application.  In the "sweep phase", the memory addresses used for these unreferenced objects are freed up by the JVM.  This essentially "deletes" the unreferenced objects and makes memory available to the heap.

While the "mark-sweep" algorithm is fairly simple, it presents problems related to fragmentation.  When memory is released, it becomes available in many fragments or segments.  Since memory is allocated in contiguous blocks, this makes it harder to assign address space and wastes unused memory.

Unlike the mark-sweep algorithm, the copy algorithm avoids memory fragmentation by copying all of the "unmarked" living objects to a new space.  This is more efficient as living memory can exist in a contiguous block.  The downside is memory is less efficiently used as memory space must be reserved as a "destination" for copied data.

This is a hybrid of the traditional mark-sweep and copy algorithm.  Mark-compact identifies unused objects similar to mark-sweep.  Unlike mark-sweep, mark-compact moves living objects to one end of boundary and reclaims the leftover space.

This is similar to the copy algorithm in that living memory is isolated from unused memory.  Unlike the copy algorithm, mark-compact moves memory instead of copying it.

While mark-compact provides a nice hybrid approach to mark-sweep and copy, it takes a performance hit with frequent manipulation and sorting of the memory space.  For these reasons, generations and regions are used with GC.

GC algorithms have their pros and cons.  While "mark sweep" is simple, it introduces fragmentation.  While copy eliminates fragmentation, it wastes memory.  While mark-compact serves as a nice hybrid, it can be inefficient.

To leverage these tradeoffs, heap space is divided into blocks called "generations".  These blocks categorize objects based on their age or how long they've been in the heap.

By categorizing objects as "young" vs "old", GC can run different algorithms best suited for the lifespan of memory...

The young generation of the heap holds all newly created objects. This space is further divided into the Eden and Survivor space:

Empirical data and research shows that 98% of objects created are short lived. For these reasons, most newly created objects are allocated to the Eden space. When the eden space becomes full, minor gc takes place. During a minor GC event, objects surviving the eden space are moved to the survivor space.

The survivor space itself exists in two spaces (S0, S1).  This is sometimes referred to as the "to" and "from" space but the important thing is that two separate spaces exist within the survivor space.

When a minor GC occurs, surviving objects from the eden space are moved to the S0 space.  The next time GC occurs, surviving objects from eden and S0 are copied into S1.  The next time GC occurs, surviving objects now living in S1 are copied (along with Eden objects) to the S0 space.

Why is this so complicated? Why can't the eden space graduate objects to the old generation if they survive GC?  It turns out the survivor space(s) act as a buffer between young and old generation.  While many objects may survive a single "minor GC event", they won't likely last more than a few cycles.  The survivor space gives objects the "opportunity" to die as it collects anything surviving up to 16 GC cycles.

Remember the copy algorithm?  By having two spaces in the survivor space, one space can be used as a destination for everything copied over from the eden and other survivor space.  This eliminates the memory fragmentation issue and works well for more short-lived objects found in the younger generation.

If an object survives enough rounds of minor GC, it will be promoted to the old generation.  Major garbage collection events are performed on the old generation.

Major GC happens less often and adopts a "mark-sweep" algorithm for cleanup.  This is better for the older generation space as most objects are long lived.  This minimizes the number of objects needing to be "marked" and resorted as per the "mark-sweep" approach.

If an object survives enough GC events, it will eventually be promoted to the old generation space.  Other objects, such as very large objects, will automatically be promoted to the old generation space.  Objects that can't fit within the eden/survivor space will be promoted to the old generation.

Older implementations of GC represent the heap space as one contiguous block of memory.  More modern implementations used with Java 8, 11+ implement a "garbage first" (G1) approach to GC.

With the G1 approach, the heap space is divided into equally sized heap regions.  Each region can represent a different generation at any given time.  While a region may sometimes represent a young generation (eden, survivor space), it may represent an old generation after being reallocated.

By subdividing the heap space into regions, multiple threads can scan regions in parallel.  Generations aren't confined to a predetermined size and can dynamically change (young space vs old space) based on the algorithm's needs.

Regions also work to identify spaces with the most garbage (fewest live objects) during global "mark" phases.  These regions are "evacuated" by copying any surviving objects to a new single region.  This compacts and frees up memory in parallel ultimately leading to faster throughput and smaller pause times.

Through the use of regions, you can see how the G1 collector performs differently than alternatives.  So what are these alternatives?

Serial GC: This is the most basic implementation of GC. Utilizing a single thread, the application is paused and GC is performed. Once GC has run and compaction is complete, the application thread resumes.

Serial GC is the most basic form of GC and is not recommended for use in environments where low latency is emphasized. You can use serial GC by specifying -XX:+UseSerialGC.

Parallel GC: Also known as "throughput collector", parallel GC works by utilizing multiple threads to perform young generation collection. A single thread is used for major GC in the old generation.

You can specify parallel GC with the -XX:+UseParallelGC option.

Parallel Old GC: Very similar to Parallel GC except that multiple threads are used across young and old generations. You can specify use of Parallel Old GC with the -XX:+UseParallelOldGC.

Concurrent Mark Sweep (CMS) GC: Unlike the already mentioned alternatives, CMS works concurrently with the application execution. Multiple threads work alongside application threads to minimize the pause time in between GC cycles.

While CMS sounds great, it only pays off if you have the CPU power to throw at it.  CMS requires more CPU and no compaction is performed with this option.

You can enable CMS with the -XX:+UseConcMarkSweepGC option (deprecated)..

G1 (Garbage first): This is the default GC used in modern Java programs. Since G1 prioritizes GC on regions having the most garbage (fewest live objects), it's considered "garbage first".

Epsilon: This GC was released as an experimental feature of JDK 11. The reason for using Epsilon is essentially to not use a memory reclamation mechanism at all. Since memory reclamation can be a costly to performance, the Epsilon GC is meant for hyper low latency applications where memory is being handled well by the developer.

You can enable Epsilon with the -XX:+UnlockExperimentalVMOptions -XX:+UseEpsilonGC.

Shenandoah: Shenandoah is relatively new (released as part of JDK 12). It's basically the newer/more improved version of today's G1 collector and can evacuate its heap concurrently with the application. While more CPU intensive than G1, Shenandoah can be thought of as a "better" version of G1.

You can enable Shenandoah with the -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGCM/span> option.

ZGC: Released with 11 and improved with 12, ZGC allows the application to continue running while performing all GC operations. ZGC prides itself on the lowest pause times and points to the future of GC in Java.

You can enable ZGC via the -XX:+UnlockExperimentalVMOptions -XX:+UseZGC option.

GC is performed automatically in Java whenever the Eden space becomes full. If there isn't enough free space around to allocate an object, GC will also be performed.

Minor GC events occur every time the Eden space reaches capacity.  A major GC event occurs whenever younger generations are full or when the older generation becomes full.

Major GC also occurs whenever you force garbage collection programatically....

Garbage collection in Java including understanding what is garbage collection, why garbage collection, how garbage collection works in Java, force garbage collection, monitoring garbage collection and interview questions.

Garbage Collection in Java

GC works automatically and isn't worth looking into for most novice developers.  Remember that Java is a robust language with years of support and testing.  The chances of you implementing GC better than the "out of the box" solutions is small.

This doesn't mean you shouldn't explore it and understand how it works.

Explain the garbage collection mechanism in Java

Explain the garbage collection mechanism in Java?

Java 11 remains the best option for reliable production deploys in 2021...

Why? It offers long term support and is latest version with LTS :)

Well. My company standard it's openJDK 8 since 2019... Cheers!

juanrios.com@localhost

thank you, I finally realize that Java 8 is the most popular version.

Evgeny Ivanov

Alexander Joseph

A discussion surrounding the different versions of Java including the difference between SE and EE as well as key features of Java 7, 8, and 9.

Which Version of Java Should You Use?

I was originally in finance and made the career change about 5 years ago. I have a formal education in business (undergraduate) and enrolled in a specialty school for additional web development training. 

the truth of the matter is programming these days is like learning a language. If you take the time to do it you can get a good paying job with just a few years of disciplined effort. 

the reason why so many people never become engineers is because of how boring and mundane modern day programming can be. It takes a really special type to enjoy sitting around all day making configuration changes but they exist. And I am one of them :) 

How long does it take to become a software engineer

UdemyAlts

Spring Boot Kafka Consumer

ChatGPT and the Future of Software Engineers

Is Kafka a programming language?

How much do software engineers make?

How do you list indexes in MongoDb?

How can you monitor garbage collection in Java?

Garbage Collection in Java

Explain the garbage collection mechanism in Java?

Which Version of Java Should You Use?

How long does it take to become a software engineer?

UdemyAlts

Spring Boot Kafka Consumer

ChatGPT and the Future of Software Engineers

Is Kafka a programming language?

How much do software engineers make?

How do you list indexes in MongoDb?

How can you monitor garbage collection in Java?

Garbage Collection in Java

Explain the garbage collection mechanism in Java?

Which Version of Java Should You Use?

How long does it take to become a software engineer?

Trending