Mirror Maker

ZooKeeper Basics
● Open Source Apache Project
● Distributed Key Value Store
● Maintains configuration information
● Stores ACLs and Secrets
● Enables highly reliable distributed coordination
● Provides distributed synchronization
● Three or five servers form an ensemble
Brokers Manage Partitions
● Messages of Topic spread across Partitions
● Partitions spread across Brokers
● Each Broker handles many Partitions
● Each Partition stored on Broker’s disk
● Partition: 1..n log files
● Each message in Log identified by Offset
● Configurable Retention Policy
Questions:
● Why do we need an odd number of ZooKeeper nodes?
● How many Kafka brokers can a cluster maximally have?
● How many Kafka brokers do you minimally need for high
availability?
● What is the criteria that two or more consumers form a
consumer group?
Zookeeper
◦ Zookeeper manages the broker and keep the list of them
◦ Zookeeper does the leader election for partition
◦ Zookeeper notifies Kafka in case of
◦ New Topic
◦ Deletion of topic
◦ New Broker
◦ Broker Dies

Demo using command line

Assuming Zookeeper & Kafka is installed, Also kafka is started on two broker (9093/94
ports)

• Create topic with 2 partition using replication factor as 1

• sh [Link] --zookeeper [Link]:2181 --create --topic first_topic --
partitions 2 --replication-factor 1

• To display topic details “sh [Link] --zookeeper [Link]:2181 –list”

• Produce the message on topic created

• sh [Link] --broker-list [Link]:9093,[Link]:9094 --topic
first_topic

• Consume the message on topic created

• sh [Link] --bootstrap-server [Link]:9093,[Link]:9094
--topic first_topic --from-beginning

• Enter couple of message on producer console and see

corresponding message getting displayed on consumer
console.
• Consumer group – If you want to test consumer group concept, execute below
command on two different terminal. You will notice message are getting
consumed as per partitions
• sh [Link] --bootstrap-server [Link]:9093,[Link]:9094
--topic first_topic --consumer-property
[Link]=mygroup1 --from-beginning
Producer Role
The primary role of a Kafka producer is to take producer properties & record as inputs and write
it to an appropriate Kafka broker. Producers serialize, partitions, compresses and load balances
data across brokers based on partitions.

Properties

Some of the producer properties are bootstrap servers, acks, [Link], [Link] [Link],
[Link] and many more. We will discuss some of these properties later in this article.
Producer record

A message that should be written to Kafka is referred to as Producer Record. A producer record
should have the name of the topic it should be written to and value of the record. Other fields like
partition, timestamp and key are optional.

Broker and metadata discovery

Bootstrap server

Any broker in Kafka cluster can act as a bootstrap server. Generally, a list of bootstrap servers is
passed instead of just one server. At least 2 bootstrap servers are recommended.
In order to send producer record to an appropriate broker, the producer first establishes a
connection to one of the bootstrap server. The bootstrap-server returns list of all the brokers
available in the clusters and all the metadata details like topics, partitions, replication factor
and so on. Based on the list of brokers and metadata details the producer identifies the leader
broker that hosts the leader partition of the producer record and writes to the broker.

Workflow
The diagram below shows the workflow of a producer.
The workflow of a producer involves five important steps:

1. Serialize
2. Partition
3. Compress
4. Accumulate records
5. Group by broker and send

Serialize

In this step, the producer record gets serialized based on the serializers passed to the
producer. Both key and value are serialized based on the serializer passed. Some of the
serializers include string serializer, byteArray serializer and ByteBuffer serializers.

Partition

In this step, the producer decides which partition of the topic the record should get written to. By
default murmur2 algorithm is used for partitioning. Murmur 2 algorithm generates a unique
hash code based on the Key passed and the appropriate partition is decided. In case the key is not
passed the partitions are chosen in a round-robin fashion.

It’s important to understand that by passing the same key to a set of records, Kafka will ensure
that messages are written to the same partition in the order received for a given number of
partitions. If you want to retain the order of messages received it’s important to use an
appropriate key for the messages. Custom partitioner can also be passed to the producer to
control which partitions message should be written to.

Compression

In this step producer record is compressed before it’s written to the record accumulator. By
default, compression is not enabled in Kafka producer. Below are supported compression types:
Compression enables faster transfer not only from producer to broker but also during replication.
Compression helps better throughput, low latency, and better disk utilization. Refer
[Link] for benchmark
details.

Record accumulator

In this step, the records are accumulated in a buffer per partition of a topic. Records are grouped
into batches based on producer batch size property. Each partition in a topic gets a separate
accumulator/buffer.
Sender thread

In this step, the batches of the partition in record accumulator are grouped by the broker to which
they are to be sent. The records in the batch are sent to a broker based on [Link] and
[Link] properties. The records are sent by the producer based on two conditions. When the
defined batch size is reached or defined linger time is reached.

Duplicate message detection

Producers may send a duplicate message when a message was committed by Kafka but the
acknowledgment was never received by the producer due to network failure and other issues.
From Kafka 0.11 to avoid duplicate messages in case of scenario stated earlier Kafka tracks each
message based on producer ID and sequence number. When a duplicate message is received for a
committed message with same producer ID and sequence number then Kafka would treat the
message as a duplicate message and will not committee message again but it will send the
acknowledgment back to the producer so the producer can treat the message as sent.

Few other producer properties

 [Link] – manage buffer memory allocated to producer
 Retries - Number of times to retry message. Default is 0. The retry may cause out of
order messages.
 [Link] - The number of messages to be sent without any
acknowledgment. Default is 5. Set this to 1 to avoid out of order message due to retry.
 [Link] - Maximum size of the message. Default 1 MB.

Summary
Based on the producer workflow and producer properties, tune the configuration to achieve
desired results. Importantly focus on below properties.

 [Link] – batch size (messages) per request

 [Link] – Time to wait before sending the current batch
 [Link] – compress messages

In part 3 of the series let’s understand Kafka producer delivery semantics and how to tune some
of the producer properties to achieve desired results.
Kafka producer delivery semantics
 Published on May 10, 2019

Sylvester Daniel

Program Architect at Mindtree - Big Data, AWS, Azure, Machine Learning & Deep Learning

10 articles

This article is a continuation of part 1 Kafka technical overview and part 2 Kafka producer
overview articles. Let's look into different delivery semantics and how to achieve those using
producer and broker properties.

Delivery semantics
Based on broker & producer configuration all three delivery semantics “at most once”, “at least
once” and “exactly once” are supported.

At most once

In at most once delivery semantics a message should be delivered maximum only once. It's
acceptable to lose a message rather than delivering a message twice in this semantic. Few use
cases of at most once includes metrics collection, log collection and so on. Applications adopting
at most semantics can easily achieve higher throughput and low latency.

At least once

In at least once delivery semantics it is acceptable to deliver a message more than once but no
message should be lost. The producer ensures that all messages are delivered for sure even
though it may result in message duplication. This is mostly preferred semantics out of all.
Applications adopting at least once semantics may have moderate throughput and moderate
latency.

Exactly once

In exactly one delivery semantics a message must be delivered only once and no message
should be lost. This is the most difficult delivery semantic of all. Applications adopting exactly
once semantics may have lower throughput and higher latency compared other 2 semantics.
Delivery Semantics summary

The table below summarizes the behavior of all delivery semantics.

Producer delivery semantics

Different delivery semantics can be achieved in Kafka using Acks property of producer and
[Link] property of the broker (considered only when acks = all).

Acks = 0

When acks property is set to zero you get at most once delivery semantics. Kafka producer
sends the record to the broker and doesn't wait for any response. Messages once sent will not be
retried in this setting. The producer uses “send and forget approach “with acks = 0.
Data loss

In this mode, chances for data loss is high as the producer does not confirm the message was
received by the broker. The message may not have even reached the broker or broker failure
soon after message delivery can result in data loss.

Acks = 1
When this property is set to 1 you can achieve at least once delivery semantics. Kafka producer
sends the record to the broker and waits for a response from the broker. If no acknowledgment is
received for the message sent, then the producer will retry sending the messages based on retry
configuration. Retries property by default is 0 make sure this is set to desired number or
[Link].

Data loss

In this mode, chances for data is moderate as the producer confirms that the message was
received by the broker (leader partition). As the replication of follower partition happens after
the acknowledgment this may still result in data loss. For example, after sending the
acknowledgment and before replication if the broker goes down this may result in data loss as
the producer will not resend the message.
Acks = All

When acks property is set to all, you can achieve exactly once delivery semantics. Kafka
producer sends the record to the broker and waits for a response from the broker. If no
acknowledgment is received for the message sent, then the producer will retry sending the
messages based on retry config n times. The broker sends acknowledgment only after replication
based on [Link] property.

For example, a topic may have a replication factor of 3 and [Link] of 2. In this case,
an acknowledgment will be sent after the second replication is complete. In order to achieve
exactly once delivery semantics the broker has to be idempotent. Acks = all should be used in
conjunction with [Link].

Data loss

In this mode, chances for data loss is low as the producer confirms that the message was received
by the broker (leader and follower partition) only after replication. As the replication of follower
partition happens before the acknowledgment data loss chances are minimal. For example,
before replication and sending acknowledgment if the broker goes down, the producer will not
receive the acknowledgment and will send the message again to the newly elected leader
partition.

Exception

When there are not enough nodes to replicate as per [Link] property then the broker
would return an exception instead of acknowledgment.
Safe producer

In order to create a safe producer that ensures minimal data loss, use below producer properties.

Producer properties

 Acks = all (default 1) – Ensures replication before acknowledgement

 Retries = MAX_INT (default 0) – Retry in case of exceptions
 [Link] = 5 (default) – Parallel connections to broker

Broker properties

 [Link] = 2 (at least 2) – Ensures minimum In Sync replica (ISR).

Acks impact

The table below summarizes the impact of acks property on latency, throughput, and durability.
Summary

Configure Kafka producer and broker to achieve desired delivery semantics based on following
properties.

 Acks
 Retries
 [Link]
 [Link]

In part 4 of the series, let’s understand Kafka consumer, consumer group and how to achieve
different Kafka consumer delivery semantics.
Kafka Monitoring
Apache kafka deals with transfering of large amount of real-time data( we can call it data
in a motion).

To assure end-to-end stream monitoring and every message is delivered from producer to
consumer.

How long messages take to be delivered, also determines the source of issue in your cluster.
We can monitor kafka with the help of metrics.

While monitoring kafka, it’s important to also monitor Zookeeper as kafka depends on it.

Why to Monitor Kafka

Kafka monitoring is important to ensure timeliness of data delivery, overall application

performance , knowing when to scale up ,
connectivity issues and ensuring data is not lost as we deal with streaming data.

Volume of data is large and there are different components involved into kafka cluster
which are:
Producer , Consumer and Broker.

To ensure every component is working fine.

Network Request Rate

Monitor and compare the network throughput per server, if possible by tracking the
number of network requests per second.
[Link]: type=RequestMetrics, name=RequestsPerSec.

Network error Rate

Error conditions include dropped network packets, error rates in responses per request
type, and the types of error(s) occurring.

network throughput with related network error rates can help diagnose the reasons for
latency.

Under-Replicated Partitions
To ensure data durability and that brokers are always available to deliver data ,
[Link]: type=ReplicaManager, name=UnderReplicatedPartitions

Total broker 04 Partitions

Simply knowing how many partitions a broker is managing can help you avoid errors and
know when it’s time to scale out. The goal should
be to keep the count balanced across brokers.
[Link]: type=ReplicaManager, name=PartitionCount – Number of partitions on the
brokers.

Log Flush Latency

Kafka Stores data by appending to existing log files .Cache based writes are flushed to
physical storage.
Your monitoring strategy should include combination of data replication and latency in the
asynchronous disk log flush time.
[Link]: type=LogFlushStats, name=LogFlushRateAndTimeMs

Consumer Message Rate

Set baselines for expected consumer message throughput and measure
fluctuations in the rate to detect latency and the need to scale the
number of consumers up and down accordingly.
[Link] type=ConsumerTopicMetrics, name=MessagePerSec, clientId=([-.w]+)
Messages consumed per sec

Consumer Max Lag

Even with consumers fetching messages at a high rate, producers
can still outspace them. This metrics works at the level of consumer
and partition , means each partition in each topic has its own lag for
a given consumer.
[Link]: type=ConsumerFetcherManager, name=MaxLag,
clientId=([-.w]+) Number of messages by which consumer lags
behind the producer.

Fetcher Lag
This metrics indicates the lag in the number of messages per follower
replica, indicating that replication has potentially stopped or has
been interrupted. Monitoring the [Link]
configuration parameter you can measure the time for which the
replica has not attempted to fetch new data from the leader.
[Link]: type=FetcherLagMetrics, name=ConsumerLag,
clientId=([-.w]+), partition=([0-9]+)

Offline Partition Count

Offline partitions represent data stores unavailable to your
application due to a server failure or restart. In kafka cluster one of
the broker server acts as a controller for managing the states of
partitions and replicas and to reassign partitions when needed.
[Link]: type=KafkaController, name=OfflinePartitionCount –
Number of partitions without an active leader.

Free Memory and Swap space Usage

Kafka performance is best when swapping is kept to minimum. To do
this set the JVM max heap size large enough to avoid frequent
garbage collecion activity, but small enough to allow space for
filesystem caching . Additionally , watch for swap usage if you have
swap enabled , watching for increases in server swapping activity, as
this can lead to kafka operations timeout.
In many cases its best to turn off swap entirely, we have to adjust our
monitoring accordingly.

Kafka Topic We can say that kafka topic is the same concept as a table in the database.

But its definetly not a table and kafka isn’t a database.

A topic is where data(messages) get published by the producer and pulled from by a consumer.

Kafka Topic Configuration

2) For changing the configuration of replication-factor of a topic : add a json script
with the
content provided below:
Assume the script name is [Link].
{"version":1,
"partitions":[
{"topic":"sendInvitation","partition":0,"replicas":[0,1,2]},
{"topic":"sendInvitation","partition":1,"replicas":[0,1,2]},
{"topic":"sendInvitation","partition":2,"replicas":[0,1,2]},
{"topic":"xyz","partition":0,"replicas":[0,1,2]},
{"topic":"xyz","partition":1,"replicas":[0,1,2]},
]}
Than execute the following command to run and apply this script:
./kafka-reassign-partitions --zookeeper localhost:2181 --reassignment-json-file
[Link] --execute

Achieving a 50% Reduction in

Cross-AZ Network Costs
from Kafka

SignalFx:
● Real-Time Cloud Monitoring Platform for Infrastructure, Micro-services and Applications
● 20 Kafka clusters in production
● 400 billion messages per day on the largest cluster
Kafka Message Compression

• Kafka supports end-to-end compression.

• Data is compressed by the Kafka producer client.
• Data is written in compressed format on Kafka
brokers, leading to savings on disk usage.
• Data is decompressed by Kafka consumer client.

• Enabling compression is as simple as setting the config

[Link] on Kafka producer client.

• Compression uses extra CPU and memory on producer/consumer.

• Snappy compression type worked best for us.

Racks changes based on broker placement across the racks