Kafka MirrorMaker

      No Comments on Kafka MirrorMaker
kafka mirrormaker

The Mirror Maker consumer needs to be client compatible with the source cluster. The Mirror Maker producer needs to be client compatible with the destination cluster.

General Considerations

MirrorMaker does not provide a guarantee of having a similar offset as there can be multiple consumers and only one single producer. During copy messages can be dropped or retried hence offset values at destination cluster can change accordingly.

MirrorMaker runs a single consumer hence it is recommended to run MirrorMaker process on the destination cluster as it improves Kafka producer latency and hence fewer failures

Make sure that destination cluster has enough free space to hold replicated data also it is recommended to create topics manually on the destination cluster but if you want them to be auto-created then please make sure that property called auto.create.topics.enable=true is set on the destination cluster

MirrorMaker runs as a single process so for Kerberos configuration it can use only a single set of credentials. Your destination cluster and source cluster should be within the same Kerberos realm. If realms are different then there should trust established between realms of Kerberos

MirrorMaker Command

/usr/hdp/current/kafka-broker/bin/kafka-mirror-maker.sh --consumer.config /home/consumer.properties --producer.config /home/producer.properties --whitelist='.*'

Producer Configs

bootstrap.servers=destination-broker1:6667,destination-broker2:6667,destination-broker3:6667
acks=all
retries=2147483647
batch.size=100
client.id=mirror_maker_producer

Consumer Configs

bootstrap.servers=source-broker1:6667,source-broker2:6667,source-broker3:6667
group.id=dp-MirrorMaker-group
exclude.internal.topics=true
client.id=mirror_maker_consumer
auto.offset.reset=earliest
auto.commit.enable=false

MirrorMaker Configs

abort.on.send.failure=true

Leave a Reply