Jump to content

How to change the number of replicas of a Kafka topic

From Costa's Wiki

1. Specify the extra replicas in a custom reassignment json file

[edit]

For example, you could create increase-replication-factor.json and put this content in it:

{"version":1,
 "partitions":[
    {"topic":"telegraf","partition":0,"replicas":[0,1,2]},
    {"topic":"telegraf","partition":1,"replicas":[1,2,0]},
    {"topic":"telegraf","partition":2,"replicas":[2,0,1]}
]}

2. Use the file with the --execute option of the kafka-reassign-partitions tool

[edit]

[or kafka-reassign-partitions.sh - depending on the kafka package]

For example:

$ ./kafka-reassign-partitions.sh --bootstrap-server kafka.kafka:9092 --reassignment-json-file increase-replication-factor.json --execute

3. Verify the replication factor with the kafka-topics tool

[edit]

[or kafka-topics.sh - depending on the kafka package]

bash-5.0# ./kafka-topics.sh --bootstrap-server kafka.kafka:9092 --topic telegraf --describe
Topic: telegraf TopicId: O4smIJNBRyqmm0z2CwunNA PartitionCount: 3       ReplicationFactor: 1    Configs: 
cleanup.policy=delete,segment.bytes=5242880,retention.ms=43200000,message.timestamp.type=LogAppendTime,retention.bytes=-1,delete.retention.ms=43200000
       Topic: telegraf Partition: 0    Leader: 0       Replicas: 0,1   Isr: 0  Adding Replicas: 1      Removing Replicas: 
       Topic: telegraf Partition: 1    Leader: 1       Replicas: 1,2   Isr: 1  Adding Replicas: 2      Removing Replicas: 
       Topic: telegraf Partition: 2    Leader: 2       Replicas: 2,0   Isr: 2  Adding Replicas: 0      Removing Replicas: 
bash-5.0# ./kafka-topics.sh --bootstrap-server kafka.kafka:9092 --topic telegraf --describe
Topic: telegraf TopicId: O4smIJNBRyqmm0z2CwunNA PartitionCount: 3       ReplicationFactor: 2    Configs: 
cleanup.policy=delete,segment.bytes=5242880,retention.ms=43200000,message.timestamp.type=LogAppendTime,retention.bytes=-1,delete.retention.ms=43200000
       Topic: telegraf Partition: 0    Leader: 0       Replicas: 0,1   Isr: 0,1
       Topic: telegraf Partition: 1    Leader: 1       Replicas: 1,2   Isr: 1,2
       Topic: telegraf Partition: 2    Leader: 2       Replicas: 2,0   Isr: 2,0
bash-5.0# 

See also: the part of the official documentation that describes how to increase the replication factor.