Kafka版本升级

阅读量

Kafka版本升级

本文档介绍kafka_2.11_2.1.1升级到2.12-2.4.1的具体操作方法. 其他版本大同小异, 详见1.5 Upgrading From Previous Versions

升级前检查项

1.确认是否有副本因子是1的Topic

1
bin/kafka-topics.sh --zookeeper localhost:2182 --describe|grep "ReplicationFactor:[[:space:]]*1"

单副本Topic在rolling restart过程中必定会受到影响

2.确认是否有只有一个Isr的Topic

1
2
3
4
5
6
7
8
9
10
11
12
13
bin/kafka-topics.sh --zookeeper localhost:2182 --describe|grep Replicas|while read line;do echo "$(awk -F'Topic:' '{print $NF}' <<< "${line}"|awk -F 'Partition:' '{print $1}'|sed 's/[ \t]*//g' ) Isr: $(awk -F'Isr:' '{print $NF}' <<< "${line}"|tr "," "\n"|sort -n|tr "\n" ","|sed 's/,$/\n/g;'|sed 's/[ \t]*//g' )";done|awk '{printf "%-64s%-4s%-23s\n",$1,$2,$3}'

test.t_batch                                                 Isr:8,13,174               
test.t_template                                              Isr:8,13,174               
test.t_user_apply                                            Isr:8,13,174               
test.t_user_order                                            Isr:8,13,174               
test.t_user                                                  Isr:8,13,174               
test.t_user_job_record                                       Isr:8,13,174 


使用如下语句输出只有一个Isr的Topic. (if (length(a) == 1)
bin/kafka-topics.sh --zookeeper localhost:2182 --describe|grep Replicas|awk '{split($NF,a,",");slen=asort(a,tA);}{if (length(a) == 1){{printf $2" : ";for (i = 1; i <= length(a); i++){if (i==length(a)&&i==1){printf tA[i];}else{printf tA[i]",";}}}{printf "\n";}}}'|awk '{printf "%-64s%-23s\n",$1,$3}'

只有一个Isr的在rolling restart过程中必定会受到影响

增加Topic副本数

当上一步检查步骤存在问题Topic时, 需要针对这些Topic增加副本, 或等待Isr>1.

本次线上升级我们已知所有Topic都只有一个副本, 所以全部需要增加副本数

增加副本会触发数据复制, 可能导致网卡打满, IO增高, 还是会影响业务

增加副本数

Kafka 支持为已有 topic 的分区增加副本因子(replication factor)，具体的方法就是使用kafka-reassign-partitions.sh并且为 topic分区增加额外的副本

理论上不影响生产和消费, 实测也确实没影响

生成json文件

我们先为__consumer_offsets增加副本, 先describe一下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
#bin/kafka-topics.sh --zookeeper localhost:2182 --topic __consumer_offsets --describe
Topic: __consumer_offsets       PartitionCount: 50      ReplicationFactor: 1    Configs: compression.type=producer,cleanup.policy=compact,segment.bytes=104857600
        Topic: __consumer_offsets       Partition: 0    Leader: 13      Replicas: 13    Isr: 13
        Topic: __consumer_offsets       Partition: 1    Leader: 174     Replicas: 174   Isr: 174
        Topic: __consumer_offsets       Partition: 2    Leader: 8       Replicas: 8     Isr: 8
        Topic: __consumer_offsets       Partition: 3    Leader: 13      Replicas: 13    Isr: 13
        Topic: __consumer_offsets       Partition: 4    Leader: 174     Replicas: 174   Isr: 174
        Topic: __consumer_offsets       Partition: 5    Leader: 8       Replicas: 8     Isr: 8
        Topic: __consumer_offsets       Partition: 6    Leader: 13      Replicas: 13    Isr: 13
        Topic: __consumer_offsets       Partition: 7    Leader: 174     Replicas: 174   Isr: 174
        Topic: __consumer_offsets       Partition: 8    Leader: 8       Replicas: 8     Isr: 8
        Topic: __consumer_offsets       Partition: 9    Leader: 13      Replicas: 13    Isr: 13
        Topic: __consumer_offsets       Partition: 10   Leader: 174     Replicas: 174   Isr: 174
        Topic: __consumer_offsets       Partition: 11   Leader: 8       Replicas: 8     Isr: 8
        Topic: __consumer_offsets       Partition: 12   Leader: 13      Replicas: 13    Isr: 13
        Topic: __consumer_offsets       Partition: 13   Leader: 174     Replicas: 174   Isr: 174
        Topic: __consumer_offsets       Partition: 14   Leader: 8       Replicas: 8     Isr: 8
        Topic: __consumer_offsets       Partition: 15   Leader: 13      Replicas: 13    Isr: 13
        Topic: __consumer_offsets       Partition: 16   Leader: 174     Replicas: 174   Isr: 174
        Topic: __consumer_offsets       Partition: 17   Leader: 8       Replicas: 8     Isr: 8
        Topic: __consumer_offsets       Partition: 18   Leader: 13      Replicas: 13    Isr: 13
        Topic: __consumer_offsets       Partition: 19   Leader: 174     Replicas: 174   Isr: 174
        Topic: __consumer_offsets       Partition: 20   Leader: 8       Replicas: 8     Isr: 8
        Topic: __consumer_offsets       Partition: 21   Leader: 13      Replicas: 13    Isr: 13
        Topic: __consumer_offsets       Partition: 22   Leader: 174     Replicas: 174   Isr: 174
        Topic: __consumer_offsets       Partition: 23   Leader: 8       Replicas: 8     Isr: 8
        Topic: __consumer_offsets       Partition: 24   Leader: 13      Replicas: 13    Isr: 13
        Topic: __consumer_offsets       Partition: 25   Leader: 174     Replicas: 174   Isr: 174
        Topic: __consumer_offsets       Partition: 26   Leader: 8       Replicas: 8     Isr: 8
        Topic: __consumer_offsets       Partition: 27   Leader: 13      Replicas: 13    Isr: 13
        Topic: __consumer_offsets       Partition: 28   Leader: 174     Replicas: 174   Isr: 174
        Topic: __consumer_offsets       Partition: 29   Leader: 8       Replicas: 8     Isr: 8
        Topic: __consumer_offsets       Partition: 30   Leader: 13      Replicas: 13    Isr: 13
        Topic: __consumer_offsets       Partition: 31   Leader: 174     Replicas: 174   Isr: 174
        Topic: __consumer_offsets       Partition: 32   Leader: 8       Replicas: 8     Isr: 8
        Topic: __consumer_offsets       Partition: 33   Leader: 13      Replicas: 13    Isr: 13
        Topic: __consumer_offsets       Partition: 34   Leader: 174     Replicas: 174   Isr: 174
        Topic: __consumer_offsets       Partition: 35   Leader: 8       Replicas: 8     Isr: 8
        Topic: __consumer_offsets       Partition: 36   Leader: 13      Replicas: 13    Isr: 13
        Topic: __consumer_offsets       Partition: 37   Leader: 174     Replicas: 174   Isr: 174
        Topic: __consumer_offsets       Partition: 38   Leader: 8       Replicas: 8     Isr: 8
        Topic: __consumer_offsets       Partition: 39   Leader: 13      Replicas: 13    Isr: 13
        Topic: __consumer_offsets       Partition: 40   Leader: 174     Replicas: 174   Isr: 174
        Topic: __consumer_offsets       Partition: 41   Leader: 8       Replicas: 8     Isr: 8
        Topic: __consumer_offsets       Partition: 42   Leader: 13      Replicas: 13    Isr: 13
        Topic: __consumer_offsets       Partition: 43   Leader: 174     Replicas: 174   Isr: 174
        Topic: __consumer_offsets       Partition: 44   Leader: 8       Replicas: 8     Isr: 8
        Topic: __consumer_offsets       Partition: 45   Leader: 13      Replicas: 13    Isr: 13
        Topic: __consumer_offsets       Partition: 46   Leader: 174     Replicas: 174   Isr: 174
        Topic: __consumer_offsets       Partition: 47   Leader: 8       Replicas: 8     Isr: 8
        Topic: __consumer_offsets       Partition: 48   Leader: 13      Replicas: 13    Isr: 13
        Topic: __consumer_offsets       Partition: 49   Leader: 174     Replicas: 174   Isr: 174

目前副本因子是1, 现在我们要把它重设成3, 创建json文件/tmp/incr_replica.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
{"version":1,
"partitions":[
{"topic":"__consumer_offsets","partition":0,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":1,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":2,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":3,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":4,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":5,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":6,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":7,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":8,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":9,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":10,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":11,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":12,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":13,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":14,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":15,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":16,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":17,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":18,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":19,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":20,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":21,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":22,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":23,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":24,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":25,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":26,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":27,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":28,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":29,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":30,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":31,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":32,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":33,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":34,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":35,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":36,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":37,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":38,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":39,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":40,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":41,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":42,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":43,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":44,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":45,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":46,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":47,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":48,"replicas":[13,174,8]},
{"topic":"__consumer_offsets","partition":49,"replicas":[13,174,8]}
]}
执行重分配工作
1
2
3
4
5
6
7
[root@bj1-mysql-dba-prod-01 kafka_2.11-2.1.1]# bin/kafka-reassign-partitions.sh --zookeeper localhost:2182 --reassignment-json-file /tmp/incr_replica.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"__consumer_offsets","partition":22,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":30,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":8,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":21,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":4,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":27,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":7,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":9,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":46,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":25,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":35,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":41,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":33,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":23,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":49,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":47,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":16,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":28,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":31,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":36,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":42,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":3,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":18,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":37,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":15,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":24,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":38,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":17,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":48,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":19,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":11,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":13,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":2,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":43,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":6,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":14,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":20,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":0,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":44,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":39,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":12,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":45,"replicas":[13],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":1,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":5,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":26,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":29,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":34,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":10,"replicas":[174],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":32,"replicas":[8],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":40,"replicas":[174],"log_dirs":["any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.
验证重分配是否成功
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
bin/kafka-reassign-partitions.sh --zookeeper localhost:2182 --reassignment-json-file /tmp/incr_replica.json --verify
Status of partition reassignment: 
Reassignment of partition __consumer_offsets-22 completed successfully
Reassignment of partition __consumer_offsets-30 completed successfully
Reassignment of partition __consumer_offsets-8 completed successfully
Reassignment of partition __consumer_offsets-21 completed successfully
Reassignment of partition __consumer_offsets-4 completed successfully
Reassignment of partition __consumer_offsets-27 completed successfully
Reassignment of partition __consumer_offsets-7 completed successfully
Reassignment of partition __consumer_offsets-9 completed successfully
Reassignment of partition __consumer_offsets-46 completed successfully
Reassignment of partition __consumer_offsets-25 completed successfully
Reassignment of partition __consumer_offsets-35 completed successfully
Reassignment of partition __consumer_offsets-41 completed successfully
Reassignment of partition __consumer_offsets-33 completed successfully
Reassignment of partition __consumer_offsets-23 completed successfully
Reassignment of partition __consumer_offsets-49 completed successfully
Reassignment of partition __consumer_offsets-47 completed successfully
Reassignment of partition __consumer_offsets-16 completed successfully
Reassignment of partition __consumer_offsets-28 completed successfully
Reassignment of partition __consumer_offsets-31 completed successfully
Reassignment of partition __consumer_offsets-36 completed successfully
Reassignment of partition __consumer_offsets-42 completed successfully
Reassignment of partition __consumer_offsets-3 completed successfully
Reassignment of partition __consumer_offsets-18 completed successfully
Reassignment of partition __consumer_offsets-37 completed successfully
Reassignment of partition __consumer_offsets-15 completed successfully
Reassignment of partition __consumer_offsets-24 completed successfully
Reassignment of partition __consumer_offsets-38 completed successfully
Reassignment of partition __consumer_offsets-17 completed successfully
Reassignment of partition __consumer_offsets-48 completed successfully
Reassignment of partition __consumer_offsets-19 completed successfully
Reassignment of partition __consumer_offsets-11 completed successfully
Reassignment of partition __consumer_offsets-13 completed successfully
Reassignment of partition __consumer_offsets-2 completed successfully
Reassignment of partition __consumer_offsets-43 completed successfully
Reassignment of partition __consumer_offsets-6 completed successfully
Reassignment of partition __consumer_offsets-14 completed successfully
Reassignment of partition __consumer_offsets-20 completed successfully
Reassignment of partition __consumer_offsets-0 completed successfully
Reassignment of partition __consumer_offsets-44 completed successfully
Reassignment of partition __consumer_offsets-39 completed successfully
Reassignment of partition __consumer_offsets-12 completed successfully
Reassignment of partition __consumer_offsets-45 completed successfully
Reassignment of partition __consumer_offsets-1 completed successfully
Reassignment of partition __consumer_offsets-5 completed successfully
Reassignment of partition __consumer_offsets-26 completed successfully
Reassignment of partition __consumer_offsets-29 completed successfully
Reassignment of partition __consumer_offsets-34 completed successfully
Reassignment of partition __consumer_offsets-10 completed successfully
Reassignment of partition __consumer_offsets-32 completed successfully
Reassignment of partition __consumer_offsets-40 completed successfully

再describe确认一下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
[root@bj1-mysql-dba-prod-01 kafka_2.11-2.1.1]# bin/kafka-topics.sh --zookeeper localhost:2182 --topic __consumer_offsets --describe
Topic:__consumer_offsets        PartitionCount:50       ReplicationFactor:3     Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
        Topic: __consumer_offsets       Partition: 0    Leader: 13      Replicas: 13,174,8      Isr: 13,8,174
        Topic: __consumer_offsets       Partition: 1    Leader: 174     Replicas: 13,174,8      Isr: 174,8,13
        Topic: __consumer_offsets       Partition: 2    Leader: 8       Replicas: 13,174,8      Isr: 8,174,13
        Topic: __consumer_offsets       Partition: 3    Leader: 13      Replicas: 13,174,8      Isr: 13,8,174
        Topic: __consumer_offsets       Partition: 4    Leader: 174     Replicas: 13,174,8      Isr: 174,13,8
        Topic: __consumer_offsets       Partition: 5    Leader: 8       Replicas: 13,174,8      Isr: 8,174,13
        Topic: __consumer_offsets       Partition: 6    Leader: 13      Replicas: 13,174,8      Isr: 13,8,174
        Topic: __consumer_offsets       Partition: 7    Leader: 174     Replicas: 13,174,8      Isr: 174,13,8
        Topic: __consumer_offsets       Partition: 8    Leader: 8       Replicas: 13,174,8      Isr: 8,174,13
        Topic: __consumer_offsets       Partition: 9    Leader: 13      Replicas: 13,174,8      Isr: 13,8,174
        Topic: __consumer_offsets       Partition: 10   Leader: 174     Replicas: 13,174,8      Isr: 174,8,13
        Topic: __consumer_offsets       Partition: 11   Leader: 8       Replicas: 13,174,8      Isr: 8,174,13
        Topic: __consumer_offsets       Partition: 12   Leader: 13      Replicas: 13,174,8      Isr: 13,8,174
        Topic: __consumer_offsets       Partition: 13   Leader: 174     Replicas: 13,174,8      Isr: 174,8,13
        Topic: __consumer_offsets       Partition: 14   Leader: 8       Replicas: 13,174,8      Isr: 8,174,13
        Topic: __consumer_offsets       Partition: 15   Leader: 13      Replicas: 13,174,8      Isr: 13,8,174
        Topic: __consumer_offsets       Partition: 16   Leader: 174     Replicas: 13,174,8      Isr: 174,8,13
        Topic: __consumer_offsets       Partition: 17   Leader: 8       Replicas: 13,174,8      Isr: 8,174,13
        Topic: __consumer_offsets       Partition: 18   Leader: 13      Replicas: 13,174,8      Isr: 13,8,174
        Topic: __consumer_offsets       Partition: 19   Leader: 174     Replicas: 13,174,8      Isr: 174,8,13
        Topic: __consumer_offsets       Partition: 20   Leader: 8       Replicas: 13,174,8      Isr: 8,174,13
        Topic: __consumer_offsets       Partition: 21   Leader: 13      Replicas: 13,174,8      Isr: 13,174,8
        Topic: __consumer_offsets       Partition: 22   Leader: 174     Replicas: 13,174,8      Isr: 174,13,8
        Topic: __consumer_offsets       Partition: 23   Leader: 8       Replicas: 13,174,8      Isr: 8,174,13
        Topic: __consumer_offsets       Partition: 24   Leader: 13      Replicas: 13,174,8      Isr: 13,8,174
        Topic: __consumer_offsets       Partition: 25   Leader: 174     Replicas: 13,174,8      Isr: 174,8,13
        Topic: __consumer_offsets       Partition: 26   Leader: 8       Replicas: 13,174,8      Isr: 8,174,13
        Topic: __consumer_offsets       Partition: 27   Leader: 13      Replicas: 13,174,8      Isr: 13,174,8
        Topic: __consumer_offsets       Partition: 28   Leader: 174     Replicas: 13,174,8      Isr: 174,8,13
        Topic: __consumer_offsets       Partition: 29   Leader: 8       Replicas: 13,174,8      Isr: 8,174,13
        Topic: __consumer_offsets       Partition: 30   Leader: 13      Replicas: 13,174,8      Isr: 13,8,174
        Topic: __consumer_offsets       Partition: 31   Leader: 174     Replicas: 13,174,8      Isr: 174,8,13
        Topic: __consumer_offsets       Partition: 32   Leader: 8       Replicas: 13,174,8      Isr: 8,174,13
        Topic: __consumer_offsets       Partition: 33   Leader: 13      Replicas: 13,174,8      Isr: 13,8,174
        Topic: __consumer_offsets       Partition: 34   Leader: 174     Replicas: 13,174,8      Isr: 174,13,8
        Topic: __consumer_offsets       Partition: 35   Leader: 8       Replicas: 13,174,8      Isr: 8,174,13
        Topic: __consumer_offsets       Partition: 36   Leader: 13      Replicas: 13,174,8      Isr: 13,8,174
        Topic: __consumer_offsets       Partition: 37   Leader: 174     Replicas: 13,174,8      Isr: 174,8,13
        Topic: __consumer_offsets       Partition: 38   Leader: 8       Replicas: 13,174,8      Isr: 8,174,13
        Topic: __consumer_offsets       Partition: 39   Leader: 13      Replicas: 13,174,8      Isr: 13,8,174
        Topic: __consumer_offsets       Partition: 40   Leader: 174     Replicas: 13,174,8      Isr: 174,8,13
        Topic: __consumer_offsets       Partition: 41   Leader: 8       Replicas: 13,174,8      Isr: 8,174,13
        Topic: __consumer_offsets       Partition: 42   Leader: 13      Replicas: 13,174,8      Isr: 13,8,174
        Topic: __consumer_offsets       Partition: 43   Leader: 174     Replicas: 13,174,8      Isr: 174,8,13
        Topic: __consumer_offsets       Partition: 44   Leader: 8       Replicas: 13,174,8      Isr: 8,174,13
        Topic: __consumer_offsets       Partition: 45   Leader: 13      Replicas: 13,174,8      Isr: 13,8,174
        Topic: __consumer_offsets       Partition: 46   Leader: 174     Replicas: 13,174,8      Isr: 174,8,13
        Topic: __consumer_offsets       Partition: 47   Leader: 8       Replicas: 13,174,8      Isr: 8,174,13
        Topic: __consumer_offsets       Partition: 48   Leader: 13      Replicas: 13,174,8      Isr: 13,8,174
        Topic: __consumer_offsets       Partition: 49   Leader: 174     Replicas: 13,174,8      Isr: 174,8,13

副本是3, 都是Isr, 这样就OK了

为其他topic增加副本
1
bin/kafka-topics.sh --zookeeper localhost:2182 --describe| grep "Topic: "| grep -v "__consumer_offsets"|awk -F':' '{print $2}' |awk '{print $1}'
1
2
3
4
5
6
7
8
9
10
11
12
13
{"version":1,
"partitions":[
{"topic":"pd.t1","partition":0,"replicas":[13,174,8]},
{"topic":"pd.t2","partition":0,"replicas":[13,174,8]},
{"topic":"pd.t3","partition":0,"replicas":[13,174,8]},
{"topic":"pd.t4","partition":0,"replicas":[13,174,8]},
{"topic":"bk.t5","partition":0,"replicas":[13,174,8]},
{"topic":"bk.t6","partition":0,"replicas":[13,174,8]},
{"topic":"bk.t7","partition":0,"replicas":[13,174,8]},
{"topic":"bk.t8","partition":0,"replicas":[13,174,8]},
{"topic":"bk.t9","partition":0,"replicas":[13,174,8]},
{"topic":"bk.t10","partition":0,"replicas":[13,174,8]}
]}
观察副本增加情况
1
2
3
4
5
6
7
8
9
10
11
12
返回必须为空
bin/kafka-topics.sh --zookeeper localhost:2182 --describe|grep "ReplicationFactor:[[:space:]]*1"

输出Isr需要>1
bin/kafka-topics.sh --zookeeper localhost:2182 --describe|grep Replicas|while read line;do echo "$(awk -F'Topic:' '{print $NF}' <<< "${line}"|awk -F 'Partition:' '{print $1}'|sed 's/[ \t]*//g' ) Isr: $(awk -F'Isr:' '{print $NF}' <<< "${line}"|tr "," "\n"|sort -n|tr "\n" ","|sed 's/,$/\n/g;'|sed 's/[ \t]*//g' )";done|awk '{printf "%-64s%-4s%-23s\n",$1,$2,$3}'

使用如下语句输出只有一个Isr的Topic. (if (length(a) == 1)
bin/kafka-topics.sh --zookeeper localhost:2182 --describe|grep Replicas|awk '{split($NF,a,",");slen=asort(a,tA);}{if (length(a) == 1){{printf $2" : ";for (i = 1; i <= length(a); i++){if (i==length(a)&&i==1){printf tA[i];}else{printf tA[i]",";}}}{printf "\n";}}}'|awk '{printf "%-64s%-23s\n",$1,$3}'

以下两个返回都需要为空
bin/kafka-topics.sh --zookeeper localhost:2182 --describe --under-replicated-partitions
bin/kafka-topics.sh --zookeeper localhost:2182 --describe --unavailable-partitions

分区重分配

增加副本数以后, prefferred replica是不变的, 在我这个例子里, 由于之前所有topic所有分区都只有一个副本112, 所以增加副本以后prefferred replica仍然是112. 虽然auto.leader.rebalance.enable默认为true, 但是由于kafka计算leader不均衡的方法是:

Kafka计算不均衡程度的逻辑实际上非常简单——该broker上的leader不是preferred replica的分区数/ broker上总的分区数。

胡夕. Apache Kafka实战 (Kindle 位置 4563-4564). 电子工业出版社. Kindle 版本.

所以没有卵用

我们需要进行分区重分配, 目的是更改prefferred replica

以往进行重分配的场景是kafka集群增加了新的broker, 比如源集群k1 k2 k3, 新增k4 k5. 原有Topic不会自动迁移到新节点上, 为了让k4 k5也能干活, 进行重分配, 将一部分topic迁移到k4 k5

重分配后应该也会产生一些leader选举, 对客户端也会造成影响

构造topics-to-move-json-file

我们需要指明一个需要进行分区重分配的topic列表(建议少量多次)

1
2
# cat /tmp/topics-to-move.json
{"topics": [{"topic":"ReconChannelMatch"}, {"topic":"ReconImportBill"}, {"topic":"__consumer_offsets"}], "version":1}

生产分配方案

1
2
3
4
5
6
7
8
# bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file /tmp/topics-to-move.json --broker-list "112,167,226" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"__consumer_offsets","partition":22,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":30,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":8,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":21,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":4,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":27,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":7,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":9,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":46,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":25,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":35,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":41,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":33,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":23,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":49,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":47,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":16,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":28,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":31,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":36,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":42,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":3,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":18,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":37,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"ReconChannelMatch","partition":0,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"ReconImportBill","partition":0,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":15,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":24,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":38,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":17,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":48,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":19,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":11,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":13,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":2,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":43,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":6,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":14,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":20,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":0,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":44,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":39,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":12,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":45,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":1,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":5,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":26,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":29,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":34,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":10,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":32,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":40,"replicas":[112,167,226],"log_dirs":["any","any","any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"__consumer_offsets","partition":49,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":38,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":16,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":27,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":8,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":19,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":13,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":2,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":46,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":35,"replicas":[167,226,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":24,"replicas":[226,167,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":5,"replicas":[167,226,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":43,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":21,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":32,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":10,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":37,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":48,"replicas":[226,167,112],"log_dirs":["any","any","any"]},{"topic":"ReconImportBill","partition":0,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":40,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":18,"replicas":[226,167,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":29,"replicas":[167,226,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":7,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":23,"replicas":[167,226,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":45,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":34,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":26,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":15,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":4,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":42,"replicas":[226,167,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":31,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":9,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":20,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":12,"replicas":[226,167,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":1,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":28,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":17,"replicas":[167,226,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":6,"replicas":[226,167,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":39,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":44,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"ReconChannelMatch","partition":0,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":36,"replicas":[226,167,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":47,"replicas":[167,226,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":3,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":25,"replicas":[112,226,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":14,"replicas":[167,112,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":30,"replicas":[226,167,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":41,"replicas":[167,226,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":22,"replicas":[112,167,226],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":33,"replicas":[226,112,167],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":11,"replicas":[167,226,112],"log_dirs":["any","any","any"]},{"topic":"__consumer_offsets","partition":0,"replicas":[226,167,112],"log_dirs":["any","any","any"]}]}
[root@bj3-canal-canal-prod-01 kafka]# cat /tmp/topics-to-move.json
{"topics": [{"topic":"ReconChannelMatch"}, {"topic":"ReconImportBill"}, {"topic":"__consumer_offsets"}], "version":1}

保存Proposed partition reassignment configuration后面的部分到

执行重分配

1
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file /tmp/move_prefferred.json --execute

执行后可以使用--verify参数来查看分区重分配执行情况

1
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file /tmp/move_prefferred.json --verify

滚动升级

以下步骤每个broker依次执行, 全部执行完成再进行下一个broker的操作

部署新版本kafka

过程略, 只解压软件包, 复制旧版的配置文件server.properties, 增加如下参数配置

1
2
3
4
5
6
7
inter.broker.protocol.version=2.1

default.replication.factor=3
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
min.insync.replicas=2

逐一升级broker

开两个窗口,分别启动一个producer和consumer

1
2
3
bin/kafka-console-producer.sh --broker-list 172.16.x.x:9092,172.16.x.x:9092,172.16.x.x:9092 --topic test_upgrade

bin/kafka-console-consumer.sh --bootstrap-server 172.16.x.x:9092,172.16.x.x:9092,172.16.x.x:9092 --topic test_upgrade

测试一下producer写数据consumer是否可以消费到, 我们在下文中称这个步骤为T1

关闭旧版本broker
1
2
3
4
5
6
7
8
9
10
# jps -m
2224 WrapperSimpleApp CloudResetPwdUpdateAgent
2432 ZooKeeperMain -server localhost:2182
9204 Jps -m
21351 QuorumPeerMain /usr/local/zookeeper/bin/../conf/zoo_test.cfg
32542 QuorumPeerMain /usr/local/zookeeper/bin/../conf/zoo.cfg
28335 ZooKeeperMain -server localhost:2182
27279 Kafka config/server.properties

# kill 27279

观察日志, 确认shutdown complete

1
2
3
tail -f /usr/local/kafka_2.11-2.1.1/logs/kafkaServer.out 

INFO [ReplicaFetcher replicaId=8, leaderId=174, fetcherId=0] Shutdown completed (kafka.server.ReplicaFetcherThread)

执行T1测试, 如果生产或消费失败, 则启动旧版本broker, 排查好问题后再次升级

使用新版本启动

确认一下命令无输出

1
bin/kafka-topics.sh --zookeeper localhost:2182 --describe --unavailable-partitions

启动

1
2
3
4
5
cd new_kafka_path
./bin/kafka-server-start.sh -daemon config/server.properties

确认无输出
bin/kafka-topics.sh --zookeeper localhost:2182 --describe --under-replicated-partitions

执行T1测试, 如果生产或消费失败, 则启动旧版本broker, 排查好问题后再次升级

切换至新协议

修改配置参数

1
inter.broker.protocol.version=2.4

逐一重启broker

Powered by Hexo and Hexo-theme-hiker

Copyright © 2013 - 2022 Fan() All Rights Reserved.

访客数 : | 访问量 :