测试librdkafka
Table of Contents
1 版本匹配问题
经过尝试，librdkafka 0.9 不向前兼容kafka 0.8.1, 因此，当使用kafka0.8.1 server的时候，需要使用对应的librdkafka版本。
2 重新编译0.8版本
首先切换到librdkafka代码的0.8 branch 然后重新编译代码
make clean ./configure make
3 进入tests目录
运行下面的命令进行测试，可以看到测试全部通过。
[dean@dell_xps_13 tests]$ make gcc -I../src -I../src-cpp -c test.c gcc -I../src -I../src-cpp ../src/librdkafka.a test.o 0003-msgmaxsize.c -o 0003-msgmaxsize.test -L../src -L../src-cpp -lrdkafka -lpthread -lz -lrt gcc -I../src -I../src-cpp ../src/librdkafka.a test.o 0012-produce_consume.c -o 0012-produce_consume.test -L../src -L../src-cpp -lrdkafka -lpthread -lz -lrt gcc -I../src -I../src-cpp ../src/librdkafka.a test.o 0001-multiobj.c -o 0001-multiobj.test -L../src -L../src-cpp -lrdkafka -lpthread -lz -lrt gcc -I../src -I../src-cpp ../src/librdkafka.a test.o 0007-autotopic.c -o 0007-autotopic.test -L../src -L../src-cpp -lrdkafka -lpthread -lz -lrt gcc -I../src -I../src-cpp ../src/librdkafka.a test.o 0010-enforcereqacks.c -o 0010-enforcereqacks.test -L../src -L../src-cpp -lrdkafka -lpthread -lz -lrt gcc -I../src -I../src-cpp ../src/librdkafka.a test.o 0008-reqacks.c -o 0008-reqacks.test -L../src -L../src-cpp -lrdkafka -lpthread -lz -lrt gcc -I../src -I../src-cpp ../src/librdkafka.a test.o 0006-symbols.c -o 0006-symbols.test -L../src -L../src-cpp -lrdkafka -lpthread -lz -lrt gcc -I../src -I../src-cpp ../src/librdkafka.a test.o 0005-order.c -o 0005-order.test -L../src -L../src-cpp -lrdkafka -lpthread -lz -lrt gcc -I../src -I../src-cpp ../src/librdkafka.a test.o 0014-reconsume-191.c -o 0014-reconsume-191.test -L../src -L../src-cpp -lrdkafka -lpthread -lz -lrt gcc -I../src -I../src-cpp ../src/librdkafka.a test.o 0004-conf.c -o 0004-conf.test -L../src -L../src-cpp -lrdkafka -lpthread -lz -lrt gcc -I../src -I../src-cpp ../src/librdkafka.a test.o 0011-produce_batch.c -o 0011-produce_batch.test -L../src -L../src-cpp -lrdkafka -lpthread -lz -lrt gcc -I../src -I../src-cpp ../src/librdkafka.a test.o 0002-unkpart.c -o 0002-unkpart.test -L../src -L../src-cpp -lrdkafka -lpthread -lz -lrt g++ -I../src -I../src-cpp ../src/librdkafka.a test.o 0013-metadata.cpp -o 0013-metadata.test -L../src -L../src-cpp -lrdkafka++ -lrdkafka -lpthread -lz -lrt 0013-metadata.cpp:38:0: warning: "_GNU_SOURCE" redefined #define _GNU_SOURCE ^ <command-line>:0:0: note: this is the location of the previous definition for i in 0003-msgmaxsize.test 0012-produce_consume.test 0001-multiobj.test 0007-autotopic.test 0010-enforcereqacks.test 0008-reqacks.test 0006-symbols.test 0005-order.test 0014-reconsume-191.test 0004-conf.test 0011-produce_batch.test 0002-unkpart.test 0013-metadata.test ; do \ LD_LIBRARY_PATH=../src:../src-cpp ./run-test.sh $i || exit 1 ; \ done ############## 0003-msgmaxsize.test ################ ### Running test 0003-msgmaxsize.test in bare mode ### Created kafka instance rdkafka#producer-0 Using topic "rdkafkatest_generic" Delivery report for message #0: Success Delivery report for message #2: Success Delivery report for message #4: Success Delivery report for message #6: Success Delivery report for message #8: Success Destroying kafka instance rdkafka#producer-0 3 thread(s) in use by librdkafka, waiting... 1 thread(s) in use by librdkafka, waiting... 0 thread(s) in use by librdkafka ### ### 0003-msgmaxsize.test in bare mode PASSED! ### ### ############## 0012-produce_consume.test ################ ### Running test 0012-produce_consume.test in bare mode ### Using topic "rdkafkatest_0012" Topic rdkafkatest_0012, testid 4029298770115172415 Created kafka instance rdkafka#producer-0 Start produce to partition 0 Produced 5000 messages to partition 0, waiting for deliveries Start produce to partition 1 Produced 5000 messages to partition 1, waiting for deliveries Destroying kafka instance rdkafka#producer-0 Created kafka instance rdkafka#consumer-1 Consuming 5000 messages from partition 0 Destroying kafka instance rdkafka#consumer-1 Created kafka instance rdkafka#consumer-2 Consuming 5000 messages from partition 1 Destroying kafka instance rdkafka#consumer-2 Created kafka instance rdkafka#consumer-3 Consuming 10000 messages from one queue serving 2 partitions Start consuming partition 0 at offset -5000 Start consuming partition 1 at offset -5000 Destroying kafka instance rdkafka#consumer-3 8 thread(s) in use by librdkafka, waiting... 0 thread(s) in use by librdkafka ### ### 0012-produce_consume.test in bare mode PASSED! ### ### ############## 0001-multiobj.test ################ ### Running test 0001-multiobj.test in bare mode ### Creating and destroying 100 kafka instances Using topic "rdkafkatest_generic" 39 thread(s) in use by librdkafka, waiting... 0 thread(s) in use by librdkafka ### ### 0001-multiobj.test in bare mode PASSED! ### ### ############## 0007-autotopic.test ################ ### Running test 0007-autotopic.test in bare mode ### NOTE! This test requires auto.create.topics.enable=true to be configured on the broker! Created kafka instance rdkafka#producer-0 Using topic "rdkafkatest_25fe0dc34f572ffc_autotopic" Delivery report for message #0: Success Delivery report for message #1: Success Delivery report for message #2: Success Delivery report for message #3: Success Delivery report for message #4: Success Delivery report for message #5: Success Delivery report for message #6: Success Delivery report for message #7: Success Delivery report for message #8: Success Delivery report for message #9: Success Destroying kafka instance rdkafka#producer-0 3 thread(s) in use by librdkafka, waiting... 1 thread(s) in use by librdkafka, waiting... 0 thread(s) in use by librdkafka ### ### 0007-autotopic.test in bare mode PASSED! ### ### ############## 0010-enforcereqacks.test ################ ### Running test 0010-enforcereqacks.test in bare mode ### Using topic "rdkafkatest_generic" Produced 100 messages, waiting for deliveries, expecting DR status 'Local: ISR count insufficient' Destroying kafka instance rdkafka#producer-0 Produced 100 messages, waiting for deliveries, expecting DR status 'Success' Destroying kafka instance rdkafka#producer-1 4 thread(s) in use by librdkafka, waiting... 1 thread(s) in use by librdkafka, waiting... 0 thread(s) in use by librdkafka ### ### 0010-enforcereqacks.test in bare mode PASSED! ### ### ############## 0008-reqacks.test ################ ### Running test 0008-reqacks.test in bare mode ### NOTE! This test requires at least 3 brokers! NOTE! This test requires default.replication.factor=3 to be configured on all brokers! Using topic "rdkafkatest_generic" Created kafka instance rdkafka#producer-0 with required acks -1 Produced 100 messages, waiting for deliveries Destroying kafka instance rdkafka#producer-0 Created kafka instance rdkafka#producer-1 with required acks 0 Produced 100 messages, waiting for deliveries Destroying kafka instance rdkafka#producer-1 Created kafka instance rdkafka#producer-2 with required acks 1 Produced 100 messages, waiting for deliveries Destroying kafka instance rdkafka#producer-2 5 thread(s) in use by librdkafka, waiting... 0 thread(s) in use by librdkafka ### ### 0008-reqacks.test in bare mode PASSED! ### ### ############## 0006-symbols.test ################ ### Running test 0006-symbols.test in bare mode ### ### ### 0006-symbols.test in bare mode PASSED! ### ### ############## 0005-order.test ################ ### Running test 0005-order.test in bare mode ### Created kafka instance rdkafka#producer-0 Using topic "rdkafkatest_generic" Produced 100000 messages, waiting for deliveries Destroying kafka instance rdkafka#producer-0 3 thread(s) in use by librdkafka, waiting... 1 thread(s) in use by librdkafka, waiting... 0 thread(s) in use by librdkafka ### ### 0005-order.test in bare mode PASSED! ### ### ############## 0014-reconsume-191.test ################ ### Running test 0014-reconsume-191.test in bare mode ### Using topic "rdkafkatest_21c2d3e15f82e64c_0014" Topic rdkafkatest_21c2d3e15f82e64c_0014, testid 8544126830059343550 Created kafka instance rdkafka#producer-0 Destroying kafka instance rdkafka#producer-0 Created kafka instance rdkafka#consumer-1 Iteration #0: Consuming from partition 0 Iteration #0: consumed 10000 messages Iteration #1: Consuming from partition 0 EOF at offset 10000 Iteration #1: consumed 0 messages Destroying kafka instance rdkafka#consumer-1 4 thread(s) in use by librdkafka, waiting... 0 thread(s) in use by librdkafka ### ### 0014-reconsume-191.test in bare mode PASSED! ### ### ############## 0004-conf.test ################ ### Running test 0004-conf.test in bare mode ### Using topic "rdkafkatest_generic" %7|1462002036.970|TOPIC|my id#producer-0| New local topic: rdkafkatest_generic %7|1462002036.970|TOPIC|my id#producer-1| New local topic: rdkafkatest_generic 1 thread(s) in use by librdkafka, waiting... 0 thread(s) in use by librdkafka ### ### 0004-conf.test in bare mode PASSED! ### ### ############## 0011-produce_batch.test ################ ### Running test 0011-produce_batch.test in bare mode ### Created kafka instance rdkafka#producer-0 Using topic "rdkafkatest_generic" Single partition: Produced 100000 messages, waiting for deliveries Destroying kafka instance rdkafka#producer-0 3 thread(s) in use by librdkafka, waiting... 1 thread(s) in use by librdkafka, waiting... 0 thread(s) in use by librdkafka Created kafka instance rdkafka#producer-1 Using topic "rdkafkatest_generic" Partitioner: Produced 100000 messages, waiting for deliveries Destroying kafka instance rdkafka#producer-1 3 thread(s) in use by librdkafka, waiting... 1 thread(s) in use by librdkafka, waiting... 0 thread(s) in use by librdkafka ### ### 0011-produce_batch.test in bare mode PASSED! ### ### ############## 0002-unkpart.test ################ ### Running test 0002-unkpart.test in bare mode ### Created kafka instance rdkafka#producer-0 Using topic "rdkafkatest_generic" Failed to produce message #0: unknown partition: good! Failed to produce message #1: unknown partition: good! Failed to produce message #2: unknown partition: good! Failed to produce message #3: unknown partition: good! Failed to produce message #4: unknown partition: good! Failed to produce message #5: unknown partition: good! Failed to produce message #6: unknown partition: good! Failed to produce message #7: unknown partition: good! Failed to produce message #8: unknown partition: good! Failed to produce message #9: unknown partition: good! Destroying kafka instance rdkafka#producer-0 3 thread(s) in use by librdkafka, waiting... 0 thread(s) in use by librdkafka ### ### 0002-unkpart.test in bare mode PASSED! ### ### ############## 0013-metadata.test ################ ### Running test 0013-metadata.test in bare mode ### Using topic "rdkafkatest_481ff4e37b4bbab8_0013" Topic rdkafkatest_481ff4e37b4bbab8_0013. 3 thread(s) in use by librdkafka, waiting... 0 thread(s) in use by librdkafka ### ### 0013-metadata.test in bare mode PASSED! ### ### [dean@dell_xps_13 tests]$
4 运行example
前面已经在test topic中写入了消息，这里需要读取出来 -p 0 指的是partition 0
[dean@dell_xps_13 examples]$ ./rdkafka_example_cpp -C -t test -p 0 % Created consumer rdkafka#consumer-0 Read msg at offset 0 hello, world
也可以对多个broker读取信息，并且采用ccb的方式，ccb指的是consumber call back，据说这个性能更高。
./rdkafka_example_cpp -C -t test -p 0 -b localhost:9093,localhost:9094,localhost:9095 -f ccb再分享一下我老师大神的人工智能教程吧。零基础！通俗易懂！风趣幽默！还带黄段子！希望你也加入到我们人工智能的队伍中来！https://blog.csdn.net/jiangjunshow