该篇博客是Kafka API的使用,自定义分区
更多高级用法请看我的博客Kafka系列
参考:
https://www.orchome.com/189
https://www.w3cschool.cn/apache_kafka/apache_kafka_workflow.html
一. 简介
kafka概念相关的介绍请看官方文档和其他博文
官方中文文档
kafka入门介绍
我们可以自定义分区策略,这样就能控制消息被发送到指定分区
二. 实现
2.1 引入依赖
主要是spring-kafka依赖
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit-dep</artifactId>
<version>4.9</version>
<scope>test</scope>
</dependency>
<!-- kafka start -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.10.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.10.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams</artifactId>
<version>0.10.1.1</version>
</dependency>
<!-- kafka end -->
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.2-beta-5</version>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.3</version>
</dependency>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>3.2.1</version>
</dependency>
</dependencies>
2.2 指定分区策略生产者
MessageInfo.java
import java.io.Serializable;
/**
* 消息实体
*/
public class MessageInfo implements Serializable {
private static final long serialVersionUID = 1L;
private String id;
private String msg;
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getMsg() {
return msg;
}
public void setMsg(String msg) {
this.msg = msg;
}
@Override
public String toString() {
return "MessageInfo{" +
"id='" + id + '\'' +
", msg='" + msg + '\'' +
'}';
}
}
自定义分区 MyPartitionor.java
import org.apache.kafka.clients.producer.Partitioner;
import org.apache.kafka.common.Cluster;
import java.util.Map;
/**
* 自定义Partitioner
*/
public class MyPartitionor implements Partitioner {
/**
* 分区数
*/
private static final Integer PARTITIONS = 5;
/**
* 根据ID与分区总长度取模来做为分区分配的策略
*/
@Override
public int partition(String topic, Object key, byte[] keyBytes,
Object value, byte[] valueBytes, Cluster cluster) {
if (null == key) {
return 0;
}
String id = String.valueOf(key);
try {
int partitionId = Integer.valueOf(id) % PARTITIONS;
return partitionId;
} catch (NumberFormatException e) {
System.out.println("Parse message key exception:" + e);
return 0;
}
}
@Override
public void configure(Map<String, ?> arg0) {
}
@Override
public void close() {
}
}
生产者 PartitionorProducer.java
import com.it.cloud.producer.dto.MessageInfo;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
import java.util.Random;
/**
* 生产者--自定义分区
*/
public class PartitionorProducer {
/**
* 消息
*
* @return
*/
private static MessageInfo createMessageInfo() {
// 随机产生1到10之间的整数
Random r = new Random();
int id = r.nextInt(10);
MessageInfo info = new MessageInfo();
info.setId(String.valueOf(id));
info.setMsg("这里是消息");
return info;
}
public static void main(String[] args) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
// 设置自定义分区
props.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, "com.it.cloud.producer.partition.MyPartitionor");
// 生产者
KafkaProducer<String, String> producer = new KafkaProducer<String, String>(props);
try {
MessageInfo info = createMessageInfo();
ProducerRecord<String, String> record = new ProducerRecord<String, String>("my-topic", info.getId(), info.getMsg());
producer.send(record);
Thread.sleep(2000);
} catch (Exception e) {
System.out.println("Send message exception" + e);
} finally {
producer.close();
}
}
}
详细参数请参考:
http://www.shixinke.com/java/kafka-configuration
https://blog.csdn.net/xiaozhu_you/article/details/91493258
源码地址
IT-CLOUD-KAFKA-CLIENT :spring整合kafka教程源码。博文在本CSDN kafka系列中。
项目推荐
IT-CLOUD :IT服务管理平台,集成基础服务,中间件服务,监控告警服务等。
IT-CLOUD-ACTIVITI6 :Activiti教程源码。博文在本CSDN Activiti系列中。
IT-CLOUD-ELASTICSEARCH :elasticsearch教程源码。博文在本CSDN elasticsearch系列中。
IT-CLOUD-KAFKA :spring整合kafka教程源码。博文在本CSDN kafka系列中。
IT-CLOUD-KAFKA-CLIENT :kafka client教程源码。博文在本CSDN kafka系列中。开源项目,持续更新中,喜欢请 Star~