本人的spark是2.2.4的,kafka是0.10的,遇到了以下问题,后续会持续更新
1、Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Failed to find data source: kafka. Please find packages at http://spark.apache.org/third-party-projects.html java.lang.ClassNotFoundException: Failed to find data source: kafka. Please find packages at http://spark.apache.org/third-party-projects.html
Caused by: java.lang.ClassNotFoundException: kafka.DefaultSource
原因是确少了sparksql跟kafka整合的包,只需添加一下依赖即可
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql-kafka-0-10_2.11</artifactId>
<version>2.2.0</version>
</dependency>
2、Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.serialization.ByteArrayDeserializer
原因是缺少了kafka的client包,添加以下依赖即可
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.12</artifactId>
<version>0.10.2.2</version>
</dependency>