SparkSQL连接Hive报错:
om.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer$.handledType()Ljava/lang/Class;
原因这是由于jackson相关包的冲突导致,或者缺少相关的包导致的,在使用spark-sql 连接hive的时候需要假如以下的依赖,将对应的Spark版本和Scala版本换成自己使用的版本即可。
<dependencies>
<!-- mysql -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.26</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.12</artifactId>
<version>2.4.5</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.module/jackson-module-scala -->
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-scala_2.12</artifactId>
<version>2.10.3</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-jaxb-annotations</artifactId>
<version>2.10.3</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.10.3</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.10.3</version>
</dependency>
</dependencies>
替换之后即可成功运行,注意需要在spark-sql的代码中开启enableHiveSupport()的支持
+------+-------------------+--------+
|uid |time |dt |
+------+-------------------+--------+
|100012|2017-02-02 17:04:02|20170202|
|100011|2017-02-02 17:05:06|20170202|
|100012|2017-02-02 17:07:07|20170202|
|100016|2017-02-02 17:08:04|20170202|
|100011|2017-02-02 18:01:01|20170202|
|100016|2017-02-02 19:03:02|20170202|
+------+-------------------+--------+