om.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer$.handledType()Ljava/lang/Class;

SparkSQL连接Hive报错:

om.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer$.handledType()Ljava/lang/Class;

原因这是由于jackson相关包的冲突导致,或者缺少相关的包导致的,在使用spark-sql 连接hive的时候需要假如以下的依赖,将对应的Spark版本和Scala版本换成自己使用的版本即可。

<dependencies>
        <!-- mysql -->
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.26</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.12</artifactId>
            <version>2.4.5</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.12</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.module/jackson-module-scala -->
        <dependency>
            <groupId>com.fasterxml.jackson.module</groupId>
            <artifactId>jackson-module-scala_2.12</artifactId>
            <version>2.10.3</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.module</groupId>
            <artifactId>jackson-module-jaxb-annotations</artifactId>
            <version>2.10.3</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.10.3</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
            <version>2.10.3</version>
        </dependency>
 </dependencies>

替换之后即可成功运行,注意需要在spark-sql的代码中开启enableHiveSupport()的支持

+------+-------------------+--------+
|uid   |time               |dt      |
+------+-------------------+--------+
|100012|2017-02-02 17:04:02|20170202|
|100011|2017-02-02 17:05:06|20170202|
|100012|2017-02-02 17:07:07|20170202|
|100016|2017-02-02 17:08:04|20170202|
|100011|2017-02-02 18:01:01|20170202|
|100016|2017-02-02 19:03:02|20170202|
+------+-------------------+--------+

猜你喜欢

转载自blog.csdn.net/qq_43081842/article/details/105905458