基于spark2.2做测试,期间遇到的一些问题。
- jar包提交
public static void main(String[] args) {
SparkConf conf = new SparkConf();
conf.setAppName("Tsoip")
SparkSession orCreate = SparkSession.builder().config(conf).enableHiveSupport().getOrCreate();
SQLContext sqlContext = new SQLContext(orCreate);
String sql = " hive相关sql ";
sqlContext.sql(sql);
}
- 异常
Final app status: FAILED, exitCode: 15, (reason: User class threw exception:
org.apache.spark.sql.AnalysisException: Table or view not found: `ods_fpos`.`sales_order_item_promotion`; line 9 pos 7;
-
解决方法
配置hive元数据:conf.set("hive.metastore.uris","thrift://fp-bd6:9083")
-
python脚本提交
# -*- coding: UTF-8 -*-
from pyspark.sql import SparkSession
if __name__ == "__main__":
spark = SparkSession.builder.appName("tSalesItem").enableHiveSupport().getOrCreate()
sql = """
hive相关sql
"""
spark.sql(sql)
- 异常
pyspark.sql.utils.AnalysisException: u"Table or view not found:`ods_fpos`.`sales_order_item_promotion`
- 解决方法
有说配置属性:
conf.set(spark.sql.warehouse.dir,hdfs://hadoop1:9000//user/hive/warehouse)
,
也有在启动脚本加hive-site.xml参数的
spark-submit --files /usr/hdp/current/hive-client/conf/hive-site.xml --master ..
试了下都不行。最后还是设置hive源数据解决:SparkSession.builder.appName("tSalesItem").config("hive.metastore.uris", "thrift://fp-bd6:9083")