制作Zeppelin的Parcel包和csd文件

其他 2019-06-12 10:11:28 阅读次数: 0
版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/weixin_43215250/article/details/90406232

  1. 配置java，maven等环境变量

    java:

    export JAVA_HOME=/usr/local/java
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tool.jar:$CLASSPATH

    maven:

    export MVN_HOME=/usr/local/maven
export PATH=$MVN_HOME/bin:$PATH

  2. 制作 Zeppelin 的 Parcel 包并校验

    注意：

    此脚本默使用Livy-0.5.0 ，Zeppelin-0.8.0版本，要求Python版本2.7/3.3或更高版本。如需修改Zeppelin版本，修改build.sh脚本中下面变量即可：

    ...
# Zeppelin安装包下载路径
ZEPPELIN_URL=http://apache.mirror.anlx.net/zeppelin/zeppelin-0.8.0/zeppelin-0.8.0-bin-all.tgz
# Zeppelin安装包的MD5值下载路径
ZEPPELIN_MD5="ee6875ac47492e37899dea103b34d176"
# 或直接使用下面配置跳过MD5校验
#ZEPPELIN_MD5="$(md5sum $( basename $ZEPPELIN_URL ))"
# Zeppelin 版本
ZEPPELIN_VERSION=0.8.1
...
# Linux系统，将el7修改为对应的系统版本即可
zeppelin_parcel_name="$zeppelin_parcel_folder-el7.parcel"
...

    运行制作parcel 包脚本

    [root@cdh01 ~]# mkdir -p ~/github/cloudera/
[root@cdh01 ~]# cd ~/github/cloudera/ 
[root@cdh01 cloudera]# git clone https://github.com/alexjbush/livy_zeppelin_cdh_csd_parcels.git
[root@cdh01 cloudera]# cd livy_zeppelin_cdh_csd_parcels	
[root@cdh01 livy_zeppelin_cdh_csd_parcels]# sh build.sh parcel

    日志

    ...
+ java -jar cm_ext/validator/target/validator.jar -f ./ZEPPELIN-0.8.1_build/ZEPPELIN-0.8.1-el7.parcel
Validating: ./ZEPPELIN-0.8.1_build/ZEPPELIN-0.8.1-el7.parcel
Validating: ZEPPELIN-0.8.1/meta/parcel.json
Validation succeeded.
+ python cm_ext/make_manifest/make_manifest.py ./ZEPPELIN-0.8.1_build
Scanning directory: ./ZEPPELIN-0.8.1_build
Found parcel ZEPPELIN-0.8.1-el7.parcel

  3. 遇到的问题：

    1. 问题一：

      [ERROR] Plugin org.apache.maven.plugins:maven-jar-plugin:2.3.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-jar-plugin:jar:2.3.1: Could not transfer artifact org.apache.maven.plugins:maven-jar-plugin:pom:2.3.1 from/to central (https://repo.maven.apache.org/maven2): Connect to repo.maven.apache.org:443 [repo.maven.apache.org/151.101.40.215] failed: Connection refused -> [Help 1]

      解决办法：

      修改maven安装目录下的conf/setting.xml文件，将中央仓库地址阿里云镜像地址。

      <mirror>
  <id>alimaven</id>
  <name>aliyun maven</name>
  <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
  <mirrorOf>central</mirrorOf>       
</mirror>

    2. 问题二：

      [INFO] Cloudera Service Schemas ........................... FAILURE [  8.483 s]
[INFO] Schema Validator ................................... SKIPPED
[INFO] Cloudera Manager Extensibility Tools ............... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8.820 s
[INFO] Finished at: 2019-05-26T15:08:45+08:00
[INFO] Final Memory: 11M/57M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project cloudera-manager-schema: Could not resolve dependencies for project com.cloudera.cmf.schema:cloudera-manager-schema:jar:5.12.0: Failed to collect dependencies at commons-cli:commons-cli:jar:1.3-cloudera-pre-r1439998: Failed to read artifact descriptor for commons-cli:commons-cli:jar:1.3-cloudera-pre-r1439998: Could not transfer artifact commons-cli:commons-cli:pom:1.3-cloudera-pre-r1439998 from/to cloudera-external (https://repository.cloudera.com/artifactory/ext-release-local/): Remote host closed connection during handshake: SSL peer shut down incorrectly -> [Help 1]

      解决办法：

      进入maven仓库下，手动动下载对应的commons-cli包。

      [root@cdh01 ~]# cd ~/.m2/repository/
[root@cdh01 repository]# cd commons-cli/commons-cli/1.3-cloudera-pre-r1439998
[root@cdh01 1.3-cloudera-pre-r1439998]# wget https://repository.cloudera.com/artifactory/ext-release-local/commons-cli/commons-cli/1.3-cloudera-pre-r1439998/commons-cli-1.3-cloudera-pre-r1439998.pom
[root@cdh01 1.3-cloudera-pre-r1439998]# wget https://repository.cloudera.com/artifactory/ext-release-local/commons-cli/commons-cli/1.3-cloudera-pre-r1439998/commons-cli-1.3-cloudera-pre-r1439998.jar

    3. 问题三

      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project cloudera-manager-schema: There are test failures.

      解决办法

      进入cm_ext目录下，运行下面打包命令

      mvn clean package -Dmaven.test.skip=true

  4. 制作 Zeppelin 的CSD文件

    [root@cdh01 livy_zeppelin_cdh_csd_parcels]# sh build.sh csd
+ set -e
+ CM_EXT_BRANCH=cm5-5.12.0
+ LIVY_URL=http://apache.mirror.anlx.net/incubator/livy/0.6.0-incubating/apache-livy-0.6.0-incubating-bin.zip
+ LIVY_MD5=55f966dc4b25ba595428dbc4634fac01
+ LIVY_VERSION=0.6.0
+ ZEPPELIN_URL=http://apache.mirror.anlx.net/zeppelin/zeppelin-0.8.1/zeppelin-0.8.1-bin-all.tgz
+ ZEPPELIN_MD5=ee6875ac47492e37899dea103b34d176
+ ZEPPELIN_VERSION=0.8.1
+ livy_service_name=LIVY
++ tr '[:upper:]' '[:lower:]'
++ echo LIVY
+ livy_service_name_lower=livy
++ basename http://apache.mirror.anlx.net/incubator/livy/0.6.0-incubating/apache-livy-0.6.0-incubating-bin.zip
+ livy_archive=apache-livy-0.6.0-incubating-bin.zip
++ basename apache-livy-0.6.0-incubating-bin.zip .zip
+ livy_folder=apache-livy-0.6.0-incubating-bin
+ livy_parcel_folder=LIVY-0.6.0
+ livy_parcel_name=LIVY-0.6.0-el7.parcel
+ livy_built_folder=LIVY-0.6.0_build
+ livy_csd_build_folder=livy_csd_build
+ zeppelin_service_name=ZEPPELIN
++ tr '[:upper:]' '[:lower:]'
++ echo ZEPPELIN
+ zeppelin_service_name_lower=zeppelin
++ basename http://apache.mirror.anlx.net/zeppelin/zeppelin-0.8.1/zeppelin-0.8.1-bin-all.tgz
+ zeppelin_archive=zeppelin-0.8.1-bin-all.tgz
++ basename zeppelin-0.8.1-bin-all.tgz .tgz
+ zeppelin_folder=zeppelin-0.8.1-bin-all
+ zeppelin_parcel_folder=ZEPPELIN-0.8.1
+ zeppelin_parcel_name=ZEPPELIN-0.8.1-el7.parcel
+ zeppelin_built_folder=ZEPPELIN-0.8.1_build
+ zeppelin_csd_build_folder=zeppelin_csd_build
+ case $1 in
+ build_livy_csd
+ JARNAME=LIVY-0.6.0.jar
+ '[' -f LIVY-0.6.0.jar ']'
+ rm -rf livy_csd_build
+ cp -rf ./livy-csd-src livy_csd_build
+ sed -i -e s/%SERVICENAME%/LIVY/ livy_csd_build/descriptor/service.sdl
+ sed -i -e s/%SERVICENAMELOWER%/livy/ livy_csd_build/descriptor/service.sdl
+ sed -i -e s/%SERVICENAMELOWER%/livy/ livy_csd_build/scripts/control.sh
+ java -jar cm_ext/validator/target/validator.jar -s livy_csd_build/descriptor/service.sdl -l 'SPARK_ON_YARN SPARK2_ON_YARN'
Validating: livy_csd_build/descriptor/service.sdl
Validation succeeded.
+ jar -cvf ./LIVY-0.6.0.jar -C livy_csd_build .
added manifest
adding: images/(in = 0) (out= 0)(stored 0%)
adding: images/icon.png(in = 1166) (out= 700)(deflated 39%)
adding: descriptor/(in = 0) (out= 0)(stored 0%)
adding: descriptor/service.sdl(in = 9551) (out= 2390)(deflated 74%)
adding: aux/(in = 0) (out= 0)(stored 0%)
adding: aux/client/(in = 0) (out= 0)(stored 0%)
adding: aux/client/livy-env.sh(in = 21) (out= 23)(deflated -9%)
adding: scripts/(in = 0) (out= 0)(stored 0%)
adding: scripts/control.sh(in = 3598) (out= 1169)(deflated 67%)
+ build_zeppelin_csd
+ JARNAME=ZEPPELIN-0.8.1.jar
+ '[' -f ZEPPELIN-0.8.1.jar ']'
+ rm -rf zeppelin_csd_build
+ cp -rf ./zeppelin-csd-src zeppelin_csd_build
+ sed -i -e s/%SERVICENAME%/ZEPPELIN/ zeppelin_csd_build/descriptor/service.sdl
+ sed -i -e s/%SERVICENAMELOWER%/zeppelin/ zeppelin_csd_build/descriptor/service.sdl
+ sed -i -e s/%LIVYSERVICENAME%/LIVY/ zeppelin_csd_build/descriptor/service.sdl
+ sed -i -e s/%LIVYSERVICENAMELOWER%/livy/ zeppelin_csd_build/descriptor/service.sdl
+ sed -i -e s/%SERVICENAMELOWER%/zeppelin/ zeppelin_csd_build/scripts/control.py
+ sed -i -e s/%LIVYSERVICENAMELOWER%/livy/ zeppelin_csd_build/scripts/control.py
+ java -jar cm_ext/validator/target/validator.jar -s zeppelin_csd_build/descriptor/service.sdl -l LIVY
Validating: zeppelin_csd_build/descriptor/service.sdl
Validation succeeded.
+ jar -cvf ./ZEPPELIN-0.8.1.jar -C zeppelin_csd_build .
added manifest
adding: images/(in = 0) (out= 0)(stored 0%)
adding: images/icon.png(in = 1166) (out= 573)(deflated 50%)
adding: descriptor/(in = 0) (out= 0)(stored 0%)
adding: descriptor/service.sdl(in = 14376) (out= 3057)(deflated 78%)
adding: aux/(in = 0) (out= 0)(stored 0%)
adding: aux/client/(in = 0) (out= 0)(stored 0%)
adding: aux/client/interpreter.livy.json(in = 2362) (out= 600)(deflated 74%)
adding: aux/client/zeppelin-env.sh(in = 21) (out= 23)(deflated -9%)
adding: aux/client/interpreter.json(in = 1753) (out= 518)(deflated 70%)
adding: aux/client/shiro.ini(in = 0) (out= 0)(stored 0%)
adding: scripts/(in = 0) (out= 0)(stored 0%)
adding: scripts/test_control.py(in = 565) (out= 273)(deflated 51%)
adding: scripts/control.py(in = 7640) (out= 2200)(deflated 71%)

  5. 查看Parcel包和csd文件

    查看Parcel包

    [root@hadoop01 livy_zeppelin_cdh_csd_parcels]# ll ZEPPELIN-0.8.1_build LIVY-0.6.0_build
LIVY-0.6.0_build:
total 93748
-rw-r--r--. 1 root root 95993481 May 26 16:30 LIVY-0.6.0-el7.parcel
-rw-r--r--. 1 root root      311 May 26 16:30 manifest.json

ZEPPELIN-0.8.1_build:
total 968732
-rw-r--r--. 1 root root       316 May 26 16:33 manifest.json
-rw-r--r--. 1 root root 991970787 May 26 16:32 ZEPPELIN-0.8.1-el7.parcel

    查看csd文件

    total 1063740
-rw-r--r--. 1 root root   96137660 May 24 20:36 apache-livy-0.6.0-incubating-bin.zip
-rw-r--r--. 1 root root       6362 May 26 14:40 build.sh
drwxr-xr-x. 6 root root       4096 May 26 13:42 cm_ext
-rw-r--r--. 1 root root      63264 May 26 13:23 kerberos.png
drwxr-xr-x. 8 root root       4096 May 26 16:03 LIVY-0.6.0
drwxr-xr-x. 2 root root       4096 May 26 16:30 LIVY-0.6.0_build
-rw-r--r--. 1 root root       5607 May 26 16:36 LIVY-0.6.0.jar
drwxr-xr-x. 6 root root       4096 May 26 16:35 livy_csd_build
drwxr-xr-x. 6 root root       4096 May 26 13:23 livy-csd-src
drwxr-xr-x. 3 root root       4096 May 26 13:23 livy-parcel-src
-rw-r--r--. 1 root root       4391 May 26 13:23 README.md
drwxr-xr-x. 9  503 wheel      4096 May 26 16:31 ZEPPELIN-0.8.1
-rw-r--r--. 1 root root  992975720 May 25 12:25 zeppelin-0.8.1-bin-all.tgz
drwxr-xr-x. 2 root root       4096 May 26 16:33 ZEPPELIN-0.8.1_build
-rw-r--r--. 1 root root       9133 May 26 16:36 ZEPPELIN-0.8.1.jar
drwxr-xr-x. 6 root root       4096 May 26 16:36 zeppelin_csd_build
drwxr-xr-x. 6 root root       4096 May 26 13:23 zeppelin-csd-src
drwxr-xr-x. 3 root root       4096 May 26 13:23 zeppelin-parcel-src

猜你喜欢

转载自blog.csdn.net/weixin_43215250/article/details/90406232
今日推荐
周排行
Access的四舍五入取整
8.23 前端学习过程
入门学习过程方向与漏洞复现总结：
操作分布式文件之八：如何批量并行读写远程文件和事务补偿处理
应邀出个教程 （搭建tensorflow跑网络环境）
Kubernetes之Pod控制器应用进阶
14-[mysql内置功能]--
HDU6212 区间dp 好题
VS2015生成代码图
验证手机号的工具类
每日归档
更多
2024-10-21(0)
2024-10-20(0)
2024-10-19(0)
2024-10-18(0)
2024-10-17(0)
2024-10-16(0)
2024-10-15(0)
2024-10-14(0)
2024-10-13(0)
2024-10-12(0)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022