Azkaban笔记 1.下载 https://github.com/azkaban/azkaban/releases 本次下载的是最新3.80.0版本。3.80.0.tar.gz 2.上传服务器,解压 3.80.0.tar.gz解压重命名/home/hadoop/azkaban 3.修改gradle配置国内镜像 vi azkaban/build.gradle repositories { maven { url 'http://maven.aliyun.com/nexus/content/groups/public/' } maven { url 'http://maven.aliyun.com/nexus/content/repositories/jcenter'} } 4.Mysql配置 vi /home/hadoop/azkaban/azkaban-exec-server/src/main/resources/conf/azkaban.properties vi /home/hadoop/azkaban/azkaban-web-server/src/main/resources/conf/azkaban.properties mysql.user=<username> mysql.password=<password> 5.利用gradle进行安装(要提前安装git) cd /home/hadoop/azkaban ./gradlew build ./gradlew clean ./gradlew installDist ./gradlew test ./gradlew build -x test 6.创建数据库azkaban,导入sql mysql > create database azkaban; cd /home/hadoop/azkaban/azkaban-db ../gradlew build installDist cd /home/hadoop/azkaban/azkaban-db/build/sql mysql -h192.168.15.45 -P3306 -uroot -p mysql > source create-all-sql-0.1.0-SNAPSHOT.sql; 7.启动azkaban-exec-server cd /home/hadoop/azkaban/azkaban-exec-server/build/install/azkaban-exec-server ./bin/start-exec.sh 8.启动azkaban-web-server cd /home/hadoop/azkaban/azkaban-web-server/build/install/azkaban-web-server ./bin/start-web.sh 报错azkaban.executor.ExecutorManagerException: No active executors found 没有找到活动的executors,需在MySQL数据库里设置端口为12321的executors表的active为1 进入mysql,update azkaban.executors set active=1; 9.web访问账号配置 azkaban-users.xml <azkaban-users> <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/> <user password="metrics" roles="metrics" username="metrics"/> <role name="admin" permissions="ADMIN"/> <role name="metrics" permissions="METRICS"/> </azkaban-users> 10.示例-1 #bar.job type=command dependencies=foo command=echo bar #foo.job type=command command=echo foo 11.示例-2 #hdfs.job type=command command=/home/hadoop/hadoop/bin/hadoop fs -mkdir /azkaban 12.示例-3 #mkdir.job type=command command=/home/hadoop/hadoop/bin/hadoop fs -mkdir -p /wordcount/input/ #putfile.job type=command dependencies=mkdir command=/home/hadoop/hadoop/bin/hadoop fs -put someword.txt /wordcount/input/someword.txt #mrwc.job type=command dependencies=putfile command=/home/hadoop/hadoop/bin/hadoop jar wordcount8.jar com.kizzle.hadoop.mpreduce.wc.WordCount someword.txt wordcount8.jar 13.示例-4 #hive.job type=command command=/home/hadoop/hive/bin/hive -f "hive_test.hql" =========>hive_test.hql use default; create table if not exists azkaban( id int, name string) row format delimited fields terminated by ','; load data local inpath 'azkaban.txt' overwrite into table azkaban; ========>azkaban.txt 1,zhangsan 2,lisi 3,wangwu 4,zhaoliu 5,qianqi 6,wangba
补充激活executor的端口号,
curl -G "hadoop01:40471/executor?action=activate" && echo