一、背景
当前在用Azkaban 3.81.4版本,使用了最简单的 azkaban-solo-server
启动,默认使用自带的嵌入式 h2
数据库,自2019年12月启用以来,执行历史数据一直存储在潜入的 h2
数据库中,已经发生了几次 azkaban-solo-server
进程假死的情况,故准备清理 h2
中的历史数据,恢复访问性能。
二、准备
-
通过阅读 azkaban 源代码得知,azkaban 支持
h2
和mysql
两种数据库,h2
连接数据库的代码 H2FileDataSource.java 显示使用相对路径读取h2
数据库文件,未设置账号或密码。 -
azkaban 源代码位置显示出大约有一下表存储了执行历史记录:
execution_dependencies execution_flows execution_jobs execution_logs executor_events executors
-
h2 官网查询得知可以使用 h2 提供的 JDBC jar 包开启命令行访问模式。
三、执行清理
-
停止
azkaban-solo-server
服务$ bin/shutdown-solo.sh Killing solo-server. [pid: 5268], attempt: 1 shutdown succeeded
-
启用 shell 方式访问
azkaban
数据库h2
,执行命令java -cp lib/h2-1.4.193.jar org.h2.tools.Shell
,在提示URL
参数时,输入jdbc:h2:./h2;IGNORECASE=TRUE
,用户和密码留空,直接回车跳过,密码会需要执行两次回车$ java -cp lib/h2-1.4.193.jar org.h2.tools.Shell Welcome to H2 Shell 1.4.193 (2016-10-31) Exit with Ctrl+C [Enter] jdbc:h2:~/test URL jdbc:h2:./h2;IGNORECASE=TRUE [Enter] org.h2.Driver Driver [Enter] User [Enter] Hide Password Password Connected Commands are case insensitive; SQL statements end with ';' help or ? Display this help list Toggle result list / stack trace mode maxwidth Set maximum column width (default is 100) autocommit Enable or disable autocommit history Show the last 20 statements quit or exit Close the connection and exit
-
查看表清单
sql> show tables; TABLE_NAME | TABLE_SCHEMA ACTIVE_EXECUTING_FLOWS | PUBLIC ACTIVE_SLA | PUBLIC EXECUTION_DEPENDENCIES | PUBLIC EXECUTION_FLOWS | PUBLIC EXECUTION_JOBS | PUBLIC EXECUTION_LOGS | PUBLIC EXECUTORS | PUBLIC EXECUTOR_EVENTS | PUBLIC PROJECTS | PUBLIC PROJECT_EVENTS | PUBLIC PROJECT_FILES | PUBLIC PROJECT_FLOWS | PUBLIC PROJECT_FLOW_FILES | PUBLIC PROJECT_PERMISSIONS | PUBLIC PROJECT_PROPERTIES | PUBLIC PROJECT_VERSIONS | PUBLIC PROPERTIES | PUBLIC RAMP | PUBLIC RAMP_DEPENDENCY | PUBLIC RAMP_EXCEPTIONAL_FLOW_ITEMS | PUBLIC RAMP_EXCEPTIONAL_JOB_ITEMS | PUBLIC RAMP_ITEMS | PUBLIC TRIGGERS | PUBLIC VALIDATED_DEPENDENCIES | PUBLIC (24 rows, 100 ms)
-
EXECUTION_LOGS
数据特征,其中LOG
字段是二进制字段,日志文件作为二进制内容存储在该字段内,UPLOAD_TIME
设定成了 bigint 存储的格林威治时间偏移量,想要按照时间范围清空历史记录的计划落空sql> select * from EXECUTION_LOGS limit 2; EXEC_ID | NAME | ATTEMPT | ENC_TYPE | START_BYTE | END_BYTE | LOG | UPLOAD_TIME 15257 | azkaban-import-ameba-data | 0 | 2 | 0 | 9943 | 1f8b0800000000000000cd9add6edb3614c7effb14bccb4d65cbfa486c03bd589c0468b73441936c17c320d01265b3914485 | 1659546382158 15257 | azkaban-calc-kpi | 0 | 2 | 0 | 51200 | 1f8b0800000000000000ed7d5b8f5c4792def3fa571c631610057755e7fdd2828c695e6451235234498dbc10063d79254baa | 1659546833421 (data is partially truncated) (2 rows, 14 ms)
-
EXECUTION_JOBS
数据特征,START_TIME
、END_TIME
两个字段仍然存储的是当前时间至格林威治时间偏移量,无法按照时间范围清空历史记录,决定清空 EXECUTION* 系列表数据sql> select * from EXECUTION_JOBS limit 20; EXEC_ID | PROJECT_ID | VERSION | FLOW_ID | JOB_ID | ATTEMPT | START_TIME | END_TIME | STATUS | INPUT_PARAMS | OUTPUT_PARAMS | ATTACHMENTS 1 | 1 | 1 | ameba-kpi-task-detail | ameba-kpi-task-detail | 0 | 1576228867567 | 1576228917591 | 70 | 1f8b080000000000000085534d6fdb300cfd2b81809e16d996932c8e4f2bb0e3805d7ad969a525ba566349863ed6a545fffb | null | null 2 | 3 | 1 | azkaban-calc-task | azkaban-calc-task | 0 | 1576467631089 | 1576467649666 | 60 | 1f8b08000000000000008553cb6ea43010fc95c8670c66605edc22ed718fb9af1abb139cc136f2235912e5dfd71e462b7680 | null | null 3 | 3 | 2 | azkaban-calc-task | azkaban-import-data | 0 | 1576560812589 | 1576561474706 | 50 | 1f8b08000000000000008553c96edb3010fd95826753fb62e916a0c71c732f46141d3112977269aa04f9f70e6537301c19b9 | 1f8b0800000000000000ab562ace2f2d4a4e55b2ca2bcdc9d1512a28ca2f2856b2aaaead050091a43bf81a000000 | null
-
执行数据清空
sql> truncate table EXECUTION_DEPENDENCIES; (Update count: 0, 2 ms) sql> truncate table EXECUTION_JOBS; (Update count: 0, 2405 ms) sql> truncate table EXECUTION_LOGS; (Update count: 0, 3851 ms) sql> truncate table EXECUTION_FLOWS; (Update count: 0, 1192 ms) sql> truncate table EXECUTORS; (Update count: 0, 3 ms) sql> truncate table EXECUTOR_EVENTS; (Update count: 0, 0 ms) sql> commit; (Update count: 0, 2 ms) sql> quit; Connection closed
-
清理前后
h2
数据库文件大小对比清理前,数据库文件大小 659M
$ ll -h -rw-r--r-- 1 root root 659M 10月 26 10:21 h2.mv.db -rw-r--r-- 1 root root 9.7M 10月 26 10:21 h2.trace.db
清理后,数据库文件大小 11M
$ ll -h -rw-r--r-- 1 root root 11M 10月 26 14:21 h2.trace.db -rw-r--r-- 1 root root 284K 10月 26 14:22 h2.mv.db
-
启动
azkaban-solo-server
服务,注意 azkaban 配置文件中 h2 数据库配置的相对路径,bin/start-solo.sh
和./start-solo.sh
启动方式不一样,会导致数据库文件所在位置不同,以下固定使用bin/start-solo.sh
启动方式$ bin/start-solo.sh