最近用到elasticsearch-jdbc插件对数据进行导入,期间遇到一些小问题,进行记录下。
bin目录下的sh脚本配置如下:
bin=/e01/elasticsearch-jdbc-2.3.4.0/bin
lib=/e01/elasticsearch-jdbc-2.3.4.0/lib
DATE=$(date +"%Y-%m-%d %H:%M:%S" -d "5 minutes ago")
#bin=$JDBC_IMPORTER_HOME/bin
#lib=$JDBC_IMPORTER_HOME/lib
echo '{
"type" : "jdbc",
"jdbc" : {
"detect_json":false,
"elasticsearch.autodiscover":true,
"elasticsearch.cluster":"elasticsearch",
"url" : "jdbc:mysql://10.4.32.79:3306/hawkeye_tenant",
"user" : "tcl",
"password" : "tcl123",
"sql" : "select id as _id,tid,skill_id,shop_id,cpro_id,shopids,worksn,addresspath,customername,customertel,bigclass,proscate_id,caller,createtime,updatetime,transfertime,case_code,case_id,engine_id,source_objid,dealstate,dealsource,worktype,sendnum,istransferwork,asktime,askdate,accepttime,departuretime,reachtime,finishtime,replytime,closetime,closereason,planstarttime,planendtime,changenum,create_username,brand_id,probcate_id,dembcate_id,case_level,business_id,phoneext,sms_receipt,cast(extfield as char) as extfield,replysource,isrepair from hw_workorder where updatetime >= \"2016-01-07 00:00:00\"",
"elasticsearch" : {
"host" : "10.4.66.56",
"port" : 9300
},
"index" : "hawkeye_tenant4",
"type" : "hw_workorder4",
"threadpoolsize" : 8
}
}' | java \
-cp "${lib}/*" \
-Dlog4j.configurationFile=${bin}/log4j2.xml \
org.xbib.tools.Runner \
org.xbib.tools.JDBCImporter
这里需要注意的是
1."detect_json":false 不对数据库的extfield的大json进行解析为对象。
2.在canal中获取数据库变更数据对es进行插入时要保持对时间字段格式的一致。