首先来看如何把数据利用datax-web,通过datax3.0,把数据从postgresql同步到mysql
这里也是给出对应的datax需要的json文件
首先去data-web构建一下需要的json,这里先添加数据源,数据源这里我添加了mysql的两个,postgresql的一个,hbase的加不上,其他博文说了,hbase我们就自己写到json中就可以了,这个有提供的模板,到时候直接复制就可以.
记得Json的底部需要加上:
,
"core": {
"transport": {
"channel": {
"speed": {
"byte": 1231412
}
}
}
}
来看一下完整的JSON
{
"job": {
"setting": {
"speed": {
"channel": 3,
"byte": 1048576
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "postgresqlreader",
"parameter": {
"username": "XVko54UY9nOe/3JQGQUikw==",
"password": "XCYVpFosvZBBWobFzmLWvA==",
"column": [
"\"id\"",
"\"name\"",
"\"mobile\"",
"\"email\"",
"\"son_json\""
],
"splitPk": "",
"connection": [
{
"table": [
"public.userinfo"
],
"jdbcUrl": [
"jdbc:postgresql://172.19.126.122:5432/test_db"
]
}
]
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"username": "yRjwDFuoPKlqya9h9H2Amg==",
"password": "XCYVpFosvZBBWobFzmLWvA==",
"column": [
"`id`",
"`name`",
"`email`",
"`mobile`",
"`son_json`"
],
"connection": [
{
"table": [
"userinfo"
],
"jdbcUrl": "jdbc:mysql://172.19.126.123:3306/nifibase?rewriteBatchedStatements=true&useUnicode=true&characterEncoding=UTF8&serverTimezone=Asia/Shanghai&useSSL=false&verifyServerCertificate=false&autoReconnect=true&failOverReadOnly=false"
}
]
}
}
}
]
},
"core": {
"transport": {
"channel": {
"speed": {
"byte": 1231412
}
}
}
}
}
这里执行就可以把数据传递过来了
---------------------
然后我们再来看将Postgresql中的数据通过datax-web导入到hbase中,这里利用Phoenix实现
这里也是给出对应的JSON文件就可以了
{
"job": {
"setting": {
"speed": {
"channel": 3,
"byte": 1048576
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "postgresqlreader",
"parameter": {
"username": "XVko54UY9nOe/3JQGQUikw==",
"password": "XCYVpFosvZBBWobFzmLWvA==",
"column": [
"\"id\"",
"\"name\"",
"\"mobile\"",
"\"email\"",
"\"son_json\""
],
"splitPk": "",
"connection": [
{
"table": [
"public.userinfo"
],
"jdbcUrl": [
"jdbc:postgresql://172.19.126.122:5432/test_db"
]
}
]
}
},
"writer": {
"name": "hbase20xsqlwriter",
"parameter": {
"batchSize": "100",
"column": [
"id",
"name",
"mobile",
"email",
"son_json"
],
"queryServerAddress": "http://127.0.0.1:8765",
"nullMode": "skip",
"table": "userinfo"
}
}
}
]
},
"core": {
"transport": {
"channel": {
"speed": {
"byte": 1231412
}
}
}
}
}
构建任务的话,这里就构建一个postgresql到mysql的,然后得到json以后再修改就可以了,这里直接给出对应的可以执行json
首先去执行对应的任务
先来清空一下hbase中的userinfo这个表,这里我用的squirrel来通过Phoenix操作的hbase
然后执行同步
执行以后可以看到已经同步过来了.
------------------------------
然后我们再来看一下如何通过datax-web来实现,从postgresql中读取数据然后同时存入到hbase表中和mysql表中去
首先我们清空一下mysql表中的数据
然后再清空一下hbase表中的数据
然后看一下json