Sqoop将HBase数据导出到mysql
sqoop不支持直接从hbase表中导出数据,但是可以借助hive来实现,具体步骤如下:
hbase表结构和数据如下所示
1. 通过Hive创建基于HBase表所对应的外部表
CREATE EXTERNAL TABLE goods_ext1 (
id int,
fname string,
fedsc string
)row format delimited fields terminated by '\t'
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
"hbase.columns.mapping" = ":key,cf-g:fname,cf-g:fdesc")
TBLPROPERTIES( "hbase.table.name" = "goods",
"hbase.mapred.output.outputtable" = "goods");
注意:一定要加分隔符row format delimited fields terminated by ‘\t’
创建成功后,hive外部表数据如下:
2. 创建对应的Hive内部表
CREATE TABLE goods_inner (
id int,
fname string,
fdesc string)row format delimited fields terminated by '\t' lines terminated by '\n';
注意:分隔符一定要加!!!
row format delimited fields terminated by ‘\t’ lines terminated by ‘\n’;
3. 将Hive外部表的数据加载到Hive内部表中
INSERT OVERWRITE TABLE goods_inner SELECT * FROM goods_ext1;
加载成功应如下所示
加载到内部表的数据如下所示:是有分隔符的,方便下一步操作
如果在建表时候不加分隔符数据会全部聚集到一起,并且在下一步可能会出现如下错误:
4. 将Hive内部表中数据导出到数据库提前创建好的目标表中
创建mysql目标表
CREATE TABLE goods_m (
id int,
fname string,
fdesc string)
charset utf8 collate utf8_general_ci;
导出数据:
sqoop export --connect jdbc:mysql://lingyun2:3306/niit?useSSL=false --username root --password 123456 -m 1 --table goods_m --export-dir hdfs://lingyun2:9000//user/hive/warehouse/niit.db/goods_inner --input-null-string "\\\\N" --input-null-non-string "\\\\N" --fields-terminated-by '\t'
导出成功如下所示: