【Hive】使用临时表保留全量数据

hive使用临时表保留全量数据

需求:
在hive环境下,a表为全量表,b表为增量表(只有当天跑的数据),

假设需要将a表中有的但b表中没有的数据仍然保留在a表,

而且需要将b表中有的但a表中没有的数据追加到a表


方案一:
使用左外关联先将a表中有的数据但b表中没有的数据过滤出来,
然后再将b表的数据与过滤出来的数据合并
---------------------创建数据(在oracle演示)

--查询b表在a表的信息
with a as(
select 1 as id, 'Lisi' as name ,'2019-10-01' as time from dual
union all
select 2 as id, 'Wangmen' as name,'2019-10-01' as time from dual
union all
select 3 as id, 'Zhaoliu' as name,'2019-10-01' as time from dual
union all
select 4 as id, 'Pangsan' as name,'2019-10-01' as time from dual
),
b as(
select 1 as id, 'Lisi' as name,'2019-10-03' as time from dual
union all
select 2 as id, 'Wangmen' as name,'2019-10-03' as time from dual
union all
select 5 as id, 'Huangsan' as name,'2019-10-03' as time from dual
)

--使用连接
select a.id, a.name,a.time
  from a
  left join b
    on a.id = b.id
 where b.id is null
union all
select b.id,b.name,b.time
  from b
;

方案二:
先将a、b表的数据合并,
然后使用分析函数row_number()进行排序,将重复的数据进行分组排序,重复的数据只保留时间最新的那一份数据即可
---------------------创建数据(在oracle演示)

--查询b表在a表的信息
with a as(
select 1 as id, 'Lisi' as name ,'2019-10-01' as time from dual
union all
select 2 as id, 'Wangmen' as name,'2019-10-01' as time from dual
union all
select 3 as id, 'Zhaoliu' as name,'2019-10-01' as time from dual
union all
select 4 as id, 'Pangsan' as name,'2019-10-01' as time from dual
),
b as(
select 1 as id, 'Lisi' as name,'2019-10-02' as time from dual
union all
select 2 as id, 'Wangmen' as name,'2019-10-02' as time from dual
union all
select 5 as id, 'Huangsan' as name,'2019-10-02' as time from dual
)

--使用连接
SELECT id
      ,NAME
      ,TIME
      ,rr
  FROM (SELECT id
              ,NAME
              ,TIME
              ,row_number() over(PARTITION BY id ORDER BY TIME DESC) AS rr
          FROM (SELECT a.id
                      ,a.name
                      ,a.time
                  FROM a a
                UNION ALL
                SELECT b.id
                      ,b.name
                      ,b.time
                  FROM b b) c) d
 WHERE d.rr = 1
;

因数据量小为得出哪个方案比较好,后续关注。

猜你喜欢

转载自blog.csdn.net/debimeng/article/details/103723739