3.8 合并数据集:合并与连接
pd的基本特性之一就是高性能的内存式数据连接join与合并merge操作。pd的主接口是merge函数。
3.8.1 关系代数
合并的理论基础是关系代数
3.8.2 数据连接的类型
merge实现三种数据连接类型:一对一,多对一,多对多。
import pandas as pd import numpy as np class display(object): """Display HTML representation of multiple objects""" template = """<div style="float: left; padding: 10px;"> <p style='font-family:"Courier New", Courier, monospace'>{0}</p>{1} </div>""" def __init__(self, *args): self.args = args def _repr_html_(self): return '\n'.join(self.template.format(a, eval(a)._repr_html_()) for a in self.args) def __repr__(self): return '\n\n'.join(a + '\n' + repr(eval(a)) for a in self.args)
一对一连接
是最简单的数据合并类型,与3.7节介绍的按列合并十分相似。
df1 = pd.DataFrame({'employee': ['Bob', 'Jake', 'Lisa', 'Sue'], 'group': ['Accounting', 'Engineering', 'Engineering', 'HR']}) df2 = pd.DataFrame({'employee': ['Lisa', 'Bob', 'Jake', 'Sue'], 'hire_date': [2004, 2008, 2012, 2014]}) display('df1', 'df2')
df1
employee | group | |
---|---|---|
0 | Bob | Accounting |
1 | Jake | Engineering |
2 | Lisa | Engineering |
3 | Sue | HR |
df2
employee | hire_date | |
---|---|---|
0 | Lisa | 2004 |
1 | Bob | 2008 |
2 | Jake | 2012 |
3 | Sue | 2014 |
若要将上边两个DF合并为一个,用merge函数:
df3 = pd.merge(df1, df2) df3
employee | group | hire_date | |
---|---|---|---|
0 | Bob | Accounting | 2008 |
1 | Jake | Engineering | 2012 |
2 | Lisa | Engineering | 2004 |
3 | Sue | HR | 2014 |
merge函数自动将两个DF共有的列employee作为键进行连接,生成一个新DF,原来DF的行索引自动丢弃,自动生成新行索引。
多对一连接
这种连接中,在需要连接的两个列中,有一列的值有重复。通过多对一连接的结果DF会保留重复值。如:
df4 = pd.DataFrame({'group': ['Accounting', 'Engineering', 'HR'], 'supervisor': ['Carly', 'Guido', 'Steve']}) display('df3', 'df4', 'pd.merge(df3, df4)')
df3
employee | group | hire_date | |
---|---|---|---|
0 | Bob | Accounting | 2008 |
1 | Jake | Engineering | 2012 |
2 | Lisa | Engineering | 2004 |
3 | Sue | HR | 2014 |
df4
group | supervisor | |
---|---|---|
0 | Accounting | Carly |
1 | Engineering | Guido |
2 | HR | Steve |
pd.merge(df3, df4)
employee | group | hire_date | supervisor | |
---|---|---|---|---|
0 | Bob | Accounting | 2008 | Carly |
1 | Jake | Engineering | 2012 | Guido |
2 | Lisa | Engineering | 2004 | Guido |
3 | Sue | HR | 2014 | Steve |
在结果的DF中多了一个supervisor列,里面有些值会因为输入数据的对应关系而有所重复。
多对多连接
如果左右两个输入的共同列都包含重复值,那么合并结果就是一种多对多连接,如:
df5 = pd.DataFrame({'group': ['Accounting', 'Accounting', 'Engineering', 'Engineering', 'HR', 'HR'], 'skills': ['math', 'spreadsheets', 'coding', 'linux', 'spreadsheets', 'organization']}) display('df1', 'df5', "pd.merge(df1, df5)")
df1
employee | group | |
---|---|---|
0 | Bob | Accounting |
1 | Jake | Engineering |
2 | Lisa | Engineering |
3 | Sue | HR |
df5
group | skills | |
---|---|---|
0 | Accounting | math |
1 | Accounting | spreadsheets |
2 | Engineering | coding |
3 | Engineering | linux |
4 | HR | spreadsheets |
5 | HR | organization |
pd.merge(df1, df5)
employee | group | skills | |
---|---|---|---|
0 | Bob | Accounting | math |
1 | Bob | Accounting | spreadsheets |
2 | Jake | Engineering | coding |
3 | Jake | Engineering | linux |
4 | Lisa | Engineering | coding |
5 | Lisa | Engineering | linux |
6 | Sue | HR | spreadsheets |
7 | Sue | HR | organization |
这三种数据连接类型可以直接与其他pd工具组合使用,从而实现各种功能。但工作的真是数据集往往不如例子的数据那样干净整洁,下面介绍更多merge功能来更好应对数据连接中的问题。
3.8.3 设置数据合并的键
merge默认将两个输入的一个或多个同名的列作为键进行合并,但由于两个输入要合并的列通常不同名,因此merge提供参数解决这个问题。
参数on的用法
最简单的方法就是直接将参数on设置为一个列名字符串或者一个包含多列名称的列表:
display('df1', 'df2', "pd.merge(df1, df2, on='employee')")
df1
employee | group | |
---|---|---|
0 | Bob | Accounting |
1 | Jake | Engineering |
2 | Lisa | Engineering |
3 | Sue | HR |
df2
employee | hire_date | |
---|---|---|
0 | Lisa | 2004 |
1 | Bob | 2008 |
2 | Jake | 2012 |
3 | Sue | 2014 |
pd.merge(df1, df2, on='employee')
employee | group | hire_date | |
---|---|---|---|
0 | Bob | Accounting | 2008 |
1 | Jake | Engineering | 2012 |
2 | Lisa | Engineering | 2004 |
3 | Sue | HR | 2014 |
这个参数只能在两个DF有共同列名的时候才可以使用。
left_on和right_on参数
有时候也要合并两个列名不同的数据集,这种情况下就可以用left_on和right_on参数来指定列名:
df3 = pd.DataFrame({'name': ['Bob', 'Jake', 'Lisa', 'Sue'], 'salary': [70000, 80000, 120000, 90000]}) display('df1', 'df3', 'pd.merge(df1, df3, left_on="employee", right_on="name")')
df1
employee | group | |
---|---|---|
0 | Bob | Accounting |
1 | Jake | Engineering |
2 | Lisa | Engineering |
3 | Sue | HR |
df3
name | salary | |
---|---|---|
0 | Bob | 70000 |
1 | Jake | 80000 |
2 | Lisa | 120000 |
3 | Sue | 90000 |
pd.merge(df1, df3, left_on="employee", right_on="name")
employee | group | name | salary | |
---|---|---|---|---|
0 | Bob | Accounting | Bob | 70000 |
1 | Jake | Engineering | Jake | 80000 |
2 | Lisa | Engineering | Lisa | 120000 |
3 | Sue | HR | Sue | 90000 |
获取的结果中会有一个多余的列,可通过DF的drop方法将其去掉:
pd.merge(df1, df3, left_on="employee", right_on="name").drop('name', axis=1)
employee | group | salary | |
---|---|---|---|
0 | Bob | Accounting | 70000 |
1 | Jake | Engineering | 80000 |
2 | Lisa | Engineering | 120000 |
3 | Sue | HR | 90000 |
left_index和right_index参数
除了合并列之外,有时候还需要合并索引:
df1a = df1.set_index('employee') df2a = df2.set_index('employee') display('df1a', 'df2a')
df1a
group | |
---|---|
employee | |
Bob | Accounting |
Jake | Engineering |
Lisa | Engineering |
Sue | HR |
df2a
hire_date | |
---|---|
employee | |
Lisa | 2004 |
Bob | 2008 |
Jake | 2012 |
Sue | 2014 |
可通过merge中left_index和//或right_index参数将索引设置为键来实现合并:
display('df1a', 'df2a', "pd.merge(df1a, df2a, left_index=True, right_index=True)")
df1a
group | |
---|---|
employee | |
Bob | Accounting |
Jake | Engineering |
Lisa | Engineering |
Sue | HR |
df2a
hire_date | |
---|---|
employee | |
Lisa | 2004 |
Bob | 2008 |
Jake | 2012 |
Sue | 2014 |
pd.merge(df1a, df2a, left_index=True, right_index=True)
group | hire_date | |
---|---|---|
employee | ||
Bob | Accounting | 2008 |
Jake | Engineering | 2012 |
Lisa | Engineering | 2004 |
Sue | HR | 2014 |
为了方便考虑,DF实现了join方法,可以按照索引进行数据合并:
display('df1a', 'df2a', 'df1a.join(df2a)')
df1a
group | |
---|---|
employee | |
Bob | Accounting |
Jake | Engineering |
Lisa | Engineering |
Sue | HR |
df2a
hire_date | |
---|---|
employee | |
Lisa | 2004 |
Bob | 2008 |
Jake | 2012 |
Sue | 2014 |
df1a.join(df2a)
group | hire_date | |
---|---|---|
employee | ||
Bob | Accounting | 2008 |
Jake | Engineering | 2012 |
Lisa | Engineering | 2004 |
Sue | HR | 2014 |
如果想将索引与列混合使用,那可以通过结合left_index与right_on,或结合left_on与right_index来实现:
display('df1a', 'df3', "pd.merge(df1a, df3, left_index=True, right_on='name')")
df1a
group | |
---|---|
employee | |
Bob | Accounting |
Jake | Engineering |
Lisa | Engineering |
Sue | HR |
df3
name | salary | |
---|---|---|
0 | Bob | 70000 |
1 | Jake | 80000 |
2 | Lisa | 120000 |
3 | Sue | 90000 |
pd.merge(df1a, df3, left_index=True, right_on='name')
group | name | salary | |
---|---|---|---|
0 | Accounting | Bob | 70000 |
1 | Engineering | Jake | 80000 |
2 | Engineering | Lisa | 120000 |
3 | HR | Sue | 90000 |
当然这些参数都适用于多个索引和多个列名。
3.8.4 设置数据连接的集合操作规则
集合操作规则是数据连接的一个重要条件。当一个值出现在一列而没有出现在另一列,就要考虑聚合操作规则了,如:
df6 = pd.DataFrame({'name': ['Peter', 'Paul', 'Mary'], 'food': ['fish', 'beans', 'bread']}, columns=['name', 'food']) df7 = pd.DataFrame({'name': ['Mary', 'Joseph'], 'drink': ['wine', 'beer']}, columns=['name', 'drink']) display('df6', 'df7', 'pd.merge(df6, df7)')
df6
name | food | |
---|---|---|
0 | Peter | fish |
1 | Paul | beans |
2 | Mary | bread |
df7
name | drink | |
---|---|---|
0 | Mary | wine |
1 | Joseph | beer |
pd.merge(df6, df7)
name | food | drink | |
---|---|---|---|
0 | Mary | bread | wine |
合并两个数据集,在name列中只有一条共同的值Mary。默认情况下结果只会包含两个输入集合的交集,这种连接方式为内连接,可用参数how设置连接方式,默认就是内连接inner:
pd.merge(df6, df7, how='inner')
name | food | drink | |
---|---|---|---|
0 | Mary | bread | wine |
how参数支持的数据连接方式还有 outer,left,right。
外连接outer返回两个输入集合的并集,所有缺失值都用NaN填充:
display('df6', 'df7', "pd.merge(df6, df7, how='outer')")
df6
name | food | |
---|---|---|
0 | Peter | fish |
1 | Paul | beans |
2 | Mary | bread |
df7
name | drink | |
---|---|---|
0 | Mary | wine |
1 | Joseph | beer |
pd.merge(df6, df7, how='outer')
name | food | drink | |
---|---|---|---|
0 | Peter | fish | NaN |
1 | Paul | beans | NaN |
2 | Mary | bread | wine |
3 | Joseph | NaN | beer |
左连接left和右连接right返回的结果分别只包含左列和右列,如:
display('df6', 'df7', "pd.merge(df6, df7, how='left')")
df6
name | food | |
---|---|---|
0 | Peter | fish |
1 | Paul | beans |
2 | Mary | bread |
df7
name | drink | |
---|---|---|
0 | Mary | wine |
1 | Joseph | beer |
pd.merge(df6, df7, how='left')
name | food | drink | |
---|---|---|---|
0 | Peter | fish | NaN |
1 | Paul | beans | NaN |
2 | Mary | bread | wine |
现在输出的行中只包含左边输入列的值。如果用how='right'的话,输出的行则只包含右边输入列的值。
3.8.5 重复列名:suffixes 参数
最后,可能会遇到两个输入DF有重名列的情况,如:
df8 = pd.DataFrame({'name': ['Bob', 'Jake', 'Lisa', 'Sue'], 'rank': [1, 2, 3, 4]}) df9 = pd.DataFrame({'name': ['Bob', 'Jake', 'Lisa', 'Sue'], 'rank': [3, 1, 4, 2]}) display('df8', 'df9', 'pd.merge(df8, df9, on="name")')
df8
name | rank | |
---|---|---|
0 | Bob | 1 |
1 | Jake | 2 |
2 | Lisa | 3 |
3 | Sue | 4 |
df9
name | rank | |
---|---|---|
0 | Bob | 3 |
1 | Jake | 1 |
2 | Lisa | 4 |
3 | Sue | 2 |
pd.merge(df8, df9, on="name")
name | rank_x | rank_y | |
---|---|---|---|
0 | Bob | 1 | 3 |
1 | Jake | 2 | 1 |
2 | Lisa | 3 | 4 |
3 | Sue | 4 | 2 |
由于输出结果中有两个重复的列名,因此merge函数自动给其添加了后缀_x,_y。当然也可以通过suffixes参数自定义后缀名:
display('df8', 'df9', 'pd.merge(df8, df9, on="name", suffixes=["_L", "_R"])')
df8
name | rank | |
---|---|---|
0 | Bob | 1 |
1 | Jake | 2 |
2 | Lisa | 3 |
3 | Sue | 4 |
df9
name | rank | |
---|---|---|
0 | Bob | 3 |
1 | Jake | 1 |
2 | Lisa | 4 |
3 | Sue | 2 |
pd.merge(df8, df9, on="name", suffixes=["_L", "_R"])
name | rank_L | rank_R | |
---|---|---|---|
0 | Bob | 1 | 3 |
1 | Jake | 2 | 1 |
2 | Lisa | 3 | 4 |
3 | Sue | 4 | 2 |
suffixes参数同样适合于任何连接方式,即使有三个或以上的重复列名时也同样适用。
3.8.6 例子:美国各州的统计数据
略