os: centos 7.4
db: postgresql 9.6
TOAST的全称是: 超尺寸属性存储技术(The Oversized-Attribute Storage Technique)。
PostgreSQL使用固定的页面尺寸(通常是8kB),并且不允许元组跨越多个额页面,因此不可能直接存储非常大的域值。
为了克服这个限制,大的域值会被压缩并/或分解成多个物理行。这些处理对用户都是透明的,只是在大部分的后端代码上有一些小的影响。
这个技术的昵称是TOAST(或者"切片面包之后的最好的东西")。
TOAST 机制也被用来提升内存中大型数据值的处理。
线外磁盘上 TOAST 存储
TOAST代码代码识别四种不同的在磁盘上存储可TOAST列的策略:
PLAIN 避免压缩或者线外存储;而且它禁用变长类型的单字节头部。这是不可TOAST数据类型列的唯一可能的策略。只是对那些不能TOAST的数据类型才有可能。
EXTENDED 允许压缩和线外存储。这是大多数可TOAST数据类型的默认策略。 首先将尝试进行压缩,如果行仍然太大,那么则进行线外存储。
EXTERNAL 允许线外存储,但是不许压缩。使用EXTERNAL将令那些在宽text和 bytea列上的子串操作更快(代价是增加了存储空间), 因此这些操作被优化为只抓取未压缩线外数据中需要的部分。
MAIN 允许压缩,但不允许线外存储(实际上,在这样的列上仍然会进行线外存储,但只是作为没有办法把行变得足以放入一页的情况下的最后手段)。
每个可TOAST的数据类型都为该数据类型的列指定了一个缺省策略, 但是一个给定表的列的存储策略可以用ALTER TABLE SET STORAGE修改。
实践
postgres=# create table tmp_t0(c0 varchar(100),c1 text);
CREATE TABLE
postgres=# insert into tmp_t0 select proname,prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc||prosrc from pg_proc where char_length(prosrc) > 1000;
INSERT 0 1
postgres=# insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
postgres=#
postgres=# \d+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+--------+-------+----------+-------+-------------
public | tmp_t0 | table | postgres | 72 kB |
(1 row)
postgres=# \d+ tmp_t0
Table "public.tmp_t0"
Column | Type | Modifiers | Storage | Stats target | Description
--------+------------------------+-----------+----------+--------------+-------------
c0 | character varying(100) | | extended | |
c1 | text | | extended | |
接下来查看 pg_class
postgres=# select pg_relation_filenode(pc.relname::varchar) as fs_filenode,
pg_relation_filepath(pc.relname::varchar) as fs_filepath,
pc.oid,pc.relname,pc.relfilenode,
pc.relpages,pc.reltuples,
pc.reltoastrelid
from pg_class pc
where 1=1
and pc.relname='tmp_t0'
and pc.reltoastrelid <> 0
;
fs_filenode | fs_filepath | oid | relname | relfilenode | relpages | reltuples | reltoastrelid
-------------+------------------+-------+---------+-------------+----------+-----------+---------------
25177 | base/13323/25177 | 25177 | tmp_t0 | 25177 | 0 | 0 | 25180
(1 row)
postgres=# select pg_relation_filenode(pc.relname::varchar) as fs_filenode,
pg_relation_filepath(pc.relname::varchar) as fs_filepath,
pc.oid,pc.relname,pc.relfilenode,
pc.relpages,pc.reltuples,
pc.reltoastrelid,
'####'::varchar as flag,
pc1.oid as toast_oid,
pc1.relname as toast_relname,
pc1.reltoastrelid as toast_reltoastrelid
from pg_class pc,
pg_class pc1
where 1=1
and pc.relname='tmp_t0'
and pc.reltoastrelid <> 0
and pc.reltoastrelid=pc1.oid
;
fs_filenode | fs_filepath | oid | relname | relfilenode | relpages | reltuples | reltoastrelid | flag | toast_oid | toast_relname | toast_reltoastrelid
-------------+------------------+-------+---------+-------------+----------+-----------+---------------+------+-----------+----------------+---------------------
25177 | base/13323/25177 | 25177 | tmp_t0 | 25177 | 0 | 0 | 25180 | #### | 25180 | pg_toast_25177 | 0
(1 row)
继续多插入些数据
postgres=# insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
postgres=# \d+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+--------+-------+----------+-------+-------------
public | tmp_t0 | table | postgres | 94 MB |
(1 row)
postgres=# select t0.c0,char_length(t0.c1) from tmp_t0 t0 order by char_length(t0.c1) desc limit 5;
c0 | char_length
----------+-------------
ts_debug | 54558720
ts_debug | 27279360
ts_debug | 27279360
ts_debug | 27279360
ts_debug | 27279360
(5 rows)
postgres=# insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
insert into tmp_t0 select c0,c1||c1 from tmp_t0;
postgres=# \d+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+--------+-------+----------+---------+-------------
public | tmp_t0 | table | postgres | 2468 MB |
(1 row)
查询更广泛些
postgres=# select pc.oid,pc.relname,pc.relfilenode,
pc.relpages,pc.reltuples,
pc.reltoastrelid
from pg_class pc
where 1=1
and pc.relname like '%toast%25177%'
;
oid | relname | relfilenode | relpages | reltuples | reltoastrelid
-------+----------------------+-------------+----------+-----------+---------------
25180 | pg_toast_25177 | 25180 | 0 | 0 | 0
25182 | pg_toast_25177_index | 25182 | 1 | 0 | 0
(2 rows)
$ ls -l |grep -i 25177;ls -l |grep -i 25180 ;ls -l |grep -i 25182;
-rw------- 1 postgres postgres 2883584 Nov 25 01:05 25177
-rw------- 1 postgres postgres 24576 Nov 25 01:01 25177_fsm
-rw------- 1 postgres postgres 1073741824 Nov 25 00:48 25180
-rw------- 1 postgres postgres 1073741824 Nov 25 01:01 25180.1
-rw------- 1 postgres postgres 409100288 Nov 25 01:06 25180.2
-rw------- 1 postgres postgres 647168 Nov 25 01:05 25180_fsm
-rw------- 1 postgres postgres 28196864 Nov 25 01:06 25182