redolog 大小的实验

时间:2021-06-27 15:42:08

前言:近日因工作需要,测试postgresql和MySQL在oltp对比测试,因结果差异太多(MySQL测试结果比较差,相同环境),寻求大神帮助,有幸得叶大师和姜大师指点,指出my.cnf配置文件innodb_log_file_size不合理(在以前的资料提示Innodb,redolog大小一般都建议128-512M大小,不然崩溃恢复将会很漫长),redolog越大,checkpoint就越少,tps相对越高,MySQL5.6redolog 大小可以超过4G,下面就测试一下不同redolog大小奔溃恢复时间

测试环境:硬件是VM,系统是centos6.6 4核4G内存,宿主硬盘是Intel SSD,测试表数据为1千万行,测试工具为sysbench0.5版本,测试方法是压力测试工具进行压力测试,然后暴力kill MySQL

跟日志相关参数:

innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 32M
innodb_log_file_size = 4096M(分别测试1G,2G,4G)
innodb_log_files_in_group = 2
innodb_max_dirty_pages_pct = 70

IO性能:因为结果出乎人意料我不得不先把我自己IO能力测试出来

16k随机读写IO,读写比例为:40:60 IOPS=2951

fio -filename=/data/fiotest -direct=1  -rw=randrw -ioengine=libaio -rwmixwrite=60 \
> -bs=16k -size=8G -numjobs=64 -runtime=600 -group_reporting -name=mytest

read : io=27676MB, bw=47231KB/s, iops=, runt=600025msec

8k随机读写IO,读写比例为:40:60 IOPS=2951

fio -filename=/data/fiotest -direct=1  -rw=randrw -ioengine=libaio -rwmixwrite=60 \

> -bs=8k -size=8G -numjobs=64 -runtime=600 -group_reporting -name=mytest

write: io=18155MB, bw=30983KB/s, iops=, runt=600046msec

4K随机读写IO  读写比例为:40:60 IOPS=3590

fio -filename=/data/fiotest -direct=1  -rw=randrw -ioengine=libaio -rwmixwrite=60 \
> -bs=4k -size=8G -numjobs=64 -runtime=600 -group_reporting -name=mytest

read : io=8416.6MB, bw=14363KB/s, iops=3590, runt=600030msec

【1】测试方法

1.1 创建测试表

./sysbench --mysql-host=192.168.80.106 --mysql-user=sysbench --mysql-password=123456 --mysql-db=sysbench --test=/opt/sysbench/tests/db/oltp.lua --oltp_tables_count=10 --oltp-table-size=1000000 --rand-init=on prepare

1.2 进行测试,其中一个会话在做压力测试,一个会话暴力杀掉MySQL sleep 300; pkill -9 mysqld

./sysbench --mysql-host=192.168.80.106 --mysql-port=3306 --mysql-user=sysbench --mysql-password=123456 --test=/opt/sysbench/tests/db/oltp.lua --oltp_tables_count=10 --oltp-table-size=10000000 --num-threads=64-oltp-read-only=off --report-interval=10 --rand-type=uniform --max-time=1200  --mysql-db=sysbench --max-requests=0 --percentile=99 run

【2】测试结果

2.1 innodb_log_file_size = 1024M

 :: mysqld_safe Starting mysqld daemon with databases from /data/mydata
-- :: [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
-- :: [Note] Plugin 'FEDERATED' is disabled.
-- :: 7f80332d07e0 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
-- :: [Note] InnoDB: Using atomics to ref count buffer pool pages
-- :: [Note] InnoDB: The InnoDB memory heap is disabled
-- :: [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
-- :: [Note] InnoDB: Memory barrier is not used
-- :: [Note] InnoDB: Compressed tables use zlib 1.2.
-- :: [Note] InnoDB: Using Linux native AIO
-- :: [Note] InnoDB: Using CPU crc32 instructions
-- :: [Note] InnoDB: Initializing buffer pool, size = .0G
-- :: [Note] InnoDB: Completed initialization of buffer pool
-- :: [Note] InnoDB: Highest supported file format is Barracuda.
-- :: [Note] InnoDB: Log scan progressed past the checkpoint lsn
-- :: [Note] InnoDB: Database was not shutdown normally!
-- :: [Note] InnoDB: Starting crash recovery.
-- :: [Note] InnoDB: Reading tablespace information from the .ibd files...
-- :: [Note] InnoDB: Restoring possible half-written data pages
-- :: [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Transaction was in the XA prepared state.
InnoDB: transaction(s) which must be rolled back or cleaned up
InnoDB: in total row operations to undo
InnoDB: Trx id counter is
-- :: [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent:
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position , file name dg6-logbin.
-- :: [Note] InnoDB: rollback segment(s) are active.
-- :: [Note] InnoDB: Waiting for purge to start
InnoDB: Starting in background the rollback of uncommitted transactions
-- :: 7f7f6dde8700 InnoDB: Rolling back trx with id , rows to undo
-- :: [Note] InnoDB: Rollback of trx with id completed
-- :: 7f7f6dde8700 InnoDB: Rolling back trx with id , rows to undo
-- :: [Note] InnoDB: Rollback of trx with id completed
-- :: 7f7f6dde8700 InnoDB: Rolling back trx with id , rows to undo
-- :: [Note] InnoDB: Rollback of trx with id completed
-- :: 7f7f6dde8700 InnoDB: Rollback of non-prepared transactions completed
-- :: [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.21-rel70.1 started; log sequence number 3446339459
-- :: [Note] Recovering after a crash using dg6-logbin
-- :: [Note] Starting crash recovery...
-- :: 7f80332d07e0 InnoDB: Starting recovery for XA transactions...
-- :: 7f80332d07e0 InnoDB: Transaction in prepared state after recovery
-- :: 7f80332d07e0 InnoDB: Transaction contains changes to rows
-- :: 7f80332d07e0 InnoDB: transactions in prepared state after recovery
-- :: [Note] Found prepared transaction(s) in InnoDB
-- :: [Note] Crash recovery finished.
/opt/app/mysql/bin/mysqld: Found a Gtid_log_event or Previous_gtids_log_event when @@GLOBAL.GTID_MODE = OFF.
/opt/app/mysql/bin/mysqld: Found a Gtid_log_event or Previous_gtids_log_event when @@GLOBAL.GTID_MODE = OFF.
-- :: [Note] RSA private key file not found: /data/mydata//private_key.pem. Some authentication plugins will not work.
-- :: [Note] RSA public key file not found: /data/mydata//public_key.pem. Some authentication plugins will not work.
-- :: [Note] Server hostname (bind-address): '*'; port:
-- :: [Note] IPv6 is available.
-- :: [Note] - '::' resolves to '::';
-- :: [Note] Server socket created on IP: '::'.
-- :: [Warning] 'user' entry 'root@dg1' ignored in --skip-name-resolve mode.
-- :: [Warning] 'proxies_priv' entry '@ root@dg1' ignored in --skip-name-resolve mode.
-- :: [Note] Event Scheduler: Loaded events
-- :: [Note] /opt/app/mysql/bin/mysqld: ready for connections.
Version: '5.6.21-70.1-log' socket: '/tmp/mysql.sock' port: Percona Server (GPL), Release 70.1, Revision

看到木有恢复时间才十几秒钟呢

2.2 innodb_log_file_size = 2048M

 :: mysqld_safe Starting mysqld daemon with databases from /data/mydata
-- :: [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
-- :: [Note] Plugin 'FEDERATED' is disabled.
-- :: 7f599fd1c7e0 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
-- :: [Note] InnoDB: Using atomics to ref count buffer pool pages
-- :: [Note] InnoDB: The InnoDB memory heap is disabled
-- :: [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
-- :: [Note] InnoDB: Memory barrier is not used
-- :: [Note] InnoDB: Compressed tables use zlib 1.2.
-- :: [Note] InnoDB: Using Linux native AIO
-- :: [Note] InnoDB: Using CPU crc32 instructions
-- :: [Note] InnoDB: Initializing buffer pool, size = .0G
-- :: [Note] InnoDB: Completed initialization of buffer pool
-- :: [Note] InnoDB: Highest supported file format is Barracuda.
-- :: [Note] InnoDB: Log scan progressed past the checkpoint lsn
-- :: [Note] InnoDB: Database was not shutdown normally!
-- :: [Note] InnoDB: Starting crash recovery.
-- :: [Note] InnoDB: Reading tablespace information from the .ibd files...
-- :: [Note] InnoDB: Restoring possible half-written data pages
-- :: [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Transaction was in the XA prepared state.
InnoDB: transaction(s) which must be rolled back or cleaned up
InnoDB: in total row operations to undo
InnoDB: Trx id counter is
-- :: [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent:
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position , file name dg6-logbin.
-- :: [Note] InnoDB: rollback segment(s) are active.
-- :: [Note] InnoDB: Waiting for purge to start
InnoDB: Starting in background the rollback of uncommitted transactions
-- :: 7f58dd9c2700 InnoDB: Rolling back trx with id , rows to undo
-- :: [Note] InnoDB: Rollback of trx with id completed
-- :: 7f58dd9c2700 InnoDB: Rollback of non-prepared transactions completed
-- :: [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.21-rel70.1 started; log sequence number 3505201579
-- :: [Note] Recovering after a crash using dg6-logbin
-- :: [Note] Starting crash recovery...
-- :: 7f599fd1c7e0 InnoDB: Starting recovery for XA transactions...
-- :: 7f599fd1c7e0 InnoDB: Transaction in prepared state after recovery
-- :: 7f599fd1c7e0 InnoDB: Transaction contains changes to rows
-- :: 7f599fd1c7e0 InnoDB: transactions in prepared state after recovery
-- :: [Note] Found prepared transaction(s) in InnoDB
-- :: [Note] Crash recovery finished.
/opt/app/mysql/bin/mysqld: Found a Gtid_log_event or Previous_gtids_log_event when @@GLOBAL.GTID_MODE = OFF.
/opt/app/mysql/bin/mysqld: Found a Gtid_log_event or Previous_gtids_log_event when @@GLOBAL.GTID_MODE = OFF.
-- :: [Note] RSA private key file not found: /data/mydata//private_key.pem. Some authentication plugins will not work.
-- :: [Note] RSA public key file not found: /data/mydata//public_key.pem. Some authentication plugins will not work.
-- :: [Note] Server hostname (bind-address): '*'; port:
-- :: [Note] IPv6 is available.
-- :: [Note] - '::' resolves to '::';
-- :: [Note] Server socket created on IP: '::'.
-- :: [Warning] 'user' entry 'root@dg1' ignored in --skip-name-resolve mode.
-- :: [Warning] 'proxies_priv' entry '@ root@dg1' ignored in --skip-name-resolve mode.
-- :: [Note] Event Scheduler: Loaded events
-- :: [Note] /opt/app/mysql/bin/mysqld: ready for connections.
Version: '5.6.21-70.1-log' socket: '/tmp/mysql.sock' port: Percona Server (GPL), Release 70.1, Revision

不科学呀,还是十几秒,那么我们改成4G,让压测运行时间更长一点,900秒以后在kill

2.3 innodb_log_file_size = 4096M

 :: mysqld_safe Starting mysqld daemon with databases from /data/mydata
-- :: [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
-- :: [Note] Plugin 'FEDERATED' is disabled.
-- :: 7fd6f83a67e0 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
-- :: [Note] InnoDB: Using atomics to ref count buffer pool pages
-- :: [Note] InnoDB: The InnoDB memory heap is disabled
-- :: [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
-- :: [Note] InnoDB: Memory barrier is not used
-- :: [Note] InnoDB: Compressed tables use zlib 1.2.
-- :: [Note] InnoDB: Using Linux native AIO
-- :: [Note] InnoDB: Using CPU crc32 instructions
-- :: [Note] InnoDB: Initializing buffer pool, size = .0G
-- :: [Note] InnoDB: Completed initialization of buffer pool
-- :: [Note] InnoDB: Highest supported file format is Barracuda.
-- :: [Note] InnoDB: Log scan progressed past the checkpoint lsn
-- :: [Note] InnoDB: Database was not shutdown normally!
-- :: [Note] InnoDB: Starting crash recovery.
-- :: [Note] InnoDB: Reading tablespace information from the .ibd files...
-- :: [Note] InnoDB: Restoring possible half-written data pages
-- :: [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Transaction was in the XA prepared state.
InnoDB: Transaction was in the XA prepared state.
InnoDB: transaction(s) which must be rolled back or cleaned up
InnoDB: in total row operations to undo
InnoDB: Trx id counter is
-- :: [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent:
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position , file name dg6-logbin.
-- :: [Note] InnoDB: rollback segment(s) are active.
-- :: [Note] InnoDB: Waiting for purge to start
InnoDB: Starting in background the rollback of uncommitted transactions
-- :: 7fd635de8700 InnoDB: Rolling back trx with id , rows to undo
-- :: [Note] InnoDB: Rollback of trx with id completed
-- :: 7fd635de8700 InnoDB: Rollback of non-prepared transactions completed
-- :: [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.21-rel70.1 started; log sequence number 3662227637
-- :: [Note] Recovering after a crash using dg6-logbin
-- :: [Note] Starting crash recovery...
-- :: 7fd6f83a67e0 InnoDB: Starting recovery for XA transactions...
-- :: 7fd6f83a67e0 InnoDB: Transaction in prepared state after recovery
-- :: 7fd6f83a67e0 InnoDB: Transaction contains changes to rows
-- :: 7fd6f83a67e0 InnoDB: transactions in prepared state after recovery
-- :: [Note] Found prepared transaction(s) in InnoDB
-- :: [Note] Crash recovery finished.
/opt/app/mysql/bin/mysqld: Found a Gtid_log_event or Previous_gtids_log_event when @@GLOBAL.GTID_MODE = OFF.
/opt/app/mysql/bin/mysqld: Found a Gtid_log_event or Previous_gtids_log_event when @@GLOBAL.GTID_MODE = OFF.
-- :: [Note] RSA private key file not found: /data/mydata//private_key.pem. Some authentication plugins will not work.
-- :: [Note] RSA public key file not found: /data/mydata//public_key.pem. Some authentication plugins will not work.
-- :: [Note] Server hostname (bind-address): '*'; port:
-- :: [Note] IPv6 is available.
-- :: [Note] - '::' resolves to '::';
-- :: [Note] Server socket created on IP: '::'.
-- :: [Warning] 'user' entry 'root@dg1' ignored in --skip-name-resolve mode.
-- :: [Warning] 'proxies_priv' entry '@ root@dg1' ignored in --skip-name-resolve mode.
-- :: [Note] Event Scheduler: Loaded events
-- :: [Note] /opt/app/mysql/bin/mysqld: ready for connections.
Version: '5.6.21-70.1-log' socket: '/tmp/mysql.sock' port: Percona Server (GPL), Release 70.1, Revision

也才二十几秒钟。

那么我们看看5.5版本是什么情况 因为5.5版本redolog大小不能超过4G,所以单个不能设置成2048M否则会报错

150607 17:15:46 InnoDB: Error: combined size of log files must be < 4 GB
150607 17:15:46 [ERROR] Plugin 'InnoDB' init function returned error.

2.4 innodb_log_file_size = 1024M

 :: mysqld_safe Starting mysqld daemon with databases from /data/mydata
:: [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
:: [Note] /opt/app/mysql/bin/mysqld (mysqld 5.5.-log) starting as process ...
:: [Note] Plugin 'FEDERATED' is disabled.
:: InnoDB: The InnoDB memory heap is disabled
:: InnoDB: Mutexes and rw_locks use GCC atomic builtins
:: InnoDB: Compressed tables use zlib 1.2.
:: InnoDB: Using Linux native AIO
:: InnoDB: Initializing buffer pool, size = .0G
:: InnoDB: Completed initialization of buffer pool
:: InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn
:: InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Doing recovery: scanned up to log sequence number
InnoDB: Transaction 203E1 was in the XA prepared state.
InnoDB: transaction(s) which must be rolled back or cleaned up
InnoDB: in total row operations to undo
InnoDB: Trx id counter is
:: InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents:
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position , file name ./mysql-bin.
InnoDB: Starting in background the rollback of uncommitted transactions
:: InnoDB: Rollback of non-prepared transactions completed
:: InnoDB: Waiting for the background threads to start
:: InnoDB: 5.5. started; log sequence number
:: [Note] Recovering after a crash using mysql-bin
:: [Note] Starting crash recovery...
:: InnoDB: Starting recovery for XA transactions...
:: InnoDB: Transaction 203E1 in prepared state after recovery
:: InnoDB: Transaction contains changes to rows
:: InnoDB: transactions in prepared state after recovery
:: [Note] Found prepared transaction(s) in InnoDB
:: [Note] Crash recovery finished.
:: [Note] Server hostname (bind-address): '0.0.0.0'; port:
:: [Note] - '0.0.0.0' resolves to '0.0.0.0';
:: [Note] Server socket created on IP: '0.0.0.0'.
:: [Warning] 'user' entry 'root@dg1' ignored in --skip-name-resolve mode.
:: [Warning] 'user' entry '@dg1' ignored in --skip-name-resolve mode.
:: [Warning] 'proxies_priv' entry '@ root@dg1' ignored in --skip-name-resolve mode.
:: [Note] Event Scheduler: Loaded events
:: [Note] /opt/app/mysql/bin/mysqld: ready for connections.
Version: '5.5.43-log' socket: '/tmp/mysql.sock' port: MySQL Community Server (GPL)

也不会很慢二十几秒钟

总结:

在5.5版本以上,修改了恢复算法,崩溃恢复快了很多,但是这里面涉及到很多因素,buffer_pool ,脏页比例,TPS等,需要多次测试,因条件限制,没办法进行大buffer_pool,高并发测试,实际物理机测试时,根据自身情况选着合理参数测试

参考资料:来自姜大神的书

在InnoDB存储引擎内部,有两种Checkpoint,分别为:

Sharp Checkpoint

Fuzzy Checkpoint

Sharp Checkpoint发生在数据库关闭时将所有的脏页都刷新回磁盘,这是默认的工作方式,即参数innodb_fast_shutdown=1。

但是若数据库在运行时也使用Sharp Checkpoint,那么数据库的可用性就会受到很大的影响。故在InnoDB存储引擎内部使用Fuzzy Checkpoint进行页的刷新,即只刷新一部分脏页,而不是刷新所有的脏页回磁盘。

这里笔者进行了概括,在InnoDB存储引擎中可能发生如下几种情况的Fuzzy Checkpoint:

Master Thread Checkpoint

FLUSH_LRU_LIST Checkpoint

Async/Sync Flush Checkpoint

Dirty Page too much Checkpoint

对于Master Thread(2.5节会详细介绍各个版本中Master Thread的实现)中发生的Checkpoint,差不多以每秒或每十秒的速度从缓冲池的脏页列表中刷新一定比例的页回磁盘。这个过程是异步的,即此时InnoDB存储引擎可以进行其他的操作,用户查询线程不会阻塞。

FLUSH_LRU_LIST Checkpoint是因为InnoDB存储引擎需要保证LRU列表中需要有差不多100个空闲页可供使用。在InnoDB1.1.x版本之前,需要检查LRU列表中是否有足够的可用空间操作发生在用户查询线程中,显然这会阻塞用户的查询操作。倘若没有100个可用空闲页,那么InnoDB存储引擎会将LRU列表尾端的页移除。如果这些页中有脏页,那么需要进行Checkpoint,而这些页是来自LRU列表的,因此称为FLUSH_LRU_LIST Checkpoint。

而从MySQL 5.6版本,也就是InnoDB1.2.x版本开始,这个检查被放在了一个单独的Page Cleaner线程中进行,并且用户可以通过参数innodb_lru_scan_depth控制LRU列表中可用页的数量,该值默认为1024,如:

mysql> SHOW VARIABLES LIKE 'innodb_lru_scan_depth'\G;

*************************** 1. row ***************************

Variable_name: innodb_lru_scan_depth

Value: 1024

1 row in set (0.00 sec)

Async/Sync Flush Checkpoint指的是重做日志文件不可用的情况,这时需要强制将一些页刷新回磁盘,而此时脏页是从脏页列表中选取的。若将已经写入到重做日志的LSN记为redo_lsn,将已经刷新回磁盘最新页的LSN记为checkpoint_lsn,则可定义:

checkpoint_age = redo_lsn - checkpoint_lsn

再定义以下的变量:

async_water_mark = 75% * total_redo_log_file_size

sync_water_mark = 90% * total_redo_log_file_size

若每个重做日志文件的大小为1GB,并且定义了两个重做日志文件,则重做日志文件的总大小为2GB。那么async_water_mark=1.5GB,sync_water_mark=1.8GB。则:

当checkpoint_age<async_water_mark时,不需要刷新任何脏页到磁盘;

当async_water_mark<checkpoint_age<sync_water_mark时触发Async Flush,从Flush列表中刷新足够的脏页回磁盘,使得刷新后满足checkpoint_age<async_water_mark;

checkpoint_age>sync_water_mark这种情况一般很少发生,除非设置的重做日志文件太小,并且在进行类似LOAD DATA的BULK INSERT操作。此时触发Sync Flush操作,从Flush列表中刷新足够的脏页回磁盘,使得刷新后满足checkpoint_age<async_water_mark。

可见,Async/Sync Flush Checkpoint是为了保证重做日志的循环使用的可用性。在InnoDB 1.2.x版本之前,Async Flush Checkpoint会阻塞发现问题的用户查询线程,而Sync Flush Checkpoint会阻塞所有的用户查询线程,并且等待脏页刷新完成。从InnoDB 1.2.x版本开始——也就是MySQL 5.6版本,这部分的刷新操作同样放入到了单独的Page Cleaner Thread中,故不会阻塞用户查询线程。

MySQL官方版本并不能查看刷新页是从Flush列表中还是从LRU列表中进行Checkpoint的,也不知道因为重做日志而产生的Async/Sync Flush的次数。但是InnoSQL版本提供了方法,可以通过命令SHOW ENGINE INNODB STATUS来观察,如:

mysql> SHOW ENGINE INNODB STATUS\G;

*************************** 1. row ***************************

Type: InnoDB

……

LRU len: 112902, unzip_LRU len: 0

I/O sum[0]:cur[0], unzip sum[0]:cur[0]

Async Flush: 0, Sync Flush: 0, LRU List Flush: 0, Flush List Flush: 111736

……

1 row in set (0.01 sec)

根据上述的信息,还可以对InnoDB存储引擎做更为深入的调优,这部分将在第9章中讲述。

最后一种Checkpoint的情况是Dirty Page too much,即脏页的数量太多,导致InnoDB存储引擎强制进行Checkpoint。其目的总的来说还是为了保证缓冲池中有足够可用的页。其可由参数innodb_max_dirty_pages_pct控制:

mysql>SHOW VARIABLES LIKE 'innodb_max_dirty_pages_pct'\G;

*************************** 1. row ***************************

Variable_name: innodb_max_dirty_pages_pct

Value: 75

1 row in set (0.00 sec)

innodb_max_dirty_pages_pct值为75表示,当缓冲池中脏页的数量占据75%时,强制进行Checkpoint,刷新一部分的脏页到磁盘。在InnoDB 1.0.x版本之前,该参数默认值为90,之后的版本都为75。