一、现象
最近发现Mysql服务器磁盘IO一直很高
[root@push-- ~]# iostat -k -d -x
Linux 3.10.-.el7.x86_64 (push--) 2019年07月05日 _x86_64_ ( CPU) Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.08 0.00 14.89 0.09 177.79 23.88 0.34 23.05 5.42 23.05 11.01 16.40
dm- 0.00 0.00 0.00 0.18 0.09 3.62 39.49 0.07 380.21 5.59 387.46 12.69 0.24
dm- 0.00 0.00 0.00 0.00 0.00 0.01 8.98 0.00 91.25 12.04 92.94 6.55 0.00
dm- 0.00 0.00 0.00 14.79 0.00 174.15 23.56 0.33 22.27 8.84 22.27 10.96 16.20 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 83.00 0.00 650.50 15.67 2.55 30.66 0.00 30.66 12.02 99.80
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 83.00 0.00 636.50 15.34 2.55 30.66 0.00 30.66 12.02 99.80 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 88.00 0.00 684.50 15.56 2.60 29.42 0.00 29.42 11.35 99.90
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 87.00 0.00 677.00 15.56 2.60 29.76 0.00 29.76 11.49 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 87.00 0.00 687.50 15.80 2.48 28.77 0.00 28.77 11.51 100.10
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 88.00 0.00 699.00 15.89 2.48 28.48 0.00 28.48 11.36 100.00 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 88.00 0.00 756.50 17.19 2.48 28.22 0.00 28.22 11.35 99.90
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 86.00 0.00 740.00 17.21 2.48 28.87 0.00 28.87 11.62 99.90
%util已经接近100%了，IO性能严重瓶颈，这台服务器有两个Mysql实例，3306是主库，3307是另一台Mysql的从库，进入从库查看从库状态
mysql> show slave status\G;
*************************** . row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 172.28.5.223
Master_User: repl
Master_Port:
Connect_Retry:
Master_Log_File: master-223.000012
Read_Master_Log_Pos:
Relay_Log_File: push---relay-bin.
Relay_Log_Pos:
Relay_Master_Log_File: master-223.000012
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno:
Last_Error:
Skip_Counter:
Exec_Master_Log_Pos:
Relay_Log_Space:
Until_Condition: None
Until_Log_File:
Until_Log_Pos:
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master:
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno:
Last_IO_Error:
Last_SQL_Errno:
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id:
Master_UUID: 165a203c--11e9-bc40-6c2b5992e592
Master_Info_File: /home/mysql-5.7./data//master.info
SQL_Delay:
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Reading event from the relay log
Master_Retry_Count:
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position:
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
row in set (0.00 sec) ERROR:
No query specified mysql>
Seconds_Behind_Master: 2345 发现从库同步延时很高，通过iotop查看
Total DISK READ : 0.00 B/s | Total DISK WRITE : 683.85 K/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 721.19 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
be/ mysql 0.00 B/s 153.28 K/s 0.00 % 72.27 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 27.51 K/s 0.00 % 8.51 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 11.79 K/s 0.00 % 7.50 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 11.79 K/s 0.00 % 7.47 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 0.00 B/s 0.00 % 7.41 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 35.37 K/s 0.00 % 7.14 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 27.51 K/s 0.00 % 6.77 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 15.72 K/s 0.00 % 6.31 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 15.72 K/s 0.00 % 5.32 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 7.86 K/s 0.00 % 5.05 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 7.86 K/s 0.00 % 5.00 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 47.16 K/s 0.00 % 4.21 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 0.00 B/s 0.00 % 3.63 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 7.86 K/s 0.00 % 3.54 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 15.72 K/s 0.00 % 3.40 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 3.93 K/s 0.00 % 3.36 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 7.86 K/s 0.00 % 3.35 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 7.86 K/s 0.00 % 2.85 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 15.72 K/s 0.00 % 2.81 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 7.86 K/s 0.00 % 2.42 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 3.93 K/s 0.00 % 2.07 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 15.72 K/s 0.00 % 1.91 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 157.21 K/s 0.00 % 1.41 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 3.93 K/s 0.00 % 1.30 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 31.44 K/s 0.00 % 0.93 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ root 0.00 B/s 0.00 B/s 0.00 % 0.61 % [kworker/:]
be/ mysql 0.00 B/s 27.51 K/s 0.00 % 0.00 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ mysql 0.00 B/s 15.72 K/s 0.00 % 0.00 % mysqld --defaults-file=/etc/mysq~mysql//mysql.sock --port=
be/ root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd-udevd
be/ root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd --system --deserialize
发现3307和3306实例占用IO最高，进入3307实例，查看sync_binlog变量
mysql> show variables like '%sync_binlog%'
-> ;
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| sync_binlog | |
+---------------+-------+
row in set (0.01 sec) mysql>
发现设置的是1，也就是说每次事务提交都会将binlog的缓存写入磁盘，严重影响磁盘效率，将变量设置为1000,
mysql> set global sync_binlog=;
Query OK, rows affected (0.00 sec) mysql> show variables like '%sync_binlog%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| sync_binlog | |
+---------------+-------+
row in set (0.00 sec)
再查看innodb_flush_log_at_trx_commit参数
mysql> show variables like '%innodb_flush_log%';
+--------------------------------+-------+
| Variable_name | Value |
+--------------------------------+-------+
| innodb_flush_log_at_timeout | |
| innodb_flush_log_at_trx_commit | |
+--------------------------------+-------+
rows in set (0.00 sec)
设置为1，也就是说每次事务提交，都会将innodb日志缓存写入磁盘，对磁盘效率影响很大，将它设置为2，每次事务提交时mysql都会把log buffer的数据写入log file，但是flush(刷到磁盘)操作并不会同时进行。该模式下，MySQL会每秒执行一次 flush(刷到磁盘)操作
mysql> set global innodb_flush_log_at_trx_commit=;
Query OK, rows affected (0.00 sec) mysql> show variables like '%innodb_flush_log%';
+--------------------------------+-------+
| Variable_name | Value |
+--------------------------------+-------+
| innodb_flush_log_at_timeout | |
| innodb_flush_log_at_trx_commit | |
+--------------------------------+-------+
rows in set (0.00 sec)
此时再查看从库同步状态
mysql> show slave status\G;
*************************** . row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 172.28.5.223
Master_User: repl
Master_Port:
Connect_Retry:
Master_Log_File: master-223.000012
Read_Master_Log_Pos:
Relay_Log_File: push---relay-bin.
Relay_Log_Pos:
Relay_Master_Log_File: master-223.000012
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno:
Last_Error:
Skip_Counter:
Exec_Master_Log_Pos:
Relay_Log_Space:
Until_Condition: None
Until_Log_File:
Until_Log_Pos:
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master:
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno:
Last_IO_Error:
Last_SQL_Errno:
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id:
Master_UUID: 165a203c--11e9-bc40-6c2b5992e592
Master_Info_File: /home/mysql-5.7./data//master.info
SQL_Delay:
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: System lock
Master_Retry_Count:
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position:
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
row in set (0.00 sec) ERROR:
No query specified
发现，同步延迟已经在很快缩小了，很快变为0，完全同步，这时通过iostat 查看io状态
[root@push-- ~]# iostat -k -d -x /s 0.00 % 0.00 % [kworker/:0H]
Linux 3.10.-.el7.x86_64 (push--) 2019年07月05日 _x86_64_ ( CPU) Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.08 0.00 14.91 0.09 178.29 23.92 0.34 23.05 5.42 23.06 11.01 16.43
dm- 0.00 0.00 0.00 0.18 0.09 3.62 39.48 0.07 380.15 5.59 387.40 12.69 0.24
dm- 0.00 0.00 0.00 0.00 0.00 0.01 8.97 0.00 91.84 12.04 93.52 6.63 0.00
dm- 0.00 0.00 0.00 14.81 0.00 174.66 23.59 0.33 22.28 8.84 22.28 10.96 16.23 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 114.00 0.00 2013.00 35.32 3.14 27.40 0.00 27.40 8.75 99.80
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 113.00 0.00 2007.50 35.53 3.14 27.65 0.00 27.65 8.83 99.80 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 108.91 0.00 3026.24 55.57 3.10 27.85 0.00 27.85 9.05 98.51
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 111.88 0.00 3121.29 55.80 3.10 27.11 0.00 27.11 8.81 98.51 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 98.00 0.00 2648.00 54.04 2.81 29.63 0.00 29.63 9.71 95.20
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 95.00 0.00 2551.50 53.72 2.81 30.59 0.00 30.59 10.02 95.20 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 103.00 0.00 2037.00 39.55 2.97 28.67 0.00 28.67 9.12 93.90
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 104.00 0.00 2051.00 39.44 2.97 28.39 0.00 28.39 9.03 93.90
发现%util依旧差不多100%，磁盘IO性能还是很低，我们再把3306实例的变量设置改一下
[root@push-- ~]# mysql -uroot -p -S /var/lib/mysql//mysql.sock
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is
Server version: 5.7.-log MySQL Community Server (GPL) Copyright (c) , , Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show variables like '%sync_binlog%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| sync_binlog | |
+---------------+-------+
row in set (0.01 sec) mysql> set global sync_binlog=;
Query OK, rows affected (0.00 sec) mysql> show variables like '%innodb_flush_log%';
+--------------------------------+-------+
| Variable_name | Value |
+--------------------------------+-------+
| innodb_flush_log_at_timeout | |
| innodb_flush_log_at_trx_commit | |
+--------------------------------+-------+
rows in set (0.00 sec) mysql> set global innodb_flush_log_at_trx_commit=;
Query OK, rows affected (0.00 sec) mysql> show variables like '%innodb_flush_log%';
+--------------------------------+-------+
| Variable_name | Value |
+--------------------------------+-------+
| innodb_flush_log_at_timeout | |
| innodb_flush_log_at_trx_commit | |
+--------------------------------+-------+
rows in set (0.00 sec) mysql>
此时，再通过iostat查看IO
[root@push-- ~]# iostat -k -d -x
Linux 3.10.-.el7.x86_64 (push--) 2019年07月05日 _x86_64_ ( CPU) Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.08 0.00 14.92 0.09 178.36 23.92 0.34 23.06 5.42 23.06 11.01 16.43
dm- 0.00 0.00 0.00 0.18 0.09 3.62 39.48 0.07 380.15 5.59 387.39 12.69 0.24
dm- 0.00 0.00 0.00 0.00 0.00 0.01 8.97 0.00 91.84 12.04 93.52 6.63 0.00
dm- 0.00 0.00 0.00 14.81 0.00 174.72 23.59 0.33 22.28 8.84 22.28 10.96 16.23 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 21.00 0.00 604.00 57.52 0.38 18.10 0.00 18.10 13.76 28.90
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 21.00 0.00 604.00 57.52 0.38 18.10 0.00 18.10 13.76 28.90 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 34.00 0.00 2220.50 130.62 1.29 37.88 0.00 37.88 12.88 43.80
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 34.00 0.00 2220.50 130.62 1.29 37.88 0.00 37.88 12.88 43.80 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 42.00 0.00 2056.00 97.90 1.29 30.67 0.00 30.67 9.40 39.50
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 42.00 0.00 2056.00 97.90 1.29 30.67 0.00 30.67 9.40 39.50 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 21.00 0.00 550.00 52.38 0.34 16.19 0.00 16.19 13.76 28.90
dm- 0.00 0.00 0.00 2.00 0.00 1.50 1.50 0.05 24.00 0.00 24.00 24.00 4.80
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 19.00 0.00 548.50 57.74 0.29 15.37 0.00 15.37 12.74 24.20 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 18.00 0.00 524.00 58.22 0.34 18.61 0.00 18.61 13.33 24.00
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm- 0.00 0.00 0.00 18.00 0.00 524.00 58.22 0.34 18.61 0.00 18.61 13.33 24.00
%util已经顺利降到40%左右了。