Benchmarks:
Ext4, data=ordered:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP iodrive-bcache 24G 1411 99 398554 18 220698 21 +++++ +++ 822438 35 6102 124 Latency 11656us 548ms 290ms 4629us 4478us 5903us 1.96,1.96,iodrive-bcache,1,1291749598,24G,,1411,99,398554,18,220698,21,+++++,+++,822438,35,6102,124,,,,,,,,,,,,,,,,,,11656us,548ms,290ms,4629us,4478us,5903us,,,,,,
Data=writeback
Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP iodrive-bcache 24G 1354 99 295877 14 238573 22 +++++ +++ 836213 36 5303 123 Latency 12591us 446ms 235ms 2401us 3868us 7345us 1.96,1.96,iodrive-bcache,1,1291754087,24G,,1354,99,295877,14,238573,22,+++++,+++,836213,36,5303,123,,,,,,,,,,,,,,,,,,12591us,446ms,235ms,2401us,3868us,7345us,,,,,,
Mysql/sysbench:
OLTP test statistics:
queries performed:
read: 420014
write: 150005
other: 60002
total: 630021
transactions: 30001 (609.82 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 570019 (11586.57 per sec.)
other operations: 60002 (1219.64 per sec.)
1card_70.30-bw.fio
[global] direct=1 rw=randrw bs=1m numjobs=4 size=20G runtime=60 group_reporting norandommap time_based rwmixread=70 [file1] filename=/dev/dm-0
1card_70.30-bw.fio.cmd
# fio /tmp/stdset/01card/dm-0/1card_70.30-bw.fio
1card_70.30-bw.fio.out
file1: (g=0): rw=randrw, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
...
file1: (g=0): rw=randrw, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
Starting 4 processes
file1: (groupid=0, jobs=4): err= 0: pid=3977
read : io=4860MB, bw=82880KB/s, iops=80, runt= 60046msec
clat (msec): min=1, max=616, avg=14.10, stdev=20.24
bw (KB/s) : min= 1703, max=65777, per=26.33%, avg=21821.86, stdev=5916.02
write: io=2159MB, bw=36819KB/s, iops=35, runt= 60046msec
clat (msec): min=4, max=694, avg=78.12, stdev=22.92
bw (KB/s) : min= 1283, max=18188, per=26.00%, avg=9572.84, stdev=1590.03
cpu : usr=0.02%, sys=2.76%, ctx=7392, majf=0, minf=120
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w: total=4860/2159, short=0/0
lat (msec): 2=48.48%, 4=11.28%, 10=2.28%, 20=1.30%, 50=6.70%
lat (msec): 100=19.40%, 250=9.82%, 500=0.68%, 750=0.06%
Run status group 0 (all jobs):
READ: io=4860MB, aggrb=82880KB/s, minb=84869KB/s, maxb=84869KB/s, mint=60046msec, maxt=60046msec
WRITE: io=2159MB, aggrb=36818KB/s, minb=37702KB/s, maxb=37702KB/s, mint=60046msec, maxt=60046msec
Disk stats (read/write):
dm-0: ios=149295/551424, merge=0/0, ticks=14049060/40254050, in_queue=54405510, util=99.88%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
1card_70.30-iops.fio
[global] direct=1 rw=randrw bs=4k numjobs=64 size=20G runtime=60 group_reporting norandommap time_based rwmixread=70 [file1] filename=/dev/dm-0
1card_70.30-iops.fio.cmd
# fio /tmp/stdset/01card/dm-0/1card_70.30-iops.fio
1card_70.30-iops.fio.out
file1: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
file1: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
Starting 64 processes
file1: (groupid=0, jobs=64): err= 0: pid=4105
read : io=103024KB, bw=1708KB/s, iops=426, runt= 60336msec
clat (usec): min=24, max=706184, avg=11551.40, stdev=7887.37
bw (KB/s) : min= 1, max= 190, per=1.67%, avg=28.59, stdev= 2.71
write: io=43588KB, bw=739759B/s, iops=180, runt= 60336msec
clat (usec): min=208, max=1280K, avg=318176.55, stdev=12388.13
bw (KB/s) : min= 2, max= 54, per=1.59%, avg=11.49, stdev= 0.42
cpu : usr=0.01%, sys=0.01%, ctx=36874, majf=0, minf=2100
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w: total=25756/10897, short=0/0
lat (usec): 50=0.37%, 100=41.99%, 250=22.63%, 500=0.42%, 750=0.36%
lat (usec): 1000=0.94%
lat (msec): 2=0.61%, 4=0.59%, 10=0.25%, 20=0.05%, 50=0.09%
lat (msec): 100=0.16%, 250=1.72%, 500=28.99%, 750=0.66%, 1000=0.09%
lat (msec): 2000=0.09%
Run status group 0 (all jobs):
READ: io=103024KB, aggrb=1707KB/s, minb=1748KB/s, maxb=1748KB/s, mint=60336msec, maxt=60336msec
WRITE: io=43588KB, aggrb=722KB/s, minb=739KB/s, maxb=739KB/s, mint=60336msec, maxt=60336msec
Disk stats (read/write):
dm-0: ios=1055/10897, merge=0/0, ticks=365050/3467370, in_queue=3841410, util=99.88%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
1card_read-bw.fio
[global] direct=1 rw=randread bs=1m numjobs=4 size=20G runtime=60 group_reporting norandommap time_based [file1] filename=/dev/dm-0
1card_read-bw.fio.cmd
# fio /tmp/stdset/01card/dm-0/1card_read-bw.fio
1card_read-bw.fio.out
file1: (g=0): rw=randread, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
...
file1: (g=0): rw=randread, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
Starting 4 processes
file1: (groupid=0, jobs=4): err= 0: pid=4017
read : io=27673MB, bw=472050KB/s, iops=460, runt= 60030msec
clat (msec): min=1, max=669, avg= 8.50, stdev= 9.91
bw (KB/s) : min= 7713, max=237093, per=25.74%, avg=121521.27, stdev=25179.88
cpu : usr=0.02%, sys=9.42%, ctx=29127, majf=0, minf=1147
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w: total=27673/0, short=0/0
lat (msec): 2=25.23%, 4=27.40%, 10=38.54%, 20=0.70%, 50=3.74%
lat (msec): 100=3.39%, 250=0.94%, 500=0.06%, 750=0.01%
Run status group 0 (all jobs):
READ: io=27673MB, aggrb=472049KB/s, minb=483379KB/s, maxb=483379KB/s, mint=60030msec, maxt=60030msec
Disk stats (read/write):
dm-0: ios=598017/0, merge=0/0, ticks=33221660/0, in_queue=33239580, util=79.20%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
1card_read-iops.fio
[global] direct=1 rw=randread bs=4k numjobs=64 size=20G runtime=60 group_reporting norandommap time_based [file1] filename=/dev/dm-0
1card_read-iops.fio.cmd
# fio /tmp/stdset/01card/dm-0/1card_read-iops.fio
1card_read-iops.fio.out
file1: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
file1: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
Starting 64 processes
file1: (groupid=0, jobs=64): err= 0: pid=4176
read : io=669868KB, bw=11067KB/s, iops=2766, runt= 60527msec
clat (usec): min=23, max=8467K, avg=33066.14, stdev=23219.72
bw (KB/s) : min= 0, max=33968, per=1.14%, avg=126.04, stdev=31.33
cpu : usr=0.01%, sys=0.05%, ctx=167560, majf=0, minf=2227
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w: total=167467/0, short=0/0
lat (usec): 50=4.74%, 100=56.60%, 250=13.19%, 500=8.84%, 750=5.13%
lat (usec): 1000=3.03%
lat (msec): 2=2.40%, 4=1.54%, 10=0.10%, 20=0.06%, 50=0.22%
lat (msec): 100=0.12%, 250=0.18%, 500=1.50%, 750=2.21%, 1000=0.08%
lat (msec): 2000=0.02%, >=2000=0.03%
Run status group 0 (all jobs):
READ: io=669868KB, aggrb=11067KB/s, minb=11332KB/s, maxb=11332KB/s, mint=60527msec, maxt=60527msec
Disk stats (read/write):
dm-0: ios=7446/0, merge=0/0, ticks=3421820/0, in_queue=3428350, util=99.40%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
1card_read-lat.fio
[global] direct=1 rw=randread bs=4k numjobs=1 size=20G runtime=60 group_reporting norandommap time_based [file1] filename=/dev/dm-0
1card_read-lat.fio.cmd
# fio /tmp/stdset/01card/dm-0/1card_read-lat.fio
1card_read-lat.fio.out
file1: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
Starting 1 process
file1: (groupid=0, jobs=1): err= 0: pid=4246
read : io=262204KB, bw=4369KB/s, iops=1092, runt= 60017msec
clat (usec): min=23, max=115860, avg=913.19, stdev=3724.58
bw (KB/s) : min= 1504, max=74744, per=100.06%, avg=4370.44, stdev=7537.30
cpu : usr=0.47%, sys=2.83%, ctx=65617, majf=0, minf=31
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w: total=65551/0, short=0/0
lat (usec): 50=12.51%, 100=27.84%, 250=53.66%, 500=0.34%, 750=0.04%
lat (usec): 1000=0.02%
lat (msec): 2=0.18%, 4=0.33%, 10=1.61%, 20=2.76%, 50=0.71%
lat (msec): 100=0.01%, 250=0.01%
Run status group 0 (all jobs):
READ: io=262204KB, aggrb=4368KB/s, minb=4473KB/s, maxb=4473KB/s, mint=60017msec, maxt=60017msec
Disk stats (read/write):
dm-0: ios=3327/0, merge=0/0, ticks=16830/0, in_queue=16830, util=28.05%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
1card_write-bw.fio
[global] direct=1 rw=randwrite bs=1m numjobs=4 size=20G runtime=60 group_reporting norandommap time_based [file1] filename=/dev/dm-0
1card_write-bw.fio.cmd
# fio /tmp/stdset/01card/dm-0/1card_write-bw.fio
1card_write-bw.fio.out
file1: (g=0): rw=randwrite, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
...
file1: (g=0): rw=randwrite, bs=1M-1M/1M-1M, ioengine=sync, iodepth=1
Starting 4 processes
file1: (groupid=0, jobs=4): err= 0: pid=3968
write: io=2882MB, bw=49132KB/s, iops=47, runt= 60066msec
clat (msec): min=5, max=212, avg=83.16, stdev= 8.96
bw (KB/s) : min= 7485, max=15086, per=25.09%, avg=12326.56, stdev=438.16
cpu : usr=0.01%, sys=1.29%, ctx=2882, majf=0, minf=119
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w: total=0/2882, short=0/0
lat (msec): 10=0.07%, 20=0.17%, 50=1.80%, 100=83.21%, 250=14.75%
Run status group 0 (all jobs):
WRITE: io=2882MB, aggrb=49132KB/s, minb=50311KB/s, maxb=50311KB/s, mint=60066msec, maxt=60066msec
Disk stats (read/write):
dm-0: ios=0/737024, merge=0/0, ticks=0/57393150, in_queue=57426360, util=99.90%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
1card_write-iops.fio
[global] direct=1 rw=randwrite bs=4k numjobs=64 size=20G runtime=60 group_reporting norandommap time_based [file1] filename=/dev/dm-0
1card_write-iops.fio.cmd
# fio /tmp/stdset/01card/dm-0/1card_write-iops.fio
1card_write-iops.fio.out
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
...
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
Starting 64 processes
file1: (groupid=0, jobs=64): err= 0: pid=4026
write: io=53440KB, bw=907881B/s, iops=221, runt= 60275msec
clat (usec): min=207, max=1231K, avg=290474.58, stdev=10591.46
bw (KB/s) : min= 3, max= 57, per=1.52%, avg=13.48, stdev= 0.31
cpu : usr=0.01%, sys=0.00%, ctx=13373, majf=0, minf=2101
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w: total=0/13360, short=0/0
lat (usec): 250=0.13%, 500=0.19%, 750=0.04%, 1000=0.01%
lat (msec): 2=0.15%, 4=0.82%, 10=0.86%, 20=0.12%, 50=0.20%
lat (msec): 100=0.46%, 250=4.03%, 500=91.06%, 750=1.46%, 1000=0.22%
lat (msec): 2000=0.25%
Run status group 0 (all jobs):
WRITE: io=53440KB, aggrb=886KB/s, minb=907KB/s, maxb=907KB/s, mint=60275msec, maxt=60275msec
Disk stats (read/write):
dm-0: ios=0/13360, merge=0/0, ticks=0/3841930, in_queue=3847320, util=99.89%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
1card_write-lat.fio
[global] direct=1 rw=randwrite bs=4k numjobs=1 size=20G runtime=60 group_reporting norandommap time_based [file1] filename=/dev/dm-0
1card_write-lat.fio.cmd
# fio /tmp/stdset/01card/dm-0/1card_write-lat.fio
1card_write-lat.fio.out
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
Starting 1 process
file1: (groupid=0, jobs=1): err= 0: pid=3961
write: io=52916KB, bw=903039B/s, iops=220, runt= 60004msec
clat (usec): min=201, max=429233, avg=4531.68, stdev=7396.63
bw (KB/s) : min= 304, max= 1390, per=100.40%, avg=884.54, stdev=108.15
cpu : usr=0.07%, sys=1.40%, ctx=13245, majf=0, minf=29
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w: total=0/13229, short=0/0
lat (usec): 250=0.83%, 500=1.68%, 750=0.29%, 1000=0.27%
lat (msec): 2=7.68%, 4=41.29%, 10=42.62%, 20=5.20%, 50=0.11%
lat (msec): 500=0.03%
Run status group 0 (all jobs):
WRITE: io=52916KB, aggrb=881KB/s, minb=903KB/s, maxb=903KB/s, mint=60004msec, maxt=60004msec
Disk stats (read/write):
dm-0: ios=0/13215, merge=0/0, ticks=0/59460, in_queue=59460, util=99.10%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%