Zoned-Storage - 对ZNS块设备进行基准测试

kxwang_

1599人浏览 · 2022-05-27 20:49:53

kxwang_ · 2022-05-27 20:49:53 发布

Zoned-Storage - 对ZNS块设备进行基准测试

fio zoned块设备支持

Flexible I/O Tester (fio) 最初是作为内核块 I/O 堆栈的测试工具编写的。然而，多年来，fio 获得了许多功能和详细的性能统计输出，从而将该工具变成了存储设备的标准基准测试应用程序。
fio 源代码可在 GitHub 上获得。

fio zoned块设备支持

fio3.9版添加了对分区块设备的支持。所有以前的版本都不为主机管理的分区块设备的写入命令顺序提供保证。仍然可以执行工作负载，但需要编写复杂的 fio 脚本。

命令行选项

对 fio 的更改以支持zoned块设备包括几个新选项，允许用户控制符合zoned块设备的工作负载。 fio 已经实现了选项 --zonemode，它允许定义在不相交的块范围内运行的工作负载。此选项被重用于定义新的 zbd 区域模式。

当 fio job 使用 zbd 区域模式时，--zonerange 选项将被忽略，--zonesize 选项会自动设置为设备区域大小。此外，读写命令的行为被修改如下。

当跨越区域边界时，读取和写入命令会被拆分。
对于顺序写入，写入流始终从区域写入指针位置开始。如果要写入的下一个zone不为空，则写入流“跳转”到该区域写入指针并继续。
对于随机写工作负载，写命令总是在写命令的目标区域的写指针位置发出。
任何针对已满（完全写入）的顺序区域的写入命令都会在发出写入 I/O 之前触发区域写入指针的reset。
默认情况下，所有读取命令始终以写入扇区为目标，即顺序写入区域的起始扇区和写入指针位置之间的扇区。可以禁用此行为，允许使用新选项 read_beyond_wp 向任何扇区发出读取命令。

此外，可以使用以下新选项添加对工作负载操作的更精细控制。

–max_open_zones 此选项限制工作负载正在写入的区域数。使用此选项，随机写入工作负载无法发出针对超过设置限制的区域的写入命令。一旦正在写入的区域变满，就会选择另一个区域并允许写入以该区域为目标，从而导致写入的区域数量恒定，始终最多等于 max_open_zones 限制。
–zone_reset_threshold 和 –zone_reset_frequency 这两个选项允许用户模拟应用程序发出的区域重置命令的执行。

除了这些选项之外，zbd 区域模式会自动启用作业同步，以确保跨多个线程或进程的工作负载可以同时执行针对同一区域的写入 I/O。

限制

正如内核支持文档中所讨论的，zoned块设备必须使用直接写入 I/O。 zbd 区域模式在启用时会通过检查是否为执行写入 I/O 的任何作业指定了选项 --direct=1 来强制执行此要求。

–offset 和 --size 选项必须指定与设备区域大小对齐的值。

zonemode=zbd 示例

本节提供各种示例，展示如何使用 fio new zbd zone mode 。在所有示例中，都使用了 15TB ZAC 主机管理的 SATA 磁盘。磁盘区域大小为 256 MiB。磁盘有 524 个常规区域，从偏移量 0 开始。磁盘的第一个顺序写入所需区域从扇区 274726912（512 B 扇区单元）开始，即字节偏移量 140660178944。

顺序写入工作负载

以下命令使用队列深度为 8 的 libaio I/O 引擎顺序写入磁盘的前 4 个顺序区域。

# fio --name=zbc --filename=/dev/sdd --direct=1 --zonemode=zbd \
      --offset=140660178944 --size=1G \
      --ioengine=libaio --iodepth=8 --rw=write --bs=256K
zbc: (g=0): rw=write, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=8
fio-3.13
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=239MiB/s][w=955 IOPS][eta 00m:00s]
zbc: (groupid=0, jobs=1): err= 0: pid=4124: Fri May 24 11:49:18 2019
  write: IOPS=938, BW=235MiB/s (246MB/s)(1024MiB/4365msec); 0 zone resets
    slat (nsec): min=5930, max=39068, avg=9729.06, stdev=2048.99
    clat (usec): min=783, max=55846, avg=8511.40, stdev=4079.36
     lat (usec): min=809, max=55854, avg=8521.19, stdev=4079.36
    clat percentiles (usec):
     |  1.00th=[ 3884],  5.00th=[ 7701], 10.00th=[ 8094], 20.00th=[ 8225],
     | 30.00th=[ 8225], 40.00th=[ 8225], 50.00th=[ 8225], 60.00th=[ 8291],
     | 70.00th=[ 8356], 80.00th=[ 8356], 90.00th=[ 8356], 95.00th=[ 8356],
     | 99.00th=[30540], 99.50th=[45351], 99.90th=[55837], 99.95th=[55837],
     | 99.99th=[55837]
   bw (  KiB/s): min=224830, max=249357, per=99.49%, avg=239009.50, stdev=9032.95, samples=8
   iops        : min=  878, max=  974, avg=933.50, stdev=35.36, samples=8
  lat (usec)   : 1000=0.02%
  lat (msec)   : 4=2.91%, 10=95.70%, 20=0.20%, 50=0.78%, 100=0.39%
  cpu          : usr=0.73%, sys=0.64%, ctx=1045, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4096,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=235MiB/s (246MB/s), 235MiB/s-235MiB/s (246MB/s-246MB/s), io=1024MiB (1074MB), run=4365-4365msec

Disk stats (read/write):
  sdd: ios=0/984, merge=0/2943, ticks=0/8365, in_queue=7383, util=24.13%

在磁盘处于此状态时，在未启用 zbd 区域模式的情况下再次执行相同的命令，fio 将尝试写入full zone，从而导致 I/O 错误。

# fio --name=zbc --filename=/dev/sdd --direct=1 \
      --offset=140660178944 --size=1G \
      --ioengine=libaio --iodepth=8 --rw=write --bs=256K
zbc: (g=0): rw=write, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=8
fio-3.13
Starting 1 process
fio: io_u error on file /dev/sdd: Remote I/O error: write offset=140660178944, buflen=262144
fio: pid=4206, err=121/file:io_u.c:1791, func=io_u error, error=Remote I/O error

zbc: (groupid=0, jobs=1): err=121 (file:io_u.c:1791, func=io_u error, error=Remote I/O error): pid=4206: Fri May 24 12:34:27 2019
  cpu          : usr=1.22%, sys=0.00%, ctx=3, majf=0, minf=16
  IO depths    : 1=12.5%, 2=25.0%, 4=50.0%, 8=12.5%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,8,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):

Disk stats (read/write):
  sdd: ios=48/3, merge=0/5, ticks=13/35, in_queue=32, util=5.39%

启用 zbd 区域模式后，在 zone 已满的情况下再次执行相同的命令会成功。

# fio --name=zbc --filename=/dev/sdd --direct=1 --zonemode=zbd \
      --offset=140660178944 --size=1G \
      --ioengine=libaio --iodepth=8 --rw=write --bs=256K
zbc: (g=0): rw=write, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=8
fio-3.13
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=243MiB/s][w=973 IOPS][eta 00m:00s]
zbc: (groupid=0, jobs=1): err= 0: pid=4220: Fri May 24 12:37:29 2019
  write: IOPS=949, BW=237MiB/s (249MB/s)(1024MiB/4316msec); 4 zone resets
    slat (nsec): min=5937, max=40055, avg=9651.92, stdev=2104.34
    clat (usec): min=795, max=36562, avg=8243.26, stdev=2031.55
     lat (usec): min=818, max=36571, avg=8252.96, stdev=2031.50
    clat percentiles (usec):
     |  1.00th=[ 3884],  5.00th=[ 7701], 10.00th=[ 8160], 20.00th=[ 8225],
     | 30.00th=[ 8225], 40.00th=[ 8225], 50.00th=[ 8225], 60.00th=[ 8291],
     | 70.00th=[ 8291], 80.00th=[ 8356], 90.00th=[ 8356], 95.00th=[ 8356],
     | 99.00th=[10159], 99.50th=[27919], 99.90th=[33817], 99.95th=[36439],
     | 99.99th=[36439]
   bw (  KiB/s): min=234538, max=249357, per=99.93%, avg=242777.88, stdev=5032.30, samples=8
   iops        : min=  916, max=  974, avg=948.25, stdev=19.70, samples=8
  lat (usec)   : 1000=0.02%
  lat (msec)   : 2=0.07%, 4=2.78%, 10=96.02%, 20=0.51%, 50=0.59%
  cpu          : usr=1.27%, sys=0.67%, ctx=1051, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4096,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=237MiB/s (249MB/s), 237MiB/s-237MiB/s (249MB/s-249MB/s), io=1024MiB (1074MB), run=4316-4316msec

Disk stats (read/write):
  sdd: ios=4/998, merge=0/2985, ticks=81/8205, in_queue=7286, util=24.38%

请注意，这种情况下的 fio 输出指出了在写入之前已重置的区域数量。

顺序读取工作负载

保留磁盘先前的状态（前四个顺序写区域已满），可以将先前的命令更改为针对写入区域的读取操作。

如果在执行此命令之前重置区域，则不会执行读取 I/O，因为 fio 将启用以查找具有写入扇区的区域。

可以使用 --read_beyond_wp 选项强制执行针对空区域的读取 I/O。

随机读写工作负载

以下命令使用 4 个job随机写入磁盘的顺序写入区域，每个job以 4 个队列深度运行（磁盘的整体队列深度为 16）。运行时间设置为 30 秒。

# fio --name=zbc --filename=/dev/sdd --direct=1 --zonemode=zbd \
      --offset=140660178944 --numjobs=4 --group_reporting=1 --runtime=30 \
      --ioengine=libaio --iodepth=4 --rw=randwrite --bs=256K
zbc: (g=0): rw=randwrite, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=4
...
fio-3.13
Starting 4 processes
Jobs: 4 (f=4): [w(4)][100.0%][w=33.0MiB/s][w=132 IOPS][eta 00m:00s]
zbc: (groupid=0, jobs=4): err= 0: pid=4425: Fri May 24 12:58:43 2019
  write: IOPS=160, BW=40.2MiB/s (42.2MB/s)(1209MiB/30064msec); 0 zone resets
    slat (nsec): min=7815, max=75710, avg=12304.05, stdev=3091.33
    clat (usec): min=1410, max=221285, avg=98856.71, stdev=29971.24
     lat (usec): min=1435, max=221295, avg=98869.07, stdev=29970.46
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   24], 10.00th=[   75], 20.00th=[   84],
     | 30.00th=[   89], 40.00th=[   94], 50.00th=[   99], 60.00th=[  105],
     | 70.00th=[  110], 80.00th=[  117], 90.00th=[  128], 95.00th=[  138],
     | 99.00th=[  194], 99.50th=[  203], 99.90th=[  218], 99.95th=[  220],
     | 99.99th=[  222]
   bw (  KiB/s): min=27092, max=138608, per=99.80%, avg=41096.13, stdev=3423.45, samples=240
   iops        : min=  103, max=  540, avg=159.05, stdev=13.38, samples=240
  lat (msec)   : 2=0.02%, 4=0.06%, 10=0.43%, 20=3.74%, 50=1.26%
  lat (msec)   : 100=47.27%, 250=47.21%
  cpu          : usr=0.09%, sys=0.15%, ctx=84537, majf=0, minf=261
  IO depths    : 1=0.1%, 2=0.2%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4836,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: bw=40.2MiB/s (42.2MB/s), 40.2MiB/s-40.2MiB/s (42.2MB/s-42.2MB/s), io=1209MiB (1268MB), run=30064-30064msec

Disk stats (read/write):
  sdd: ios=0/4807, merge=0/0, ticks=0/474905, in_queue=470163, util=9.89%

zbc_report_zones 可用于在此工作负载执行结束时探索磁盘的状态。

# zbc_report_zones -ro full -n /dev/sdd
Device /dev/sdd:
    Vendor ID: ATA xxxx xxxxxxxxxxx xxxx
    Zoned block device interface, Host-managed zone model
    29297213440 512-bytes sectors
    3662151680 logical blocks of 4096 B
    3662151680 physical blocks of 4096 B
    15000.173 GB capacity
    Read commands are unrestricted
    Maximum number of open sequential write required zones: 128
    0 zone from 0, reporting option 0x05
# zbc_report_zones -ro closed -n /dev/sdd
Device /dev/sdd:
    Vendor ID: ATA xxxx xxxxxxxxxxx xxxx
    Zoned block device interface, Host-managed zone model
    29297213440 512-bytes sectors
    3662151680 logical blocks of 4096 B
    3662151680 physical blocks of 4096 B
    15000.173 GB capacity
    Read commands are unrestricted
    Maximum number of open sequential write required zones: 128
    4498 zones from 0, reporting option 0x04
# zbc_report_zones -ro imp_open -n /dev/sdd
Device /dev/sdd:
    Vendor ID: ATA xxxx xxxxxxxxxxx xxxx
    Zoned block device interface, Host-managed zone model
    29297213440 512-bytes sectors
    3662151680 logical blocks of 4096 B
    3662151680 physical blocks of 4096 B
    15000.173 GB capacity
    Read commands are unrestricted
    Maximum number of open sequential write required zones: 128
    128 zones from 0, reporting option 0x02

这表明写入了 4498+128=4626 个区域，没有一个顺序写入区域完全写入（没有完整区域）。将操作模式切换为读取，可以随机读取上次运行中写入的扇区。

# fio --name=zbc --filename=/dev/sdd --direct=1 --zonemode=zbd \                
      --offset=140660178944 --numjobs=4 --group_reporting=1 --runtime=30 \      
      --ioengine=libaio --iodepth=4 --rw=randread --bs=256K
...
fio-3.13
Starting 4 processes
Jobs: 4 (f=4): [r(4)][0.0%][r=31.0MiB/s][r=124 IOPS][eta 23d:02h:02m:24s]
zbc: (groupid=0, jobs=4): err= 0: pid=4494: Fri May 24 13:24:08 2019
  read: IOPS=118, BW=29.7MiB/s (31.1MB/s)(894MiB/30156msec)
    slat (usec): min=6, max=183, avg= 7.86, stdev= 8.75
    clat (usec): min=1252, max=1537.8k, avg=133931.63, stdev=123584.49
     lat (usec): min=1260, max=1537.8k, avg=133939.56, stdev=123584.42
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[   16], 10.00th=[   20], 20.00th=[   35],
     | 30.00th=[   53], 40.00th=[   72], 50.00th=[   96], 60.00th=[  127],
     | 70.00th=[  169], 80.00th=[  218], 90.00th=[  296], 95.00th=[  368],
     | 99.00th=[  558], 99.50th=[  625], 99.90th=[  869], 99.95th=[  919],
     | 99.99th=[ 1536]
   bw (  KiB/s): min=15855, max=50152, per=100.00%, avg=30364.05, stdev=1762.17, samples=240
   iops        : min=   60, max=  195, avg=117.22, stdev= 6.92, samples=240
  lat (msec)   : 2=0.22%, 4=0.06%, 10=1.40%, 20=8.44%, 50=18.26%
  lat (msec)   : 100=23.15%, 250=33.30%, 500=13.47%, 750=1.51%, 1000=0.17%
  cpu          : usr=0.06%, sys=0.12%, ctx=82815, majf=0, minf=1281
  IO depths    : 1=0.1%, 2=0.2%, 4=99.7%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3577,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=29.7MiB/s (31.1MB/s), 29.7MiB/s-29.7MiB/s (31.1MB/s-31.1MB/s), io=894MiB (938MB), run=30156-30156msec

Disk stats (read/write):
  sdd: ios=3565/0, merge=0/0, ticks=475774/0, in_queue=472274, util=11.93%

直接访问 sg I/O 引擎

SCSI 通用直接访问接口也可以与 zbd 区域模式一起使用，只要使用块设备文件 (/dev/sdX) 来指定磁盘即可。如果使用 SCSI 通用节点文件 (/dev/sgY) 指定磁盘，则不会启用 zbd 区域模式。
下面的示例说明了使用 sg I/O 引擎和 8 个作业对顺序写入区域执行 64KB 随机写入工作负载。

# fio --name=zbc --filename=/dev/sdd --direct=1 --zonemode=zbd \
      --offset=140660178944 --size=1G --numjobs=8 --group_reporting=1 \
      --ioengine=sg --rw=randwrite --bs=64K
...
fio-3.13
Starting 8 processes
zbc: (groupid=0, jobs=8): err= 0: pid=4792: Fri May 24 14:18:52 2019
  write: IOPS=2007, BW=125MiB/s (132MB/s)(8192MiB/65278msec); 32 zone resets
    clat (usec): min=148, max=327494, avg=1589.26, stdev=4060.13
     lat (usec): min=149, max=327497, avg=1590.55, stdev=4060.13
    clat percentiles (usec):
     |  1.00th=[   848],  5.00th=[   988], 10.00th=[  1090], 20.00th=[  1172],
     | 30.00th=[  1221], 40.00th=[  1287], 50.00th=[  1369], 60.00th=[  1434],
     | 70.00th=[  1500], 80.00th=[  1631], 90.00th=[  2180], 95.00th=[  2638],
     | 99.00th=[  3064], 99.50th=[  3392], 99.90th=[ 24773], 99.95th=[ 77071],
     | 99.99th=[291505]
   bw (  KiB/s): min=39001, max=178678, per=100.00%, avg=131457.71, stdev=3143.56, samples=1015
   iops        : min=  606, max= 2791, avg=2052.81, stdev=49.16, samples=1015
  lat (usec)   : 250=0.07%, 500=0.12%, 750=0.38%, 1000=4.89%
  lat (msec)   : 2=81.56%, 4=12.76%, 10=0.05%, 20=0.06%, 50=0.05%
  lat (msec)   : 100=0.03%, 250=0.02%, 500=0.01%
  cpu          : usr=0.14%, sys=0.20%, ctx=251460, majf=0, minf=139
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,131072,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=125MiB/s (132MB/s), 125MiB/s-125MiB/s (132MB/s-132MB/s), io=8192MiB (8590MB), run=65278-65278msec

Disk stats (read/write):
  sdd: ios=278/0, merge=0/0, ticks=1953/0, in_queue=1677, util=0.39%

注意 SCSI 通用直接访问绕过块层 I/O 调度程序。对于分区块设备，这意味着启用最后期限 I/O 调度程序区域写入锁定以提供写入命令顺序保证。但是，zbd 模式可确保作业之间互斥，以便对同一区域进行写访问。这种同步本质上与区域写锁定相同，并且执行所有写命令而没有任何错误。

区域写入流

典型的分区块设备兼容应用程序将顺序写入区域，直到该区域已满，然后切换到另一个区域并继续写入。多个线程可以以这种方式运行，每个线程在不同的区域上运行。

可以使用选项 --rw_sequencer 以及在 --rw=randwrite 参数末尾指定的许多 I/O 操作来模拟这种典型行为。下面是一个示例脚本，其中 4 个作业使用 512KB 写入操作（即每 256 MB 区域 512 个 I/O）按顺序将区域写入到满。正在写入的区域是在每个作业的不相交区域范围内随机选择的。这由偏移量、大小和 rw 参数控制。实现这种工作负载的脚本文件streams.fio 如下所示。

#
# streams.fio: 4 write streams
#

[global]
ioengine=psync
direct=1
thread=1
bs=512K
continue_on_error=none
filename=/dev/sdd
group_reporting=1
zonemode=zbd

[stream1]
offset=140660178944
size=3714878275584
rw=randwrite:512
rw_sequencer=sequential
io_size=2G

[stream2]
offset=3855538454528
size=3714878275584
rw=randwrite:512
rw_sequencer=sequential
io_size=2G

[stream3]
offset=7570416730112
size=3714878275584
rw=randwrite:512
rw_sequencer=sequential
io_size=2G

[stream4]
offset=11285295005696
size=3714878275584
rw=randwrite:512
rw_sequencer=sequential
io_size=2G

此脚本执行的结果如下所示。

# fio streams.fio
stream1: (g=0): rw=randwrite, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
stream2: (g=0): rw=randwrite, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
stream3: (g=0): rw=randwrite, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
stream4: (g=0): rw=randwrite, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1
fio-3.13
Starting 4 threads
Jobs: 1 (f=1): [_(3),w(1)][98.3%][w=159MiB/s][w=318 IOPS][eta 00m:01s]
stream1: (groupid=0, jobs=4): err= 0: pid=5161: Fri May 24 15:01:21 2019
  write: IOPS=285, BW=143MiB/s (150MB/s)(8192MiB/57362msec); 0 zone resets
    clat (usec): min=992, max=5788.2k, avg=12731.27, stdev=95315.03
     lat (usec): min=996, max=5788.2k, avg=12742.01, stdev=95315.03
    clat percentiles (usec):
     |  1.00th=[   1106],  5.00th=[   2024], 10.00th=[   2114],
     | 20.00th=[   2278], 30.00th=[   2769], 40.00th=[   3687],
     | 50.00th=[   3884], 60.00th=[   4047], 70.00th=[   4228],
     | 80.00th=[   4817], 90.00th=[   5342], 95.00th=[  50594],
     | 99.00th=[ 196084], 99.50th=[ 258999], 99.90th=[ 876610],
     | 99.95th=[2071987], 99.99th=[4731175]
   bw (  KiB/s): min= 9211, max=840925, per=100.00%, avg=185793.45, stdev=44491.80, samples=352
   iops        : min=   17, max= 1642, avg=361.42, stdev=86.97, samples=352
  lat (usec)   : 1000=0.23%
  lat (msec)   : 2=1.95%, 4=54.69%, 10=37.05%, 20=0.15%, 50=0.90%
  lat (msec)   : 100=2.34%, 250=2.16%, 500=0.42%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2000=0.04%, >=2000=0.06%
  cpu          : usr=0.15%, sys=0.12%, ctx=16505, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16384,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=8192MiB (8590MB), run=57362-57362msec

Disk stats (read/write):
  sdd: ios=144/16333, merge=0/0, ticks=573/208225, in_queue=192356, util=28.64%