ZFS Write Performance (Impact of fragmentation)

At Delphix we are constantly trying to improve performance. Some of the enhancements come from customers where we see performance pathologies that need to be addressed while some come from our understanding of the bottlenecks in Delphix OS. In the past few months, George Wilson and Adam Leventhal made significant improvements to how writes are handled in ZFS. In this multi part blog post, I will talk about a benchmark we created to  measure improvements in ZFS write performance as we make changes to the OS. In this post, I will talk about the benchmark setup and run. I will show some of the results from this benchmark on Delphix OS. In part two, I will present data and some analysis on the bottlenecks discovered and how they are going to be addressed.

The Benchmark

The benchmark is a simple program that writes random data to random offsets in a file. I am pleasantly surprised at how much I was able to learn from this simple 20-line program. Hopefully there is information here that would help some of you as well. The program was thought up by Adam and George as a way to observe the behavior of ZFS when a zpool is fragmented.

 

  randFd = open("/dev/urandom", O_RDONLY);
  fd = open(argv[2], O_CREAT|O_WRONLY|O_APPEND, 0777);
  .
  .
  while (1) {
    read(randFd, buf, sizeof(char)*BLOCKSIZE);
    pwrite(fd, buf, BLOCKSIZE, (lrand48() % (test_size / BLOCKSIZE)) * BLOCKSIZE);
  }

 

I ran the program on a clean zpool and observed the performance of the system using DTrace. The program fragments the pool as it writes to random offsets. I wanted to measure the steady state throughput achieved as a function of the zpool usage (10% full, 50% full etc.). This will demonstrate interesting ZFS pathologies and track our progress from release to release.

Benchmark Setup

Before each run, I created a new zpool and a filesystem with record size of 8k, using the following commands.

zpool create domain0 c3t1d0
zfs create -o recordsize=8k -o compression=on domain0/tests

 

I used an 8 vCPU VM, with 48GB of RAM and 32GB of pool size for the Delphix server. I used a small pool size so I could fill it quickly. While 32GB is far smaller than a typical customer’s pool, I believe that the results are still representative. I ran eight concurrent instances of the program to generate load. (I also tested the benchmark with fewer than eight writers, it showed similar characteristics, but took longer to reach steady state.)

Benchmark Run

Once I created a zpool, I populated it with random data using “dd”.

dd if=/dev/urandom of=/domain0/tests/testfileX.dat bs=8k count=Y

I monitored the file system using dtrace as the pool was getting full. The output below shows a histogram of IOs, by size, being performed at the backend storage while the above “dd” command is running. The majority of writes are 64K and 128K in size, even though I did 8k writes at the user level. This aggregation is performed at the IO layer in ZFS.

2013 Feb 16 18:45:42 (14.999s elapsed) /engineering/uday/scripts/io.d

  bytes                                               write                                             
           value  ------------- Distribution ------------- count    
             256 |                                         0        
             512 |@                                        309      
            1024 |                                         217      
            2048 |@                                        281      
            4096 |@                                        230      
            8192 |@                                        231      
           16384 |                                         176      
           32768 |                                         119      
           65536 |@@@@@                                    2352     
          131072 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@          14321    
          262144 |                                         0        

                                avg latency         iops   throughput
write                                7311us       1302/s    142394k/s

total                                             1302/s    142394k/s

 

ZFS divides each IO device into a few hundred regions called “Metaslabs”. The figure 1 below shows a visualization of the zpool after it is filled to 25% using "dd" command above. Each cell represents one metaslab. The pool here has 256, 128MB Metaslabs. The percentage value is the amount of free space in each metaslab. As you can see, around 32 metaslabs are full, accounting for 8GB of data.

At this point, the pool is compact -most metaslabs are either full or empty and consume minimal space on disk. When I started my program with the zpool in the above state, I saw an initial spurt of high write throughput followed by a steep fall in throughput stabilizing after some time. The figure 2 below depicts the zpool after the program ran for 30 mins -note that pool is still only 25% full. This image did not change even after running the program for a few hours longer, the benchmark had reached steady state.

As you can see the pool is fairly fragmented at this point. The pool is not compact, 2.5x times as many metaslabs are used. The majority of metaslabs have regions of free space. So what is going on here? When the program started writing to random blocks in the zpool, ZFS tried to place the random blocks in contiguous region of the pool by allocating new metaslabs. Due to copy-on-write, every time a block is overwritten it will create a hole somewhere in the pool. New data blocks were allocate to new metaslabs until ZFS found free regions within currently allocated metaslabs. At this point, ZFS started filling those holes before allocating to new metaslabs. Hence the number of metaslabs used did not increase after a point in the test. The chart in Fig 3 below shows write throughput during the benchmark, at different usage level of the zpool.

After the initial high write throughput, there is a precipitous drop with the benchmark settling at a steady state throughput. Steady state throughput is computed using average of throughput from the final 50 samples in the experiment. The steady state throughput is proportional to the amount of free space in the zpool at the beginning of the benchmark run. The chart in Fig 4 below plots this throughput against zpool occupancy.

Conclusion

The experiments show an interesting performance pathology as ZFS pools become increasingly full and fragmented. Visualizing the metaslabs and their usage through the benchmark run, helped understand the reasons behind the drop in performance. The Fig.4 above shows a tight correlation between amount of free space in the pool and the random write throughput achieved. This test provides an interesting benchmark for future ZFS performance work. We can both work to push the curve to the right, delaying the point at which performance degrades, and push the curve up, improving the worst case. The simple, 20-line C program proved to be a fountain of useful information. In my next blog post I'll dissect the results in more detail, and provide some insight into what ZFS is doing -- and how we might improve it.