Performance tests on an Infortrend EonStor 16TB RAID

Author: L.S.Lowe. File: raidperf10. Tests performed 200708. This minor update: 20120725. Part of Guide to the Local System.

Note: also see my most recent Infortrend RAIDs and the index.

These are some performance tests on a Infortrend EonStor RAID system, A24U-G2421-1, attached via a LSI22320RB-F scsi HBA card, also known as LSI22320-R, with file-system sizes of 1.7 TB and 7.5 TB.

I have a LSI22320RB-F card fitted in a PowerEdge 1950, and attached to a transtec RAID (rebadged Infortrend EonStor). The operating system is currently CentOS 5 32-bit.

See my page RAID 2 TB limit for information on what was necessary to get this setup working with a 8TB file-system. Also see my page Performance tests on ext4 and xfs on an Infortrend RAID for more recent information.

Read performance

Here are some results for some block read-rate tests done by bonnie++, with various read-ahead buffer settings. The read-ahead setting was set by the command:
         blockdev --setra $rab /dev/$dev
where $rab is the read-ahead buffer size in 512-byte sectors; this was equivalent in 2.6 kernel to doing:
         echo $rabkb > /sys/block/$dev/queue/read_ahead_kb 
where $rabkb is the read-ahead buffer size in kBytes. The system default is 128 kBytes.

The RAID has a 1GB cache and the server has 2GB RAM. The RAID was configured for RAID-6, with a stripe-size of 256 kBytes originally, though the stripe-size was changed back to 128 kBytes (the factory default) because this performed slightly better. The bonnie++ file-size was always 8GB.

Each read-ahead buffer setting was tested with 24 variations, which is why there are 24 data points per read-ahead value: 4 different file-system setups for ext2 and ext3, and 6 different bonnie++ buffer-size settings (8kB to 256kB) within each file-system setting. It demonstrates that the read-ahead buffer size is the dominant performance determinator, and that different file-systems and options for ext2/ext3 affect the read performance hardly at all. Very large sizes of 8MB or larger have a minor adverse affect on ext3 read performance. The default read-ahead buffer size of 128 kB gives around 65% of the optimum performance, which was achieved with settings in the range 1-4 MB, so it's worth tuning this size to 1 or 2 MB. The best block read-rate was around 240 MBytes/sec.

Write performance for different ext2/ext3 filesystem options

The block write performance for different file-system types was measured. These types were: ext2, ext3j data=journal, ext3o data=ordered, ext3w data=writeback. The bonnie++ file size was 8 GB in all cases, and the buffer size was 8 kB to 256 kB, as indicated along the horizontal axis. The write performance was, as you'd expect, unaffected by read-ahead buffer size! The ext2 performance clearly wins, at around 245 MBytes/sec, but ext3 with the default option of data=ordered is a reasonable runner-up.

Timings for mkfs on an 7.5 TB RAID

A mkfs -t ext3 -E stride=32 with the default number of inodes (one per 8192 bytes) on this 10b filesystem took 23m53s. It was then tuned as follows: tune2fs -c 0 -i 0 -r 102400 devicename. This tunes the system reserved area to a mere 400 MB (should be enough!) and turns off regular full fsck checking, so occasional full fsck checks will have to be done by hand, at a convenient time.
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sdc 7208165240 182888 7207572752 1% /disk/10b
Since we are going to put mainly large files on this filesystem, we can make do with a lot fewer inodes. Fewer inodes give me a small gain in available file-space, and reduce a full fsck time a bit (see a section below), though it's a case of diminishing returns. I settle on a bytes-per-inode of 65536, eight times the default.

A mkfs -t ext3 -E stride=32 -i 65536 on this 10b filesystem took 26m44s. Surprising it wasn't less than before, but there we go. Same tuning. We gain an extra 100 GB of space, for what it's worth!

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sdc 7308286072 182888 7307693584 1% /disk/10b

Timings for normal fsck using journal on an 7.5 TB RAID

This was tested with a file-system formatted with ext3, 41% full, 3M/114M used/total inodes, 741M/1830M blocks, as follows:
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sdc 7308286072 2952254708 4355621764 41% /disk/10b

The data on this disk was arranged with 3 directories each containing 1000 sub-directories each containing 1000 1MB flat-files.

The time for a journal recovery after a deliberate system crash was under 1 second, as shown:

# time fsck.ext3 /dev/sdc
e2fsck 1.39 (29-May-2006)
10b: recovering journal
10b: clean, 3003020/114423808 files, 741760799/1830768640 blocks

real 0m0.930s
user 0m0.397s
sys 0m0.077s

Timings for full fsck on an 7.5 TB RAID

The time for a full fsck.ext3 (with -f and -p options), was measured for several different levels of disk occupancy. The data on the disk was arranged with one to six directories, each containing 1000 sub-directories each containing 1000 1MB flat-files. The time for a full fsck.ext3 (with -f and -p options), again with a 41% full file-system, was 2h33m: this is not unexpected for this size of ext3 filesystem, and is rarely required:
# date; time fsck.ext3 -p -f /dev/sdc; date
Wed Mar 28 20:04:23 BST 2007
10b: 3003020/114423808 files (6.4% non-contiguous), 741760799/1830768640 blocks

real 153m7.227s
user 1m17.915s
sys 2m3.596s
Wed Mar 28 22:37:30 BST 2007
This was re-tested for various other disk utilisations, and different max-inode settings. The results are below, sorted by time taken:

mkfs.ext3 -i option used nodes total inodes total 4kB blocks used blocks timing (minutes)
65536 0.0M 114M 1830M 0% 34m
65536 1.0M 114M 1830M 14% 70m
8192 1.0M 915M 1830M 15% 111m
8192 2.0M 915M 1830M 28% 143m
819200 3.0M 9M 1830M 40% 141m
65536 3.0M 114M 1830M 41% 153m
8192 3.0M 915M 1830M 42% 170m
819200 4.0M 9M 1830M 54% 180m
8192 4.0M 915M 1830M 55% 204m
8192 5.0M 915M 1830M 69% 236m
819200 6.0M 9M 1830M 81% 261m
8192 6.0M 915M 1830M 82% 274m


An important thing to do of course is to use tune2fs to set maximum mount count and check interval to suitable values, so that a full fsck is done at the system manager's convenience, not at the convenience of the system after a power-failure when you're trying to get everything back online quickly!

Also see my page Performance tests on ext4 and xfs on an Infortrend RAID for more recent information.