Performance tests for ext4 over nfs on an Infortrend RAID

Author: L.S.Lowe. File: raidperf11nfs. This update: 20081212. Part of Guide to the Local System.

Read/write performance over nfs to ext4 and effect of tuning

Currently very disorganised notes to accompany another page on native ext4 and xfs performance.

Set up with a default no-options NFS mount of the ext4 filesystem on a client. Server and client have gigabit ethernet connectivity, so maximum network speed was around 125 MB/s. Tests were done using dd, which reports speeds in MB/s (not MiB/s). The records in/out lines from dd have been omitted after the first. No kernel tuning is in effect. Server system remains Fedora10, client systems are Centos 5 and SL5.2.
client$ dd if=/dev/zero of=/disk/11a/copy/10GiBx bs=131072 count=80000
80000+0 records in
80000+0 records out
10485760000 bytes (10 GB) copied, 151.084 seconds, 69.4 MB/s
client$ dd if=/dev/zero of=/disk/11a/copy/10GiBy bs=131072 count=80000
10485760000 bytes (10 GB) copied, 153.358 seconds, 68.4 MB/s
With read-ahead buffer on server defaulted at 256 sectors:
client$ dd if=/disk/11a/copy/10GiBx of=/dev/null bs=131072
10485760000 bytes (10 GB) copied, 99.8303 seconds, 105 MB/s
With read-ahead buffer on server set to 4096 sectors:
client$ dd if=/disk/11a/copy/10GiBy of=/dev/null bs=131072
10485760000 bytes (10 GB) copied, 89.2208 seconds, 118 MB/s
The above was with mount option barrier=1 (default). With barrier=0, the following write timing applied, which was no different:
client$ dd if=/dev/zero of=/disk/11a/copy/10GiBz bs=131072 count=80000
10485760000 bytes (10 GB) copied, 152.414 seconds, 68.8 MB/s
With a netcat connection, rather than NFS, the write speed was as follows:
server# nc -v -l myport > /disk/11a/copy/10GiBnc
client$ dd if=/dev/zero bs=131072 count=80000 | nc -v myserver myport
Connection to myserver myport succeeded!
10485760000 bytes (10 GB) copied, 111.042 seconds, 94.4 MB/s
Going back to NFS, and increasing the RPCNFSDCOUNT to a larger number (24) in place of the default 8, gave the following improvement:
client$ dd if=/dev/zero of=/disk/11a/copy/10GiB24a bs=131072 count=80000
10485760000 bytes (10 GB) copied, 109.926 seconds, 95.4 MB/s
However, returning RPCNFSDCOUNT to 8, and restarting nfsd gave no real reduction in performance!
client$ dd if=/dev/zero of=/disk/11a/copy/10GiB08a bs=131072 count=80000
10485760000 bytes (10 GB) copied, 111.204 seconds, 94.3 MB/s
After a reboot of the server,  local barrier=0 mount, and NFS remount:
client$ dd if=/dev/zero of=/disk/11a/copy/10GiB08d bs=131072 count=80000
10485760000 bytes (10 GB) copied, 154.725 seconds, 67.8 MB/s

That client was shared with other network-busy tasks, so I changed to a quiet client on system SL5.2. Freshly booted server to kernel 2.6.27.7-134.fc10.i686, file-system mounted with barrier=0, NFS mounted without special options:

client9$ dd if=/dev/zero of=/disk/11a/copy/10GiB-9-08a bs=131072 count=80000
10485760000 bytes (10 GB) copied, 141.419 seconds, 74.1 MB/s
client9$ dd if=/dev/zero of=/disk/11a/copy/10GiB-9-08b bs=131072 count=80000
10485760000 bytes (10 GB) copied, 140.635 seconds, 74.6 MB/s
client9$ dd if=/dev/zero of=/disk/11a/copy/10GiB-9-24a bs=131072 count=80000
10485760000 bytes (10 GB) copied, 140.415 seconds, 74.7 MB/s
Now same client and server but using nc/netcat not NFS:
server8# nc -v -l port > /disk/11a/copy/10GiBnc3
client9$ dd if=/dev/zero bs=131072 count=80000 | nc -v server port
Connection to server port succeeded!
80000+0 records in
80000+0 records out
10485760000 bytes (10 GB) copied, 112.81 seconds, 93.0 MB/s
With UDP mount in place of tcp mount, slightly better than TCP:
10485760000 bytes (10 GB) copied, 149.506 seconds, 70.1 MB/s

/proc/sys/net/core/rmem_max and rmem_default 131071 and 112640 on the server.
With /proc/sys/net/core/rmem_max and rmem_default set to 256k, and network and nfs restarted:
10485760000 bytes (10 GB) copied, 134.64 seconds, 77.9 MB/s
no improvement, so reset these (by rebooting!).

Mount with bg,hard,intr,nolock,tcp but no nfs rsize or wsize options:
10485760000 bytes (10 GB) copied, 138.039 seconds, 76.0 MB/s
10485760000 bytes (10 GB) copied, 138.319 seconds, 75.8 MB/s
10485760000 bytes (10 GB) copied, 137.818 seconds, 76.1 MB/s
10485760000 bytes (10 GB) copied, 137.311 seconds, 76.4 MB/s
10485760000 bytes (10 GB) copied, 133.869 seconds, 78.3 MB/s

Mount with bg,hard,intr,nolock,tcp,rsize=32768,wsize=32768 :
10485760000 bytes (10 GB) copied, 167.588 seconds, 62.6 MB/s

With no rsize or wsize, seeing some large Recv-Q sizes in a netstat -n | grep 2049 on the server:
tcp   1202672      0   server8:2049         client9:958          ESTABLISHED

whereas with rsize=32768,wsize=32768, max is
tcp   526400      0    server8:2049         client9:958          ESTABLISHED

mount with bg,hard,intr,nolock,tcp,rsize=131072,wsize=131072:
10485760000 bytes (10 GB) copied, 150.168 seconds, 69.8 MB/s

bg,hard,intr,nolock,tcp,rsize=262144,wsize=262144
10485760000 bytes (10 GB) copied, 131.356 seconds, 79.8 MB/s
10485760000 bytes (10 GB) copied, 138.867 seconds, 75.5 MB/s


bg,hard,intr,nolock,tcp,rsize=524288,wsize=524288
10485760000 bytes (10 GB) copied, 142.459 seconds, 73.6 MB/s

bg,hard,intr,nolock,tcp,rsize=1048576,wsize=1048576
10485760000 bytes (10 GB) copied, 139.349 seconds, 75.2 MB/s

rejig network so that joined to common gig switch,
bg,hard,intr,nolock,tcp
10485760000 bytes (10 GB) copied, 142.875 seconds, 73.4 MB/s


L.S.Lowe