Performance tests for ext4 over nfs on an Infortrend
RAID
Author: L.S.Lowe. File: raidperf11nfs. This update: 20081212.
Part of Guide to the Local System.
Read/write performance over nfs to ext4 and effect of tuning
Currently very disorganised
notes to accompany another page on native
ext4 and xfs performance.
Set up with a default no-options NFS mount of the ext4 filesystem on a
client. Server and client have gigabit ethernet connectivity, so
maximum network speed was around 125 MB/s. Tests were done using dd, which reports speeds in MB/s
(not MiB/s). The records in/out
lines from dd have been omitted after the first. No kernel tuning is in
effect. Server system remains Fedora10, client systems are Centos 5 and
SL5.2.
client$ dd if=/dev/zero of=/disk/11a/copy/10GiBx bs=131072 count=80000
80000+0 records in
80000+0 records out
10485760000 bytes (10 GB) copied, 151.084 seconds, 69.4 MB/s
client$ dd if=/dev/zero of=/disk/11a/copy/10GiBy bs=131072 count=80000
10485760000 bytes (10 GB) copied, 153.358 seconds, 68.4 MB/s
With read-ahead buffer on server defaulted at 256 sectors:
client$ dd if=/disk/11a/copy/10GiBx of=/dev/null bs=131072
10485760000 bytes (10 GB) copied, 99.8303 seconds, 105 MB/s
With read-ahead buffer on server set to 4096 sectors:
client$ dd if=/disk/11a/copy/10GiBy of=/dev/null bs=131072
10485760000 bytes (10 GB) copied, 89.2208 seconds, 118 MB/s
The above was with mount option barrier=1 (default). With barrier=0,
the following write timing applied, which was no different:
client$ dd if=/dev/zero of=/disk/11a/copy/10GiBz bs=131072 count=80000
10485760000 bytes (10 GB) copied, 152.414 seconds, 68.8 MB/s
With a netcat connection, rather than NFS, the write speed was as
follows:
server# nc -v -l myport > /disk/11a/copy/10GiBnc
client$ dd if=/dev/zero bs=131072 count=80000 | nc -v myserver myport
Connection to myserver myport succeeded!
10485760000 bytes (10 GB) copied, 111.042 seconds, 94.4 MB/s
Going back to NFS, and increasing the RPCNFSDCOUNT to a larger number
(24) in place of the default 8, gave the following improvement:
client$ dd if=/dev/zero of=/disk/11a/copy/10GiB24a bs=131072 count=80000
10485760000 bytes (10 GB) copied, 109.926 seconds, 95.4 MB/s
However, returning RPCNFSDCOUNT to 8, and restarting nfsd gave no real
reduction in performance!
client$ dd if=/dev/zero of=/disk/11a/copy/10GiB08a bs=131072 count=80000
10485760000 bytes (10 GB) copied, 111.204 seconds, 94.3 MB/s
After a reboot of the server, local barrier=0 mount, and NFS
remount:
client$ dd if=/dev/zero of=/disk/11a/copy/10GiB08d bs=131072 count=80000
10485760000 bytes (10 GB) copied, 154.725 seconds, 67.8 MB/s
That client was shared with other network-busy tasks, so I changed to a
quiet client on system SL5.2. Freshly booted server to kernel
2.6.27.7-134.fc10.i686, file-system mounted with barrier=0, NFS mounted
without special options:
client9$ dd if=/dev/zero of=/disk/11a/copy/10GiB-9-08a bs=131072 count=80000
10485760000 bytes (10 GB) copied, 141.419 seconds, 74.1 MB/s
client9$ dd if=/dev/zero of=/disk/11a/copy/10GiB-9-08b bs=131072 count=80000
10485760000 bytes (10 GB) copied, 140.635 seconds, 74.6 MB/s
client9$ dd if=/dev/zero of=/disk/11a/copy/10GiB-9-24a bs=131072 count=80000
10485760000 bytes (10 GB) copied, 140.415 seconds, 74.7 MB/s
Now same client and server but using nc/netcat not NFS:
server8# nc -v -l port > /disk/11a/copy/10GiBnc3
client9$ dd if=/dev/zero bs=131072 count=80000 | nc -v server port
Connection to server port succeeded!
80000+0 records in
80000+0 records out
10485760000 bytes (10 GB) copied, 112.81 seconds, 93.0 MB/s
With UDP mount in place of tcp mount, slightly better than TCP:
10485760000 bytes (10 GB) copied, 149.506 seconds, 70.1 MB/s
/proc/sys/net/core/rmem_max and rmem_default 131071 and 112640 on the
server.
With /proc/sys/net/core/rmem_max and rmem_default set to 256k, and
network and nfs restarted:
10485760000 bytes (10 GB) copied, 134.64 seconds, 77.9 MB/s
no improvement, so reset these (by rebooting!).
Mount with bg,hard,intr,nolock,tcp but no nfs rsize or wsize options:
10485760000 bytes (10 GB) copied, 138.039 seconds, 76.0 MB/s
10485760000 bytes (10 GB) copied, 138.319 seconds, 75.8 MB/s
10485760000 bytes (10 GB) copied, 137.818 seconds, 76.1 MB/s
10485760000 bytes (10 GB) copied, 137.311 seconds, 76.4 MB/s
10485760000 bytes (10 GB) copied, 133.869 seconds, 78.3 MB/s
Mount with bg,hard,intr,nolock,tcp,rsize=32768,wsize=32768 :
10485760000 bytes (10 GB) copied, 167.588 seconds, 62.6 MB/s
With no rsize or wsize, seeing some large Recv-Q sizes in a netstat -n
| grep 2049 on the server:
tcp 1202672 0
server8:2049
client9:958
ESTABLISHED
whereas with rsize=32768,wsize=32768, max is
tcp 526400
0
server8:2049
client9:958
ESTABLISHED
mount with bg,hard,intr,nolock,tcp,rsize=131072,wsize=131072:
10485760000 bytes (10 GB) copied, 150.168 seconds, 69.8 MB/s
bg,hard,intr,nolock,tcp,rsize=262144,wsize=262144
10485760000 bytes (10 GB) copied, 131.356 seconds, 79.8 MB/s
10485760000 bytes (10 GB) copied, 138.867 seconds, 75.5 MB/s
bg,hard,intr,nolock,tcp,rsize=524288,wsize=524288
10485760000 bytes (10 GB) copied, 142.459 seconds, 73.6 MB/s
bg,hard,intr,nolock,tcp,rsize=1048576,wsize=1048576
10485760000 bytes (10 GB) copied, 139.349 seconds, 75.2 MB/s
rejig network so that joined to common gig switch,
bg,hard,intr,nolock,tcp
10485760000 bytes (10 GB) copied, 142.875 seconds, 73.4 MB/s
L.S.Lowe