Transfer rates Nikhef → BNL

Atlas reported transfer rate errors from Nikhef to BNL, starting end of February. Since this coincides with the commissioning of the new oliebollen storage nodes, there is a reason to look at these machines in particular.

I took a number of measurements by transferring large files from our storage nodes to the BNL SRM system with the lcg-cp command.

The transfer rates were not consistent, but the achievable rates were roughly as follows:

srcdeststreamsrate (KB/s)

The command-lines used were:

lcg-cp -v -n 10  srm:// srm://

(for a large data file on oliebol-01);

lcg-cp -v -n 1  srm:// srm://

(for a large file on biet-1).

By taking packet captures and plotting I/O rates (thanks to wireshark) we can compare the network flow in these different situations.


Bytes per 0.01s during transfer with 10 streams from oliebol-01

The worst case, with 10 streams from oliebol-01, resulted in TCP frame sizes that never exceeded 2962 bytes.


Bytes per 0.01s during transfer with a single stream from oliebol-01

The best rate from oliebol-01 was obtained by using a single stream. But this resulted in an enormous flood of packets. The ~7 second capture amounted to about 200MB of data (compared to about 4MB for the other captures). The largest frames were 33370 bytes.


Bytes per 0.01s during transfer with a single stream from biet-1

Comparing with 10 streams from biet-1, we see frame sizes as large as 64294, and more of them bunched together.

Date: 2014-04-10

Author: Dennis van Dok

Org version 7.8.11 with Emacs version 24

Validate XHTML 1.0