Transfer rates Nikhef → BNL
Atlas reported transfer rate errors from Nikhef to BNL, starting end of February. Since this coincides with the commissioning of the new oliebollen storage nodes, there is a reason to look at these machines in particular.
I took a number of measurements by transferring large files from
our storage nodes to the BNL SRM system with the lcg-cp command.
The transfer rates were not consistent, but the achievable rates were roughly as follows:
| src | dest | streams | rate (KB/s) |
|---|---|---|---|
| biet-1 | dcdoor12.usatlas.bnl.gov | 1 | 15000 |
| oliebol-01 | dcdoor10.usatlas.bnl.gov | 1 | 30000 |
| biet-1 | dcdoor07.usatlas.bnl.gov | 10 | 130000 |
| oliebol-01 | dcdoor13.usatlas.bnl.gov | 10 | 500 |
The command-lines used were:
lcg-cp -v -n 10 srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/rucio/mc12_8TeV/7d/5d/AOD.01230091._000076.pool.root.1 srm://dcsrm.usatlas.bnl.gov/pnfs/usatlas.bnl.gov//BNLT0D1/templon/test.data
(for a large data file on oliebol-01);
lcg-cp -v -n 1 srm://tbn18.nikhef.nl/dpm/nikhef.nl/home/atlas/atlasdatadisk/rucio/mc12_8TeV/22/3c/AOD.01230064._000016.pool.root.1 srm://dcsrm.usatlas.bnl.gov/pnfs/usatlas.bnl.gov//BNLT0D1/templon/test.data
(for a large file on biet-1).
By taking packet captures and plotting I/O rates (thanks to wireshark) we can compare the network flow in these different situations.
Bytes per 0.01s during transfer with 10 streams from oliebol-01
The worst case, with 10 streams from oliebol-01, resulted in TCP frame sizes that never exceeded 2962 bytes.
Bytes per 0.01s during transfer with a single stream from oliebol-01
The best rate from oliebol-01 was obtained by using a single stream. But this resulted in an enormous flood of packets. The ~7 second capture amounted to about 200MB of data (compared to about 4MB for the other captures). The largest frames were 33370 bytes.
Bytes per 0.01s during transfer with a single stream from biet-1
Comparing with 10 streams from biet-1, we see frame sizes as large as 64294, and more of them bunched together.