SLD: S-Link Driver for Linux.

Current version is 1.0.3 (july 14^th, 2000):

Background

At Nikhef investigations are in progress for the Atlas project (check their website for information). One of the topics is S-Link, which is to be used to pump data from one system (perhaps a detector) to a fast computer (to process the data). One possibility to connect an S-Link LDC (Link Destination Card) to a generic PC is to use the CERN SSPCI interface. This driver works only with that interface. The assignment indicated the following requirements (among others):

high-speed, to be reached by:

zero-copy architecture: put data directly into the address space of the user space program
use DMA to put the data in memory

handle variable-size events with a maximum eventsize of 2048 bytes or 512 datawords (an event is a S-Link controlword, followed by multiple datawords and another controlword)
interrupt driven
differentiate between control- and data-words

From a few conversations with people, it was clear that this driver would not be used in a generic-use Linux environment, which allowed to take decisions to be made different from the generic-use case. One of the first decisions in this manner was to avoid system calls, since they are expensive with regard to CPU power.
lmbench showed that a simple system call costs 0.5814 microseconds (on a 700Mhz PentiumIII), so at the expected datarate of 60MB/s with 1.5KB per event and 3 system calls per event (2 controlwords and 1 block datawords) this would mean 40960 events, or a whopping 122880 system calls per second. This would require about 71.4 milliseconds already for the simple system call.

The SSPCI interface uses the AMCC s5933 Matchmaker chip. The documentation is available from the AMCC website. AMCC only has the errata for the QE version available. Here are also the QB and QC errata and a letter about the transistion to QE, which are not available from the AMCC website (duh!).

Features

fully interrupt driven during continuous use,
a timer task to handle out-of-buffer situations and the last controlword,
data interface without system call overhead as long as data is available, however, this does require a non-generic user program,
supports multi-DMA operation (when an event is larger than the configured buffer size),
compile time choice to use a bottom-half or not (if you don't use it, you'll get a few nasty surprises, though!)
the select system call is supported,
multiple interface support (currently 4, but compile time changeable),
works with regular, unpatched Linux kernels (no need for a bigphysarea patch),
it is a `dumb' driver: the user space application needs to evaluate, process and implement error checking and higher-level protocols.

Implementation and use

Before starting the development of the driver, so things were written down in some kind of `design' document. It is available in the original Star Writer format, but also as HTML and Word '95.

The driver has been developed as a kernel module and uses a dynamic major number. As long as you have only one driver using dynamic majors, or always load drivers in the same order, the driver will get the same major number.

There are two configurable items on this driver and those are the buffer size and the number of buffers that are used. The lower limit on these two is 128 bytes and 10, respectively (just to make sure that the driver will function somewhat) and the upper limit is PAGE_SIZE for the buffer size (which is 4KB on Intel systems) and unlimited for the number of buffers (in the driver anyway!).

Internally the driver maintains a circular queue with databuffers and it fills the buffers at the head of the queue. In user space some program takes a buffer from the tail of the queue, processes the data in it and frees it again. Some code from the driver needs to be compiled with the program to access this interface. The basic operation of the user program looks like this (for clarity error checking of select() was removed):

infoblock_t *blockPtr;
do
{
    blockPtr = getFullInfoBlock( privDataPtr );
    if ( blockPtr == NULL )
    {
        select( sldFd );
        blockPtr = getFullInfoBlock( privDataPtr );
    }
    handleBlock( blockPtr );
    freeInfoBlock( privDataPtr );
}
while ( TRUE );

Sldtest offers the options to display S-Link data and control words, driver informational messages, statistics and various other things. It uses the driver the way any other application would use the driver, so it's a good example when you're developing your own application. The other program is sldstat, a program like iostat and vmstat. It displays one line of information at a time with the most basic statistical information from the driver. A program in this style only has the ioctl interface to perform any useful work. This allows for control of the return lines and getting the status of the S-Link and statistical information.