In your everyday work with PRDF data, you'll find that a lot can be accomplished with the ready-made tools. Almost every week, someone comes up with a new creative way to use them in combination with perl, grep, wc, and so on to get some answers about the data structure without actually programming anything.

The most versatile tools we have are

• the dlist utility, which lets you list the packets contained in a given event;

• the ddump utility, which lets you dump and look at the contents of a packet in various ways;

• the dpipe utility, which copies events from one destination to another with a lot of flexibility; you can also use this to sift through a data stream in a simple way.

All utilities have a -h switch to provide some online help.

Let's start with dlist. It lists the packets it finds in the Event, and calls the identify() function on each of them, so you'll see a short identification message for each packet found.

dlist can open any input stream, and its default is to open an ET pool. With -f, you tell that name you give is actually a filename of a data file, and dlist will open the file. If you give -T, you tell dlist to open a test input stream, Events made up on the fly for test purposes. Their packet contents are predictable,

$ dlist -T
Packet   1001    26  0 (Unformatted)  30006 (ID4EVT)
Packet   1002    16  0 (Unformatted)  30005 (ID2EVT)
Packet   1003    10  0 (Unformatted)  30006 (ID4EVT)

Here's an example that reads from a file. Only the first few lines of output are shown:

$ dlist -f /common/buffer2/eventdata/EVENTDATAxxx_P01-000002* | more
Packet  14001    52  0 (Unformatted)   714 (IDGL1)
Packet  14008    12  0 (Unformatted)   814 (IDGL1P)
Packet  14002     9  0 (Unformatted)   701 (IDBBC_LL1)
Packet  11033   537  0 (Unformatted)  1011 (IDMUTC_FPGA)
Packet  11035   537  0 (Unformatted)  1011 (IDMUTC_FPGA)
Packet  11036   537  0 (Unformatted)  1011 (IDMUTC_FPGA)
Packet  11037   537  0 (Unformatted)  1011 (IDMUTC_FPGA)
Packet  11038   537  0 (Unformatted)  1011 (IDMUTC_FPGA)
Packet  11039   537  0 (Unformatted)  1011 (IDMUTC_FPGA)
Packet  11040   537  0 (Unformatted)  1011 (IDMUTC_FPGA)
Packet  11025   537  0 (Unformatted)  1011 (IDMUTC_FPGA)
Packet  11026   537  0 (Unformatted)  1011 (IDMUTC_FPGA)
Packet  11027   537  0 (Unformatted)  1011 (IDMUTC_FPGA)
Packet  11028   537  0 (Unformatted)  1011 (IDMUTC_FPGA)
Packet  11029   537  0 (Unformatted)  1011 (IDMUTC_FPGA)
Packet  11030   537  0 (Unformatted)  1011 (IDMUTC_FPGA)
Packet  11031   537  0 (Unformatted)  1011 (IDMUTC_FPGA)
Packet  11032   537  0 (Unformatted)  1011 (IDMUTC_FPGA)
Packet  11017   537  0 (Unformatted)  1011 (IDMUTC_FPGA)
<more lines deleted...>

dlist reads, by default, the first event it considers a data event. It will silently skip special events such as Begin-run, End-run, and scaler events. You have to ask for them explicitly with the -t (type) switch.

There are numerous switches that modify the behavior of dlist. You can ask for a given event number with the -e switch. (Note that this is the event number in the event header, not its position in the file).

With the -c n switch, You can also ask for the event in the nth position (dlist simply skips n-1 events).

You can ask for a number of events listed in a row with the -n <numberofevents> switch. The default is just one event. If you specify 0, you will read all events from the data stream (e.g., if you read from a file, read all the events from the file.)

You can combine -e or -c with -n, for example, start with event 15 and list the next 5:

dlist -f -e 15 -n 5 <filename>

or start with the event in the 22nd position in the file and list 5:

dlist -f -c 22 -n 5 <filename>

Another useful switch is -i, which causes each event listed to identify itself.

$ dlist -T -i
 -- Event 1 Run:  1331 length:    68 frames:   1 type:  1 (Data Event) 
Packet   1001    26  0 (Unformatted)  30006 (ID4EVT)
Packet   1002    16  0 (Unformatted)  30005 (ID2EVT)
Packet   1003    10  0 (Unformatted)  30006 (ID4EVT)

There is also the -I switch, which will print very detailed information about the packets. Usually something used by the experts.

So for example, let's say that we wish to find out if all packets we expect in a given event are indeed present or not. We list the packets and pipe the output into wc:

$ dlist -f /common/buffer2/eventdata/EVENTDATAxxx_P01-0000020578-0000.PRDFF | wc -l
    166

Ok, 166 packets. Does event 15 have 166 packets, too?

$ dlist -f -e 15 /common/buffer2/eventdata/EVENTDATAxxx_P01-0000020578-0000.PRDFF | wc -l
    166

While dlist lists the packets in an event by calling the packet's identify() function, ddump dumps the contents of one or all packets of a given event according to the user's wishes. Again, ddump -h will print some help text.

As much as possible, ddump supports the same event selection switches as dlist ( -t, -e, -c, -n), and also the same notation to identify the type of input stream (default Et pool, -f denotes file, -T test stream). It also has an -i switch to identify the event.

As described in the previous chapter, the packets support generic dumps and regular dumps. You request a generic dump with the -g switch. In that case, you can pick the style (decimal, hexidecimal, or octal) with -d for decimal, -o for octal, and hex as the default.

You can specify a particular packet id to be dumped with the -p <id> switch.

Here is an example where we dump packet 1003 with a generic dump from a test stream, requesting decimal numbers:

$ ddump -T  -g -d -p 1003
Packet   1003    10  0 (Unformatted)  30006 (ID4EVT)
    0 |         -12         -7       -854      14943 

and with in-depth packet information (with the -I switch):

$ ddump -T -g -d -p 1003 -I
*** Packet id         1003
    Hdr version       1
    Hdr length        6
    Packet Length     10
    Packet Status     0
    Debug Length      0
    Error Length      0
    Structure         0 (Unformatted)  
    Descriptor Words: 1
    Endianism:        1 (Little Endian) 
    Padding:          0
Packet   1003    10  0 (Unformatted)  30006 (ID4EVT)
    0 |         -12         -7       -854      14943 

A lot of debugging takes place with the ddump utility. Here is an example you you quickly look for a few beam clock entries in the Emcal packet 8100. If you ask for a non-generic dump, the output is

> ddump -p 8100 -f /common/buffer2/eventdata/EVENTDATAxxx_P01-0000023434-0000.PRDFF 
Packet   8100    21  0 (Unformatted)  1108 (IDEMC_FPGA0SUP)
  Detector id:               4
  Event number:              1
  Module address:            1
  Flag Word:              8000
  Beam Clock Counter:       c6
  AMU cell 0:               1d
  AMU cell 1:               19
  AMU cell 2:               1c
  Long. Parity word       7cca
  Sumary word             6220
 channel      time high-post low-post high-pre low-pre
 -----------------------------------------------------
    0 |         0        0        0        0        0 
    1 |         0        0        0        0        0 
<stuff deleted>

If you run this for a few events with the -n switch, and grep for "Beam", you see the values for a few consecutive events, and you can see that they are somehat reasonable:

$ ddump -p 8100 -f -n 10 /common/buffer2/eventdata/EVENTDATAxxx_P01-0000023434-0000.PRDFF | grep Beam
  Beam Clock Counter:       c6
  Beam Clock Counter:       ba
  Beam Clock Counter:       ca
  Beam Clock Counter:        2
  Beam Clock Counter:       8f
  Beam Clock Counter:       ee
  Beam Clock Counter:       df
  Beam Clock Counter:       c2
  Beam Clock Counter:       b2
  Beam Clock Counter:       72

You can further process that with awk if you like, and strip off the "Beam Clock Counter:" text:

$ ddump -p 8100 -f -n 10 /common/buffer2/eventdata/EVENTDATAxxx_P01-0000023434-0000.PRDFF | grep Beam | awk -F: '{print $2}'
       c6
       ba
       ca
        2
       8f
       ee
       df
       c2
       b2
       72

This is a very simple example how to get some answers quick by just running ddump.

The last utility we want to mention here is dpipe, which has also many applications beyond its primary purpose, which is to move (gleaned from the UNIX "pipe" concept) events from an input stream to an output stream. It has lots of switches somewhat consistent with dlist and ddump (the -c, -e, -n switches for event selection, and -i for event identification). Its general form is

dpipe [options] <input stream> <output stream>

Where the type of streams both default to Et Pool.

The -s (source) and -d (destination) switches tell dpipe what kind of streams it is dealing with. Input can be any of file, et pool, or Test stream, denoted by -s file, -s Test, or -s etpool (which is the default). The output can be -d f (file) or -d e (again the default), or -d n (nowhere), which comes in handy at times. You cannot write to a test stream.

The "file", "Test", "etpool", and "nowhere" keywords can be shortened to the first letter.

Here is a command that acts as a simple data logger. It reads 100 events from the Et pool "/tmp/Partition01" and writes it to a file:

$ dpipe -d f -n 100 /tmp/Partition01 100Events.prdf

Add the -i switch, and you see which events go to the file.

Also, if you want to write a compressed prdf file, add the -z switch. That can be slow, but if you don't care, the file sizes can be dramatically smaller.

There is also a wait switch -w <seconds> so you can throttle the speed.

The dpipe command has lots of applications that can make your life easier. Let's say that the event with the sequence number 62 in a data file is somehow funny, and you want to look at this event over and over again, while you make changes to your software. Since the data streams are just that, streams, they are not direct access files. You have to read the events as they come, and cannot "jump" to the desired event fast.

With dpipe, you can extract that event to a file which then holds only this event, and you are much faster reading the event over and over again:

$ dpipe -s f -d f  -e 62 -n 1 -i  /common/buffer2/eventdata/EVENTDATAxxx_P01-0000023434-0000.PRDFF my_event.prdf
 -- Event 62 Run: 23434 length: 163317 frames:   1 type:  1 (Data Event) 
$

If you interpret the switches, you read "source is a file, destination is a file, start with event with sequence number 62, and go for one event, and please identify the event". We added the -i switch so we can see that we pick the right one. Now our file contains just that one event:

$ dlist -i -f my_event.prdf | more
 -- Event 62 Run: 23434 length: 163317 frames:   1 type:  1 (Data Event) 
Packet  14001    52  0 (Unformatted)   714 (IDGL1)
Packet  14008    12  0 (Unformatted)   814 (IDGL1P)
Packet  14002     9  0 (Unformatted)   701 (IDBBC_LL1)
Packet  11035    25  0 (Unformatted)  1111 (IDMUTC_FPGA0SUP)
< .. lines deleted ..> 

You may wonder what the "nowhere" destination is for. It's usually used in conunction with -i, and used to list all events in a stream:

$ dpipe -s f -d n -i /common/buffer2/eventdata/EVENTDATAxxx_P01-0000023434-0000.PRDFF 
 -- Event -1 Run: 23434 length:   192 frames:   1 type:  9 (Begin Run Event) 
 -- Event -2 Run: 23434 length:   104 frames:   1 type: 14 (Scaler Event) 
 -- Event 1 Run: 23434 length: 121203 frames:   1 type:  1 (Data Event) 
 -- Event 2 Run: 23434 length: 49135 frames:   1 type:  1 (Data Event) 
 -- Event 5 Run: 23434 length: 60037 frames:   1 type:  1 (Data Event) 
 -- Event 7 Run: 23434 length: 71433 frames:   1 type:  1 (Data Event) 
 -- Event 8 Run: 23434 length: 165019 frames:   1 type:  1 (Data Event) 
 -- Event 10 Run: 23434 length: 48855 frames:   1 type:  1 (Data Event) 
 -- Event 3 Run: 23434 length: 165300 frames:   1 type:  1 (Data Event) 
 -- Event 4 Run: 23434 length: 47004 frames:   1 type:  1 (Data Event) 
 -- Event 6 Run: 23434 length: 47621 frames:   1 type:  1 (Data Event) 
 -- Event 9 Run: 23434 length: 48284 frames:   1 type:  1 (Data Event) 
 -- Event 11 Run: 23434 length: 46455 frames:   1 type:  1 (Data Event) 
 -- Event 12 Run: 23434 length: 166335 frames:   1 type:  1 (Data Event) 
 -- Event 13 Run: 23434 length: 48255 frames:   1 type:  1 (Data Event) 
 -- Event 15 Run: 23434 length: 48318 frames:   1 type:  1 (Data Event) 
 -- Event 16 Run: 23434 length: 45424 frames:   1 type:  1 (Data Event) 
 -- Event 18 Run: 23434 length: 48591 frames:   1 type:  1 (Data Event) 
 -- Event 20 Run: 23434 length: 169283 frames:   1 type:  1 (Data Event) 
 -- Event 22 Run: 23434 length: 45744 frames:   1 type:  1 (Data Event) 
 -- Event 24 Run: 23434 length: 62511 frames:   1 type:  1 (Data Event) 
 -- Event 25 Run: 23434 length: 166924 frames:   1 type:  1 (Data Event) 
 -- Event 14 Run: 23434 length: 45513 frames:   1 type:  1 (Data Event) 
 -- Event 17 Run: 23434 length: 67210 frames:   1 type:  1 (Data Event) 
 -- Event 19 Run: 23434 length: 47113 frames:   1 type:  1 (Data Event) 
< .. lines deleted ..> 

One of the greatest abuses of this command came last year, when the then-analysis macros had a very unsophisticated event loop that would require the number of events as input, so people would run

$ dpipe -s f -d n -i /common/buffer2/eventdata/EVENTDATAxxx_P01-0000023434-0000.PRDFF | wc -l
      78

just get one number. Doing that routinely is very wasteful - last year's data were usually written in big-endian format, and were hosted on NFS-mounted disks, so in order to get one number out, you would read the whole file through nfs, and byte-swap the whole file. With pmonitor, or a decent macro event loop, there is no need to know that number up front, you can just run through all events in a file.

Still, running what I call the "dpipe dry run" can give you some useful answers. Let's say that you experience an unexplicable segmentation fault running through that file in our event 62. Although care has been taken that corrupt data structures in a file don't cause the event library to segfault, there's always the possibility that this file is corrupt in a new creative way which defeats the safeguards. As the first check, run the file through the dpipe dry run. If that succeeds, you know that the outer structure of the file is formally ok. As the next step, run all events through a (non-generic) ddump -n 0 (all events). This gives an excessive amount of output, and since you see the segfault in event 62, you may want to skip a few events in the beginning using the -e or -c switch. Be sure not to request a generic dump with -g, which would be a much less comprehensive test. If that succeeds as well, the chances that a corrupt data structure causes the problem are very, very small. (They are not zero, we have seen a bug with the TOF decoding that passed that test).