bu_cms_history/DCC2Testing

SiteMap (Historical BU CMS wiki main page)

16 Dec 2009 - BU (hazen)

Confirm yesterday's results. "-z 500 635" gives 0 BCNT errors. "-z 500 634" gives 0x15 errors. Last BX which is ok seems to be 2929 (0xb71). Default orbit gap settings for DCC2 are 3485-3494 (far away). Confirm that they are at the defaults: 0x24 = -da7-d9d (ok).

Let's try testing with fixed triggers.

 Orbit Length:       -o 3563 BX
 Trigger Delay:      -d 2915 BX
 Orbit Count:        -n 0 orbits
 Trigger Spacing:    -s 8 BX
 Triggers per orbit: -t 10 triggers
 Repeat period:      -r 10 orbits

Here we have 10 triggers spaced 8 BX spanning the critical end-of-orbit region. Overall rate ~ 10kHz. NO BCNT errors seen.

Even at a punishing rate (> 100kHz with 10 timesamples) where the HTR give steady OW, BZ, EE there are still *no* BCNT errors. Hmm....

 Orbit Length:       -o 3563 BX
 Trigger Delay:      -d 2600 BX
 Orbit Count:        -n 0 orbits
 Trigger Spacing:    -s 8 BX
 Triggers per orbit: -t 50 triggers
 Repeat period:      -r 3 orbits

Random 49kHz doesn't do it with 10ts. Try 49kHz with 20ts. Nope.

              RDY on: 00000000 6271d575                BSY on: 00000000 00000000  
              OFW on: 00000000 0c8ffb48                SYN on: 00000000 00000000  
              RUN on: 00000000 6f01ef65  
            L1A Trig: 00108be6          Events Built: 00108ba9  
        SLink Events: 00000000            VME Events: 00000080  
            Cal Trig: 00000000  
     L1 EvN Mismatch: 00000000       L1 BcN Mismatch: 00108bae       Bunch count err: 00000000  
     Rule 5 violated: 00000000  

Try 64kHz with 20ts. Aha!

              RDY on: 00000000 24cfeb7a                BSY on: 00000000 00000000  
              OFW on: 00000000 0dd47563                SYN on: 00000000 00000000  
              RUN on: 00000000 32a471ad  
            L1A Trig: 00070637          Events Built: 000705f4  
        SLink Events: 00000000            VME Events: 00000080  
            Cal Trig: 00000000  
     L1 EvN Mismatch: 00000000       L1 BcN Mismatch: 000705fa       Bunch count err: 0002db80  

Oh, but the HTR is seeing them too. Maybe a TTCvi problem? Try 64kHz with only 2TS. Still see errors.

              RDY on: 00000000 5697778d                BSY on: 00000000 00000000  
              OFW on: 00000000 00000000                SYN on: 00000000 00000000  
              RUN on: 00000000 56979755  
            L1A Trig: 0010efeb          Events Built: 0010efeb  
        SLink Events: 00000000            VME Events: 00000080  
            Cal Trig: 00000000  
     L1 EvN Mismatch: 00000000       L1 BcN Mismatch: 0010efeb       Bunch count err: 0005a15d  
     Rule 5 violated: 00000000  
 HTR   :  OW BZ EE RL LE LW OD CK BE
 HTR  0:  -- -- -- -- -- -- 10efeb -- 108 4d2a35 08 2c 4d8fd4 bfeda484 4cd5d0
 HTR  1:  -- -- -- -- -- -- 10efeb -- 108 -- 4cd240 bfeda44c bfeda448 -- --

Could be our Xilinx board too. Check with scope w/ delayed trigger. BC0 looks rock solid. Can see it move if we change the orbit length, so I think the scope setup is good. How to probe BC0 on a TTCrx?

Without any L1A (TTCvi input source set to 4) still see them at a rate of roughly 0x2000 per second (10kHz). This is the BC0 rate, so for some reason when the Xilinx board is set at 64kHz randoms we get a BCNT error every orbit, even though the orbit spacing looks fine on the scope?

15 Dec 2009 - BU (hazen)

Investigating BCNT errors.

 Firmware versions:
 xilinx:  0x3022  CRC: 0x00b1d268
    lrb:  0x010e  CRC: 0x006bfb4b
    vme:  0x0104  CRC: 0x00410ee6

Using "random_100kHz_rules", which is:

  cms1
 ~/src/TTS_ctrl > ./random_100kHz_rules
 Orbit Length:       -o 3563 BX
 Trigger Delay:      -d 100 BX
 Orbit Count:        -n 10 orbits
 Trigger Spacing:    -s 25 BX
 Triggers per orbit: -t 4 triggers
 Repeat period:      -r 0 orbits
 Random threshold:   -p 155 / 65535
 TTS latency         -l 0 BXn (0 sec)
 TTS sample mask     -m 0
 TTC cmd BCN         -w 1000
 Allow L1A in gap    -g 0
 Gap window          -z 250 500
 Trigger rule 1: no more than 1 triggers per 3 BX
 Trigger rule 2: no more than 2 triggers per 25 BX
 Trigger rule 3: no more than 3 triggers per 100 BX
 Trigger rule 4: no more than 4 triggers per 250 BX
 Rule enable mask: 0xf

Looking on the scope, I see triggers starting 6.2us (248 BX) after BC0, continuing to 12.5us (500 BX) before BC0. After ~ 10s we have.

              RDY on: 00000002 f458c59c                BSY on: 00000000 00000000  
              OFW on: 00000000 0b32a6c9                SYN on: 00000000 00000000  
              RUN on: 00000002 ff8b8b55  
            L1A Trig: 0007a0bd          Events Built: 0007a0bd  
        SLink Events: 00000000            VME Events: 00000080  
            Cal Trig: 00000000  
     L1 EvN Mismatch: 00000000       L1 BcN Mismatch: 0007a0bd       Bunch count err: 0000001e  
     Rule 5 violated: 00000000  

Let's try fiddling with the gap settings a bit. Change to "-z 750 750".

              RDY on: 00000000 56a9b5ae                BSY on: 00000000 00000000  
              OFW on: 00000000 02e01d1f                SYN on: 00000000 00000000  
              RUN on: 00000000 5989f295  
            L1A Trig: 00080e38          Events Built: 00080e38  
        SLink Events: 00000000            VME Events: 00000080  
            Cal Trig: 00000000  
     L1 EvN Mismatch: 00000000       L1 BcN Mismatch: 00080e38       Bunch count err: 00000000  
     Rule 5 violated: 00000000  

Aha! no BCNT errors! Let's try some other settings.

 -z 250 500   1e
 -z 750 750    0
 -z 500 500   4e
 -z 750 500   25
 -z 500 750    0

OK, so it's the end of the orbit which is the problem.

 -z 500 600   12
 -z 500 700    0
 -z 500 650    0
 -z 500 675    0
 -z 500 680    0
 -z 500 625   11
 -z 600 642    0
 -z 600 630   16
 -z 600 635    0
 -z 600 632   14
 -z 600 633   11
 -z 600 634   11
 -z 600 635    0

Aha! Sending triggers within 634 BX of BC0 seems to trigger a DCC2 bug.

06 Apr 2009 - BU (hazen)

Testing with new DCC firmware at high rates. Set all 6 HTRs to 16TS+4TP. This results in a 208 word payload from each HTR, and a total FED payload of 0x9da (2522) 32-bit words. Initialize for 100kHz random... TTS limits the rate to 61kHz.

Register dump dump_2009-04-06-1421.txt .

No errors reported. According to my calculation this is ~ 600MB/sec!

  2522 * 4 * 61e3 = 615e6

04 Apr 2009 - BU (hazen)

Testing in preparation for GR next week. DCC firmware is LRB version 101, DCC version 3000 rev 2009-03-03_1536.

AOK in Building 28 setup with xDAQ. 20 TS, 4 spigots active, took several 10k runs. Data is in /data/spool on cmsmoe6, log in .txt file in ~daqowner/hazen.

High-rate tests at BU

6 HTRs connected to DCC spigots 0-11.

Using HTR firmware 00D85. With 10TS / 0TP, AOK up to about 28kHz. After that all the counters light up.

Update to HTR firmware 00D86. Run at 100kHz - rate limit to ~ 86kHz. See lots of error counts:

 >dcc/stat
 0000 0000000f
: DCC_err
 TTS: 1000 RDY
   slink down
   slink full
   MB full
   MB empty
 0004 30008001
:  run mode
   rule 5 BSY
 0008 00000000
 (spy prescale)  0018 001c9090
 (sych ctrl)  0080 00000000
 (page) 00000632
 (mon words)
 0088 3a903b10
 (ttc)   0000000f
 (slink)
         HTR mis:  010e84f 010e84f 010e84f 010e84f 010e84f 010e84f 010e84f 010e84f 010e84f 010e84f 010e84f 010e84f
         HTR blk:  010e84f 010e84f 010e84f 010e84f 010e84f 010e84f 010e84f 010e84f 010e84f 010e84f 010e84f 010e84f
        HTR word:  895f626 895f52a 895f4ac 895f52a 895f626 895f4ac 895f52a 895f42e 895f4ac 895f52a 895f626 895f4ac
        HTR Cerr:  010e6c6 010e6c3 010e6c3 010e6c3 010e6c6 010e6c3 010e6c3 010e6c3 010e6c3 010e6c3 010e6c3 010e6c3
        HTR Uerr:  010e6c4 010e6c7 010e6ca 010e6c8 010e6c4 010e6c9 010e6c8 010e6cb 010e6c9 010e6c8 010e6c7 010e6ca
     L1A EvN mis:  010e6cf 010e6ce 010e6cf 010e6cf 010e6cf 010e6cf 010e6cf 010e6cf 010e6cf 010e6cf 010e6cf 010e6cf
     L1A BcN mis:  010e84f 010e84f 010e6f6 010d912 010e840 010e84f 010e84f 010e84f 010e84f 010e84f 010e556 010e4f2
     L1A OrN mis:  010e84f 010e84f 010e84f 010e84f 01085c8 010e84f 010e84f 010e84f 010e84f 010c73c 010e846 0042967
     L1A Skipped:
      L1A Padded:
     HTR CRC Err:  010e6c4 010e6c7 010e6ca 010e6c8 010e6c4 010e6c9 010e6c8 010e6cb 010e6c9 010e6c8 010e6c7 010e6ca
               RDY on: 00000000 5a8257df                BSY on: 00000000 00000000
               OFW on: 00000000 089c2cfe                SYN on: 00000000 00000000
               RUN on: 00000000 631ea9b5
             L1A Trig: 0010e857          Events Built: 0010e84f
         SLink Events: 00000000            VME Events: 00000080
             Cal Trig: 00000000
      L1 EvN Mismatch: 0010e6cf       L1 BcN Mismatch: 0010e84f       Bunch count err: 00000000
      Rule 5 violated: 00000000
 HTR   :  OW BZ EE RL LE LW OD CK BE
 HTR  0:  c2 c2 c2 c2 c2 c2 c2 c2 c2 688 253 d0 4d291e 4cd240 bff21f0c

There are almost as may Cerr and Uerr as blocks. Full register dump: dump_2009-03-04_1428.txt .

At 64kHz there are no errors reported.

Now, tweak the BcN offset (set register 0x20 to 0). No more EvN, BcN or OrN mis-match. With 10TS / 0TP at 100kHz, TTS rate limits to 74kHz and there are ''no errors''. Why does this work now?

Change register 0x20 to 1 so there are always BcN mismatch again. Still no other errors. The only other thing changed was that now we use this little script to setup the TTC system using DCCdiagnose.exe:

 #
 # DCC script to take data with new DCC (TTC setup only!)
 #
    ttc/trig 4
    sleep 0.1

    ttc/write 0x82 0xf000        # reset BGO fifos

    ttc/write 0x80 0xff64        # enable external orbit, disable triggers
    ttc/write 0x92 10            # inhibit 0 delay (250ns)
    ttc/write 0x94 10            # inhibit 0 duration (250ns)
    ttc/write BData0 0x00800000  # write one word (BCR, cmd=01) to fifo 0

    ttc/write 0x90 0xd           # enable BG0 channel 0

 # NOTE: do the following for ECR and BCR:
 #  ttc/cmd 2
 #  ttc/cmd 0x28

The procedure now is:

  1>  .../DCC2Tool.exe -x std_init.dcc2
  2>  DCCdiagnose.exe -x ttc_setup.dcc     (above script)
  1>  > ttc/cmd 2
  1>  > ttc/cmd 0x28
  1>  > ttc/trig 1                         (enable triggers)
  2>  > dcc/stat                           (show DCC2 counters)

With this procedure we see no errors at 100kHz (limited to 74kHz). Try increasing data volume to 10TS+5TP. More or less ok, but something is funny with HTR2. See dump_2009-03-04_1521.txt .

 >dcc/stat
 0000 0000000f
: DCC_err
 TTS: 1000 RDY
   slink down
   slink full
   MB full
   MB empty
 0004 30008001
:  run mode
   rule 5 BSY
 0008 0040fe00
 (spy prescale)  0018 001c9090
 (sych ctrl)  0080 00000000
 (page) 00000000
 (mon words)
 0088 28202820
 (ttc)   0000000f
 (slink)
         HTR mis:
         HTR blk:  013117e 013117e 013117e 013117e 013117e 013117e 013117e 013117e 013117e 013117e 013117e 013117e
        HTR word:  a214af0 a214af0 a214af0 a214af0 a214af0 a214af0 a214af0 a214af0 a214af0 a214af0 a214af0 a214af0
        HTR Cerr:
        HTR Uerr:
     L1A EvN mis:
     L1A BcN mis:
     L1A OrN mis:
     L1A Skipped:
      L1A Padded:
     HTR CRC Err:
               RDY on: 00000000 5c68fc3d                BSY on: 00000000 00000000
               OFW on: 00000000 19498278                SYN on: 00000000 00000000
               RUN on: 00000000 75b2a735
             L1A Trig: 0013117e          Events Built: 0013117e
         SLink Events: 00000000            VME Events: 0013117e
             Cal Trig: 00000000
      L1 EvN Mismatch: 00000000       L1 BcN Mismatch: 00000000       Bunch count err: 0000005a
      Rule 5 violated: 00000000
 HTR   :  OW BZ EE RL LE LW OD CK BE
 HTR  2:  40 40 40 40 40 40 40 40 40 bfe8e03c 4cd288 01 -- -- 4d291e

Now try 10TS+10TP. Much the same... basically OK. NOTE: due to HTR or htr.exe bug, this is really 10TS+4TP. See dump_2009-03-04_1525.txt .

 >dcc/stat
 0000 0000000f
: DCC_err
 TTS: 1000 RDY
   slink down
   slink full
   MB full
   MB empty
 0004 30008001
:  run mode
   rule 5 BSY
 0008 0040e100
 (spy prescale)  0018 001c9090
 (sych ctrl)  0080 00000000
 (page) 00000000
 (mon words)
 0088 11f011f0
 (ttc)   0000000f
 (slink)
         HTR mis:
         HTR blk:  01056e1 01056e1 01056e1 01056e1 01056e1 01056e1 01056e1 01056e1 01056e1 01056e1 01056e1 01056e1
        HTR word:  8ae2788 8ae2788 8ae2788 8ae2788 8ae2788 8ae2788 8ae2788 8ae2788 8ae2788 8ae2788 8ae2788 8ae2788
        HTR Cerr:
        HTR Uerr:
     L1A EvN mis:
     L1A BcN mis:
     L1A OrN mis:
     L1A Skipped:
      L1A Padded:
     HTR CRC Err:
               RDY on: 00000000 6ad95c05                BSY on: 00000000 00000000
               OFW on: 00000000 15a877f0                SYN on: 00000000 00000000
               RUN on: 00000000 8081f8cd
             L1A Trig: 001056e1          Events Built: 001056e1
         SLink Events: 00000000            VME Events: 001056e1
             Cal Trig: 00000000
      L1 EvN Mismatch: 00000000       L1 BcN Mismatch: 00000000       Bunch count err: 00000058
      Rule 5 violated: 00000000
 HTR   :  OW BZ EE RL LE LW OD CK BE
 HTR  2:  77 77 77 77 77 77 77 -- -- bffa222c 4cd288 01 -- -- 4d291e

Now we try again with setting NTS and NTP. This time we really have TS=16+TP=4. Then we get errors repeatedly. See dump_2009-03-04_1609.txt and status display below.

 >dcc/stat
 0000 b7f7ad90
: TTC_err DCC_err
 TTS: 0001             OFW
   Bit 4
   TTCrx sgl err
   TTCrx dbl err
   Bit 10
   Bit 11
   TTCrx OVFL
   DCC nRDY
   HTR 0 full
   HTR 1 full
   HTR 2 full
   HTR 4 full
   HTR 5 full
   HTR 6 full
   HTR 7 full
   HTR 8 full
   HTR 9 full
   HTR 10 full
   HTR 12 full
   HTR 13 full
   HTR babbling
 0004 30008001
:  run mode
   rule 5 BSY
 0008 00000000
 (spy prescale)  0018 001c9090
 (sych ctrl)  0080 00000000
 (page) 000009da
 (mon words)
 0088 0c401040
 (ttc)   0000000f
 (slink)
         HTR mis:  002ca 002c9 00185 002c9 002b2 002c9 002c9 002b4 002c9 002c9 002c9 002b4
         HTR blk:  002fc 002fc 002fc 002fc 002fc 002fc 002fc 002fc 002fc 002fc 002fc 002fc
        HTR word:  20eb8 20f84 1e7ac 20c54 1f3a0 209f0 209f0 20390 20060 209f0 1fd30 209f0
        HTR Cerr:  001e9 0014d 001e1 00219 0019f 00199 0023e 00239 00246 00183 00237 001fd
        HTR Uerr:  001f0 00164 00207 00228 001b8 001a9 001f6 001fd 00258 00192 00232 001f5
     L1A EvN mis:  002ca 002c9 00185 002c9 002b2 002c9 002c9 002b4 002c9 002c9 002c9 002b4
     L1A BcN mis:  002c8 002c4 0010e 002c7 00297 002c5 002c7 00296 002c4 002c5 002c7 00297
     L1A OrN mis:  002c4 002b8 00028 002c4 00206 002ae 002c4 00289 002ae 002ae 002c4 00289
     L1A Skipped:
      L1A Padded:
     HTR CRC Err:  001f0 00164 00207 00228 001b8 001a9 001f6 001fd 00258 00192 00232 001f4
               RDY on: 00000000 321bccc7                BSY on: 00000000 00000000
               OFW on: 00000000 1645c8c6                SYN on: 00000000 00000000
               RUN on: 00000000 4861a65d
             L1A Trig: 0000033c          Events Built: 000002fc
         SLink Events: 00000000            VME Events: 00000080
             Cal Trig: 00000000
      L1 EvN Mismatch: 000002c9       L1 BcN Mismatch: 000002c7       Bunch count err: 00000000
      Rule 5 violated: 00000000
 HTR   :  OW BZ EE RL LE LW OD CK BE
 HTR  0:  54 54 54 54 54 54 54 94 94 24 1990 4d8fd4 b7f78ab0 b7f7aae0 b7f7ad90
 HTR  1:  06 06 06 06 06 06 06 01 01 bfe3ffc0 688 253 d0 4d291e 4cd240
 HTR  2:  0a 0a 0a 0a 0a 0a 0a 01 01 4d2a35 08 1c 4d8fd4 bfe3fff4 4cd5d0
 HTR  3:  11 11 11 11 11 11 11 11 11 -- bfe3faec 4cd288 01 -- --
 HTR  4:  0c 0c 0c 0c 0c 0c 0c 07 07 bfe3fb50 bfe3fb1c 4d2a35 08 1c 4d8fd4
 HTR  5:  02 02 02 02 02 02 02 02 02 01 4d91c0 bfe3f420 4ce5ce b7f76a80 4d91c0
 HTR  7:  0e 0a 0e 0e 0a 0e 0e 0c 0c bfe3fb80 4ce5ce -- 4d8fd4 bfe3ffc0 8049ddf
 HTR  8:  01 01 01 01 01 01 01 01 01 4d8fd4 bfe3fff4 4cd52a 01 b7f767f0 bfe3fbe0
 HTR  9:  02 02 02 02 02 02 02 -- -- 01 b7f762d8 2b30858 24 -- bfe3fc0c
 HTR 10:  01 01 01 01 01 01 01 -- -- 2b23c71 -- 2b23c71 04 04 01
 HTR 11:  03 03 03 03 03 03 03 02 02 ffffffe8 bfe3fcc8 2a6ba21 2b30820 2a699a4 706318

S-Link output validation

Take a few short runs at low rate, capture S-Link output using fedkit_dump_receiver. Convert back to binary with undump_SLINK.cc and check... AOK.

Don't have the tools to do a better job... best check this at CERN.

New HTR firmware 00E86

Programming HTR firmware 00E86 into all HTRs for a quick test. Seems OK.