bu_cms_history/SN_003

SiteMap (Historical BU CMS wiki main page)

S/N 003

2007-09-18 (hazen, CERN)

DEAD MB?! SHIP TO BOSTON.

Swapped out because of alleged S-Link problems. Now has "Deep FW" problems too. Power up. All DCC LEDs are on. Motherboard LEDs (at top) are off!

Try checking the PCI devices with DCCrepair:

 ./DCCrepair.exe sbs:0 8 -c -p
 ...
 Detailed scan of card in slot 8
 PCI configuration information
 VSI0 base of 0x28000000 will be used to access PCI configuration space
 Bus Dev Alias PCI ID     Device ID  CSR      BAR0     BAR1     BAR2     BAR3     BAR4     BAR5
   0   0  br3 28000000          ??  1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff
   0   1 log3 1000ffff          ??  1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff
   0   2  uv2 1000ffff          ??  1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff
   0   3   bc 1000ffff          ??  1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff
   0   4   lc 1000ffff          ??  1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff
   1   0  br2 1000ffff          ??  1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff
   1   1 log2 1000ffff          ??  1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff
   1   2 mip3 1000ffff          ??  1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff
   1   3 mip4 1000ffff          ??  1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff
   1   4 mip5 1000ffff          ??  1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff
   2   0   -- 1000ffff          ??  1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff
   2   1 log1 1000ffff          ??  1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff
   2   2 mip0 1000ffff          ??  1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff
   2   3 mip1 1000ffff          ??  1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff
   2   4 mip2 1000ffff          ??  1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff 1000ffff

Ok, this guy is really sick. (See the DCCrepair documentation for an example of a healthy output). Somehow even the first PCI device does not appear on config space. This display should look reasonable for even an empty motherboard, so tentatively we suspect the motherboard.

Next step is to try JTAG on the motherboard. (See the instructions )

Hook up the Xilinx, cable, get all the files from here . The three Xilinx CPLDs (2 XC9536 and XC95108) all verify with the corresponding firmware files.

Hook up the Altera cable, program local control. Still no go... initialization fails completely with all tools.

Take off all LRBs and S-Link plate as a last resort. No improvement. Note that MB LEDs come from local control chip.

2008-01-07 (hazen, BU)

Arrived with LRB3 partly unplugged, no jumper board. Re-seat all boards, power up. LEDs are dead (no MB leds, all DCC leds on).

Remove all daughterboards from MB. Same symptoms. Check DC power: 2.5V, 3.3V and 5V all OK.

Check clocks: 64Mhz, 32MHz, 16MHz all ok.

Program LC with Altera JTAG cable. Programming is ok. Connect with Xilinx adapter and use impact to verify the 3 CPLDs. All verify ok.

Try to use DCCdiagnose 'pci/lc_check', get an error:

  >pci/lc_check
  ERROR - LCProg::writeFlash() -> ERROR: writing to offset 0x0
  ERROR - CheckAccesstoFlash: lc_programmer.cc:79
 LCProg::writeFlash() -> Item "lc_mem" not found.
      (HAL::AddressTable::checkItem)
  Flash check FAILED

This seems like a HAL address table problem! Or maybe not? Here's the DCCdiagnose 'map' output for a good motherboard:

 ******************************************************************************************************
  *                   VME Address Table : DDC Address Table
  *                         item       AdrSpace  map AM  width   address      mask  r  w  description
  ******************************************************************************************************
                         lc_conf                     09      4  00002000  ffffffff  1  1  Motherboard Local Control
                        br2_conf                     09      4  00010000  ffffffff  1  1  PCI-2 bridge
                          lc_mem                     09      4  00030000  ffffffff  1  1  BAR0
                        br3_conf                     09      4  00000000  ffffffff  1  1  PCI-3 bridge
                              u2                     09      4  00000000  ffffffff  1  1  U2 PCI Configuration space base address
                           lc_io                     09      4  00040000  ffffffff  1  1  BAR1
                         u2_vrai                     39      4  0000d000  ffffffff  1  1  U2 VRAI
                     u2_vrai_top                     39      4  0000e000  ffffffff  1  1  U2 VRAI
  ******************************************************************************************************

And here is the sick one. Notice that on the bad guy lc_fmem appears but not lc_mem....

  *************************************************************************************************
  *                   VME Address Table : DDC Address Table
  *                         item       AdrSpace  map AM  width   address      mask  r  w  description
  ******************************************************************************************************
                         lc_conf                     09      4  00002000  ffffffff  1  1  Motherboard Local Control
                        br2_conf                     09      4  00010000  ffffffff  1  1  PCI-2 bridge
                        br3_conf                     09      4  00000000  ffffffff  1  1  PCI-3 bridge
                              u2                     09      4  00000000  ffffffff  1  1  U2 PCI Configuration space base address
                           lc_io                     09      4  00040000  ffffffff  1  1  BAR1
                         lc_fmem                     09      4  00030000  ffffffff  1  1  BAR0
                         u2_vrai                     39      4  00009000  ffffffff  1  1  U2 VRAI
                     u2_vrai_top                     39      4  0000a000  ffffffff  1  1  U2 VRAI
  ******************************************************************************************************

This one is for Mr Wu!