Accelerator Update for February 9, 2009 - February 11, 2009
Monday Feb 9
The day shift began with Operations monitoring the Tevatron (TeV) store 6787 (154.75E30) and the Recycler stash (157.5E10), with the Antiproton Source stacking (29.0E10), with the Linac, Booster, and Main Injector (MI) operating normally, with the MiniBooNE and NuMI experiments taking beam, and with the Collider Detector at Fermilab (CDF) and D-Zero (D0) experiments using the colliding beams.
Pbar experts began a 10-minute target scan study around 9:55 AM.
A MI expert began a longitudinal bunch measurement study at 10:19 AM. Some problems delayed this study for about 45 minutes.
At 2:50 PM, the P1 transfer line beam permit tripped off when the output of a power supply (I:V714) went too high. Everything reset.
At 3:10 PM, the TeV sector C4 air compressor tripped off again. Operators managed to restart it before the store was lost. Experts were searching for the cause of the trips. They added a spare compressor.
Operations reported at 4 PM that the Recycler had stashed antiprotons eight times during the day shift.
Operations terminated store 6787 at 4:05 PM.
Operations prepared D0 for a one-hour access at 4:24 PM.
Operations put the TeV through a wet squeeze at 5:57 PM.
Operations put the TeV into shot setup at 6:50 PM.
At 7:41 PM, Operations reported a LCW leak.
Operators checked the service buildings; around 7:53 PM they discovered an Accumulator power supply (A:LQ) had a leaking hose. Operations called in an expert.
Operations established store 6789 at 8:15 PM with an initial luminosity of 320.88E30.
Operations reported at 9:47 PM that the A:LQ power supply was repaired.
Pbar resumed stacking at 10 PM.
At 11:23 PM, A:LQ failed to return to its nominal position. Operators recycled the busses.
Pbar resumed stacking at 11:32 PM.
Operations reported at midnight that the Recycler had stashed four times during the evening shift.
Tuesday Feb 10
The midnight shift began with Operations monitoring store 6789 (172.18E30) and the stash (50.9E10), with Pbar stacking (18.1E10), and with NuMI and MiniBooNE taking beam.
At 4:37 AM, A:LQ tripped off due to another LCW leak. Operations called in an expert.
Operations put Pbar into standby mode at 4:39 AM.
A Pbar power supply expert reported at 6:50 AM that he had A:LQ back online.
Pbar resumed stacking at 7:09 AM.
Operations reported at 8:04 AM that the Recycler had stashed four times during the midnight shift.
At 8:57 AM, Booster RF stations BRF6, 7, and 13 all tripped off. They were reset.
A MI expert began a 90-minute study at 9:49 AM.
At 10:58 AM, TEL1 tripped off on a vacuum indication. It reset.
Mechanical group experts reported at 11:18 AM that they had replaced the overload heaters for the TeV sector C4 air compressors.
At 11:49 AM, BRF7 tripped off. An RF expert investigated and said something “funny” was happening in the East Gallery. Operations lowered Booster intensities.
A Booster expert began tuning around 12:05 PM.
At 12:31 PM, BRF5 began tripping off due to sparking.
Operations terminated store 6789 at 2:39 PM.
Operations prepared the TeV for a proton only study at 2:59 PM.
Operations reported at 3.54 PM that the Recycler had stashed seven times during the day shift.
At 4:57 PM, BRF5 tripped off two times. It reset.
Operations put the TeV through a wet squeeze at 5:45 PM.
Operations put the TeV into shot setup at 6:28 PM.
At 6:37 PM, the P2 transfer line beam permit tripped off due to a power supply (HTF13) going out of tolerance. Everything reset.
At 6:44 PM, the TeV sector C4 air compressor tripped off. Operators reset it.
Operations established store 6792 at 7:44 PM with an initial luminosity or 314.37E30.
At 9:28 PM, the P1 transfer line beam permit tripped off due to V714 going out of tolerance. Everything reset.
At 10:42 PM, Operations reported that that a Pbar Debuncher kicker (D:EKIK) had stopped firing. Operators investigated and found its ramp was missing.
Around 11 PM, Operations reported the transformer that powered the Cross Gallery, MCR, the Computer Room, and the MAC room had failed – no power to these areas. The TeV quenched at six houses: A4, B1, C4, D1, B3, and C2. Store 6792 was lost. The stash was lost. Pbar managed to hold on to a small stack.
At 11:18 PM, smoke was observed in the TeV sector B3 area. The Fire Department and operators investigated. Later, it was discovered that the dump switches for the sectors B3 and C2 service buildings had failed.
At 11:45 PM, the UPS in the Computer Room started to fail. The MCR lost controls and readbacks for the complex.
Wednesday Feb 11
The midnight shift began with the complex down due to a power failure. Investigation and recovery efforts began. The electronic logbook was down. The following is a report on machine statuses near the end of the shift, ~8 AM.
The Plans for Wednesday
Power was restored to the Cross Gallery and other areas by a large generator. There is a spare transformer in the Proton experimental area, but it must be checked out before it’s moved. The plans for today are to investigate, recover what can be recovered, and repair damage.
Comments and Suggestions
|last modified 02/11/2009 email Fermilab|
|Security, Privacy, Legal|