Accelerator Update for January 25, 2008 - January 28, 2008

Friday January 25

The day shift began with the Tevatron (TeV) down due to a power supply problem, with Operations monitoring the Recycler stash (201.5E10), with the Antiproton Source stacking (32.7E10), with the Main Injector (MI), Linac, and Booster operating normally, with the Collider Detector at Fermilab (CDF) and D-Zero (D0) waiting for the next store, and with the NuMI and SciBooNE / MiniBooNE (BNB) experiments taking beam.

Operations reported at 8:18 AM that experts were working on the TeV corrector magnet power supply (T:HA24) and on a Linac RF station (LRF2).

Operations reported at 8:51 AM that experts from the water group were working on air compressor at TeV sectors B2, A3, and A4.

Operations reported at 9:53 AM that the TeV T:HA24 work was completed; they found and repaired a bad butt splice on a cable.

At 9:59 AM, a Booster RF station (BRF3) tripped off and caused the East Anode Power Supply (EAPS) to trip. An expert investigated.

Operations put the TeV through a dry squeeze at 10:29 AM.

Operations reported at 11:32 AM that the Pelletron had been turned off and experts were accessing MI-31.

At 1:20 PM, Operations received a TeV kicker (F17K2) temperature alarm. A TeV expert investigated and said that they would need an access to fill the kicker.

Operations prepared CDF and D0 for an access that began around 1:56 PM.

Operations reported at 2:25 PM that LRF2 was repaired.

Recycler experts began a study period at 2:46 PM.

D0 and CDF came out of access by 2:56 PM.

At 3:30 PM Operations established beam to BNB.

At 3:44 PM, a Linac Klystron RF station (KRF4) tripped off. It reset.

At 3:50 PM, Operations reported that beam was not making it to NuMI or to the Pbar target due to a possible RF problem in MI.

At 4:20 PM, Operations reported that the stash of about 170E10 was lost. Operators investigated and discovered that the stash was lost at the same time as the MI ramp dump, which was due to a quadrupole QCL bus ground fault (the QCL measures the difference between the focusing quadrupole and defocusing quadrupole current buses). Also, experts had rebooted the Low Level RF system at the same time. Later it was discovered that a LCW leak on a quadrupole magnet (Q631) had caused the ground fault.

At 5:22 PM, the MI safety system dropped. Operations called in the interlock experts. Later it was discovered that the safety system had dropped and reset when the stash was lost and this might have been the root cause for the loss.

At 5:45 PM, a Pbar Debuncher RF station (DRF1-3) tripped off. It reset.

At 6:39 PM, the Pelletron LCW cooling temperatures came into alarm. Operators contacted a Pelletron expert and then prepared MI-31 for an access. An Operator and a Recycler expert bled off air in the system, but this did not resolve the temperature problem.

Operation put the TeV through a dry squeeze at 7:01 PM.

At 7:18 PM, during the dry squeeze, a TeV corrector magnet power supply (T:HB15) tripped off.

Interlock experts reported at 7:57 PM that they found and replaced a pair of bad relays at MI-8. Unfortunately, MI was still off due to problems with Q631.

Operations raised the ground fault current limit and established a 120 GeV ramp in MI at 8:27 PM.

Operations resumed sending beam to BNB at 8:41 PM.

At 9:08 PM, Operations was about to send beam to Pbar and NuMI when the Booster beam permit dropped. Operators investigated and discovered a problem with the Booster RF system. Operators investigated further and reported that all the ramp information had been wiped from the cards. Operators restored the necessary information.

Operations had the power supply replaced for T:HB15 by 9:10 PM.

Operations injected beam into the TeV at 9:23 PM and a TeV expert began a study period.

Operations reported at 10:02 PM that MI was slipstacking.

At 10:59 PM, KRF4 tripped off. It reset.

Operations reported at 11:37 PM that there was a problem with chiller at MI-31. Water technicians were investigating.

Saturday January 26

The midnight shift began with TeV experts conducting studies, with Operations monitoring the stash (10.6E10), with Pbar stacking (61.8E10), with NuMI and BNB taking beam, and with water system experts still working on the Pelletron’s LCW system.

Operations reported at 12:23 AM that the Pelletron LCW system was regulating normally. The experts finally got all the air out of the system.

One TeV expert completed his study and another began a different study at 2:08 AM.

At 2:47 AM, Operations reported that the MI Power Supply Interlock Loop (MI PSL) had tripped off twice on quadrupole ground fault. Operators contacted an expert who instructed operators to increase the limit.

The TeV studies ended around 6:52 AM.

At 7:13 AM, KRF7 tripped off. It reset.

Operations put the TeV into shot setup at 7:23 AM.

At 7:34 AM BRF10 tripped off. It reset.

At 7:50 AM, BRF1 tripped off. It reset.

Operations reported at 8:05 AM that the Recycler had stashed antiprotons four times during the midnight shift.

At 8:39 AM, during loading of protons into the TeV, Operations reported that the SBD SyncLite
had a TCPort connection failure. Operators reported that the MI SBD was also down. Operations contacted an expert. This was repaired by 9:11 AM.

Operations established store 5869 at 9:32 AM with an initial luminosity of 134.33E30.

At 10:09 AM, KRF3 tripped off. It reset.

At 10:37 AM, BRF9 tripped off. It reset.

At 1:09 PM, the Booster East Anode Power Supply (EAPS) tripped off on a high cabinet temperature indication and would not reset. BRF3 showed multiple trips at the same time. Operators investigated and found that BRF3 had a LCW flow fault as well as the rest of the RF stations in the east gallery. After operators had reset the stations, the EAPS reset.

Pbar resumed stacking at 1:38 PM.

At 4:33 PM LRF3 tripped off. It reset.

At 6:09 PM, Operations reported that a TeV sector D11 separator sparked. This caused a small hit (~4E30) to the luminosity.

At 6:26 PM, LRF 3 tripped off for the first of three times in two minutes. It reset.

Operations reported at 8:01 PM that the Recycler had stashed six times during the last 12 hours.

At 8:22 PM, LRF2 tripped off for the first of about five times in about 2.5 hours. It reset.

Operations terminated store 5869 at 11:03 PM.

Operations put the TeV into shot setup at 11:13 PM.

Sunday January 27

The midnight shift began with the TeV in shot setup, with Operations monitoring the stash, with Pbar stacking, and with NuMI and BNB taking beam.

Operations established store 5870 at 12:46 AM with an initial luminosity of 208.07E30.

At 2:57 AM, BRF1 tripped off for the first of two times in about five hours. It reset.

At 3:15 AM, KRF2 tripped off. It reset.

AT 4:03 AM, BRF10 tripped off. It reset.

At 5:16 AM, KRF7 tripped off. It reset.

Operations reported at 7:59 AM that the Recycler had stashed five times during the last 12 hours.

At 8:15 AM, LRF2 tripped off. It reset.

At 8:44 AM, Operators attempted to turn on a number of vacuum pumps around the complex that had tripped off over the last 24 hours or so. Most of them turned on except for a few in MI. An expert reported that the panel that fed power to these pumps, which was located in the tunnel, had tripped and would need repair. This would require an access at some later time.

MI and Booster experts began tuning their respective machines at 10:24 AM.

At 11:13 AM, BRF10 tripped off for the first of two times in a bit more than an hour. It reset.

At 2:23 PM, BRF9 tripped off. It reset.

At 2:32 PM, BRF10 tripped off. It reset.

At 3:33 PM, a Booster expert arrived to try and solve some problems with the tune that was causing the MI beam problem.

At 4:01 PM, BRF10 tripped off twice in 26 minutes. It reset.

At 4:43 PM, KRF3 tripped off. It reset.

At 7:09 PM, MIRF15 tripped off. It reset.

At 7:18 PM, LRF1 tripped off for the first off of for times on LCW flow faults. It reset.

At 7:23 PM, KRF3 tripped off. It reset.

Operations reported at 8 PM that the Recycler had stashed seven times during the last 12 hours.

Operations terminated store 5870 at 9:32 PM.

Operations put the TeV into shot setup at 9:43 PM.

At 10:36 PM, KRF2 tripped off. It reset.

At 11:42 PM, Operations reported that a couple of TeV collimators were out of tolerance. Operators contacted an expert and then readjusted the collimators.

Monday January 28

The midnight shift began with the TeV in shot setup, with Operations monitoring the stash, with Pbar stacking, and with NuMI and BNB taking beam.

Operations established store 5871 at 12:05 AM with an initial luminosity of 252.18E30.

At 1:04 AM, KRF7 tripped off. It reset.

At 2:42 AM, CDF experimenters reported that they were having problems with their solenoid.

At 3:20 AM, KRF4 tripped off. It reset.

At 4:35 AM, LRF2 tripped off for the first of four times in about two hours. It reset.

At 7:38 AM, KRF7 tripped off. It reset.

Operations reported at 7:57 AM that the Recycler had stashed six times during the last 12 hours.

The Plans for Monday

The plans for today are to stack, stash, and store. The Run Coordinator has proposed an access to MI beginning around 6 AM Wednesday to replace the bad quadrupole (Q631) magnet. This will take 10 — 12 hours to replace. Booster and Linac will also be allowed to access.

Accelerator Update Archive

More Information
For Tevatron luminosity charts and the current status of Fermilab's accelerators and detectors (live!), please go to Fermilab Now.

Comments and Suggestions
What do you think about the Accelerator Updates? Please send comments and suggestions to: accelupdates@fnal.gov.