Accelerator Update for October 3, 2008 - October 6, 2008

Friday Oct 3

The day shift began with Operations monitoring the Tevatron (TeV) store 6469 (39.75E30) and the Recycler stash (120.2E10), with the Antiproton Source (Pbar) stackingg (146.9E10), with the Linac, Booster, and the Main Injector (MI) operating normally, with the MiniBooNE and NuMI experiments taking beam, and with the Collider Detector at Fermilab (CDF) and DZero (D0) using the colliding beams.

TeV experts reported at 9:03 AM that they had already started crystal collimator studies.

Power supply experts reported at 10:11 AM that they were conducting thermal scans of the TeV main bus knife switches.

At 10:28 AM, the MI Power Supply interlock Loop (MIPSLOOP) tripped off on a volts-to-ground (ground fault) indication at MI-40. When operators reset the trip and turned back on, MIPSLOOP tripped off again, but this time on an overcurrent indication at MI-20L. Experts investigated.

At 11:05 AM, Operations received a report that the TeV sector E2 air conditioner was making a lot of noise. An AC technician investigated and found a bad fan blade. He had the air conditioner working about three hours later.

Water group experts reported at 11:45 AM that they had replaced the bad power supply for the TeV LCW at CUB.

A water group expert reported at 11:51 AM that they had started the regeneration of column 1 at CUB.

A power supply expert reported at 12:10 PM that MI-20L and MI-30L were both back online. NuMI resumed taking beam.

TeV experts completed their crystal collimator study at 12:20 PM.

At 12:26 PM, a MI RF station (MIRF8) tripped off and would not reset. RF experts investigated and later reported that the station needed major repairs. They bypassed the station.

Operations terminated store 6469 at 12:29 PM.

Operations put the TeV into shot setup at 12:43 PM.

At 1:28 PM, a Linac Klystron RF station (KRF6) tripped off. It reset.

The Neutron Therapy Facility reported at 1:50 PM that they had finished treating their last patient for the day.

At 2:25 PM, KRF5 tripped off. It reset.

Operations established store 6470 at 2:33 PM with an initial luminosity of 87.45E30.

A Recycler expert began a study at 2:40 PM.

At 5:27 PM, LRF2 tripped off. It reset.

At 6:34 PM, the D0 solenoid tripped off. Their experts investigated.

At 7:48 PM, LRF5 tripped off. It reset.

At 7:57 PM, MIRF5 tripped off. It reset.

A Pbar expert began a one-hour study at 8:17 PM.

At 8:45 PM, Operations reported that the TeV sector C17 separator had sparked twice. Fortunately, there was no loss of luminosity.

At 10:43 PM, LRF5 tripped off for the first of two times in 14 minutes. It reset.

Operations reported at midnight that the Recycler had stashed antiprotons once during the evening shift.

Saturday Oct 4

The midnight shift began with Operations monitoring store 6470 and the stash (123.7E10), with Pbar stacking (96.7E10), and with the MiniBooNE and NuMI experiments taking beam.

Operations terminated store 6470 around 12:05 AM.

Operations put the TeV into shot setup at 12:25 AM.

Operations established store 6471 at 1:46 AM with an initial luminosity of 83.7E30.

At 2:05 AM, LRF5 tripped off for the first of two times in three minutes. It reset.

At 6:09 AM, an Accumulator septum (A:ISEP1V) tripped off.

At 6:14 AM, A:ISEP2V tripped off. They both reset.

Operations disabled beam to MiniBooNE at 7 AM. They will remain off for the duration of the shutdown. Operations prepared MI-12 for an access.

At 7:59 AM, the Pbar beam permit tripped off on a target air pressure indication. Operations called in an expert.

At 7:59 AM, the TeV suffered a quench at sector B11. Store 6470 was lost. Operators investigated and found that the TeV sector B1 cold compressor had tripped off.

Operations reported at 8 AM that the Recycler had stashed once during the midnight shift.

A Pbar expert reported at 8:22 AM that he reset the blower for the target.

At 8:08 AM, BRF2 tripped off. It would not reset.

At 9:37 AM, the NuMI beam permit tripped off for the first of two times in five minutes due to a power supply (E:V100) regulation problem.

At 9:43 AM, Operators investigated a water leak in the mechanical support area. They reported that the leak was from a bad valve in an ICW return line. This leak will be repaired later.

At 10:54 AM, LRF5 tripped off. It reset.

A TeV expert began a chromaticity study at 1:50 PM.

At 1:54 PM, KRF6 tripped off. It reset.

At 2:20 PM, Operations aborted the TeV beam due to a closed LCW valve at CUB, leak alarms, and return pressure alarms. Operations investigated and found there was work occurring at CUB; the workers had brought a TeV column online and that dropped the level of the spheres and caused the inlet valve to close.

At 3:13 PM, KRF6 tripped off for the first of two times in 26 minutes on a switch balance indication. It reset.

Pbar stopped stacking at 4:21 PM.

A Pbar expert began a study at 4:41 PM.

At 5:01 PM, Operations reported that MI was tripping off on injection. Operators investigated and replaced a power supply and a control card.

NuMI resumed taking beam at 5:26 PM.

At 6:15 PM, KRF6 tripped off. It reset.

Operations put the TeV through a wet squeeze at 6:38 PM.

At 6:48 PM, LRF5 tripped off for the first of three times in 17 minutes. It reset.

At 7:06 PM, KRF6 tripped off. It reset.

Operations put the TeV into shot setup at 7:26 PM.

At 8:18 PM, LRF5 tripped off. It reset.

Operations established store 6475 at 8:39 PM with an initial luminosity of 107.4E30.

At 8:43 PM, LRF5 tripped off for the first of four times in an hour. The station would not reset after the last trip. Pbar experts investigated.

Operations began preparing MI for an access at 9:51 PM.

Operations began preparing NuMI for access at 10:47 PM.

Sunday Oct 5

The midnight shift began with Operations monitoring store 6475 and the stash.

Operations began preparing Booster for access at 12:12 AM.

Interlock experts began preparing MI and Booster for interlock testing at 12:13 AM.

At 2:35 AM, operators reported that the LCW expansion tank for LRF5 was low. They also reported that there were some sharp changes to the supply water temperature and to the chilled water input water temperature.

At 2:45 AM, KRF6 tripped off. It reset.

Operations reported at 3:07 AM that they were looking for a Linac LCW leak.

At 4:56 AM, Operations reported a LRF5 power amplifier LCW leak.

Operations terminated store 6475 at 5:07 AM. Operations began preparing the TeV for an access.

Operations prepared CDF and D0 for access at 5:41 AM.

Interlock experts reported at 5:45 AM that they had completed the TeV interlock testing.

TeV experts reported at 6:09 AM that they were beginning their hunt for the intermittent ground fault.

Operations prepared Linac for access at 7:32 AM.

Operations reported at 9:50 AM that the Booster had been prepared for access.

A water expert reported at 9:52 AM that the Linac 55 LCW pump #5 had tripped off on Sunday morning and the control system brought up pump #4 up. This was the cause of the temperature spikes.

Cryo system experts at CHL reported at 10:51 AM that they were turning off all the TeV cold compressors.

A Pbar expert reported at 10:56 AM that the Pbar 95 LCW system had a leak somewhere. Operators began searching for the leak.

Operators reported at 11:01 AM that they found the Pbar LCW leak on an Accumulator quadrupole magnet (A5Q1) supply and return hoses.

Operators reported at 11:40 AM that they found more Pbar LCW leaks on Pbar Debuncher quadrupole magnets D4Q5 and D4Q7.

Linac experts reported at 2:29 PM that they were beginning to repair KRF2 and LRF5.

Operations reported at 4:21 PM that the TeV LCW system had lost about 2000 gallons of LCW since early this morning. The also reported a rise in humidity for sector B.

Operators reported at 5:43 PM that they found the TeV LCW leak in the TeV sector F18 area; it was coming from the P2 line. The leaking magnet was valved out. It will be repaired later.

Monday Oct 6

The midnight shift began with the Fermi accelerator complex in shutdown.

Operations reported at 6:27 AM that they were turning off various devices in preparation for the Lab wide power outage.

The Master Substation power outage began at 7 AM. It was over and the power was restored by 7:26 AM.

Operations reported at 7:50 AM:

• Operators and mechanical support experts were recovering the Tevatron vacuum
• The Cryo Coordinator was directing recovery of Tevatron cryo systems
• The water group was recovering Tevatron LCW
• FESS was recovering CUB
• FESS will begin working on Tevatron water pond pump strainers
• Controls experts were addressing a failure of the MI3 CAMAC Front End.
• The MINOS absorber was in access
• Linac was in access
• Operators were restoring CAMAC crates across the complex
• Many, many other jobs were in progress

The Plans for Monday

The weeklong shutdown of the accelerators for maintenance and repairs has begun.

Accelerator Update Archive

More Information
For Tevatron luminosity charts and the current status of Fermilab's accelerators and detectors (live!), please go to Fermilab Now.

Comments and Suggestions
What do you think about the Accelerator Updates? Please send comments and suggestions to: accelupdates@fnal.gov.