Accelerator Update for April 9, 2008 - April 11, 2008

Wednesday April 9

The day shift began with the Tevatron (TeV) in quench recovery, with Operations monitoring the Recycler stash (397.6E10) and the Antiproton Source (Pbar) stack (32.7E10), with the Recycler, the Main Injector, the Booster and Linac either in access or conducting studies, with the Booster off except for sending protons to MI for the next storee, with the Collider Detector at Fermilab (CDF) and DZero (D0) waiting for the colliding beam, and with the NuMI and SciBooNE / MiniBooNE (BNB) experiments off due to the Booster kicker magnet (MP02) problems.

Operations reported at 8 AM that controls experts were trying to understand the TeV MDAT problems.

An expert began a Pbar stacktail study at 8:35 AM.

Operations reported the following status at 8:36 AM: the TeV sectors A2 and A3 power supply work was completed, interlock experts had completed the electrical safety system work at A2, and the wet engine work at TeV sectors C1, 2, 3, and D1 continued.

Operations put the TeV through a QBS ramp at 9:15 AM.

Cryo system experts reported at 9:27 AM that they had completed their wet engine repairs at C1, 2, and 3. They started work on D1 and that would be their last repair.

Operations performed a successful abort test of the TeV at 10 AM.

A power supply expert reported at 10:58 AM that the TeV quadrupole power supply (QDD1) was repaired.

Operations reported at 11:25 AM that an expert replaced a cable for the Recycler VSA and rebooted everything.

Operations turned off the BNB power supplies at 11:37 AM per an expert’s request.

At 1:24 PM, Operations reported that controls experts had fixed a TeV MDAT crate problem, but, unfortunately, this caused the TeV ramp and the low beta magnets to trip off.

Operations had the TeV ramping by 1:48 PM.

Operations put the TeV through a dry squeeze at 1:59 PM.

TeV experts began an orbit smooth at 3:07 PM.

Operations preformed a wet squeeze at 3:42 PM to validate the orbit smooth.

The Pbar stacktail study ended at 4:48 PM.

Operations put the TeV through a dry squeeze at 5:30 PM.

A Pbar expert began a study at 5:38 PM.

Operations put the TeV into shot setup at 5:52 PM.

Operations turned off the Booster Gradient Magnet Power Supply (GMPS) at 6:42 PM.

At 6:56 PM, a Linac Klystron RF station (KRF3) tripped off. It reset.

Operations had the Booster prepared for access at 7:09 PM.

At 7:13 PM, the TeV suffered a quench at sectors A1, all of B, C1 and 4, D1, and E1. In total, nine houses quenched. Operators investigated.

At 7:15 PM, a building alarm brought the Fermi Fire Fighters and operators to the TeV sector D0 compressor room. The Fire Fighters reported a haze along the ceiling. Operators reported that all three compressors were running and there were no signs of smoke or tripped fire heads.

At 7:39 PM, the cryo system technicians reported that the TeV sector B1 house was venting gas. Operators verified that there was helium relief valve stuck open. Operations prepared the area for CHL operators to access the roof.

Operations reported at 8:24 PM that the TeV quench clock event announcement ($4E) and the low beta quench announcement ($5E) did not go out and this was why so many houses quenched.

Operations started the cryo recovery for TeV sectors C1 and E1 at 8:34 PM.

The Pbar study ended at 8:36 PM.

At 9:56 PM, the Cryo Coordinator reported a suspected Kautzky valve leak in TeV sector B19.

At 10:12 PM, a Booster expert reported the smell of smoke in the Booster West Gallery. Operators investigated and then called in the Fermi Fire Department. The Fire Fighters used their thermal imaging equipment, but could not find a hot spot.

At 10:30 PM, Operators tried to turn Booster back on, but BRF16 would not turn on. Operators investigated and reported that a bad smell was coming from its modulator.

At 10:53 PM, a Booster vacuum pump (B:IP171) tripped off. It reset.

At 11:21 PM, cryo system techs reported a stuck Kautzky valve on the sector B19-2 dipole magnet. They will have to warm the house to replace the valve.

At 11:24 PM, KRF4 tripped off. It reset.

At midnight, experts reported that a TeV sector B11 separator spark was the probable cause for the quench.

Thursday April 10

The midnight shift began with the TeV recovering from a quench, with Operations monitoring the stash (4.5E10), with Pbar not stacking (0.5E10), and with NuMI and BNB off due to the MP02 replacement.

Operations reported at 12:01 AM that cryo experts were replacing Kautzky valves, a Recycler expert was conducting an Electron Cooling study, and Pbar experts were conducting studies.

Cryo system techs reported at 1:51 AM that they had repaired all the helium leaks.

At 1:59 AM, KRF7 tripped off. It reset.

At 2:55 AM, LRF3 tripped off. It reset.

At 3:46 AM, KRF2 tripped off. It reset.

At 4:43 AM, KRF7 tripped off. It reset.

Operators prepared Pbar for access at 6 AM.

At 6:03 AM, LRF2 tripped off. It reset.

A controls expert began investigating the $4E announcement problems at 7:09 AM.

Operators prepared Linac for access at 7:31 AM.

Operations reported at 7:32 AM that the MP02 replacement work was beginning.

A controls expert reported at 9:01 AM that the lightning strikes had damaged a “request drive” card at TeV sector A2 and the A6 CAMAC crate.

Operations reported at 9:13 AM that the following areas were in access: Pbar rings and transport, Linac, Booster, NuMI, and the Pelletron.

A Linac expert began work on LRF3 at 9:26 AM.

A TeV expert put the TeV abort kickers into their conditioning mode at 9:31 AM.

Operations prepared MI for an access at 10:29 AM.

A cryo system expert reported at 10:31 AM that the TeV sector E2 wet engine had a cracked connecting rod bearing. They began their repairs.

The TeV came out of access at 10:59 AM.

Operations began a TeV QBS checkout at 12:04 PM.

Linac came out of access around 12:43 PM.

Operations established beam to the Linac dump at 1:52 PM. The Preaccelerator was running on the H- Source.

A Recycler expert rebooted the Recycler low level RF system at 2:04 PM.

D0 came out of access at 2:43 PM.

Operations reported at 2:50 PM that the MI-31 interlocks had been intentionally dropped to access the Pelletron on Friday. A Recycler expert reported that the Pelletron had some serious problems, which would have to be investigated before it could operate.

Operations began ramping the TeV at 2:58 PM.

Operations put the TeV through a dry squeeze at 3:11 PM.

Pbar came out of access at 3:15 PM.

Operations prepared D0 and CDF for access at 3:57 PM.

Operations prepared Linac for an access at 4:42 PM. They came out of access about 40 minutes later.

Operations reported at 5:58 PM that TeV experts had put the TeV separators and abort kickers into conditioning mode.

MI came out of access at 6:40 PM.

At 9:58 PM, KRF7 tripped off. It reset.

Booster came out of access at 10:06 PM.

Friday April 11

The midnight shift began with the TeV at 150 GeV, with the TeV separators and abort kickers conditioning, Linac sending beam to its absorber, and with everything else off.

Experts reported at 7:42 AM that an Accumulator sextupole power supply (A:SEX10) was repaired.

The Plans for Friday and the Weekend

The Run Coordinator has to wait until this afternoon to get a status report on MP02 and the Pelletron before he can make any plans for today or the weekend.

Accelerator Update Archive

More Information
For Tevatron luminosity charts and the current status of Fermilab's accelerators and detectors (live!), please go to Fermilab Now.

Comments and Suggestions
What do you think about the Accelerator Updates? Please send comments and suggestions to: accelupdates@fnal.gov.