Accelerator Update for November 30, 2007 - December 3, 2007

Friday November 30

The day shift began with Operations monitoring the Tevatron (TeV) store 5761 (99.40E30) and the Recycler stash (162.3E10), with the Antiproton Source (Pbar) stacking (23.5E10), with the Main Injector (MI), Booster, and Linac operating normally, with the Collider Detector at Fermilab (CDF) and D-Zero (D0) using the colliding beams, and with the NuMI and SciBooNE / MiniBooNE experiments taking beam.

At 9:19 AM, Pbar tripped off due to bad beam positions. Operations turned off Pbar, put in a NuMI only timeline, and investigated.

Operations turned Pbar back on at 9:30 AM. An operator had accidentally tuned on the wrong device.

An MI expert began conducting 45 minutes of aperture scans at 10:55 AM.

MI experts began a collimator and a slipstacking study at 1:40 PM.

Operations reported at 2:44 PM that Pbar set a new stacking record with 24.30 mA for one hour.

At 2:49 PM, Operations reported a problem with the TeV sector D1 vacuum.

Operations terminated store 5761 at 4:21 PM.

Operations began preparing the TeV and D0 areas for an access at 4:29 PM.

At 5:30 PM, the Linac Klystron distribution water skid tripped off and caused the Linac beam permit to trip. An expert investigated and reported that the skid tripped off due to an over temperature indication and that the chilled water pressure was at half of its normal value. Operations contacted the Duty Mechanic and he headed out to CUB to investigate further. Everything was eventually reset.

The TeV access into sector D began at 5:31 PM.

A TeV expert reported at 7:31 PM that the vacuum leak was in between two sector D17 collimators. Vacuum technicians began their repairs.

D0 came out of access at 9:20 PM.

Vacuum experts reported at 10:12 PM that they had completed their work on the D17 bellows and vacuum pump.

Operations began the TeV turn on at 10:49 PM.

Saturday December 1

The midnight shift began with the TeV turn on in progress, with Operations monitoring the stash (294.5E10), with Pbar stacking (145.4E10), and with NuMI and Sci/MiniBooNE taking beam.

At 12:44 AM, Operations reported a problem the 400MeV area PLC. Operators investigated and then called in experts. Operations turned off Pbar and put in a NuMI only timeline.

Operations put the TeV through a dry squeeze at 1:19 AM.

Controls experts reported at 4:07 AM that the 400 MeV area PLC problem was due to a bad CPU. Operations contacted a PLC expert.

Operations reported that the 400 MeV line PLC was back on line by 8:12 AM.

Operations began working to reestablish beam to Sci/MiniBooNE, Pbar, and NuMI at 8:47 AM.

Operations put the TeV through a wet squeeze at 9:20 AM.

At 10:12 AM, Operations reported having problems with a TeV collimator (T:E02H). It seemed to be moving in the wrong direction. Operations called in an expert.

Operations put the TeV through a dry squeeze at 11:20 AM.

Operations began preparing the TeV for an access at 11:55 AM to check the sector E0 collimator position. An expert had come in and swapped out controller cards. He found a bent pin. However, he said the collimator was probably cocked and would have to be moved manually.

At 12:49 PM, collimator experts reported that the E02H collimator was cocked, had a crushed limit switch, and its transmission was messed up. They began repairs.

The TeV collimator work was completed and Operations began the TeV turn on at 3:30 PM.

Operations put the TeV through a dry squeeze at 3:50 PM.

At 4:09 PM, Operations reported that all the Linac Klystron RF stations tripped off again due to another distribution water skid trip. Operators investigated.

Linac resumed sending beam at 4:56 PM.

Operations put the TeV through a dry squeeze at 5:55 PM.

Operations put the TeV into shot setup at 6:23 PM.

At 7:15 PM, Operations reported that all of the TeV QPM nodes dropped off the link. Operators contacted an expert. With help from the experts, operators fired all the HFUs, power cycled the QPMs, and everything reset.

Operations began the TeV turn on at 9:06 PM.

At 9:21 PM, Operations quenched the TeV at sectors DA and DB while trying to figure out what was happening with the QPM. Operations reported that they should have loaded a QBS ramp.

Operations began the TeV turn on at 11:24 PM.

Sunday December 2

The midnight shift began with the TeV turn on in progress, with Operations monitoring the stash, with Pbar stacking, and with NuMI and Sci/MiniBooNE taking beam.

Operations put the TeV through a dry squeeze at 12:29 AM.

Operations put the TeV into shot setup at 12:53 AM.

Operations established store 5768 at 2:43 AM with an initial luminosity of 186.83E30.

The Pelletron’s front-end stopped responding at 5:09 AM. Operators investigated.

The Pelletron’s front end was still down at 6:23 AM. Operators contacted experts and with their instructions began a restart of the Accelnet computer.

At 6:38 AM, all of the Pbar TWTs tripped off. Operators contacted an expert who discovered that just one PLC was not responding. After a reset everything came back up.

Operations reported at 7:52 AM that the Recycler had stashed antiprotons twice during the last 12 hours.

At 10:05 AM, the Klystron distribution water skid tripped off. Operators tried to reset the station, but it would not stay on. Operations called in an expert.

At 11:28 AM, Operations reported a problem with temperatures at TeV sector A3. Experts reported that this was just a thermometry crate problem; it would be looked at later.

Linac resumed sending beam at 11:57 AM.

At 12:36 PM, the Klystron distribution skid tripped off. An expert was here and investigated.

The Linac expert managed to reset the distribution skid, but still couldn’t find what was causing the bad status bit.

Linac resumed sending beam at 3:54 PM.

At 5:17 PM, the Klystron water skid tripped. The Linac expert bypassed the bad bit and then worked on the water skid. Operations set up a special alarm page to watch the Klystron water skid system.

Linac resumed sending beam at 6:07 PM.

Operations terminated store 5768 at 7:01 PM.

Operations put the TeV into shot setup at 7:15 PM.

Operations reported at 8:11 PM that the Recycler had stashed three times during the last 12 hours.

At 8:28 PM, the TeV suffered an abort that was pulled by a sector F1 BLM. Operators investigated and found that a stacking pulse was missed. Operators contacted an expert.

Operations put the TeV through a dry squeeze at 9:29 PM.

Operations put the TeV into shot setup at 9:54 PM.

Monday December 3

The midnight shift began with the TeV in shot setup, with Operations monitoring a stash, with Pbar stacking, with NuMI in access, and with Sci/MiniBooNE taking beam.

Operations established store 5770 at 12:50 AM with an initial luminosity of 190.99E30.

At 4:08 AM, a Booster RF station (BRF3) tripped off for the first of many, many times. It reset.

At 6:55 AM, Operations reported that BRF3 tripped off and would not reset. Operators bypassed the station and brought on BRF19 to take its place.

A RF expert investigated the Booster RF problems at 7:21 AM. Later, experts reported that repairs would require some downtime.

Operations reported at 7:55 AM that the Recycler had stashed twice during the last 12 hours.

The Plans for Monday

The plans for today are to implement the new Recycler damper system. The Run Coordinator will allow MI experts to conduct an aperture scan and slipstacking studies today. He will also allow Booster to conduct any repairs it needs to fix BRF3.

Accelerator Update Archive

More Information
For Tevatron luminosity charts and the current status of Fermilab's accelerators and detectors (live!), please go to Fermilab Now.

Comments and Suggestions
What do you think about the Accelerator Updates? Please send comments and suggestions to: accelupdates@fnal.gov.