Accelerator Update for February 2, 2007 - February 5, 2007

Friday February 2

The day shift began with Operations monitoring the Recycler stash (178.9E10), with the Antiproton Source (Pbar) stacking (42.5E10), with the Tevatron (TeV) being prepared for an access, with the Linac, Booster, Main Injector (MI), Recycler operating normally, with MiniBooNE and NuMI off, with the Collider Detector at Fermilab (CDF) and D-Zero (D0) in access.

Operations prepared Pbar for an access at 9:14 AM.

NuMI resumed taking beam at 10:01 AM.

Pbar experts came out of access and operations began turning on the machine at 10:03 AM.

At 10:26 AM, a Linac Klystron RF station (KRF6) tripped off. Operators investigated. Experts reported that the station lost its phase lock.

D0 came out of access at 11:06 AM.

The Neutron Therapy Facility (NTF) reported that they had completed treatment on their last patient at 11:55 AM.

The TeV access ended at 11:59 AM.

Operations received the Run Conditions for running beam to the Meson MTest area at 1:38 PM.

Switchyard (SY) experts began tuning beam to the Meson absorber at M01 at 1:54 PM.

Booster RF experts reported at 2:49 PM that there was problem with BRF9 and it was in the tunnel.

A MI expert began an hour-long slipstacking study at 3:10 PM.

Operations reported at 3:56 PM that the Recycler had stashed antiprotons twice during the day shift.

Operators interlocked the CDF collision hall around 4:20 PM.

At 6:48 PM, Operations reported problems with the Recycler’s stochastic cooling system. Operations contacted an expert who told them to set the cooling manually.

The NuMI beam permit tripped due to bad beam positions on their target at 7:51 PM. The permit reset okay.

The Cryo System Coordinator reported at 10:21 PM that the vacuum in the TeV A18 region had started to degrade.

Operations began preparing the TeV for an access at 11:28 PM. Vacuum technicians would be in around midnight to check out the A18 vacuum.

Operations reported at midnight that the Recycler had stashed twice during the evening shift.

Saturday February 3

The midnight shift began with Operations monitoring the stash (314.0E10) and stack (72.3E10), with the TeV in access, and with NuMI taking beam.

Vacuum techs came out of the TeV at 1:18 AM.

At 1:36 AM, Operations reported that BRF6 had tripped off for the first of several times during the midnight shift. It reset okay.

Operations put the TeV through a dry squeeze at 3:04 AM.

A power supply expert powered the TeV A2 sextupole power supplies at 7:07 AM.

A Meson expert began tuning beam to MTest at 9:42 AM.

Operations reported at 10:37 AM that the new TeV sextupoles had been commissioned and were operational.

Operations injected protons into the TeV at 11:01 AM for studies.

At 11:06 AM, Operations received a fire alarm for the F0 service building. Operators noted that there were no readbacks coming from equipment in the building, which pointed to a yard breaker trip.

At 11:26 AM, Operations reported that the power for the F0 compressor room was down. The #2 compressor had failed and caused the breaker trip. Operators and the Duty Electrician reset the yard breaker and then reset and turned on the other three compressors.

Operations loaded the TeV with six bunches of protons. They will put the TeV through a wet squeeze for a few hours.

The Meson expert reported at 12:27 PM that he had tuned beam to MT4. He was finished for the time being. Operations disabled the slow spill.

TeV experts reported at 1:11 PM that the TeV tunes didn’t look right. They continued to work with the orbits and tunes.

At 1:45 PM, a Linac RF station (LRF1) tripped off. An expert who happened to be around helped get the station back on by raising the power amplifier’s filament current.

Operations put the TeV through a dry squeeze at 3:07 PM.

At 4 PM, Operations reported that Pbar stacking and production readbacks had stopped. After operators had tried several restarts an expert was called.

At 4:32 PM, Operations reported a MI CAMAC link driver problem. Operations contacted Controls experts.

At 4:48 PM, the Pbar expert reported that the readbacks had stopped when the stack had reached 200 mA due to a program error.

The Pbar expert restarted a device at 5:17 PM that enabled a believable stack rate.

Operations reported at 5:49 PM that the CAMAC problem was due to a hung serial link controller in the MI-3 front end. A power cycle fixed the problem.

Operations reported at 6:09 PM that the TeV tune problems were continuing. Experts continued their investigations.

At 6:13 PM, the Pelletron tripped off due to a bad MI beam position. An expert helped operators fix the problem.

At 7:39 PM, the Meson MS2 LCW system alarmed. There were no readbacks from power supplies and other equipment. Operators and the Duty Mechanic and Electrician investigated.

At 7:57 PM, a Pelletron expert reported that the Pelletron hadn’t been regulating properly since it had tripped off earlier. He said that he would investigate the problem.

MI experts completed at barrier bucket study at 8:02 PM.

Operations reported at 8:08 PM that the Meson MS2 area and equipment were back in service.

At 9:10 PM, Operations reported more Recycler cooling problems. Earlier, operators had found that the cooling was wrongly in the Gated Mode. They turned the devices off that controlled this. This time, operators again found these devices on and turned them off. They were not sure why these devices were getting enabled.

At 9:18 PM, the MI-3 serial link driver (SLD) failed again. An expert came in and replaced the module. MI resumed sending beam at 10:06 PM.

At 10:18 PM, the MI-3 SLD hung up again. Operations contacted an expert who suggested changing a 290 card. Operations reset the front end (FE).

Operations put the TeV into shot setup at 11:41 PM.

Sunday February 4

The midnight shift began with the TeV in shot setup, with Operations monitoring the stash (302.5E10) and stack (227.7E10), with MI, Recycler, Linac, and Booster operating normally, with NuMI taking beam, with MiniBooNE off, and with CDF and D0 waiting for the colliding beams.

Operations established store 5208 at 1:26 AM with an initial luminosity of 146.56E30.

At 1:49 AM, MI-3 problems have crippled some automated checks related to the shots.

At 2:04 AM, operators began changing fuses and swapping cards at MI-3 with no help. Operations called in an expert.

A Controls expert arrived at 3:56 AM and began his investigation. He was still searching for the cause of the problem at 5:28 AM.

Another Controls expert came in at 9:11 AM.

At 2:02 PM, a TeV RF station (TRF2) tripped off.

Another Controls expert came in at 2:29 PM to help with the MI Controls problems.

At 2:34 PM, store 5208 aborted due to the TeV sector C3 vacuum. Operators investigated, but found no vacuum problems. Later, operators changed out a vacuum controller card just to be safe.

Operations prepared CDF for an access at 3:01 PM. They came out of access at 4:30 PM.

At 4:52 PM, Operations reported a problem with the MTest interlocks. Operations contacted an expert who reported he would investigate the problem on Monday morning. The area was already off.

Operations put the TeV through a dry squeeze at 5:33 PM.

Operations called in more Controls experts at 7:59 PM to replace the ones that had been here all day working on the MI-3 SLD problems.

At 11 PM, Operations reported a TeV kicker at sector E48 had tripped off and wouldn’t reset. Operators investigated and then called a TeV expert.

Monday February 5

The midnight shift began with a TeV kicker problem, with Operations monitoring the stash (98.7E10) and stack (131.9E10), with MI-3 CAMAC problems continuing, with the Recycler, Linac, and Booster operating normally, with NuMI off due to MI-3 problems, with MiniBooNE off, and with CDF and D0 waiting for the colliding beams.

Operators, with advice from an expert, managed to get the TeV kicker (T:E48K) on at 12:13 AM.

At 2:43 AM, a Recycler kicker (R:KPS2A) alarmed on a temperature fault. This kicker was used to inject antiprotons into the Recycler and had its own special cooling fluid. Antiprotons cannot be loaded into the Recycler with out it.

Operations, at the request of Controls experts, established beam to NuMI at 5:08 AM.

At 5:12 AM, Operations reported that a Pbar Debuncher separator at AP50 (D:ISEPV) wouldn’t turn on. Operations called in an expert.

Controls experts working on the MI-3 CAMAC problem reported at 5:24 AM that they finally tracked down a crate that was offline due to a bad power supply and controller.

At 6:38 AM, the power supply expert said that the Debuncher separator’s power supply had a bad charging capacitor. He began repairs.

At 6:15 AM, a Booster vacuum leak caused the MI-8 beam line’s beam valve to close, halting all downstream beam. Two vacuum pumps had tripped off. They eventually reset.

At 7:15 AM, cryo system techs at CHL reported a power glitch. Operations contacted people. Later, the Duty Electrician reported that feeders 36 and 37 were off.

Operations established beam to NuMI at 7:24 AM.

Operations reported at 7:33 AM that power had been restored to CHL. They were starting up their compressors.

Operations reported at 8:02 AM that D:ISEPV was repaired.

At 8:09 AM, Operations reported that the MS2 LCW system was off.

The Plans for Monday

The plans for today are to turn on the TeV later this morning after CHL recovers and to repair Pbar, Recycler, and the MS2 LCW problems. Hopefully, Operations will be able to establish a store this afternoon.

Accelerator Update Archive

More Information
For Tevatron luminosity charts and the current status of Fermilab's accelerators and detectors (live!), please go to Fermilab Now.

Comments and Suggestions
What do you think about the Accelerator Updates? Please send comments and suggestions to: accelupdates@fnal.gov.