Accelerator Update for April 4, 2008 - April 7, 2008

Friday April 4

The day shift began with Operations monitoring the Recycler stash (81.1E10), with the Antiproton Source (Pbar) stacking (34.0E10), with the Recycler, the Main Injector, and Linac, operating normally, with Booster having septa problems, but still running, with the Collider Detector at Fermilab (CDF) waiting for colliding beam, with DZero (D0) in access, and with the NuMI and SciBooNE / MiniBooNE (BNB) experiments taking beam when available.

At 8:12 AM, a Booster septa power supply (MP02) tripped off. An expert was investigating the problem.

TeV experts reported at 8:13 AM that they had cleaned some contacts and reterminated a cable for a Tevatron (TeV) sector A0 kicker (PAK4).

A TeV instrumentation expert reported at 8:15 AM that he was rebooting the TeV BLMs for sectors A1 – A4.

D0 came out of access at 8:28 AM.

Operations prepared the Pelletron for a 30-minute access at 8:57 AM.

Interlock experts began safety system checks for the Switchyard (SY) at 9:05 AM. They completed their testing 30-minutes later.

Operations reported at 9:30 AM that the Pbar target had been moved to a new disk.

At 9:33 AM, Linac Klystron RF station (KRF7) tripped off. It reset.

At 9:43 AM, MP02 tripped off. Booster experts responded.

At 9:46 AM, a Pbar vacuum pump (IP403) tripped off. It reset.

Operations put Pbar into standby at 9:59 AM while Booster experts investigated MP02 problems.

An RF expert began work on a MI RF station (MIRF12) at 10:10 AM.

Interlock experts reported at 10:20 AM that they were done with SY for the day.

Operations reported at 10:22 AM that they had put the TeV through a dry squeeze earlier.

Cryo system experts reported at 10:46 AM that they were working on the TeV sector D1 wet engine cryo vacuum.

A Booster expert told Operations to run some Booster beam. Operations sent the Booster beam at 11:22 AM. MP02 tripped off at 11:23 AM. Operators reset it and it tripped off again at 11:25 AM and then again 45 minutes later.

A MI expert began a coalescing tune up at 12:15 PM.

At 1:04 PM, a Linac expert reported that LRF3 was sparking. Experts had the station back on line within seven minutes.

At 1:26 PM, LRF1 tripped off. It reset.

At 2:01 PM, a Linac expert reported that LRF3 wouldn’t last more that another 30 minutes.

Operations turned off Pbar for 15 minutes at 2:29 PM so technicians could change out the vacuum CIA crates.

At 2:31 PM, Linac experts began work on LRF3. Pbar experts began a study.

Operations established store 6023 at 3:06 PM with an initial luminosity of 123.46E30.

Operations reported at 3:57 PM that the Recycler had stashed antiprotons twice during the day shift.

Linac experts brought LRF3 back online at 4:19 PM.

At 5:03 PM, BRF1 and 19 tripped off. They reset.

At 5:19 PM, Operations turned off the Booster beam and prepared the area for access. Booster experts were accessing to connect diagnostics to MP02 devices.

Booster came out of access at 7 PM.

Operations resumed sending beam to Pbar and NuMI at 7:07 PM.

At 7:47 PM, Operations prepared the Recycler for a stash and made two transfers with very poor efficiency. Operators investigated and found that the Recycler transverse damper was off. It was turned on at 8:02 PM.

At 8:11 PM, a Booster vacuum pump (IPRF23) tripped off and caused BRF13 and 14 to trip off. Everything reset.

At 8:26 PM, KRF2 tripped off. It reset.

At 8:56 PM, Operations prepared the Recycler for a stash. The first transfer was inefficient. Operators investigated and discovered that the MI orbit smooth came on at the wrong time during the stash. This was fixed.

At 10:06 PM, KRF3 tripped off. It reset.

At 11:39 PM, Operations reported that the Recycler VSA had gotten stuck for the third time during measurements on this shift. Operators reported that it would work for a while after each restart.

Operations reported at midnight that the Recycler had stashed three times during the evening shift.

Saturday April 5

The midnight shift began with Operations monitoring store 6023 (70.2E30) and the stash (70.2E10), with Pbar stacking (39.3E10), and with NuMI and BNB taking beam.

At 2:34 AM, KRF2 tripped off for the first of two times in about an hour. It reset.

At 4:12 AM, a Pbar Debuncher vacuum pump (D:IP403) tripped off for the first of two times in 82 minutes. It reset.

At 4:34 AM, Operations reported more problems with the Recycler VSA program.

At 6:10 AM, the TeV sector D11 separator sparked and caused a loss of luminosity.

At 7:38 AM, Operations contacted experts to look at the Recycler VSA.

Operations terminated store 6023 at 7:39 AM.

Operations prepared D0 for an access at 7:43 AM.

Operations reported at 7:54 AM that the Recycler had stashed five times during the midnight shift.

At 8:20 AM, LRF3 tripped off. It reset.

Operations prepared CDF for an access at 8:23 AM.

At 8:34 AM, a TeV vacuum device (T:D1TC7R) came into alarm. An expert had operators close two vacuum valves (CV7U and CV7D). Later, he recommended that a vacuum pump be swapped out.

At 8:35 AM, LRF4 tripped off. It reset.

CDF came out of access at 9:51 AM.

Operations prepared the TeV for an access at 10:21 AM.

Vacuum technicians accessed the TeV at 11:33 AM.

Operators changed out the regulator for (VD37) at 11:39 AM, which was a TeV corrector magnet.

At 12:44 PM, Operations responded to a code “brush fire” issued for the TeV B2 area. Experts were conducting a scheduled prairie burn and the fire jumped the TeV pond. Dried grass had burned up and over the TeV berm before it was put out. No harm occurred. Operations inform all the necessary people.

D0 came out of access at 1:29 PM.

At 2:04 PM, BRF7 tripped off. It reset.

At 2:59 PM, Operations changed out a TeV feeddown sextupole regulator (C:S4C2B) due to a fuse fault indication.

Operations put the TeV through a dry squeeze at 3:31 PM.

At 4:19 PM, Operations received a high temperature alarm for the TeV A3 cryo system building. Operators investigated and found that there was a portable heater running. It was turned off.

Operations put the TeV through a wet squeeze at 4:19 PM.

Operations reported at 4:24 PM that the TeV sector D1 vacuum had improved since the new pump was installed.

At 4:25 PM, LRF1 tripped off. It reset.

At 4:40 PM, a Booster vacuum pump (IPRF17) tripped off and caused the BRF7and 8 to trip off. Everything reset.

Operations put the TeV into shot setup at 4:43 PM.

Operations established store 6026 at 6:36 PM with an initial luminosity of 256.37E30.

At 6:51 PM, LRF4 tripped off. It reset.

Operations reported at 7:56 PM that the Recycler had stashed six times during the last 12 hours.

At 8:05 PM, KRF 7 tripped off. It reset.

At 8:43 PM, KRF3 tripped off. It reset.

At 9:33 PM, LRF3 tripped off. It reset.

Sunday April 6

The midnight shift began with Operations monitoring store 6026 and the stash, with Pbar stacking, and with NuMI and BNB taking beam.

At 12:51 AM, KRF3 tripped off. It reset.

At 1:11 AM, MIRF4 tripped off and continued to trip off after each reset. Operators bypassed the station.

At 1:27 AM, Operations reported that MI beam positions had been bad since operators put in the MIRF4 cavity short. Operations reconfigured the NuMI beam permit box to reflect the missing RF station.

Operations reported at 1:39 AM, that sending a “VSA NOW” command had fixed the problem with the odd longitudinal traces.

At 1:56 AM, a Booster vacuum pump (IPRF6) tripped off. It reset.

At 2 AM, MIRF4 tripped off due to a vacuum problem. Operations reported that MI slipstacking was not good and that they could barely run three turns of MI beam for stacking and NuMI due to very bad beam positions.

At 2:17 AM, LRF4 tripped off. It reset.

Operations reported at 2:22 AM that they were running MI at nine turns, but that bad beam positions were starting to get worse.

At 2:55 AM, Operations reported they were running MI at three turns.

Operations reported at 3:18 AM that they were rebooting the MI damper front end (FE).

At 3:32 AM, Operations reported that they were running MI at seven turns.

A MI expert reported at 4:07 AM that he could not find out what the problem was from home, so was coming in to the lab. He arrived about 30 minutes later.

At 5:35 AM, KRF4 tripped off. It reset.

Operations reported at 5:41 AM that another MI expert was on her way in to help with slipstacking.

An RF expert conferred with Operations about the MIRF4 problems at 7:47 AM.

Operations reported at 7:57 AM that the Recycler had stashed seven times during the last 12 hours.

At 9:07 AM, a MI south anode power supply tripped off and caused MIRF stations 1-6 to all trip off. Everything reset. Operations lowered MI to 5 turns.

Operations reported 9:08 AM that an RF expert was working on MIRF4.

At 9:19 AM, Operations reported that a heat exchanger (HE) temperature had been in alarm since midnight. They contacted an expert to investigate.

At 9:24 AM, Operations reported that they were contacting an expert about the VSA transverse emittance problems.

At 10:06 AM, Operations called in another RF expert to look at MIRF4.

At 10:35 AM, the Booster East Anode Power Supply (EAPS) tripped off. It reset.

At 10:35 AM, LRF4 tripped off. It reset.

At 11:54 AM, KRF4 tripped off. It reset.

At 12:34 PM, MI suffered a vacuum burst.

Operations put Pbar into standby at 12:50 PM due to the MI vacuum problem.

At 1:17 PM, Operations reported that the P150 transfer line was having a vacuum problem.

Operations terminated store 6026 some time around 2:15 PM.

Operations put the TeV into shot setup at 2:38 PM.

At 4:14 PM, KRF7 tripped off. It reset.

Operations turned off the Pelletron at 4:48 PM in preparation for an access.

Operations established store 6027 at 4:59 PM with an initial luminosity of 280.81E30.

At 5:27 PM, Pelletron experts reported that they thought the temperature problems were due to a bad condenser coil on the AC chiller. Later, the experts said the problem was with the condenser fans. Operations contacted the FESS on call person.

At 6:28 PM, BRF7 tripped off. It reset.

At 7:33 PM, the BNB horn tripped on a ground fault indication. It reset.

Operations reported at 7:35 PM that the air conditioning expert was inspecting the Pelletron chiller condenser unit.

At 8:02 PM, the BNB horn tripped off again. It reset.

Operations reported at 8:06 PM that the Recycler had stashed five times during the last 12 hours.

At 8:30 PM, a BNB beam detector tripped off on a failsafe voltage. Operators replaced the detector at 8:55 PM.

At 9:05 PM, the 400 MeV area beam permit tripped off due to bad beam positions and continued to trip after every reset. Operators discovered a problem with the 400 MeV area chopper. Operators called in several Booster experts.

Operations turned on the Pelletron at 9:33 PM for about five minutes to lower the antiprotons emittances.

Operations prepared the Linac 400 MeV area for access at 10:30 PM.

At 10:50 PM, a MI kicker (I:KPS4A) tripped off and took down the MI beam permit. Everything reset.

Booster experts reported at 11:17 PM that they had reterminated a chopper cable.

At 11:32 PM, the NuMI beam permit tripped off due to bad beam positions. It reset.

Operations reported at 11:44 PM that Pbar, NuMI, and BNB were all taking beam.

Monday April 7

The midnight shift began with Operations monitoring store 6027 and the stash, with Pbar stacking, and with NuMI and BNB taking beam.

Operations turned on the Pelletron at 12:43 AM for about 45 minutes.

Operations turned on the Pelletron around 1:29 AM.

At 1:48 AM, LRF4 tripped off. It reset.

At 3:27 AM, BRF6 tripped off. It reset.

At 3:28 AM, the BNB horn tripped off for the first of three times in two minutes. It reset.

At 4:24 AM, BRF9 and 10 tripped off on a vacuum high pressure. They reset.

At 5:29 AM, the Pbar “PBKICK” mode went through some irregularities and caused the sweeping magnet power supplies to trip off. Everything reset.

At 5:44 AM, the MI power supply interlock loop faulted. It reset.

At 6:56 AM, KRF2 tripped off. It reset.

At 7:19 AM, Operations turned off the Pelletron because of temperature problems. Operations contacted experts.

Operations reported at 7:58 AM that the Recycler had stashed five times during the last 12 hours.

The Plans for Monday

The Run Coordinator reported that the Pelletron chiller looked like it was fixed. He also reported that the old MP02 was still operating and seemed to be doing okay. The Booster experts will decide later today if they will try going to higher intensities. D0 has some major problems with two muon scintillator systems that are offline and are limiting their ability to take data.

Accelerator Update Archive

More Information
For Tevatron luminosity charts and the current status of Fermilab's accelerators and detectors (live!), please go to Fermilab Now.

Comments and Suggestions
What do you think about the Accelerator Updates? Please send comments and suggestions to: accelupdates@fnal.gov.