Accelerator Update for March 28, 2008 - March 31, 2008

Friday March 28

The day shift began with Operations monitoring the Tevatron (TeV) store 6008 (38.83E30) and the Recycler stash (253.7E10), with the Antiproton Source stacking (23.9E10), with the Main Injector (MI), Linac, and Booster operating normally, with the Collider Detector at Fermilab (CDF) and DZero (D0) using the colliding beams, and with NuMI and the SciBooNE / MiniBooNE (BNB) experiments taking beam.

At 8:13 AM, a Linac Klystron RF station (KRF6) tripped off. It reset.

Operations reported at 8:15 AM that a MI expert rebooted the MI flying wire front end (FE).

At 8:55 AM, two MI RF stations (MIRF12 & 17) tripped off. They reset.

Operations terminated store 6008 at 9:03 AM.

Operations prepared CDF and D0 for access at 9:13 AM.

At 10:11 AM, LRF2 tripped off for the first of three times in three minutes. It reset.

At 10:30 AM, Operations reported that a MI-8 corrector magnet power supply (I:HT830D) was running close to its limit. They also discovered that there was a large orbit oscillation. An expert began tuning the upstream trim magnets to fix the oscillation.

The CDF experimenters dropped their interlocks at 11:32 AM while coming out of access. Operators had the experiment resecured about 40 minutes later.

At 12:27 PM, Operations reported a TeV power supply (T:HB11) had a fuse fault. Operators replaced the supply.

Operations put the TeV through a dry squeeze at 1:15 PM.

The Neutron Therapy Facility (NTF) reported at 1:49 PM that they had completed treatment on their last patient for the day.

At 2:07 PM, Operations reported that they were having problems extracting beam from MI and sending it to the TeV. Experts investigated.

At 3:44 PM, LRF2 tripped off. It reset.

Operations reported at 3:54 PM that after several unsuccessful TeV reboots, an expert unplugged the debug port connector for the FE and injections were successful.

Operations reported at 3:56 PM that the Recycler stashed antiprotons four times during the day shift.

Operations put the TeV through a wet squeeze around 4:20 PM.

Operations put the TeV into shot setup around 6:02 PM.

Operations established store 6011 at 7:56 PM with an initial luminosity of 260.55E30.

At 8:11 PM, KRF3 tripped off. It reset.

At 8:15 PM, KRF2 tripped off. It reset.

At 11:52 PM, Operations reported that they were running NuMI at a lower intensity (six turns) due to beam position problems in MI. Experts will investigate further on Saturday morning.

Operations reported at midnight that the Recycler stashed three times during the evening shift.

Saturday March 29

The midnight shift began with Operations monitoring store 6011 (141.46E30) and the stash (161.6E10), with Pbar stacking (9.1E10), and with NuMI and BNB taking beam.

At 5:23 AM, Operations reported that Linac was having problems with the Debuncher SRM. Also, the vernier had no gradient output. Operations called in an expert.

At 6:58 AM, LRF4 tripped off. It reset.

A Linac expert reported at 7:29 AM that he found and fixed a phase problem.

Operations reported at 7:59 AM that the Recycler stashed five times during the midnight shift.

Operations terminated store 6011 at 9:34 AM.

Operations had D0 and CDF prepared for access by 9:51 AM.

At 11:03 AM, a Pbar beam valve closed (BV728) and a Debuncher septum (D:ISEPV) power supply tripped off on a ground fault indication. The supply reset, but the beam valve wouldn’t open. Operators talked with experts and managed to open the valve by 11:26 AM.

At 11:56 AM, Operations reported a problem with beam positions in MI. Operations called in an expert.

At 12:54 PM, KRF2 tripped off. It reset.

At 1:26 PM, D0 reported that they were having problems with their forward muon triggering system. They were calling in experts.

At 1:37 PM, D:ISEPV tripped for the first of two times in one minute.

At 2:05 PM, D0 reported that they had fixed two of their three problems and were coming out of access.

Operations put the TeV through a dry squeeze at 2:30 PM.

At 2:52 PM, BV728 closed. It opened.

Operations put the TeV into shot setup at 2:54 PM.

At 3:36 PM, the P1 transfer line beam permit tripped off due to a power supply (HT701) being out of tolerance. Operators managed to get the beam line back on about five minutes later.

At 4:06 PM, a number of Pbar Debuncher beam valves closed. Operators were able to open the valves.

At 4:49 PM, D:ISEPV tripped off and BV728 closed for the first of two times in about 30 minutes. Everything reset. Operators investigated, but found no obvious problems.

Operations established store 6013 at 5:09 PM with an initial luminosity of 280.9E30.

At 6:47 PM, D:ISEPV tripped off on a ground fault. It reset.

Pbar experts reported at 7:56 PM that they think the BV728 trips are due to an electrical problem. Also, they don’t see a correlation between the D:ISEPV ground fault trips and the BV728 closings. Their investigations continued.

Operations reported at 7:58 PM that the Recycler stashed six times during the last 12 hours.

At 8:36 PM, BRF7 tripped off. It reset.

At 10:29 PM, KRF4 tripped off. It reset.

At 11:13 PM, BV728 closed. It opened.

Sunday March 30

The midnight shift began with Operations monitoring store 6013 and the stash, with Pbar stacking, and with NuMI and BNB taking beam.

At 12:04 AM, the NuMI beam permit tripped off due to a power supply (E:V108) going out of tolerance. Everything reset.

At 12:55 AM, the MI vacuum computer (MIVAC) hung up. A reboot fixed the problem.

At 3:33 AM, LRF1 tripped off for the first of many times over the next three hours. It reset okay. After the last trip, Operations called in a Linac expert.

At 5:13 AM, a Booster vacuum pump (B:IPRF23) tripped off and caused BRF13 & 14 to trip off.

At 7:41 AM, Operations reported an alarm for a NuMI LCW system running hot. An expert investigated and said that the temperatures were well below critical and needed no immediate intervention.

Operations reported at 7:56 AM that the Recycler stashed eight times during the last 12 hours.

At 9:53 AM, MIRF1 tripped off and took down the NuMI beam permit. Everything reset.

Operations terminated store 6013 at 10:30 AM.

At 10:44 AM, Operations reported a MI High Level RF vacuum alarm. Operations investigated and found that all the MIRF vacuum pumps were off. Operators contacted experts and informed many people of the problem.

Operations prepared D0 and CDF for access at 10:47 AM.

Operations reported at 11:28 AM that they had closed all the MI RF vacuum valves after a vacuum burst. The 604 section was the only place that vacuum was not holding. After conferring with a vacuum expert, Operators attempted to turn on vacuum pumps in the 601 and 602 areas, but the current draw on the pumps was too high and they were turned off.

At 12:04 PM, Operations aborted the Recycler stash (345.0E10) and prepared MI for an access.

Operations prepared Linac for an access at 12:42 PM.

At 1:18 PM, D0 experimenter accidentally dropped their interlocks.

Mechanical technicians reported at 1:51 PM that they were flushing the heat exchanger at MI20. After the flush gets completed, LCW techs and controls experts will calibrate the three-way mixing valve.

Vacuum experts reported at 2:40 PM that the MI vacuum leak was narrowed to an area in between beam valves 604 and 605, but they haven’t located the leak.

Vacuum experts reported at 3:33 PM that they think they found the MI vacuum leak on a bellows downstream of BV604. Later they reported that the leak was actually on the gaskets. The 603 and 604 areas would have to be vented to atmosphere before repairs could begin.

Operations reported that D0 came out of access and Operators search and secured the area at 6:05 PM.

Linac came out of access at 6:41 PM.

At 7:16 PM, vacuum techs reported that they found another MI vacuum leak on the 603 side of the valve that was venting to atmosphere.

Operations put the TeV through a dry squeeze at 7:42 PM.

Operations reported at 7:56 PM that the Recycler stashed twice times during the last 12 hours, but the entire stash was aborted due to the MI vacuum leak and access.

Operations reported at 8:49 PM that LRF1 was still down. KRF4 was running, but might not stay running. Experts were looking at problems with the Klystron Debuncher water skid SRM.

At 9:23 PM, LRF2 tripped off. It reset.

Linac experts began work on KRF4 at 9:53 PM.

Linac experts reported at 10:30 PM that they were replacing the driver for LRF1.

MI vacuum techs came out of access at 11:53 PM. They had completed their vacuum work and the beam line was pumping down.

Monday March 31

The midnight shift began with no TeV store, no Recycler stash due to MI vacuum work, with no Pbar stacking, and with no beam to NuMI and BNB due to LRF1 problems.

Operations reported at 12:26 AM that operators were turning on the MI power supplies and that MI was up and ramping. The MI RF would be kept off until vacuum improved.

At 12:36 AM, KRF6 tripped off. It reset.

Linac experts reported at 12:59 AM that LRF1 was up and running. Operators established beam to the Linac dump.

Operations reported at 6:19 AM that the MI RF vacuum was good enough to turn on the RF system.

At 6:27 AM, Operations reported a regulation problem with a NuMI quadrupole power supply (E:QD111). Operations informed an expert and turned off the NuMI beam.

Pbar resumed stacking at 6:34 AM.

Operations put the TeV through a dry squeeze at 7:54 AM.

At 7:57 AM, MIRF1 tripped off. It reset.

The Plans for Monday

The plans for today are to determine what to do about the NuMI magnet E:QD111. This magnet is located in MI. Operations halted stashing the Recycler due to a possible MI access. The magnet E:QD111 has an internal water leak. Mechanical experts believe that if this magnet is baked for a while it might resume normal operations. It would take 2 days to replace this magnet.

Accelerator Update Archive

More Information
For Tevatron luminosity charts and the current status of Fermilab's accelerators and detectors (live!), please go to Fermilab Now.

Comments and Suggestions
What do you think about the Accelerator Updates? Please send comments and suggestions to: accelupdates@fnal.gov.