Accelerator Update for July 07, 2008 - July 09, 2008

Monday July 7

The day shift began with Operations monitoring the Tevatron (TeV) store 6273 (278.87E30) and the Recycler (RR) stash (10.1E10), with the Antiproton Source stacking (58.2E10), with the Main Injector, Booster, and Linac operating normally, with the Collider Detector at Fermilab (CDF) and DZero (D0) using the colliding beams, and with the SciBooNE / MiniBooNE (BNB) and Meson MTest experiments taking beam.

At 8:11 AM, Operations reported that they had lost the beam permits for the P1 and P2 transfer lines, Switchyard (SY), MI, and Pbar. FIRUS indicated that two breakers had tripped off at the Kautz Road substation. The store, stack, and stash all remained intact. The High Voltage Coordinator investigated.

Operations put Pbar into standby at 8:16 AM.

Operations reported at 8:28 AM that the harmonic filter at the Kautz Road substation had tripped.

A water expert reported at 8:39 AM that he was turning off the LCW pump at MI-30.

At 9:09 AM, the MI ramp tripped off for the first of two times in nine minutes. The cause was traced to the differential water pressure at MI-30. The LCW pump alarm was bypassed.

MI resumed sending beam around 9:18 AM.

Operations turned off MI at 9:52 AM to allow Kautz Road substation work.

A CUB expert raised the regulating temperature for the Pbar’s LCW system by 5 degrees to 95˚ F at 10:10 AM.

Operations resumed sending beam to BNB at 10:14 AM.

Operations prepared MI for access at 10:19 AM.

At 11:26 AM, the Pbar 4-8 momentum TWTs tripped off due to a chiller problem at AP-50. An expert investigated and reported that the chiller was running but there were no flow readings. He thought the problem was in the tunnel.

Instrumentation experts rebooted the Recycler BPMs at 11:51 AM.

Operations reported at 12:16 PM that Linac experts were conducting a RF station (LRF5) study.

Operations reported at 12:25 PM that the MI ramp was reestablished.

At 12:35 PM, Operations reported that the Pbar AP3 beam stop had been moving for quite a while, but it hadn’t made it to its out position. Operations contacted an expert.

Operations resumed sending beam to BNB at 12:53 PM.

FESS personnel began work at 12:54 PM on the Booster East Gallery cooling ductwork repairs.

Pbar resumed stacking at 1:04 PM.

Operations reported at 1:12 PM that the Pelletron was turned off.

Operations established beam to the Meson MTest experiment at 1:27 PM.

Operations reported at 1:57 PM that the AP3 beam stop was closed. Pbar cannot transfer antiprotons to the Recycler. Experts investigated.

Operations halted stacking at 3:10 PM and put Pbar into standby in anticipation of an access to investigate the AP3 beam stop problem.

Operations prepared Pbar for access at 3:48 PM.

At 4:15 PM a Booster vacuum pump tripped off (IPRF23) and caused BRF13 & 14 to trip off. Everything reset.

Linac experts reported at 4:34 PM that they had completed work on the Linac Klystron vernier water turbine.

At 5:38 PM, the TeV sector E1 compressor tripped off. Cryo system experts will turn on another compressor after this store ends.

At 5:39 PM, LRF2 tripped off. It reset.

Pbar experts reported at 5:56 PM that the AP3 beam stop motor was spinning. The linkage between the motor and beam stop had broken. Experts were already preparing a new motor for installation.

Linac experts reported at 5:58 PM that they were modifying LRF5 for the low level RF study they plan to begin.

FESS reported at 7:10 PM that they had completed their dual temperature water repairs in the Booster East Gallery near the Low-Level RF room. The chiller was back online and the air conditioning units were operating normally.

A mechanical expert reported at 7:17 PM that the beam stop motor had been installed and was being tested. (The picture below shows the assembled unit.)

Operations began turning on the transport power supplies at 7:41 PM.

At 8:30 PM, Linac experts reported that they were having problems with the new low level RF system for LRF5. Operations requested that they revert everything back to the normal operating mode.

Operations reestablished beam to BNB at 8:43 PM.

At 10:16 PM, Operations reported that the MI beam did not look good and they could not fix it. Operations contacted experts.

Operations disabled the MTest beam at 10:41 PM. They would not receive beam during the Pbar access.

Operations discovered at 11:13 PM that the Booster damper was not working and causing beam problems in MI. They called in a damper expert.

At 11:40 PM, KRF1 tripped off. It reset.

Tuesday July 8

The midnight shift began with Operations monitoring store 6273 (66.52E30) and the stash (34.8E10), with Pbar stacking (17.5E10), and with BNB taking beam.

At 1:22 AM, the Linac Klystron Vernier tripped off. It reset.

At 1:48 AM, a Booster expert reported that the BEX3 power supply was not ramping. Operators replaced its control card.

At 1:55 AM, KRF3 tripped off. It reset.

At 2:36 AM, Operations reported that the BEX3 power supply was still not ramping. The expert moved the input reference cable and the power supply began ramping. This cable will be replaced during the day shift.

Operations was able to resume stashing at 3:10 AM.

At 4:22 AM, KRF4 tripped off. It reset.

Operations established beam to MTest at 6:42 AM.

Operations reported at 7:58 AM that the Recycler had stashed antiprotons six times during the midnight shift.

At 8:16 AM, the 400 MeV area tripped off due to bad beam positions. It reset.

Operations terminated store 6273 at 8:40 AM.

Operations began at 8:47 AM preparing the TeV for an access.

At 8:54 AM, Operations reported that a TeV sector D2 VCB would not open. Operations contacted an expert.

At 9:22 AM, interlock experts began investigating the TeV electrical safety system for a possible ground fault.

At 9:37 AM, a power supply expert reported that the TeV sector D2 VCB might have a faulty controller. They had the controller replaced about 15 minutes later.

At 9:39 AM, KRF1 tripped off. It reset.

Operations prepared CDF for access at 9:59 AM.

Operations prepared D0 for access at 10:06 AM.

At 11:19 AM, LRF4 and 5 tripped off. They reset.

Interlock experts reported at 11:32 AM that they were finished with their search at TeV sector A3. They found corrosion on a board.

Both CDF and D0 came out of access at 11:53 AM.

Pbar resumed stacking at 12:04 PM.

Operators reported at 12:29 PM that they had completed their search and secure of TeV sector A.

At 12:40 PM, the Pbar Debuncher Septa (D:ISEPV) tripped off on a ground fault indication, which caused two beam valves (BV210D and BV309) to close. Operators reset the air bit and the valves opened.

At 1:09 PM, KRF5 tripped off. It reset.

At 1:48 PM, a TeV sector E1 wet engine tripped off and dumped the ramp. Cryo system experts investigated.

At 2:01 PM, the Cryo Coordinator reported that there was a problem with the TeV sector E1 IO crate. He sent out experts to investigate.

At 2:10 PM, the Pelletron tripped off and would not reset. Experts investigated.

At 2:31 PM, LRF2, 3, 4, & 5 all tripped off for the first of three times in three minutes. They all reset. Linac experts investigated.

The Cryo Coordinator reported at 2:53 PM that the TeV sector E1 wet engine was fixed and the cryo system was recovering.

At 2:55 PM, BRF3, 9 and 12-18 all tripped off. Eventually, everything reset.

At 3:23 PM, Operations reported that the MI-10 LCW temperatures were nearing their trip point of 100˚ F. Operations disabled the MTest beam to help lower the temperature.

Booster experts began a study at 3:44 PM.

Operations reported at 4:06 PM that the Recycler had stashed four times during the day shift.

Operations reported at 4:13 PM that the TeV sector E1 cryo system had cooled down. Operations began the TeV turn on.

At 4:27 PM, the TeV suffered a quench at sectors BD, BB, DA, & DB. Experts investigated.

At 4:29 PM, the Pbar sweeper magnet (M:USWSTA) tripped off. An expert investigated and later reported that Sweeper “A” would not reset. Operations turned on Sweeper “B.”

Operations reported at 6:04 PM that the quench was probably due to a warm slug of helium gas. Operators discovered some TeV quadrupole lead flows that were off.

At 6:15 PM, Operations reported having trouble with the low beta magnet heater tests during the TeV checkout. Operations called in TeV experts.

At 6:59 PM, Operations reported that the HFU test for the B0 and D0 QPMs did not work. Operations rebooted the B0 QPM, but there were still problems. Operators contacted a computer expert.

At 7:19 PM, KRF5 tripped off. It reset.

At 7:20 PM, Operations reported that operators had put Pbar into standby due to a chiller problem at AP-50.

At 8:09 PM, Operations reported that the Pbar problem was due to the 4-8 momentum TWT system and not the AP-50 chiller. Operations called in experts.

Operations reported at 8:43 PM that they had transferred all antiprotons out of Pbar in preparation for a Pbar access. The Recycler stash was at ~404E10.

Computer experts reported at 9:50 PM that they had replaced the TeV B0 QPM VME processor and span boards, but problems remained. One of the experts reported that there were some software problems. A software expert was contacted.

At 10:23 PM, LRF5 tripped off. It reset.

At 10:53 PM, the Recycler stash was lost. A MI-50 bulk power supply (R:C501) tripped off on a regulator control power fault. An expert was called in.

Operations reported at 10:57 PM that the software expert found a problem with a parameter boot script for the QPMs. When the system began working, operators discovered a bad 10-volt VFC chassis. Experts began their repairs.

Operations reported at 11:50 PM that the Recycler had stashed two times during the evening shift.

Wednesday July 9

The midnight shift began with TeV QPM investigations, with Operations monitoring the stash (48.6E10) and stack (22.3E10), and with BNB taking beam.

At 1:13 AM, the Pbar beam permit tripped off for the first of two times in about 80 minutes due to the target dump temperature. It reset.

TeV experts reported at 1:24 AM that they had found two bad ribbon cables on the 10-volt VFC chassis.

Pbar resumed stacking at 2:38 AM.

TeV experts reported at 2:45 AM that they had replaced the two bad ribbon cables, but the QPM showed that the chassis had four bad cards. They reinstalled the original chassis. The QPM showed three bad cards. They continued to investigate.

TeV experts reported at 3:08 AM that they reseated the processor and span cards and everything began operating normally.

Operations established beam to MTest at 4:15 AM.

Operations put the TeV through a dry squeeze at 4:26 AM.

Operations put the TeV through another dry squeeze at 5:42 AM. Operators reported that they were still having abort problems from the TeV sector C1 BLM system with no beam in the TeV. Operations contacted experts.

Operations reported at 5:55 AM that with expert help they managed to inject protons into the TeV.

At 6:54 AM, Operations reported a problem with the TeV “Smooth Proton Only” aggregate. Operations contacted experts.

Operations reported at 7:22 AM that they rebooted the TeV Front End.

Operations reported at 7:50 AM that the Recycler had stashed five times during the midnight shift.

The Plans for Wednesday

Accelerator Update Archive

More Information
For Tevatron luminosity charts and the current status of Fermilab's accelerators and detectors (live!), please go to Fermilab Now.

Comments and Suggestions
What do you think about the Accelerator Updates? Please send comments and suggestions to: accelupdates@fnal.gov.