Accelerator Update for June 13, 2008 - June 16, 2008

Friday June 13

The day shift began with Operations monitoring Tevatron (TeV) store 6216 (78.4E30) and the Recycler (RR) stash (163.5E10), with the Antiproton Source stacking (26.7E10), with the Main Injector, Booster, and Linac operating normally, with the Collider Detector at Fermilab (CDF) and DZero (D0) using the colliding beams, and with the SciBooNE / MiniBooNE (BNB) and the Meson MTest experiments taking beam.

At 8:16 AM, a Linac Klystron RF station (KRF4) tripped off. It reset.

Operations put Pbar into standby mode at 8:38 AM.

Two MI experts began a coalescing study at 9:25 AM.

At 10:04 AM, LRF5 tripped off. It reset.

Operations reported at 10:17 AM that the AC technician would be turning off the Booster Low Level RF room AC unit. He was going to replace the air dryer.

At 10:21 AM, BRF7 tripped off. It reset.

At 10:40 AM, KRF3 tripped off. It reset.

At 11:41 AM, KRF4 tripped off. It reset.

The MI coalescing study ended at 11:49 AM.

At 1:17 PM, three Linac RF stations (LRF3, 4, and 5) tripped off. They all reset.

At 1:38 PM. LRF3 tripped off. It reset.

At 2:04 PM, the P1 and P2 transfer lines tripped off due to many power supplies tripping off. Everything reset. The cause of the trips was not found.

Operations reported at 2:19 PM that the Booster LLRF room AC unit was still not operational. The AC tech replaced its air dryer, but now it needs a new expansion valve.

Operations reported at 3:16 PM that Linac experts had turned off KRF4 for maintenance.

Operations loaded an energy conservation timeline into the Time Line Generator (TLG) at 3:17 PM.

Operations reported at 4 PM that the Recycler had stashed antiprotons four times during the day shift.

Linac experts reported at 4:04 PM that they had completed the cavity regulation work on KRF4.

Klystrons power the side coupled cavities pictured above

A Booster expert reported at 4:05 PM that the LLRF room temperature was cooling down.

At 4:39 PM, Operations reported that the Recycler had “frozen” after the last stash. Experts reported that the resistive wall monitor had stopped working. It took about an hour, but they managed to get the machine back to normal running.

Operations terminated store 6216 at 5:03 PM.

Operations put the TeV through a wet squeeze at 5:51 PM.

Operations disabled the MTest beam at 6:04 PM.

At 6:54 PM, LRF5 tripped off. It reset.

Operations put the TeV into a quiet time for CDF and D0 at 6:55 PM.

Cryo system experts reported at 7:12 PM that they had changed out the TeV sector E1 actuator (EVLH).

By 7:14 PM, both D0 and CDF were done with the quiet time and Operations put the TeV through a dry squeeze.

Operations put the TeV into shot setup at 7:53 PM.

Operations established store 6220 at 9:21 PM with an initial luminosity of 281.2E30.

At 10:20 PM, Operations reported problems with Booster beam positions. Operations contacted experts and then proceeded with an injection tune. This helped Booster’s efficiencies.

At 11:22 PM, Operations reported Recycler VSA problems.

At 11:30 PM, Operations reported that something caused a shift in the 400MeV area steering trim magnet power supplies. Operators investigated and found that something upstream in Linac had caused the shift.

Operations reported at midnight that the Recycler had stashed four times during the evening shift.

Saturday June 14

The midnight shift began with Operations monitoring store 6220 (173.37E30) and the stash (86.8E10), with Pbar stacking (36.9E10), and with BNB taking beam.

Operators reported at 12:08 AM that they had tuned on the Pbar beam.

At 1:58 AM, KRF6 tripped off on a coil comparator fault. It reset.

At 4:23 AM, the Linac Vernier tripped off on a LCW flow fault. It reset.

Operations established beam to MTest at 6:25 AM.

Operations reported at 7:58 AM that the Recycler had stashed six times during the midnight shift.

At 8:21 AM, Operations reported a low level alarm from the TeV sector A0 nitrogen Dewars. Operators investigated and discovered a leak. An expert arranged for a repair and for a nitrogen delivery. He also thought that there might be another leak and asked the Duty Mechanic to check CUB, but there was no leak there.

At 10:10 AM, Operations reported that Booster was having more beam position problems. Operators tuned.

At 10:12 AM, Operations reported losing about 10E10 antiprotons from the Recycler stash. Operations contacted experts. Operators investigated and believed the problem was due to a time line event ($2B) relearning in.

At 11:49 AM, a MiniBooNE expert reported that their detector temperatures were rising. The AC technician investigated and fixed the problem.

Operations terminated store 6220 at 12:19 PM.

At 12:21 PM, Operations reported that the temperature was climbing for the TeV sector A3 electronics room. The air conditioning technician reported that the AC compressor had failed. There already was a portable AC unit in the room. The AC tech lowered its set point.

At 12:26 PM, KRF6 tripped off. It reset.

Operations put the TeV into shot setup at 12:31 PM.

Operations established store 6221 at 2:23 PM with an initial luminosity of 280.9E30.

At 4:12 PM, KRF5 tripped off. It reset.

At 4:35 PM, LRF4 tripped off and though operators did everything they could, the station would not reset. Operators contacted an expert.

Operations disabled the MTest beam at 6:13 PM.

A Linac expert reported at 6:49 PM that he had reset LRF4.

Operations reported at 8 PM that the Recycler had stashed five times during the last 12 hours.

Sunday June 15

The midnight shift began with Operations monitoring store 6221 and the stash, with Pbar stacking, and with BNB taking beam.

Operations established beam to MTest at 6:10 AM.

At 7:19 AM, Operations reported that the MI-20 VCB had opened and would not reset. Operations contacted an expert.

At 7:22 AM, Operations reported a TeV sector D48 separator spark. Operators suspected that this was due to the storm passing over.

At 7:27 AM, after a multitude of lightning flashes, store 6221 quenched at sector B4. Also, the B4 cold compressor tripped off. It reset.

At 7:31 AM, the MI power supply interlock loop tripped off. The Recycler emittances grew and the Recycler lost about 3E10 of antiprotons. A Recycler expert reported that the stochastic cooling had started heating the beam. Operators turned off the transverse cooling system and emittances got better, but the Recycle had lost more beam, about 20E10 in total.

Operations reported at 8:04 AM that the Recycler had stashed eight times during the last 12 hours, but also lost about 20E10 of antiprotons.

Operations reported at 8:11 AM that a power supply expert was investigating the MI-20 VCB problem.

At 8:15 AM, Operations reported that the TeV sectors A-E electrical safety system suffered ground faults and the D0 and B0 low beta magnets quenched. Operations contacted an interlock expert who suggested operators try a master reset. This reset the system.

At 8:24 AM, Operations reported a problem with the TeV sector B4 separators. Operators reported that there was a problem with the separator control cards (C465). They changed out the cards.

At 8:41 AM, Operations reported that certain bulk power supplies had tripped off during the electrical safety system trip and they would not reset. An interlock expert was called in.

At 9:09 AM, KRF5 tripped off. It reset.

Operations prepared CDF and D0 for access at 9:36 AM.

Operators reported at 9:43 AM that they had recovered the Recycler stochastic cooling system. Emittances dropped back to normal and life times were improving.

Pbar resumed stacking at 10:09 AM.

At 10:12 AM, Pbar stopped stacking. Operations reported that a Pbar Debuncher kicker (D:EKIK) had tripped off and would not reset. Operations called in experts.

Operations put Pbar into standby mode at 10:41 AM.

Operations established beam to MTest at 11:58 AM.

D0 and CDF were out of access by 12:19 PM.

A kicker expert reported at 12:50 PM that D:EKIK was back on line. Pbar resumed stacking.

At 1:28 PM, Operations reported problems with the TeV sectors B48 and C17 separators. Operators investigated and discovered that the B48 separators were in run mode and conditioning mode at the same time. Operations contacted experts.

At 1:32 PM, Operations reported that the Meson SWICs and SEM were not working and they would not reset; they could not see the beam. Operations called in experts.

At 4:24 PM, a Meson expert reported that the SWIC and SEM problems were due to a special T-Clock repeater card. Another expert was called in.

Operations put the TeV through a QBS ramp at 6:36 PM.

At 6:57 PM, Operations reported more problems with the TeV sector B48 separator. Operators replaced the control card; an expert reported that the problem was with a vertical separator current limit reference and replacing the card had fixed the problem.

At 7:10 PM, the BNB horn tripped off. It reset.

At 7:15 PM, Operations reported that the lightning strikes this morning had caused four separator spark counters to fail. An expert came in and reported that they were broken and he could not find any spares. Operations bypassed the counters.

A controls expert repaired the Meson SEM problem at 7:41 PM.

Operations reported at 8 PM that the Recycler had stashed once during the last 12 hours.

At 8:10 PM, KRF7 tripped off. It reset.

At 8:15 PM, a controls expert, who was here and heard about the separator problem, replaced a C185 CAMAC card, which enabled the separator spark counters to resume counting.

Operations disabled the MTest beam at 8:28 PM.

Operations put the TeV through a dry squeeze at 9:03 PM.

Operations put the TeV through a wet squeeze at 9:35 PM.

Operations put the TeV through another wet squeeze at 10:03 PM.

At 10:12 PM, LRF1 tripped off. It reset.

Operations put the TeV through another wet squeeze at 10:45 PM.

Operations put the TeV into shot setup at 11:07 PM.

Monday June 16

The midnight shift began with the TeV in shot setup, with Operations monitoring the stash, with Pbar stacking, and with BNB taking beam.

At 12:15 AM, Operations reported that the shot setup was abandoned. Too few protons and antiprotons had made it to the TeV and those that did, were not well coalesced.

Operations put the TeV through a dry squeeze at 12:20 AM.

Operations put the TeV into shot setup at 1:02 AM.

Operations established store 6226 at 2:31 AM with an initial luminosity of 274.31E30.

At 3:36 AM, the Linac Vernier tripped off. It reset.

At 4:08 AM, LRF2 tripped off. It reset.

At 5:53 AM, MI had a watchdog trip. It took several tries, but everything eventually reset.

Operations established beam to MTest at 6:22 AM.

At 7:30 AM, a MI-8 vacuum pump (IP827) tripped off. It reset.

Operations reported at 7:59 AM that the Recycler had stashed five times during the last 12 hours.

The Plans for Monday

The plans for today are to stack, stash, and store. The Run Coordinator will allow experts to make up water in the TeV and MI LCW systems.

Accelerator Update Archive

More Information
For Tevatron luminosity charts and the current status of Fermilab's accelerators and detectors (live!), please go to Fermilab Now.

Comments and Suggestions
What do you think about the Accelerator Updates? Please send comments and suggestions to: accelupdates@fnal.gov.