Accelerator Update for September 12, 2008 - September 15, 2008

Friday Sept 12

The day shift began with Operations monitoring the Tevatron (TeV) store 6413 (182.51E30) and the Recycler (RR) stash (92.4E10), with the Antiproton Source (Pbar) stack (41.5E10), with the Linac, Booster, or the Main Injector (MI) operating normally, with the MiniBooNE, NuMI, and the Meson MTest experiments taking beam, and with the Collider Detector at Fermilab (CDF) and DZero (D0) using the colliding beams.

At 8:08 AM, a Linac Klystron RF station (KRF4) tripped off. It reset.

A Booster expert reported at 9:38 AM that he was taking the East Anode Power Supply offline for about 20 minutes so he could isolate BRF7 for repairs.

The Neutron Therapy Facility reported at 9:56 AM that they had completed their treatment on the last patient for the day.

MI experts reported at 10:59 AM that they were tuning coalescing.

At 12:22 PM, LRF5 tripped off. It reset.

At 2:18 PM, the P1 beam permit tripped off due to a power supply’s ( HT702) output current moving out of tolerance. The NuMI permit also tripped off due to beam in the kicker gap.

Booster RF experts brought BRF7 back online at 3:20 PM.

At 3:37 PM, the TeV sector A1 wet engine tripped off. Cryo system experts at CHL promptly restarted it.

A Pbar expert reported at 3:59 PM that he had completed an AP2 line quadrupole magnet’s centering measurements.

Operations reported at 4:10 PM that the Recycler had stashed antiprotons five times during the day shift.

At 5:43 PM, LRF1 tripped off. It reset.

Operations disabled the beam to MTest at 6:05 PM.

At 9:42 PM, LRF4 tripped off. It reset.

Operations terminated store 6413 at 9:45 PM.

Operations put the TeV into shot setup at 9:58 PM.

Operations established store 6414 at 11:50 PM with an initial luminosity of 282.2E30.

At 11:54 PM, KRF1 tripped off. It reset.

Operations reported at 11:59 PM that the Recycler had stashed four times during the evening shift.

Saturday Sept 13

The midnight shift began with Operations monitoring store 6414 (257.74E30) and the stash (51.4E10), with the Pbar stacking (31.9E10), and with the MiniBooNE and NuMI experiments taking beam.

At 4:11 AM, Operations reported that they were receiving many high water alarms due to the heavy rain.

Operations established beam to MTest at 4:39 AM.

At 6:17 AM, the Pbar lithium lens tripped off on a ground fault indication. Operations called in experts. Operators reported that water did not seem to be the cause of the trip.

Operations loaded a NuMI only time line into the Time Line Generator (TLG) at 6:18 AM.

At 6:32 AM, BRF6 tripped off. It reset.

Operations reported at 8:01 AM that the Recycler had stashed five times during the midnight shift.

At 8:09 AM, BRF16 tripped off for the first of two times in 45 minutes. It reset.

Operations had the Pbar vault prepared for access at 8:46 AM.

At 8:55 AM, Operations reported water coming down the back wall in Linac near LRF4.

At 9:29 AM, an expert reported that the NWA service building in the Fixed Target area was flooded.

At 12:24 PM, a NuMI expert reported that water was pouring down a cable tray to the elevator landing in MI-65. Another expert was called in.

At 12:40 PM, a Booster vacuum pump (IPRF17) tripped off and caused BRF7 & 8 to also trip off. Everything reset.

At 1:06 PM, the TeV sector B4 electronics room air conditioner tripped off. Operations contacted the Duty Electrician.

At 1:18 PM, the TeV sector F4 air compressor tripped off. It reset.

At 2:25 PM, a Code One building alarm came in for NML. Operators investigated and reported smoke. Water had found its way into a transformer panel. FESS personnel isolated the building power. Experts investigated.

Operations reported that an RF expert had completed Pbar Delta study at 2:38 PM. An Accumulator power supply (A:QT) had tripped off. Operations contacted a power supply expert.

A Pbar expert asked operators at 2:59 PM to take a tour of the AP-10 tunnel to look for water leaks. Operators reported that the area was dry.

Operations prepared the Pbar rings for access at 3:04 PM.

At 3:57 PM, an expert reported that one of three switchboards for the main building power at NML had shorted and there was other water damage. FESS personnel installed portable sump pumps to pump out NML.

At 5:48 PM, operators reported that the TeV sector B3 service building was under water. Road and ground personnel built a sandbag dam around a transformer and switchgear.

Operations disabled the MTest beam at 6:15 PM.

At 6:51 PM, Operations reported that MIRF3 had tripped off three times. It reset, but operators left the station off after the last trip and informed an expert.

Operations reported at 8 PM that the Recycler had stashed once during the last 12 hours.

At 9:15 PM, the MI south anode power supply tripped off. It reset.

At 10:09 PM, the MiniBooNE beam permit tripped off. It reset.

At 10:56 PM, Operations turned off the Linac beam due to a vacuum pump that had tripped off for the Buncher. The pump reset and Linac resumed sending beam.

Sunday Sept 14

The midnight shift began with Operations monitoring store 6414 and the stash, with Pbar off and with no stack due to a lithium lens problem, and with the MiniBooNE and NuMI experiments taking beam.

At 12:17 AM, the MI-40 Upper VCB opened on an “oil level” indication. It would not reset. Operators investigated and reported an oil leak from the transformer. The fire department responded. Operations contacted a transformer expert.

At 12:49 AM, experts reported that the Accumulator magnet A1Q6 had faulted. They disconnected the magnet and shielded it from the dripping water.

At 1 AM, the MI VCBs opened and shut off the main bus power supplies. Operations called in high voltage experts.

A Pbar expert reported at 1:11 AM that the new lithium lens had been successfully pulsed.

Operations reported at 1:32 AM that they had rebooted MECAR after it had locked up during the MI power supply turn on.

At 3:07 AM, FESS reported that they had opened five fire hydrants to keep Casey’s pond water from overflowing.

At 3:21 AM, Operations reported Booster beam problems. Operators tuned.

At 3:56 AM, the MI power supply interlock loop tripped off. It reset.

Operations reestablished beam to NuMI at 4:14AM.

At 5:44 AM, Operations reported that QXR problems were stopping MI from sending beam to Switchyard and the MTest experiment. A QXR expert was called in.

At 6:16 AM, LRF5 tripped off for the first of three times in 26 minutes. It reset.

At 7:43 AM, a Booster vacuum pump (IP61) tripped off. It reset.

Operations established beam to the Accumulator around 8 AM.

A TeV expert began a crystal collimator study at 8:26 AM.

At 8:51 AM, Operations reported that the MI-20 LCW pump had tripped off earlier this morning.

Operations established stacking at 9:01 AM. A Pbar expert tuned.

At 9:12 AM, Operations reported that they were having trouble with the Pbar beam permit. Experts investigated.

The QXR expert reported at 9:13 AM that he had to adjust the tunes to get QXR back in line.

At 9:37 AM, LRF5 tripped off. It reset.

Operations reported at 9:56 AM the Pbar had resumed stack at full intensity.

At 10:08 AM, Operations reported that a Meson quadrupole magnet was showing a ground fault indication. Operators investigated.

At 11 AM, the TeV sector C4 air compressor tripped off. It reset.

Operations reported at 11:20 AM that the TeV studies had ended.

Operations terminated store 6414 at 11:31 AM.

Also at 11:31 AM, LRF5 tripped off for the first of three times in 14 minutes. It reset.

At 11:36 AM, KRF2 tripped off. It reset.

Operations prepared D0 and CDF for an access at 11:47 AM.

Operations prepared TeV sectors A-E for an access at 12:43 PM.

At 12:43 PM, a Pbar kicker (PBKICK) tripped off. An expert brought it back up.

At 2:25 PM, operators reported that there was much water in the TeV tunnel at B3. The breaker for the sump and the emergency sump were tripped. The Duty Electrician investigated. Also, the vacuum was getting bad at B3. A vacuum expert was called in.

At 2:47 PM, operators reported that B4 was also full of water flowing from B3. The humidity probably caused the breaker trip.

At 3:56 PM, Operations reported that they had lost Klystron RF stations 1-6 and LRF4. Operations contacted a Linac expert.

Operations put Pbar into standby at 3:59 PM.

A Linac expert reported at 4:50 PM that water skid #3 had tripped off. Operators reset the contactor and Linac resumed sending beam at 4:56 PM.

AT 5:27 PM, MiniBooNE tripped off. It reset.

At 5:44 PM, FESS reported that they would be cleaning a water strainer behind the Switchyard service building.

TeV came out of access at 5:47 PM.

Operations disabled the MTest beam at 6:01 PM.

At 6:31 PM, Operations reported that a 10-volt VFC card was out of tolerance at TeV sector C4.

At 7:02 PM, LRF5 tripped off. It reset.

Operators reported at 7:19 PM that Roads and Grounds personnel had completed their dam at B3 to hold back the rising water.

Operators reported at 7:34 PM that they had swapped out the VFC card at TeV sector C4.

Operations reported at 7:58 PM that the TeV turn on had already begun.

Operations reported at 8:11 PM that the Recycler had stashed four times during the last 12 hours.

Operations put the TeV through a dry squeeze at 9:12 PM.

At 9:36 PM, the MI south APS tripped off for the first of three times in six minutes. MIRF15 had also tripped off both times. Operations called in an RF expert.

A RF expert reported at 9:49 PM that there was something shorting out the MI south APS. Operations called in a power supply expert.

Operations put Pbar into standby at 9:50 PM.

At 9:56 PM, Operations reported that LRF5 had tripped off twice. It reset.

At 10:11 PM, KRF5 tripped off. It reset.

A RF expert reported at 10:49 PM that MIRF3 was causing the MI south APS trips due to a bad modulator. He began repairs.

At 11:32 PM, LRF4 tripped off. It reset.

Operations reported at 11:34 PM all the MIRF was operational. Beam to Pbar, NuMI, and MiniBooNE resumed two minutes alter.

Operations put the TeV through a wet squeeze at 11:50 PM.

Monday Sept 15

The midnight shift began with the TeV in a dry squeeze, with Operations monitoring the stash, with Pbar stacking, and with the MiniBooNE and NuMI experiments taking beam.

At 12:48 AM, Operations received a conductivity alarm for LRF3. Operations first contacted a Linac expert and then a LCW expert. The Operations called in two water technicians to change out the DI bottle for LRF3.

At 12:56 AM, BRF15 and 16 tripped off. They reset.

Operations reported at 1:31 AM that they had successfully completed an orbit smooth.

At 1:46 AM, KRF3 tripped off. It reset.

Also at 1:46 AM, Operations put the TeV into shot setup.

A Linac expert reported at 1:59 AM that the DI bottle could be removed safely.

Another Linac expert reported at 2:30 AM that the PLC controller for LRF3 was broken.

A water expert came in to investigate the conductivity problems. He reported at 3:25 AM that neither the skid nor the DI bottles were the problem. He checked and determined it was the conductivity probe. The sensor was turned off; it will get repaired in later in the morning.

At 3:27 AM, Operations reported that they had ended a reverse proton tune up prematurely that left the TeV in the wrong helix state. They put the TeV through another dry squeeze to correct this.

At 4:13 AM, a Linac quadrupole power supply (QPS230) tripped off. It reset.

Operations restarted the TeV shot setup at 4:22 AM.

At 4:40 AM, Operations reported that the Pbar autotune program was not working. They disabled the program and conducted all tuning by hand.

At 6:12 AM, LRF5 tripped off. It reset.

At 6:32 AM, the Klystron Vernier tripped off. It reset.

At 7:18 AM, the TeV sector B11 vertical separator sparked a couple of times and knocked out a huge chunk of luminosity.

Operations established store 6419 at 7:31 AM with an initial luminosity of 128.23E30.

Operations established beam to MTest at 7:51 AM.

Operations reported at 7:59 AM that the Recycler had stashed once during the last 12 hours.

The Plans for Monday

The plans for today are to stack, stash, and store.

Accelerator Update Archive

More Information
For Tevatron luminosity charts and the current status of Fermilab's accelerators and detectors (live!), please go to Fermilab Now.

Comments and Suggestions
What do you think about the Accelerator Updates? Please send comments and suggestions to: accelupdates@fnal.gov.