Operation Summary for July 15, 2011 - July 18, 2011
Friday July 15
The day shift began with Operations monitoring the Tevatron (TEV) store 8894 (223.19E30) and the Recycler stash (63.6E10), with the Antiproton Source (Pbar) stacking (25.9E10), with the Main Injector (MI), Linac, and Booster operating normally, with the MiniBooNE experiment taking beam, and with the Collider Detector at Fermilab (CDF) and D-Zero (D0) using the colliding beams.
During the day shift, the Neutron Therapy Facility (NTF) began treating patients at 8:14 AM. At 8:31 AM, MiniBooNE personnel asked to have their beam turned off for 1.5 hours so that NuMI personnel could remove equipment stored in the MI-12 service building. Operations sent about 1.5 hours of beam to MTA starting at 12:10 PM. MI personnel began a coalescing study at 12:19 PM. Operators reported at 2:36 PM that a beam position problem took down the beam permit at MI-30; MI personnel investigated and thought that a power glitch might have been the cause. Operators rebooted the VME. NTF finished treating patients at 2:23 PM. Operators turned off the Linac beam for about a half hour while Linac technicians changed the Debuncher LCW pump at 2:56 PM.
During the evening shift, The MI coalescing studies ended at 5:05 PM. The H- Source tripped off when the I- Source sparked; everything reset. Operations terminated store 8894 at 7:09 PM. Operators prepared D0 for a one-hour access at 7:28 PM. Operators reported at 10:38 PM that the TEV locked up at 90 GeV; they called a Controls person who suggested operators change out a controls card, which fixed the problem. Operators then put the TEV through three HEP ramps, and at 11:45 through a dry squeeze. Operations put the TEV into shot setup at 12:50 PM.
The midnight shift began with TEV in shot setup, with Operations monitoring the stash (397.3E10), with Pbar stacking (59.2E10), and with MiniBooNE taking beam.
During the midnight shift, after a lot of controls problems, Operations established store 8896 at 12:41 AM with an average initial luminosity (AIL) of 399.63E30. Operators reported short downtimes due to MI beam permit trips on MECAR, a Linac RF station (LRF3), and a Linac Klystron RF station (KRF1).
During the day shift, a Water Works tech started the initial preparations for the CUB column #1 regeneration. Operations terminated store 8896 at 12:34 PM. Operators prepared D0 for 45-minute access at 12:34 PM. Operations put the TEV into shot setup at 3:20 PM.
During the evening shift, at 4:22 PM, operators reported a problem with the Pelletron. Operations established store 8899 at 4:22 PM with an AIL of 342.59E30. Operations reported at 5:32 PM that the TEV sector F4 electronics room temperature came into alarm; FESS personnel placed a spot cooler in the room.
Sunday July 17
The midnight shift began with Operations monitoring store 8899 and the stash, with Pbar stacking, and with MiniBooNE taking beam.
During the midnight shift, operators conducted a MI coalescing check at 2 AM. Operations terminated store 8899 at 5:51 AM and put the TEV into shot setup. At 6:22 AM, operators reported that the Pelletron tripped off many times during shot setup; operators notified Electron Cooling personnel. Operations established store 8900 at 6:29 AM with an AIL of 380.85E30. Operators reported short downtimes due to the Linac Klystron Buncher, LRF1 and 3, and due to a Booster beam detector.
During the day shift, at 10:16 AM, MIRF5 tripped off and wouldn’t reset; operators bypassed the station. At 10:53 AM, BRF16 tripped off many times over a 20-minute period; it reset. Operators reported at 12:33 PM that the temperature for the MI-12 service building was climbing fast and at 12:44 PM they were forced to turn off beam to MiniBooNE. FESS personnel brought the east side air conditioner (AC) online for MI-12 at 1:31 PM.
During the evening shift, operators rebooted the MI-10 BPM VME at 5:24 PM. FESS personnel added a portable AC unit to MI-12 at 5:48 PM. Operators reported that the MI-10 electronics room temperature alarmed; FESS personnel reported that both AC units in the room were off. At 6:45 PM, Water Works personnel went to CUB to put column #1 into recirculation mode, but found a three-inch pipe above the housing leaking; they valved the pipe off and will repair it tomorrow. FESS personnel cleaned the coils for one of the MI-10 AC units and brought it online at 8:25 PM; the room temperature began to drop. The MI-12 service building temperatures dropped down to 90˚ F; Operations resumed sending beam to MiniBooNE. At 11:05 PM, the MI-12 service building temperatures went back on the rise; operators notified FESS personnel. Operations terminated store 8900 at 11:20 PM and put the TEV into shot setup. At 11:56 PM, operators reported problems with the MI BPMs on reverse injections of protons into the TEV; this held off shot setup.
Monday July 18
The midnight shift began with shot setup on hold due to a MI/TEV reverse injection problem, Operations monitoring the stash, with Pbar stacking, and with MiniBooNE taking beam.
During the midnight shift, operators called in a TEV person at 12:38 AM to investigate the TEV injection problem. Operators reported TEV beam position problems at D0, D1, D2, and E4. At 3:59 AM, all the MI BLMs stopped responding; operators rebooted the system. At 4:05 AM, all the TEV BPMs stopped responding; a general reboot did not help. Operators rebooted each system individually. Operators reported at 6:50 AM that the MI-12 building temperatures began to rise; operators talked with a MiniBooNE person and then disabled beam to MiniBooNE at 6:57 AM. Operators reported at 7:25 AM that the TEV had been aborting on reverse proton injections through out the shift.
The Plans for Monday
The plans for today were to get the TEV reverse proton injection problem fixed, but a TEV person reported that they had found and fixed the problem. The Run Coordinator wants to establish a store as soon as possible and then try to survive the high temperatures.
Comments and Suggestions
|last modified 07/18/2011 email Fermilab|
|Security, Privacy, Legal|