Summary notes of the fifty-first meeting of the LHC Commissioning Working Group

 

Tuesday July 15th, 14:00

CCC conference room 874/1-011

Persons present

 

Minutes of the Previous Meeting and Matters Arising

The minutes of the 48th meeting from 17 June had just been distributed prior to this meeting. There were no additional comments. Roger mentioned that Alick was in the process of finalizing the minutes of the 50th meeting.

Roger reported that numerous discussions had taken place about sending first beam into the machine. The aim now was to inject first beam in the middle of August. This schedule was being driven by AB safety concerns, which required the full machine to be ready. Safety issues for an earlier injection tests concerned LHCB in particular. Mike asked about the exact reason and whether the procedures could not be modified by Ghislain Roy to accommodate any safety concern.  Paul explained that if one departed from the official conditions, CERN first needed to perform a safety study, which involved a substantial checking and, especially, an additional approval by Paris. It could have been done if more time were available. Rudiger commented that the resulting delay was about 2-3 weeks, compared with an earlier plan from the preceding week.

Roger announced that this was the 51st and final meeting of the LHCCWG, which would in the future be combined with the LTC. The LHC commissioning team would however retain the Tuesday afternoon slot as a possible forum for discussion.

 

Status of Collimation System and Commissioning Plans Without and With Beam (Ralph)

Ralph started with a status overview of the collimation system. The collimator status was excellent. In March 08 the installation of the initial 7 TeV collimation system had been completed. As discussed in a previous meeting, a few collimators had to be delayed. The most important collimators missing are the two-beam collimators in IR2 and IR8. These suffered from a vacuum problem. As a consequence, presently, it was assumed that the beta* in IR2 and I8 should stay larger than 6 m, so that the triplets remain in the shadow of the arc apertures, a proposal which was accepted by various committees. The full phase-1 collimation system would be installed in the first shut down after the 2008 run.

 

Roger posed a question about the exact limit on beta* in IR8, and Oliver added that LHCb would wish to operate with the smallest possible beta* at low intensity. Ralph replied that Thomas Weiler was studying this question in detail, and the answer would be reported later. Oliver then pointed out that many flexible cables in the tunnel appeared to be made from rubber. He asked whether this was not a concern with regard to radiation.  Ralph assured him that, at the collimator locations, every component was fully metallic, including the flexible water cables. He pointed to a photo from his slides to prove this case.

 

Ralph highlighted that all collimator units are plug-in modules and can readily be exchanged. He presented a photo of a special absorber protecting the dogleg bend, which alone would gain a factor of 5 in performance. The next picture showed the tertiary collimators in front of a triplet – these collimators are protecting and cleaning at the same time.

 

The distribution of measured jaw flatness peaked around 30 micron. Most collimator jaws reached the specified 40 micron tolerance. The collimator jaws closest to the beam consist of fiber-reinforced graphite and have a minimum gap of about 0.3 mm. Tertiary collimators from tungsten exhibit a larger gap of 0.6-0.7 mm.

 

The collimator beam commissioning would be efficient only with a thorough and successful collimator hardware commissioning done beforehand. The collimator hardware commissioning procedure had been published and approved. An MTF system for the collimator hardware commissioning had been set up (EDMS document no. 882140). The collimator hardware commissioning was scheduled to end on 21 July.

 

A recent issue was the lifetime of the cage rail system supporting the jaw motion. Ralph presented two example units. The middle rail moves with half the speed. A creeping problem for such type of device had been known, but not advertised by industry. No problem had been seen in a prototype test over 32,000 cycles, but already the early series production had revealed an issue. Luckily, for unrelated production reasons, the cage design for later collimator jaws had been changed. Ralph summarized the pertinent measurements on series collimators, presenting a few examples. In some cases, the collimators might have blocked after less than 1000 cycles. Many tests had been done. The measured creeping for some rails with inox cages reached values up to 300 mm per km, corresponding to 17 micron per cycle (1 cycle = 60 mm), significantly higher than expected by the manufacturer. The reason for the unusually high creeping rate was not understood. As a countermeasure an anti-creeping tool “ACT” had quickly been developed. This tool proved to nicely work in the laboratory, showing an ability to extend the cage lifetime to up to 60,000 cycles in the lab. A retrofitting in the tunnel was presently ongoing for 28 collimators which were most badly affected.  Ralph underlined two reasons why this had to be done prior to the first beam. One had to two with the assembly, and the second reasons was the radiation level after operation that would make prolonged manual work at the collimators difficult. The intervention time required was 6 h per collimator. The total retrofitting for 28 collimators was manageable during two weeks. With the ACT tool in place all collimator would conform to the specified lifetime goal of more than 20 years (or more than 22,000 cycles). Nevertheless in view of the past trouble, Ralph recommended to minimize the number of cycles and to avoid unnecessary cycling of collimators.

 

The next item addressed by Ralph was the controls readiness and remote commissioning. The low, middle, and top level software were operational and could support the coherent motion of many collimators. He presented the results of remotely executing a synchronized ramp function of 7 collimators, and in particular the measured jaw position error, i.e. the difference between requested and measured position for each jaw of the ensemble.  

 

Concerning collimator interlocks, Ralph first reviewed the position interlocks based on comparing the jaw position and gap size of a collimators with limits determined from beam energy, beta function, and time. He distinguished between warning limits and dump limits. Further jaw movement is prevented when the dump limit is reached. An automatic verification procedure for the generation of interlocks and the requested system response had already been successfully tested for some collimators.

 

Frank asked if one would not better try to prevent jaw motion after reaching the warning limit, thereby avoiding a beam dump. Ralph replied if beam dumps induced by collimator position interlocks were to occur frequently one could consider such option.

 

Other types of interlock are collimator temperature interlocks, again with warning and dump thresholds, and BLM based interlocks. Ralph showed a photo of a collimator with pumping domes flanked by two BLMs on either side (ionization chambers and SEM). The BLM read-out rate would need to be at least 1 Hz. A table summarized the estimated settings of the damage interlock levels (slides 30 and 31), which should provide protection against melting of collimator components.

 

The plan for beam commissioning assumed that the collimators would be fully operational, that one could drive the collimators following any function, that one could set any warning and interlock level, that the jaw position interlock would have been commissioned without beam already, that the temperature interlock would ensure collimator safety, and that the other auxiliary systems would be operational, in particular the BLMs at 1 Hz read-out rate. The beam commissioning plan then just meant to decide what collimators to use when, and at which settings, and to perform the beam-based calibration of the collimators.

 

Ralph only lost a few words about the beam-based calibration, the scheme for which had been reported many times before. Special calibration runs would be needed for this, namely an estimated 6 shifts of about 8 hours each. In 2008 the calibration would be done manually. An automatic beam-based calibration should be available for 2009. The development was under discussion with the CO group.

 

Several collimator operational modes can be distinguished corresponding to an increasing level of accelerator control and monotonically rising higher intensity reach, namely mode 1: primary collimators and protection collimators only; mode 2: primary collimators, absorbers and protection collimators; mode 3: full 2008 system (5 instead of 11 secondary collimators in IR7 per beam); and mode 4: full phase-1 system, as installed for 2009.

 

Realistic assumptions are made for the efficiency, which due to various errors can be a factor of 10 below the ideal value (PhD Thesis of Chiara Bracco). Ralph presented tables summarizing the commissioning settings at various energies and modes, as well as the resulting performance (slides 35 and 36).

 

Oliver asked how one would measure the local cleaning inefficiency during commissioning. Ralph responded that he had already once presented this. The strategy was to relate BLM readings and quench limits. Oliver then asked if this would be done in early commissioning. Ralph confirmed that he would of course like to do this early, but this depended on the overall priorities.

 

Ralph S. asked whether the quoted orbit tolerances on slide 36 referred to certain individual collimators or e.g. to the relative alignment of primaries and secondaries. Ralph answered that these tolerances specified how much the orbit (and/or the beta beat) were allowed to change anywhere around the machine.

 

Now Ralph showed a scenario for closing the collimators with relaxed tolerances during early energy ramps: the tolerance for transient beta beating would increase from 10% at injection to 40% a top energy. Similarly, the tolerance on the collimator settings would loosen from 0.4 to 1.6 sigma. Slide 41 listed the preliminary TCT settings for 5 TeV as a function of beta*. Ralph emphasized that for operation beyond 156 bunches one would need to strongly work on the imperfections, especially on the orbit in the dispersion suppressors.

 

Excellent news was that for phase 2, recently a factor 30 potential gain in cleaning efficiency had been found.

 

Ralph next summarized the collimation plan on a single slide (43):

·        Pilot bunch - Start with primary collimators only and with protection elements; keep the collimators open during first ramps and close as we learn about the ramp; bring in additional collimators and test ramp functions & stability; end of ramp compatible with beta* = 11 m.

·        43 bunches and/or squeeze to beta* = 2 m - Use the full installed system, according to collimator hierarchy; close during ramp with optimized ramp settings (maximum tolerances); squeeze without closing collimators (end of ramp compatible with beta* = 2 m).

·        156 bunches and/or squeeze to beta* = 0.55 m - Close collimators to nominal settings and tight tolerances; squeeze with tight collimator settings; compatible with beta* = 0.55 m.

·        75 ns running (push intensity and luminosity) - Reduce imperfections (machine & collimators) and improve machine stability.

 

Frank asked for the meaning of “imperfections at collimators”. Ralph responded that this mainly referred to beam-based calibration. Roger asked when we would close the collimators in phase 3. Ralph replied that this closure would be needed below 2 m beta*; it could of course be tested earlier at larger beta*.

 

The milestones ahead in July and August comprised: installation of ACT tools, approval of all collimators for operation, moving all collimator together in dry runs, closing IR3 collimators for injection test, temperature interlocks checks, and preparation of functions.

 

Elliot asked whether one would need all BLMs at 1 Hz – or only a subset. Ralph anwered that only those near the collimators needed to be read at the 1 Hz rate. Mike reported that Marek had changed the rate of the relevant BLM readings (plus others) to 1 Hz. Stefano pointed out that the  1-Hz rate was not required for all integration times. Mike asked if Ralph would now hand over the collimators to operation. Ralph replied, yes, as soon as some outstanding issues were fixed.

 

In response to questions by Rudiger and Mike related to a future collimator move from IR7 to IR3, Ralph mentioned that a budget estimate had been produced, and that discussion was ongoing with Steve. He emphasized that a rapid decision was needed. Following a comment of Rudiger, Ralph clarified that the collimators would be moved during the first shut down.

 

Gianluigi asked when the interlocks could be checked and if this might possibly be earlier than August. Ralph responded that he first wanted to verify the sensors; where some issues had been found. The system could be ready for checks in the last week of July, but Ralph’s priority was to make sure that the collimators were fully operational and working. It was commented, by Rudiger or Gianluigi, that an interlock test on BIC by BIC basis could be done.

 

Ezio asked about the intensity limit for each collimation commissioning phase. Ralph clarified that the limits quoted always referred to the total intensity and assumed that all bunches had the same lifetime; the number of bunches indicated on slide 40 reflected the standard commissioning scenarios.

 

 

Results from Collimator Remote Commissioning (Stefano)

Stefano gave the last LHCCWG talk, on behalf of many people. His presentation covered the following subjects: collimator controls status, results of remote commissioning, beam commissioning in TI2, open issues and conclusions.

 

Starting with the status of the collimator controls, Stefano reviewed the scheme of the top-level controls architecture. The responsibility for the middle-level architecture had been moved between groups, and a few people had been nervous about this system, but Stefano now was happy to report the positive results.

 

Stefano showed the display of a single collimator control unit, which had been deployed since 2006. A new version including warning and dump levels was under construction with E. Veyrunes from OP. A fixed display tool (Nuno and Veyrunes) illustrated the status of each collimator in a symbolic form.

 

Oliver inquired about the meaning of white colored collimators. Stefano explained that the white color indicated a communication problem with middleware, while red meant an error.

 

Stefano next sketched the general idea of function-based collimator settings, and this  presented an example from the nominal ramp to 5 TeV.

 

Oliver asked whether one could select which collimators were displayed, which Stefano answered in the affirmative. This selection of collimators was indeed aided by the “Collimator TRIMapp” shown on the following slide 11. TRIMapp was a convenient tool for selecting sets of collimators, with the purpose to set all secondaries e.g.

 

Now commenting on the LSA integration, Stefano showed how collimators are accessed via “equip state”, which allows for checking statuses, and provides the “drive” functionality for settings and thresholds, software triggers, etc.

 

Behind the collimator control is a complicated collimator parameter space which determines the motion of collimators in units of sigma. The beam-based parameters are an input needed. An example was the change in optics values during a squeeze.

 

About 13,000 logging variables exist for the collimator, not including the BLMs. Ralph pointed out that the logging rate might not be 1 Hz but probably 1/2 Hz. Stefano alleviated such concerns, saying that one could still have a local logging on demand at higher frequencies. He would follow this up.

 

A special request had been made to also log collimator settings changes. This was implemented: every time a change in the trim occurs, both the settings and the threshold changes are recorded in the logging, for permanent storage. Slide 22 showed the collimator LSA database for illustration. EDMS document no. 906937 describes the hardware parameters for the 2008 LHC collimators.

 

The next major theme was the result of remote commissioning for the collimators, addressing function-driven motion; synchronized ramp functions; commissioning sequences for MP; EMC results; and preparation for beam operation.

 

Quoting the OP cold checkout meeting from November 2007, the cold checkout should be focused on performing global simultaneous system checks; verifying interfaces; and testing the management & validation of measurement data. Stefano stressed that the 400 degrees of freedom would translate to about 2000 limit functions for the collimators.

 

As an example of function-driven motion, the pictures on slide 28 overlaid the setting function with the requested function. The setting error was below the minimum step size of 5 micron. However, what mattered most was the real position of the jaw which was measured independently (and would be discussed later). Stefano showed “dummy” settings for parking, coarse, nominal and ramp, used for simulating the LHC cycle with a group of 7 collimators in dry runs. The maximum measured error in the collimator position amounted to about 30 micron.

 

Ralph commented that the collimators had been under vacuum but the positioning calibration had not yet been redone under vacuum. The error was expected to decrease further after re-calibration.

 

Stefano presented sequences of collimator jaw movements developed for MPS commissioning where individual jaw positions were moved across the warning and interlock thresholds in regular patterns. Interlocks needed to be checked both on the collimator corner position and on the gap size. Stefano. The test was done at 1 Hz. Later the post-mortem buffer with 100-Hz could be used if necessary. The machine-protection check takes 3 min. per collimator; all collimators can be tested together. Stefano pointed out that the BIC status changes should also be logged in parallel to verify the removal of the beam permit.

 

Rudiger remarked that something similar was being done for the PIC, namely it was tried to read its status, and to look for the user permit. He proposed to discuss this development with Stefano offline, as this tool, just being prepared, could prove very powerful and be deployed here too.

 

Now turning to the problems encountered, Stefano recalled a magnetic interference in TI2 of collimators first seen two years ago. Collimators were close to the powering cables, and as a result the LVDT readings were strongly affected by the powering of the main circuits, mirroring the LHC and CNGS cycles with amplitudes of 100 micron. As a countermeasure A. Masi had installed a magnetic shielding for the collimators. In 2007 the TI2 pulsing with a few magnets showed 80 micron fake collimator motion, while in 2008, with the shielding in place, the motion was reduced to about 10 micron, except for one bad sensor, which had been conspicuous in other regards as well. Stefano next showed results of EMC tests in the LHC itself. The typical noise found was below 10 micron. One bad example, TCCSG in IP6, exhibited a 250 micron fake motion, a problem that was not yet understood, but perhaps related to ventilation. EMC tests would continue in parallel with HWC.

 

Ralph suggested doing such tests also while moving all collimators together. Mike asked for the danger posed by the fake readings. Stefano replied that the danger was related to the fact that the affected signals were interlocked.

 

Another potential problem observed was that after a fast large-step motion with 2 mm/s, the LVDT signals showed a significant overshoot of about 0.5% followed by drifts over 100s of seconds. This problem was fixed by exchanging the electronic card. This problem was unlikely to be encountered in standard LHC operation since the collimators would be moved much more slowly.

 

Stefano continued with unabated energy to discuss the beam commissioning at TI2 with three TCDI collimators in mid-June. Single bunches with about 5e10 protons per bunch had been employed for these studies, which lasted 4-5 hours, and explored the beam-based set up of transfer-line collimators. Stefano remarked that the beam conditions had been excellent and intensity stable to within 10-20%. The beam-based set up had been accomplished by scanning one jaw at a time, and by fitting the BLM readings to the losses that were expected for a Gaussian distribution.

 

Oliver asked whether the distribution had to be Gaussian. Frank observed that the BLM reading at the intersection of the fitted curves corresponding to the jaws on either side was not at half height. Rudiger commented that the measurement had been done over many shots, and that a large error was to be expected.

 

Stefano presented results from a second type of beam-based measurement were a sliding gap of constant width had been moved across the beam. This second measurement also yielded the beam centre, in agreement with the result from the other method.

 

The parameters determined by beam-based measurements are fed into the LSA database, which also contains a trim history of the collimator settings. The protection settings computed from the beam-based procedure were successfully verified using orbit bumps generated by Verena.

 

The beam tests brought to light a new surprise, namely for one sensor a significant LVDT noise induced by the beam when the beam impacted on the collimator. The sensor readout started to jump, even to values outside the mechanical limits of the collimator. The effect disappeared after about one hour for no obvious reason. This was the very sensor that had also remained sensitive to TI2 pulsing. It will probably be replaced.

 

Items on which work was presently ongoing included timing (for the moment only software trigger; need to update system to dynamic payload; taking advantage of last FESA release); alarms (warnings and errors defined; the alarm property needs to be introduced in FESA in a few weeks); post-mortem (PM buffer will be filled with 100 Hz data); MCS (the first version from Verena is being tested); temperature (controls soon to be made available); BLM acquisition (tests limited by availability of BLM concentrator); and other movable devices (TCDQ, TDI and Roman Pots).

 

Mike asked whether post-mortem was triggered by timing or internally. Stefano replied that it was certainly triggered by internal faults; he could not comment about the timing.

 

The specifications on FESA interface to control other movable devices were under approval in EDMS. The middle-level software was copied from the collimator application. First results for TCDQ were shown, using the corresponding LSA application.

 

After some summarizing remarks, Stefano announced that the next milestone was to move all available collimators through a full nominal 5-TeV cycle.

 

Mike remarked that the results presented by Stefano were very impressive. Ralph S. commented that at present all settings were presumably computed for the nominal orbit & optics. He asked about checks that collimators were not driven into the beam. Ralph replied that the collimation set up relied on the reproducibility of orbit and optics, as well as (Rudiger added) on the feedback. Answering another question by Ralph S., Ralph confirmed that automated measurements were not foreseen. Stefano commented that the orbit feedback should provide a control at the 0.3 sigma level. Ralph explained that the EIC would need to make sure that the machine was set up correctly. Rudiger observed that if a beam were to move against a collimator the beam would be dumped. The danger was that the orbit moved somewhere else, not at a collimator. Such cases would need to be caught by the SIS interlock. Ralph S. inquired whether for injecting into an empty machine any rough check was planned. It was said that the EICs could monitor the BPMs at the collimators in such cases. Frank suggested an automatic warning mechanism to alert the CCC personnel. Elliot and Rudiger commented that some tasks could be handled by the sequencer.

 

Oliver asked what would happen if one jaw got stuck and one could not move that jaw anymore. Ralph responded that the collimation team could look at this problem within a few hours, run the COLLTRACK programme and make a decision.  He pointed out that only a few collimators were critical, and that even if one jaw was stuck the LHC could still work using the other, opposing jaw. For detailed predictions one would need to run SIXTRACK, requiring more time. A factor 10 speed could be gained by using the computing grid.

 

AOB

This was the 51st and last LHCCWG meeting. Roger thanked all working-group members for their significant efforts during the past 2.5 years.

 

 

 

 Reported by Frank