Summary notes of the fifty-first meeting of the
LHC Commissioning Working Group
Tuesday July 15th, 14:00
CCC conference room 874/1-011
Minutes of the Previous Meeting and Matters
Arising
The minutes of the 48th
meeting from 17 June had just been distributed prior to this meeting. There
were no additional comments. Roger mentioned that Alick was in the process of
finalizing the minutes of the 50th meeting.
Roger reported that numerous
discussions had taken place about sending first beam into the machine. The aim
now was to inject first beam in the middle of August. This schedule was being driven
by AB safety concerns, which required the full machine to be ready. Safety
issues for an earlier injection tests concerned LHCB in particular. Mike asked
about the exact reason and whether the procedures could not be modified by
Ghislain Roy to accommodate any safety concern. Paul explained that if one departed from the
official conditions, CERN first needed to perform a safety study, which
involved a substantial checking and, especially, an additional approval by Paris.
It could have been done if more time were available. Rudiger commented that the
resulting delay was about 2-3 weeks, compared with an earlier plan from the
preceding week.
Roger announced that this was the 51st
and final meeting of the LHCCWG, which would in the future be combined with the
LTC. The LHC commissioning team would however retain the Tuesday afternoon slot
as a possible forum for discussion.
Status of Collimation System and Commissioning
Plans Without and With Beam (Ralph)
Ralph started with a status overview of the collimation system.
The collimator status was excellent. In March 08 the installation of the
initial 7 TeV collimation system had been completed. As discussed in a previous
meeting, a few collimators had to be delayed. The most important collimators
missing are the two-beam collimators in IR2 and IR8. These suffered from a
vacuum problem. As a consequence, presently, it was assumed that the beta* in
IR2 and I8 should stay larger than 6 m, so that the triplets remain in the
shadow of the arc apertures, a proposal which was accepted by various committees.
The full phase-1 collimation system would be installed in the first shut down
after the 2008 run.
Roger posed a question about the exact
limit on beta* in IR8, and Oliver
added that LHCb would wish to operate with the smallest possible beta* at low
intensity. Ralph replied that Thomas Weiler was studying this question in
detail, and the answer would be reported later. Oliver then pointed out that
many flexible cables in the tunnel appeared to be made from rubber. He asked
whether this was not a concern with regard to radiation. Ralph assured him that, at the collimator
locations, every component was fully metallic, including the flexible water
cables. He pointed to a photo from his slides to prove this case.
Ralph highlighted that all collimator
units are plug-in modules and can readily be exchanged. He presented a photo of
a special absorber protecting the dogleg bend, which alone would gain a factor
of 5 in performance. The next picture showed the tertiary collimators in front
of a triplet – these collimators are protecting and cleaning at the same time.
The distribution of measured jaw flatness peaked around 30 micron.
Most collimator jaws reached the specified 40 micron tolerance. The collimator
jaws closest to the beam consist of fiber-reinforced graphite and have a
minimum gap of about 0.3 mm. Tertiary collimators from tungsten exhibit a
larger gap of 0.6-0.7 mm.
The collimator beam commissioning would
be efficient only with a thorough and successful collimator hardware commissioning done beforehand. The collimator hardware
commissioning procedure had been published and approved. An MTF system for the
collimator hardware commissioning had been set up (EDMS
document no. 882140). The collimator hardware commissioning was scheduled to
end on 21 July.
A recent issue was the lifetime of
the cage rail system supporting the jaw motion. Ralph presented two example
units. The middle rail moves with half the speed. A creeping problem for such
type of device had been known, but not advertised by industry. No problem had
been seen in a prototype test over 32,000 cycles, but already the early series
production had revealed an issue. Luckily, for unrelated production reasons,
the cage design for later collimator jaws had been changed. Ralph summarized
the pertinent measurements on series collimators, presenting a few examples. In
some cases, the collimators might have blocked after less than 1000 cycles.
Many tests had been done. The measured creeping for some rails with inox cages reached
values up to 300 mm per km, corresponding to 17 micron per cycle (1 cycle = 60
mm), significantly higher than expected by the manufacturer. The reason for the
unusually high creeping rate was not understood. As a countermeasure an
anti-creeping tool “ACT” had quickly been developed. This tool proved to nicely
work in the laboratory, showing an ability to extend the cage lifetime to up to
60,000 cycles in the lab. A retrofitting in the tunnel was presently ongoing
for 28 collimators which were most badly affected. Ralph underlined two reasons why this had to be
done prior to the first beam. One had to two with the assembly, and the second
reasons was the radiation level after operation that would make prolonged
manual work at the collimators difficult. The intervention time required was 6
h per collimator. The total retrofitting for 28 collimators was manageable during
two weeks. With the ACT tool in place all collimator would conform to the
specified lifetime goal of more than 20 years (or more than 22,000 cycles).
Nevertheless in view of the past trouble, Ralph recommended to minimize the
number of cycles and to avoid unnecessary cycling of collimators.
The next item addressed by Ralph was
the controls readiness and remote commissioning. The low, middle, and top level
software were operational and could support the coherent motion of many
collimators. He presented the results of remotely executing a synchronized ramp
function of 7 collimators, and in particular the measured jaw position error,
i.e. the difference between requested and measured position for each jaw of the
ensemble.
Concerning collimator interlocks, Ralph first reviewed the position interlocks based on comparing
the jaw position and gap size of a collimators with limits determined from beam
energy, beta function, and time. He distinguished between warning limits and
dump limits. Further jaw movement is prevented when the dump limit is reached. An
automatic verification procedure for the generation of interlocks and the
requested system response had already been successfully tested for some
collimators.
Frank asked if one would not better
try to prevent jaw motion after reaching the warning limit, thereby avoiding a
beam dump. Ralph replied if beam dumps induced by collimator position
interlocks were to occur frequently one could consider such option.
Other types of interlock are collimator temperature interlocks, again
with warning and dump thresholds, and BLM
based interlocks. Ralph showed a photo of a collimator with pumping domes
flanked by two BLMs on either side (ionization chambers and SEM). The BLM
read-out rate would need to be at least 1 Hz. A table summarized the estimated
settings of the damage interlock levels (slides 30 and 31), which should
provide protection against melting of collimator components.
The plan for beam commissioning assumed that the collimators would be fully
operational, that one could drive the collimators following any function, that
one could set any warning and interlock level, that the jaw position interlock
would have been commissioned without beam already, that the temperature interlock
would ensure collimator safety, and that the other auxiliary systems would be
operational, in particular the BLMs at 1 Hz read-out rate. The beam commissioning plan then just meant to decide what
collimators to use when, and at which settings, and to perform the beam-based
calibration of the collimators.
Ralph only lost a few words about
the beam-based calibration, the
scheme for which had been reported many times before. Special calibration runs
would be needed for this, namely an estimated 6 shifts of about 8 hours each. In
2008 the calibration would be done manually. An automatic beam-based
calibration should be available for 2009. The development was under discussion
with the CO group.
Several collimator operational modes can be distinguished corresponding to
an increasing level of accelerator control and monotonically rising higher
intensity reach, namely mode 1: primary collimators and protection collimators
only; mode 2: primary collimators, absorbers and protection collimators; mode
3: full 2008 system (5 instead of 11 secondary collimators in IR7 per beam);
and mode 4: full phase-1 system, as installed for 2009.
Realistic assumptions are made for the
efficiency, which due to various errors can be a factor of 10 below the ideal
value (PhD Thesis of Chiara Bracco). Ralph presented tables summarizing the
commissioning settings at various energies and modes, as well as the resulting
performance (slides 35 and 36).
Oliver asked how one would measure the
local cleaning inefficiency during
commissioning. Ralph responded that he had already once presented this. The
strategy was to relate BLM readings and quench limits. Oliver then asked if
this would be done in early commissioning. Ralph confirmed that he would of
course like to do this early, but this depended on the overall priorities.
Ralph S. asked whether the quoted orbit tolerances on slide 36 referred
to certain individual collimators or e.g. to the relative alignment of
primaries and secondaries. Ralph answered that these tolerances specified how
much the orbit (and/or the beta beat) were allowed to change anywhere around
the machine.
Now Ralph showed a scenario for closing
the collimators with relaxed tolerances
during early energy ramps: the tolerance for transient beta beating would
increase from 10% at injection to 40% a top energy. Similarly, the tolerance on
the collimator settings would loosen from 0.4 to 1.6 sigma.
Slide 41 listed the preliminary TCT settings for 5 TeV as a function of beta*. Ralph
emphasized that for operation beyond 156 bunches one would need to strongly work
on the imperfections, especially on the orbit in the dispersion suppressors.
Excellent news was that for phase 2,
recently a factor 30 potential gain in cleaning efficiency had been found.
Ralph next summarized the collimation plan on a single slide (43):
·
Pilot bunch - Start
with primary collimators only and with protection elements; keep the
collimators open during first ramps and close as we learn about the ramp; bring
in additional collimators and test ramp functions & stability; end of ramp
compatible with beta* = 11 m.
·
43 bunches and/or squeeze to beta* = 2 m - Use the full installed system, according to collimator
hierarchy; close during ramp with
optimized ramp settings (maximum tolerances); squeeze without closing
collimators (end of ramp compatible with beta* = 2 m).
·
156 bunches and/or squeeze to beta* = 0.55 m - Close collimators to nominal settings and tight
tolerances; squeeze with tight collimator settings; compatible with beta* =
0.55 m.
·
75 ns running (push intensity and luminosity) - Reduce imperfections (machine & collimators) and improve
machine stability.
Frank asked for the meaning of “imperfections
at collimators”. Ralph responded that this mainly referred to beam-based
calibration. Roger asked when we would close the collimators in phase 3. Ralph
replied that this closure would be needed below 2 m beta*; it could of course be
tested earlier at larger beta*.
The milestones ahead in July and
August comprised: installation of ACT tools, approval of all collimators
for operation, moving all collimator together in dry runs, closing IR3
collimators for injection test, temperature interlocks checks, and preparation
of functions.
Elliot asked whether one would need
all BLMs at 1 Hz – or only a subset.
Ralph anwered that only those near the collimators needed to
be read at the 1 Hz rate. Mike reported that Marek had changed the rate
of the relevant BLM readings (plus others) to 1 Hz. Stefano pointed out that the 1-Hz rate was not
required for all integration times. Mike asked if Ralph would now hand over the
collimators to operation. Ralph replied, yes, as soon as some outstanding
issues were fixed.
In response to questions by Rudiger
and Mike related to a future collimator
move from IR7 to IR3, Ralph mentioned that a budget estimate had been
produced, and that discussion was ongoing with Steve. He emphasized that a
rapid decision was needed. Following a comment of Rudiger, Ralph clarified that
the collimators would be moved during the first shut down.
Gianluigi asked when the interlocks
could be checked and if this might possibly be earlier than August. Ralph
responded that he first wanted to verify the sensors; where some issues had
been found. The system could be ready for checks in the last week of July, but
Ralph’s priority was to make sure that the collimators were fully operational
and working. It was commented, by Rudiger or Gianluigi, that
an interlock test on BIC by BIC basis could be done.
Ezio asked about the intensity limit
for each collimation commissioning phase. Ralph clarified that the limits quoted
always referred to the total intensity and assumed that all bunches had the
same lifetime; the number of bunches indicated on slide 40 reflected the
standard commissioning scenarios.
Results from
Collimator Remote Commissioning (Stefano)
Stefano gave the last LHCCWG talk,
on behalf of many people. His presentation covered the following subjects: collimator
controls status, results of remote commissioning, beam commissioning in TI2,
open issues and conclusions.
Starting with the status of the collimator controls,
Stefano reviewed the scheme of the top-level controls architecture. The
responsibility for the middle-level architecture had been moved between groups,
and a few people had been nervous about this system, but Stefano now was happy
to report the positive results.
Stefano showed the display of a single
collimator control unit, which had
been deployed since 2006. A new version including warning and dump levels was
under construction with E. Veyrunes from OP. A fixed display tool (Nuno and
Veyrunes) illustrated the status of each collimator in a symbolic form.
Oliver inquired about the meaning of
white colored collimators. Stefano explained that the white color indicated a communication
problem with middleware, while red meant an error.
Stefano next sketched the general
idea of function-based collimator settings, and this presented an example from the nominal
ramp to 5 TeV.
Oliver asked whether one could select
which collimators were displayed, which Stefano answered in the affirmative.
This selection of collimators was indeed aided by the “Collimator TRIMapp”
shown on the following slide 11. TRIMapp was a convenient tool for selecting
sets of collimators, with the purpose to set all secondaries e.g.
Now commenting on the LSA integration, Stefano showed how
collimators are accessed via “equip state”, which allows for checking statuses,
and provides the “drive” functionality for settings and thresholds, software
triggers, etc.
Behind the collimator control is a
complicated collimator parameter space which determines the motion of collimators
in units of sigma. The beam-based parameters are an input needed. An example
was the change in optics values during a squeeze.
About 13,000 logging variables exist
for the collimator, not including the BLMs. Ralph pointed out that the logging
rate might not be 1 Hz but probably 1/2 Hz. Stefano alleviated such concerns,
saying that one could still have a local logging on demand at higher
frequencies. He would follow this up.
A special request had been made to
also log collimator settings changes. This was implemented: every time a change
in the trim occurs, both the settings and the threshold changes are recorded in
the logging, for permanent storage. Slide 22 showed the collimator LSA database
for illustration. EDMS document
no. 906937 describes the hardware parameters for the 2008 LHC collimators.
The next major theme was the result of remote commissioning for the
collimators, addressing function-driven motion; synchronized ramp functions; commissioning
sequences for MP; EMC results; and preparation for beam operation.
Quoting the OP cold checkout meeting
from November 2007, the cold checkout
should be focused on performing global simultaneous system checks; verifying
interfaces; and testing the management & validation of measurement data.
Stefano stressed that the 400 degrees of freedom would translate to about 2000 limit
functions for the collimators.
As an example of function-driven
motion, the pictures on slide 28 overlaid the setting function with the
requested function. The setting error was below the minimum step size of 5
micron. However, what mattered most was the real position of the jaw which was
measured independently (and would be discussed later). Stefano showed “dummy”
settings for parking, coarse, nominal and ramp, used for simulating the LHC
cycle with a group of 7 collimators in dry runs. The maximum measured error in
the collimator position amounted to about 30 micron.
Ralph commented that the collimators
had been under vacuum but the positioning calibration had not yet been redone
under vacuum. The error was expected to decrease further after re-calibration.
Stefano presented sequences of
collimator jaw movements developed for MPS commissioning where individual jaw
positions were moved across the warning and interlock thresholds in regular
patterns. Interlocks needed to be checked both on the collimator corner position
and on the gap size. Stefano. The test was done at 1
Hz. Later the post-mortem buffer with 100-Hz could be used if necessary. The
machine-protection check takes 3 min. per collimator; all collimators can be
tested together. Stefano pointed out that the BIC status changes should also be
logged in parallel to verify the removal of the beam permit.
Rudiger remarked that something
similar was being done for the PIC, namely it was tried to read its status, and
to look for the user permit. He proposed to discuss this development with
Stefano offline, as this tool, just being prepared, could prove very powerful
and be deployed here too.
Now turning to the problems encountered, Stefano recalled
a magnetic interference in TI2 of
collimators first seen two years ago. Collimators were close to the powering
cables, and as a result the LVDT readings were strongly affected by the
powering of the main circuits, mirroring the LHC and CNGS cycles with
amplitudes of 100 micron. As a countermeasure A. Masi had installed a magnetic
shielding for the collimators. In 2007 the TI2 pulsing with a few magnets
showed 80 micron fake collimator motion, while in 2008, with the shielding in
place, the motion was reduced to about 10 micron, except for one bad sensor,
which had been conspicuous in other regards as well. Stefano next showed results
of EMC tests in the LHC itself. The
typical noise found was below 10 micron. One bad example, TCCSG in IP6,
exhibited a 250 micron fake motion, a problem that was not yet understood, but
perhaps related to ventilation. EMC tests would continue in parallel with HWC.
Ralph suggested doing such tests
also while moving all collimators together. Mike asked for the danger posed by
the fake readings. Stefano replied that the danger was related to the fact that
the affected signals were interlocked.
Another potential problem observed
was that after a fast large-step motion with 2 mm/s, the LVDT signals showed a
significant overshoot of about 0.5% followed by drifts over 100s of seconds. This
problem was fixed by exchanging the electronic card. This problem was unlikely
to be encountered in standard LHC operation since the collimators would be
moved much more slowly.
Stefano continued with unabated
energy to discuss the beam commissioning at TI2 with three TCDI collimators in
mid-June. Single bunches with about 5e10 protons per bunch had been employed
for these studies, which lasted 4-5 hours, and explored the beam-based set up of transfer-line
collimators. Stefano remarked that the beam conditions had been excellent
and intensity stable to within 10-20%. The beam-based set up had been
accomplished by scanning one jaw at a time, and by fitting the BLM readings to
the losses that were expected for a Gaussian distribution.
Oliver asked whether the
distribution had to be Gaussian. Frank observed that the BLM reading at the intersection
of the fitted curves corresponding to the jaws on either side was not at half
height. Rudiger commented that the measurement had been done over many shots, and
that a large error was to be expected.
Stefano presented results from a second type of beam-based measurement
were a sliding gap of constant width had been moved across the beam. This
second measurement also yielded the beam centre, in agreement with the result
from the other method.
The parameters determined by
beam-based measurements are fed into the LSA database, which also contains a trim
history of the collimator settings. The protection settings computed from the
beam-based procedure were successfully verified using orbit bumps generated by
Verena.
The beam tests brought to light a
new surprise, namely for one sensor a significant LVDT noise induced by the
beam when the beam impacted on the collimator. The sensor readout started to
jump, even to values outside the mechanical limits of the collimator. The
effect disappeared after about one hour for no obvious reason. This was the
very sensor that had also remained sensitive to TI2 pulsing. It will probably
be replaced.
Items on which work was presently
ongoing included timing (for the moment only software trigger; need to update
system to dynamic payload; taking advantage of last FESA release); alarms
(warnings and errors defined; the alarm property needs to be introduced in FESA
in a few weeks); post-mortem (PM buffer will be filled with 100 Hz data); MCS
(the first version from Verena is being tested); temperature (controls soon to
be made available); BLM acquisition (tests limited by availability of BLM
concentrator); and other movable devices (TCDQ, TDI and Roman Pots).
Mike asked whether post-mortem was
triggered by timing or internally. Stefano replied that it was certainly
triggered by internal faults; he could not comment about the timing.
The specifications on FESA interface
to control other movable devices were under approval in EDMS. The middle-level
software was copied from the collimator application. First results for TCDQ
were shown, using the corresponding LSA application.
After some summarizing remarks,
Stefano announced that the next milestone was to move all available collimators
through a full nominal 5-TeV cycle.
Mike remarked that the results
presented by Stefano were very impressive. Ralph S. commented that at present all
settings were presumably computed for the nominal orbit & optics. He asked
about checks that collimators were not driven into the beam. Ralph replied that
the collimation set up relied on the reproducibility of orbit and optics, as
well as (Rudiger added) on the feedback. Answering another question by Ralph S.,
Ralph confirmed that automated measurements were not foreseen. Stefano
commented that the orbit feedback should provide a control at the 0.3 sigma level.
Ralph explained that the EIC would need to make sure that the machine was set
up correctly. Rudiger observed that if a beam were to move against a collimator
the beam would be dumped. The danger was that the orbit moved somewhere else,
not at a collimator. Such cases would need to be caught by the SIS interlock.
Ralph S. inquired whether for injecting into an empty machine any rough check
was planned. It was said that the EICs could monitor the BPMs at the
collimators in such cases. Frank suggested an automatic warning mechanism to
alert the CCC personnel. Elliot and Rudiger commented that some tasks could be
handled by the sequencer.
Oliver asked what would happen if
one jaw got stuck and one could not move that jaw anymore. Ralph responded that
the collimation team could look at this problem within a few hours, run the COLLTRACK
programme and make a decision. He
pointed out that only a few collimators were critical, and that even if one jaw
was stuck the LHC could still work using the other, opposing jaw. For detailed
predictions one would need to run SIXTRACK, requiring more time. A factor 10 speed
could be gained by using the computing grid.
This was the 51st and last
LHCCWG meeting. Roger thanked all working-group members for their significant efforts
during the past 2.5 years.