Report of the Run Coordination Task Force

 

Version 2.2

March 18, 1999

Task Force Members

Vince Cianciolo, Achim Franz, Steve Pate, Michael Sivertz (Chair)

 

  1. Introduction

    2. The Task Force met during the December Core Week to go over the charge, and to collect points of importance which must be included in the recommendations.

    Every member of the Task Force brought strong opinions about various aspects of Run Coordination, but there was unanimous agreement that the most important aspect was a clear chain of command. The collaboration needs to select competent people to fill the positions in Run Coordination. This should be easy to do with a base of over 400 collaborators. Then the collaboration must grant authority to those people and make clear where the responsibility rests for all of the tasks associated with running the experiment.

    From this starting point, all of the other recommendations are details. What follows are the detailed recommendations of the Task Force.

     

  2. Run Coordination hierarchy

    3. The person responsible for directing the operation of the detector is the Run Coordinator (RC).

    The person supervising the day to day operations of the detector is the Period Coordinator (PC).

    The person responsible for activities of the Shift Crew during a shift is the Shift Leader (SL).

    The Institutional Board (IB), in consultation with the Executive Council (EC), shall determine the algorithm by which the shift commitment of PHENIX collaborating institutions is calculated. This applies to shift assignments for the PC, SL and the Shift Crew. It shall be the responsibility of the IB members or their designates to communicate with the RC regarding shift assignments. The Task Force requests that the shift assignment algorithm be made as simple as possible in order to reduce the bookkeeping load on the RC.

    The Shift Crew commitment shall be one full week of shifts, and shifts shall be assigned in one week blocks. The PC shift commitment shall be for two weeks.

    The shift commitment shall be allocated irrespective of all other service tasks.

    1. Run Coordinator

      2. The Run Coordinator shall be appointed by the Spokesperson in consultation with the EC. This appointment will be for a period of one year, beginning approximately 12 weeks prior to each 37 week running period. This will allow the RC time to prepare for the activities associated with the up-coming run.

      The RC shall be resident at BNL during the appointment.

      It is the duty of the RC to report to the Spokesperson and EC on a monthly basis, or as requested by same.

      Before each 37 week running period starts, the RC will send out shift assignments to the collaborating institutions via the IB members. The shift assignments will be a calendar with shifts allocated to institutions. There will be three specific roles to be assigned; Period Coordinator, Shift Leader, and Shift Crew. The IB members will respond to the RC by giving the names of people for each assigned shift. In case of scheduling conflicts, it is up to the institution, not the RC, to arrange shift trades. If IB members do not respond to the RC request in a timely manner, the Spokesperson will take action to ensure compliance.

       

    2. Period Coordinator

      3. Each Period Coordinator (PC) will serve for a two week term of duty.

      Each IB member shall identify at least one person from their institute who can act as Period Coordinator. The RC will approve the list of PC nominees submitted by the IB members. The RC shall draw up a PC Shift Schedule allocating two week slots to the collaborating institutions, according to the algorithm determined by the EC. The RC will send this PC Shift List to the IB members. The IB members shall sign up their nominee on the PC Shift Schedule. If a collaborating institution does not have a qualified nominee, then a representative from that institution shall come to PHENIX and be trained for the position by the RC.

       

    3. Shift Leader

    Each Shift Leader (SL) will serve for a one week term of duty.

    The RC shall draw up a SL Shift Schedule allocating one week slots to the collaborating institutions. The IB members will sign up their nominee on the SL Shift Schedule, pending the approval of the RC. The RC will approve the list of SL nominees submitted by the IB members. If a collaborating institution does not have a qualified nominee, then a representative from that institution shall come to PHENIX and be trained for the position by the RC.

    A SL will be on hand at all times during running periods to direct activities on the shift. The person selected as SL will remain SL for the duration of the shift period (one week). Each person who serves as SL will also serve as the Alternate PIC for Building 1008.

  3. Responsibilities of the Run Coordination Team

    4.  

    1. Run Coordinator

      2.  

      The RC shall have the following responsibilities:

      1. Approve people to fill Shift Leader and Period Coordinator shift slots.
      2. Maintain the Run Coordination web site which will include a web-accessible Shift List.
      3. Maintain a list of mandatory training requirements for PHENIX Shift Crew members.
      4. Act as the liaison to RHIC management with respect to operations.
      5. Coordinate activities in advance of and during accesses to the interaction region during the running period.
      6. Chair an open weekly meeting in which hardware and software subsystems report their status.
      7. Report monthly to the EC on the status of the experiment.
      8. Work closely with the EC to make sure the physics program is being carried out in a safe and effective manner.
      9. Maintain On-Call and Off-Call Experts lists.
    2. Period Coordinator

      3.  

      The PC shall have the following responsibilities during each two week shift period:

      1. Supervise the safe day-to-day operations of the experiment.
      2. Chair a daily meeting reviewing the previous day's events and plan the events of the next day.
      3. Act as the liaison to the RHIC operations crew.
      4. Attend the daily RHIC Operations meeting.
      5. Organize the activities of experts who have been called in.
      6. Write a web-accessible report summarizing in detail the events of each two week shift period.
      7. Report weekly at the RC meeting.
      8. Evaluate the performance of the shift crew and rearrange shift members to improve shift functioning, if required.

       

    3. Shift Leader

      4.  

      The SL shall have the following responsibilities:

      1. Make sure the other shift personnel are carrying out their duties.
      2. Serve as Alternate PIC for building 1008.
      3. Take care of proper logging of activities and events.
      4. Summarize shift activity in the web-accessible shift log.
      5. Provide sufficient overlap between shift changes to ensure smooth running.
      6. Act as the immediate liaison between the counting house to the outside world.
      7. Act as liaison to the RHIC operations crew in the absence of the PC.
      8. Alert the PC and/or RC in case of problems and emergencies.
      9. Call in experts when problems arise which require their attention.
    4. Shift Crew

      5. During running periods there shall be a minimum of four people on shift at all times. This will include a SL and three other members of the shift crew. For later reference these shift members will be called "Shift Leader", "Shift Assistant 1", "Shift Assistant 2", and "Shift Assistant 3". During watch periods, such as when there are machine studies or during start up, there shall be a minimum of two people on shift at all times.

      The Shift Crew shall have the following responsibilities:

      1. Log the shift activities in the electronic and paper log books.
      2. Read the log book entries from the previous shift before the beginning of the shift.
      3. Monitor the subsystem operations and performance.
      4. Alert subsystem experts if there appears to be a problem with their subsystem. (See following section for Emergency Procedure Handling.)
      5. Assist in emergency access situations.
      6. Monitor the data for Quality Assurance.

       

      If a shift crew member does not show up (or arrives late for shift) the PC shall determine the most appropriate way of dealing with the situation.

       

    5. Shift Definition

      6. The shift quantum is one full week of shifts. A person coming to BNL to take shifts should expect to take them in one week (7 shifts) blocks. The persons taking the same shift for one week will therefore serve together for that week and form a single "shift crew".

      The role of Shift Assistant 3 is a special one in that people taking this shift can split their assignment. Some collaborators will not always be able to commit to a week of shifts at a time, so when they are given a week of shifts they can trade part of their shift with other collaborators in order to make a more convenient schedule for both people. There will be a web site managed by the RC dedicated to facilitating shift trading.

    6. Shift Times
      1. There will be two overlapping cycles of three shifts per day:

      Cycle A: The Shift Leader and Shift Assistant 1 shall take shifts on Cycle A.

      "Owl" Shift 00:00 – 08:00

      "Day" Shift 08:00 – 16:00

      "Evening" Shift 16:00 – 24:00

      Cycle B: The Shift Assistant 2 and Shift Assistant 3 shall take shifts on Cycle B.

      "Rooster" Shift 04:00 – 12:00

      "Physicist" Shift 12:00 – 20:00

      "Vampire" Shift 20:00 – 04:00

       

    7. Shift Cycles

      8. In order to keep the detector from collapsing in chaos every Monday as the new group of shifters replaces the old group, it is proposed that the new shift cycles be staggered. There are two methods of staggering under consideration. The decision of which method to implement will be left to the RC, as more running experience is gained.

      Method A: In the first method, only one member of the three-person shift crew will be replaced at any particular time. Each week-long period of shifts for the "Shift Leader" will begin on Monday. The new week of shifts for "Shift Assistant 1" will begin on Wednesday. The new week of shifts for "Shift Assistant 2" will begin on Thursday. The new week of shifts for "Shift Assistant 3" will begin on Friday.

      Method B: In the second method, only one shift crew will be replaced at any particular time. The Owl (and Rooster) Shift Crew will be replaced on Mondays, Day (and Physicist) Shift Crew will be replaced Wednesday, and Evening (and Vampire) Shift Crew will be replaced on Fridays.

       

    8. Operational Governance of the Experiment

    At any given moment, the following persons are in charge of the immediate operations of the experiment:

    Run Coordinator

    Period Coordinator

    Shift Leader

    All others persons, including experts who have been called in, must act in accordance with the instructions of these people. All major changes to the running conditions of the experiment must be cleared with the RC in advance. Such changes should be discussed and planned at the weekly and daily meetings, if time permits.

  4. Emergency Procedure Handling

    5. During the operation of PHENIX there will be situations in which the shift crew, following documented experimental operating procedures, is instructed to "call an expert." There will be other situations when the proper course of action is not even documented. Such situations (which may or may not constitute an emergency) cannot and should not be handled solely by the shift crew.

    The task force recommends that the Run Coordinator maintain a list of On-Call Experts. This list should include at least two persons for each subsystem (the primary and an alternate) and the task force can foresee additional on-call requirements not strictly related to any given subsystem (e.g., the Period Coordinator, infrastructure, etc.). Each person on the list must be locally available to respond to such situations involving their area of expertise. In addition, the task force recommends that PHENIX provide cellular phones to each primary On-Call Expert.

    Even with this list carefully maintained, situations may arise in which neither the primary nor the alternate On-Call Expert can be contacted. For this reason, the Task Force recommends that the Run Coordinator maintain a second list - Off-Call Experts. This list would have the same organization as the On-Call Experts list, broken down by subsystem with additional categories as required. Persons on this list should be listed in order of contact preference so that, e.g., the first person is effectively the second on-call alternate, the second person is the third on-call alternate, etc. Persons on this list would not have any responsibility to be locally available but could provide back up expertise in emergencies.

  5. Training requirements for the shift crew

Every PHENIX Shift Crew member will take all required BNL and RHIC training courses. The PHENIX Safety Coordinator shall supply the RC with a list of all BNL and RHIC Training Requirements for people on shift. PHENIX as a whole will take responsibility for preparing materials sufficient to train Shift Crew members in the safe and productive operation of PHENIX. The RC shall maintain a web-accessible list of all required BNL, RHIC and PHENIX-specific training requirements. All shifters are advised to check the RC web site well in advance of their shifts to ensure that they have completed all the required training. Each PHENIX Shift Crew member is ultimately responsible for realizing the great responsibility that being on shift entails and for taking the shift training seriously.

Each subsystem will supply thoroughly documented operating procedures. The call-list protocol recommended by the task force will provide significant incentive to subsystems to make these procedures comprehensive and easy to understand since any situation not found in the procedures will require calling in the designated On-Call Expert. Training for shift duty shall include a thorough reading of the documentation pertaining to normal shift duties and a perfunctory understanding of possible non-standard situations and procedures.

This documentation shall:
    1. Include a general overview of the subsystem.
    2. Define "normal operation".
    3. Delineate normal operating procedures.
    4. Outline steps that can be taken by the shift crew should something go wrong.
    5. Indicate when it is time to call in an expert.

      6. In addition to studying the documentation provided, new Shift Crew members are encouraged to audit shifts prior to start of their first official shift. "Auditing" is the practice of sitting in on an active shift and watching everything that goes on during the shift. Advance arrangement should be made with the RC to avoid crowds in the counting room.

      The task force recommends that the PC and SL make a point of mentoring Shift Crew members not trained for these jobs. This will increase the pool of candidates from which new nominees can be picked.

    6. Interaction with RHIC
      1. The RC shall be primarily responsible for interaction between PHENIX Operations and RHIC. All PHENIX Operations issues that impact RHIC shall be directed through the RC. The RC shall attend the weekly RHIC Operations Meetings.
      2. The PC shall attend the daily RHIC Operations Meetings.
      3. The SL shall be responsible for communicating with RHIC on all issues having to do with the normal operation of PHENIX. Exceptional issues will first be directed to the PC and if necessary to the RC. Normal operational communication will include reports of beam quality, luminosity, polarization, etc.
      4. All requests for access shall be handled by the RC.

       

       

    7. Special rules for the Engineering Run and Start-Up

      8. RHIC is a new machine and currently no detailed time-scale for the startup is available. This makes it difficult to set up a formal procedure.

      In the coming months each subsystem will define its role, i.e. what detector in which configuration, and what will be their needs in the ER, such as gas, HV, and LV. This will require prior approval by the DC and RHIC/PHENIX Safety.

      The RC for the ER (Achim Franz) will soon submit a request to IB members for a list of people available for the ER shifts (PC, SL, and Shift Crew). On a similar time scale a list of experts for each subsystem will be defined so that the On-Call Expert list can be made up.

      As soon as detectors are put on operating gas, round the clock shift coverage will start. There will be a minimum of two person per shift in watch mode. All PHENIX groups shall cover these shifts, not only the subsystems using gaseous detectors. Subsystems shall be allowed at this time to setup and monitor ancillary systems, HV, LV, cooling, etc. Full documentation about emergency procedures is required prior to operating the detectors in full running mode.

      If RHIC is close to getting beam in the machine DAQ and GL-1 experts shall be on call. They will be given priority to set up a GL-1 trigger and timing. Priorities shall be set in advance during the daily meeting, allowing one subsystem to change run, trigger or timing configurations for their tests, which may influence the other systems running in parallel.

    8. Summary of Recommendations

      9. The broad recommendations of the Task Force can be summarized as follows:

      1. The Institutional Board, in consultation with the Executive Council, will determine how responsibility for shifts is to be assigned to the collaborating institutions.
      2. The Run Coordinator oversees operation of PHENIX and reports to the EC. The appointment is for one year.
      3. The Period Coordinator oversees day to day activities and reports to the RC. The appointment is for two weeks.
      4. The Shift Leader oversees the activities on shift. Each shift is staffed by four people, one of which is identified as the Shift Leader. Shift assignments are for one week, except for the Shift Assistant 3 slot which may be subdivided at the convenience of the shifter.