Transcription
ICARUS: Remote shift from homeUmut Kose, CERN
From “Overview presentation for the April 2020, SBN Program Management Group”SBN Document 17767-v2Completing system stabilizationCollaboration working on plan for staged turn-on with very limited on site access required toconnect cables, turn on equipment, etc.
Before COVID-19, ICARUS detector monitored from Fermilab’s Remote Operations Center by both localshifters and other shifters travelled to Femilab. Cryogenic and Strain Gauges are monitored by CERN teamas well.Off-site Remote Operations Centers at other institutes (Colorado State University and CERN) was setupand tested beginning of the year.As part of the evolving response to the COVID-19 situation, remote shift scheme revisited allowing shiftersto perform duties from their laptops and/or desktops.
Prerequisites:Fermilab services account (ECL)Fermilab Kerberos accountrequest an account to ICARUS-GATEWAY01/02 machineFermilab “Fermi” Windows domain accountrequest an access to iFix machine (howard@fnal.gov)RSA token for Fermilab VPN connections (soft tokens are delivered by email, an app to beinstalled on your mobile phone)To do:Call Fermilab Service Desk to get your passwords (specify which password): 1 630 840 2345SSH tunnel to specific machines to view Slow Controls (DCS), DAQ, Grafana DAQ monitoring.Remote Desktop program to view iFix cryogenics display “ppd-ifix2.fnal.gov”.
ICARUS Operations Wiki PageMany useful pages explaining what to do on shift and operations/wiki
Shift scheduleShift nameTime (FNAL time) [CERN]Early morning00:00 – 08:00 [07:00 – 15:00]Work hours08:00 – 16:00 [15:00 – 23:00]Evening16:00 – 00:00“The assignment has been compiled with the usual random assignment of a week per each institution taking into account the shiftsalready done until now. Night shifts are assigned to institutions based in Europe and swing shifts are assigned to US basedinstitutions. Day shifts are assigned to both locations.”
Proposed shift assignment for May 2020 (C. Montanari, sent by email today)Assigned to CERN team
Remote shift from homeOption 1: Using your PC, follow the instructions in Icarus-operations wiki page orOption 2: Connect to the server in ideasquare via Remote Desktop program.
Option 1: Using your PC, follow the instructions in Icarus-operations wiki page.FermilabVPNCisco AnyConnect: [Fermilab Service password & RSA token]In terminal: kinit –f username@FNAl.GOV [Kerberos password]DAQ-VNC:ssh -KL 8443:icarus-evb01.fnal.gov:443 8443DCS-VNC:ssh -KL 9443:icarus-evb06.fnal.gov:443 9443GRAFANA DAQ:ssh -KL localhost:10080:localhost:10080 icarus@icarus-gateway01.fnal.gov'ssh –KL localhost:10080:localhost:10080 0iFixppd-ifix2.fnal.gov FERMI\username; FERMI windows domain password
Web camera link and passwords for DAQ-VNC, DCS-VNC and GRAFANA DAQ monitoring pages are given in
How to start the shift and what to do?Check shift bulletin board, SLACK::SBN::Icarus-shift-operations, and ELOGCheck your connections before contacting previous shifter.Get in touch with previous shifter before start your shift ( 15 min before)Make ECL entry “Shift check-in” (Tag: Control Room, General)Immediately after shift check-in make ECL entry “Start of shift”Make ECL entries every 1 hour on the status of cryogenic by using “Cryogenics checklist Form"Make ECL entries every 2 hours on the status of the run by using "DAQ Commisioning Form“.The daily plan is to take data continuously from the WE mini-crates, and the shifter’s duty consists simply inperiodically verifying the status of the DAQ and online monitoringAfter the new shifter has arrived and you have handed things over, make ECL entry “Shift check-out”
In case of alarms or problem during your shift, contact the relevant experts:Use SLACK::SBN::icarus shift operations to get help from other shifters or experienced peopleLet me know if you do not have slack access!
ECL page[Fermilab Service password]
Option 2: Remote shift through the server in ideasquare Need to be at CERN network, therefore connect first to “cernts.cern.ch” machine (NICE password used) withRemote Desktop Connection From “cernts.cern.ch” machine connect to the server [128.141.xxx.xxx] with Remote Desktop Connection(contact me for the password)
Shift check-inShift startShift check-out
Run Number isalso given hereYou can check the date/time tosee whether DAQ is runningand monitoring tool is updating
DAQ-VNChttps://localhost:9443
ECL – DAQ inputEvery 2 hours!Grafana pageICARUS online monitoring page
VNC-DCShttps://localhost:8443To be checked every 1 hour and filled into the ECL.Check for alarms! In case of alarm call Claudio and Michael.Compare with iFix monitoring display!
iFix cryogenic monitoring tool
Sometime you get an error of “SBN ICARUS ALARM SUM A CV: Connection NOT established with node”
There should be no “Alarm” at the “Overview” tab.
The pressure parameters have to be checked.Especially if the pressure gets smaller than 50 mbar call cryo team: Claudio and Michael
Tank-1 internalCheck the temperatures and levelsTake snapshots time to time.If temperature difference between two valuesof the temperatures become higher than 47K,call cryo team: Claudio/Michael!Usually these temperature are around 87-91K:so if you find a value around 45K or (moreprobable) a value around 130K you have tocall!Tank-2 internal
ECL - CRYO
If you decide using the server in ideasquare for Remote shift, contact to get more info how toaccess the server (for password).In case of any problem, do not hesitate to contact me.
Currently the control room in ideasquare
Fermilab “Fermi” Windows domain account request an access to iFix machine (howard@fnal.gov) RSA token for Fermilab VPN connections (soft tokens are delivered by email, an app to be installed on your mobile phone) To do: Call Fermilab Service Desk to get
