Transcription
Challenges of Implementing Substation HardwareUpgrades for NERC CIP Version 5 Compliance toEnhance CybersecurityJ. Matt Cole, PEPower Delivery ServicesSargent & Lundy LLCChattanooga, TN, USAjoseph.m.cole@sargentlundy.comIn 2007, FERC approved 83 NERC Reliability Standards.NERC CIP Standards were also approved in 2007 andincluded 9 standards with 45 requirements. The CIP standardswere put in place to ensure that utilities actively secure theirassets for operating the Bulk Power System (BPS). In 2010,FERC approved NERC CIP Version 3 standards. Three yearslater, FERC approved additional NERC standards that addednew cybersecurity enhancements in the NERC CIP Version 5(V5) standards. Implementing the V5 standards will result insignificant progress in preventing cyber risks to the BPS.Abstract— To minimize the vulnerability of the electric powergrid to cyberattacks the North American Electric ReliabilityCorporation (NERC), under the jurisdiction of the FederalEnergy Regulatory Commission (FERC), has enacted andenforced cybersecurity standards that continue to evolve astechnology and the nature of these threats advance. TheseCritical Infrastructure Protection (CIP) standards issued byNERC require utilities to meet an aggressive timeline forregulatory compliance. To supplement in-house resources,utilities rely on consultants and suppliers to meet NERC’s fastapproaching deadlines, such as implementing the latest NERCCIP Version 5 (V5) standards. This paper discusses majordesign challenges faced when upgrading substation equipmentfor cybersecurity enhancements and, specifically, the hardwareimprovements implemented for the NERC CIP V5 conversions.II.In 2013, 256 cyberattacks were reported by the IndustrialControl Systems Cyber Emergency Response Team (ICSCERT). Over 50% of these attacks were utility related.Index Terms – Communication system security, Communicationnetworks, IP networks, Power system protection, Substationprotection.I.PREVIOUS CYBER ATTACKS AND THREATSUnderscoring the need to harden the country’s utilities towithstand these threats, Presidential Policy Directive-21(PPD-21) for Critical Infrastructure Security and Resiliencewas issued in 2013. This directive was ordered to “strengthenand maintain secure, functioning, and resilient criticalinfrastructure – including assets, networks, and systems – thatare vital to public confidence and the Nation’s safety,prosperity, and well-being [3].”INTRODUCTIONA major blackout in 1965 effecting portions ofsoutheastern Canada and northeastern United States was theimpetus for forming NERC with utilities complying on avoluntary basis. A blackout in 2003 impacting 50 millionpeople in the midwestern and northeastern U.S. and Ontario,Canada, resulted in NERC becoming certified as a reliabilityorganization by FERC, with standards compliance becomingmandatory. A recent part of NERC’s mission to reduce theprobability of future blackouts is to protect the grid and itscritical equipment from cyber threats.In May 2014, the Department of Homeland Security(DHS) reported that a utility in the U.S. was breached bycyber hackers that gained access control of major operationsthrough the internet.Another incident reported by the ICS-CERT was that acompany’s Supervisory Control and Data Acquisition(SCADA) system was hacked through a cellular modemwithout special access controls in place. The SCADA systemwas fortunately disconnected from the main controls forscheduled maintenance.The urgent need for electric power companies to addressthis issue has been driven by previous cyberattacks thataffected utilities and by the significant impact that widespread loss of power has on the nation and the economy. Overthe last ten years, the U.S. government has informed all utilitycompanies which include power, telecommunications, gas,and water that they were extremely vulnerable to possiblecyber threats.CNN had reported in 2007 about the Idaho LabExperiment named “Aurora”, which was an experimentalcyberattack to cause a power generator to self-destruct. “DHSacknowledged the experiment involved controlled hacking1
into a replica of a power plant's control system. Sourcesfamiliar with the test said researchers changed the operatingcycle of the generator, sending it out of control. Some expertsfear bigger, coordinated attacks could cause widespreaddamage to electric infrastructure that could take months to fix[4].”III.In 2014, NERC initiated a program to help utilities movefrom CIP V3 to CIP V5 standards. The objective of thistransition program was to advance the industry’sunderstanding of the security requirements for CIP V5standards with the expectations for compliance andenforcement.Table 1 below shows the NERC CIP V5 standards subject tofineable violations after April 2016.NERC CIP VERSION 5 REGULATORY STANDARDSUtilities are being faced with a large culture change.NERC Standards are pushing utilities to have more cybercontrol of their equipment, especially Internet Protocol (IP)network routable equipment. Significant resources have beenassigned to prepare rollout of V5, which is an ongoing effort.The requirements for CIP V5 include: Training employees and promoting awareness ofpossible cyber attacks including physical securityvulnerabilities and phishing/social network attacks. Creating a specific plan and procedures for V5conversions. Identifying critical (high/medium impact) and noncritical (low impact) substations that may or may notaffect the BPS. Identifying the Bulk Electric System’s (BES’s)critical and non-critical power assets within anElectronic Security Perimeter (ESP). Identifying the Critical Cyber Assets (CCAs) andnon-CCAs within an ESP. Identifying all Entry Access Points (EAPs) within theESP. Implementing Firewalls, Virtual Private Networks(VPNs) and restricted access data gateways. Upgrading Hardware and Softwaresubstations and control centers. Implementing Network Change Detection andConfiguration Management Improvements. Implementing Access Controls with Logging andEvent Management. Improving File and Data Monitoring. Requiring more stringent user account managementwith password restrictions and more frequentpassword changes. Ensuring all equipment inventory asset lists arecomplete and accurately documented.atNERC LEBES Cyber System CategorizationSecurity Management ControlsPersonnel & TrainingElectronic Security Perimeter(s)Physical Security of BES Cyber SystemsSystem Security ManagementIncident Reporting and Response PlanningRecovery Plans for BES Cyber SystemsConfiguration Change Management andVulnerability AssessmentsCIP-011-1Information ProtectionCIP-014-1,2Physical SecurityTable 1. NERC CIP V5 StandardsThis paper focuses on the design challenges powercompanies face with upgrading substation equipment whileimplementing NERC CIP V5 standards (particularly CIP-003,CIP-005, CIP-007 and CIP-011) in order to avoid regulatoryfines and Potential Violations (PV’s) after April 2016. Goodrecord keeping, thorough documentation and validation of asbuilt configurations are all essential to fully comply with thesestandards and to any issues during the NERC audits.IV.SUBSTATION DESIGN ASSESSMENTSSome utilities are moving ahead with integratedsubstations incorporating advanced remote access capabilitiesto end user equipment using remote dial-up or IP networkssuch as Wide Area Network and Local Area Networks (WAN& LAN).Remote access controls demand a robusttelecommunications transport system due to accommodatelarge amounts of data back and forth between the substationand the control center. Computer servers and networkingequipment process data at the main control center, whichcould reside hundreds of miles away.bothOther utilities with less advanced telecommunicationssystems and non-remote access controls may decide to remainin a disconnected IP state to comply with current NERC CIPstandards.As part of the assessment, a utility must identify thecritical stations, critical BES power assets and CCAs within anESP. The entity must also identify all EAPs. Each BES andCCA asset must be clearly identified and categorized.The objective is to protect the IP network before it ishacked. V5 forces utilities to protect, control, and adequatelymonitor all CCAs within the ESP and Physical SecurityPerimeter (PSP). NERC CIP mandates enforcement foraccess control at the cyber and physical levels which affectsprotection and controls (P&C) and telecommunicationsequipment.Upgrading substation equipment for NERC CIP V5regulatory requirements has been challenging and timeconsuming. Not only does it involve upgrading P&C andtelecommunications equipment to help protect from cybersecurity attacks while limiting physical security access, but itrequires replacing legacy and obsolete equipment as well.2
Obsolete equipment can pose a security risk and may not becompatible with new NERC hardened equipment. Vendors nolonger support legacy equipment with security patches orfirmware upgrades, greatly limiting the availability ofreplacement parts.The router gateway is connected to the fiber network via theFO electronics device. An Ethernet switch is connected to therouter and is set up with virtual LANs (VLANs). One VLANhas all the non-critical IP routable connections and the otherVLAN has the CCA routable connections. Figure 1 alsoshows the P&C relays connected in different ways for remoteaccess (yellow & red). Some of the intelligent electronicdevices (IEDs) are connected to the Ethernet switch and someare connected to a communications processor via a serial RS232 protocol connection from a dial-up modem leased linecircuit.Some of the new, resilient systems required at substationsfor protection against cyberattacks for NERC CIP V5standards are:firewalls, VPNs, restricted access datagateways and Intrusion Prevention Systems/IntrusionDetection Systems (IPS/IDS).As an engineering partner for several utilities, Sargent &Lundy, LLC (S&L) supported the upgrade of several mediumimpact substations and control houses from 2014 through2016 to comply with NERC CIP V5 standards. Most clientshad already implemented NERC CIP V3 standards andcompleted a portion of the V5 conversion.The SCADA system is shown connected to the Ethernetswitch with a serial connection to the Human MachineInterface (HMI) and Programmable Logic Controllers (PLCs).All the network and CCA equipment shown in thesubstation reside within the ESP (red dashed box) with othernon-CCA equipment shown within the PSP (blue dashed box).The EAPs show the external routable communication pathsthat pass traffic in and out of the ESP/PSP.S&L was tasked with helping utilities implement thedesign changes for the CIP-003-5, 005-5, 006-5, 007-5 and011-1 standards as shown in Table 1, with the main emphasison CIP-005-5 “Electronic Security Perimeter”. S&L’s majorinvolvement was performing the design, updating outstandingas-builts as required, ensuring the inventory asset lists wereaccurate while providing necessary construction support. Theprojects kicked-off with various site visits walk downs todetermine a total list of equipment that requiredmodification/upgrades for the controlled ESP. The assessmentincluded recommendations for replacing any legacy orobsolete equipment.Also shown in the PSP are the physical security videomonitors and card readers along with the batteries andchargers.V.A. Design issuesFor CIP V5 conversion, the entire substation IP networkhad to be upgraded. VLANs separating critical and noncritical assets on the same Ethernet switch were no longeracceptable as they were in V3. The V5 conversion forced thesubstation LAN upgrade to have two separate hardwarenetworks with different router interfaces – one being forCCAs and the other for non-CCAs. Each network wasequipped with a new router and firewall with VPN capability.All routable P&C and telecommunications devices wereseparated and connected to the appropriate critical and noncritical networks. An IPS/IDS had to be installed andconnected to both networks for preventing and detectingmalicious traffic and unauthorized access to the network.Ethernet switches had to be replaced due to IPS/IDSincompatibility and visibility limitations.Shown in Figure 1 is a basic telecommunications singleline diagram showing a typical or existing substation LANconfiguration with the major P&C and telecommunicationsequipment AP)(ESP)F.O. ELECTRONICSCONTROLCENTERV1 – VLAN1V2 – VLAN2DIAL-UPMODEMROUTERCOMM PROCDFR, METERsRELAYs(IEDs)GPS CLOCKV1ETHERNET SWITCHDIAL-UPCOMM PROCTELECOMM ALARMsTOTELECOMMEQUIPMENT48VDCBATTCHARGER DISTRIBUTION PNLV2PHYSICALSECURITYBELL COMPANYAn intelligent data gateway was installed for securitymanagement and access control. It was necessary to upgradesome of the legacy SCADA Remote Terminal Units (RTUs)and Human Machine Interface (HMI) systems due toincompatibility issues with the new data gateway system. Asthe SCADA RTUs and HMI systems were upgraded, somestations had PLCs that were using serial interface connectionsback to SCADA that required upgrades as well. These serialinterfaces were replaced with IP Ethernet interfaces. The newconfiguration allowed the PLCs to connect to an Ethernetswitch on the critical network and talk to SCADA on the samenetwork instead of connecting directly to the SCADA RTUvia a slow serial CONVERSION TO CIP V5RELAYs(IEDs)(ESP)CARD READERs(PSP)Figure 1. Existing Substation LAN ConfigurationAt the top right of Figure 1 is the control center connectingto a fiber network WAN. The substation is connected to thesame fiber network via a Fiber Optics (FO) electronics device.Due to the increase in network components and datamonitoring traffic between the substation’s controlled ESPand the control center, the telecommunications transport3
equipment and bandwidth for the new networks were replacedwith faster, reliable and failover redundancy transports.FIBERNETWORK(WAN)B. Upgrading Legacy and Obsolete EquipmentObsolete telecommunications alarm units were replaceddue to the vendor no longer supporting or supplying parts.Legacy telecommunications processors that were less secureand more vulnerable were replaced to continue providing IPconnectivity to the substation LAN while providing remoteaccess via serial connections to older (non-IP routable)protective relays or IEDs. Less secure and obsolete protectiveline relays that were using dial-up for remote access werereplaced with new IP connection relays. Legacy GlobalPositioning Systems (GPS) clock receivers were upgraded toprovide accurate timing for the IP routable devices includingthe new IP ALLROUTERETHERNET SWITCHPHYSICALSECURITYTELECOMM ALARMsCONTROLCENTERROUTERRELAYs(IEDs)ETHERNET SWITCH(VPN)NON-CRITICALCRITICALFIREWALLROUTERGPS CLOCKROUTER(ESP)F.O. ELECTRONICSHMIPLCsSCADAETHERNET SWITCHDFRCOMM IALRELAY(IEDs)(ESP)Due to additional network equipment, data gateway andthe IPS/IDS, the substation telecommunications 48VDC loadhad increased over 10 amps of continuous DC load.Substation battery studies were required for the 48VDCsystems. As a result, over 50% of the substations in S&L’sscope of work, had batteries and chargers that were inadequatefor the additional loads per the client’s standard and the IEEE485 standard.This included upgrading some 48VDCdistribution panels due to being completely full or undersizedfor the future DC load increases.CARD READERsFigure 2. CONSTRUCTIONDue to the limited construction resources and the fastapproaching NERC CIP V5 mandate to be compliant by April2016, the client’s construction crews started work before somedesigns were complete.With the amount of substation design changes that wentalong with NERC CIP V5 conversions and the extremely tightdesign completion schedules, resources were allocated asnecessary to meet or accelerate all the client’s designschedules. This reduced costs by allowing installation work tobegin earlier than anticipated and put the project on track tomeet the April 2016 deadline.Engineering support during construction, includingtraining for the new HMI units that were being installed, waskey to achieving the successful implementation of all theNERC CIP V5 upgrades.The major equipment described earlier that had to bereplaced and/or installed for CIP V5 is as follows: New NERC Substation LAN ConfigurationVI.In addition to new hardware installations, upgrades andreplacements for V5 compliance, some of the drawings anddocument controls were not accurate or up to date. Several asbuilt drawing corrections were necessary during the designphase. CHARGER DISIBUTION PNL(PSP)Additional equipment that was installed per NERC CIP V5was a third IP network for the access card readers and videosurveillance equipment for physical security access andmonitoring. 48VDCBATTAdvanced routers and Ethernet switches withfirewalls, VPNs and other securityIPS/IDS and Intelligent Access Data GatewaysFO electronics or microwave (MW) radio transportsSCADA RTUs and HMI SystemsPLC Serial to IP InterfacesLegacy telecommunications alarm units andcommunications processorsLegacy protective line relaysIncompatible GPS clock receiversDial-Up leased line remote access converted tosecure IP accessBatteries, chargers and distribution panelsVII. CONCLUSIONUpgrading substation hardware for NERC CIP V5conversions has been complicated by the discovery of legacyand obsolete equipment that must be replaced to enable therequired cybersecurity improvements. Another challenge islack of up to date as-built documentation not adequatelyrepresenting the current design configuration. Also, theincrease in DC power loads from additional substationequipment that is required for V5 may result in the need toupgrade, some batteries, chargers and distribution panels tohandle larger capacities. It is recommended that contingenciesbe included in resource plans, budgets and schedules whenbeginning the process of modifying substation hardware forNERC CIP standards.Shown in Figure 2 is the revised telecommunicationssingle line diagram from Figure 1 with the typical LANmodifications for V5 compliance.VIII. REFERENCES[1] NERC CIP Standards, CIP Version 5, 2013. [Online]. andards.aspx.4
[7] Tempered Networks, Seattle, WA. “NERC CIP Alignment,” compliance/.[8] The Fortinet, Sunnyvale, CA. “Securing ICS Infrastructure for NERCCompliance and Beyond,” unpublished. [Online]. Available:http://www.fortinet.com/resource -nerc.html.[9] GarrettCom Inc., (2007, July). NERC/CIP Compliance: Headache orOpportunity? Utility Automation & Engineering T&D. [Online].Available: http://www.garrettcom.com/nerc cip opportunity.htm[2] NERC CIP Standards, CIP V5 Transition Program, 2014. [Online].Available: .aspx.[3] "Presidential Policy Directive - Critical Infrastructure Security andResilience." www.whitehouse.gov. US Government, 12 Feb. [4] CNN.com., (2007, September). Staged cyber attack reveals [5] Hector J. Altuve Ferrer and Edmund O. Schweitzer, III, “InformationSecurity” in Modern Solutions for Protection, Control, and Monitoringof Electric Power Systems. Oregon, IL: Quality Books, Inc., 2010, pp.263-279.[6] RAD, Mahway, NJ. “NERC CIP Compliance Guide,” unpublished.[Online]. Available: eport/33162/.5
NERC CIP VERSION 5 REGULATORY STANDARDS Utilities are being faced with a large culture change. NERC Standards are pushing utilities to have more cyber control of their equipment, especially Internet Protocol (IP) network routable equipment. Significant resources have been assigned
