Transcription
A Hybrid Communications NetworkApproach for Advanced Applications on theModern GridMark A. Carella, PEManager – Distribution Control and Integrationnationalgrid
Private Telecom NetworkKey Stats Private NetworkFiber Route Miles . 1311T-1 Circuits . . .400Protection .250Radio Circuits .350EMS Circuits 500Voice Circuits 400Total Circuits . . 1,900Radio Sites . 81Comm Rooms 74Alarm Status Systems 125Batteries & Chargers 125NE & Upstate NY Leased CircuitsT-1's .500EMS .1473GEMS .243Radio .124Other .1311Total Lines . 3651% BW Leased vs Private 53% (approximate)2
Private Telecom NetworkMore than 1300 route miles of backbone fiber Most backbone fiber is either wholly owned by National Grid or by others with National Grid having IndefeasibleRight to Use (IRU) Optical transport is SONET platform with DWDM on backboneLegacy Equipment Alcatel Lucent DMX Platform with some DDM-2000 Approximately 141 optical nodes deployed, mostly at transmission substations and Control Centers GE JMUX in substations for protective relay applications RFL IMUX in substations for protective relay applications Digital cross connect systems Tellabs 532 (1/0)Microwave Sites Approximately 81 radio tower sites Mostly Alcatel MDR and Aviat (Harris) microwave equipment deployed Microwave radios use TDM platform at 155Mbs or 45Mbs3
Electric Substations4
Microwave / Radio Tower Sites5
Transmission SubstationCommunicationsPhysical Security AMAG card readers and access control system Video camerasMonitoring and Control: Energy Managements System (EMS) - EMS RTU data & Wide Area Network (WAN) Metering – Wired and Wireless for Power Control Gas Energy Managements System (GEMS) – RTU and Wide Area Network (WAN) Digital Fault Recorders (DFR’s) for event analysisSystem Protection: Bulk power system protection per NPCC CriteriaDispatch Radio Radio communications from Dispatch and Control Centers to field operations mobile radioTelephone Talk lines for NPCC Black Start Procedures E9116
Typical Transmission SubstationBackup ControlCenterBackupSCADASUBSTATION BSONET RingProtectiveRelay B-AProtectiveRelay B-APrimarySCADAPOTSControl CenterSUBSTATION AProtectiveRelay A-BProtectiveRelay A-BPOTSSUBSTATION CProtectiveRelay C-ANational Grid FiberSONET Optical TransportAll fiber routes separated by 3 feet minimumPOTSProtectiveRelay A-C7
Distribution & Sub-TransmissionPrivate RF Network RI VVO 4RF Private Licensed 900MHz. Point-to-Multipoint Radio Areas in MECO / NECO regions Upgrades to older GE MAS systems w/ 60 remote units New IP based 4RF units currently considered where network availability is reasonable to achieve4RF Private Licensed 700 MHz. Point-to-Multipoint Radio System Nantucket IslandGE MDS Private (Unlicensed) 900MHz. Point-to-Multipoint Radio System Trilliant Private 5.8GHz. Radio System - 35 devicesWestern New York, 200 DevicesUpstate NY DA Schemes with Private Point-to-Point 900MHz. Radio Systems UtiliNET Radios 35 Devices S&C SpeedNet Radios 30 DevicesWorcester Smart Energy Solutions Demonstration 200 “Grid” Devices on 3.65GHz. Point-to-Multipoint Private WiMAX Network 15,000 electric meters on Private 900MHz. mesh network / cellular backhaul8
Distribution & Sub-TransmissionLegacy Communications National Grid has also remotely monitored and controlled thousands of reclosers utilizing cellular communications thatwere provided in a hosted, Software as a Service (SaaS), environment. Annual SaaS cost was 400,000 for communication with 1732 reclosers at 70kb/device/month. Analog updates once a week. Binary updates twice a day statically, with unsolicited report-by-exception.SaaS offerings provided to National Grid: Cellular communications AT&T 2G & 3G network, monitoring and maintenance Web portal Visibility to the recloser data, event based actions, group notifications and commissioning aid. DNP Server SCADA interface. The AT&T 2G service was sunset at the end of 2016. To maintain communications beyond 2016, all 1732 recloser locations required a site visit to upgrade the hardware and alsoupgrade our SaaS environment.9
New Hybrid CommunicationNetwork Architecture Due to end of 2G service, we had to make a choice: Upgrade existing SaaS environment Take it internal for more control and flexibility Our service territory already has: Existing pockets of private RF infrastructure. Remote locations where RF network buildout is not feasible. The concept of a hybrid connectivity model became critical forsustainability.10
New Hybrid CommunicationNetwork Architecture Experience gained through our “Smart Grid” and Volt / VAr Optimization (VVO)pilots also illustrated the need for a flexible communications model. The model must support various types of communications technologies, asapplication requirements can vary greatly. Modern network architectures for system critical operations require: Low-latency High degree of availability, reliability, serviceability Security Redundancy These systems will vary in degrees of cost and complexity.11
Private IP Network Overview Layer 3 MPLS virtual private network solution facilitatessecure connectivity to multiple hub locations, enablingsystem redundancy, and failover capability.12
High-Level MPLS VRF Topology This network architecture would allow data from field devices tointegrate to both primary and secondary back-office systems (e.g.,data concentrators, SCADA, management portal, etc.) Cyber security was a primary focus during the entire networkdesign and was embedded from inception to implementation,complete with PEN testing and validation. New virtual routing and forwarding (VRF) solutions were identifiedto establish desired connectivity to all required facilities.13
High-Level MPLS VRF TopologyHigh Level MPLS VRF TopologyNational GridCNISite #1Existing VRFPrimary PathSite #2Site #3New VRF for Accessfrom VSTIGs to Site #2and Site #3 for NewEngland Reclosers OnlyNew VRF for Accessfrom VSTIGs to Site #1and Site #2 for New YorkReclosers OnlyVSTIGS #2VSTIGS #1Data CenterExisting VRFSecondary PathNew VRF for NewEngland VzW AccessNew VRF for New YorkVzW AccessEach VzW VRF will have aSeparate IP Address Pool14
Device Connectivity Both legacy and modern control equipment deployed across the service territory, both Serial and Ethernetconnectivity was required. Verizon 4G / LTE Cellular based multi-service connected routers were used to replace the legacy cellular basedmodems. The router’s port forwarding functionality, on-board terminal server, and firewall, was leveraged tosimultaneously accommodate both serial and Ethernet communications in a single, secure, package. This approach results in a truly “plug and play”, IP addressable, solution for any device on the network. These new edge routers communicate through the Verizon Cloud (routed via MPLS) to National Grid ownedRTUs / data concentrators located at System Control Centers. The PI Data Historian system is being used to archive operational data for analytics. Currently have 700Mhz system integrated into MPLS/PIP network Currently integrating satellite links to the MPLS/ PIP network for additional flexibility.15
Device Connectivity Example16
Device Redundency Example17
18
Implementation Challenges Strict risk management policies resulted in extensive time required to implement and test individual networkchanges. 90 required network changes Each network ‘change’ was implemented individually and required a subsequent 48hr window for testingand validation. Limited Staffing level to build and maintain this new network. Training and Documentation New technology (i.e., hardware) required training for all impacted Engineering and Operations personnel. New processes rolled out to all impacted stakeholders. Engineering Protection and Telecom Operations Meter and Test O&M Overhead Operation Extensive reference documentation required for sustainability.19
Functional Network Benefits Remote management of field devices through a secure Corporate portal: Device configuration Device security Remote Record Retrieval 2-way file transfer (remote firmware, settings, etc.) Network and security management embedded in the design of the network architectureallowing for real-time monitoring and proactive action should anomalies be detected orwhen general maintenance is required (e.g., Syslog, SNMP, etc.)FUNCTIONALITYLegacy Telemetry SolutionNew Network ArchitectureSCADANear RemoteConfigurationRemote sSupportNOYESDataHistorianLIMITEDYES20
THANK YOU
4RF Private Licensed 700 MHz. Point-to-Multipoint Radio System Nantucket Island GE MDS Private (Unlicensed) 900MHz. Point-to-Multipoint Radio System Western New York, 200 Devices Upstate NY DA Schemes with Private Point-to-Point 900MHz. Radio Systems UtiliNET Radios 35 Devices S&C SpeedNet Radios 30 Devices
