KASEYA INCIDENT RESPONSE August 9, 2021 - CU*Answers

Transcription

KASEYA INCIDENTRESPONSEAugust 9, 2021CONFIDENTIAL

CONTENTSINCIDENT OVERVIEW . 3GENERAL TIMELINE SUMMARY . 4INCIDENT DETAILS. 5AUDITLINK COMMUNICATION WITH OTHER VENDORS . 7RESPONSE TO MICHIGAN DIFS . 8CALL WITH KASEYA . 9WHAT WOULD CU*ANSWERS HAVE DONE IF AFFECTED? . 9LIMITATIONS WITH VENDOR RISK MANAGEMENT. 10COMMUNICATIONS WITH CLIENTS . 10Kaseya Incident Response Page 2 of 16

DATE(S) OF INCIDENTJuly 2, 2021 July 23, 2021DATE OF REPORTAugust 9, 2021SENSITIVE DATA EXPOSED?NOINCIDENT OVERVIEWOn Friday, July 2, CU*Answers received a communication from Kaseya stating all Kaseya clients should shutdown their Kaseya VSA Remote Monitoring and Management (RMM) systems. Kaseya VSA is a primaryproduct CU*Answers uses to monitor and manage client and CU*Answers workstations, though not the onlyone. After confirming the validity of the communication, CU*Answers shut down all Kaseya VSA systems onthe network.Disabling Kaseya VSA affected CU*Answers monitoring of its internal infrastructure and that of our managedclients. While Kaseya VSA remained disabled in anticipation of remediation from Kaseya, CU*Answerscontinued to provide managed services through other tools or used workarounds whenever possible.CU*Answers also followed guidance provided by the vendor and from external resources such as CISA.CU*Answers prepped the Kaseya VSA systems in accordance with this guidance in anticipation of thepatches. CU*Answers reached out to our critical vendors requesting whether they were affected by this event.Our team also responded to an inquiry from the State of Michigan Department of Insurance and FinancialServices.On July 11, CU*Answers was able to restore Kaseya VSA services for both internal and managed clients. Toensure appropriate security was in place for clients, CU*Answers developed a plan with Kaseya to allowclients to have remote access through VPN connectivity only. Between July 14 and July 23, CU*Answersworked with clients on an individual basis to create the necessary VPN infrastructure to allow remote access.In addition to the details of CU*Answers’ response below, CU*Answers has included some additionalinformation and opinions regarding this incident. These include: AuditLink’s communication with our critical vendors. CU*Answers’ response to the Michigan Department of Insurance and Financial Services inquiry. A recap of our post-mortem call with Kaseya. A brief description of our general response if the CU*Answers network is affected by ransomware. An opinion on the role and limitations of vendor risk assessments regarding service provider andsupply chain incidents.Please reach out to our teams if you have further questions.

GENERAL TIMELINE SUMMARYAs this was a highly complex event, the following is a basic timeline of events and responses to this incident:DATEJuly 2, 2021EVENTS AND RESPONSESKaseya notifies CU*Answers of the ransomware attack.CU*Answers confirms validity of the notification and shuts down the Kaseyaproducts. CU*Answers activates its Incident Response Protocol. CU*Answers followsguidance from Kaseya, the FBI, and CISA regarding responses to this attack.CU*Answers notifies clients of the incident and the powering off the Kaseya systems.CU*Answers performed a manual review of indicators of compromise based oninformation available through the press and online communities.CU*Answers receives a tool from Kaseya to test whether the Kaseya VSA systemwas compromised. The tool reported no evidence of compromise.July 4, 2021July 6, 2021July 7, 2021July 8, 2021July 11, 2021July 13, 2021July 14 – July23, 2021July 23, 2021CU*Answers receives a tool from Trend Micro, our Anti-Malware provider.CU*Answers runs an aggressive scan, and the tool reported no evidence ofcompromise.CU*Answers receives a new tool from Kaseya to test for compromise. CU*Answersruns the new tool, and again receives a report of no evidence of compromise.CU*Answers provides clarification on what services were impacted by the Kaseyaoutage, and what services are still fully functional for managed clients.AuditLink sends out a request to CU*Answers’ critical vendors asking whether theirservices were impacted by the Kaseya event. No vendor reported an impact.CU*Answers receives an inquiry from the Michigan DIFS asking whether CU*Answerswas impacted by the Kaseya attack.CU*Answers receives instructions on how to prepare Kaseya VSA systems for thepatch and spends the next few days prepping the systems in anticipation of thepatch.CU*Answers responds to the Michigan DIFS inquiry.CU*Answers notifies clients of a two-step plan to re-enable Kaseya systems afterpatching.Kaseya releases the patch for the vulnerability. CU*Answers immediately appliesthe patch and performs recommended hardening of the devices. Kaseya VSAsystems were brought back online same day. Per the plan announced on July 8,CU*Answers did not turn on client access immediately. Otherwise, managed clientsupport was returned to normal.CU*Answers announces the plan to allow access to the Kaseya VSA ManagementGUI through VPN access from each client’s own network only. CU*Answers reachesout to each individual client to explain the process for creating this new accesscontrol.CU*Answers works with individual clients to bring up remote access to the KaseyaVSA system through VPN.CU*Answers has a conference call with Kaseya to discuss the event, lessonslearned, and provide additional information on future support and securityenhancements.Kaseya Incident Response Page 4 of 16

INCIDENT DETAILSOn Friday, July 2, the notorious cyber-criminal gang known as REvil used a vulnerability to attack on-siteKaseya VSA systems with the intent to distribute ransomware to clients. The attack date of July 2 was almostcertainly chosen intentionally to catch victims off-guard before the Fourth of July U.S. holiday. At 2:58 PMET, several blast emails were sent by Kaseya, and the CU*Answers Accounting Team promptly notified ourInternal Audit Team. Internal Audit communicated with Network Services, and the Network Services Teamrallied the key players to determine the validity of the communication and responses. CU*Answers shut downall Kaseya systems by 3:20 PM ET. Notification to clients went out same day at 4:15 PM ET. On the sameday, CU*Answers did a manual review of known/rumored indicators of compromise (accounts andprocedures) based on information in the press/community. No evidence of compromise was found, and as ittuned out the reports from the press and various communities were later confirmed to be accurate.Because this incident affected a service provider, CU*Answers could not remediate this issue on our own. OurIncident Response Protocol dictated we engage as follows after shutting down the Kaseya systems:1. Review all systems to determine whether client or internal CU*Answers assets had beencompromised.2. Contact vendors of other network security controls to determine what options CU*Answers wouldhave if additional layered controls were needed.3. Provision of services to clients and review service alternatives during the time Kaseya VSA wasoffline.4. Pressure Kaseya into posting updates on status of their remediation.5. Update clients on the status.On Sunday, July 4, our teams received the tool from Kaseya to test systems for compromise. CU*Answerschecked each system to determine whether there was evidence of vulnerability. CU*Answers performed thetesting and determined no systems were affected by this attack. Kaseya systems were taken back offline oncethe test was completed.In addition, Network Services leveraged tools from our Anti-Malware provider, Trend Micro. NetworkServices ran an aggressive scan of all systems on July 4, and there was no evidence of compromise inCU*Answers or managed client networks.Kaseya Incident Response Page 5 of 16

On Tuesday, July 6, Kaseya offered additional tools to detect vulnerability. CU*Answers ran these new toolsthe same day, and these tools also returned no evidence of compromise.CU*Answers Network Services continued to provide service to clients and the network during the outages.Services that were provided through Kaseya were audited and we determined what services could be providedmanually or through other vendors and systems. The service impact to managed clients and to CU*Answersown network was as follows:Managing Microsoft Windows Systems. Kaseya is not the only network management tool we usefor managing client networks, but it is a primary one for Microsoft Windows systems.For Microsoft Windows Systems, unavailable services included an inability to gather system log files,some system status/availability alerting was not active, automation was not available, and the abilityfor CU*Answers to remote into systems proactively for maintenance was not available. Details on theimpacted services included the following:Daily ChecksKaseya Server Alarms (low disk alarms, Kaseya events, etc) – Alarm Monitoring module inKaseyaSAN & Virtualization checks (VMHosts, SAN health) – Kaseya Network Monitoring modulein KaseyaMachines Missing Trend – View in Kaseya’s Manage Agents moduleWeekly ChecksWeekly Server Report verification – generated & delivered via KaseyaFailed Logon Report, Patch Management Report review – generated & delivered via KaseyaWorkstations missing Kaseya agentsEnsure all PCs are in proper OU – Requires remoting onto client’s DCPCs assigned a LAN Cache – LAN Caches assigned via KaseyaNo Errors Deploying 3rd Party Software – Kaseya-generated reportMachines assigned to a 3rd Party Patching Profile – 3rd Party Profiles built in KaseyaEnsure machines have AV installed – No discovery built into Trend for missing AVAll client checks not listed had established workarounds and were completed onschedule.All Other Infrastructure. Network Services was still able to execute and monitor critical servicessuch as communications to CU*Answers, managed hosting, firewalls/remote access systems, endpointsecurity and backup services.At the same time, CU*Answers continued to follow FBI/CISA/Kaseya guidance on managing this incident.On July 7, Kaseya distributed a On Premises VSA Startup Readiness Guide to CU*Answers and other Kaseyaclients. Over the next few days CU*Answers went through the steps to prepare the systems for the Kaseyapatch, as described in the Readiness Guide. The seven steps included the following:Step 1 – Ensure your VSA server is isolatedKaseya Incident Response Page 6 of 16

Step 2 – Check your System for Indicators of Compromise (IOC)Step 3 – Patching the Operating Systems of the VSA ServersStep 4 – Using URL Rewrite to control access to VSA through IIS and Firewall RulesStep 5 – Install FireEye AgentStep 6 – Remove Pending Scripts/JobsStep 7 - VSA SQL Database AssessmentCU*Answers was advised not to start up our VSA Application until this VSA patch was applied.Acting in accordance with Kaseya instructions, CU*Answers announced to clients on July 8 a two-step processfor reactivating Kaseya systems after patching. First step was CU*Answers Network Services access to VSA,including restoration of all agent functions at credit unions. Second step was client access to VSA systems.On July 11, Kaseya released the patches to CU*Answers for immediate implementation. CU*Answers appliedthose updates the same day, as well as performing additional system hardening recommended by Kaseya. Inaccordance with other recommended guidelines, CU*Answers also put additional controls in place to furtherreduce the network footprint of the environment. CU*Answers received assurances that by patching andhardening, the system was now safe to operate. Therefore, CU*Answers brought Kaseya VSA services online,and returned managed client support to normal.CU*Answers also announced that client access into the environment would remain disabled as NetworkServices addressed minor functionality issues, and to confirm with Kaseya the best options for implementingclient access in the most secure and convenient way possible.By July 13, CU*Answers developed a secure approach in consultation with the vendor for remote clientaccess. CU*Answers announced to clients that remote client access to Kaseya VSA required VPN access, andaccess to the Kaseya VSA management interface would now only be possible from a client’s own network.Network Services contacted each client individually to review the requirement and start the VPNconfigurations. In addition, a new feature is that dual approval is required to start using new agent procedures.A new agent procedure was how the REvil attack executed delivery of ransomware, so dual approval requiresa second person to approve the agent procedure, rather than an attacker that may have already compromisedVSA credentials.On July 14, VPN access was built out for managed clients. SonicWall also offered a tool to assist with gettingall clients connected with remote access to the Kaseya system. From July 14 to July 23, CU*Answers workedwith clients on an individual basis to restore connectivity with the new VPN connection. After all clients wereconfirmed to be working, the incident was closed.On July 23, CU*Answers also met with Kaseya to discuss the event, lessons learned, and future enhancementsin light of the attack. The CU*Answers team also received individual technical support around the event toassist our team’s understanding of the event. A recap of the call is included below.AUDITLINK COMMUNICATION WITH OTHER VENDORSAs we did with the Orion SolarWinds incident, AuditLink made inquiries to our critical vendors to determineif any were affected by this Kaseya issue. On July 6, the following inquiry was sent out:Good afternoon,Kaseya Incident Response Page 7 of 16

We have recently been made aware of the Kaseya VSA Supply-Chain Ransomware Incident and arereaching out to critical vendors of CU*Answers as a part of our vendor management to determine theimpact, if any, and the steps being taken for containment and remediation. If you or one of yourvendors in your supply chain may have been impacted by this most recent compromise in some way,please let us know which system(s) or vendor(s) that would include and what impact you are able toascertain at this time. We understand that a problem of this nature and scope may take time to fullyidentify and uncover, but even early information could be valuable in restoring the supply chain andproviding security to clients.No critical vendors responded they were adversely affected by the Kaseya incident.RESPONSE TO MICHIGAN DIFSOn July 6, CU*Answers received a communication from the State of Michigan Department of Insurance andFinancial Services (DIFS) requesting information if we experienced an impact from the Kaseya event:Dear CEOs,As you are all aware we are in the middle of another major ransomware incident. Below isinformation regarding the Kaseya incident and guidance provided by the Cybersecurity &Infrastructure Security Agency (CISA). CISA and the FBI are strongly urging organizations to followthe guidance below.CISA-FBI Guidance for MSPs and their Customers Affected by the Kaseya VSA Supply-ChainRansomware Attack CISAUpdate Regarding VSA Security Incident KaseyaPlease continue to take prudent action to ascertain if you, or your vendors, have been impacted.Please let the Office of Credit Unions know as soon as possible if you have experienced or areanticipating an impact from this event.As always, do not hesitate to contact me with any questions/concerns.Regards,Christopher M. Fournier, CISA, CSCUES, BSACSInformation Technology ManagerOffice of Credit UnionsMichigan Department of Insurance and Financial ServicesOn July 8, CU*Answers responded as follows:Good afternoon Mr. FournierCU*Answers does use the affected Kaseya VSA products, here is a timeline of activity around ourresponse to this ongoing incident:Friday - 7/2 1:30 PM - notified by Kaseya that we should shut these products down until furthernotice. We did so within 30 minutes and notified all of our credit union clients. Kaseya at this timeKaseya Incident Response Page 8 of 16

listed 1) lack of admin access to the VSA console and 2) presence of a particular script as being signsof compromise. We reviewed systems and confirmed neither of these conditions were present.Sunday – 7/4 – Kaseya published a IOC toolkit and procedure doc. We obtained this and, afterremoving VSA systems physically from the network ran through the process finding no indication ofcompromise. Trend Micro, our endpoint provider also release signatures that would indicatecompromise and we ran those on CU*A and all managed client networks finding no indications ofcompromise.Tuesday – 7/6 – Kaseya updated their IOC tool and procedure – we repeated our testing with the sameresults – no indication of compromise.Today – Kaseya released guidance to further harden these VSA servers which we are confirming wehave in place, in expectation of patch installation which they are saying will be on Sunday.Further we are reviewing FBI/CISA guidance as we work through the process.At this point we have told our CUs that we have no indication of compromise and we will keep oursystems offline until Kaseya’s remediation is complete and a patch is available. While our creditunion’s are without the benefit of the Kaseya tool, we have carried forward critical managementprocesses with work arounds to ensure oversight is maintained. We applied the out of band Microsoftpatch last night and continue to monitor that situation as well.Let me know what questions you have.Matt Sawtell CU*Answers Network Services VP Managed Technology ServicesMichigan DIFS acknowledged the response on July 13 and appreciated the detail in CU*Answers’ response tothe inquiry.CALL WITH KASEYAOn July 23, Kaseya had a call with CU*Answers to discuss the incident and response. Kaseya explained theirplan was to update functionality around security and alerting. Kaseya believed this attack was a somewhatisolated incident, not minimizing what happened to the 51 affected clients. Kaseya believes they are the mostsecure RMM out there as a result of this incident and response. Kaseya agreed to work with CU*Answersdirectly on some of the technical issues CU*Answers questioned during the incident to enhance our team’stechnical knowledge. Kaseya was also asked to ensure they upgrade their incident response notification.CU*Answers has been promised additional details from Kaseya in a final report on the incident. CU*Answerswill share what we can, under our current contract, once this report has been published.WHAT WOULD CU*ANSWERS HAVE DONE IF AFFECTED?CU*Answers could have been one of the victims attacked by REvil before the notification went out and theKaseya systems were shut down. An obvious question is CU*Answers’ plan if our environment was affectedby this or a future ransomware attack. While our response could potentially change depending on the actualcircumstances of the attack, CU*Answers has both a recovery strategy and backups to critical systems torestore from. Core operations have real-time replication, while other important systems have encryptedbackups transmitted to an offsite location. We would never promise a ransomware attack would not bedisruptive to our operations, but we also believe CU*Answers is well positioned to recover. CU*Answers hasKaseya Incident Response Page 9 of 16

had discussions with other service providers that had customers affected by ransomware to understand theirapproach, which has almost universally mirrored our disaster recovery plan.CU*Answers has more information included in our knowledge base.LIMITATIONS WITH VENDOR RISK MANAGEMENTAnother reasonable question is on the CU*Answers vendor management response to this incident. In ouropinion, Kaseya was responsive, clear, and timely with their communications and released tools to identify ifsystems have been compromised. Kaseya also worked diligently to get patches published and provideadditional control recommendations. Kaseya was attacked because their offering is part of a service providerand supply chain system. Attacking service providers and the supply chain represents a “one to many”equation: successfully compromise one provider and attack everyone else down the chain. Criminalorganizations and state sponsored actors have learned that compromising a distributed service provider such asOrion SolarWinds changes the economy of the effort. Note that SolarWinds was a vastly different type ofattack, where SolarWinds own source code was compromised. Kaseya’s attack was against a vulnerability thatrequired patching. Our response would have been significantly different if Kaseya had suffered the same typeof attack as SolarWinds.Kaseya is not considered a critical vendor under CU*Answers’ vendor management program becauseCU*Answers does not provide Kaseya with sensitive information, and we can migrate to competitive products.CU*Answers maintained our core services internally and to managed clients even though Kaseya wasunavailable for several days. Any attempt to reclassify and place additional due diligence on supply chainvendors has two significant limitations. First, deciphering the upstream vendors of contracted providers can betricky or impossible. For example, a service provider could have several supply chain vendors as part of theirinfrastructure (Orion SolarWinds, SonicWall, AWS, etc.) that present these vulnerabilities. While AuditLinkhas done a great job reaching out to our critical vendors to see what their responses have been, such as they didwith Orion SolarWinds, not every vendor will provide us with reliable responses. There is no effective wayfor us to know, for example, whether critical patches or updates have been applied by our service providers.The other serious limitation is that in many ways these attacks are vendor agnostic. The issue is not whichvendor provides the service, but rather the nature of the service itself. For example, if you were to predict anetwork monitoring system that could be affected by a supply chain attack, Orion SolarWinds would be alikely target because Orion is one of the largest service providers with government and enterprise-levelservices. On the other hand, however, Kaseya is small with just a fraction of the overall market share forautomated IT management services. Kaseya is emphatically not a likely target for this kind of supply chainattack. So, while we can flag vendors that have a documented failure to update their systems with criticalpatches provided they have a SOC report or other documentation, we do not have an effective way ofpredicting who might actually be a target. Going without a service such as Kaseya means patching must bedone on a manual basis, which significantly increases the chance of human error. Attempting to develop ourown product is time intensive and challenging, and not an option for systems that require physical hardware.COMMUNICATIONS WITH CLIENTSThe initial communication and updates posted to CU*BASE Alerts regarding Kaseya were as follows:July 2 Communication to Clients and Initial Kaseya VSA CommunicationKaseya Incident Response Page 10 of 16

Clients,CU*Answers received a notice from Kaseya today advising us to shut down our Kaseya VSAsystems. We have followed the directions and have shut down all Kaseya VSA systems, as well asany network access to those systems as an added precaution. We have reviewed systems for indicatorsof compromise but have not found any.The current impact will be that we cannot monitor your systems during this time and services such asremote desktop sharing and automated patching will be unavailable.We will continue to monitor the situation with Kaseya for updates throughout this incident.If you have any questions or concerns please let us know.The notice CU*Answers received from Kaseya is below:------------------We are experiencing a potential attack against the VSA that has been limited to a small number of onpremises customers only as of 2:00 PM EDT today.We are in the process of investigating the root cause of the incident with an abundance of caution butwe recommend that you IMMEDIATELY shutdown your VSA server until you receive further noticefrom us.Its critical that you do this immediately, because one of the first things the attacker does is shutoffadministrative access to the VSA.We will be updating our support article at the link s/4403440684689July 3 CommunicationGood afternoon everyone – wanted to get an update out as we are at the 24-hour mark.At this time our Kaseya systems remain offline and this will continue until we get some actionableinformation on the event.We will continue to monitor the situation throughout the weekend and provide updates.If you have any questions please let us know.-----You can read the most recent update from Kaseya /4403440684689July 5 CommunicationGood evening – here is what has happened since the last update on 7/3:Kaseya Incident Response Page 11 of 16

1) Kaseya released an IOC (Indication of Compromise) tool that would report the signs ofcompromise they’ve seen from affected installation across their user base. We were able to run againstour Kaseya implementations and the tool reported that none of those indicators were present on oursystems.2) Trend Micro release signatures that would indicate compromise at client sites. Our Trend Micromanagement console did not detect indicators present at any client sites.3) Kaseya has continues to update the timeline to recover their SAAS environment. Once that iscomplete, which they anticipate in the next 24 hours, they will start building out a release schedule forthe patch we will need to apply to our systems.We will continue to keep our systems offline. This will not affect your ability to use CU*Base or otherline of business applications. It does mean that our ability to remotely monitor and manage yournetwork, as well as automated patch application will still be unavailable. We will have work around inplace to provide remote support should you need it at open of business tomorrow.We will continue to monitor the situation. If you have any questions please let us know.You can continue to monitor Kaseya’s update /4403440684689July 6 CommunicationOn the afternoon of Friday 7/2/21, we received notice from Kaseya, one of our network managementtool vendors, that their software was being targeted by bad actors attempting to spread ransomware.Because attacks were actively taking place, Kaseya advised that their tools be shut down until theycould develop and publish appropriate patches and mitigations. CU*Answers Network Services tookimmediate action to activate our Incident Response Plan and shut down our Kaseya managementsystems.We have since been provided with a set of tools from Kaseya to determine if our systems weretargeted by bad actors prior to shut down, and our results have been clean. At this point we have noindication that our systems were compromised by bad actors.Service Impact: Kaseya is not the only network management tool we use for managing clientnetworks, but it is a primary one for Microsoft Windows systems. For those and some others we arenot able to gather system log files, some system status/availability alerting is not active, automation isnot available, and our ability to remote into systems proactively for maintenance is not available. Weare still able to execute and monitor critical services such as communications to CU*Answers,managed hosting, firewalls/remote access systems, endpoint security and backup services.We are still here to serve you! CNS is still available to serve you during this time. We have alternatemethods for assisting you during service calls. These methods may take a little longer than normal, orwe may need your assistance such as jumping on a screen share through Zoom. We ask that youplease be patient with us during this time until we are able to return all of our systems to productionuse.Do you have a contingency plan? The good thing about using commercial tools for networkmanagement is that we can pivot to another vendor should that be necessary. If Kaseya shows signs ofnot returning to normal use for an extended period of time, CNS will initiate plans to switch to anKaseya Incident Response Page 12 of 16

alternate vendor. However, at the moment we expect they will have fixes available this week and weshould be able to return to normal operations soon.July 7 CommunicationKaseya continues to work on releasing fixes for the vulnerability. Our Kaseya servers remain offlinepending their release. Once we have the updates, they will be evaluated by our team prior to returningsystems to production.CNS remains ready to assist you and your team. We are still able to execute and monitor criticalservices such as communications to CU*Answers, managed hosting, firewalls/remote access systems,endpoint security and backup services. Please contact the Help Desk at x266 for as

down their Kaseya VSA Remote Monitoring and Management (RMM) systems. Kaseya VSA is a primary product CU*Answers uses to monitor and manage client and CU*Answers workstations, though not the only one. After confirming the validity of the communication, CU*Answers shut down all Kaseya VSA systems on the network.