Transcription
Logs for IncidentResponseAnton Chuvakin, Ph.D., GCIA, GCIH, GCFAChief Logging EvangelistMitigating Risk. Automating Compliance.LogLogic ConfidentialMonday, June 23, 20081
Logs for Incident InvestigationsA few thoughts to start us off All attackers leave traces. Period! It is just that you don’t always know whatand whereAnd almost never know why Logs are the place to look, firstMitigating Risk. Automating Compliance.Confidential Monday, June 23, 20082
GoalsLearn/refresh about logs and loggingRefresh our knowledge of incident responsepracticesLearn how various logs are used at variousstages of incident responseLearn about log forensicsLearn how to insider-proof loggingMitigating Risk. Automating Compliance.Confidential Monday, June 23, 20083
Outline - IIncident Response (IR) ProcessLogs OverviewLogs Usage at Various Stages of theResponse ProcessHow Log from Different Sources Help IRMitigating Risk. Automating Compliance.Confidential Monday, June 23, 20084
Outline - IIStandards and Regulation Affecting Logs andIncident ResponseIncident Response vs ForensicsLog Analysis and Incident Response MistakesBonus: Logs vs InsidersBonus: Logs Honeytokens Case StudyMitigating Risk. Automating Compliance.Confidential Monday, June 23, 20085
Incident Response ProcessIncident Response ProcessMitigating Risk. Automating Compliance.Confidential Monday, June 23, 20086
Incident Response Methodologies: SANSSANS Six-Step E]radication[R]ecovery[F]ollow-UpMitigating Risk. Automating Compliance.Confidential Monday, June 23, 20087
Incident Response Methodologies: NISTNIST Incident Response 800-611. Preparation2. Detection and Analysis3. Containment , Eradication and Recovery4. Post-incident ActivityMitigating Risk. Automating Compliance.Confidential Monday, June 23, 20088
Why Have a Process?It helps – Predictability– Efficiency– Auditability– Constant ImprovementStage 1Stage 2aStage 4Mitigating Risk. Automating Compliance. Monday, June 23, 2008Stage 2cStage 3It shrinks – Indecision– Uncertainty– Panic!ConfidentialStage 2b9
Example: Worm “Mitigation” in a Large Company circa 2002 AD Worm hitsPanic initial response in parallel (urgh! )Mitigation investigation at the same timeTwo walking steps forward and 10 runningsteps back Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200810
From Incident Response to LogsFrom Incident Response toLogsMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200811
DefinitionsLog record related towhatever activities occurringon an information system,event record standard definitions are coming soon:CEE standard by MITRE(http://cee.mitre.org)Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200812
Terms and DefinitionsLoggingAuditingMonitoringEvent reportingLog analysisAlertingMessage – some system indicationthat an event has transpiredLog or audit record – recordedmessage related to the eventLog file – collection of the aboverecordsAlert – a message usually sent tonotify an operatorDevice – a source of securityrelevant logsMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200813
Login? Logon? Log in? 18 Dec 17 15:45:57 10.14.93.7 ns5xp: NetScreen device id ns5xp systemwarning-00515: Admin User netscreen has logged on via Telnet from10.14.98.55:39073 (2002-12-17 15:50:53) 57 Dec 25 00:04:32:%SEC LOGIN-5-LOGIN SUCCESS:Login Success[user:yellowdog] [Source:10.4.2.11] [localport:23] at 20:55:40 UTC Fri Feb 282006 122 Mar 4 09:23:15 localhost sshd[27577]: Accepted password for kyle from::ffff:192.168.138.35 port 2895 ssh2 13 Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Success AuditENTERPRISE Account LogonLogon attempt by: MICROSOFT AUTHENTICATION PACKAGE V1 0 Logon account: POWERUSER Source Workstation: ENTERPRISE Error Code: 0xC000006A 4574Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200814
Log Data OverviewFrom Where?What logs?Audit logsFirewalls/intrusion preventionTransaction logsRouters/switchesIntrusion logsIntrusion detectionConnection logsServers, desktops, mainframesSystem performance recordsBusiness applicationsUser activity logsDatabasesVarious alerts and othermessagesAnti-virusVPNsMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200815
Devices that Log: An Attempt at a Comprehensive ListNetwork gear: routers, switches,Security gear: firewall, IDS, VPN, IPS, etcAccess control: RAS, AD, directory servicesSystems: OS (Unix, Windows, VMS, i5/OS400, etc)Applications: databases, email, web, client applicationsMisc: physical access, other non-IT technologiesOther: just about everything with the CPU Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200816
CONFIGURE LOGGINGConfigure LoggingMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200817
GuidanceFirewalls and network gear– Connections, access, firewall healthUnix– Syslog, PS accounting, binary auditWindows– Windows event logsMail servers– Email traffic, errors, accessWeb servers– Access, errorsMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200818
Cisco IOS Boxes#config term#logging 10.1.1.1#write memNOTE:There are more options available, but this gets the default log settingMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200819
Cisco Cats #set logging server 10.1.1.1#set logging server enableNOTE:There are more options available, but this gets the default log settingMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200820
Unix/Linux SyslogDefault syslog is OK, butleaves a lot out Nov 12 00:19:01 sparky su: (to nobody) root on noneNov 12 00:19:01 sparky PAM-unix2[14270]: sessionstarted for user nobody, service suNov 12 00:39:18 sparky PAM-unix2[14270]: sessionfinished for user nobody, service suSyslog514/UDPEasy centralized loggingMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200821/etc/syslog.conf## Warnings in one file#*.crit/var/log/warn## save the rest in one file#mail.*/var/log/maillog## enable this, if you want to keep all messages# in one file*.* @lx1000.example.com
Unix/Linux: Other LogsProcess accounting– Install psacct package– create a file e.g. /var/logs/audit– start accounting e.g. chkconfig psaccton and /etc/init.d/psacct startDetailed kernel audit– Solaris BSM, HP-UX, AIX Audit, SELinux– complex!Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200822
Windows LoggingMain Windows event logs:Application logSecurity logSystem logDomain controllers have two extra logs– File Replication service log– DNS Server logsMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200823
Windows Audit PolicyMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200824
Web ServersWeb servers ship with sensible logging defaultsApache– access log, error log, SSL logsMS IIS– W3C Extended files inc:\win\system32\logfiles\extXXXXXX.log– Errors in Windows Event logsMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200825
Email ServersSendmail– Syslog into /var/log/mailogMS Exchange– Errors in Windows Event logs– Plus, file-based logs (SMTP, diagnostics,message tracking, subject, etc) – use ExchangeManagerMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200826
DatabasesOracle– Change init.ora file to have audit trail db– Restart the database– Run audit statements: audit {statement privilege}[by user] [by {session access}] [ whenever{successful unsuccessful}];Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200827
Oh, Horror! – Musings on Logging DefaultsServer OS (Unix, Windows): authentication – yes, fileaccess - noDatabases: authentication – yes, changes – no, dataaccess – noFirewalls - connection blocked – yes, connectionsallowed – sometimes, configuration changes - noMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200828
Natural Flow of Log Management: How People Enable Logs1.Firewalls, network gear2.Other network security gear3.Servers (Unix, then Windows)4.Other server applications (web, mail)5.Databases6.Applications7.DesktopsMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200829
LOG ANALYSISLog AnalysisMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200830
Log Analysis: WhySituational awareness and new threat discovery–Who is doing what on a serverGetting more value out of the network and securityinfrastructure–Firewall logs for IDMeasuring security (metrics, trends, etc)–Top users by bandwidth from firewall logsCompliance and regulations (oh, my!)–Report on access to credit card data in a databaseIncident response (last, but not least!)Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200831
Log Analysis: Why NOT“Real hackers don’t get logged!” Why bother? No, really Too much data ( x0 GB per day)Too hard to doIs this device lying to me? No tools “that do it for you”– Or: tools too expensiveMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200832
Log Analysis Basics: Summary1.2.3.4.5.6.7.ManualFilteringSummarization and reportsSimple visualizationLog searchingCorrelationLog Data miningMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200833
Log Analysis Basics: ManualManual log review– Just fire your trusty ‘tail’, ‘more’, “notepad”, ‘vi’,Event Viewer, etc and get to it! Pros:– Easy, not tools required (neither build nor buy)Cons:– Try it with 10GB log file one day – Boring as Hell! Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200834
Log Analysis Basics: FilteringLog Filtering– Just show me the bad stuff; here is the list(positive)– Just ignore the good stuff; here is the list(negative or “Artificial Ignorance”)Pros:– Easy result interpretation: see- act– Many tools or write your ownCons:– Patterns beyond single messages?– Neither good nor bad, but interesting?Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200835
Log Analysis Basics: SummarySummarization and reports– Top X Users, Connections by IP,Pros:– Dramatically reduces the size of data– Suitable for high-level reportingCons:– Loss of information by summarizing– Which report to pick for a task?Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200836
Log Analysis Basics: VisualizationVisualization, from simple to 4D– A pie chart worth a thousand words?Pro– You just look at it – and know what it means andwhat to doCon– You just look at it – and hmmm .Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200837
Log Analysis Basics: SearchSearch– User specifies a time period, a log source or all,and an expression; gets back logs that match(regex vs Boolean)Pro– Easy to understand– Quick to doCon– What do you search for?– A LOT of data back, sometimesMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200838
Log Analysis Basics: CorrelationCorrelation– Rule-basedand other“Correlation” algorithmsPro– Highly automatedCon– Needs rules written by experts– Needs tuning for each siteMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200839“correlation”and
Log Analysis Basics: Log Data MiningLog mining– Algorithms that extract meaning from raw dataPro– Promises fully-automated analysisCon– Still research-grade technologyMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200840
Select Log Analysis ToolsLog collection–Syslog-ng, kiwi, ntsyslog, LASSO, DAD, Apache2syslog, etcSecure centralization–Stunnel, ssh, free IPSec VPNsPre-processing–LogPP, MS LogParser (Windows - text)Storage–MySQL or design your ownAnalysis – oooh, a tough one! ––––SEC for rule-based correlationSLCT and loghound for simple clusteringOSSEC, OSSIM, Prelude for [some] intelligenceSwatch, logwatch, logsentry, other match-n-bug scriptsMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200841
Back to Incident ResponseBack to Incident Response:How Logs HelpMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200842
Reminder: SANS Incident Response radication[R]ecovery[F]ollow-UpMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200843
Logs at Various Stage of Incident ResponsePreparation: verify controls, collect normal usage data,baseline, etcIdentification: detect an incident, confirm incident, etcContainment: scope the damage, learn what else is “lost”,what else the attacker visited/tried, etcEradication: preserving logs for the future, etcRecovery: confirming the restoration, etcFollow-Up: logs for “peaceful” purposes (training, etc) aswell as preventing the recurrenceMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200844
Using Logs at Preparation Stage1: PVerify ControlsOngoing MonitoringChange Management Support“If you know the cards, you’d live on an island” In general, verifying that you have control over yourenvironmentMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200845
Example 1 Logging Infrastructure for OptimumResponseMonitoring infrastructure based on NSM philosophy:netflow packet content logs (NIDS, etc)Pre- and post-incident monitoringUseful even if deployed after the incident, but mostuseful if deployed prior to itMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200846
Using Logs at Identification StageDetect Intrusion, Infections and AttacksObserve Attack Attempts, Recon and SuspiciousActivityPerform Trend Analysis and Baselining for AnomalyDetectionMine the Logs for Hidden Patterns, Indicating Incidentsin the Making “What is Out There?”2: IMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200847
Example 2 FTP Hack CaseServer stopsFound ‘rm-ed’ by the attackerWhat logs do we have?Forensics on an image to undelete logsClient FTP logs reveals Firewall confirms!Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200848
Sidetrack: What if Not Prepared – Logging DefaultsUnix (typical): system messages, login/logout, failuresWindows: system messages, login/logout, failuresWeb servers: access (some details), errorsDatabases: errors, restarts, NO access or changesFirewalls: varies (denied, NO allowed)Proxies: access, cachingVPNs: connections, login/logouts, errorsNIDS/NIPS: alerts, failuresMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200849
Sidetrack: What if Not Prepared – Local RetentionUnix (typical): varies – weeks to daysWindows: days to hours (!)Web servers: varies – weeks to daysDatabases: retainedFirewalls: no data (or days to hours)Proxies: no dataVPNs: no dataNIDS/NIPS: no data (or weeks to days)Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200850
Using Logs at Containment StageAssess Impact of the Infection, Compromise, Intrusion,etcCorrelate Logs to Know What You Can [Still] TrustVerify that Containment Measures Are Working“What Else is Hit?”3:CMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200851
Example 3 But Did It Spread?“A classic”: regular desktop starts scanning internallyCut from the network soon after: an incident isdeclaredAn impressive array of malware is discovered; AV isdeadProblem solved? Did it infect anybody else?!Logs from firewalls and flow to the rescue Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200852
Using Logs at Eradication StagePreserving the Log Evidence from Previous Stages–Especially if court action is likely or possible (see Forensics)Confirming that Backups are Safe–Using Logs, How Else? “Is it Gone?4: EMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200853
Example 4 Logs for [Possible] LitigationDeliberations on the log retention (and destruction!)policy: IDS, VPN, firewalls, servers – oh, my!Decided: IDS – longest; server – next; firewalls, VPN –shortestCase: financial information leaked to the mediaInvestigation points to a specific userDid he do it?!!Well, the answer died with 6-mo old VPN logs Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200854
Using Logs at Recovery StageIncreased Post-Incident MonitoringWatch for RecurrenceWatch for Related Incidents Elsewhere“Better Safe than Sorry”5: RMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200855
Example 5 When They Come Back Password guessing hack: non-root account passwordguessedIRC bot, scanning, phishing site setup, etcPassword changed; attacker files cleanedMore guessing attempts across the network– are thosethe same folks?Will they succeed again?Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200856
Using Logs at Follow-Up StageTrain Analysts, Responders and AdministratorsCreate Management ReportsVerify and Audit Newly Implemented Controls“We know we are OK”6: FMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200857
Example 6 Logs for Responder ating Risk. Automating Compliance.Confidential Monday, June 23, 200858
Sidetrack: Incident Record Keeping and Log RetentionRetention policy for routine and incident logs#1: Human action logs – the longest!–Logs created during incident responseBefore planning any log retention policy changes –define incident and routine log retentionSpecifically Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200859
LOG RETENTION – A TRIVIAL MATTER?Log RetentionMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200860
What is Log Retention?Q: When is “log storage” considered “log retention”?A: Log retention Log storage Accessibility Log destructionMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200861
What is NOT Retention?A database that stores a few fields from each logA tape closet with log data tapes that were neververifiedA syslog server that just spools logs into filesMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200862
Sidetrack: Why Destroy Log Data?Log destruction? You’ve got to be kidding Why you need to destroy logs sometimes?How to destroy them?Mitigating Risk. Automating Compliance.Confidential Monday, June 23, 200863
Retention Time QuestionI have the answer! No, not really.Regulations?– Unambiguous: PCI – keep’em for 1 year–1 yr 1 mo is also common (and so it 39 mos)Tiered retention strategy– Online– Nearline– Offline/tapeMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200864
Example: Retention StrategyType network storage tierIDS DMZ online 90 daysFirewall DMZ online 30 daysServers internal online 90 daysALL DMZ archive 3 yearsCritical internal archive 5 yearsOTHER internal archive 1 yearMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200865
Retention Strategy HOWTO1.Assess applicable compliance requirements2.Look at risk posture3.Look at various log source and their log volumes4.Review available storage options5.Decide on tiersMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200866
Log Storage Options1.RDBMSOracle, MySQL, etc2.Flat files“Files ”: Compressed, indexed, etc3.HybridCombine #1 and #24.Proprietary datastoreBuild from scratch to store logsMitigating Risk. Automating Compliance.Confidential Monday, June 23, 200867
What Makes It “Accessible?”Why store logs? Duh, so you can get to them later!Use case for logsResponse time toget logsSeconds to minutesRegulatoryTime frame oflogs neededWeeks tomonthsMonths toyearsUp to yearsE-DiscoveryUp to yearsHours to
Select Log Analysis Tools {Log collection – Syslog-ng, kiwi, ntsyslog, LASSO, DAD, Apache2syslog, etc {Secure centralization – Stunnel, ssh, free IPSec VPNs {Pre-processing – LogPP, MS LogParser (Windows - text) {Storage – MySQL or design your own {Analysi
