WIXLIB

Click to edit Master title style IntegrityClick to editMaster text stylesMonitoring Second level Thirdlevel in the filesystemDetectingchanges Fourth level Fifth levelKlaus MöllerWP8-T1Webinar, 7th of August 2020Publicwww.geant.org07/08/201 www.geant.org1

What is Integrity Monitoring? “the process of validating the integrity of operating systemfiles and directories” Integrity: file/directory content and metadata are unchangedwith regards to a given “known good” state However: changes to files and directories are intented––Updates: OS, software etc.Configuration changes: users, network (addresses), settings, etc. Detecting unauthorized or unintended changes–Those made by attacks or mistakes File Integrity Monitoring (FIM)2 www.geant.org

Why is (File) Integrity Monitoring useful? Assessing the impact of integrity violations–Which changes were made? I.e. new firewall rules, new users, changed daemon/serviceconfigurations, unparseble configurations/librariesChanged binaries/libraries/kernel (modules/drivers), .Additionally installed or removed files?––Crypto-miners, Spam-SW, phishing pages, AV, firewall, FIM, What happened? How did it happen? Detecting unintended changes–Critial: configuration mistakes that open weaknesses Empty passwords, disabling authentication, 3 www.geant.org

Integrity monitoring workflow4. Act––––Valid state?Change deliberately orunintentionally?Consequences?Preventable in the future?1. Plan: what to monitor, howto monitor––Systems, files, directoriesAttributes: content, permissions,etc.3. Check: at regularintervals––Examine attributes ofmonitored files &directoriesCompare exam results withbaseline Report2. Do: Take Baseline–Record valid state(s)4 www.geant.org

Plan: what systems should be monitored? Rule of thumb: By order of impact/mission criticality––Look at your Business Impact Analysis (if present)Identity management, authentication databases/servers –Systems storing your mission critical data –Database servers, file servers, backup serversSecurity critical systems –I.e. KDCs, Domain Controllers, LDAP servers with authentication informationCompromising these will compromise most other systemsFirewalls, SIEM, loghost, Other mission critical systems Webservers, application servers, load balancers, VM-hosts, centralswitches/routers, central DNS, central Email, HR, CRM, 5 www.geant.org

Plan: what files should be monitored? Trusted computing base–Kernel, kernel modules/drivers –Binaries, libraries –/bin, /usr/bin, /lib, /usr/libC:\Windows\System32Directories in PATH (Linux) or %PATH% (Windows)System configuration –/boot, /lib/modules, C:\BOOTMGR, C:\Boot\BCDLinux/Unix: /etcWindows: RegistryCritical files in Home directories /.ssh/authorized keys, /.config6 www.geant.org

Plan: Limited checks Sockets, named pipes, IPC objects––– Symlinks– Not all FIMs will monitor where the symlink points toConfidential data––– Reading (i.e. checksumming) will likely blockInode number will change when socket gets re-created at bootPermissions, ownership, major/minor device number can be monitoredKey material, esp. private keysNo text diffsMay show up in text diffs or logsTemporary filesystems/directories––/tmp, /usr/tmp, /var/tmp, /dev/shm, /run/user/, /etc/mntabPermissions (sticky bit) are OK7 www.geant.org

Plan: What to exclude from checking? Ephemeral/dynamic file systems–– Network file systems–– /proc, /sys, /dev, /etc/mntabToo many changes in operation to be usefulNFS, CIFS, AFS, etc.Check these on the server – not over the networkRemovable media––USB/flash drives, CD/DVD/BD, Floppy(?)Content will change with different media mounted8 www.geant.org

Plan: what attributes should be monitored? Content, of course–––– Permissions/ACLs––– S-UID/S-GID bitsWrite permissions on configuration files for ordinary users?Read permissions for world appearing on confidential data?Owner, Group– Complete file? – that’s called a backup;)Usually cryptographic checksums: SHA256, (too often still MD5, SHA1)For very large files ( 1 GByte), checksumming may take too longFull content for small (vital) text files - allows diff to show changesSystem binaries/libraries should be owned by rootSize––Binaries, libraries should not change size – except through updatesOthers (log files) should only grow – what about log rotation?9 www.geant.org

Plan: what attributes could be monitored? Device ID, Inode:– Number of links:–––– Somebody might have replaced stuff with mounts to another filesystemEach file in a directory has one link to it, plus itself and the parent directoryHidden files/directories will show up as mismatch on link countWorks well on Ext2/3/4, vfat, and (old) standard Unix filesystemsDoes not work with modern filesystems: XFS, Btrfs, on (Windows), Change (of metadata/inode: Linux/Unix)B(orn) or D(eleted) timestamps – if supported by filesystemTimestamps can be changed by attackers Even creation with root privileges & anti-forensic tools10 www.geant.org

Do: Baselining Naive: find / -print0 xargs -0 sha256sum /tmp/my.dbBaseline must be secured against tampering/loss–––– Best done by keeping on a central serverSame goes for configuration of the FIMIf kept locally, sign digitally, check before useAvailability issues: deleted locally, no network, how to act then.Baseline must be taken form a “known good”/valid/legal state–––After a fresh/complete install?After initial setup?Patches, updates, later installs?11 www.geant.org

Check How often to check?–––– What to report?– Depends, anywhere between once/hour and once/dayMore checks – more work, more load on the systemsOTOH: checking more often may spot attacks earlierIdeal: real-time monitoring for changes (Linux: inotify system call)Need actionable data: Report Background enough information to draft aplan to act uponHow to report changes?–––Log messages (syslog, eventlog) – best to SIEM/central loghostEmail (standalone systems)Console log?12 www.geant.org

Act: WorkflowCheck rly?FalsePositivesAdaptConfig13 www.geant.org

FIM: tabaseFile systemeditAdminConfig14 www.geant.org

How to start? Begin small–– Observe, adapt, expand––– One or two servers, only a handful of filesCan be implemented on spare hardwareLearn how and when changes happen and whyAdapt your configurationWrite down in knowledge baseExpand bit-by-bit––Have a plan (what to monitor)It’s better to observe too few things than too much15 www.geant.org

Wazuh Live Demonstration Configuring syscheckAdding/deleting a fileChanging the content of a fileLooking into events/reports16 www.geant.org

Wazuh: AgentSource: https://documentation.wazuh.com Full Host Intrusion Detection System (HIDS)––––Syscheck: Integrated FIMRootcheck: configuration check & rootkit detectionLog collector: Event & log file monitoring/forwarding (Filebeat)Modules Manager: Place to plug-in user defined (scan) modules17 www.geant.org

Wazuh: Server Analysis Daemon–Decodes and analyses incoming logs & events Remote Daemon: Agent management Elasticstack: Kibana, Filebeat–Analysis (ElasticSearch)–Log/Event forwarding reception (Filebeat)–Dashboard (Kibana)Source: https://documentation.wazuh.com18 www.geant.org

Limits of FIM: Malware Some malware doesn’t write anything to the filesystem––– Rootkits hide files/directories from every user–– What’s not there, can’t be foundBut most malware needs a means of persistence: Autostart keys, kernel modules,boot loader/parameters, etc.This will leave tracesWhat is not visible can’t be checked or seenBut hiding a file/subdir also alters the parent directory: Timestamps, Link counts,etc.A thorough check will detect somethingBut it’s up to the admin to pick up on strange reports19 www.geant.org

Limits of FIM: File signature evasion Find a collision,–––I.e. a file that has the same cryptographic hash sum as the originalCan be done with weak/broken hash algorithms, like MD5 or SHA1Very rare in practice Do not confuse with cases where valid Authenticode signatures were usedThese were made with leaked/stolen certificatesMitigation–––Multiple checksums – attacker has to find collisions for all employedhash algorithmsStronger hash algorithms: SHA256, SHA-512, SHA-3, etc.Full content comparision, i.e. diff20 www.geant.org

Other uses for file hashes: Virustotal Unknown file, good ormalicious?–Scan with your own Anti-Virus –– What if it says nothing?Use more AV-Scannerhttps://virustotal.comCan’t/won’t send file––––Malware upload blockedMay contain sensitiveinformationSearch by cryptographic hashmd5, sha1, sha25621 www.geant.org

What have you learned? What integrity monitoring (at the OS level) is How to do integrity monitoring How to configure the integrity monitoring softwareWhat has been left out? Boot process integrity–TPM, secure boot (MS), EVM/LMA (Linux) Binary signing under (elfsign - Linux, Authenticode - Windows) Cryptographic signing of files (PGP, S/MIME)22 www.geant.org

Click to edit Master title style Click to edit Master text styles Second level Third levelThankyou Fourth levelAny questions? Fifth levelNext module: Network 1st Hop Security, 11th of August 2020www.geant.org07/08/20 GÉANT Association on behalf of the GN4 Phase 2 project (GN4-2).The research leading to these results has received funding fromthe European Union’s Horizon 2020 research and innovationprogramme under Grant Agreement No. 731122 (GN4-2).23 www.geant.org23

References Kim, Gene H.; Spafford, Eugene H. (1994). "The Designand Implementation of Tripwire: A File System IntegrityChecker"– Lawrence Grim: “IDS: File Integrity Checking”– ction/ids-file-integrity-checking-35327OSSEC Host-Based Intrusion Detection Guide– https://dl.acm.org/doi/10.1145/191177.191183Rory Bray, Daniel Cid, Andrew Hay, Syngress, 2008, ISBN: 978-1597492409Host integrity monitoring using OSIRIS and Samhain–Brian Wotring, Syngress, 2005, ISBN-13: 978-159749018424 www.geant.org

Some Open Source FIM software Tripwire: the grandparent of many FIM software (1992)– Aide: Advanced Intrusion Detection Environment– http://afick.sourceforge.net/Samhain: Linux FIM with additional monitoring of kernel datastructures– https://aide.github.io/Afick: Another File Integrity ChecKer– tps://www.la-samhna.de/samhain/OSSEC, Wazuh: Full open source HIDS with FIM h.com/25 www.geant.org

Wazuh Live Demonstration Wazuh Server Appliance–– Kali Linux– m/vm/wazuh3.10.2 7.3.2.ovahttps://www.kali.org/downloads/Windows 10 from Microsoft Evaluation -365/windows26 www.geant.org

Click to edit Master title style Click to edit Master text styles Second level Third levelBackupmaterial Fourth levelFifthitlevelStuff that didn’t makedue to time constraintswww.geant.org07/08/20 GÉANT Association on behalf of the GN4 Phase 2 project (GN4-2).The research leading to these results has received funding fromthe European Union’s Horizon 2020 research and innovationprogramme under Grant Agreement No. 731122 (GN4-2).27 www.geant.org27

Decoding Wazuh file modes/*/* definetypes.types. */*/S IFDIRS IFDIRS IFCHRS IFCHRS IFBLKS IFBLKS IFREGS IFREGS IFIFOS IFIFOS IFLNKS IFLNKS IFSOCKS y. */*/CharacterCharacter device.device. */*/BlockBlock device.device. */*/RegularRegular file.file. */*/FIFO.FIFO. */*/SymbolicSymbolic link.link. */*/Socket.Socket. */*//*/* ProtectionProtection bits.bits. */*/#define#define S ISUIDS ISUID0400004000#define#define S ISGIDS ISGID0200002000#define#define S ISVTXS ISVTX0100001000/*/* SetSet useruser IDID onon execution.execution. */*//*/* SetSet groupgroup IDID onon execution.execution. */*//*/* SaveSave swappedswapped texttext afterafter useuse (sticky).(sticky). */*/#define#define S IREADS IREAD#define#define S IWRITES IWRITE#define#define S IEXECS IEXEC/*/* ReadRead byby owner.owner. */*//*/* WriteWrite byby owner.owner. */*//*/* ExecuteExecute byby owner.owner. */*/04000400020002000100010028 www.geant.org