Transcription
Digital Forensics withOpen Source Tools
Digital Forensics withOpen Source ToolsCory AltheideHarlan CarveyTechnical EditorRay DavidsonAMSTERDAM BOSTON HEIDELBERG LONDONNEW YORK OXFORD PARIS SAN DIEGOSAN FRANCISCO SINGAPORE SYDNEY TOKYOSyngress is an imprint of Elsevier
Acquiring Editor: Angelina WardDevelopment Editor: Heather SchererProject Manager: Andre CuelloDesigner: Joanne BlankSyngress is an imprint of Elsevier225 Wyman Street, Waltham, MA 02451, USA 2011 Elsevier, Inc. All rights reserved.No part of this publication may be reproduced or transmitted in any form or by any means, electronicor mechanical, including photocopying, recording, or any information storage and retrieval system,without permission in writing from the publisher. Details on how to seek permission, furtherinformation about the Publisher’s permissions policies and our arrangements with organizations suchas the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website:www.elsevier.com/permissions.This book and the individual contributions contained in it are protected under copyright by thePublisher (other than as may be noted herein).NoticesKnowledge and best practice in this field are constantly changing. As new research and experiencebroaden our understanding, changes in research methods or professional practices, may becomenecessary. Practitioners and researchers must always rely on their own experience and knowledgein evaluating and using any information or methods described herein. In using such information ormethods they should be mindful of their own safety and the safety of others, including parties forwhom they have a professional responsibility.To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assumeany liability for any injury and/or damage to persons or property as a matter of products liability,negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideascontained in the material herein.Library of Congress Cataloging-in-Publication DataApplication submittedBritish Library Cataloguing-in-Publication DataA catalogue record for this book is available from the British Library.ISBN: 978-1-59749-586-8Printed in the United States of America11 12 13 1410 9 8 7 6 5 4 3 2 1Typeset by: diacriTech, IndiaFor information on all Syngress publications visit our website at www.syngress.com
ContentsAbout the Authors .xiAcknowledgments.xiiiIntroduction .xvCHAPTER 1 Digital Forensics with Open Source Tools . 1Welcome to “Digital Forensics with Open Source Tools” .1What Is “Digital Forensics?” .1Goals of Forensic Analysis.2The Digital Forensics Process .3What Is “Open Source?” .4“Free” vs. “Open”.4Open Source Licenses .5Benefits of Open Source Tools .5Education .5Portability and Flexibility .6Price .6Ground Truth .7Summary .7References .8CHAPTER 2 Open Source Examination Platform . 9Preparing the Examination System .9Building Software.9Installing Interpreters .10Working with Image Files .10Working with File Systems .10Using Linux as the Host .10Extracting Software .11GNU Build System.12Version Control Systems .16Installing Interpreters .16Working with Images .19Using Windows as the Host .26Building Software.26Installing Interpreters .27Working with Images .31Working with File Systems .34Summary .37References .37v
viContentsCHAPTER 3 Disk and File System Analysis . 39Media Analysis Concepts.39File System Abstraction Model .40The Sleuth Kit .41Installing the Sleuth Kit.41Sleuth Kit Tools .42Partitioning and Disk Layouts .52Partition Identification and Recovery .52Redundant Array of Inexpensive Disks .53Special Containers.54Virtual Machine Disk Images .54Forensic Containers .55Hashing .56Carving .58Foremost .59Forensic Imaging .61Deleted Data .61File Slack .62dd .64dcfldd .65dc3dd .66Summary .67References .67CHAPTER 4 Windows Systems and Artifacts . 69Introduction .69Windows File Systems .69File Allocation Table .69New Technology File System.71File System Summary .77Registry .78Event Logs .84Prefetch Files.87Shortcut Files .89Windows Executables .89Summary .93References .93CHAPTER 5 Linux Systems and Artifacts . 95Introduction .95Linux File Systems.95
ContentsFile System Layer.96File Name Layer .99Metadata Layer .101Data Unit Layer .103Journal Tools .103Deleted Data .103Linux Logical Volume Manager.104Linux Boot Process and Services .105System V .105BSD .107Linux System Organization and Artifacts .107Partitioning .107Filesystem Hierarchy.107Ownership and Permissions .108File Attributes .109Hidden Files .109/tmp.109User Accounts .110Home Directories .112Shell History .113ssh .113GNOME Windows Manager Artifacts .114Logs .116User Activity Logs .116Syslog .117Command Line Log Processing .119Scheduling Tasks .121Summary .121References .121CHAPTER 6 Mac OS X Systems and Artifacts . 123Introduction .123OS X File System Artifacts .123HFS Structures .123OS X System Artifacts .129Property Lists .129Bundles .130System Startup and Services .130Kexts .131Network Configuration .131Hidden Directories .132vii
viiiContentsInstalled Applications .133Swap and Hibernation dataData .133System Logs .133User Artifacts .134Home Directories .134Summary .141References .141CHAPTER 7 Internet Artifacts. 143Introduction .143Browser Artifacts .143Internet Explorer.144Firefox .147Chrome .154Safari .156Mail Artifacts .161Personal Storage Table .161mbox and maildir .163Summary .166References .166CHAPTER 8 File Analysis. 169File Analysis Concepts.169Content Identification .170Content Examination .171Metadata Extraction .172Images .175JPEG .178GIF .183PNG .184TIFF .185Audio .185WAV .185MPEG-3/MP3.186MPEG-4 Audio (AAC/M4A) .186ASF/WMA .188Video .189MPEG-1 and MPEG-2 .189MPEG-4 Video (MP4).189AVI .190ASF/WMV .190
ContentsMOV (Quicktime) .191MKV.192Archives .192ZIP .192RAR .1937-zip .195TAR, GZIP, and BZIP2 .195Documents.196OLE Compound Files (Office Documents) .197Office Open XML .201OpenDocument Format .204Rich Text Format .205PDF.206Summary .210References .210CHAPTER 9 Automating Analysis and Extending Capabilities . 211Introduction .211Graphical Investigation Environments .211PyFLAG .212Digital Forensics Framework .221Automating Artifact Extraction.229Fiwalk .229Timelines .231Relative Times .233Inferred Times .234Embedded Times .236Periodicity .236Frequency Patterns and Outliers (Least Frequencyof Occurrence) .237Summary .239References .239APPENDIX A Free, Non-open Tools of Note . 241Introduction .241Chapter 3: Disk and File System Analysis.242FTK Imager .242ProDiscover Free .242Chapter 4: Windows Systems and Artifacts .244Windows File Analysis.244Event Log Explorer .244Log Parser.245ix
xContentsChapter 7: Internet Artifacts.247NirSoft Tools .247Woanware Tools .247Chapter 8: File Analysis .248Mitec.cz: Structured Storage Viewer.248OffVis .249FileInsight.250Chapter 9: Automating Analysis and Extending Capabilities.250Mandiant: Highlighter .250CaseNotes .252Validation and Testing Resources .253Digital Corpora .253Digital Forensics Tool Testing Images .253Electronic Discovery Reference Model.254Digital Forensics Research Workshop Challenges .254Additional Images .254References .255Index. 257
About the AuthorsCory Altheide is a security engineer at Google, focused on forensics and incidentresponse. Prior to Google, Cory was a principal consultant with MANDIANT, aninformation security consulting firm that works with the Fortune 500, the defenseindustrial base, and banks of the world to secure their networks and combat cybercrime. In this role he responded to numerous incidents for a variety of clients inaddition to developing and delivering training to corporate and law enforcementcustomers.Cory also worked as the senior network forensics specialist in the NationalNuclear Security Administration’s Information Assurance Response Center (NNSAIARC). In this capacity he analyzed potentially hostile code, performed wirelessassessments of Department of Energy facilities, and researched new forensic techniques. He also developed and presented hands-on forensics training for various DoEentities and worked closely with members of the Southern Nevada Cyber CrimesTask Force to develop their skills in examining less common digital media.Cory has authored several papers for the computer forensics journal DigitalInvestigation and was a contributing author for UNIX and Linux Forensic Analysis(2008) and The Handbook of Digital Forensics and Investigation (2010). Additionally, Cory is a recurring member of the program committee of the Digital ForensicsResearch Workshop.Harlan Carvey (CISSP) is a vice president of Advanced Security Projects withTerremark Worldwide, Inc. Terremark is a leading global provider of IT infrastructureand “cloud computing” services based in Miami, Florida. Harlan is a key contributorto the Engagement Services practice, providing disk forensics analysis, consulting,and training services to both internal and external customers. Harlan has providedforensic analysis services for the hospitality industry and financial institutions, aswell as federal government and law enforcement agencies. Harlan’s primary areas ofinterest include research and development of novel analysis solutions, with a focus onWindows platforms. Harlan holds a bachelor’s degree in electrical engineering fromthe Virginia Military Institute and a master’s degree in the same discipline from theNaval Postgraduate School. Harlan resides in Northern Virginia with his family.xi
AcknowledgmentsCory AltheideFirst off I want to thank Harlan Carvey. In addition to serving as my coauthor andsounding board, he has been a good friend and colleague for many years. He hasproven to be one of the most consistently knowledgeable and helpful individualsI have met in the field. Harlan, thanks again for adding your considerable expertise tothe book and for never failing to buy me a beer every time I see you.I also thank Ray Davidson for his work as technical editor. His early insights andcommentary helped focus the book and made me target my subsequent writing onthe intended audience.Tremendous thanks go out to the “usual suspects” that make the open sourceforensics world the wonderful place it is. First, thank you to Wietse Venema and DanFarmer for creating open source forensics with “The Coroner’s Toolkit.” Thanks toBrian Carrier for picking up where they left off and carrying the torch to this day.Simson Garfinkel, you have my gratitude for providing the invaluable resource that isthe Digital Forensics Corpora. Special thanks to Eoghan Casey, who first encouragedme to share my knowledge with the community many years ago.To my parents, Steve and Jeanine Altheide, thank you for buying my first Commodore-64 (and the second and the third). Thanks to my brother Jeremy Altheideand the Old Heathen Brewing Company for producing some of the finest beersaround someday.I express infinite gratitude to my incredible wife Jamie Altheide for her neverending patience, love, and support during the research and writing of this book.Finally, I thank my daughters Winter and Lily for reminding me every day that I willnever have all the answers, and that’s okay.Harlan CarveyI begin by thanking God for the many blessings He’s given me in my life, the first ofwhich has been my family. I try to thank Him daily, but I find myself thinking thatthat’s not nearly enough. A man’s achievements are often not his alone, and in myheart, being able to write books like this is a gift and a blessing in many ways.I thank my true love and the light of my life, Terri, and my stepdaughter, Kylie.Both of these wonderful ladies have put up with my antics yet again (intently staringoff into space, scribbling in the air, and, of course, my excellent imitations taken fromsome of the movies we’ve seen), and I thank you both as much for your patience asfor being there for me when I turned away from the keyboard. It can’t be easy to havea nerd like me in your life, but I do thank you both for the opportunity to “put pen topaper” and get all of this stuff out of my head. Yes, that was a John Byrne reference.Finally, whenever you meet Cory, give him a thundering round of applause. Thisbook was his idea, and he graciously asked me to assist. I, of course, jumped at thechance to work with him again. Thanks, Cory.xiii
IntroductionINTENDED AUDIENCEWhen writing a technical book, one of the first questions the authors must answeris “Who is your audience?” The authors must then keep this question in mind at alltimes when writing. While it is hoped that this book is useful to everyone that readsit, the intended audience is primarily two groups.The first group is new forensic practitioners. This could range from students whoare brand new to the world of digital forensics, to active practitioners that are stillearly in their careers, to seasoned system administrators looking to make a careerchange. While this book is not a singular, complete compendium of all the forensicknowledge you will need to be successful, it is, hopefully, enough to get you started.The second audience is experienced digital forensics practitioners new to opensource tools. This is a fairly large audience, as commercial, proprietary tools havehad a nearly exhaustive hold on working forensic examiners. Many examiners operating today are reliant upon a single commercial vendor to supply the bulk of theirexamination capabilities. They rely on one vendor for their core forensic platformand may have a handful of other commercial tools used for specific tasks that theirmain tool does not perform (or does not perform well). These experienced examinerswho have little or no experience with open source tools will also hopefully benefitgreatly from the content of this book.LAYOUT OF THE BOOKBeyond the introductory chapter that follows, the rest of this book is divided up intoeight chapters and one Appendix.Chapter 2 discusses the Open Source Examination Platform. We walk throughall the prerequisites required to start compiling source code into executable code,install interpreters, and ensure we have a proper environment to build software onUbuntu and Windows. We also install a Linux emulation environment on Windowsalong with some additional packages to bring Windows closer to “feature parity”with Linux for our purposes.Chapter 3 details Disk and File System Analysis using the Sleuth Kit. TheSleuth Kit is the premier open source file system forensic analysis framework. Weexplain use of the Sleuth Kit and the fundamentals of media analysis, disk and partition structures, and file system concepts. We also review additional core digitalforensics topics such as hashing and the creation of forensic images.Chapter 4 begins our operating system-specific examination chapters withWindows Systems and Artifacts. We cover analysis of FAT and NTFS file systems,including internal structures of the NTFS Master File Table, extraction and analysisof Registry hives, event logs, and other Windows-specific artifacts. Finally, becausexv
xviIntroductionmalware-related intrusion cases are becoming more and more prevalent, we discusssome of the artifacts that can be retrieved from Windows executable files.We continue on to Chapter 5, Linux Systems and Artifacts, where we discuss analysis of the most common Linux file systems (Ext2 and 3) and identification, extraction, and analysis of artifacts found on Linux servers and desktops.System level artifacts include items involved in the Linu
Cory has authored several papers for the computer forensics journal Digital Investigation and was a contributing author for UNIX and Linux Forensic Analysis (2008) and The Handbook of Digital Forensics and Investigation (2010). Addition-ally, Cory is a recurring member of the program committee of the Digital Forensics Research Workshop.
