WIXLIB

Data IntegritySHA and MD5 ChecksumsLooking at Differences Between DataCompressing Data and Working with Compressed DatagzipWorking with Gzipped Compressed FilesCase Study: Reproducibly Downloading DataPart III.114115116118119120120Practice: Bioinformatics Data Skills7. Unix Data Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125Unix Data Tools and the Unix One-Liner Approach: Lessons fromProgramming PearlsWhen to Use the Unix Pipeline Approach and How to Use It SafelyInspecting and Manipulating Text Data with Unix ToolsInspecting Data with Head and TaillessPlain-Text Data Summary Information with wc, ls, and awkWorking with Column Data with cut and ColumnsFormatting Tabular Data with columnThe All-Powerful GrepDecoding Plain-Text Data: hexdumpSorting Plain-Text Data with SortFinding Unique Values in UniqJoinText Processing with AwkBioawk: An Awk for Biological FormatsStream Editing with SedAdvanced Shell TricksSubshellsNamed Pipes and Process SubstitutionThe Unix Philosophy 71631651691691711738. A Rapid Introduction to the R Language. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175Getting Started with R and RStudioR Language BasicsSimple Calculations in R, Calling Functions, and Getting Help in RVariables and AssignmentVectors, Vectorization, and IndexingWorking with and Visualizing Data in RLoading Data into Rviii Table of Contents176178178182183193194

Exploring and Transforming DataframesExploring Data Through Slicing and Dicing: Subsetting DataframesExploring Data Visually with ggplot2 I: Scatterplots and DensitiesExploring Data Visually with ggplot2 II: SmoothingBinning Data with cut() and Bar Plots with ggplot2Merging and Combining Data: Matching Vectors and Merging DataframesUsing ggplot2 FacetsMore R Data Structures: ListsWriting and Applying Functions to Lists with lapply() and sapply()Working with the Split-Apply-Combine PatternExploring Dataframes with dplyrWorking with StringsDeveloping Workflows with R ScriptsControl Flow: if, for, and whileWorking with R ScriptsWorkflows for Loading and Combining Multiple FilesExporting DataFurther R Directions and 32542572602619. Working with Range Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263A Crash Course in Genomic Ranges and Coordinate SystemsAn Interactive Introduction to Range Data with GenomicRangesInstalling and Working with Bioconductor PackagesStoring Generic Ranges with IRangesBasic Range Operations: Arithmetic, Transformations, and Set OperationsFinding Overlapping RangesFinding Nearest Ranges and Calculating DistanceRun Length Encoding and ViewsStoring Genomic Ranges with GenomicRangesGrouping Data with GRangesListWorking with Annotation Data: GenomicFeatures and rtracklayerRetrieving Promoter Regions: Flank and PromotersRetrieving Promoter Sequence: Connection GenomicRanges with SequenceDataGetting Intergenic and Intronic Regions: Gaps, Reduce, and Setdiffs inPracticeFinding and Working with Overlapping RangesCalculating Coverage of GRanges ObjectsWorking with Ranges Data on the Command Line with BEDToolsComputing Overlaps with BEDTools IntersectBEDTools Slop and FlankCoverage with BEDToolsTable of 324327329330333335 ix

Other BEDTools Subcommands and pybedtools33610. Working with Sequence Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339The FASTA FormatThe FASTQ FormatNucleotide CodesBase QualitiesExample: Inspecting and Trimming Low-Quality BasesA FASTA/FASTQ Parsing Example: Counting NucleotidesIndexed FASTA Files33934134334434634935211. Working with Alignment Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355Getting to Know Alignment Formats: SAM and BAMThe SAM HeaderThe SAM Alignment SectionBitwise FlagsCIGAR StringsMapping QualitiesCommand-Line Tools for Working with Alignments in the SAM FormatUsing samtools view to Convert between SAM and BAMSamtools Sort and IndexExtracting and Filtering Alignments with samtools viewVisualizing Alignments with samtools tview and the Integrated GenomicsViewerPileups with samtools pileup, Variant Calling, and Base Alignment QualityCreating Your Own SAM/BAM Processing Tools with PysamOpening BAM Files, Fetching Alignments from a Region, and IteratingAcross ReadsExtracting SAM/BAM Header Information from an AlignmentFile ObjectWorking with AlignedSegment ObjectsWriting a Program to Record Alignment StatisticsAdditional Pysam Features and Other SAM/BAM 8839139412. Bioinformatics Shell Scripting, Writing Pipelines, and Parallelizing Tasks. . . . . . . . . . 395Basic Bash ScriptingWriting and Running Robust Bash ScriptsVariables and Command ArgumentsConditionals in a Bash Script: if StatementsProcessing Files with Bash Using for Loops and GlobbingAutomating File-Processing with find and xargsUsing find and xargsFinding Files with findx Table of Contents396396398401405411411412

find’s Expressionsfind’s -exec: Running Commands on find’s Resultsxargs: A Unix PowertoolUsing xargs with Replacement Strings to Apply Commands to Filesxargs and ParallelizationMake and Makefiles: Another Option for Pipelines41341541641841942113. Out-of-Memory Approaches: Tabix and SQLite. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425Fast Access to Indexed Tab-Delimited Files with BGZF and TabixCompressing Files for Tabix with BgzipIndexing Files with TabixUsing TabixIntroducing Relational Databases Through SQLiteWhen to Use Relational Databases in BioinformaticsInstalling SQLiteExploring SQLite Databases with the Command-Line InterfaceQuerying Out Data: The Almighty SELECT CommandSQLite FunctionsSQLite Aggregate FunctionsSubqueriesOrganizing Relational Databases and JoinsWriting to DatabasesDropping Tables and Deleting DatabasesInteracting with SQLite from PythonDumping 545845946514. Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467Where to Go From Here?468Glossary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 471Bibliography. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483Table of Contents xi

PrefaceThis book is the answer to a question I asked myself two years ago: “What bookwould I want to read first when getting started in bioinformatics?” When I beganworking in this field, I had programming experience in Python and R but little else. Ihad hunted around for a terrific introductory text on bioinformatics, and while Ifound some good books, most were not targeted to the daily work I did as a bioinfor‐matician. A few of the texts I found approached bioinformatics from a theoretical andalgorithmic perspective, covering topics like Smith-Waterman alignment, phylogenyreconstruction, motif finding, and the like. Although they were fascinating to read(and I do recommend that you explore this material), I had no need to implementbioinformatics algorithms from scratch in my daily bioinformatics work—numerousterrific, highly optimized, well-tested implementations of these algorithms alreadyexisted. Other bioinformatics texts took a more practical approach, guiding readersunfamiliar with computing through each step of tasks like running an aligner ordownloading sequences from a database. While these were more applicable to mywork, much of those books’ material was outdated.As you might guess, I couldn’t find that best “first” bioinformatics book. Bioinformat‐ics Data Skills is my version of the book I was seeking. This book is targeted towardreaders who are unsure how to bridge the giant gap between knowing a scripting lan‐guage and practicing bioinformatics to answer scientific questions in a robust andreproducible way. To bridge this gap, one must learn data skills—an approach thatuses a core set of tools to manipulate and explore any data you’ll encounter during abioinformatics project.Data skills are the best way to learn bioinformatics because these skills utilize timetested, open source tools that continue to be the best way to manipulate and explorechanging data. This approach has stood the test of time: the advent of highthroughput sequencing rapidly changed the field of bioinformatics, yet skilled bioin‐formaticians adapted to this new data using these same tools and skills. Nextgeneration data was, after all, just data (different data, and more of it), and masterbioinformaticians had the essential skills to solve problems by applying their tools toxiii

this new data. Bioinformatics Data Skills is written to provide you with training inthese core tools and help you develop these same skills.The Approach of This BookMany biologists starting out in bioinformatics tend to equate “learning bioinformat‐ics” with “learning how to run bioinformatics software.” This is an unfortunate andmisinformed idea of what bioinformaticians actually do. This is analogous to think‐ing “learning molecular biology” is just “learning pipetting.” Other than a few simpleexamples used to generate data in Chapter 11, this book doesn’t cover running bioin‐formatics software like aligners, assemblers, or variant callers. Running bioinformat‐ics software isn’t all that difficult, doesn’t take much skill, and it doesn’t embody anyof the significant challenges of bioinformatics. I don’t teach how to run these types ofbioinformatics applications in Bioinformatics Data Skills for the following reasons: It’s easy enough to figure out on your own The material would go rapidly out of date as new versions of software or entirelynew programs are used in bioinformatics The original manuals for this software will always be the best, most up-to-dateresource on how to run a programInstead, the approach of this book is to focus on the skills bioinformaticians use toexplore and extract meaning from complex, large bioinformatics datasets. Exploringand extracting information from these datasets is the fun part of bioinformaticsresearch. The goal of Bioinformatics Data Skills is to teach you the computationaltools and data skills you need to explore these large datasets as you please. These dataskills give you freedom; you’ll be able to look at any bioinformatics data—in any for‐mat, and files of any size—and begin exploring data to extract biological meaning.Throughout Bioinformatics Data Skills, I emphasize working in a robust and reprodu‐cible manner. I believe these two qualities—reproducibility and robustness—are toooften overlooked in modern computational work. By robust, I mean that your work isresilient against silent errors, confounders, software bugs, and messy or noisy data. Incontrast, a fragile approach is one that does not decrease the odds of some type oferror adversely affecting your results. By reproducible, I mean that your work can berepeated by other researchers and they can arrive at the same results. For this to bethe case, your work must be well documented, and your methods, code, and data allneed to be available so that other researchers have the materials to reproduce every‐thing. Reproducibility also relies on your work being robust—if a workflow run on adifferent machine yields a different outcome, it is neither robust nor fully reproduci‐ble. I introduce these concepts in more depth in Chapter 2, and these are themes thatreappear throughout the book.xiv Preface

Why This Book Focuses on Sequencing DataBioinformatics is a broad discipline, and spans subfields like proteomics, metabolo‐mics, structure bioinformatics, comparative genomics, machine learning, and imageprocessing. Bioinformatics Data Skills focuses primarily on handling sequencing datafor a few reasons.First, sequencing data is abundant. Currently, no other “omics” data is as abundant ashigh-throughput sequencing data. Sequencing data has broad applications acrossbiology: variant detection and genotyping, transcriptome sequencing for gene expres‐sion studies, protein-DNA interaction assays like ChIP-seq, and bisulfite sequencingfor methylation studies just to name a few examples. The ways in which sequencingdata can be used to answer biological questions will only continue to increase.Second, sequencing data is terrific for honing your data skills. Even if your goal is toanalyze other types of data in the future, sequencing data serves as great example datato learn with. Developing the text-processing skills necessary to work with sequenc‐ing data will be applicable to working with many other data types.Third, other subfields of bioinformatics are much more domain specific. The wideavailability and declining costs of sequencing have allowed scientists from all disci‐plines to use genomics data to answer questions in their systems. In contrast, bioin‐formatics subdisciplines like proteomics or high-throughput image processing aremuch more specialized and less widespread. Still, if you’re interested in these fields,Bioinformatics Data Skills will teach you useful computational and data skills that willbe helpful in your research.AudienceIn my experience teaching bioinformatics to friends, colleagues, and students of anintensive week-long course taught at UC Davis, most people wishing to learn bioin‐formatics are either biologists, or computer scientists/programmers. Biologists wishto develop the computational skills necessary to analyze their own data, while theprogrammers and computer scientists wish to apply their computational skills to bio‐logical problems. Although these two groups differ considerably in biological knowl‐edge and computational experience, Bioinformatics Data Skills covers material thatshould be helpful to both.If you’re a biologist, Bioinformatics Data Skills will teach you the core data skills youneed to work with bioinformatics data. It’s important to note that Bioinformatics DataSkills is not a how-to bioinformatics book; such a book on bioinformatics wouldquickly go out of date or be too narrow in focus to help the majority of biologists. Youwill need to supplement this book with knowledge of your specific research and sys‐tem, as well as the modern statistical and bioinformatics methods that your subfieldPreface xv

uses. For example, if your project involves aligning sequencing reads to a referencegenome, this book won’t tell you the newest and best alignment software for yourparticular system. But regardless of which aligner you use, you will need to have athorough understanding of alignment formats and how to manipulate alignment data—a topic covered in Chapter 11. Throughout this book, these general computationaland data skills are meant to be a solid, widely applicable foundation on which themajority of biologists can build.If you’re a computer scientist or programmer, you are likely already familiar withsome of the computational tools I teach in this book. While the material presented inBioinformatics Data Skills may overlap knowledge you already have, you will stilllearn about the specific formats, tools, and approaches bioinformaticians use in theirwork. Also, working through the examples in this book will give you good practice inapplying your computational skills to genomics data.The Difficulty Level of Bioinformatics Data SkillsBioinformatics Data Skills is designed to be a thorough—and in parts, dense—book.When I started writing this book, I decided the greatest misdeed I could do would beto treat bioinformatics as a subject that’s easier than it truly is. Working as a professio‐nal bioinformatician, I routinely saw how very subtle issues could crop up andadversely change the outcome of the analysis had they not been caught. I don’t wantyour bioinformatics work to be incorrect because I’ve made a topic artificially simple.The depth at which I cover topics in Bioinformatics Data Skills is meant to prepareyou to catch similar issues in your own work so your results are robust.The result is that sections of this book are quite advanced and will be difficult forsome readers. Don’t feel discouraged! Like most of science, this material is hard, andmay take a few reads before it fully sinks in. Throughout the book, I try to indicatewhen certain sections are especially advanced so that you can skip over these andreturn to them later.Lastly, I often use technical jargon throughout the book. I don’t like using jargon, butit’s necessary to communicate technical concepts in computing. Primarily it will helpyou search for additional resources and help. It’s much easier to Google successfullyfor “left outer join” than “data merge where null records are included in one table.”Assumptions This Book MakesBioinformatics Data Skills is meant to be an intermediate book on bioinformatics. Tomake sure everyone starts out on the same foot, the book begins with a few simplechapters. In Chapter 2, I cover the basics of setting up a bioinformatics project, and inChapter 3 I teach some remedial Unix topics meant to ensure that you have a solidxvi Preface

grasp of Unix (because Unix is a large component in later chapters). Still, as an inter‐mediate book, I make a few assumptions about you:You know a scripting languageThis is the biggest assumption of the book. Except for a few Python programsand the R material (R is introduced in Chapter 8), this book doesn’t directly relyon using lots of scripting. However, in learning a scripting language, you’vealready encountered many important computing concepts such as working witha text editor, running and executing programs on the command line, and basicprogramming. If you do not know a scripting language, I would recommendlearning Python while reading this book. Books like Bioinformatics ProgrammingUsing Python by Mitchell L. Model (O’Reilly, 2009), Learning Python, 5th Edition,by Mark Lutz (O’Reilly, 2013), and Python in a Nutshell, 2nd, by Alex Martelli(O’Reilly, 2006) are

Bioinformatics Data Skills ISBN: 978-1-449-36737-4 US 49.99 CAN 57.99 “ Mostxisting e bioinformatics texts focusnlgorithms o a and theories. Bioinformatics Data Skills akest a refreshingly practical approachy b providing aomprehensive c introduction to the techniques, tools, and b