CS2043 - Unix Tools & Scripting

Transcription

CS2043 - Unix Tools & ScriptingCornell University, Spring 20141Instructor: Bruno AbrahaoJanuary 22, 20141Slides evolved from previous versions by Hussam Abu-Libdeh and David SlaterInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

AdministriviaWhen: January 22nd - March 5th; 3 times a weekWhere: MWF 11:15 - 12:00 in Hollister B14Drop Deadline: February 5th, two weeks into the courseGrade: Successful/UnsuccessfulWorkload: 5 or 6 assignmentsInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

AdministriviaBruno’s e-mail: abrahao at cs.cornell.eduWebsite: http://www.cs.cornell.edu/courses/cs2043/CMS: http://cms.csuglab.cornell.eduTAs and office hours: TBAPiazza: Ask questions, have discussions with course staff andclassmatesText: No official text. Everything we need is in the Unixdocumentation: man (together with the Web) is gonna beyour best friend.Try to give us feedback early: This is a short course!Instructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Course GoalsOverall Goal: Intensive training on basic and advanced Unix toolsand scripting.We will learn how to string together simple programs toperform extremely powerful tasks.Some of the topics include:Shell, Unix filesystem, basic toolsCombining tools/commands (pipe’ing)Advanced toolsRegular expressionsStream manipulationScriptingShell scriptingPython scriptingInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

What are scripts?Programs written for a special run-time environment that caninterpret and automate the execution of tasks, which couldalternatively be executed one-by-one by a human operator(Wikipedia).Interpreted (rather than compiled)Little use of data structuresMostly used for stream processingInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

A simple exampleWe need to change the name convention of one million files:24-09-2007-picturename.jpgshould be2007-09-24-picturename.jpgfor fn in *.jpgdo mv fn ‘echo fn \sed ‘s/([0-9] )-([0-9] )-([0-9] )/\3-\2-\1/’‘doneInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

PrerequisitesNot assuming any previous experience with the UNIXenvironmentThis course is a combination of the former CS2042 (tools)and CS2044 (scripting)Basic understanding of programming desirable(but probably not even necessary)Instructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Why you should learn this stuff?If (you’ve never used Unix before and) you think you’re a poweruser now, just wait. You don’t know what real power is.William E. Shotts, Jr., The Linux Command LineInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Why you should learn this stuff?Enables us to accomplish and automate complicated tasksthat would require arduous manual labor.A rich set of small commands and utilities that can becombined in unlimited ways to perform complex custom tasks.Rewarding: Extremely useful computer skill that will berelevant many years from now.Not limited to preconfigured combinations or menus, as inpersonal computer systems. Unix is a well-stocked toolbox,not a giant do-it-all Swiss Army Knife.It is fun!Software Engineering interviewsInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Typical software engineering interview questionWrite a sequence of Unix commands that tells you what programsyou use the most.history awk ’{print 2}’ sort uniq -c sort -nr head229 screen146 exit136 ls81 vi64 w47 math43 cp33 cd25 who23 historyInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

What Is Unix?One of the first widely-used operating systemsBasis for many modern OSesHelped set the standard for multi-tasking, multi-user systemsInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Short history of UNIX’60s The ambitious project MULTICS (Multiplexed Informationand Computing System) fails, but a number of seminal ideas(like pipes and shells) are proposed’69 Ken Thompson, Dennis Ritchie (et al.) start working on a filesystem, and name their system UNICS, which is later changedto UNIX.UNIX was “small, simple and clean”, and distributed freely tomany universities, where it becomes popularInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Short history of UNIX’73 Thompson and Ritchie rewrote UNIX in C (while most of theoperating systems at that time were written in assembly)’81 Berkley UNIX 4.1 BSD: vi, C shell, virtual memory’91 Linux, GNU, and others: similar to UNIX, but their sourcecode rewritten, very popular and widespread, freeMany Linux Distributions: Ubuntu, Fedora, Debian, .Currently, X/Open is responsible for developing UNIXInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Unix ArchitectureInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Unix shellsA shell is a program that allows the user to interact with the UNIXsystem:read user’s input and parses itevaluates special characterssetup pipes, redirections, and background processingfind and setup programs for executionInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Unix ShellsThere are primarily two “families” of unix shells:Bourne shell (AT&T) sh ksh bashC shell (Berkley) csh tcshWe focus on bash: easy syntax and default in many systemsInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

UNIX FlavorsBerkeley Software Distribution (BSD)GNU/LinuxMac OS XSun’s SolarisIBM AIXHP-UXSilicon Graphics IRIXInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Berkeley Software DistributionDeveloped by students and faculty at UC BerkeleyForked from the proprietary version back in the 80sHas since split into many additional flavors - namely,NetBSD, OpenBSD, and FreeBSDSpawned a popular open-source software license (the BSDLicense!)Primary competitor to Linux among free OSesInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Advantages/Disadvantages: BSDProsReliable and very secureClean codeUsable on almost anything that uses electricityMost flexible licenseFree!ConsConservative: slow progressLeast community/professional supportYou thought Linux was for nerdy outsiders?!Instructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

SUN SolarisCommercial offshoot of BSDDesigned to run on Sun’s SPARC servers, since ported to x86Most of the source code was recently released for theOpenSolaris projectInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Advantages/Disadvantages: SolarisProsBuilt specifically for the hardware it runs onScales really well as system size/load increasesLots of support from Sun as well as the communityConsYou are paying for Sun’s support and probablythe hardwarePrimarily for server use, not superdesktop-friendlyInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Linux!Pieced together by a Finnish guy named Linus Torvaldsstarting in 1991Built over the internet using message boards (Usenet)Designed to a UNIX-like standard, but not a direct descendantNote:Linux technically only refers to the OSs core, or kernel - withoutother programs it cant really do anything.Instructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Unix ArchitectureInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Free Software MovementGNU Gnu is Not UnixMovement in the 80s to build a free OSCreated many very popular toolsUnix like but uses no Unix codeStallman says:There really is a Linux, and these people areusing it, but it is just a part of the system theyuse. Linux is the kernel: the program in thesystem that allocates the machines resourcesto the other programs that you run. Linux isnormally used in combination with the GNUoperating system: the whole system is basicallyGNU with Linux added, or GNU/Linux.Instructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Torvalds says:Think of Richard Stallman as the greatphilosopher and think of me as the engineer.Instructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

GNU/Linux distrosLike BSD, GNU/Linux has a variety of flavors called“distributions”. These versions generally have different designgoals (security, speed, desktop use) and package a unique set oftools with the kernel to achieve them.Hundreds of distributions, such as RedHat, Ubuntu, SuSE,Slackware, Gentoo, etc.Saying “GNU/Linux” every time is tedious, so we will just refer tothe entire system as “Linux”.Instructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Advantages/Disadvantages: LinuxProsHuge community support baseFree (unless you want professional support)Free software to do almost anything“wine” allows you to run almost any windows programSome distributions are privacy preservingConsLacks some widely-used software (Office,Photoshop etc)Instructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Mac OS XBuilt on a BSD-based kernel, which was renamed “Darwin”Arguably the most popular desktop version of UNIXA pretty, easy to use experience built on a powerful frameSteve Jobs Says:What can the fully compliant UNIX technology inLeopard do? It can run any POSIX-compliant sourcecode. Help you make the most of multicore systems.Put a new tabbed-interface Terminal at yourfingertips. Introduce a whole host of new featuresthat make life easier for every developer. Really, whatcant it do?Instructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Advantages/Disadvantages: OSXProsUser friendly and just worksFully-featured GUI with a powerful terminalSupports most of the software the others lackConsDefinitely paying for this one!Closed-source, not as flexible as LinuxOnly runs on hardware purchased from Apple(without breaching the EULA)Instructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Why Linux?IT’S FREEMore widely used than BSD or SolarisEasy to find beginner’s guides online if you need themBasic tools are pretty much standardizedInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

We’re gonna live on a Unix terminalWe’ll interact with the Unix Shell: a text based programlauncher.Mouse, windows, and clicks are from another (distant) world!Instructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Internet MemeInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Getting to a UNIX ShellIf you are registered for the course, you have an account with theCS undergrad lab.http://www.csuglab.cornell.edu/userinfo/You can ssh (secure remote login) into the machines in the lab.Instructions are available at the csuglab webpage. If you use MSWindows, download Putty (free) to connect.Example Login:ssh ha232@linus.csuglab.cornell.eduInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Installing Linux on your machinePreferred method as you get superuser powers and can installprograms using a package manager.Dual boot (set up automatically for you) so that you can haveWindows and Linux side by side.To install Linux:1Choose a Linux distribution and download the Linux iso file(CD image file)www.debian.org (stable, but slow updates)www.ubuntu.com (User friendly, privacy problems)www.gnewsense.org (only free rn the iso image into a CD and boot your computer fromthe CDFollow the simple install wizard and enjoy! :-)Instructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

MS WindowsIf you have a Windows machine, there are a few other options:cygwin: a Linux-like environment for Windows(http://www.cygwin.com/)any linux live cd (http://www.livecdlist.com)Linux on a flash drive!VMWare: Unix environment in Virtual Machine withinWindowsInstructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

AppleOSX: Install xcode and the pagckage manager Macports(www.macports.org)IPad or IPhone: Download app, such as “SSH Term Pro”(commercial)Instructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Next timeWe’ll get our hands dirtyReminder: Homework (possibly a survey) will be posted inthe course website. Stay tuned!Instructor: Bruno AbrahaoCS2043 - Unix Tools & Scripting

Overall Goal: Intensive training on basic and advanced Unix tools and scripting. We will learn how to string together simple programs to perform extremely powerful tasks. Some of the topics include: Shell, Unix lesystem, basic tools Combining too