WIXLIB

Bareos OverviewBareos Overviewwww.bareos.org

Agenda1. What is Bareos / Bareos Features2. Bareos Architecture3. Installation4. Workflow (run jobs, restore)5. Configuration6. Plugins7. Roadmap

What is Bareos?

BareosNetwork based multi-platform backup solutionLicense: AGPL, 100% open sourcehttps://github.com/bareos/Core written in C/C Forked from Bacula in 2010First release in 2013 (bareos-12.4.3)One major release every yearcurrent: bareos-16.2.4 (16.2.5 soon)

Multi-platformInstallation packages forbareos.org/bareos.comCentOS, Debian, Fedora, openSUSE, RHEL, SLES,Ubuntu, Univention Corporate ServerWindows 32/64 bitMac ClientFreeBSDUNIX: AIX, HP-UX, SolarisDistributionsArch Linux, Debian, FreeBSD, Gentoo, Ubuntu Universe

Network Backup with Bareos

FeaturesAll common features of a backup solution are supportedFull, Differential and Incremental backupsAlways IncrementalBackup ManagementVolume ManagementRetention periodsFlexible SchedulingFlexible network setupDirector, Storage Daemon, File Daemon

FeaturesDifferent User Interfacesbconsole, bareos-webui (PHP), bat (QT-GUI, deprecated)

FeaturesSecurityChallenge-response authenticationTLSClient: Data EncryptionTape: LTO encryption (hardware, keys stored in BareosCatalog)Audit LogSecure Erase CommandACLsFile Daemon: restricted mode

FeaturesAPI / scriptingPlugin SupportC/C and Python pluginsintegration with/in other softwaree.g. Relax-and-Recover

Bareos Architecture

Bareos Architecture

File DaemonRuns on Client Computerread, write, verify filesread, write ACLs, attributesmake VSS snapshotschecksum calculationcompression/encryptionrun scriptsPlugin interface (C , Python)

Storage Daemondevice access (disk, tape, cloud)media changer controlread barcodes labelsMultiple Storage Daemonsrun Migration and CopyJobs on/to multiplelocationshandle media errorsPlugin interface (C , Python)

Catalogstores information about all files, media, jobsPostgreSQL/MySQL/SQLite

Directorhandles catalogmedia and pool handlingschedulingtrigger jobsbackup levelmessages, statistics and reportsrun scriptsPlugin interface (C , Python)

Network ConnectivityNormally:Connection are only made when required.Director connects to SD and FD.Tells the SD that it will receive a connection from the FDsoon.Tells the FD to connect to the SD.Other options:Passive Client:Director tells SD to connect to FDClient Initiated Connection:FD connects to Director

Installation of Bareos

Installing a Bareos Server1. Install the database of your choice2. Add Bareos repository3. Install Bareos packages, matching your database4. Prepare the Bareos database table/usr/lib/bareos/scripts/create bareos database/usr/lib/bareos/scripts/make bareos tables/usr/lib/bareos/scripts/grant bareos privileges5. Start the daemons Running Backup Server for Self-Backups

Bareos latest/bareos, bareos-bat, bareos-bconsole, bareos-client, bareoscommon, bareos-database-common, bareos-databasemysql, bareos-database-postgresql, bareos-databasesqlite3, bareos-database-tools, bareos-director, bareosdirector-python-plugin, bareos-filedaemon, bareosfiledaemon-ceph-plugin, on-ldap-python-plugin, bareos-filedaemonpython-plugin, bareos-storage, bareos-storage-ceph, bareosstorage-glusterfs, bareos-storage-python-plugin, bareosstorage-tape, bareos-tools, bareos-traymonitor, bareos-vadpdumper, bareos-vmware-plugin, bareos-vmware-vix-disklib5,bareos-webui, libfastlz

Bareos PackagesPrivate instance of http://openbuildservice.org/

WindowsWindows installer:cross-compiled on Linux (http://openbuildservice.org/)configuration of the Windows Firewallsilent install options / OPSI packagesFD, SD and Director can be selecteddebug package also installs sourcecode

Windows InstallationFD, SD and Director can be selectedconfiguration of the Windows Firewallsilent install options / OPSI packagesdebug package also installs sourcecodecross-compiled on Linuxhttp://openbuildservice.org/

Workflow

Workflowbconsole: start jobadmin@linux: bconsoleConnecting to Director bareos:91011000 OK: bareos-dir Version: 16.2.4 (01 July 2016)Enter a period to cancel a command.*Interactive Console to a Bareos DirectorTCP connection to the Directorhelp will list the available commands

Start workingbconsole: start job*runA job name must be specified.The defined Job resources are:1: backup-bareos-fd2: RestoreFiles3: CopyToTape4: BackupClient15: BackupCatalogSelect Job resource (1-5): 4

Workflowbconsole: start jobRun Backup jobJobName: at:NativeFileSet: Full SetPool:File (From Job resource)Storage: File (From Job resource)When:2017-01-30 16:30:59Priority: 10OK to run? (yes/mod/no): yesJob queued. JobId 3You have messages.*

Workflowbconsole: job message*messages30-Jan 16:31 bareos-dir JobId 3: No prior Full backup Job record found.30-Jan 16:31 bareos-dir JobId 3: No prior or suitable Full backupfound in catalog. Doing FULL backup.30-Jan 16:31 bareos-dir JobId 3: Start Backup JobId 3,Job BackupClient1.2017-01-30 16.31.05 0730-Jan 16:31 bareos-dir JobId 3: Using Device "FileStorage"to write.30-Jan 16:31 bareos-sd JobId 3: Volume "File-0001" previouslywritten, moving to end of data.30-Jan 16:31 bareos-sd JobId 3: Ready to append to end ofVolume "File-0001" size 3241954330-Jan 16:31 bareos-sd JobId 3: Elapsed time 00:00:01, Transferrate 32.38 M Bytes/second30-Jan 16:31 bareos-dir JobId 3: Bareos bareos-dir 16.2.4 (01Jul16):Build OS:x86 64-suse-linux-gnu suseopenSUSE Leap 42.1 (x86 64)JobId:3

Workflowbconsole: start job a second timeRun Backup jobJobName: at:NativeFileSet: Full SetPool:File (From Job resource)Storage: File (From Job resource)When:2017-01-30 16:40:59Priority: 10OK to run? (yes/mod/no): yesJob queued. JobId 4You have messages.*

WorkflowBconsole job, second run: incremental*messages30-Jan 16:41 bareos-dir JobId 4: Start Backup JobId 4, Job BackupClient1.201730-Jan 16:41 bareos-dir JobId 4: Using Device "FileStorage" to write.30-Jan 16:41 bareos-sd JobId 4: Volume "File-0001" previously written, moving to end30-Jan 16:41 bareos-sd JobId 4: Ready to append to end of Volume "File-0001" size 30-Jan 16:41 bareos-sd JobId 4: Elapsed time 00:00:01, Transfer rate 0 Bytes/second30-Jan 16:41 bareos-dir JobId 4: Bareos bareos-dir 16.2.4 (01Jul16):Build OS:x86 64-suse-linux-gnu suse openSUSE Leap 42.1 (x86 64)JobId:4Job:BackupClient1.2017-01-30 16.41.45 08Backup Level:Incremental, since 2017-01-30 16:31:08Client:"bareos-fd" 16.2.4 (01Jul16) x86 64-suse-linux-gnu,suse,openSUFileSet:"Full Set" 2017-01-30 16:29:42Pool:"File" (From Job resource)Catalog:"MyCatalog" (From Client resource)Storage:"File" (From Job resource)Scheduled time:30-Jan-2017 16:41:42Start time:30-Jan-2017 16:41:47

WorkflowWebui: run

WorkflowWebui: list jobs

WorkflowWebui: list joblog

WorkflowWebui: restore

Bareos Configuration

Bareos ConfigurationConfiguration is done in config filesEach daemon has its own config directoryusually in /bareos-fd.d/bconsole:/etc/bareos/bconsole.conf

FileSet: Definition what to backupFileSet {Name "LinuxAll"Include {Options {Signature MD5One FS NoFS Type btrfsFS Type ext4FS Type zfs}File /}Exclude {File /tmp}}

FileSet: let client decide, what tobackupFileSet {Name "LinuxClientDefinedList"Include {Options {Signature MD5}File "\\X/etc/bareos/backup-paths.list"}}\\X \\ file eva

Schedule: Definition when to run abackupSchedule {Name "WeeklyCycle"Run Full 1st sun at 23:05Run Differential 2nd-5th sun at 23:05Run Incremental mon-sat at 23:05}

Client: Definition of a ClientClient {Name bareos-fdAddress 192.168.0.1Password "lecCqzgBjxgM0J3 1adiuLzhy0cPGIHrdYMdtGHMbvKX"}

Job: Definition of a Jobcombines the other resources to a runnable backup jobJob {Name "backup-bareos-fd"Client "bareos-fd"FileSet "LinuxAll"Schedule "WeeklyCycle"Storage "File"Messages "Standard"Full Backup Pool "Full"Differential Backup Pool "Differential"Incremental Backup Pool "Incremental"[.]}#########name of this resourcewhat client to backup?which files to backup?when to backup?where to backup?where to send messages?write Full Backups into "Full" Poolwrite Diff Backups into "Differentiwrite Incr Backups into "Incrementa

Pool: FullPool {Name FullPool Type BackupRecycle yesAutoPrune yesVolume Retention 365 daysMaximum Volume Bytes 50GMaximum Volumes 100Label Format "Full-"}######Bareos can automatically recycle VolumesPrune expired volumesHow long should the Full Backups be kept? (#06)Limit Volume size to something reasonableLimit number of Volumes in PoolVolumes will be labeled

Pool: IncrementalPool {Name IncrementalPool Type BackupRecycle yesAutoPrune yesVolume Retention 30 daysMaximum Volume Bytes 1GMaximum Volumes 100Label Format "Incremental-"}######Bareos can automatically recycle VolumesPrune expired volumesHow long should the Incremental Backups be kept?Limit Volume size to something reasonableLimit number of Volumes in PoolVolumes will be labeled

Add A Clientbareos 16.2: manuallybareos 16.2:client: install bareos-filedaemonserver: "configure add client"server: copy generated client configuration to clientclient: restart bareos-filedaemon

Add A ClientClient:add Bareos repositoryinstall the package bareos-filedaemonServer:linux# bconsole*configure add client name client2-fd address 192.168.0.2 password secretCreated resource config file "/etc/bareos/bareos-dir.d/client/client2-fd.conf" fd/bareosfd.d/director/bareos-dir.confcopy filedaemon configuration to clientlinux# scp reos-fd.d/director/bareodir.conf irector/Client: restart bareos-filedaemon

Add A Client: Verify*status client client2-fdConnecting to Client client2-fd at 192.168.0.2:9102.*estimate listing job BackupClient1 client client2-fdConnecting to Client client2-fd at 192.168.0.2:9102lrwxrwxrwx1 rootroot7 2016-09-28 23:14:12lrwxrwxrwx1 rootroot7 2016-09-28 23:14:12./usr/sbin/a2disconf - /usr/sbin/a2enconf -

Add A Job*configure add job name backup-client2-fd client client2-fd jobdefs DefaultJobCreated resource config file "/etc/bareos/bareos-dir.d/job/client2-job.conf" .*status schedule job backup-client2-fd days 3.DateScheduleOverrides Thu 02-Feb-2017 21:00 WeeklyCycleLevel IncrementalFri 03-Feb-2017 21:00 WeeklyCycleLevel IncrementalSat 04-Feb-2017 21:00 WeeklyCycleLevel Full.*run job client2-jobJob queued. JobId 256*wait jobid 256JobId 256JobStatus OK (T)

Plugin: Backup using Pipesuses a pipe to backup a servicebackup as a virtual fileFileSet {Name "postgresql-all"Include {Options {signature MD5compression gzip}Plugin "bpipe:file /POSTGRESQL/dump.sql:reader pg dumpall -U postgres:writer psql}}

Plugin: MySQL / MariaDBuses Percona xtrabackupIncremental backups (for INNODB tables)HotbackupPoint-In-Time RecoveryFileSet {.Plugin "python:module path /usr/lib64/bareos/plugins:module name bareos-fd-percona:myc.}

Plugin: VMwareVMware Vstorage API supportallows backup of VMware virtual machinessupports Changed Block Tracking (Incremental backups)only used/changed blocks are backed up/restoredFileSet {.Plugin "python:module path /usr/lib64/bareos/plugins/vmware plugin:module name bareos.}

NDMP supportStorage systems often provide a NDMP backup interfaceNetApp, Isilon, .Bareos support NDMPFull and Incremental backupsSingle File restore

Volume access by native toolsAccess backup data without running Bareos Daemonscommand line tools:bls, bextract, bcopy, bscan, bcrypto

In depth: AlwaysIncremental BackupScheme

Conventional backupscheme1. daily incremental backups kept for one week2. weekly differential backups kept for three weeks3. monthly full backups kept for half year

Job availability for conventional backup scheme

Data being moved in conventional backup scheme

Problems with conventionalbackup scheme1. Full data is copied over the network in regular intervals2. Identical Data is copied from client multiple times3. Job history loss caused by retention expiry

always incremental backup schemeBasic conceptOnly changes are copied from the clients - alwaysincrementalExisting data from the client is consolidated with the newincremental information (keep history)The consolidation happens without client interactionMinimized number of incrementals is kept to have adefined change history