WIXLIB

www.it-ebooks.info

www.it-ebooks.info

OpenStack Operations Guideby Tom Fifield, Diane Fleming, Anne Gentle,Lorin Hochstein, Jonathan Proulx, Everett Toews,and Joe Topjianwww.it-ebooks.info

OpenStack Operations Guideby Tom Fifield, Diane Fleming, Anne Gentle, Lorin Hochstein, Jonathan Proulx, Everett Toews, and JoeTopjianCopyright 2014 OpenStack Foundation. All rights reserved.Printed in the United States of America.Published by O'Reilly Media, Inc. , 1005 Gravenstein Highway North, Sebastopol, CA 95472.O'Reilly books may be purchased for educational, business, or sales promotional use. Online editions arealso available for most titles (http://my.safaribooksonline.com). For more information, contact our corpo‐rate/institutional sales department: 800-998-9938 or corporate@oreilly.com .Editors: Andy Oram and Brian AndersonProduction Editor: Kristen BrownCopyeditor: John PierceProofreader: Amanda KerseyMay 2014:Indexer: Judith McConvilleInterior Designer: David FutatoCover Designer: Karen MontgomeryIllustrator: Rebecca DemarestFirst EditionRevision History for the First Edition2014-04-21: First releaseSee http://oreilly.com/catalog/errata.csp?isbn 9781491946954 for release details.Nutshell Handbook, the Nutshell Handbook logo, and the O'Reilly logo are registered trademarks ofO'Reilly Media, Inc. OpenStack Operations Guide, the image of a crested agouti, and related trade dress aretrademarks of O'Reilly Media, Inc.Many of the designations used by manufacturers and sellers to distinguish their products are claimed astrademarks. Where those designations appear in this book, and O'Reilly Media, Inc., was aware of a trade‐mark claim, the designations have been printed in caps or initial caps.While every precaution has been taken in the preparation of this book, the publisher and authors assumeno responsibility for errors or omissions, or for damages resulting from the use of the information con‐tained herein.978-1-491-94695-4[LSI]www.it-ebooks.info

Table of ContentsPreface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiPart I.Architecture1. Example Architectures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Example Architecture—Legacy Networking (nova)OverviewDetailed DescriptionOptional ExtensionsExample Architecture—OpenStack NetworkingOverviewDetailed DescriptionExample Component ConfigurationParting Thoughts on Architectures3479991119232. Provisioning and Deployment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Automated DeploymentDisk Partitioning and RAIDNetwork ConfigurationAutomated ConfigurationRemote ManagementParting Thoughts for Provisioning and Deploying OpenStackConclusion252628282929303. Designing for Cloud Controllers and Cloud Management. . . . . . . . . . . . . . . . . . . . . . . . . 31Hardware ConsiderationsSeparation of ServicesDatabase323334iiiwww.it-ebooks.info

Message QueueConductor ServicesApplication Programming Interface tion and AuthorizationNetwork Considerations3435353636373737384. Compute Nodes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39Choosing a CPUChoosing a HypervisorInstance Storage SolutionsOff Compute Node Storage—Shared File SystemOn Compute Node Storage—Shared File SystemOn Compute Node Storage—Nonshared File SystemIssues with Live MigrationChoice of File 0414242434344444545455. Scaling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47The Starting PointAdding Cloud Controller NodesSegregating Your CloudCells and RegionsAvailability Zones and Host AggregatesScalable HardwareHardware ProcurementCapacity PlanningBurn-in Testing4749505151535354546. Storage Decisions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55Ephemeral StoragePersistent StorageObject StorageBlock StorageOpenStack Storage ConceptsChoosing Storage Backendsiv 555555575758Table of Contentswww.it-ebooks.info

Commodity Storage Backend TechnologiesConclusion60627. Network Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63Management NetworkPublic Addressing OptionsIP Address PlanningNetwork TopologyVLAN Configuration Within OpenStack VMsMulti-NIC ProvisioningMulti-Host and Single-Host NetworkingServices for NetworkingNTPDNSConclusionPart II.6364646567676768686868Operations8. Lay of the Land. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71Using the OpenStack Dashboard for AdministrationCommand-Line ToolsInstalling the ToolsAdministrative Command-Line ToolsGetting CredentialsInspecting API CallsServers and ServicesDiagnose Your Compute NodesNetwork InspectionUsers and ProjectsRunning InstancesSummary7171727273757678798081829. Managing Projects and Users. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83Projects or Tenants?Managing ProjectsAdding ProjectsQuotasSet Image QuotasSet Compute Service QuotasSet Object Storage QuotasSet Block Storage Quotas8384848586868990Table of Contentswww.it-ebooks.info v

User ManagementCreating New UsersAssociating Users with ProjectsCustomizing AuthorizationUsers Who Disrupt Other UsersSummary92929394979710. User-Facing Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99ImagesAdding ImagesSharing Images Between ProjectsDeleting ImagesOther CLI OptionsThe Image Service and the DatabaseExample Image Service Database QueriesFlavorsPrivate FlavorsHow Do I Modify an Existing Flavor?Security GroupsGeneral Security Groups ConfigurationEnd-User Configuration of Security GroupsBlock StorageBlock Storage Creation FailuresInstancesStarting InstancesInstance Boot FailuresUsing Instance-Specific DataAssociating Security GroupsFloating IPsAttaching Block StorageTaking SnapshotsLive SnapshotsInstances in the DatabaseGood 810810911011211211311411511611711. Maintenance, Failures, and Debugging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119Cloud Controller and Storage Proxy Failures and MaintenancePlanned MaintenanceRebooting a Cloud Controller or Storage ProxyAfter a Cloud Controller or Storage Proxy RebootsTotal Cloud Controller FailureCompute Node Failures and Maintenancevi Table of Contentswww.it-ebooks.info119119119120120121

Planned MaintenanceAfter a Compute Node RebootsInstancesInspecting and Recovering Data from Failed InstancesVolumesTotal Compute Node Failure/var/lib/nova/instancesStorage Node Failures and MaintenanceRebooting a Storage NodeShutting Down a Storage NodeReplacing a Swift DiskHandling a Complete FailureConfiguration ManagementWorking with HardwareAdding a Compute NodeAdding an Object Storage NodeReplacing ComponentsDatabasesDatabase ConnectivityPerformance and miannuallyDetermining Which Component Is BrokenTailing LogsRunning Daemons on the 3313413512. Network Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137Using “ip a” to Check Interface StatesVisualizing nova-network Traffic in the CloudVisualizing OpenStack Networking Service Traffic in the CloudFinding a Failure in the PathtcpdumpiptablesNetwork Configuration in the Database for nova-networkManually Deassociating a Floating IPDebugging DHCP Issues with nova-network137138139145146147148148149Table of Contentswww.it-ebooks.info vii

Debugging DNS IssuesTroubleshooting Open vSwitchDealing with Network NamespacesSummary15215315415513. Logging and Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157Where Are the Logs?Reading the LogsTracing Instance RequestsAdding Custom Logging StatementsRabbitMQ Web Management Interface or rabbitmqctlCentrally Managing Logsrsyslog Client Configurationrsyslog Server ConfigurationStackTachMonitoringProcess MonitoringResource AlertingMetering and Telemetry with CeilometerOpenStack-Specific ResourcesIntelligent 16316416516516616716816914. Backup and Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171What to Back UpDatabase BackupsFile System BackupsComputeImage Catalog and DeliveryIdentityBlock StorageObject StorageRecovering BackupsSummary17117217217217317317317317317415. Customization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175Create an OpenStack Development EnvironmentCustomizing Object Storage (Swift) MiddlewareCustomizing the OpenStack Compute (nova) SchedulerCustomizing the Dashboard (Horizon)viii Table of Contentswww.it-ebooks.info175178184189

Conclusion18916. Upstream OpenStack. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191Getting HelpReporting BugsConfirming and PrioritizingBug FixingAfter the Change Is AcceptedJoin the OpenStack CommunityHow to Contribute to the DocumentationSecurity InformationFinding Additional Information19119219319419419419519519617. Advanced Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197Differences Between Various DriversImplementing Periodic TasksSpecific Configuration TopicsSecurity Configuration for Compute, Networking, and StorageHigh AvailabilityEnabling IPv6 SupportPeriodic Task Frequency for ComputeGeographical Considerations for Object Storage19719819919919919919920018. Upgrades. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201Pre-Upgrade Testing EnvironmentPreparing for a RollbackUpgradesHow to Perform an Upgrade from Grizzly to Havana—UbuntuImpact on UsersUpgrade ConsiderationsPerform a BackupManage RepositoriesUpdate Configuration FilesUpgrade Packages on the Controller NodeStop Services, Update Database Schemas, and Restart Services on theController NodeUpgrade Packages and Restart Services on the Compute NodesUpgrade Packages and Restart Services on the Block Storage NodesHow to Perform an Upgrade from Grizzly to Havana—Red Hat EnterpriseLinux and DerivativesImpact on UsersUpgrade ConsiderationsTable of 205207207208209210210210 ix