WIXLIB

Installation IssuesInstallation IssuesIssueDescriptionWorkaroundAMIF-4876: Nodesadded to wronglocationWhen the Lattus system has racks with identical names but indifferent locations, the nodes are all added in one location duringnode initialization.N/AAMIF-4803: Agent notreachable afternetwork initializationin a Lattus multi-geo3-site setupWhen initializing a node, the agent does not recover from thenetwork initialization. Only restarting the agent solves this issue.Restart theagent.AMIF-4660: Cleanupon a partiallyinitialized node canbreak initialization ofthe next nodeWhen you clean up a node whose initialization has failed betweenconfiguring the network and configuring DSS, the node keeps itsstatic IP address. This might cause the next node you initialize tofail, because both nodes might have the same static IP address.N/AAMIF-4530: Possibleto create cachecluster and Arakooncluster with nonhealthy disksIt is possible to create cache clusters and Arakoon clusters withdisks that have the status Degraded or Autodecommissioning,instead of failing as expected.N/AAMIF-4444:Initializing a nodedoes not generate avalidation error whenselecting a virtual IPWhen initializing a node, selecting a virtual IP address of themanagement node does not produce a validation error.N/AAMIF-4452: Lattusinstallation cancontinue withoutspecified root disksThe Lattus installation can proceed even when there are no rootdisks found. The root disks are specified in the HAL configurationfiles with the provided bus locations. When the bus location or rootdisks are not found, the first two found hard disks are used for theinstallation.N/AAMIF-4302: Installershows wronghardware type whenusing generichardwareWhen installing a management node or a remote node using USB ongeneric hardware, the installer shows “CTRLGeneric” (genericcontroller node) as the hardware type. The installer does not detectwhether you are installing a storage node or controller node. Duringthe initialization, the hardware type is updated with the correct type.N/AThis information is visible in the installation logs in the CMC.Lattus Release Notes8

Installation IssuesIssueDescriptionWorkaroundAMIF-4246: Cannotclean up node withstatus “Installing”When a node has the status “Installing”, but the installation of thenode has failed, it is not possible to clean up the node. This situationcan for example occur in an installation with incorrect routes.Set the statusof the node to“Failed,” andthen removethe node fromthe CMC.AMIF-77: After theinitial configurationstep on theManagement Node,an unclear error isproduced because theadmin password haschanged.The following error appears:N/AAMIF-992: There is nocheck for the queueinitialization file’sextension.Uploading a file with an extension that differs from .csv leads tofailed queue file upload.N/AAMIF-1810:- Adeprecation warningmessage appearswhen initializing a QShell environmentThis is because an older version of the pycrypto package (2.0.1) isused, and newer versions are available.N/AAMIF-2020: Installingthe controller nodefails with error: Invalidpartitiontable:recursivepartition on /dev/sddWhen the USB drive is left inserted into the machine after OSinstallation, this error can occur during the tstring:'error',faultdetail:"Authentication Failed"The behavior of these newer versions is currently being tested.This error occurs with specific types of USB drives.If youencounter thisissue, clean upthe machineand restart theinstallation.Remove theUSB drive afterrebooting themachine, priorto initializing.AMIF-1858: Whenadding a node to anexisting environment,the default login andpassword areassignedLattus Release NotesWhen you add a node to an already set up environment, the node isinitialized with the default node login/password combination.After adding anode, executethe procedureto change thepassword forthe rootaccount.9

Installation tion fails fornode that wasremoved from thequeueIn the following scenario:Remove thenode that failedto install inorder to avoidhaving thefailure.lllBuild up a node queue with more than two servers and a fewstorage nodes.Select all nodes and initialize the queue.Remove the second node from the queue, while the first node isbeing initialized.In this case, the running “Initializing X nodes” still tries to processthe removed node, resulting in an error.AMIF-3082: Lots ofDEBUG.osis.utilsmessages on theconsole during themanagement nodeInstallationWhen you install the management node successful and reboot forthe first time, you will receive the following debug licy.pyThese debug osis messages are benign and can safely be ignored.AMIF-3732: 1MBSSD storage ismissing on anothernon-CMC controllerwhile creating objectMetaStoreThe size of the MetaStore cannot be greater than the remainingspace on the SSDs. If it is greater, that SSD will not be available toadd to the MetaStore).N/AAMIF-3084: rpcbinderror messagesappear after rebootingan installedmanagementcontrollerAfter installing and rebooting the management controller, the bootlog might show you error messages like the following:N/ALattus Release Notesrpcbind: Cannot open ‘ file name ’ file forreading, errno 2 (No such file or directory)These errors are benign and have no further impact on the furtherinstallation of your environment.10

PUT, GET, and REPAIR Operation IssuesPUT, GET, and REPAIR Operation IssuesIssueDescriptionWorkaroundAMIF-4959: Initializingreplacement disk failsbecause of missingArakoon directoriesWhen you initialize a replacement disk and there is no Arakoontlf and Arakoon home directory, you receive an errorN/ADSS-1886: Command linerejects objects which areincompatible with AXRobject name rulesWhen you use the command line or Q-Shell on an object that isnot compatible with the AXR naming conventions, you receivean error, even when you work in an S3 bucket or when theobject name is S3 compatible.N/ADSS-1882: Outstandingobjects and incompletemultipart uploads are notdeletedWhen you delete an AXR directory or S3 bucket, you do notdelete possible outstanding objects or upload parts of anincomplete multipart upload. You will no longer see these‘objects,’ but they remain on disk.N/ADSS-1805: Wrong cachesize after removing LRUobjectWhen a cache cluster has reached its maximum capacity, itstarts removing entries using an LRU method.N/ADSS-1146: Rebalancingfor small file support doesnot workWith the introduction of small file support, rebalancing hasbeen disabled because the spreading of the full copy was notdone in a homogenous way, and a rebalance could put toomuch strain on a single disk.A workaround isdescribed in KBarticle BSP038.DSS-1658: Storage fullmessage is misleading in adistributed setupIn a distributed setup, it is possible that you receive a messageindicating that “Storage is full”. This message is by design, andin most situations means there are too many offlineblockstores.N/ADSS-1518: Syncstorestatus “Full” is notimmediately honoredWhen setting the status of a syncstore to “Full”, you can stillupload files via the AXR interface for a while, but not via the QShell interface. After a while it is no longer possible through theAXR interface.N/ATypeError: has master() takes exactly 5non-keyword arguments (4 given).It removes a file from the disk, and should subtract the size ofthis file from the current cache size. However, it subtracts thesize of the new file that will be stored in the cache (whichtriggered the LRU cleanup), instead of the size of the file it hasdeleted from disk.When setting the status back to write modus, it again takes acouple of minutes before you can start uploading files via theAXR interface.Lattus Release Notes11

PUT, GET, and REPAIR Operation IssuesIssueDescriptionWorkaroundDSS-1484: Deleting anobject via the AXRinterface randomly failswith an Unauthorized errorDeleting an object using the AXR interface might randomly failwith the error: “HTTP 401 error: Unauthorized dueN/ADSS-1508: Files withwhite spaces in their namecannot be uploaded viaAXRUploading a file fails when using the web form and the AXRinterface, and when the file has white spaces in its name.N/ADSS-1474: A failed PUToperation writes metadatato Arakoon anyway, butcannot be deletedafterwardsWhen a PUT operation fails, there is still metadata written toArakoon. It is impossible to remove the loose metadata.N/ADSS-1436: Object nameswith a leading forwardslash cannot be handled bythe Lattus S3 interfaceThe Lattus S3 interface cannot handle objects whose namesbegin with a forward slash (“/”). Also, the authentication failswith a leading forward slash.N/ADSS-1443: Many GEToperations makes the logfile grow too fastMany GET operations makes the BitSpread log file grow toofast. This typically occurs with low bandwidth setups. You seea lot of lines in the log file appear with the following content:N/Ato an exception in Arakoon while handlingthe request”.wr sb duration 10.0sDSS-601: Pulling out aswitch doesn’timmediately result inoperations taking thebandwidth of the remainingswitchWhen one of the back-end switches fails, all existing putoperations fail (until they time out). New puts that are startedafter the failure will succeed, but could also time out.After all blockstores have been contacted, there will be nofurther connection attempts towards the old switch, andtherefore no more reasons for failure.Adjust thetimeout valueson the clientdaemon.Restart theclient daemons.Depending upon the size of the objects and superblocks, it cantake up to 15 minutes before the full bandwidth capacity of theremaining NIC is used. After restarting the client daemons, ittakes up to 3 minutes to achieve a steady state situation.Lattus Release Notes12

PUT, GET, and REPAIR Operation IssuesIssueDescriptionWorkaroundDSS-747: When two backend switches fail shortlyone after one the other, putand get operations willhangAs stated in another known issue, it can take a while beforeenough connections are migrated from one network to another.If your failed switch is restored, it will take a while before allconnections are migrated back and well balanced over bothswitches.Adjust thetimeout valueson the clientdaemon.Restart theclient daemons.In cases where the second failure happens too quickly after thefirst, not enough connections are migrated back, and ongoingput/get operations will hang because there aren’t enoughavailable connections.DSS-549: Unclear errormessages when creatingan already existing dir or adir with a missing parent dirCreating a directory that already exists results in a HTTP 405“Method not allowed” message.DSS-115: Namespacedeletion removes data fromthe blockstores in a serialfashionDeleting the data of a namespace removes the data from theblockstores in a serial fashion (that is, one blockstore at atime).DSS-611: Upload buttonon Client daemon webinterface doesn’t workwhen authentication isenabledThis is caused because the upload button does a redirect to alocation on which the logged-in user doesn’t have sufficientrights. There is no API available that provides these rights tothe user.N/ADSS-869: Inconsistentoutput of REST directoryentries listingWhen requesting directory entries in REST using the variousapplication types, the output is not consistent.N/ADSS-1037: Unable toupload file when IFNONE MATCH HTTPHeader is defined anduser/password are alsodefinedDue to a bug in the libcurl, when trying to PUT a file whiledefining a user, a password and an IF NONE MATCH HTTPheader, you get a failure instead of success.Lattus Release NotesN/ACreating a directory while its parent directory doesn’t existproduces a HTTP 409 “Conflict” message.N/AWhen a system has run full, it takes up to the n-th blockstorebefore the system can ingest data again. If these deleteshappen in parallel fashion, the space should be freedimmediately on all blockstores, making the system usableimmediately.While the output for listing directories in XML is fine, it is notwhile using JSON.N/AThe curl initially writes (PUT) a file of 0 bytes. Then the curlmakes a second PUT request on the same connection to writethe actual contents of the file. Since the file already existsbecause of the first PUT, the second PUT fails.13

S3 REST Interface IssuesIssueDescriptionWorkaroundDSS-1108: Client daemonreturns an empty HTTP200 response instead of anerror if not enoughblockstores are onlineIf two or more storage daemons are stopped in yourenvironment and you try to read a file that was stored on thosestorage daemons, you should receive an error.N/AInstead, the AXR and S3 interface return a HTTP 200 responsewith a correct header, but without either empty content or anincompleteread() message.S3 REST Interface IssuesIssueDescriptionWorkaroundDSS-1866: URL withuppercase charactersincorrectly validatedIf the URL of an S3 request contains uppercase characters,such as mybucket.WorldCom.cloud.com, the bucket is notfound due to an incorrect URL validation.N/ADSS-1738: Create AXRdirectory in S3 bucketIt is possible to create an object with the "directory" type inS3 buckets via the CLI and the Q-Shell. This type of object issupported only in AXR namespaces; S3 is not aware ofdirectories.N/ADSS-1728: Query parametersare treated with casesensitivityWhen executing an S3 request with query parameters, thestring of the query parameters was treated with casesensitivity. The case of the string should be ignored.N/ADSS-1721: HTTP 501 NotImplemented for multipartupload with Java SDKWhen you use the Java SDK for a multipart upload, youreceive an HTTP 501 - Not Implemented error, due to anextra line in the POST request (REQUEST LINE: “0”).N/ADSS-1717: UTF-8 charactersnot displayed correctly in XMLresponseThe Lattus S3 interface doesn’t display UTF-8 characterscorrectly in XML responses. For example, “é” is representedby two pipe signs “ ”N/ADSS-1718: Signing a canonicalresource request is casesensitiveWhen signing a canonical resource request, the Lattus S3Interface does not respect the case sensitivity of thecanonical s an incorrect “Location”When completing a multipart upload via Lattus S3, thelocation property of the object points to s3.amazonaws.cominstead of the pn domain, as defined in the setup.N/ADSS-1481: “Byte offset read”operation behavior same asAmazon S3When reading the last byte of a file in Lattus, the complete filewas returned, not an exception “HTTP 416 Requested RangeNot Satisfiable” as expected.N/ALattus Release Notes14

S3 REST Interface IssuesIssueDescriptionWorkaroundDSS-1659: Get Bucket doesnot take CommonPrefixes intoaccount with max-keyparametersWhen the max-keys parameter is enabled and you perform aGET Bucket ( list objects), you get all common prefixes andone object key, where only one common prefix is expected.N/ADSS-1670: HEAD request failswith “403:Forbidden” on bucketwith enabled anonymous GETrequestWhen you have a bucket with “anonymous GET request”enabled and you execute a HEAD request, you receive anHTTP error “403:Forbidden”.N/ADSS-1641: Authentication errorwhen using Java SDK toAmpliStor S3 interface viaproxyWhen there is a proxy between a Java application and theAmpliStor S3 interface, there are authentication errors due toa wrong string to sign (missing trailing slash).N/ADSS-1568: “GET /?versioning”Response cannot be parseddue to missing new linecharacterThe response of the request “GET /?versioning” cannot beparsed due to a missing new line character after the ?xmlversion "1.0" encoding "UTF-8"? element.N/ADSS-1569: Different responsesbetween AWS S3 and LattusS3 for some time zone formatsIn Lattus, some x-amz-date time zone formats might returndifferent responses (internal server error, forbidden)compared to AWS S3.N/ADSS-1561: S3 requests failwhen using unsupported timezonesWhen using unsupported time zones (for example IranStandard Time - IRST), an S3 request fails with an HTTP 500Internal Server Error.N/ADSS-1566: Uncompletedmultipart objects are notdeleted when their bucket isdeletedWhen deleting a bucket that contains incomplete multipartuploads, these incomplete parts are not removed from theblockstore directories, but they should be removed as well.N/ADSS-1543: Socket timeoutwhen updating a multipartobjectWhen you update an object or multipart object with a regularPUT request without update permissions, you can get asocket timeout instead of an “access denied” response.N/ADSS-1521: Sorting order formultipart uploads in progressare not based on creation timeWhen listing multipart upload in progress, the sorting order isby object name. If the object name has multiple multipartuploads associated with it, they should be ordered bycreation time, but they aren’t.N/ALattus Release Notes15

S3 REST Interface IssuesIssueDescriptionWorkaroundDSS-1498: Cancelled multipartupload has difference betweenbehavior of BitSpread andAmazonAfter cancelling a multipart upload and then trying to uploadother parts of the multipart, there is a difference in thebehavior.N/AAmazon: socket timeoutBitSpread: S3ResponseError: 404 Not Found. ?xml version "1.0" encoding "UTF-8"? Error Code NoSuchUpload /Code Message The specified multipart uploaddoes not exist.The upload ID might be invalid, or themultipart upload might have been abortedor completed. /Message RequestId /RequestId Resource testfile7 /Resource /Error DSS-1393: S3 GET requestswith multiple byte ranges is notconsistent with the Amazon S3interfaceWhen sending a GET request with the “Range” parameterhaving multiple byte ranges in the header, the whole object isreturned instead of the requested range.DSS-1398: Force-creating abucket on a syncstore with aspecial name generatesunclear HTTP error responsesForce-creating a bucket in a syncstore with an syncstore IDthat is not nicely 32 HEX characters, raises unclear errorresponses.There is also a concatenation in the Lattus S3 interface.N/AFor example:llLattus Release NotesN/AA syncstore with “aaaabbbbccccddddeeeeffff00001111”returns 404 Syncstore Not Found.A syncstore with “spaghetti arrabiata” returns 400 BadRequest, where it should be 400 Bad Syncstore ID16

S3 REST Interface IssuesIssueDescriptionWorkaroundDSS-1262: Issuing simple GET/ on the S3 service returnsHTTP 500 errorThe GET / command pointed in a browser at an S3 endpointresults in an HTTP 500 error.N/ADSS-1263: PUT fails viaS3cmd with extra ‘/’ in the pathnameIf you try something similar to the following, you will receivean HTTP 403: SignatureDoesNotMatch error:N/ADSS-1271: Copying an objectwith an invalid option in the “xamz-metadata-directive” failswith an error response that isinconsistent with AmazonAWSIf you copy an object using the S3 interface and a value otherthan “COPY” or “REPLACE” is set in the “x-amz-metadatadirective” header, the error message sent in the HTTPresponse by the Amazon AWS S3 Interface is different fromthe message returned by the Lattus S3 interface.N/ADSS-1281: Creating bucketswith illegal characters returns adifferent response thanAmazonIf you create (a)

Lattus-D product is initially qualified for use with the CommVault Simpana 10 Cloud Connector technology when using the S3 protocol to communicate. Purpose of This Release This document announces the release of the Lattus 3.4.4 release. This release includes fixes described in the Resolved Issues section.