Transcription
Catalogue Open Data Protocol (OData API)User GuideContent1.Introduction . 31.1. Purpose. 31.2. Change register. 31.3. Structure of the document . 31.4. Acronyms . 41.5. Reference Documents. 42.Open Data Protocol overview . 62.1. Entity Data Model concept . 6Entity Type .7Entity Set .73.ONDA OData Entity Data Model . 83.1. ONDA Entity Types . 8Product .8Metadata .83.2. ONDA Entity Sets. 94.How to create ONDA OData queries . 104.1. Query string options .10Query option top . 11Query option skip . 12Query option count . 12Query option select . 12Query option format . 13Query option orderby . 13Query option search . 13Query option filter [To be implemented] . 14Built-in functions [To be implemented] . 14Operators . 145.ONDA OData queries by examples . 165.1. Querying all the Entities of the ONDA OData API .165.2. Querying products in the Catalogue .165.3. Querying Products, showing all Metadata together with their Properties .165.4. Querying Products, showing all the Properties (but not the quicklook) and Metadata: .175.5. Querying a single Product in the archive .175.6. Search product metadata .185.7. Search a specific metadatum .18Catalogue Open Data Protocol – User Guide Issue 1.7, February 2019 Ref.: COPE-SERCO-IF-18-0285Page 1 of 22ONDA by Serco is a registered trademark of Serco Italia S.p.A.
5.8. Querying the products (paging) .185.9. Filter the products on time-based criteria.18Filtering the Products by Creation date . 18Filtering the Products by Sensing Time (start and stop) . 195.10.Filtering the Products using the file name.205.11.Download full product from its ID .205.12.Sort products by creation date .205.13.Search products with specific metadata .215.14.Discover the products over a predefined Area Of Interest (AOI): Geographical Search21POLYGON . 21Table of FiguresFigure 1 – EDM key concepts . 6Figure 2 – URI components scheme . 10Table IndexTable 1 – Acronyms and Abbreviations . 4Table 2 – Reference Documents . 4Table 3 – ONDA entity sets. 9Table 4 – Query Options . 11Table 5 – OData built-in functions . 14Table 6 – Comparison and Logical Operators . 15Catalogue Open Data Protocol – User Guide Issue 1.7, February 2019 Ref.: COPE-SERCO-IF-18-0285Page 2 of 22ONDA by Serco is a registered trademark of Serco Italia S.p.A.
1. Introduction1.1. PurposeThis document provides end users guidance on usage of the ONDA Open Data Protocol(OData) API, exposed by the Catalogue, which allows interactive data discovery and download,via computer programs/scripts.1.2. Change irst Issue1.127/06/2018Review1.205/07/2018Sections 3.1.1,3.1.2, 3.2, 5.4Metadatas Entity has beenrenamed to Metadata1.309/07/2018Section 1.5Links updatedownloadable and footprintproperties have been added.Section 1.5, 3.1.1and 3.1.21.403/09/2018Section 4.1.6 and5.12Additional references tometadata indexes have beenadded.Correction of the queryconcerning the sort by creationdate.ONDA by Serco registeredtrademark added1.526/11/2018Footer1.629/01/2019Par. 5.8Updated1.728/02/2019New Paragraphs5.3, 5.4 and 5.7have been addedPossibility to search a specificmetadatum of a product hasbeen implemented1.3. Structure of the documentThis document is composed of the following sections: Section 1 is this Introduction. Section 2 contains an overview of what the Open Data Protocol (OData) is, adescription of the Entity Data Model (EDM) and its Entities.Catalogue Open Data Protocol – User Guide Issue 1.7, February 2019 Ref.: COPE-SERCO-IF-18-0285Page 3 of 22ONDA by Serco is a registered trademark of Serco Italia S.p.A.
Section 3 provides a description of the ONDA OData Entity Data Model, its Entities andrelated Properties. Section 4 provides the basic criteria to build an OData query by means of differentfilters. Section 5 contains an exhaustive list of examples to use the ONDA OData API exposedby the Catalogue. Beginners to OData can also leverage this Section as a structuredway to learn.1.4. AcronymsTable 1 – Acronyms and AbbreviationsAcronymDefinitionADApplicable DocumentAPIApplication Programming InterfaceASCIIAmerican Standard Code for Information InterchangeEDMEntity Data ModelESAEuropean Space AgencyHTTPHypertext Transfer ProtocolHTTPSHyper Text Transfer Protocol SecureJSONJavaScript Object NotationRDReference DocumentUUIDUniversally Unique IdentifierURIUniform Unique IdentifierURLUniform Resource Locator1.5. Reference DocumentsTable 2 – Reference DocumentsIDDocument TitleReferenceRD-1OData v4 -2Metadata for Sentinel-1 sentinel-1/RD-3Metadata for Sentinel-2 sentinel-2/RD-4Metadata for Sentinel-3 sentinel-3/Catalogue Open Data Protocol – User Guide Issue 1.7, February 2019 Ref.: COPE-SERCO-IF-18-0285Page 4 of 22ONDA by Serco is a registered trademark of Serco Italia S.p.A.
RD-5Metadata for Envisat envisat/RD-6Metadata for Copernicus Land copernicus-land/RD-7Metadata for Copernicus Marine copernicus-marine/RD-8Metadata for Landsat-8 landsat-8-geotiff/Catalogue Open Data Protocol – User Guide Issue 1.7, February 2019 Ref.: COPE-SERCO-IF-18-0285Page 5 of 22ONDA by Serco is a registered trademark of Serco Italia S.p.A.
2. Open Data Protocol overviewOData (Open Data Protocol) is a standard that defines conventions, rules and formats forhandling data on the web using Hypertext Transfer Protocol (HTTP/HTTPS) requests.OData is based on the Representational State Transfer (REST) architecture, which allowresources – identified using Uniform Resource Identifiers (URIs) – to be published and edited byWeb clients using simple HTTP messages. The OData specification defines a set of rules forconstructing URIs to identify the data and metadata exposed by an OData server as well as aset of reserved URI query string operators.In the case of the ONDA, the OData interface exposed by the Catalogue allows browsing,selecting and downloading of EO products in the Catalogue itself.2.1. Entity Data Model conceptThis Section provides a high-level description of the Entity Data Model (EDM): the abstractdata model that must be used to describe the data exposed by an OData service.The main concepts in describing the structure of data in EDM are depicted in Figure 1 anddescribed in the following paragraphs.Figure 1 – EDM key conceptsCatalogue Open Data Protocol – User Guide Issue 1.7, February 2019 Ref.: COPE-SERCO-IF-18-0285Page 6 of 22ONDA by Serco is a registered trademark of Serco Italia S.p.A.
As we will see in detail in this Section, the EDM organizes entities into a simple hierarchy. Eachentity is part of an entity set (2.1.2), and each entity set belongs to an entity container. Entities,each of which is of some entity type (2.1.1), also have a simple structure: they containproperties, each of which contains data that this entity holds. To describe the data in properties,the EDM defines a variety of data types, such as String, Boolean, Int16, Int32, Binary andDateTime. Special properties – called navigation properties – represent associations andimplement connections between entities.Entity TypeAn Entity Type is the fundamental building block for describing the structure of data in theEDM. Entity types represent a specific type of data.Each Entity Type comprises: a unique name a unique key (for uniquely identifying instances of Entity Types and allowing Entity Typeinstances to participate in relationships) data in form of Properties Navigation Properties (optional)Entity SetEntitySets are collections of instances of EntityTypes. Each instance of an entity within anentity set can be accessed by its unique key.Catalogue Open Data Protocol – User Guide Issue 1.7, February 2019 Ref.: COPE-SERCO-IF-18-0285Page 7 of 22ONDA by Serco is a registered trademark of Serco Italia S.p.A.
3. ONDA OData Entity Data ModelThis Section specifies the ONDA OData Entity Data Model. The current version of the ONDAOData service is based on OData v4.0 [RD-1].3.1. ONDA Entity TypesEach Entity and its properties are listed in the Tables of the following paragraphs.ProductEntityType nameProductDescriptionThe Product Entity represents the fundamental data element which can be managed by the ONDA.Corresponding EntitySet nameProductsPropertiesTypeDescriptionId etBooleanUnique identifier of the Product.Product NameTime the Product was archivedStatus of the DownloadableNested EntityStringBinaryBooleanSize of the ProductVirtual path used to classify the product inENS filesystem.Footprint (polygon or multi-poly)Quicklook of the product (if available)Status of downloadabilityMetadataMetadataEntityType nameMetadataDescriptionThe Metadata Entity contains the full set of metadata (attributes) extracted at ingestion time for each specificmission (e.g. Sentinel-1, Sentinel-2, Sentinel-3, etc)Corresponding EntitySet nameMetadataPropertiesID (key)NameValueNested EntityTypeDescriptionStringStringStringUnique ID of the Metadatum ( name)Name of the MetadatumValue of the Metadatum (if present)Catalogue Open Data Protocol – User Guide Issue 1.7, February 2019 Ref.: COPE-SERCO-IF-18-0285Page 8 of 22ONDA by Serco is a registered trademark of Serco Italia S.p.A.
NoneThe following link refer to the lists of all the Metadata nested properties of the Product entityType (see Paragraph 3.1). These properties change depending on the Product mission. Metadata Index S1: see [RD-2], Metadata Index S2: see [RD-3], Metadata Index S3: see [RD-4], Metadata for Envisat products: see [RD-5], Metadata for Copernicus Land products: see [RD-6], Metadata for Copernicus Marine products: see [RD-7], Metadata for Landsat-8: see [RD-8].3.2. ONDA Entity SetsThe entity sets are divided in two categories:-The top-level EntitySets, contained in the highest framework. These entity sets areaccessible with the following query:https://[ONDA Catalogue Hostname]/dias-catalogue/EntitySet-The nested EntitySets, accessible via an association starting from a top level EntitySet. The OData query described above does not recognize entities that are nestedwithin other entities. This Entity Set can be explored through the “parent” Entity Set bymeans of the key property.Table 3 – ONDA entity setsONDA entity setProductsMetadataEntity set levelFirst levelNestedCatalogue Open Data Protocol – User Guide Issue 1.7, February 2019 Ref.: COPE-SERCO-IF-18-0285Page 9 of 22ONDA by Serco is a registered trademark of Serco Italia S.p.A.
4. How to create ONDA OData queriesIn this Section we describe the rules for constructing URIs allowing access to data Entities andProperties listed in Section 0.The general scheme summarizing the URI components is depicted in Figure 2.Figure 2 – URI components schemeA URI used by an OData service has up to three significant parts: The service root (Scheme Host:Port ServiceRoot); The resource path; The query string options.For Example:The service root URI identifies the root of an Odata service. For the ONDA Odata interface, theScheme is the protocol HTTPS, Host represents the ONDA server IP address or hostname[ONDA Catalogue Hostname]).The ResourcePath identifies the resource to be interacted with and enables any aspect of thedata model exposed by an Odata service to be addressed. For the ONDA interface, it isProducts.QueryOptions are additional parameters for the query which specify which data are returnedand how they are formatted. We will analyse the query options in the Section 4.1, in order tohelp the user go through the most common and useful scenarios.4.1. Query string optionsOData supports various kinds of query options for querying data. System query options arequery string parameters that control the amount and order of the data returned for the resourceidentified by the URL. The names of all system query options are prefixed with a dollarcharacter, ( ). A query string starts with a question mark (?), and the query options areseparated by an ampersand (&). The asterisk (*) is used to specify all values. Each query optioncan be set on a particular value with ( ).Catalogue Open Data Protocol – User Guide Issue 1.7, February 2019 Ref.: COPE-SERCO-IF-18-0285Page 10 of 22ONDA by Serco is a registered trademark of Serco Italia S.p.A.
A query string option (QueryOption of Figure 2) can be represented in this way: query option A{ value 1}& & query option N{ value 2}The query options admitted by the Data Hub service are listed in the following Table:Table 4 – Query OptionsQuery OptionDescriptionReference topdetermines the maximum number of records to returnSection 0 skiprequests the number of items in the queried collection that areSection 0to be skipped and not included in the result countallows clients to request a count of the matching resourcesincluded with the resources in the response.Section 4.1.3 selectspecifies a subset of properties to return /allows clients torequests a specific set of properties for each entitySection 4.1.4 formatspecifies the HTTPS response format e.g. XML or JSON/allows clients to request a response in a particular formatSection 4.1.5 searchrestricts the result to include only those entities matching thespecified search expression. The definition of what it means to Section 4.1.7match is dependent upon the implementation.Query option topA data service URI with a top System Query Option identifies a subset of the Entities in anEntitySet identified by the Resource Path section of the URI.This subset is formed by selecting only the first M items of the set, where M is an integer greaterthan or equal to zero specified by this query option. If a value less than zero is specified, theURI should be considered malformed.Syntax:https://[ONDA Catalogue Hostname]/dias-catalogue/Products? top M ONDA example:List the last 10 Sentinel-1 products published on the ONDA Catalogue:https://[ONDA Catalogue Hostname]/dias-catalogue/Products? search "name:S1*"& top 10Catalogue Open Data Protocol – User Guide Issue 1.7, February 2019 Ref.: COPE-SERCO-IF-18-0285Page 11 of 22ONDA by Serco is a registered trademark of Serco Italia S.p.A.
Query option skipA data service URI with a skip System Query Option identifies a subset of the Entities in anEntitySet identified by the Resource Path section of the URI.That subset is defined by seeking N Entities into the EntitySet and selecting only the remainingEntities (starting with Entity N 1). N is an integer greater than or equal to zero specified by thisquery option. If a value less than zero is specified, the URI should be considered malformed.The syntax is:https://[ONDA Catalogue Hostname]/dias-catalogue/Products? skip N A client can request a particular page of items by combining top and skip.Query option countThe count system query option allows clients to request a count of the matching resourcesincluded with the resources in the response.Syntax:https://[ONDA Catalogue Hostname]/dias-catalogue/EntitySet/ countONDA example:Total number of all products in the ONDA Catalogue:https://[ONDA Catalogue Hostname]/dias-catalogue/Products/ countTotal number of Sentinel-2 products in the ONDA Catalogue:https://[ONDA Catalogue Hostname]/dias-catalogue/Products/ count? search "name:S2*"Query option selectThe select system query option allows the clients to requests a limited set of properties foreach entity. The value of a select System Query Option is a comma-separated list of selectionclauses.Syntax:https://[ONDA Catalogue Hostname]/dias-catalogue/Entity? select Property 1[,Property 2]ONDA example:Querying the ID property of all the Productshttps://[ONDA Catalogue Hostname]/dias-catalogue/Products? select idQuerying the Name and CreationDate properties of all the Productshttps://[ONDA Catalogue Hostname]/dias-catalogue/Products? select name,creationDateCatalogue Open Data Protocol – User Guide Issue 1.7, February 2019 Ref.: COPE-SERCO-IF-18-0285Page 12 of 22ONDA by Serco is a registered trademark of Serco Italia S.p.A.
Query option formatThe format system query option allows clients to request a response in a particular format. Thedefault format is XML. Valid values for the format query string option are: atom, xml, jsonONDA example:Display all products in the archive in Json format:https://[ONDA Catalogue Hostname]/dias-catalogue/Products? format jsonDisplay all products in the archive in XML format:https://[ONDA Catalogue Hostname]/dias-catalogue/Products? format xmlQuery option orderbyThe orderby system query option allows clients to request resources in either ascending ordescending order.Syntax:https://[ONDA Catalogue Hostname]/dias-catalogue/Entity? orderby property [asc desc]ONDA example:Querying product sorted by ascending creation datehttps://[ONDA Catalogue Hostname]/dias-catalogue/Products? orderby creationDate%20ascQuery option searchA URI with a search System Query Option identifies a subset of the Entities from an EntitySetidentified by the Resource Path section of the URI. The subset is determined by selecting onlythe Entities that satisfy the predicate expression specified by the query option.Syntax:https://[ONDA Catalogue Hostname]/dias-catalogue/Entity? search " keyword : values [AND keyword : value ]"Depending on the keyword, the value(s) can be specified as a single value or range of values.Search keywords can be combined with each other using Operators.ONDA example:Querying Products in a specific creation date rangeCatalogue Open Data Protocol – User Guide Issue 1.7, February 2019 Ref.: COPE-SERCO-IF-18-0285Page 13 of 22ONDA by Serco is a registered trademark of Serco Italia S.p.A.
https://[ONDA Catalogue Hostname]/dias-catalogue/Products? search uerying Sentinel-1 MSI Productshttps://[ONDA Catalogue Hostname]/dias-catalogue/Products? search "instrumentShortName:MSI"Querying Sentinel-3 SRAL Productshttps://[ONDA Catalogue Hostname]/dias-catalogue/Products? search "instrumentShortName:SRAL"Querying products with SLC product typehttps://[ONDA Catalogue Hostname]/dias-catalogue/Products? search "productType:%20IW SLC 1S"Query option filter [To be implemented]A URI with a filter System Query Option identifies a subset of the Entities from an EntitySetidentified by the Resource Path section of the URI. The subset is determined by selecting onlythe Entities that satisfy the predicate expression specified by the query optionOData supports aset of basic predicates and built-in functions for search, including operators. They aredescribed in the following Paragraphs.Built-in functions [To be implemented]A set of functions is defined for use with filter system query option. The following Tabledescribes these OData functions, specifying the ones that are available in the current ONDAversion.Table 5 – OData built-in functionsString functionsFunctionSyntaxsubstringofbool substringof(string p0, string p1)endswithbool endswith(string p0, string p1)startswithbool startswith(string p0, string p1)DescriptionThe substringof function returns records withnames containing a particular string at anyposition.The endswith function returns true if the firstparameter string value ends with the secondparameter string value, otherwise itreturns false.The startswith function returns true if the firstparameter string value starts with the secondparameter string value, otherwise itreturns false.OperatorsThe operators supported in the expression language are shown in the following table, specifyingthe ones that are available in the current ONDA version.Catalogue Open Data Protocol – User Guide Issue 1.7, February 2019 Ref.: COPE-SERCO-IF-18-0285Page 14 of 22ONDA by Serco is a registered trademark of Serco Italia S.p.A.
Table 6 – Comparison and Logical OperatorsOperatorSymbolDescriptionEqualseqThe eq operator returns true if the left operand is equal to the rightoperand, otherwise it returns false.The null value is equal to itself, and only to itself.Not equalsneThe ne operator returns true if the left operand is not equal to the rightoperand, otherwise it returns false.The null value is not equal to any value but itself.gtThe gt operator returns true if the left operand is greater than the rightoperand, otherwise it returns false.If any operand is null, the operator returns false.For Boolean values true is greater than false.Greater than or equalgeThe ge operator returns true if the left operand is greater than or equalto the right operand, otherwise it returns false.If only one operand is null, the operator returns false. If both operandsare null, it returns true because null is equal to itself.Less thanltThe lt operator returns true if the left operand is less than the rightoperand, otherwise it returns false.If any operand is null, the operator returns false.leThe le operator returns true if the left operand is less than or equal tothe right operand, otherwise it returns false.If only one operand is null, the operator returns false. If both operandsare null, it returns true because null is equal to itself.andThe and operator returns true if both the left and right operands evaluateto true, otherwise it returns false.The null value is treated as unknown, so if one operand evaluates to nulland the other operand to false, the and operator returns false. All othercombinations with null return null.OrorThe or operator returns false if both the left and right operands bothevaluate to false, otherwise it returns true.The null value is treated as unknown, so if one operand evaluates to nulland the other operand to true, the or operator returns true. All othercombinations with null return null.NotnotThe not operator returns true if the operand returns false, otherwise itreturns false.The null value is treated as unknown, so not null returns null.Wildcard*any sequence of zero or more charactersLogical operatorsGreater thanLess than or equalAndCatalogue Open Data Protocol – User Guide Issue 1.7, February 2019 Ref.: COPE-SERCO-IF-18-0285Page 15 of 22ONDA by Serco is a registered trademark of Serco Italia S.p.A.
5. ONDA OData queries by examplesThis Section contains an exhaustive list of examples to query and filter Entities and Properties,and therefore can be used as a “beginner user manual”.5.1. Querying all the Entities of the ONDA OData APISyntax:https://[ONDA Catalogue Hostname]/dias-catalogue/Response Payload:5.2. Querying products in the CatalogueThe following OData URI returns all the products stored in the ONDA Catalogue. Each productrecord includes the Id, the product name and its properties, the link for download and the link tometadata.Syntax:https://[ONDA Catalogue Hostname]/dias-catalogue/ProductsBy default it provides a list of 100 records sorted by creation date and arranged in descendingorder.5.3. Querying Products, showing all Metadata together with theirPropertiesThe following OData URI returns all the products stored in the ONDA Catalogue, with theirproperties, and showing also all their Metadata in the response payload.Syntax:https://[ONDA Catalogue Hostname]/dias-catalogue/Products? expand MetadataUsing skip and top options, it is possible to choose how many products are requested.Catalogue Open Data Protocol – User Guide Issue 1.7, February 2019 Ref.: COPE-SERCO-IF-18-0285Page 16 of 22ONDA by Serco is a registered trademark of Serco Italia S.p.A.
5.4. Querying Products, showing all the Properties (but not thequicklook) and Metadata:It is also possible to show, together with all the Metadata, only a subset of chosen Properties,specifying which of them.For example:https://[ONDA Catalogue Hostname]/diascatalogue/Products? expand Metadata& select eudopath,footprint,downloadable5.5. Querying a single Product in the archiveSyntax:https://[ONDA Catalogue Hostname]/dias-catalogue/Products(Id)For example:https://[ONDA Catalogue 97-8812-81fa52d6bd1a)The request returns an individual entity of type Product by the given Id '1307ac04-597e-4b978812-81fa52d6bd1a'Catalogue Open Data Protocol – User Guide Issue 1.7, February 2019 Ref.: COPE-SERCO-IF-18-0285Page 17 of 22ONDA by Serco is a registered trademark of Serco Italia S.p.A.
5.6. Search product metadataSyntax:https://[ONDA Catalogue Hostname]/dias-catalogue/Products(Id)/MetadataFor example:https://[ONDA Catalogue 97-8812-81fa52d6bd1a)/MetadataThe request returns all the metadata of the Product identified by the Id '1307ac04-597e-4b978812-81fa52d6bd1a'.5.7. Search a specific metadatumSyntax:https://[ONDA Catalogue meta
In the case of the ONDA, the OData interface exposed by the Catalogue allows browsing, selecting and downloading of EO products in the Catalogue itself. 2.1. Entity Data Model concept This Section provides a high-level description of the Entity Data Model (EDM): the abstract
