Transcription
NOAA EnvironmentalData ManagementUpdate for Unidata SAC 2014-10-08Jeff de La Beaujardière, PhDNOAA Data Management Architectjeff.deLaBeaujardiere@noaa.gov 1 301-713-7175
NOAA data are unique,valuable, and irreplaceable Many observing systems: 10 satellites 3 buoy networks 120 weather radars 200 tide gauges 17 ships 10 aircraft 5 supercomputers human observers animal telemetry Scope is from the bottom of theocean to the surface of the Sun Wide variety of data collected for different purposes Many formats & dissemination methods Operational & legacy systems(slide adapted from "NOAA 101" briefing)jeff.deLaBeaujardiere@noaa.gov2014-09-092
Vision for NOAA Data ManagementAll NOAA environmental data are to oaa.govDiscoverablefor all types of users and applications.2014-09-09Authorities: NOAA Administrative Order 212-15 (2010) OMB Open Data Policy (2013) OSTP Public Access to Research Results memo (2013)3
EDMC Procedural pData Management PlanningDescribe how you will preserve,document and distribute your data.(2011) In revision 2014Data Center process for approvingarchive requests.(2008) Reviewed 2014; no changeAssign persistent identifiers todatasets and encourage citation.(in preparation 2014)Data AccessData Sharing by GranteesOcean Data AcquisitionsData DocumentationGrantees write data sharing plan,and share data within 2 years.(2012; to be revised 2015 per PARR)Conversion of NAO 216-101 toEDMC PD(1990; in preparation 2015)2014-09-09How to apply ISO 19115 metadatafor discovery, use & understanding.(2011; no change)Make data accessible, preferablyvia on-line services. REcommendspecific service and formats forparticular classes of data.(in preparation 2014)jeff.deLaBeaujardiere@noaa.govArchive ApprovalData Citation4
NOAA Environmental Data Management FrameworkData Management taLifecycleDataLifecyclePurpose:To organize, guide andsupport NOAAenvironmental datamanagement .php5
White House Policies (2013)Public Access toResearch ResultsOpen Data Policy(OSTP memo 2013-02-22)(Exec. Order /files/microsites/ostp/ostp public access memo ntResearchDataDM 13/m-13-13.pdf6
NOAA PARR Plan Highlights (draft pending OSTP approval)(per OSTP Public Access to Research Results Memorandum)– Consider whether to archive at NOAA National Data Center Requires grantees to include project DM Plan in proposals. Requires submission of final manuscripts to NOAA Central Library.jeff.deLaBeaujardiere@noaa.gov States NOAA will continue existing EDM efforts to ensure data areaccessible, usable, and archived. Assigns responsibility to NOAA Programs and PMs to properlymanage data they produce. Requires grant programs to include summary DM Plan inannouncements for data likely to result from grant.– Visible after 1 year embargo– FY 2016 for new/current intramural data and publications– FY 2017 for new extramural publications– FY 2018 for new extramural data2014-09-09 If plan approved, new provisions take effect in:7
Goal: “Earth Observations Common Framework”Any user tool able to connect to any Earth Observation data mericalModelsData.gov &Other oaa.govEO Common FrameworkTHREDDSData Search & Discovery ServicesCatalogTDS, IDD,ncWMS, sharedncSOSstandardsncISO,ACDDCF, NetCDF,UDUNITSData DocumentationCompatible Formats and Vocabularies2014-09-09DataSourcesData Access odels8
Data Discovery Activitiesdata.noaa.govEstablishedNov 2013Inclusionmandatoryper US OpenData Policyjeff.deLaBeaujardiere@noaa.govHarvests fromexistingmetadatacollectionsOn AmazonFederalGeoCloud2014-09-09Starting tocomputestatistics (e.g.,# with dataaccess URL)9
Data Accessibility Activities NWS Integrated Dissemination Program (IDP) EnterpriseGeospatial Services– Operational hosting at NCEP for NWS & NOAA– THREDDS Data Servers– CF conventions for in situ data– OpenDAT/Unidata Linked Servers (OPULS) grant IOOS THREDDS & Sensor Observation Services Data Center Cloud Pilot Big Data Partnership RFI ( BEDI)– See https://www.fbo.gov/index?s opportunity&mode form&id cdbfd2f6b096dfe93aecae44b67fcc40&tab core& cview 12014-09-09– Copy of NOAA data in Cloud with computing capability– 1st RFI issued Feb 2014; 2nd RFI issued– Industry Day 2014-10-17jeff.deLaBeaujardiere@noaa.gov Unified Access Framework10
Conceptual Model of NOAA Big Data PartnershipcustomersCustomer 1network roduct/App #1App #2App #3integrationanalysisfunctionsfunctionsworking copy of dataagency security boundaryAgency Service Tiermastercopy of dataCatalogAccess vicesmaximum standardizationcommercial cloudproviderCustomer 3jeff.deLaBeaujardiere@noaa.govapplication &product providersCustomer 2maximum diversity(RFI issued 2014-02)11
Data Usability ActivitiesMetadata training webinarsConversion to ISO metadata standard at NODCISO export of NMFS InPort MetadataATRAC metadata editor at NCDCMetadata metrics & diagnostics at NGDCjeff.deLaBeaujardiere@noaa.gov 2014-09-0912
Data Preservation Activities Ongoing preservation & stewardship activitiesat NOAA National Data Centers Common Submission Interface (CSI) Machine-to-Machine Interface (M2M)– National Data Buoy Center using NetCDF groupsfor monthly archive packages2014-09-09 Project: Assignment of permanent dataset IDsto archival datasetsjeff.deLaBeaujardiere@noaa.gov– Over 1.5 PB ingested in FY2014– CLASS interface improvements13
Dataset IdentifierProjectData &Metadataused inPublishedPaperor otherworkNOAA NationalData Center(NCDC, NGDC, NODC)assignscitesIDresolves tolandingpage2014-09-09Three NOAA dataset IDs assigned as ofJuly 2013. Target: 20-30 by Feb 2014.List: http://goo.gl/KGr0Wylinks tojeff.deLaBeaujardiere@noaa.govsubmitted to14
NOAA DOIs Assigned (data & pubs)NOAA DOIs assigned to date:http://search.datacite.org/ui?&q 10.7289# of DOIsjeff.deLaBeaujardiere@noaa.gov2014-09-09Date15
Big Earth Data Initiative (BEDI)– OSTP Earth Observations Assessment– NOAA Observing Systems of Record– USGCRP National Climate Assessment Inter-agency activity coordinated through US Group onEarth Observations (USGEO) Data Management WorkingGroup2014-09-09– Starting more detailed discussion on specific services &approachesjeff.deLaBeaujardiere@noaa.gov 2M FY2015 funding request Improve discoverability, accessibility, & usability of data Focus on "high value" datasets, e.g. from:16
Closing Wishes(Jeff’s opinion, not NOAA statement)jeff.deLaBeaujardiere@noaa.gov2014-09-09 Comprehensive view of NOAA usage of Unidata technologies –difficult to get piecemeal from the inside Bullet-proof software for operational use– Reliable and high-performance– Easy installation– IT security certification & CIO pre-approval Scalable distributions for Cloud use– Glad to see Cloud in 5-year strategy– Use cases include serving Cloud-hosted data, analytical toolsrunning on the Cloud, and efficient transmission of data intoCloud from provider facilities Philosophical question: how can we better leverage, and reducecompartmentalization between, federally-funded activitiesincluding NSF/Unidata, NSF/EarthCube, NOAA activities, etc?17
3 buoy networks 120 weather radars 200 tide gauges 17 ships 10 aircraft 5 supercomputers human observers animal telemetry Scope is from the bottom of the ocean to the surface of the Sun (slide adapted from "NOAA 101" briefing) jeff.deLaBeaujardiere@noaa.gov 2014-09-09 2 NOAA data are unique,
