Environmental Scenario Search Engine

Information about Environmental Scenario Search Engine

Published on June 15, 2007

Author: funnyside

Source: authorstream.com

Content

ESSEEnvironmental Scenario Search Engine for the Data Services Grid:  ESSE Environmental Scenario Search Engine for the Data Services Grid Mikhail Zhizhin, Geophysical Center Russian Academy of Sciences [email protected] Eric Kihn, National Geophysical Data Center NOAA [email protected] www.wdcb.ru:  www.wdcb.ru Geophysical Center Russian Academy of Sciences World Data Centers for Solid Earth and Solar-Terrestrial Physics Environmental data archives – paper, tapes, files, databases, e-journals… International network for geophysical data exchange with the US, Japan, China, … Computer center, Linux cluster, fiber optics Part of the European GRID infrastructure EGEE, Russian GRID Virtual Organization e-Earth 50 years ago – International Geophysical Year – IGY1957:  50 years ago – International Geophysical Year – IGY1957 Total data volume ~ 1 Gb Exchange ~ 1 Mb/year Yesterday – databases, Internet, web – Y2K :  Yesterday – databases, Internet, web – Y2K Total data volume ~ 1 Tb Exchange ~ 1 Gb/year Tomorrow – Electronic Geophysical Year – EGY2007 :  Tomorrow – Electronic Geophysical Year – EGY2007 Total data volume ~ 1 Pb Exchange ~ 1 Tb/year Slide6:  Boulder Moscow Kamchatka Nagoya Sydney Grahamstown SPIDR – Space Physics Interactive Data Resource SPIDR 3 SPIDR 2 http://spidr.ngdc.noaa.gov Beijing Cross-disciplinary data exchange:  Cross-disciplinary data exchange Users need data from different disciplines Rapid growth of the data volume and data demand requires new tools for the data management and the data mining “Metcalfe’s law” for databases:  'Metcalfe’s law' for databases The utility of N independent data sets seems to increase super-linearly One can find N(N-1) ≈ N2 relations between data sources, that is their utility grows ≈ N2 It is more efficient ot use several data sources than one archive Sources of data inflation?:  Sources of data inflation? New versions Derived data products Reanalysis Products of Level 1 (NASA terminology) take 10% of the Level 0 volume, but the number of the Level 1 products is increasing. If the volume of the Level 0 data grows as N, then the volume of Level 1 data is growing as N2. Observations + Model = Reanalysis:  Observations + Model = Reanalysis Direct observations, including raw and processed data, e.g. meteorological station or satellite. Numerical model 'knows' physics, uses direct observations as boundary values, e.g. Global Circulation Model. Input data volume (irregular grid) is less than the output volume (regular grid). Reanalysis – accumulated output of the numerical model runs based on the direct observations for a long time period, say 50 years. D-day reanalysis – morning (after ECMWF):  D-day reanalysis – morning (after ECMWF) June 6th, 1944, midnight June 6th, 1944, 6 AM D-day reanalysis – evening(after ECMWF):  D-day reanalysis – evening (after ECMWF) June 6th, 1944, 6 PM June 6th, 1944, 12 AM Data inflation after reanalysis:  Data inflation after reanalysis Modern global atmospheric circulation model (GCM) at 2.5o (latitude) x 2.5o (longitude) x 20 (levels) = 106 gridpoints. GCM outputs 'high-frequency' data every six hours of simulation time, so ~ 1 Gb of data per simulation day . By contrast, the world-wide daily meteorological observational data collected over the Global Telecommunications System, is ~ 200 Mb. As an extreme, to run the GCM for 50 years of simulation time will provide 40 Tb of data. Space Weather Reanalysis:  Input: ground and satellite data from SPIDR Space weather numerical models Output: high-resolution representation of the near-Earth space Space Weather Reanalysis ESSE solutions:  ESSE solutions Do not use data files, use distributed databases Optimize data model for the typical data request Virtualize data sources using grid (web) services Metadata schema describes parameters, grids, formulas for virtual parameters (e.g., wind speed from U- and V-wind) Search for events in the environment by the 'scenario' in natural language terms Translate the scenario into the parallel request to the databases using fuzzy logic ESSE architecture:  ESSE architecture Fuzzy logic engine performs searching and statistical analysis of the distribution of the identified events Parallel mining of several distributed data sources, possibly from different subject areas Both the fuzzy logic engine and data sources implemented as Grid (web) services Interfaces and data structures can be obtained from the definitions of the web-services (WSDL) Web services and prototype user interface are installed on two mirror servers: Boulder, US Moscow, Russia Parallel database cluster (NCEP reanalysis):  Parallel database cluster (NCEP reanalysis) ESSE “time series” data model:  ESSE 'time series' data model Indexed lat-lon grids of time series in BLOBs What is fuzzy logic?:  What is fuzzy logic? Fuzzy logic uses set membership values between and including 0 and 1, allowing for partial membership in a set. Fuzzy logic is convenient for representing human linguistic terms and imprecise concepts ('slightly', 'quite', 'very'). Fuzzy membership functions What good is fuzzy logic for ESSE?:  What good is fuzzy logic for ESSE? Fuzzy engine allows to build queries in human linguistic terms: (VERY LARGE 'wind speed') AND (AVERAGE 'surface temperature') AND ('relative humidity' ABOUT 60%) You can use the same terms for different value ranges: AVERAGE TEMPERATURE for Africa is not the same as for Syberia. Results are given as a list of 'most likely' events. Each event is assigned a value, representing its 'likeliness'. Slide21:  'High' Wind 'Average' Temperature 'About' 60% Humidity Prototype workflow and UI:  Prototype workflow and UI Prototype UI implemented as a web-application Discover data sources by keyword-based metadata search Use predefined weather events (e.g. 'ice storm', 'flood') Define the event as a combination of fuzzy conditions on a set of environmental parameters (e.g. 'high temperature and low relative humidity') Review statistics for the detected events Visualize the selected event as time series plots or contour maps Download the event data in self-describing format (NetCDF or HDF) to the user’s workstation Setting spatial locations:  Setting spatial locations Select a set of 'probes' (representing spatial locations of interest, e.g. New York) where the desired event may occur. Defining fuzzy search criteria:  Defining fuzzy search criteria Select several parameters for the event from a list. Set the fuzzy constraints on the parameters for the event (e.g. 'very high temperature', 'very high humidity'). Working with scenarios:  Working with scenarios The user may search for a desired scenario by describing several subsequent events Search Results:  Search Results 'Score' represents the 'likeliness' of each event in a numerical form. The results page provides links to visualization and data export pages. Visualizing event as time series:  Visualizing event as time series Visualizing event in 5D:  Visualizing event in 5D Visualizing event from satellites :  Visualizing event from satellites What do we get at the end?:  What do we get at the end? Using the 'time machine', we can see the weather on the D-day, or the Rita hurricane, or the typical September day in San Diego. Statistics to estimate risk from natural disasters, global climate change, realistic weather in movies, computer games, simulators When Tim Berners-Lee uses semantic web to find a photo of the Eiffel Tower on a sunny summer day, ESSE can provide a list of sunny days to be merged with the list of images named with 'eiffel'

Related presentations


Other presentations created by funnyside

HCM Luzern
18. 06. 2007
0 views

HCM Luzern

lecture12
12. 10. 2007
0 views

lecture12

basketball
01. 08. 2007
0 views

basketball

Atacama Large Millimeter Array
15. 06. 2007
0 views

Atacama Large Millimeter Array

gen mtgmar07 slides
18. 06. 2007
0 views

gen mtgmar07 slides

THAKKAR ANJALI artificial blood
16. 10. 2007
0 views

THAKKAR ANJALI artificial blood

20030815 RepICFA HN
27. 09. 2007
0 views

20030815 RepICFA HN

Owen
03. 10. 2007
0 views

Owen

14BBlackHoles
07. 10. 2007
0 views

14BBlackHoles

Noda06
09. 10. 2007
0 views

Noda06

reptile
10. 10. 2007
0 views

reptile

leskov
15. 10. 2007
0 views

leskov

461
17. 10. 2007
0 views

461

Europe Presentation
19. 10. 2007
0 views

Europe Presentation

RadOnc
22. 10. 2007
0 views

RadOnc

schnittsysteme
23. 10. 2007
0 views

schnittsysteme

ch1 10 part2
23. 10. 2007
0 views

ch1 10 part2

firma end Presentation
19. 10. 2007
0 views

firma end Presentation

sta291 5c
29. 10. 2007
0 views

sta291 5c

ACM India Studies 2005
26. 11. 2007
0 views

ACM India Studies 2005

Mirabel
01. 12. 2007
0 views

Mirabel

ghana
10. 12. 2007
0 views

ghana

DanishCitationEventJB final
15. 10. 2007
0 views

DanishCitationEventJB final

aleynikov
26. 10. 2007
0 views

aleynikov

ShortageChicago Seago 2
29. 10. 2007
0 views

ShortageChicago Seago 2

Daylight Saving Time
07. 11. 2007
0 views

Daylight Saving Time

unity
21. 08. 2007
0 views

unity

Fad Diets
21. 08. 2007
0 views

Fad Diets

NACADA Combined Workshop
21. 08. 2007
0 views

NACADA Combined Workshop

TeraGrid SC06 1
22. 10. 2007
0 views

TeraGrid SC06 1

plane parallel 2 1
24. 10. 2007
0 views

plane parallel 2 1

cronologia storia contemporanea
13. 11. 2007
0 views

cronologia storia contemporanea

IDitarod
16. 11. 2007
0 views

IDitarod

Axiome Von Neumann Morgenstern
19. 11. 2007
0 views

Axiome Von Neumann Morgenstern

Dog bite sepsis
19. 11. 2007
0 views

Dog bite sepsis

Girish
21. 11. 2007
0 views

Girish

EcoValidity
17. 12. 2007
0 views

EcoValidity

matthew
29. 12. 2007
0 views

matthew

Cyatheaceae Classification
02. 01. 2008
0 views

Cyatheaceae Classification

RFF Shinnecock
03. 01. 2008
0 views

RFF Shinnecock

AustralianPoetry
03. 01. 2008
0 views

AustralianPoetry

dbl dutch
22. 11. 2007
0 views

dbl dutch

Lecture10 overheads
21. 08. 2007
0 views

Lecture10 overheads

Deborah Russell Presentation
14. 11. 2007
0 views

Deborah Russell Presentation

AAPT v3
01. 08. 2007
0 views

AAPT v3

EScannapieco
15. 11. 2007
0 views

EScannapieco

Successful  Case  Studies
29. 09. 2007
0 views

Successful Case Studies

Interv Hajji
23. 10. 2007
0 views

Interv Hajji

khelif edson rep
09. 10. 2007
0 views

khelif edson rep

aadcrocodile
11. 10. 2007
0 views

aadcrocodile

matricole
16. 10. 2007
0 views

matricole

the three little pigs
14. 02. 2008
0 views

the three little pigs

Exon HKSB
30. 10. 2007
0 views

Exon HKSB

June06SPCCUpdateJVan voorhis
07. 11. 2007
0 views

June06SPCCUpdateJVan voorhis

cultural impact hispanics tn
24. 02. 2008
0 views

cultural impact hispanics tn

SE 05 Punctuation I
29. 02. 2008
0 views

SE 05 Punctuation I

GaryCorbett e
04. 03. 2008
0 views

GaryCorbett e

Radiation Hazards
06. 03. 2008
0 views

Radiation Hazards

Portoroz
04. 10. 2007
0 views

Portoroz

jdean c10
25. 03. 2008
0 views

jdean c10

SoS
25. 03. 2008
0 views

SoS

MVestergaard SantaFe06
29. 11. 2007
0 views

MVestergaard SantaFe06

PowerPointTour
02. 11. 2007
0 views

PowerPointTour

tal daly
10. 04. 2008
0 views

tal daly

matz011
16. 04. 2008
0 views

matz011

installaware faq
17. 04. 2008
0 views

installaware faq

ATMSELong
18. 04. 2008
0 views

ATMSELong

36329938
22. 04. 2008
0 views

36329938

ITCan
10. 03. 2008
0 views

ITCan

kejian4
11. 10. 2007
0 views

kejian4

061213 dialog north korea petrov
11. 10. 2007
0 views

061213 dialog north korea petrov

Session6 Adm Wilcocks
07. 11. 2007
0 views

Session6 Adm Wilcocks

high technology surgery
30. 04. 2008
0 views

high technology surgery

Robot Convoy final presentation
07. 01. 2008
0 views

Robot Convoy final presentation

harry truman
18. 06. 2007
0 views

harry truman

goldrush
18. 06. 2007
0 views

goldrush

GMK
18. 06. 2007
0 views

GMK

gatech
18. 06. 2007
0 views

gatech

fusion be
18. 06. 2007
0 views

fusion be

Fun Facts updated 3 05
18. 06. 2007
0 views

Fun Facts updated 3 05

Geomatic AAU 07032007 forkortet
18. 06. 2007
0 views

Geomatic AAU 07032007 forkortet

General Spring Intro
18. 06. 2007
0 views

General Spring Intro

intervento Ambrosi Legge125
18. 06. 2007
0 views

intervento Ambrosi Legge125

insostegno
18. 06. 2007
0 views

insostegno

Inhalts angabe
18. 06. 2007
0 views

Inhalts angabe

Implant Simulations
18. 06. 2007
0 views

Implant Simulations

implantate berlin
18. 06. 2007
0 views

implantate berlin

Imber stud
18. 06. 2007
0 views

Imber stud

hsm bend2
18. 06. 2007
0 views

hsm bend2

history mat storia dei diritti
18. 06. 2007
0 views

history mat storia dei diritti

headquarters structure
18. 06. 2007
0 views

headquarters structure

Department Of Conservation
23. 10. 2007
0 views

Department Of Conservation

Hippo
18. 07. 2008
0 views

Hippo

pa3 disare ppt
28. 09. 2007
0 views

pa3 disare ppt

Sudan Polio Update 03 05
21. 10. 2007
0 views

Sudan Polio Update 03 05

bradford ancer
21. 08. 2007
0 views

bradford ancer

Delaunay Triangulations
15. 06. 2007
0 views

Delaunay Triangulations

Bayesian analysis and problems
15. 06. 2007
0 views

Bayesian analysis and problems

Biofuels - Think Big
15. 06. 2007
0 views

Biofuels - Think Big

AVO  - Virtual Observatory
15. 06. 2007
0 views

AVO - Virtual Observatory

PHOBOS Collaboration
15. 06. 2007
0 views

PHOBOS Collaboration

gennaipm
18. 06. 2007
0 views

gennaipm

sigcse2003
07. 12. 2007
0 views

sigcse2003

Introductionto Kyrgyzstan
26. 11. 2007
0 views

Introductionto Kyrgyzstan

HIT SAN
18. 06. 2007
0 views

HIT SAN

Metzenbaum Shelley
23. 10. 2007
0 views

Metzenbaum Shelley

ppt CHAP018
24. 02. 2008
0 views

ppt CHAP018

lesson3 warm up
07. 04. 2008
0 views

lesson3 warm up

shelving with lcc
19. 02. 2008
0 views

shelving with lcc

25ConnectionsV5 p2
18. 10. 2007
0 views

25ConnectionsV5 p2

2506043
15. 11. 2007
0 views

2506043

TravelChapter7 3
27. 03. 2008
0 views

TravelChapter7 3

french rfp
18. 06. 2007
0 views

french rfp

BEFxii Coronado
09. 04. 2008
0 views

BEFxii Coronado

spectrumastrotalk
28. 11. 2007
0 views

spectrumastrotalk

metadata frames
05. 10. 2007
0 views

metadata frames

Dancefinal
27. 11. 2007
0 views

Dancefinal