Published on September 22, 2014
Quontra Solutions: Quontra Solutions we offer online IT training with placements, project assistance in different platforms with real time industry consultants to provide quality training for all it professionals, corporate clients and students etc… http:// www.QuontraSolutions.com/ Email : [email protected] Agenda: Agenda Introduction to DW/BI Concepts Extract, Transform & Load (ETL) Business Intelligence / Reporting DW / BI Concepts: DW / BI Concepts Information Management Strategy: Information Management Strategy Structure the systems and data relationships to provide user-friendly customer access to data in order to provide decision-making information . Adding Value to Data: Adding Value to Data Information Pyramid: Information Pyramid What is a Data Warehouse?: What is a Data Warehouse? A copy of data from one or more On-line Transaction Processing (OLTP) systems specifically structured for Query, Reporting and Analysis (QRA). Data is typically at a summarized level to limit the size and complexity of the data warehouse Data is usually cleansed and merged to create an “apples to apples” comparisons OLTP Systems Data Warehouse End-User Reporting The Idea Behind Data Warehousing: The Idea Behind Data Warehousing Extract & Transform Meaningful & Easy Access Data Warehouse POS FINANCIALS INVENTORY Source Systems BILLING ? Information Data Framework Architecture: Metadata Management Project Management & Quality Assurance Source Systems External Data Data Transformation Data Extraction Data Cleansing Data Integration Data Improvement Operational Data Store Data Warehouse OLAP Query Information Delivery Data Mining Operations & Systems Management Datamarts Enterprise Reporting Framework Architecture Business Intelligence: Business Intelligence Information Data Sales Business Activity Marketing Service Customers Manage Inventory Maintain Accounts Product Performance Sales Performance Customer Behavior Marketing Performance Inventory Monitoring Financial Performance Business Intelligence Integrated Meaningful Consistent Validated Easy to Use Leveragable Timely PowerPoint Presentation: Issues Multiple Versions of the Truth Different definitions for similar data Multiple Reporting Tools and Reports Impacts Performance of Oracle Financials Corp. Chemical O&G Oracle Financials Passport Domestic Oracle Financials European Oracle Financials Passport Adage Maximo TOLAS Novistar Energy Financials Aberdeen Oracle Financials PREMAS Intl. Systems Aberdeen Data Warehouse O&G Data Warehouse Production Operations Financial Operations Data Warehouses Existing Reporting Systems P2000 DFW Merak Tobin Others InPower HR Data Warehouse Chemhouse PowerPoint Presentation: Corp . Chemical O&G Oracle Financials Passport? Domestic Oracle Financials European Oracle Financials Passport Adage Maximo TOLAS Novistar Energy Aberdeen Oracle Financials PREMAS Intl. Systems Aberdeen Data Warehouse O&G Data Warehouse P2000 DFW Merak Tobin Others “Centralized” Data Warehouse(s) Dashboards & KPI Mgmt Budget & Forecasting Application Peoplesoft Consolidated Analysis & Reporting Solution (Cognos Business Intelligence) Possibly Phased Out or Integrated Possibly Phased Out or Integrated Production Operations Financial Operations Data Warehouses Consolidated Reporting Systems Existing Reporting Systems Benefits Single Version of the Truth Business Intelligence environment leads to Timely Analysis & Reporting Consolidated Tool Enables Multiple Economies of Scale - Massaging & Distribution - Desktop/Maintenance - Transaction System Load’ Chemhouse STANDARDIZE, CONSOLIDATE, MINIMIZE, & SIMPLIFY Phased Out HR Data Warehouse Data Warehouse Roles: Data Warehouse Roles Manager Planning and management of entire product or project lifecycle; May assist in ETL & BI Interface design and development Data Warehouse Architect – Applies knowledge of technology options, platforms, and design techniques across product and project lifecycle; responsible for design of overall warehouse process ETL Specialist – Analysis and design of extraction, transformation, and loading strategy; development of ETL scripts and procedures Business Intelligence Specialist – Design and development of multidimensional-cubes & reports; performance and tuning of chosen technologies Web Interface Specialist – Design and development of application interface elements; coordinates interfaces between application components ETL: ETL PowerPoint Presentation: Plan/Forecast/ Analysis Source Systems Data Extract Cleansing and Integration Process Operational Data Store Data Warehouse OLAP Query Datamarts Information Analysis Data Mining Data Visualization Global / Dept/ Business Unit Summary and Analysis Metadata Management Executive Information Systems External Data Data Transformation Data Extraction and Transformation Applying business rules to turn data into useable information Clean up and standardization of consumers, vendors, products, etc. Integration of disparate internal and external data Can be 70% - 80% of effort Issues - Can be difficult and time consuming to define business rules - Extraction tools automate only the more simple tasks Project Management & Quality Assurance Operations & Systems Management Data Extraction and Transformation ETL Challenges: ETL Challenges Warehouse Complexity Destructive vs. Incremental Loads Integration (mergers/acquisitions) Data integration (consolidation of information) Process integration (consolidation of processes) Smaller windows of opportunity Make decision in a shorter period of time due to competitive, global market Global marketplace (DW timing updates) High-profile e-Business initiatives Satisfying requirements Data Volume Source Inclusion Extract Timing Warehouse Complexity + + = ETL - The “Heavy Lifting”: ETL - The “Heavy Lifting” Challenge to develop efficient, consistent methods of gathering and cleansing heterogeneous data Capture and load of data from multiple source systems (both internal and external) Integrates data into a single source Cross-system mapping to standard identifiers (surrogate keys) Aggregation for information delivery and BI initiatives ETL Tools - Only Half the Story: ETL Tools - Only Half the Story Half the story: ETL Tools Extract, Transform, and Load data Transport data between sources and targets Document data element changes (metadata) Administer run-time processes and operations Scheduling Error management Audit logs Statistics ETL Tools – Core Components: ETL Tools – Core Components Databases/Files/ Legacy Apps Metadata Repository Load Extract Transform Target Adapters Source Adapters Databases/Files Runtime Metadata Services Design Manager Metadata Import/Export ETL - The Options: ETL - The Options Software Products “Homegrown” Solutions SQL*LOADER, PL/SQL, GATEWAYS & LINKS Developer-bound Little cohesion between components Niche Players (Oracle Warehouse Builder / DataJunction) Enhanced Scheduling & Logging Not Multi-Warehouse Oriented Informatica Powermart Great UI Powerful Scheduling & Logging High Price Proprietary Transform Language ETL - The Reality: Oracle RDBMS RDBMS Flat File Mainframe DBMS ERP “New” Sources O/S Scripts 3GL Code SQL Scripts PERL Scripts Interface Apps Loader Utility COBOL Code Data Repository ETL - The Reality Informatica Powermart: Informatica Powermart Repository Manger Designer Workflow Manager Workflow Monitor Business Intelligence / Reporting: Business Intelligence / Reporting What is Business Intelligence?: What is Business Intelligence? Business Intelligence is the transformation of data into information you can use to drive your business. There are a number of vendors that have developed Business Intelligence software. Kerr-McGee uses Cognos . Business Intelligence Tools: Source Systems Data Extract Cleansing and Integration Process Data Warehouse Metadata Management Project Management & Quality Assurance External Data Data Transformation Operational Data Store Business Intelligence Tools Business Intelligence Tools Combination of applications and tools Provide analysis, presentation and reporting facilities for users Tailored to meet diverse needs of executives, mgrs , analysts Data may reside in ODS, data warehouse or data mart Issues How do you choose the right tool or tools? Datamarts OLAP Query Plan/Forecast/ Analysis Information Analysis Data Mining Data Visualization Global / Dept/ Business Unit Summary and Analysis Executive Information Systems Project Management & Quality Assurance Operations & Systems Management Categorize Information Needs: Highly Summarized Moderately Summarized Integrated Operational Data Business Users Management Business Analysts Market Researchers Executive Categorize Information Needs Financial analysts, product managers, etc.. Senior Management Salespersons, line managers, administrative staff, etc.. Information Delivery Mechanisms: Information Delivery Mechanisms Operational Trends Web or C/S Wireless Mobile Operational Trends Web or C/S (in millions) 1998 1999 2000 2001 2002 Net Revenues $x,xxx $x,xxx $x,xxx $x,xxx $x,xxx Net income xxx xxx xxx xxx x,xxx Earnings per share x.xx x.xx x.xx x.xx x.xx Return on net revenues xx% xx% xx% xx% xx% Cash & s/t investments $xxx $xxx $xxx $xxx $xxx Total Assets $xxx $xxx $xxx $xxx $xxx Shareholder Equity xxx xxx xxx xxx x,xxx Operational Trends Web or C/S Predefined Summaries Specialized Algorithms Directed Analysis Ad-hoc Queries Standardized Operational Reporting Data Visualization Delivery Mechanism Considerations Integrated with Operations? Detailed Reporting only? Real-time or based on a Periodic Business Cycle (Financials) Tethered or ‘disconnected’? C/S Web Wireless PowerPoint Presentation: B.I. Infrastructure All things Cognos: All things Cognos Cognos is a vendor. The suite of applications we’ve bought and use from Cognos are: Access Manager – (Security) Upfront – (Portal – http:/intranet/kmbi) PowerPlay – (reports/cubes) Impromptu – (Web based PDF reports) NoticeCast – (conditional report notifications) *Visualizer – (graphic depictions of data warehouse/cube information) Terminology: Terminology Cube - A multidimensional way to analyze information, designed to provide quick answers to the who, what, why, when, and where business questions. Drill Down - Going from a summarized view to a more detailed view of information within the same cube Drill Across - Linking data from One Subject Area to Another (General Ledger to Accounts Payable) Drill Through - Linking to source data using selected filters Powerplay Web - On-Line Analysis Tool for cubes (slice/dice, drill down, drill across & drill through) Newsbox - A web based folder used to store views of data (reports). Every KMBI user has their own personal newsbox. Cognos - Upfront: Cognos - Upfront Upfront - Portal Management Cognos - PowerPlay: Cognos - PowerPlay PowerPlay – web reports/slicing and dicing/data analysis, based on cubes. More Information on Cognos website : http://www.cognos.com/products/businessintelligence/analysis/ Cognos - Impromptu: Cognos - Impromptu Impromptu – printable reports (in PDF) that may/or may not be produced with prompts for filtered information. Cognos - Visualizer: Cognos - Visualizer Visualizer – interactive graphic depictions of data warehouse/cube information Developing Visualizations: Developing Visualizations Initial Project meeting should include: Client - gives input on look and feel, data requirements, timelines Project Manager – ensures project is feasible within budget and time restraints at the onset and through out the project. Data Warehouse Architect – ensures all the needed data is in the data warehouse. Business Intelligence Specialist (cube builder) – ensures all the needed data is in the cube, in the correct format Web Interface Specialist (visualization builder) – works with cube builder and client to established look and feel, navigation, chart styles, etc..