LAI, MIT Research Seminar 12/09/2009

Information about LAI, MIT Research Seminar 12/09/2009

Published on December 9, 2009

Author: giombetti

Source: authorstream.com

Content

The influence of quality requirements on the suitability of software cost estimation methods LAI Research Seminar – 9 Dec 2009 : The influence of quality requirements on the suitability of software cost estimation methods LAI Research Seminar – 9 Dec 2009 Marc Giombetti <[email protected]> Outline : Outline Personal introduction My experience at LAI - MIT Motivation Advantages and limitations of different software cost estimation methods Understanding & estimating software quality A controlled experiment – COCOMO II vs. Expert judgment Conclusions & Future work Undergraduate studies in Computer Science in Kaiserslautern, Germany : Undergraduate studies in Computer Science in Kaiserslautern, Germany Bachelor of Science in Computer Science 07’– TU Kaiserslautern Bachelor Thesis conducted externally at Fraunhofer IESE “Evaluating The Architectural Coverage Of Runtime Traces” Supervision: Prof. Dr. Dieter Rombach Areas of interest: Software Architectures Software Engineering Software Quality Assurance Slide 6: München Graduate studies in Computer Science in Munich : Graduate studies in Computer Science in Munich Master of Science in Computer Science 10’– TU München Master thesis conducted at LAI, MIT “The influence of quality requirements on the suitability of software cost estimation models” Supervision: Prof. Dr. Dr. h.c. Manfred Broy Dr. Stefan Wagner (TUM) Dr. Ricardo Valerdi (LAI, MIT) Slide 8: My experience at My experience at LAI - MIT : My experience at LAI - MIT Duration: 6 months - 10 June – 14 December 2009 Different style of academic work Lectures (Cost estimation and measurement systems) Conferences Second International Engineering Systems Symposium - Engineering Systems - June 15-17, 2009 at MIT 3rd International Symposium on Empirical Software Engineering and Measurement – October 15-16 - Lake Buena Vista, Florida MIT Conference on Systems Thinking for Contemporary Challenges - October 22-23 at MIT 24th International Forum on COCOMO and Systems/Software Cost Modeling – November 2-5 at MIT Things I learned in the 6 months at LAI - MIT : Things I learned in the 6 months at LAI - MIT I definitely got a more holistic view on things A new way of thinking (and questioning things) Designing and conducting experiments Statistical methods Interview techniques Presenting work at conferences American culture English language Slide 12: Now to the fun stuff  Research overview : Research overview Analysis of the influences of quality requirements on the suitability of software cost estimation methods. Understanding the perception of quality for human estimators (cf. mental models) Experiment to compare model estimates to expert estimates with regard to quality.  Selection of the right methods by understanding what the actual influences are. Cost estimation in software engineering : Cost estimation in software engineering “Prediction is very difficult, especially if it's about the future“ – Niels Bohr B. Boehm, C. Abts, A.W. Brown, S. Chulani, B.K. Clark, E. Horowitz, R. Madachy, D.J. Reifer, and B. Steece, Software Cost Estimation with Cocomo II, Prentice Hall PTR, 2000. Decision making for projects Tracking projects Reduction of uncertainty over time:Cone of Uncertainty: Is the ability to estimate costs and keep projects on track improving over time? : Is the ability to estimate costs and keep projects on track improving over time? Standish Group Chaos Report Many reasons for failure: technical reasons vs. reasons related to team/human reasons Implausible deadlines due to wrong estimates demotivate teams. Influence factors Quality & Risk Expectations External pressure Integration of my research in the LAI context : Integration of my research in the LAI context D. Nightingale, Lecture Notes “ESD.61J / 16.852J: Integrating the Lean Enterprise”, Massachusetts Institute of Technology, 2009 Formal estimation models or expert judgment ? (1/2) : Formal estimation models or expert judgment ? (1/2) M. Jørgensen and B. Boehm, “Software Development Effort Estimation: Formal Models or Expert Judgment?,” IEEE Software, vol. 26, 2009, pp. 14-19. Formal estimation models or expert judgment […] figure out when, and under what conditions, each method would be best.  This research adjusts this question and asks: When are formal models more useful than expert judgment; given a set of specific quality requirements. COnstructive COst MOdel: COCOMO : COnstructive COst MOdel: COCOMO First version developed by Dr. Barry Boehm in the 1970s. Parametric cost estimation model Estimates the development effort based on: Project characteristics / Type of project Project size in Kilo Delivered Source Instructions (KDSI). Effort adjustment factors EAF (intermediate/detailed model) Calibration based on static analysis of 63 projects EFFORT = a ∙ EAF ∙ (KDSI)b Constructive Cost MOdel: COCOMO : Constructive Cost MOdel: COCOMO COCOMO cost drivers:EAF = Effort Adjustment Factors Product of the cost drivers: EFFORT = a ∙ EAF ∙ (KDSI)b organic: EFFORT = 2,4 ∙ EAF ∙ (KDSI)1,05 semi-detached: EFFORT = 3,0 ∙ EAF ∙ (KDSI)1,12 embedded: EFFORT = 3,6 ∙ EAF ∙ (KDSI)1,20 COCOMO II : COCOMO II COCOMO II was developed during the 1990s Adjustment to modern development lifecycles Calibration on a broader set of data 161 projects.Software process maturity effects and process-driven quality estimation. Reuse driven approaches involving COTS packages, reengineering, application compositions and application generation capabilities Under continuous development Many similar models and derivatives: COSYSMO, COQUALMO, COCOTS, COSECMO, … Expert estimation methods : Expert estimation methods Top-down and bottom-up estimation as expert estimation methods using Work Breakdown Structures. Top-down: A hierarchical decomposition of the system into progressively smaller components is used to estimate the size of a software component Bottom-up: Individuals assess each component; estimates are summed to calculate the total estimate. Formal estimation models or expert judgment ? (2/2) : Formal estimation models or expert judgment ? (2/2) Formal Estimation Models COCOMO, FP based models Mechanical quantification step Dependent on calibration data Objective & repeatable ? Industry usage? – Trust in a method which is not fully understood by PM? Expert judgment Checklist-based estimation, top-down, bottom-up, “gut feelings” Judgment-based quantification step Dependent on individual estimator Biased towards optimism Only 15% of the journal papers analyzed judgment-based effort estimation* The difference between formal estimation models and expert judgment is not always clear  Combination of methods * M. Jørgensen and M. Shepperd, “A systematic review of software development cost estimation studies,” IEEE Transactions on Software Engineering, vol. 33, 2007, pp. 33-53. Software Quality : Software Quality Software quality is an important factor for the success of software of any kind. How much software quality assurance is enough?* For several industries quality is not negociatable! One of the challenges of software quality is that "everyone feels they understand it“** Quality requirements need to be taken into consideration while estimateing the cost of future projects. * L. Huang and B. Boehm, “How Much Software Quality Investment Is Enough: A Value-Based Approach,” Software, IEEE, vol. 23, 2006, pp. 88-95.   ** P.B. Crosby, Quality Is Free, Signet, 1980. ISO/IEC 9126 - Software Quality Model : ISO/IEC 9126 - Software Quality Model International Organization for Standardization, ed., “ISO/IEC 9126-1:2001 - Software engineering - Product quality - Part 1: Quality model,” 2001. Software quality is the totality of features and characteristics of a software product that bear on its ability to satisfy stated or implied needs. The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (1/4) : The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (1/4) System requirements and project settings Domain: Financial application Type: Web application Development team size: 1 Project Manager1 Senior developer,1 QA, 3 Junior developers Expert estimation: Graduate students Security: 13 students Usability: 15 students Engineering vs. computer science students The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (2/4) : The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (2/4) Estimation for development time (no schedule compression) Expert judgment : Top-down or bottom-up approach H1 H2 The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (3/4) : The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (3/4) Research hypotheses H10: In the case of additional security requirements There is no significant difference between the average development time estimated using expert estimation and the intermediate COCOMO II model estimate. H20: In the case of additional usability requirements: There is no significant difference between the average development time estimated using expert estimation and the intermediate COCOMO II model estimate. The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (4/4) : The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (4/4) Experiment Goals G.1. Analyze the relative impact of security and usability requirements on the development time of a web-application using COCOMO II. G.2. Finding evidence if there is a significant difference between the schedule estimate for the web application enhanced by security requirements made using expert estimation compared to COCOMO II. G.3. Finding evidence if there is a significant difference between the schedule estimate for the web application enhanced by usability requirements made using expert estimation compared to COCOMO II. Experiment: COCOMO II Estimate : Experiment: COCOMO II Estimate Experiment: Results (1/3) - Total : Experiment: Results (1/3) - Total Experiment: Results (2/3) – Engineering vs computer science students : Experiment: Results (2/3) – Engineering vs computer science students Experiment Results (3/3) : Experiment Results (3/3) T-Test & Wilcoxon signed rank test at 5% significance level: Reject H1  There IS a significant difference between the estimates of expert estimatiors and COCOMO II for the security enhancement. Reject H2  There is NO significant difference between the estimates of expert estimatiors and COCOMO II for the usability enhancement. Threats to validity Students in the role of expert estimators* Classroom exercise: The requirements provided to the students were simplified, but chosen to be comparable to real requirements (in size & complexity). Selection of quality requirements Small dataset * P. Berander, “Using students as subjects in requirements prioritization,” Proceedings of the 2004 International Symposium on Empirical Software Engineering, IEEE Computer Society, 2004, pp. 167-176. Conclusions & Future work : Conclusions & Future work Perception of quality attributes influences the estimators, but not the COCOMO model estimate. Human estimators were pretty good at estimating usability, but underestimated the development time for high security requirements. Influence of quality for cost estimation – Improvements Preferable: Software professionals with real estimation experience Focus on other quality aspects (maintainability, reliability,…) Conduction of a second experiment on the influence of usability (Focus on FP counting & COCOMO II) Future work: COCOMO II – Estimating the influence of usability : Future work: COCOMO II – Estimating the influence of usability Slide 36: Questions? Slide 37: Thank you! References : References D. Nightingale, Lecture Notes “ESD.61J / 16.852J: Integrating the Lean Enterprise”, Massachusetts Institute of Technology, 2009 Standish Group International, “The Chaos Report,” 1995,2004,2006,2009. M. Jørgensen and B. Boehm, “Software Development Effort Estimation: Formal Models or Expert Judgment?,” IEEE Software, vol. 26, 2009, pp. 14-19.   M. Jørgensen and M. Shepperd, “A systematic review of software development cost estimation studies,” IEEE Transactions on Software Engineering, vol. 33, 2007, pp. 33-53. L. Huang and B. Boehm, “How Much Software Quality Investment Is Enough: A Value-Based Approach,” Software, IEEE, vol. 23, 2006, pp. 88-95. P.B. Crosby, Quality Is Free, Signet, 1980.     International Organization for Standardization, ed., “ISO/IEC 9126-1:2001 - Software engineering - Product quality - Part 1: Quality model,” 2001. P. Berander, “Using students as subjects in requirements prioritization,” Proceedings of the 2004 International Symposium on Empirical Software Engineering, IEEE Computer Society, 2004, pp. 167-176. Slide 39: Backup Slides Expert estimation approach chosen by participants : Expert estimation approach chosen by participants Confidence of estimators : Confidence of estimators

Related presentations


Other presentations created by giombetti

Freiheit im Internet
01. 07. 2008
0 views

Freiheit im Internet

Modellierung mit UML 2
01. 07. 2008
0 views

Modellierung mit UML 2