# RR 06 Covert Comparison of SC Model Correlation

Information about RR 06 Covert Comparison of SC Model Correlation

Published on January 16, 2008

Author: Riccard

Source: authorstream.com

Comparison of Spacecraft Cost Model Correlation Coefficients:  Comparison of Spacecraft Cost Model Correlation Coefficients Raymond P. Covert The Aerospace Corporation 15049 Conference Center Drive, CH1-410 Suite 600 Chantilly, VA 20151 E-mail: [email protected] © 2002 The Aerospace Corporation Outline:  Outline Background Calculating Estimate Uncertainty Importance of Correlation Types of Correlation Introduction Frequency Distribution of Correlation Coefficients Comparing Coefficients Between Models Conclusions Calculating Estimate Uncertainty:  Calculating Estimate Uncertainty Total cost variance, s2 s2=sT[r]s [r] = Correlation matrix (full matrix) s = Vector of standard deviations (cost space) Excel Commands SIGMA_TOT=SQRT(MMULT(MMULT(TRANSPOSE(SIGMA),RHO),SIGMA)) Correlation is Essential in calculating variance! Correlation Coefficients:  Correlation Coefficients Different types of correlation Pearson's product-moment (linear) correlation - measure of the linearity between two random variables Spearman's rank correlation - measure of the monotonicity between two random variables. Where uncertainties between two variables are linear - little difference between linear and rank correlation When uncertainties have non-linear relationship - Answers can be remarkably different Spreadsheet-based Monte Carlo simulations use rank correlation May give a different answer than analytically derived Uncertainty and Residuals:  Uncertainty and Residuals Percentage error or standard error are a measure of residual errors Uncertainty and risk calculations Use residual errors to represent uncertainty Correlation between residuals Cost vs. Weight 0 500 1000 1500 2000 2500 3000 0 20 40 60 80 100 Weight (lbs) Cost (\$K) Deriving Correlation Coefficients:  Deriving Correlation Coefficients Sample calculation using randomly generated numbers Introduction:  Introduction Correlation must be included when calculating total variance of an estimate Sometimes correlation coefficients are “guessed” We know how to calculate correlation coefficients for pairs of Cost Estimating Relationships (CERs) Do/will these correlation coefficients apply to other models? How inaccurate would this assumption be? Compared analytically derived correlation coefficients from two spacecraft cost models Unmanned Spacecraft Cost Model Version 7 (USCM7) Small Spacecraft Cost Model Version 2000 (SSCM2000) Are correlation coefficients the same? USCM7 Sample Correlation Coefficients:  USCM7 Sample Correlation Coefficients Sample correlation coefficients for USCM7 Weight based, Mean Unbiased Percentage Error (MUPE) CERs Average correlation coefficient = 0.160 Should these coefficients be used for all spacecraft cost models? Correlation and Causality:  Correlation and Causality Natural tendency to relate causes of correlation to value of the coefficients. Correlation does not necessarily imply a causal relationship between two random variables. It is a measure of tendency of one WBS element to cost more than estimated while another WBS element is over or under estimated. Statistically determined If correlation coefficients were due to causal relationships, coefficients between two spacecraft cost models would be similar for similar WBS pairs To be demonstrated Correlation Coefficients for SSCM 2000:  Correlation Coefficients for SSCM 2000 Applied the same method of calculating correlation coefficients in SSCM as we did for USCM7 Derived correlation coefficients for SSCM Average correlation coefficient = 0.051 Need to compare to USCM-7 Similarity of Correlation Distribution:  Similarity of Correlation Distribution Both models show similar distribution of correlation coefficients Comparing Correlation Coefficients:  Comparing Correlation Coefficients Can’t directly compare correlation between USCM7 and SSCM SSCM does not have a NR/T1 breakout, USCM7 does All correlation coefficients appear to be different and random Combine Nonrecurring (NR) +First Flight Unit (T1) data for USCM to have an “apples to apples” comparison Sum NR+T1 for actual costs (Used USCM derived T1 data) Sum NR+T1 for estimates Calculate mean unbiased percentage error (MUPE) Correlate pairs of columns of MUPE data Compare models with similar estimate content (NR+T1) USCM7* and SSCM Correlation:  USCM7* and SSCM Correlation Are correlation coefficients similar? * Modified NR+T1 Comparison of Correlation Coefficients:  Comparison of Correlation Coefficients We subtracted similar correlation coefficients to find the difference between USCM7 and SSCM There are major differences We should not use one set of correlation coefficients for all models The correlation is statistically based, not causal Correlation coefficients are different Correlation Coefficient Differences:  Correlation Coefficient Differences The mean of the differences of correlation coefficients between USCM7 and SSCM are large Due to effects of combining NR & T1 from USCM7 Different correlation between totals and NR and T1 Mean = 0.378 Correlation Distribution with Modified USCM7:  Correlation Distribution with Modified USCM7 NR+T1 correlation distribution is biased and shaped differently Conclusions:  Conclusions Correlation coefficients are very different between two similar spacecraft models If similar, it would provide evidence for causality They are different, which implies the lack of causality Don’t use correlation coefficients derived for a different model Correlation coefficients are distributed similarly between two similar spacecraft models Perhaps there is a natural distribution (lognormal) We can use this to check if “guesses” at correlation coefficients mirror statistics Correlation is biased and redistributed when new estimators are used (NR+T1) Future Research:  Future Research Look for correlation in software CERs Regress equations for NR+T1 in USCM-7 Determine cause of biases and redistribution of correlation coefficients Apply correlation analysis to cost risk Develop better risk databases Segregate risk data Apply principles of determining correlation References:  References Covert, Raymond, “Correlation Coefficients in the USCM 7 Database”, 3rd Annual Joint ISPA/SCEA International Conference, Tyson's Corner, VA, June 14, 2000. Taylor, John, An Introduction to Error Analysis, University Science Books, Mill Valley, CA 1982. Nguyen, P., et al., Space and Missile Systems Center Unmanned Spacecraft Cost Model Seventh Edition, Space and Missile Systems Center, Cost Division, Los Angeles AFB, CA, August 1994. Summers, P., et al., Small Spacecraft Cost Model Version 2000, The Aerospace Corporation, El Segundo, CA, 2000. Book, Stephen A., "Cost Risk analysis: A Tutorial", in conjunction with the Risk Management Symposium Co sponsored by USAF Space and Missile Systems Center and The Aerospace Institute, Manhattan Beach, CA, 2 June 1997 Book, Stephen A, "Why Correlation Matters in Cost Estimating", 32nd Annual DoD Cost Analysis Symposium, Williamsburg, VA, 2-5 February, 1999 Garvey, Paul R, " Do Not Use Rank Correlation in Cost Risk Analysis", 32nd Annual DoD Cost Analysis Symposium, Williamsburg, VA, 2-5 February, 1999 Lurie,, Philip M. and Goldberg, Matthew S., " Simulating Correlated Random Variables", 32nd Annual DoD Cost Analysis Symposium, Williamsburg, VA, 2-5 February, 1999

09. 01. 2008
0 views

10. 01. 2008
0 views

11. 01. 2008
0 views

11. 01. 2008
0 views

11. 01. 2008
0 views

15. 01. 2008
0 views

15. 01. 2008
0 views

17. 01. 2008
0 views

21. 01. 2008
0 views

22. 01. 2008
0 views

22. 01. 2008
0 views

23. 01. 2008
0 views

04. 02. 2008
0 views

05. 02. 2008
0 views

08. 02. 2008
0 views

12. 02. 2008
0 views

09. 01. 2008
0 views

16. 01. 2008
0 views

25. 01. 2008
0 views

08. 01. 2008
0 views

28. 01. 2008
0 views

05. 02. 2008
0 views

06. 02. 2008
0 views

07. 02. 2008
0 views

20. 02. 2008
0 views

20. 02. 2008
0 views

16. 01. 2008
0 views

25. 02. 2008
0 views

14. 02. 2008
0 views

28. 02. 2008
0 views

03. 03. 2008
0 views

08. 03. 2008
0 views

12. 03. 2008
0 views

14. 03. 2008
0 views

15. 03. 2008
0 views

19. 03. 2008
0 views

20. 03. 2008
0 views

21. 03. 2008
0 views

15. 04. 2008
0 views

16. 04. 2008
0 views

17. 04. 2008
0 views

18. 04. 2008
0 views

21. 04. 2008
0 views

22. 04. 2008
0 views

28. 01. 2008
0 views

24. 04. 2008
0 views

07. 05. 2008
0 views

08. 05. 2008
0 views

30. 04. 2008
0 views

02. 05. 2008
0 views

02. 05. 2008
0 views

06. 02. 2008
0 views

21. 01. 2008
0 views

24. 01. 2008
0 views

20. 01. 2008
0 views

11. 02. 2008
0 views

23. 01. 2008
0 views

21. 02. 2008
0 views

26. 01. 2008
0 views

14. 04. 2008
0 views

12. 01. 2008
0 views

05. 02. 2008
0 views

31. 01. 2008
0 views

24. 01. 2008
0 views

31. 03. 2008
0 views

23. 01. 2008
0 views

10. 04. 2008
0 views

25. 02. 2008
0 views

08. 04. 2008
0 views