RR 06 Covert Comparison of SC Model Correlation

Information about RR 06 Covert Comparison of SC Model Correlation

Published on January 16, 2008

Author: Riccard

Source: authorstream.com

Content

Comparison of Spacecraft Cost Model Correlation Coefficients:  Comparison of Spacecraft Cost Model Correlation Coefficients Raymond P. Covert The Aerospace Corporation 15049 Conference Center Drive, CH1-410 Suite 600 Chantilly, VA 20151 E-mail: [email protected] © 2002 The Aerospace Corporation Outline:  Outline Background Calculating Estimate Uncertainty Importance of Correlation Types of Correlation Introduction Frequency Distribution of Correlation Coefficients Comparing Coefficients Between Models Conclusions Calculating Estimate Uncertainty:  Calculating Estimate Uncertainty Total cost variance, s2 s2=sT[r]s [r] = Correlation matrix (full matrix) s = Vector of standard deviations (cost space) Excel Commands SIGMA_TOT=SQRT(MMULT(MMULT(TRANSPOSE(SIGMA),RHO),SIGMA)) Correlation is Essential in calculating variance! Correlation Coefficients:  Correlation Coefficients Different types of correlation Pearson's product-moment (linear) correlation - measure of the linearity between two random variables Spearman's rank correlation - measure of the monotonicity between two random variables. Where uncertainties between two variables are linear - little difference between linear and rank correlation When uncertainties have non-linear relationship - Answers can be remarkably different Spreadsheet-based Monte Carlo simulations use rank correlation May give a different answer than analytically derived Uncertainty and Residuals:  Uncertainty and Residuals Percentage error or standard error are a measure of residual errors Uncertainty and risk calculations Use residual errors to represent uncertainty Correlation between residuals Cost vs. Weight 0 500 1000 1500 2000 2500 3000 0 20 40 60 80 100 Weight (lbs) Cost ($K) Deriving Correlation Coefficients:  Deriving Correlation Coefficients Sample calculation using randomly generated numbers Introduction:  Introduction Correlation must be included when calculating total variance of an estimate Sometimes correlation coefficients are “guessed” We know how to calculate correlation coefficients for pairs of Cost Estimating Relationships (CERs) Do/will these correlation coefficients apply to other models? How inaccurate would this assumption be? Compared analytically derived correlation coefficients from two spacecraft cost models Unmanned Spacecraft Cost Model Version 7 (USCM7) Small Spacecraft Cost Model Version 2000 (SSCM2000) Are correlation coefficients the same? USCM7 Sample Correlation Coefficients:  USCM7 Sample Correlation Coefficients Sample correlation coefficients for USCM7 Weight based, Mean Unbiased Percentage Error (MUPE) CERs Average correlation coefficient = 0.160 Should these coefficients be used for all spacecraft cost models? Correlation and Causality:  Correlation and Causality Natural tendency to relate causes of correlation to value of the coefficients. Correlation does not necessarily imply a causal relationship between two random variables. It is a measure of tendency of one WBS element to cost more than estimated while another WBS element is over or under estimated. Statistically determined If correlation coefficients were due to causal relationships, coefficients between two spacecraft cost models would be similar for similar WBS pairs To be demonstrated Correlation Coefficients for SSCM 2000:  Correlation Coefficients for SSCM 2000 Applied the same method of calculating correlation coefficients in SSCM as we did for USCM7 Derived correlation coefficients for SSCM Average correlation coefficient = 0.051 Need to compare to USCM-7 Similarity of Correlation Distribution:  Similarity of Correlation Distribution Both models show similar distribution of correlation coefficients Comparing Correlation Coefficients:  Comparing Correlation Coefficients Can’t directly compare correlation between USCM7 and SSCM SSCM does not have a NR/T1 breakout, USCM7 does All correlation coefficients appear to be different and random Combine Nonrecurring (NR) +First Flight Unit (T1) data for USCM to have an “apples to apples” comparison Sum NR+T1 for actual costs (Used USCM derived T1 data) Sum NR+T1 for estimates Calculate mean unbiased percentage error (MUPE) Correlate pairs of columns of MUPE data Compare models with similar estimate content (NR+T1) USCM7* and SSCM Correlation:  USCM7* and SSCM Correlation Are correlation coefficients similar? * Modified NR+T1 Comparison of Correlation Coefficients:  Comparison of Correlation Coefficients We subtracted similar correlation coefficients to find the difference between USCM7 and SSCM There are major differences We should not use one set of correlation coefficients for all models The correlation is statistically based, not causal Correlation coefficients are different Correlation Coefficient Differences:  Correlation Coefficient Differences The mean of the differences of correlation coefficients between USCM7 and SSCM are large Due to effects of combining NR & T1 from USCM7 Different correlation between totals and NR and T1 Mean = 0.378 Correlation Distribution with Modified USCM7:  Correlation Distribution with Modified USCM7 NR+T1 correlation distribution is biased and shaped differently Conclusions:  Conclusions Correlation coefficients are very different between two similar spacecraft models If similar, it would provide evidence for causality They are different, which implies the lack of causality Don’t use correlation coefficients derived for a different model Correlation coefficients are distributed similarly between two similar spacecraft models Perhaps there is a natural distribution (lognormal) We can use this to check if “guesses” at correlation coefficients mirror statistics Correlation is biased and redistributed when new estimators are used (NR+T1) Future Research:  Future Research Look for correlation in software CERs Regress equations for NR+T1 in USCM-7 Determine cause of biases and redistribution of correlation coefficients Apply correlation analysis to cost risk Develop better risk databases Segregate risk data Apply principles of determining correlation References:  References Covert, Raymond, “Correlation Coefficients in the USCM 7 Database”, 3rd Annual Joint ISPA/SCEA International Conference, Tyson's Corner, VA, June 14, 2000. Taylor, John, An Introduction to Error Analysis, University Science Books, Mill Valley, CA 1982. Nguyen, P., et al., Space and Missile Systems Center Unmanned Spacecraft Cost Model Seventh Edition, Space and Missile Systems Center, Cost Division, Los Angeles AFB, CA, August 1994. Summers, P., et al., Small Spacecraft Cost Model Version 2000, The Aerospace Corporation, El Segundo, CA, 2000. Book, Stephen A., "Cost Risk analysis: A Tutorial", in conjunction with the Risk Management Symposium Co sponsored by USAF Space and Missile Systems Center and The Aerospace Institute, Manhattan Beach, CA, 2 June 1997 Book, Stephen A, "Why Correlation Matters in Cost Estimating", 32nd Annual DoD Cost Analysis Symposium, Williamsburg, VA, 2-5 February, 1999 Garvey, Paul R, " Do Not Use Rank Correlation in Cost Risk Analysis", 32nd Annual DoD Cost Analysis Symposium, Williamsburg, VA, 2-5 February, 1999 Lurie,, Philip M. and Goldberg, Matthew S., " Simulating Correlated Random Variables", 32nd Annual DoD Cost Analysis Symposium, Williamsburg, VA, 2-5 February, 1999

Related presentations


Other presentations created by Riccard

NIC RoHS WEEE
09. 01. 2008
0 views

NIC RoHS WEEE

Chapter 4 The Colonies copy 2
10. 01. 2008
0 views

Chapter 4 The Colonies copy 2

PP066
11. 01. 2008
0 views

PP066

wedding brochure
11. 01. 2008
0 views

wedding brochure

HarlemRenaissance
11. 01. 2008
0 views

HarlemRenaissance

pakisnl
15. 01. 2008
0 views

pakisnl

CS4705phonetWEB
15. 01. 2008
0 views

CS4705phonetWEB

Spiritual Warfare
17. 01. 2008
0 views

Spiritual Warfare

Paper 16 QinetiQ
21. 01. 2008
0 views

Paper 16 QinetiQ

ACP EU 16h30 v5
22. 01. 2008
0 views

ACP EU 16h30 v5

l05a
22. 01. 2008
0 views

l05a

Symposium
23. 01. 2008
0 views

Symposium

Can you hear me Now 1
04. 02. 2008
0 views

Can you hear me Now 1

Lecture 2 Semiotics History
05. 02. 2008
0 views

Lecture 2 Semiotics History

newtech
08. 02. 2008
0 views

newtech

Narrativepwh
12. 02. 2008
0 views

Narrativepwh

APS LAX talk for web
09. 01. 2008
0 views

APS LAX talk for web

vv
16. 01. 2008
0 views

vv

smt exam3
25. 01. 2008
0 views

smt exam3

massac1
08. 01. 2008
0 views

massac1

McDougall Crop Protection Market
28. 01. 2008
0 views

McDougall Crop Protection Market

Aquator presentation
05. 02. 2008
0 views

Aquator presentation

feeding children
06. 02. 2008
0 views

feeding children

D31831 D21461 Beowulf
07. 02. 2008
0 views

D31831 D21461 Beowulf

Unisys Migrating to MOSS
20. 02. 2008
0 views

Unisys Migrating to MOSS

sunum
20. 02. 2008
0 views

sunum

Comet
16. 01. 2008
0 views

Comet

pps 311
14. 02. 2008
0 views

pps 311

mvwzju14
28. 02. 2008
0 views

mvwzju14

the pinkle winkle
03. 03. 2008
0 views

the pinkle winkle

DVC sacred feminine
08. 03. 2008
0 views

DVC sacred feminine

dealer
12. 03. 2008
0 views

dealer

Mixed Methods Presentation
15. 03. 2008
0 views

Mixed Methods Presentation

Get Rich Quick
19. 03. 2008
0 views

Get Rich Quick

cyber attacks
20. 03. 2008
0 views

cyber attacks

The Greek Culture
15. 04. 2008
0 views

The Greek Culture

MartinArchery
16. 04. 2008
0 views

MartinArchery

sixhats
17. 04. 2008
0 views

sixhats

natural step presentation
18. 04. 2008
0 views

natural step presentation

3 Sports marketing
21. 04. 2008
0 views

3 Sports marketing

TonyWinterbottom
22. 04. 2008
0 views

TonyWinterbottom

skrzypiec2
28. 01. 2008
0 views

skrzypiec2

I3P at UCB 2005 11 17
24. 04. 2008
0 views

I3P at UCB 2005 11 17

rosenberg
08. 05. 2008
0 views

rosenberg

05 AACSB Presentation 2 6
30. 04. 2008
0 views

05 AACSB Presentation 2 6

ChallengeSMOT0706
02. 05. 2008
0 views

ChallengeSMOT0706

Web Integration
02. 05. 2008
0 views

Web Integration

German
06. 02. 2008
0 views

German

Prager at Conference
21. 01. 2008
0 views

Prager at Conference

Relationship with sports media
24. 01. 2008
0 views

Relationship with sports media

cispres
11. 02. 2008
0 views

cispres

Habitat for Humanity
23. 01. 2008
0 views

Habitat for Humanity

01 IntroduccionACsharp
21. 02. 2008
0 views

01 IntroduccionACsharp

eckhart presentation
26. 01. 2008
0 views

eckhart presentation

corporatesponsors
14. 04. 2008
0 views

corporatesponsors

13 Presentacion 2005
12. 01. 2008
0 views

13 Presentacion 2005

s Wedding
05. 02. 2008
0 views

s Wedding

eCommerce Artisans
31. 01. 2008
0 views

eCommerce Artisans

hydrogeneconomychall enges
24. 01. 2008
0 views

hydrogeneconomychall enges

BrainDrainjan11
31. 03. 2008
0 views

BrainDrainjan11

clemmons injuctions
23. 01. 2008
0 views

clemmons injuctions

Panasonic
10. 04. 2008
0 views

Panasonic

20050615report 2
25. 02. 2008
0 views

20050615report 2

Graying of the Northstate
08. 04. 2008
0 views

Graying of the Northstate