Nuisance parameter, systematic uncertain

Information about Nuisance parameter, systematic uncertain

Published on June 15, 2007

Author: funnyside

Source: authorstream.com

Content

Nuisance parameters and systematic uncertainties:  Nuisance parameters and systematic uncertainties Glen Cowan Glen Cowan Royal Holloway, University of London [email protected] www.pp.rhul.ac.uk/~cowan IoP Half Day Meeting on Statistics in High Energy Physics University of Manchester 16 November, 2005 Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Vague outline:  Vague outline Glen Cowan I. Nuisance parameters and systematic uncertainty II. Parameter measurement Frequentist Bayesian III. Estimating intervals (setting limits) Frequentist Bayesian IV. Conclusions Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Statistical vs. systematic errors :  Statistical vs. systematic errors Glen Cowan Statistical errors: How much would the result fluctuate upon repetition of the measurement? Implies some set of assumptions to define probability of outcome of the measurement. Systematic errors: What is the uncertainty in my result due to uncertainty in my assumptions, e.g., model (theoretical) uncertainty; modeling of measurement apparatus. The sources of error do not vary upon repetition of the measurement. Often result from uncertain value of, e.g., calibration constants, efficiencies, etc. Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Nuisance parameters :  Nuisance parameters Glen Cowan Suppose the outcome of the experiment is some set of data values x (here shorthand for e.g. x1, ..., xn). We want to determine a parameter q, (could be a vector of parameters q1, ..., q n). The probability law for the data x depends on q : L(x| q) (the likelihood function) E.g. maximize L to find estimator Now suppose, however, that the vector of parameters: contains some that are of interest, and others that are not of interest: Symbolically: The are called nuisance parameters. Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Example #1: fitting a straight line:  Example #1: fitting a straight line Glen Cowan Data: Model: measured yi independent, Gaussian: assume xi and si known. Goal: estimate q0 (don’t care about q1). Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Case #1: q1 known a priori:  Case #1: q1 known a priori Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester For Gaussian yi, ML same as LS Minimize c2 → estimator Come up one unit from to find Slide7:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Correlation between causes errors to increase. Standard deviations from tangent lines to contour Case #2: both q0 and q1 unknown Slide8:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester The information on q1 improves accuracy of Case #3: we have a measurement t1 of q1 Slide9:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester The ‘tangent plane’ method is a special case of using the profile likelihood: The profile likelihood is found by maximizing L (q0, q1) for each q0. Equivalently use The interval obtained from is the same as what is obtained from the tangents to Well known in HEP as the ‘MINOS’ method in MINUIT. Profile likelihood is one of several ‘pseudo-likelihoods’ used in problems with nuisance parameters. See e.g. talk by Rolke at PHYSTAT05. Slide10:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester The Bayesian approach In Bayesian statistics we can associate a probability with a hypothesis, e.g., a parameter value q. Interpret probability of q as ‘degree of belief’ (subjective). Need to start with ‘prior pdf’ p(q), this reflects degree of belief about q before doing the experiment. Our experiment has data x, → likelihood function L(x|q). Bayes’ theorem tells how our beliefs should be updated in light of the data x: Posterior pdf p(q|x) contains all our knowledge about q. Slide11:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Case #4: Bayesian method We need to associate prior probabilities with q0 and q1, e.g., Putting this into Bayes’ theorem gives: posterior Q likelihood  prior ← based on previous measurement reflects ‘prior ignorance’, in any case much broader than Slide12:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Bayesian method (continued) Ability to marginalize over nuisance parameters is an important feature of Bayesian statistics. We then integrate (marginalize) p(q0, q1 | x) to find p(q0 | x): In this example we can do the integral (rare). We find Slide13:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Digression: marginalization with MCMC Bayesian computations involve integrals like often high dimensionality and impossible in closed form, also impossible with ‘normal’ acceptance-rejection Monte Carlo. Markov Chain Monte Carlo (MCMC) has revolutionized Bayesian computation. Google for ‘MCMC’, ‘Metropolis’, ‘Bayesian computation’, ... MCMC generates correlated sequence of random numbers: cannot use for many applications, e.g., detector MC; effective stat. error greater than √n . Basic idea: sample multidimensional look, e.g., only at distribution of parameters of interest. Slide14:  Although numerical values of answer here same as in frequentist case, interpretation is different (sometimes unimportant?) Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Example: posterior pdf from MCMC Sample the posterior pdf from previous example with MCMC: Summarize pdf of parameter of interest with, e.g., mean, median, standard deviation, etc. Slide15:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Case #5: Bayesian method with vague prior Suppose we don’t have a previous measurement of q1 but rather some vague information, e.g., a theorist tells us: q1 ≥ 0 (essentially certain); q1 should have order of magnitude less than 0.1 ‘or so’. Under pressure, the theorist sketches the following prior: From this we will obtain posterior probabilities for q0 (next slide). We do not need to get the theorist to ‘commit’ to this prior; final result has ‘if-then’ character. Slide16:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Sensitivity to prior Vary () to explore how extreme your prior beliefs would have to be to justify various conclusions (sensitivity analysis). Try exponential with different mean values... Try different functional forms... Slide17:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Example #2: Poisson data with background Count n events, e.g., in fixed time or integrated luminosity. s = expected number of signal events b = expected number of background events n ~ Poisson(s+b): Sometimes b known, other times it is in some way uncertain. Goal: measure or place limits on s, taking into consideration the uncertainty in b. Widely discussed in HEP community, see e.g. proceedings of PHYSTAT meetings, Durham, Fermilab, CERN workshops... Slide18:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Setting limits Frequentist intervals (limits) for a parameter s can be found by defining a test of the hypothesized value s (do this for all s): Specify values of the data n that are ‘disfavoured’ by s (critical region) such that P(n in critical region) ≤ g for a prespecified g, e.g., 0.05 or 0.1. (Because of discrete data, need inequality here.) If n is observed in the critical region, reject the value s. Now invert the test to define a confidence interval as: set of s values that would not be rejected in a test of size g (confidence level is 1 - g ). The interval will cover the true value of s with probability ≥ 1 - g. Equivalent to Neyman confidence belt construction. Slide19:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Setting limits: ‘classical method’ E.g. for upper limit on s, take critical region to be low values of n, limit sup at confidence level 1 - b thus found from Similarly for lower limit at confidence level 1 - a, Sometimes choose a = b = g /2 → central confidence interval. Slide20:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Likelihood ratio limits (Feldman-Cousins) Define likelihood ratio for hypothesized parameter value s: Here is the ML estimator, note Critical region defined by low values of likelihood ratio. Resulting intervals can be one- or two-sided (depending on n). (Re)discovered for HEP by Feldman and Cousins, Phys. Rev. D 57 (1998) 3873. Slide21:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Nuisance parameters and limits In general we don’t know the background b perfectly. Suppose we have a measurement of b, e.g., bmeas ~ N (b, b) So the data are really: n events and the value bmeas. In principle the confidence interval recipe can be generalized to two measurements and two parameters. Difficult and rarely attempted, but see e.g. talk by G. Punzi at PHYSTAT05. G. Punzi, PHYSTAT05 Slide22:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Bayesian limits with uncertainty on b Uncertainty on b goes into the prior, e.g., Put this into Bayes’ theorem, Marginalize over b, then use p(s|n) to find intervals for s with any desired probability content. Controversial part here is prior for signal s(s) (treatment of nuisance parameters is easy). Slide23:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Cousins-Highland method Regard b as ‘random’, characterized by pdf (b). Makes sense in Bayesian approach, but in frequentist model b is constant (although unknown). A measurement bmeas is random but this is not the mean number of background events, rather, b is. Compute anyway This would be the probability for n if Nature were to generate a new value of b upon repetition of the experiment with b(b). Now e.g. use this P(n;s) in the classical recipe for upper limit at CL = 1 - b: Result has hybrid Bayesian/frequentist character. Slide24:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester ‘Integrated likelihoods’ Consider again signal s and background b, suppose we have uncertainty in b characterized by a prior pdf b(b). Define integrated likelihood as also called modified profile likelihood, in any case not a real likelihood. Now use this to construct likelihood ratio test and invert to obtain confidence intervals. Feldman-Cousins andamp; Cousins-Highland (FHC2), see e.g. J. Conrad et al., Phys. Rev. D67 (2003) 012002 and Conrad/Tegenfeldt PHYSTAT05 talk. Calculators available (Conrad, Tegenfeldt, Barlow). Slide25:  Glen Cowan Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester Interval from inverting profile LR test Suppose we have a measurement bmeas of b. Build the likelihood ratio test with profile likelihood: and use this to construct confidence intervals. See PHYSTAT05 talks by Cranmer, Feldman, Cousins, Reid. Wrapping up:  Wrapping up Glen Cowan I’ve shown a few ways of treating nuisance parameters in two examples (fitting line, Poisson mean with background). No guarantee this will bear any relation to the problem you need to solve... At recent PHYSTAT meetings the statisticians have encouraged physicists to: learn Bayesian methods, don’t get too fixated on coverage, try to see statistics as a ‘way of thinking’ rather than a collection of recipes. I tend to prefer the Bayesian methods for systematics but still a very open area of discussion. Statistics in HEP, IoP Half Day Meeting, 16 November 2005, Manchester

Related presentations


Other presentations created by funnyside

HCM Luzern
18. 06. 2007
0 views

HCM Luzern

lecture12
12. 10. 2007
0 views

lecture12

basketball
01. 08. 2007
0 views

basketball

Atacama Large Millimeter Array
15. 06. 2007
0 views

Atacama Large Millimeter Array

gen mtgmar07 slides
18. 06. 2007
0 views

gen mtgmar07 slides

THAKKAR ANJALI artificial blood
16. 10. 2007
0 views

THAKKAR ANJALI artificial blood

20030815 RepICFA HN
27. 09. 2007
0 views

20030815 RepICFA HN

Owen
03. 10. 2007
0 views

Owen

14BBlackHoles
07. 10. 2007
0 views

14BBlackHoles

Noda06
09. 10. 2007
0 views

Noda06

reptile
10. 10. 2007
0 views

reptile

leskov
15. 10. 2007
0 views

leskov

461
17. 10. 2007
0 views

461

Europe Presentation
19. 10. 2007
0 views

Europe Presentation

RadOnc
22. 10. 2007
0 views

RadOnc

schnittsysteme
23. 10. 2007
0 views

schnittsysteme

ch1 10 part2
23. 10. 2007
0 views

ch1 10 part2

firma end Presentation
19. 10. 2007
0 views

firma end Presentation

sta291 5c
29. 10. 2007
0 views

sta291 5c

ACM India Studies 2005
26. 11. 2007
0 views

ACM India Studies 2005

Mirabel
01. 12. 2007
0 views

Mirabel

ghana
10. 12. 2007
0 views

ghana

DanishCitationEventJB final
15. 10. 2007
0 views

DanishCitationEventJB final

aleynikov
26. 10. 2007
0 views

aleynikov

ShortageChicago Seago 2
29. 10. 2007
0 views

ShortageChicago Seago 2

Daylight Saving Time
07. 11. 2007
0 views

Daylight Saving Time

unity
21. 08. 2007
0 views

unity

Fad Diets
21. 08. 2007
0 views

Fad Diets

NACADA Combined Workshop
21. 08. 2007
0 views

NACADA Combined Workshop

TeraGrid SC06 1
22. 10. 2007
0 views

TeraGrid SC06 1

plane parallel 2 1
24. 10. 2007
0 views

plane parallel 2 1

cronologia storia contemporanea
13. 11. 2007
0 views

cronologia storia contemporanea

IDitarod
16. 11. 2007
0 views

IDitarod

Axiome Von Neumann Morgenstern
19. 11. 2007
0 views

Axiome Von Neumann Morgenstern

Dog bite sepsis
19. 11. 2007
0 views

Dog bite sepsis

Girish
21. 11. 2007
0 views

Girish

EcoValidity
17. 12. 2007
0 views

EcoValidity

matthew
29. 12. 2007
0 views

matthew

Cyatheaceae Classification
02. 01. 2008
0 views

Cyatheaceae Classification

RFF Shinnecock
03. 01. 2008
0 views

RFF Shinnecock

AustralianPoetry
03. 01. 2008
0 views

AustralianPoetry

dbl dutch
22. 11. 2007
0 views

dbl dutch

Lecture10 overheads
21. 08. 2007
0 views

Lecture10 overheads

Deborah Russell Presentation
14. 11. 2007
0 views

Deborah Russell Presentation

AAPT v3
01. 08. 2007
0 views

AAPT v3

EScannapieco
15. 11. 2007
0 views

EScannapieco

Successful  Case  Studies
29. 09. 2007
0 views

Successful Case Studies

Interv Hajji
23. 10. 2007
0 views

Interv Hajji

khelif edson rep
09. 10. 2007
0 views

khelif edson rep

aadcrocodile
11. 10. 2007
0 views

aadcrocodile

matricole
16. 10. 2007
0 views

matricole

the three little pigs
14. 02. 2008
0 views

the three little pigs

Exon HKSB
30. 10. 2007
0 views

Exon HKSB

June06SPCCUpdateJVan voorhis
07. 11. 2007
0 views

June06SPCCUpdateJVan voorhis

cultural impact hispanics tn
24. 02. 2008
0 views

cultural impact hispanics tn

SE 05 Punctuation I
29. 02. 2008
0 views

SE 05 Punctuation I

GaryCorbett e
04. 03. 2008
0 views

GaryCorbett e

Radiation Hazards
06. 03. 2008
0 views

Radiation Hazards

Portoroz
04. 10. 2007
0 views

Portoroz

jdean c10
25. 03. 2008
0 views

jdean c10

SoS
25. 03. 2008
0 views

SoS

MVestergaard SantaFe06
29. 11. 2007
0 views

MVestergaard SantaFe06

PowerPointTour
02. 11. 2007
0 views

PowerPointTour

tal daly
10. 04. 2008
0 views

tal daly

matz011
16. 04. 2008
0 views

matz011

installaware faq
17. 04. 2008
0 views

installaware faq

ATMSELong
18. 04. 2008
0 views

ATMSELong

36329938
22. 04. 2008
0 views

36329938

ITCan
10. 03. 2008
0 views

ITCan

kejian4
11. 10. 2007
0 views

kejian4

061213 dialog north korea petrov
11. 10. 2007
0 views

061213 dialog north korea petrov

Session6 Adm Wilcocks
07. 11. 2007
0 views

Session6 Adm Wilcocks

high technology surgery
30. 04. 2008
0 views

high technology surgery

Robot Convoy final presentation
07. 01. 2008
0 views

Robot Convoy final presentation

harry truman
18. 06. 2007
0 views

harry truman

goldrush
18. 06. 2007
0 views

goldrush

GMK
18. 06. 2007
0 views

GMK

gatech
18. 06. 2007
0 views

gatech

fusion be
18. 06. 2007
0 views

fusion be

Fun Facts updated 3 05
18. 06. 2007
0 views

Fun Facts updated 3 05

Geomatic AAU 07032007 forkortet
18. 06. 2007
0 views

Geomatic AAU 07032007 forkortet

General Spring Intro
18. 06. 2007
0 views

General Spring Intro

intervento Ambrosi Legge125
18. 06. 2007
0 views

intervento Ambrosi Legge125

insostegno
18. 06. 2007
0 views

insostegno

Inhalts angabe
18. 06. 2007
0 views

Inhalts angabe

Implant Simulations
18. 06. 2007
0 views

Implant Simulations

implantate berlin
18. 06. 2007
0 views

implantate berlin

Imber stud
18. 06. 2007
0 views

Imber stud

hsm bend2
18. 06. 2007
0 views

hsm bend2

history mat storia dei diritti
18. 06. 2007
0 views

history mat storia dei diritti

headquarters structure
18. 06. 2007
0 views

headquarters structure

Department Of Conservation
23. 10. 2007
0 views

Department Of Conservation

Hippo
18. 07. 2008
0 views

Hippo

pa3 disare ppt
28. 09. 2007
0 views

pa3 disare ppt

Sudan Polio Update 03 05
21. 10. 2007
0 views

Sudan Polio Update 03 05

bradford ancer
21. 08. 2007
0 views

bradford ancer

Delaunay Triangulations
15. 06. 2007
0 views

Delaunay Triangulations

Bayesian analysis and problems
15. 06. 2007
0 views

Bayesian analysis and problems

Biofuels - Think Big
15. 06. 2007
0 views

Biofuels - Think Big

AVO  - Virtual Observatory
15. 06. 2007
0 views

AVO - Virtual Observatory

PHOBOS Collaboration
15. 06. 2007
0 views

PHOBOS Collaboration

gennaipm
18. 06. 2007
0 views

gennaipm

sigcse2003
07. 12. 2007
0 views

sigcse2003

Introductionto Kyrgyzstan
26. 11. 2007
0 views

Introductionto Kyrgyzstan

HIT SAN
18. 06. 2007
0 views

HIT SAN

Metzenbaum Shelley
23. 10. 2007
0 views

Metzenbaum Shelley

ppt CHAP018
24. 02. 2008
0 views

ppt CHAP018

lesson3 warm up
07. 04. 2008
0 views

lesson3 warm up

shelving with lcc
19. 02. 2008
0 views

shelving with lcc

25ConnectionsV5 p2
18. 10. 2007
0 views

25ConnectionsV5 p2

2506043
15. 11. 2007
0 views

2506043

TravelChapter7 3
27. 03. 2008
0 views

TravelChapter7 3

french rfp
18. 06. 2007
0 views

french rfp

BEFxii Coronado
09. 04. 2008
0 views

BEFxii Coronado

spectrumastrotalk
28. 11. 2007
0 views

spectrumastrotalk

metadata frames
05. 10. 2007
0 views

metadata frames

Dancefinal
27. 11. 2007
0 views

Dancefinal