evaluating quant research

Information about evaluating quant research

Published on August 20, 2007

Author: Roxie

Source: authorstream.com

Content

Slide1:  Reviewer’s Secrets Revealed! James R. Van Scotter, Ph.D., ISDS Department, Louisiana State University About This Presentation:  About This Presentation Is it about learning how to review papers? About This Presentation:  About This Presentation ..or, is it about how reviewers evaluate your papers? About This Presentation:  About This Presentation …or, both! Basic Questions for Reviewers:  Should this paper be published? (Why or why not?) Does it contribute to scientific knowledge? (How?) Can the paper be strengthened? Basic Questions for Reviewers Criteria:  Schwab’s (1985) Criteria 1. Technical merit Internal validity, construct validity, causal issues. 2. Craftsmanship Organization, clarity 3. Significance or importance Tests a reasonable theory in non-trivial way. Improves an existing body of empirical research. Has implications for an important policy issue. Criteria Source: Schwab, D. (1985). Reviewing empirically-based manuscripts: Perspectives on Process. In L. Cummings andamp; R. Frost (Eds.). Publishing in the organizational sciences. Homewood, IL: Richard D. Irwin, 171-181. Criteria:  Why I Recommended Rejecting Your Manuscript… No theory Theory should explain what the variables mean, how they are related and why they are expected to be related to one another. Concepts and operationalization not aligned Variables do not reflect the concepts or the logical links have not been developed. Criteria Source: Daft, R. (1985). Why I recommended rejecting your manuscript and what you can do about it. In L. Cummings andamp; R. Frost (Eds.). Publishing in the organizational sciences. Homewood, IL: Richard D. Irwin, 193-209 Criteria:  Why I Recommended Rejecting Your Manuscript… 3. Insufficient definition - Theory Concepts are not defined or justified. It is not clear why variables were included. Some were included simply because they had appeared in other studies. 4. Insufficient rationale – Design Lack of explanation for study procedures, description of sample, etc. Nothing is obvious or assumed. Criteria Source: Daft, R. (1985). Why I recommended rejecting your manuscript and what you can do about it. In L. Cummings andamp; R. Frost (Eds.). Publishing in the organizational sciences. Homewood, IL: Richard D. Irwin, 193-209 Criteria:  Why I Recommended Rejecting Your Manuscript… 5. Insufficient research design Usually fatal. Using survey questions to study subtle processes. Using undergrad sample for strategy research (CEO, CIO level). 6. Lack of organization and flow No single story line. Mismatch between tests and hypotheses. 7. Amateurish style and tone Overly negative approach to previous literature. Exaggerated sense of importance. Logical gaps in writing. Criteria Source: Daft, R. (1985). Why I recommended rejecting your manuscript and what you can do about it. In L. Cummings andamp; R. Frost (Eds.). Publishing in the organizational sciences. Homewood, IL: Richard D. Irwin, 193-209 Criteria:  Leading Reasons for Rejection by JIS Lack of theory Weakness in Design Poor Motivation for the Study Other Reasons for Rejection Lack of Fit with Journal Insufficient Development Insufficient Contribution Methodological Problems Criteria Source: Murthy, U. andamp; Wiggins, C., (2002). Why manuscripts are rejected: An analysis of JIS rejections. Journal of Information Systems, 16, 1, 41-48. Review Process:  Peer review is a complex social act. All referees are not experts in the areas they review. Recommendations are based on the editor’s or reviewer’s personal preferences. 3. Reviewers’ recommendations may conflict. 4. When recommendations become too detailed, reviewers become ghost writers. 5. Nobel prize winning research has been rejected. Review Process Source: Bedeian, A. (2004). Peer Review and the social construction of knowledge In the management discipline. Academy of Management Learning and Education, 3, 2, 198-216. Review Process:  Survey of Reviewer Practices Typical reviewer spends 5.4 hours per paper. They reviewed 1.5 papers per month. Two dominant roles: Critic and Coach. Review Process Source: Jauch, L. andamp; Wall, J. (1989). What do they do when they get your manuscript: A survey of Academy of Management Reviewer Practices. Academy of Management Journal, 32, 1, 157-173. Process:  Developmental Reviewing More than just avoiding a negative tone. Two main ingredients: Helping the author(s) understand what is wrong. Helping the author(s) figure out how to fix it. Process Source: Feldman, D. (2004). Editorial: Being a developmental reviewer: Easier said than done. Journal of Management, 30, 2, 161-164. Process:  Developmental Reviewing 3. Procedure: Start with overall evaluation. Describe aspects of the paper that are strong/weak. Identify major and minor issues. Suggest solutions for methods problems. Suggest areas that should be cut or expanded. Process Source: Feldman, D. (2004). Editorial: Being a developmental reviewer: Easier said than done. Journal of Management, 30, 2, 161-164. Process:  Developmental Reviewing 4. Specific recommendations. No sample is perfect. Help the author(s) make the best use of the data. b. Give examples of improvements (e.g., clearer hypotheses) you would like to see. If appropriate, provide references for papers with an alternative perspective. Process Source: Feldman, D. (2004). Editorial: Being a developmental reviewer: Easier said than done. Journal of Management, 30, 2, 161-164. What to Do:  What to Do What to Do:  What to Do 'Science depends on qualitative, common-sense knowing … (Campbell, 1979, p.50)' Campbell, D. T. (1979). Degrees of freedom and the case study, in T. D. Cook, andamp; C. S. Reichardt (Eds.), Qualitative and quantitative methods in evaluation research, Beverly Hills, CA: Sage Publications, pp.49-67. Influential Articles:  Analysis of Influential Articles 1. Obvious contribution to knowledge 2. Results of major theoretical significance 3. Useful new theory/theoretical framework 4. Generates new research 5. Provides new and exciting ideas 6. Integrates several areas 7. Integrates existing theories into a simpler framework Influential Articles Source: Sternberg, R. andamp; Gordeeva, T. (1996). The anatomy of impact: What Makes an article influential. Psychological Science, 7, 2, 69-75. Slide19:  A research paper should be like a present. The general concept should be appealing, the contents should be valuable, and everything necessary should be wrapped up in the package. Six General Goals of Science:  Six General Goals of Science Organize andamp; categorize things (typologies) Explain past events Predict future events Control future events Provide a sense of understanding Generalize results Portions Adapted from: Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company. How does science accomplish its goals?:  How does science accomplish its goals? Formulating hypotheses andamp; descriptive statements that: Are derived from theories Are falsifiable Can be tested empirically Portions Adapted from: Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company. How does science accomplish its goals?:  2. Developing models that summarize important attributes/relationships Portions Adapted from: Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company. How does science accomplish its goals? How does science accomplish its goals?:  3. Collecting data Manipulating relevant variables Measuring relevant variables Controlling and/or measuring nuisance variables Portions Adapted from: Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company. How does science accomplish its goals? How does science accomplish its goals?:  4. Using empirical data to test the hypotheses Portions Adapted from: Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company. How does science accomplish its goals? How does science accomplish its goals?:  5. Generalizing research results to a larger context Portions Adapted from: Reynolds, P. D. (1971). A primer in theory construction. Indianapolis, The Bobbs-Merrill Company. How does science accomplish its goals? General Principles:  General Principles General Principles:  General Principles Research as story telling and good writing. 2. Favor simple explanations (parsimony). 3. Use models to summarize important relationships. Portions adapted from: Ableson, R. P. (1995). Statistics as principled argument. Hillsdale, NJ: Lawrence Erlbaum Publishers. General Principles:  General Principles Collecting data to test models/hypotheses. 5. Using comparison as the evaluation framework. 6. Ruling out alternative explanations. Portions adapted from: Ableson, R. P. (1995). Statistics as principled argument. Hillsdale, NJ: Lawrence Erlbaum Publishers. Story Telling:  Story Telling From general to specific From specific to general Hour glass shape Story Telling:  Story Telling Introduction Literature Review Methods Results Discussion Conclusion andamp; Recommendations Story Telling:  1. Why is the topic important and interesting? Theoretical andamp; practical value Appropriateness for journal Timeliness Novelty Story Telling Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Story Telling:  2. Literature review Link to important literature Logic for hypotheses andamp; propositions Context for study (samples/methods) Use literature to develop arguments Thorough exposition Accurate interpretation (including nuances) c. Lead to a conclusion about what is needed d. Establish incremental value of the paper Story Telling Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Story Telling:  3. Conceptual development Adequacy andamp; scope of research hypotheses Clarity and logical coherence (one main story line) Explains how things work without leaps of faith or logical gaps 4. Have a point (not just a point of view) Story Telling Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Parsimony:  Parsimony Simple explanations are better. Simple explanations are more likely to be testable. Parsimony:  Parsimony Parsimony is related to goals of scientific research Simplifying complex issues Separating important from unimportant factors Generalizing across time, place, samples Providing a benchmark Slide36:  Theoretical statement goes here Theoretical statements should be concise enough to fit on a bumper sticker. …and simple enough that your grandmother could understand them. Models:  Models Simplify communication of complex relationships Consistent with idea that measures of the same variable are interchangeable Encourage thinking about alternative explanations Models:  Models 4. Testing process must be guided by theory, not the modeling method. 5. All 6 models above have the same fit (Bollen, 1989, p.70) Methods:  Methods Sample design 1. Representativeness andamp; generalizability Appropriateness for research question Demographics for subjects Research setting andamp; context 2. Suitability for tests andamp; inference (statistical assumptions) Sampling method (random, purposive, convenience) Comparison with population, or samples in related studies Sufficiency (sample size, response rate, attrition) Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Methods:  Methods Measurement Justify constructs Purpose Theory andamp; Prior research Operationalization of constructs Use well known instruments when possible Use archival data when appropriate Use multiple sources of measures when possible Avoid single item measures Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Methods:  Methods Measurement 3. Distinguish between measures of perceptions, intentions versus behaviors and outcomes. 4. Avoid criterion contamination or assess andamp; report it. 5. Use appropriate level of analysis; avoid ecological fallacy. Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Methods:  Methods Measurement 6. Provide appropriate reliability evidence. Internal consistency Inter-rater Test-retest, Alternate forms Average Variance Extracted 7. Provide appropriate validity evidence. Convergent andamp; discriminant Construct Content Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Methods:  Methods Design 1. Use pilot studies Test manipulations/treatments and instruments Test sampling methods and procedures 2. Use standard administration methods to avoid biasing results 3. Use ethical procedures; comply with rules for treatment of human subjects Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Methods:  Methods Design (cross-sectional studies) 4. Use designs appropriate to the hypotheses Tests of alternative models/explanations Differential predictions NOT the general null hypothesis 5. Control for statistical and sampling artifacts Restriction in range Common method bias Uses appropriate control variables Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Methods:  Methods Design (experimental/quasi-experimental) 6. Use random assignment procedures; provide information on comparability of groups. 7. Use adequate manipulations/perform checks. 8. Avoid threats to validity. Internal andamp; external threats Experimenter bias Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Framework for Evaluation:  Framework for Evaluation Empirical research tests relationships between observed variables Framework for Evaluation:  Framework for Evaluation Theories focus on relationships between Latent (unobservable) variables Framework for Evaluation:  Framework for Evaluation Latent Independent Variable Latent Dependent Variable Observed Independent Variable Observed Dependent Variable What we want to know What we can test Construct validity Construct validity Statistical Conclusion Validity Statistical Conclusion Validity Construct validity Methods:  Methods Comparative Evaluation 1. Tests of different groups Natural categories Treatment/control groups; multiple treatment groups 2. Tests of alternative hypotheses Competing explanations Nested models Goal is to find evidence that contradicts one explanation and supports another Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Methods:  Methods Data Analysis Use appropriate statistics Demonstrate awareness of major assumptions Conduct supplemental analyses suggested by the study 2. Avoid capitalization on chance Examine outliers as needed Control experiment-wise error rate 3. Report effect sizes, confidence intervals, significance levels Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Ruling Out Alternatives:  Ruling Out Alternatives Some sources of errors Design Sample Instruments Experimenter andamp; situational errors Demand artifacts Results:  Results Summarize results briefly Link findings to hypotheses and interpret them Relate findings to other studies Derive specific theoretical and practical implications Provide suggestions for future research Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Story Telling:  Story Telling Writing quality, simple sentences, proper grammar, active voice Logical organization andamp; sense of direction 3. Succinct and shows attention to detail 4. Tell a consistent, integrated story Portions adapted from: Campion, M. (1993). Article Review Checklist: A Criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 1-14. Story Telling:  Why should this paper NOT be published? Does the theoretical development make sense? Do the authors explain why AB? Are there any logical gaps or leaps of faith? Story Telling Story Telling:  Why should this paper NOT be published? Do the authors test what they said they would? Are the data and methods appropriate? What are the theoretical implications? What does this paper add to the literature? Story Telling Additional:  Journals with 75% or more articles by IS Authors Journal Journal Pct Ranking MIS Quarterly .96 A Information Systems Research .95 A Journal of Management Information Systems .92 A Communications of the AIS .91 B The Data Base for Advances in Info Systems .84 A International Journal of Electronic Commerce .80 A Journal of the AIS .80 B Journal of Global Information Management .76 ? Additional Source: Chua, C., Cao, L., Cousins, K., andamp; Straub, D. (2002). Measuring researcher production in information systems. Journal of the AIS, 3, 145-215. Publish or Perish:  The 7 P’s Proper Prior Planning Prevents Poor Publication Performance Publish or Perish Source: Unknown Additional References:  Additional References Bhattacharjee, S., Tung, Y., andamp; Pathak, B. (2004). Author experiences with the IS journal review process. Communications of the AIS, 13, 629-653. Mingers, J. (2002). The long and winding road: Getting papers published In top journals. Communications of the AIS, 8, 330-339. Slide59:  The End

Related presentations


Other presentations created by Roxie

Apricot 2004 Sinkholes
07. 10. 2007
0 views

Apricot 2004 Sinkholes

Mattel
02. 11. 2007
0 views

Mattel

Option
17. 04. 2008
0 views

Option

Value of Nonpartisan
14. 04. 2008
0 views

Value of Nonpartisan

stifelcapital
13. 04. 2008
0 views

stifelcapital

m lecture 1 2007
10. 04. 2008
0 views

m lecture 1 2007

renewable report summary
09. 04. 2008
0 views

renewable report summary

Costa Rica
07. 04. 2008
0 views

Costa Rica

pptrick
30. 03. 2008
0 views

pptrick

060516202103
28. 03. 2008
0 views

060516202103

field to flour
04. 10. 2007
0 views

field to flour

wolff
27. 11. 2007
0 views

wolff

danielle
11. 12. 2007
0 views

danielle

WP6 CurriculumDevelopment
01. 11. 2007
0 views

WP6 CurriculumDevelopment

paris 23jan
02. 11. 2007
0 views

paris 23jan

why b entrepreneur
07. 11. 2007
0 views

why b entrepreneur

Miss Earth 2007
13. 11. 2007
0 views

Miss Earth 2007

Lecture15 Discourse
20. 08. 2007
0 views

Lecture15 Discourse

oasis case study
20. 08. 2007
0 views

oasis case study

4121
20. 08. 2007
0 views

4121

mediaplan
20. 08. 2007
0 views

mediaplan

soren mateu kast prop13
20. 08. 2007
0 views

soren mateu kast prop13

The Given New Principle
14. 12. 2007
0 views

The Given New Principle

FadorFact
20. 08. 2007
0 views

FadorFact

Socia lContent
11. 08. 2007
0 views

Socia lContent

Staff Wellness
11. 08. 2007
0 views

Staff Wellness

SAR presentation 2005 2006
11. 08. 2007
0 views

SAR presentation 2005 2006

Typhoid 2004b
11. 08. 2007
0 views

Typhoid 2004b

Presentation Al Jisr
23. 10. 2007
0 views

Presentation Al Jisr

CL 07 lec6 2 15
15. 11. 2007
0 views

CL 07 lec6 2 15

large animal power point
03. 10. 2007
0 views

large animal power point

patents
03. 10. 2007
0 views

patents

meta analysis 09 2006
20. 08. 2007
0 views

meta analysis 09 2006

li2 2007 gender
09. 07. 2007
0 views

li2 2007 gender

Picture Chinese in Your Future
12. 10. 2007
0 views

Picture Chinese in Your Future

socratesswissranking 28 11 06
18. 10. 2007
0 views

socratesswissranking 28 11 06

Nonverbal
09. 07. 2007
0 views

Nonverbal

MJ
09. 07. 2007
0 views

MJ

Milk Fortification
09. 07. 2007
0 views

Milk Fortification

MakeUp
09. 07. 2007
0 views

MakeUp

allexp 052404
05. 10. 2007
0 views

allexp 052404

lamm1
01. 01. 2008
0 views

lamm1

schindel
11. 08. 2007
0 views

schindel

BuildingStrategyMaps v5
24. 02. 2008
0 views

BuildingStrategyMaps v5

03n 0312 ts00002 vol1
29. 02. 2008
0 views

03n 0312 ts00002 vol1

OMA TP 2003 0354
20. 08. 2007
0 views

OMA TP 2003 0354

Digital Reference Shelf
22. 11. 2007
0 views

Digital Reference Shelf

anspaugh
23. 11. 2007
0 views

anspaugh

Neolithic Europe
26. 03. 2008
0 views

Neolithic Europe

KAT PRESENTATION
21. 11. 2007
0 views

KAT PRESENTATION

Prezentacja o rekrutacji
18. 03. 2008
0 views

Prezentacja o rekrutacji

Met en zonder MakeUp Antwoord
09. 07. 2007
0 views

Met en zonder MakeUp Antwoord

media body image
09. 07. 2007
0 views

media body image

Diario Intimo de los gatos 1913
19. 06. 2007
0 views

Diario Intimo de los gatos 1913

globus
19. 06. 2007
0 views

globus

Article VIII
21. 09. 2007
0 views

Article VIII

Montreal Summer School2005
31. 12. 2007
0 views

Montreal Summer School2005

El Dulce Aroma Del Cafe 2118
19. 06. 2007
0 views

El Dulce Aroma Del Cafe 2118

El Caballo Y Cerdo 2117
19. 06. 2007
0 views

El Caballo Y Cerdo 2117

El pajaro de la paz 1853
19. 06. 2007
0 views

El pajaro de la paz 1853

El genio 2046
19. 06. 2007
0 views

El genio 2046

El buen uso de los espejos
19. 06. 2007
0 views

El buen uso de los espejos

El ladrillazo 2092
19. 06. 2007
0 views

El ladrillazo 2092

2005 JRC Workshop Smrz
06. 03. 2008
0 views

2005 JRC Workshop Smrz

El camino de las estrellas 2075
19. 06. 2007
0 views

El camino de las estrellas 2075

El sustento de la pareja 1935
19. 06. 2007
0 views

El sustento de la pareja 1935

CJ Addiction
21. 09. 2007
0 views

CJ Addiction

BLUE BEAUTY
16. 06. 2007
0 views

BLUE BEAUTY

Biler Kvinder
16. 06. 2007
0 views

Biler Kvinder

2CF Japanese Fishing Story
16. 06. 2007
0 views

2CF Japanese Fishing Story

2007 0505 Fishoree
16. 06. 2007
0 views

2007 0505 Fishoree

2006 Function Presentation
16. 06. 2007
0 views

2006 Function Presentation

El mejor doctor 2076
19. 06. 2007
0 views

El mejor doctor 2076

BWands
04. 01. 2008
0 views

BWands

El Hombre Perfecto 2059
19. 06. 2007
0 views

El Hombre Perfecto 2059

Maria project
09. 07. 2007
0 views

Maria project

13 14
14. 11. 2007
0 views

13 14

El hipopotamo y la tortuga 2119
19. 06. 2007
0 views

El hipopotamo y la tortuga 2119

Ejercicios Espirituales 2116
19. 06. 2007
0 views

Ejercicios Espirituales 2116

CSU Presentation
20. 08. 2007
0 views

CSU Presentation

Jeff
28. 12. 2007
0 views

Jeff

El Respeto 2120
19. 06. 2007
0 views

El Respeto 2120

17579
20. 08. 2007
0 views

17579

Peru Seminar
20. 08. 2007
0 views

Peru Seminar

temptations
11. 08. 2007
0 views

temptations

SAB SA CorpStdPgm05608
21. 09. 2007
0 views

SAB SA CorpStdPgm05608

Edificios colmena 1993
19. 06. 2007
0 views

Edificios colmena 1993

eval
20. 08. 2007
0 views

eval

Takahiro Sato
29. 11. 2007
0 views

Takahiro Sato

tfpresent
11. 08. 2007
0 views

tfpresent

LMK preze
09. 07. 2007
0 views

LMK preze

pods03
29. 10. 2007
0 views

pods03

Social Engineering ISACA RIFE v5
20. 08. 2007
0 views

Social Engineering ISACA RIFE v5

Volkow505
20. 08. 2007
0 views

Volkow505

Towards Extracting Personality 4
11. 08. 2007
0 views

Towards Extracting Personality 4

Dios y el Zodiaco 1737
19. 06. 2007
0 views

Dios y el Zodiaco 1737

sanford poster
11. 08. 2007
0 views

sanford poster