Experimentally Validated Surveys: Potential for Studying Cognitive and Behavioral Issues in Management
Abstract
In management, addressing behavioral and cognitive issues empirically is particularly challenging, as it requires combining both internal and external validity with multilevel data collection. The traditional quantitative methodologies used in this area, which are surveys and laboratory experiments, are ill-suited for this purpose. This paper presents a promising, innovative methodology, the experimentally validated survey (EVS), which allows researchers to face this challenge. The EVS relies on the implementation of survey questions previously validated by a controlled experiment. This paper adopts a threefold approach: descriptive, achieved by proposing a structured review of the recent body of literature; practical, achieved by discussing technical issues that one may encounter while running an EVS; and critical, achieved by discussing the advantages and limitations of such methodology for management scholars. Our main contribution is initiating a dynamic movement toward the importation of this innovative methodology into management research.
Downloads
References
Abell, P., Felin, T. & Foss, N. (2008). Building micro-foundations for the routines, capabilities, and performance links. Managerial and Decision Economics, 29(6), 489–502. doi: 10.1002/mde.1413
Aguinis, H. & Lawal, S. O. (2012). Conducting field experiments using eLancing’s natural environment. Journal of Business Venturing, 27(4), 493–505. doi: 10.1016/j.jbusvent.2012.01.002
Allinson, C. W., Chell, E. & Hayes, J. (2000). Intuition and entrepreneurial behaviour. European Journal of Work and Organizational Psychology, 9(1), 31–43. doi: 10.1080/135943200398049
Barbosa, S., Fayolle, A. & Smith, B. (2019). Biased and overconfident, unbiased but going for it: How framing and anchoring affect the decision to start a new venture. Journal of Business Venturing, 34(3), 528–557. doi: 10.1016/j.jbusvent.2018.12.006
Barsade, S. G. (2002). The ripple effect: Emotional contagion and its influence on group behavior. Administrative Science Quarterly, 47(4), 644–675. doi: 10.2307/3094912
Bekir, I., El Harbi, S., Grolleau, G. & Sutan, A. (2015). Efficiency, equality, positionality: What do people maximize? Experimental vs. hypothetical evidence from Tunisia. Journal of Economic Psychology, 47(C), 77–84. doi: 10.1016/j.joep.2015.01.007
Bird, B. & Schjoedt, L. (2009) Entrepreneurial behavior: Its nature, scope, recent research, and agenda for future research. In A. L. Carsrud & M. Brännback (Eds.), Understanding the entrepreneurial mind (pp. 327–358). Springer. doi: 10.1007/978-1-4419-0443-0_15
Bird, B., Schjoedt, L. & Baum, J. R. (2012). Editor’s introduction. Entrepreneurs’ behavior: Zlucidation and measurement. Entrepreneurship, Theory and Practice, 36(5), 889–913. doi: 10.1111/j.1540-6520.2012.00535.x
Bohm, P. (1972). Estimating demand for public goods: An experiment. European Economic Review, 3(2), 111–130. doi: 10.1016/0014-2921(72)90001-3
Busenitz, L. W. (1999). Entrepreneurial risk and strategic decision making: It’s a matter of perspective. Journal of Applied Behavioral Science, 35(3), 325–340. doi: 10.1177/0021886399353005
Busenitz, L. W. & Barney, J. B. (1997). Differences between entrepreneurs and managers in large organizations: Biases and heuristics in strategic decision-making. Journal of Business Venturing, 12(1), 9–30. doi: 10.1016/s0883-9026(96)00003-1
Caliendo, M., Fossen, F. M. & Kritikos, A. S. (2009). Risk attitudes of nascent entrepreneurs–new evidence from an experimentally validated survey. Small Business Economics, 32, 153–167. doi: 10.1007/s11187-007-9078-6
Campbell, D. T. & Stanley, J. C. (1966) Experimental and quasi-experimental designs for research. Houghton Mifflin Company. doi: 10.4135/9781483346229.n137
Chandler, G. N. & Lyon, D. W. (2001). Issues of research design and construct measurement in entrepreneurship research: The past decade. Entrepreneurship, Theory and Practice, 25(4), 101–114. doi: 10.1177/104225870102500407
Chang, J. B., Lusk, J. L. & Norwood, F. B. (2009). How closely do hypothetical surveys and laboratory experiments predict field behavior? American Journal of Agricultural Economics, 91(2), 518–534. doi: 10.1111/j.1467-8276.2008.01242.x
Charness, G. (2010). Laboratory experiments: Challenges and promise: A review of “Theory and Experiment: What are the Questions?” by Vernon Smith. Journal of Economic Behavior & Organization, 73(1), 21–23. doi: 10.1016/j.jebo.2008.11.005
Charness, G., Gneezy, U. & Kuhn, M. A. (2013). Experimental methods: Extra-laboratory experiments-extending the reach of experimental economics. Journal of Economic Behavior & Organization, 91, 93–100. doi: 10.1016/j.jebo.2013.04.002
Chen, S. S., Ho, K. Y. & Ho, P. H. (2014). CEO overconfidence and long-term performance following R&D increases. Financial Management, 43(2), 245–269. doi: 10.1111/fima.12035
Churchill, G. A. (1979). A paradigm for developing better measures of marketing constructs. Journal of Marketing Research, 16(1), 64–73. doi: 10.2307/3150876
Coppola, M. (2014). Eliciting risk-preferences in socio-economic surveys: How do different measures perform? The Journal of Socio-Economics, 48, 1–10. doi: 10.1016/j.socec.2013.08.010
Croson, R., Anand, J. & Agarwal, R. (2007). Using experiments in corporate strategy research. European Management Review, 4(3), 173–181. doi: 10.1057/palgrave.emr.1500082
Crowne, D. P. & Marlowe, D. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24(4), 349–354. doi: 10.1037/h0047358
Ding X., Hartog, J. & Sun, Y. (2010). Can we measure individual risk attitudes in a survey? IZA working paper, n°4807.
Dohmen T., Falk, A., Huffman, D. & Sunde, U. (2005). Explaining trust: Disentangling the roles of beliefs and risk preference. IZA Working Paper.
Dohmen, T., Falk, A., Huffman, D., Sunde, U. et al. (2011). Individual risk attitudes: Measurement, determinants, and behavioral consequences. Journal of the European Economic Association, 9(3), 522–550. doi: 10.1111/j.1542-4774.2011.01015.x
Douglas, E. J. & Shepherd, D. A. (2000). Entrepreneurship as a utility maximizing response. Journal of Business Venturing, 15(3), 231–251. doi: 10.1016/s0883-9026(98)00008-1
Falk, A., Becker, A., Dohmen, T. J., Huffman, D. & Sunde, U. (2016). The preference survey module: A validated instrument for measuring risk, time, and social preferences. SSRN Electronic Journal, 1–66. doi: 10.2139/ssrn.2725874
Falk, A. & Fehr, E. (2003). Why labour market experiments? Labour Economics, 10(4), 399–406. doi: 10.1016/s0927-5371(03)00050-2
Fehr, E., Fischbacher, U., Von Rosenbladt, B., Schupp, J. et al. (2002). A nation-wide laboratory: Examining trust and trustworthiness by integrating behavioral experiments into representative surveys. SSRN Electronic Journal, 1–23. doi: 10.2139/ssrn.385120
Felin, T., Foss, N. J. & Ployhart, R. E (2015). The microfoundations movement in strategy and organization theory. The Academy of Management Annals, 9(1), 575–632. doi: 10.1080/19416520.2015.1007651
Galasso, A. & Simcoe, T. S. (2011). CEO overconfidence and innovation. Management Science, 5(8), 1469–1484. doi: 10.1287/mnsc.1110.1374
Gillespie, R. (1991) Manufacturing knowledge: A history of the Hawthorne experiments. Cambridge University Press.
Gneezy, U. & Imas, A. (2017) Lab in the field: Measuring preferences in the wild. In A. Banerjee & E. Duflo (Eds.), Handbook of field experiments (vol. 1, pp. 439–464). Elsevier. doi: 10.1016/bs.hefe.2016.08.003
Goldstein, W. M. & Hogarth, R. M. (1997). Judgment and decision research: Some historical context. In W. M. Goldstein & R. M. Hogarth (Eds.), Cambridge series on judgment and decision making. Research on judgment and decision making: Currents, connections, and controversies (pp. 3–65). Cambridge University Press. doi: 10.1002/9780470752937
Graf, L., König, A., Enders, A. & Hungenberg, H. (2012). Debiasing competitive irrationality: How managers can be prevented from trading off absolute for relative profit. European Management Journal, 30(4), 386–403. doi: 10.1016/j.emj.2011.12.001
Grazzini, F. & Boissin, J.-P. (2013). Managers’ mental models of small business acquisition; the case of the SME French Transfert market. M@n@gement, 16(1), 49–87. doi: 10.3917/mana.161.0049
Grégoire, D., Binder, J. & Rauch, A. (2019). Navigating the validity tradeoffs of entrepreneurship research experiments: A systematic review and best-practice suggestions. Journal of Business Venturing, 34(2), 284–310. doi: 10.1016/j.jbusvent.2018.10.002
Hainmueller, J., Hangartner, D. & Yamamoto, T. (2015). Validating vignette and conjoint survey experiments against real-world behavior. Proceedings of the National Academy of Sciences, 112, 2395–2400. doi: 10.1073/pnas.1416587112
Hardeweg, B., Menkhoff, L. & Waibel, H. (2013). Experimentally validated survey evidence on individual risk attitudes in rural Thailand. Economic Development and Cultural Change, 61(4), 859–888. doi: 10.1086/670378
Harrison, G. W. & List, J. A. (2004). Field experiments. Journal of Economic Literature, 42(4), 1009–1055. doi: 10.1257/0022051043004577
Hisrich, R., Langan-Fox, J. & Grant, S. (2007). Entrepreneurship research and practice: A call to action for psychology. American Psychologist, 62(6), 575–589. doi: 10.1037/0003-066x.62.6.575
Hogarth, R. M. (1987). Judgement and choice: The psychology of decision (2nd ed.). John Wiley.
Huber, L. R., Sloof, R. & Van Praag, M. (2014). The effect of early entrepreneurship education: Evidence from a field experiment. European Economic Review, 72(C), 76–97. doi: 10.1016/j.euroecorev.2014.09.002
James, H. (2006). Family capitalism: Wendels, Haniels, Falcks, and the continental European model. Harvard University Press.
Johnson, N. D. & Mislin, A. (2012). How much should we trust the world values survey trust question? Economics Letters, 116(2), 210–212. doi: 10.1016/j.econlet.2012.02.010
Kahneman, D. (2003). Maps of bounded rationality: Psychology for behavioral economics. American Economic Review, 93(5), 1449–1475. doi: 10.1257/000282803322655392
Kahneman, D., Slovic, P. & Tversky, A. (1982), Judgement under uncertainty: Heuristics and biases. Cambridge University Press. doi: 10.1017/cbo9780511809477
Kirk, R. (1995). Experimental design, procedures for the behavioural sciences. Brooks/Cole Publishing Company.
Laroche, H. & Nioche, J. P. (2015). L’approche cognitive de la stratégie d’entreprise. Revue Française de Gestion, 8(253), 97–120. doi: 10.3166/rfg.160.81-108
Lavrakas P. J. (2008). Encyclopedia of survey research methods. Sage. doi: 10.4135/9781412963947
List, J., Sadoff, A. & Wagner, M. (2011). So you want to run and experiment, now what? Some simple rules of thumb for optimal experimental design. Experimental Economics, 14(4), 439–457. doi: 10.1007/s10683-011-9275-7
McNeil, B., Pauker, S., Sox, H. & Tversky, A. (1982). On the elicitation of preferences for alternative therapies. The New England Journal of Medicine, 306(21), 1259–1262. doi: 10.1056/nejm198205273062103
Mintzberg, H. (1973). The nature of managerial work. Harper & Row.
Mitchell, R., Busenitz, L., Bird, B., Caglio, C. et al. (2007). The central question in entrepreneurial cognition research 2007. Entrepreneurship Theory and Practice, 31(1), 1–27. doi: 10.1111/j.1540-6520.2007.00161.x
Mitchell, R., Busenitz, L., Lant, T., McDougall, P. et al. (2002). Toward a theory of entrepreneurial cognition: Rethinking the people side of entrepreneurship research. Entrepreneurship, Theory and Practice, 27(2), 93–104. doi: 10.1111/1540-8520.00001
Mizik, N. (2010). The theory and practice of myopic management. Journal of Marketing Research, 47(4), 594–611. doi: 10.1509/jmkr.47.4.594
Molina-Azorin, J. F., Lopez-Gamero, M. D., Pereira-Moliner, J. & Pertusa-Ortega, E. M. (2012). Mixed methods studies in entrepreneurship research: Applications and contributions. Entrepreneurship and Regional Development, 24(5–6), 425–456. doi: 10.1080/08985626.2011.603363
Moore, D. & Flynn, F. J. (2008). The case for behavioral decision research in organizational behavior. The Academy of Management Annals, 2(1), 399–431. doi: 10.1080/19416520802211636
Moore, D. A. & Healy, P. J. (2008). The trouble with overconfidence. Psychological Review, 115(2), 502–517. doi: 10.1037/0033-295x.115.2.502
Paulhus, D. L. (1991). Measurement and control of response bias. In J. P. Robinson, P. R. Shaver & L. S. Wrightsman (Eds.), Measures of social psychological attitudes (vol. 1, pp. 17–59). Academic Press. doi: 10.1016/b978-0-12-590241-0.50006-x
Phan, P. & Wright, M. (2018). Advancing the science of human cognition and behavior. Academy of Management Perspectives, 32(2), 179–181. doi: 10.5465/amp.2018.0058
Powell, T., Lovallo, D. & Fox, C. (2011). Behavioral strategy. Strategic Management Journal, 32(13), 1369–1386. doi: 10.1002/smj.968
Sadoff, S. & Samek, A. (2019). Can interventions affect commitment demand? A field experiment on food choice. Journal of Economic Behavior & Organization, 158(C), 90–109. doi: 10.1016/j.jebo.2018.11.016
Shepherd, D.A., Williams, T. A. & Patzelt, H. (2015). Thinking about entrepreneurial decision making: Review and research agenda. Journal of Management, 41(1), 11–46. doi: 10.1177/0149206314541153
Simon, H. A. (1947). Administrative behavior. Macmillan.
Simon, H. A (1955). A behavioral model of rational choice. Quarterly Journal of Economics, 69(1), 99–118. doi: 10.2307/1884852
Simon, H. A. (1979). Rational decision making in business organizations. American Economic Review, 69(4), 493–513.
Song, Y., Liu Y., Lanaj, M. W. K., Johnson, R. E. et al. (2018). A social mindfulness approach to understanding experiences customer mistreatment: A within person field experiment. Academy of Management Journal, 61(3), 994–1020. doi: 10.5465/amj.2016.0448
Stein, J. C. (1988). Takeover threats and managerial myopia. Journal of Political Economy, 96(1), 61–80. doi: 10.1086/261524
Stein, J. C. (1989). Efficient capital markets, inefficient firms: A model of myopic corporate behavior. Quarterly Journal of Economics, 104(4), 655–669. doi: 10.2307/2937861
Thaler, R. & Sunstein C. R. (2008). Nudge: Improving decisions about health, wealth, and happiness. Yale University Press.
Tashakkori, A. & Teddlie, C. (2003). Handbook of mixed methods in social & behavioral research. Sage.
Vieider, F., Lefebvre, M., Bouchouicha, R., Chmura, T. et al. (2015). Common components of risk and uncertainty attitudes across contexts and domains: Evidence from 30 countries. Journal of the European Economic Association, 13(3), 421–452. doi: 10.1111/jeea.12102
Vischer, T., Dohmen, T., Falk, A., Huffman, D. et al. (2013). Validating an ultra-short survey measure of patience. Economics Letters, 120(2), 142–145. doi: 10.1016/j.econlet.2013.04.007
Vo, L.-C., Culié, J.-D. & Mounoud, E. (2016). Microfoundations of decoupling: From a coping theory perspective. M@n@gement, 19(4), 248–276. doi: 10.3917/mana.194.0248
Williams, D., Wood, M., Mitchell, R. & Urbig, D. (2019). Applying experimental methods to advance entrepreneurship research: On the need for and publication of experiments. Journal of Business Venturing, 34(2), 215–223. doi: 10.1016/j.jbusvent.2018.12.003
Zellmer-Bruhn, M., Caligiuri, P. & Thomas, D. C. (2016). From the editors: Experimental designs in international business research. Journal of International Business Studies, 47(4), 399–407. doi: 10.1057/jibs.2016.12
Zikmund, G., Babin, B. J., Carr, J. C. & Griffin, M. (2013). Business research methods. Cengage Learning Custom.
Authors retain copyright of their work, with first publication rights granted to the AIMS.
Published by 