UNPLUGGED

Unleashing the Potential of Crowd Work: The Need for a Post-Taylorism Crowdsourcing Model

Ioanna Lykourentzou¹, Lionel P. Robert Jr.² and Pierre-Jean Barlatier^3*

¹Department of Information and Computing Sciences, Utrecht University, Utrecht, The Netherlands
²School of Information, University of Michigan, Ann Arbor, MI, USA
³EDHEC Business School, Nice Cedex 3, France

 

 

Abstract

Crowdsourcing, that is, the process of outsourcing a task to large numbers of people who contribute their work, knowledge, or experience and get monetary remuneration in return, has become a multi-billion-dollar industry and is revolutionizing work (Grewal-Carr & Bates, 2016; Ye et al., 2017). In this article, we use the terms crowdsourcing and paid crowdsourcing interchangeably, to refer this to this new labor model, to clearly distinguishing it from its nonmonetary counterparts. For more than a decade now, paid crowdsourcing has been used for a variety of tasks, leading to impressive results in terms of efficiency and timeliness (Kim & Robert, 2019). A distinction is made between micro-tasks, that is, brief tasks that do not require advanced skills, and macro-tasks, that is, more challenging long-lasting tasks that require specific advanced skills (Grier, 2013; Khan et al., 2019). Crowdsourcing platforms act then as intermediaries between the requesters who own the problems and are in need of a workforce, and the workers who accept to execute tasks. Such platforms can also manage relationships between requesters and workers and provide services like the handling of payments or practical support like verifying time worked (Robert, 2019).

Hence, gig economy entrepreneurs succeeded in creating platforms, in which labor could be purchased and dispensed with on demand, in virtual marketplaces such as Figure Eight or Amazon Mechanical Turk (AMT), engendering therefore a ‘commodification’ of labor (Wood et al., 2019). This has been particularly true for micro-tasks that can be clearly decomposed to the individual level, such as image recognition, sentiment analysis, and content moderation (Khan et al., 2019). In such platforms, labor is fragmented into tightly packaged tasks, which are spatially and temporally distributed across the crowdsourcing platform’s networks through algorithmically enhanced arm’s-length market transactions (Wood et al., 2019). As a consequence, although for the requester, this represents a very efficient access process to an on-demand workforce, the huge majority of online workers are left without legal rights and labor protection, no access to healthcare, and with their time spent on work-related activities, such as breaks, training, and job searching, being largely unpaid (Wood et al., 2019). This lack of social welfare for crowd workers has recently raised concerns and criticism. Nevertheless, to date, crowdsourcing platforms manage to avoid labor regulations by framing themselves only as intermediaries between requesters and crowd workers (Wood et al., 2019), in a similar manner to the way online content providers (e.g. YouTube) frame themselves as a conduit rather than a publisher of content (Gillespie, 2010).

Despite their massive 24/7 available workforce, crowdsourcing platforms have been unable to fully leverage the wisdom of the crowd. Innovation, creativity, and wicked problem-solving, all moving forces of today’s knowledge economy, still remain difficult to address at-scale through crowdsourcing (Kim & Robert, 2019).

Why? Applying old tools to new needs

But why? Although paid crowdsourcing affords the opportunity to engage the crowd in a wide range of work (Brabham 2008), most crowdsourcing work is still primarily focused on micro-tasking. This leads to a fundamental misalignment between the way crowdsourcing is currently approached and the conditions needed to fully leverage it to address complex and creative problems. The study of crowdsourcing platforms for innovation fully illustrates this point.

Indeed, in line with the rise of open innovation paradigm (Chesbrough, 2003), which praises notably the development of inbound innovation practices that promote access to external knowledge and expertise for in-house use, promising crowdsourcing initiatives supporting innovation have flourished. Innovation crowdsourcing platforms like Innocentive, NineSigma, TopCoder, Threadless, or Quirky have been champions in raising funds for their development (Van Alstyne et al., 2017), while in parallel, most of the Fortune 500 firms have developed their own innovation crowdsourcing initiatives (such as P&G’s Connect + Develop) and/or partnered with such platforms. Social media-based crowdsourcing offers a tremendous potential for open innovation and can be strategically used by companies to leverage dynamic capabilities and new business models (Mention et al., 2019).

However, the innovation performance of these platforms is still very limited, as their business model focuses almost exclusively on ‘innovation contests’, that is, helping clients select a single best idea among multiple individually contributed ones. This approach may be particularly relevant for product or service innovation, where ideas are considered as raw material (e.g. Fiat, LEGO, and P&G), but it does not support the efficient combination of expertise required to solve complex, wicked problems like the ones NASA or BP (during the Mexico Gulf crisis) is confronted with (Kim & Robert, 2019; Van Alstyne et al., 2017).

How Taylorism’s assumptions limit today’s crowdsourcing

From a managerial perspective, this focus on micro-tasking has led current crowd labor management algorithms and mainstream crowd labor markets to be predominantly influenced by Taylorism. Taylorism is a century-old management paradigm (Taylor, 1911), which focuses on the efficient production of as much work as possible, through extreme specialization and repetitive work only on designated task parts. It emphasizes managerial supervision and allows minimal interactions among the workers, since it perceives the latter as irresponsible and prone to social loafing. Algorithmically mediated crowd labor markets function in a similar manner. Their workflows decompose incoming tasks to repetitive, piece-rate subtasks (Alkhatib et al., 2017), which can be accomplished individually and without collaboration.

Consequently, relying on these guiding managerial principles has conferred crowd work several shortcomings and limits:

• Workers as human Central Processing Units (CPUs). Crowd workers are often presented as unemotional agents who work in a predictable manner. For example, major crowdsourcing platforms hide crowd workers behind a digital wall of ‘quality guarantee’ options (performance levels, automatic removal in case of low output, maximum tasks per worker, and so on). Clients never or rarely directly interact with workers (Wood et al., 2019), but instead, use an interface that allows them to filter workers by selecting certain options, as if they were fine-tuning an automated service (Irani, 2015).
• Micro-tasking dominates. Platforms and research primarily focus on work that can be decomposed to standalone micro-tasks, which can be accomplished without collaboration, and in a competitive for the workers environment. Micro-tasking is limited to problems that are well-defined and require a known skillset with limited dependencies. Unfortunately, many of today’s problems, from creating a new product to formulating an R&D approach, are non-decomposable, but instead they are complex and ill-defined, with a high degree of dependencies among their diverse knowledge domains (Majchrzak & Malhotra, 2013). Adherence to Taylorism principles limits our potential to address such problems in a scalable manner.
• Collaboration is not supported. Consistent with Taylorism, which does not value worker collaboration for fear of social loafing, current platforms discourage worker collaboration and interaction by design. As a result, workers are left isolated, with little or no access to peer social support mechanisms (Gray et al., 2016).
• Pay-per-output focus. Today’s paid crowd labor markets rely primarily on a single pay scheme to incentivize crowd workers: per-piece monetary compensation. Other payment schemes such as a fixed salary or providing healthcare benefits are not employed at-scale. This is fully in line with the output-based incentive structure advocated by Taylorism as well as neoliberalism. The pay-per-output model means that the time spent on work-related activities (such as breaks, training, or waiting for work) is a cost born by the crowd workers and not the platform, and it entails significant unpaid ‘work-for-labor’ costs for the workers (Berg, 2016; Florisson & Mandl, 2018). Pay-per-output increased worker fatigue and stress, further adding to the dehumanization of workers and their social isolation (Derksen, 2014).
• Rigid hierarchical work-structure. Crowdsourcing platforms use a two-layered, top-down coordination model, where task requesters dictate, and workers execute. All decisions regarding how work is to be performed are made without input from the workers. Unfortunately, this model is effective only for tasks that are decomposable (to atomic micro-tasks), homogeneous (all micro-tasks are similar in topic, difficulty, and expertise requirements), and close-ended. Consequently, complex tasks are often not assigned to the crowd.
• Ethical-legal questions. The pay-per-piece crowd labor model has raised legal and ethical questions, as illustrated by the crowd worker class actions against CrowdFlower (henceforth Figure Eight) and the ongoing legislative discussion on re-classifying Uber/Lyft drivers as a ‘third legal category of workers’ (Seiner, 2017). Even though labor regulations exist in most countries, crowdsourcing platforms often distance themselves from these regulations by labeling themselves as technology providers and mediators between independent contractors (Wood et al., 2019). Other scholars have already highlighted the power imbalances and the role of law and policy in these issues (Ekbia & Nardi, 2017; Milland, 2016; Silberman et al., 2018). The elements that underlie these political forces are likely to both further the need for change and to help to shape it. That withstanding, crowd workers are underpaid and overworked (Cherry, 2009) and have limited intellectual property rights (IPRs), ownership of their work or other benefits, making it difficult to attract and retain a highly skilled workforce.

Design solutions for post-Taylorism crowdsourcing

We should acknowledge that Taylorism is still viewed as an effective approach to micro-tasking and is still in use today. There are also scholars who see inherent conflicts in the creation of value for the platform, requester, and workers (Deng et al., 2016). Nevertheless, we believe that the best case for value creation is a move away from micro-tasking. Even if this argument is not shared by all, it is clear that to harness the crowd’s full potential, we need to move away from Taylorism and acknowledge that 21st-century work challenges cannot be treated with 20th-century work processes alone.

Later, we describe, through four design rules, how a shift from Taylorism can help develop online crowd workplaces able to address complex and ill-defined macro-tasks, at-scale.

Endorse, support, and leverage social networking

Despite the lack of support by platforms, crowd workers form unofficial social networks (El Maarry et al., 2018; Gray et al., 2016). These networks act as moral support and training structures, allowing workers to share the administrative costs of using the platform, exchange employment opportunities, benefit from peer training, and make social connections. However, these networks are often ad hoc, not systematically organized, and discoverable only through word-of-mouth. Recognizing their importance means incorporating social networking tools, micro-blogging, Q&A forums, chat rooms, personal profile pages, wikis, etc., within the crowd platform. It also means accepting a power shift away from the task requesters to the crowd because the information exchanged inside the social network will inevitably increase crowd workers’ expectations for better treatment and fairness. The time spent by workers in these networks should be properly acknowledged in the form of authorship recognition and financial payment (Kankanhalli et al., 2005). Finally, platforms should make these spaces safe for sharing, which includes providing opt-in provisions and fully disclosing the use of information.

Encourage collaboration

Research has highlighted the necessity of collaboration to address complex problems (Woolley et al., 2015). Collaboration enables individuals to combine multiple viewpoints to reach a solution (Kim & Robert, 2019; Robert et al., 2008), and research methods exist to optimally team people up based on their personality (Lykourentzou et al., 2016), preferences (Lykourentzou et al., 2017), or using matching algorithms (Basu Roy et al., 2014). Platforms should incorporate these approaches to better address complex tasks and to foster collaboration not only among crowd workers but also between crowd workers and clients. Relationships can reinforce trust and improve labor performance (Robert et al., 2008; Wood et al., 2019). The risk of disintermediation, where workers contact the client outside the platform, is expected to be low in this case because crowdsourcing makes it inefficient for the client to maintain working relationships with dozens or hundreds of crowd workers, outside the platform.

Value and invest in the crowd worker

Next-generation crowd platforms can improve retention by treating workers as trusted, capable, and responsible individuals. Concrete steps to attract and retain a highly skilled workforce include fair compensation, providing training, re-hiring, and rewarding competent workers, and promoting ownership of one’s work. Platforms could also offer flexible remuneration schemes, from salary to freelance payment, accompanied by other forms of compensation such as healthcare benefits. Certain platforms have gradually started to adopt such provisions, recognizing the need to offer their crowd workers more support and incentives (Aloisi, 2016). A recent example is the ‘Crowdsourcing Code of Conduct 2.0’ (http://crowdsourcing-code.com/), signed by eight major German platforms, which foresees e-learning training possibilities to create a more motivating and fulfilling work environment. Such development options could be further evaluated by the community and the task providers to make sure that they benefit all sides. Some of our recommendations, although distinct from, will align well with job enrichment programs suggested by other scholars (Deng et al., 2016).

Leverage a multilayered self-coordination

To alleviate the problems caused by the current two-layered, top-down work coordination model, recent studies suggest decentralizing the workflow and relying on worker self-organization. The tools needed to support this process differ based on the nature and complexity of the task. For open-ended tasks like the product design, software development, and game production, where the required expertise is known a priori, it is necessary to put in place tools that support peer-review, peer-mentoring, profiling, and expertise-building models (Valentine et al., 2017). For ill-defined tasks like the end-to-end groundbreaking innovation, which are currently not handled by commercial crowdsourcing platforms (Felin & Zenger, 2014; Kim & Robert, 2019), but which, nonetheless, represent the next frontier of crowdsourcing work, we must envision tools that allow workers to participate and even lead the workflow design process, establish social status, allocate roles, and monitor progress. Scholars have already highlighted the potential of role assignment in crowdsourcing (Valentine et al., 2017) and the emerging reality of subcontracting tasks by crowdworkers to other crowdworkers (Morris et al., 2017; Taylor & Joshi, 2018; Wood et al., 2019, for a review see Kim & Robert, 2019). To accomplish this, crowds need tools that act as distributed consensus mechanisms (e.g. based on the blockchain technology), enable synchronous or asynchronous discussion, and support progress self-monitoring, expert search, and task/person recommendations (Basu Roy et al., 2014). These tools can enable bottom-up worker self-coordination, which offers performance guarantees and is flexible enough to handle generic complex tasks in an ad hoc manner. Finally, new production models and governance structures would need to deal with IPR issues of the distributed complex work. Although these structures are new for crowd work, best practices could be transferred from other domains such as Open Source communities. Among shared IPR management models that could be adapted for crowd work we find: (1) structured license schemes developed by the participating designer worker base in collaboration with IPR owners, platforms, and users, (2) cocreating via the so-called ‘YouTube model’, that is, mixed licensing rules to enable cocreation on existing IPR-protected content, and (3) double licensing models (Ballardini et al., 2016).

Post-Taylorism crowdsourcing: A new deal among crowd workers, crowdsourcing platforms, and task requesters

The managerial and organizational consequences of such a paradigm shift are tremendous. Our post-Taylorism crowdsourcing model and its inherent design rules implies radically new insights regarding the crowd worker (e.g. highly skilled, trusted, and autonomous), their interactions within the crowdsourcing process (e.g. social interactions, networking, and collaboration), and their organization (e.g. self-coordination). All these represent managerial challenges and standpoints for further research.

The commodification of labor engendered by the current model of crowdsourcing is a too ad hoc process for dealing with macro-tasks, where the criteria for success and excellence differ from project to project. From this perspective, in the new model, we propose that crowd workers will be more active and less substitutable actors: ‘Experts solvers are indispensable, not interchangeable’ (Van Alstyne et al., 2017). Consequently, this leads to a new deal of power balance regarding the crowd workers, crowdsourcing platforms, and task requesters, which, in turn, appeals to new managerial settings. For instance, macro-task requesters, who post open-ended and innovation-oriented work that relies on the creativity and expertise of crowd workers, need to develop the crowd worker’s willingness to participate through incentives. Recent works highlight that the ‘classic’ monetary reward model of incentives is not relevant anymore in these new settings, and macro-task requesters need to find the right balance between extrinsic (e.g. money) and intrinsic (e.g. work feedback or career growth) motivators (Lykourentzou et al., 2019). Moreover, this necessary empowerment of crowd workers calls for crowd management innovations, in developing more flexible algorithms to align with macro-task management requirements, espousing crowd worker autonomy, creativity, and initiative-taking (Lykourentzou et al., 2019). As highlighted by Fixson and Marion (2016) in their analysis of Quirky’s initial failure, ‘specialist’ crowd workers do not like to be managed and want to be part of the collective decision-making process.

From an organizational standpoint, macro-task crowdsourcing administration calls for organizational innovations as well. Organizing crowdsourcing means organizing human resource allocation at a very large scale, largely beyond the traditional span of control of managers and, therefore, requires computation in defining algorithms to leverage scalability. The inherent complexity of macro-tasks generates uncertainty, epistemic interdependencies, and non-decomposability that makes them difficult to articulate, modularize, and predefine actions needed to achieve them. To overcome this organizational challenge, Valentine et al. (2017) suggest the creation of ‘flash organizations’ that involve defining ad hoc formal structures such as roles, teams, and hierarchies that allow coordination in delineating responsibilities, interdependencies, and information flows without prespecifying all actions. Their approach computationally structures activities around complex workflows and is characterized by two main principles. First, a deindividualized role hierarchy (as can be found in organizations like movie crews, disaster response teams, or the army) where collaboration is based on workers’ knowledge of the roles rather than their knowledge of each other; second, a continuous reconfiguration of the organization, done by changing roles or adding teams. Recent research highlights other types of structuring algorithms that focus more on teamwork, rotating workers according to their viewpoints or ideas in different team combinations (Salehi & Bernstein, 2018). All these recent developments of the organization of macro-task crowdsourcing platforms, although still mostly at research level, demonstrate a will to switch from direct supervision toward mutual adjustment designs that support self-coordination.

Finally, the rise of a new, post-Taylorism model of crowdsourcing entails both managerial and organizational challenges that pave the way for future research. Our proposed model facilitates the combination of expertise by multiple workers through collaboration, to achieve macro-tasks such as end-to-end innovation. The central focus is, henceforth, to design crowdsourcing platforms that provide value for both task requesters and crowd workers, from a virtuous, ecosystem-like perspective. This should be possible only in succeeding in raising awareness, work engagement, and platform commitment for a next-gen crowdsourcing platform design.

Conclusion

Paid crowdsourcing is a labor model that is dominated by micro-tasking, which is based on rigid work principles derived from Taylorism. Unfortunately, these principles are not suited to adequately address macro-tasking and the accrued complexity of modern problems. In this article, we plead for the emergence of a new, post-Taylorism crowdsourcing model based on crowd worker collaboration, trust, and fair treatment, which are better suited to address the wicked problems and grand challenges of the 21st century. To support this vision, we propose four design rules, namely, (1) leveraging crowd social networks, (2) encouraging teamwork and collaboration, (3) valuing and investing in the crowd worker, and (4) facilitating multilayered self-coordination. We accompany each rule with the tools and technologies that can be used in practice and highlight further research avenues from this perspective.

References

Alkhatib, A., Bernstein, M. S. & Levi, M. (2017). Examining crowd work and gig work through the historical lens of piecework. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 4599–4616). ACM, Denver Colorado USA. doi: 10.1145/3025453.3025974

Aloisi, A., Commoditized Workers. Case Study Research on Labour Law Issues Arising from a Set of ‘On-Demand/Gig Economy’ Platforms (May 1, 2016). Comparative Labor Law & Policy Journal, 37(3), 2016, Available at SSRN: http://dx.doi.org/10.2139/ssrn.2637485

Ballardini, R. M., Lindman, J. & Ituarte, I. F. (2016). Co-creation, commercialization and intellectual property-challenges with 3D printing. European Journal of Law and Technology, 7(3), 1–37.

Basu Roy, S., Lykourentzou, I., Thirumuruganathan, S., Amer-Yahia, S. & Das, G. (2014). Optimization in knowledge-intensive crowdsourcing. ArXiv E-Prints, arXiv:1401.1302.

Berg, J. (2016). Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers. Comparative Labor Law and Policy Journal, 37(3), 506–543.

Cherry, M. A. (2009). Working for (virtually) minimum wage: Applying the fair labor standards act in cyberspace. Pacific McGeorge School of Law Research Paper. Retrieved from https://ssrn.com/abstract=1499823

Chesbrough, H. (2003) Open Innovation: The New Imperative for Creating and Profiting, from Technology. Harvard Business School Press, Boston.

Brabham, D. C. (2008). Crowdsourcing as a model for problem solving. Convergence, 14(1), 75–90. doi: 10.1177/1354856507084420

Deng, X., Joshi, K. D. & Galliers, R. D. (2016). The duality of empowerment and marginalization in microtask crowdsourcing: Giving voice to the less powerful through value sensitive design. MIS Quarterly, 40(2), 279–302. doi: 10.25300/MISQ/2016/40.2.01

Derksen, M. (2014). Turning men into machines? Scientific management, industrial psychology, and the ‘human factor’. Journal of the History of the Behavioral Sciences, 50(2), 148–165. doi: 10.1002/jhbs.21650

Ekbia, H. R. & Nardi, B. A. (2017). Heteromation, and other stories of computing and capitalism. MIT Press.

El Maarry, K., Milland, K. & Balke, W. (2018). A fair share of the work? The evolving ecosystem of crowd workers. In H. Akkermans, K. Fontaine, & I. Vermeulen (Eds.), Proceedings of the 10th ACM conference on web science (pp. 145–152). Association for Computing Machinery. doi: 10.1145/3201064.3201074

Felin, T. & Zenger, T. R. (2014). Closed or open innovation? Problem solving and the governance choice. Research Policy, 43(5), 914–925. doi: 10.1016/j.respol.2013.09.006

Fixson, S. K. & Marion, T. J. (2016). A case study of crowdsourcing gone wrong. Harvard Business Review, December 15. Retrieved from https://hbr.org/2016/12/a-case-study-of-crowdsourcing-gone-wrong

Florisson, R. & Mandl, I. (2018). Platform work: Types and implications for work and employment – Literature review. Working paper WPEF18004 Eurofound, Dublin.

Gillespie, T. (2010). The politics of ‘platforms’. New Media & Society, 12(3), 347–364. doi: 10.1177/1461444809342738

Gray, M., Shoaib, S., Kulkarni, D. & Suri, S. (2016). The crowd is a collaborative network. In Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing (pp. 134–147). ACM, San Francisco California USA. doi: 10.1145/2818048.2819942

Grewal-Carr, V. & Bates, C. (2016). The three billion Enterprise crowdsourcing and the growing fragmentation of work. Deloitte LLP. Retrieved from https://www2.deloitte.com/content/dam/Deloitte/de/Documents/Innovation/us-cons-enterprise-crowdsourcing-and-growing-fragmentation-of-work%20(3).pdf

Grier, D. A. (2013). Crowdsourcing for dummies. John Wiley & Sons.

Irani, L. (2015). The cultural work of microwork. New Media & Society, 17(5), 720–739. doi: 10.1177/1461444813511926

Kankanhalli, A., Tan, B. C. Y. & Wei, K. (2005). Contributing knowledge to electronic knowledge repositories: An empirical investigation. MIS Quarterly, 29(1), 113–143. doi: 10.2307/25148670

Khan, J., Lykourentzou, I., Papangelis, K. & Markopoulos, P. (2019). Macro-task crowdsourcing: Engaging the crowds to address complex problems. Springer Human Computer Interaction Series. Springer Nature Switzerland.

Kim, S. & Robert, L. P. (2019). Crowdsourcing coordination: A review and research agenda for crowdsourcing coordination used for macro-tasks. In J. Khan, I. Lykourentzou, K. Papangelis, & P. Markopoulos (Eds.), Macro-task crowdsourcing: Engaging the crowds to address complex problems (pp. 17–43). Springer Human Computer Interaction Series. Springer, Cham, New York USA. doi: 10.1007/978-3-030-12334-5_2

Lykourentzou, I., Antoniou, A., Naudet, Y. & Dow, S. P. (2016). Personality matters: Balancing for personality types leads to better outcomes for crowd teams. In D. Gergle & M. R. R. Morris (Eds.), Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (pp. 260–273). Association for Computing Machinery. doi: 10.1145/2818048.2819979

Lykourentzou, I., Khan, V. J., Papangelis, K. & Markopoulos, P. (2019). Macrotask crowdsourcing: An integrated definition. In V.–J. Khan, K. Papangelis, I. Lykourentzou & P. Markopoulos (Eds.), Macrotask crowdsourcing (pp. 1–13). Springer.

Lykourentzou, I., Kraut, R. E. & Dow, S. P. (2017). Team dating leads to better online ad hoc collaborations. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (pp. 2330–2343). ACM, Portland Oregon USA. doi: 10.1145/2998181.2998322.

Majchrzak, A. & Malhotra, A. (2013). Towards an information systems perspective and research agenda on crowdsourcing for innovation. Journal of Strategic Information Systems, 22(4), 257–268. doi: 10.1016/j.jsis.2013.07.004

Mention, A. L., Barlatier, P.-J. & Josserand, E. (2019). Using social media to leverage and develop dynamic capabilities for innovation. Technological Forecasting and Social Change, 144, 242–250. doi: 10.1016/j.techfore.2019.03.003

Milland, K. (2016). A mechanical turk worker’s perspective. Journal of Media Ethics, 31(4), 263–264. doi: 10.1080/23736992.2016.1228813

Morris, M. R., Bigham, J. P., Brewer, R., Bragg, J., Kulkarni, A., Li, J. & Savage, S. (2017). Subcontracting Microwork. In G. Mark & S. Fussell (Eds.), CHI ‘17: Proceedings of the 2017 CHI Conference on human factors in computing systems (pp. 1867–1876). Association for Computing Machinery. doi: 10.1145/3025453.3025687

Robert, L. P., Dennis, A. R. & Ahuja, M. K. (2008). Social capital and knowledge integration in digitally enabled teams. Information Systems Research, 19(3), 314–334. doi: 10.1287/isre.1080.0177

Robert, L. P. (2019). Crowdsourcing controls: A review and research agenda for crowdsourcing controls used for macro-tasks. In J. Khan, I. Lykourentzou, K. Papangelis, & P. Markopoulos (Eds.), Macro-task crowdsourcing: Engaging the crowds to address complex problems (pp. 45–126). Springer Human Computer Interaction Series. doi: 10.1007/978-3-030-12334-5_3

Salehi, N. & Bernstein, M. S. (2018). Hive: Collective design through network rotation. Proceedings of the ACM on Human-Computer Interaction, 2, 1–26. doi: 10.1145/3274420

Seiner, J. (2017). Tailoring class actions to the on-demand economy. Ohio State Law Journal, 78(1), 21–71.

Silberman, M. S., Tomlinson, B., LaPlante, R., Ross, J., Irani, L. & Zaldivar, A. (2018). Responsible research with crowds: Pay crowd-workers at least minimum wage. CACM, 61(3), 39–41. doi: 10.1145/3180492

Taylor, F. W. (1911). The principles of scientific management. Harper & Row.

Taylor, J. & Joshi, K. (2018). How IT leaders can benefit from the digital crowdsourcing workforce. MIS Quarterly Executive, 17(4), 281–295. doi: 10.17705/2msqe.00002

Valentine, M. A., Retelny, D., To, A., Rahmati, N., Doshi, T. & Bernstein, M. S. (2017). Flash organizations: Crowdsourcing complex work by structuring crowds as organizations. In G. Mark & S. Fussell (Eds.), Proceedings of the 2017 CHI conference on human factors in computing systems (pp. 3523–3537). Association for Computing Machinery. doi: 10.1145/3025453.3025811

Van Alstyne, M. W., Di Fiore, A. & Schneider, S. (2017). 4 mistakes that kill crowdsourcing efforts. Harvard Business Review, July 21. Retrieved from https://hbr.org/2017/07/4-mistakes-that-kill-crowdsourcing-efforts

Wood, A. J., Graham, M., Lehdonvirta, V. & Hjorth, I. (2019). Networked but commodified: The (dis)embeddedness of digital labour in the gig economy. Sociology, 53(5), 931–950. doi: 10.1177/0038038519828906

Woolley, J., Madsen, T. L. & Sarangee, K. (2015). Crowdsourcing or expertsourcing: Building and engaging online communities for innovation? In Conference Paper Presented at DRUID15, (pp. 15–17). 15–17 June 2015, Rome.

Ye, T., You, S. & Robert, L. (2017). When does more money work? Examining the role of perceived fairness in pay on the performance quality of crowdworkers. In Proceedings of the International AAAI Conference on Web and Social Media, 11(1), 327–336. Retrieved from https://ojs.aaai.org/index.php/ICWSM/article/view/14876