Building crowds to produce high volumes of specialized work is an experiment I have been focused on for the last two years. It’s been exhilarating thus far, and I expect it will remain so.

To help you build a crowdsourcing company or execute on a crowdsourcing project, I thought I would put together a blog series on building crowds.

Part One: Put Clear Processes in Place

Part one continues below and covers the high level “how it gets done” and then shifts to a focus on how to make sure you avoid the largest pitfalls that will crush a project. (Teaser: ambiguity and quality control)

Part Two: Create Forums for Discussion

Once you have the processes ready, you need to make sure that you’re able to engage in bi-directional conversations with the crowd. (Simple Summary: Q&A with site/project/contest administrators and in the future you hopefully use the community itself for self-monitoring/regulation)

Part Three: Establish a Business Model to Sustain the Crowd

My thinking is that this is among the most important parts of any crowdsourcing project. If you don’t have the right model in place (whether that be simple monetary incentives or somewhat more complex incentives like social status), your crowdsourcing project will fail. I like to think of crowdsourcing like an orchestra, and the conductor is just one player.

Topic One: Building Crowds Requires Clear Processes

Leveraging crowds working collaboratively is much more than the interface people use to exchange information or the software for doing so. The interface and the technology used by the crowd to interact and produce something (i.e. the point of crowdsourcing) is really only a part of what is built when managing large crowds. If, say, half the software you build is for connecting all the outputs from the crowd into an aggregate (i.e. creating crowdsourcing projects), the other half you build is [hopefully] a result of identifying the needs related to managing crowds and executing on solutions to those needs.

In an ideal world, we could anticipate what all the major problems will be and build for those problems before we encounter them. That never happens, so when we start out we need to focus on the high-impact problems. Let’s examine two of those obvious needs.

One very basic principle that all crowdsourcing software, platforms, contests, and games must account for at the very beginning of design is quality control.

Here are a few ways to achieve quality control:

• Peer Review — have the work submitted by one worker reviewed by another worker (or more).
• Compare Worker Answers — have multiple workers complete the same task and compare the answers to find the real answer. It’s a type of peer review actually.
• Make a Video Game — one advantage to games is that they really control the environment (in terms of inputs and outputs). It’s hard to break and if the incentives are correctly aligned (i.e. there’s no incentive or way to cheat), making games could be really effective at generating accurate crowdsourced information. It’s just a question of scope.
• Strictly Control Access — the buzz word for this is ‘curation.’ A lot of crowdsourcing companies are talking about curation as the method for regulating who has access. It starts to breakdown for some situations. For example, does a four person crowd count as “crowdsourcing” or is that just “old fashioned freelance outsourcing”? (Depends on who you ask.)

There are numerous ways to create infrastructures for quality control. Whether you use one of the commonly used methods above alone or in combination or something different all together, make sure you think about this as one of the first problems your crowdsourcing company should try to solve.

Once you have a clear quality control process in place and the minimum software built to accomplish the tasks, there is one last, very easy to do step: Avoid Ambiguity.

I can provide a lot of funny examples of where I’ve seen this messed up, including by myself.
One of the clearest in my mind is for this image:

We asked whether the quality of this image was good, fair or poor. To our credit, we provided examples of each quality type to help set expectations. Where we went wrong was that we made this into a test. Knowing ‘good’ is a really sharp image, ‘fair’ has some blurriness around the edges but is clear and ‘poor’ is pretty much ‘crap’, what would you have said?

Another recent example makes me laugh every time I think of it. It’s also an example of how your attempts at quality control can backfire if you make assumptions or leave things ambiguous.

Here’s the description from a profile we evaluated.

We prompted our workers to evaluate the profile with the following:

To be acceptable, the section must:
Be understandable and complete
Note that you can ignore spelling, grammatical and punctuation errors
Not contain links to other sites
Do the text sections meet these conditions?

The goal of the evaluations were maintain minimum quality levels but the people working on the project were initially surprised this got through because they had set the situation up as a quality test.

It took a few moments to realize the evaluation parameters didn’t mention “threatening to hurt people or children” as a site no-no. Oops! Back to the drawing board!

So, when starting to build a crowd, make sure you have quality control measures in place (even if it just means starting with something rudimentary) and try to avoid ambiguity in your instructions to others or how you’ve framed your expectations.

Edit: Part Two of building better crowds is available here.