Archive for crowdsourcing at servio

Aligning Incentives & Expectations When Crowdsourcing

I’m happy to report that over the last week we transitioned all of our writing projects at Servio to a modified workflow.

Before explaining the change and why I’m happy (and how it relates to incentives and expectations), it’s important to understand how all work gets approved on CloudCrowd, Servio’s crowdsourcing platform.

Luckily it’s quite simple: all work is peer reviewed a minimum of one time, and frequently up to three or even four times, depending on context or work complexity. This means that if I, as a worker, am instructed to “go find an image of a camera on a white background measuring 300 by 300 px” then another worker will verify that the image I’ve submitted meets all the criteria outlined in the project instructions.

The peer worker reviewing my submission makes a determination to approve my work or reject it. If approved, I get paid. If rejected, I do not.

Here we can see that workers have a very strong economic incentive to provide quality, accurate work because failure to do so means any time devoted to the project is a waste of time. I think of this process as creating and leveraging social accountability to ensure quality. The peer review process is one way Servio guarantees quality, and it’s extremely powerful. I discuss a few other quality control techniques here and here.

Prior to the change in workflows, the writing process at Servio followed this same two-step process:
1st — a writer would receive instructions on what topic and length to write on. There would likely be structural and tone requirements too. (You can view a fairly extensive set of options for custom content writing on Servio.)
2nd — the “peer” in this case would be an editor, not a writer.

On the surface this seems fine. It even mimics the real-world in many ways. Writers write and editors edit. Seems fine because we’re mimicking the real world and known social forms—one of my (personal) requirements for how crowdsourcing workflows should flow.

So where’s the problem you ask?

The editors reviewing written content were instructed to approve a document if there were less than two grammatical errors. If there were more than two or if there were any spelling errors, the document should be rejected. In this case, the writer would receive no payment for the work.

At first glance, this may seem totally acceptable. Of course the writer shouldn’t be paid for submitting writing that has grammatical or spelling errors. However, we have to remember the perspectives of a writer and an editor are very different. Editors know all sorts of nitty-gritty details on grammar, frequently details the common writer, including myself, won’t remember (or even care to). Editors on Servio are pretty hardcore. You have to really know your stuff if you want to keep up.

This situation created an expectation imbalance in which the editors were holding writers to unrealistic expectations. I’m not trying say that writers shouldn’t know as much about grammar as possible. I am saying that small slips in grammar rules should not be a determination in whether you get paid for the bulk of your work.

Remember, if the submitted writing is rejected because of a couple of comma splices and a misuse of parallelism in a series, then the writer gets nothing. If the writing project was for a 2000 word article on the risks of Oxycontin/Oxycodone addiction or Lithium withdrawal, subjects that would require fairly extensive research to write on, then a writer may have two hours of work thrown out for fairly minor reasons. Talk about a negative incentive to participate! The result is low worker morale, low throughput, and frustration between editors and writers.

I’ll go into detail on other techniques Servio uses to avoid wasted work like this another time—we’ve released a series of really, really awesome feature/workflow enhancements starting in 2010 to address this overall problem.

For now, we can contrast the prior workflow (the editors reviewing writer work) to the new workflow.

The New Workflow

Rather than having editors reviewing writer work directly, writers review writer submissions. After a writer’s article is approved (by another writer during the peer-review process), the article is automatically routed to an editor who is responsible for addressing deeper facets of the writing. Here’s a portion of what we expect from editors:

Proofreading: Proofreading is merely to check a written text for errors in spelling and grammar.
Editing: Editing is the process of selecting and preparing language. This means, editors are required to make improvements to word choice and sentence structure, otherwise it’s merely proofreading.

Note that we don’t expect a trivial proofreading of the content. We expect the editors to focus on improving the writing, from sentence structure to word choice.

By isolating these project expectations—that is, by allowing writers to remain responsible exclusively for adherence to topic/focus/tone requirements and routing completed articles to an edit-only work queue—we’re able to achieve a much higher quality in final article while also better aligning expectations for both writers and editors.

The result is a lot less stress for the writers (no longer held to unrealistic expectations) and better pay (less rejected work for small errors means the average earning per submitted article will go up). Another result is faster completion of each writing assignment (primarily due to lower rejection rates). In a business that sells on quality and scale, this is a pretty big deal.

For business reasons, I cannot explain why we originally started with the first flawed workflow, but I can say that I’m extremely happy with where we are now. (Feedback on the worker support forums indicates workers are happy with this too!)

The Important Lesson

When designing the workflows and writing instructions to workers, it is important to properly aligned expectations and incentives (pay, points, rewards, etc). And as I illustrated above, small changes in the workflow can have major impacts on quality, worker satisfaction (hugely important) and throughput.

Presentation at CrowdConf 2011: Building Killer Crowds to Maximize ROI

Below is the presentation I gave at CrowdConf 2011. I focused on building and maintaining crowds. My general premise is that for a crowdsourcing company one of the top assets is the crowd. I discuss a few tips for insuring you have a strong crowd that understands expectations and holds itself accountable to those expectations. I focused on the minimum tools needed, different quality control mechanisms, and provided a few examples of where we learned some lessons (at Servio).

Thoughts on CrowdConf 2011

Because I was hosting a breakout session for CrowdConf 2011, I was pretty excited going into the conference. Who would be there? What cool demos would I get to see? Who would have the next big idea [that could I use it at Servio]?

Below I run through each of the companies and highlight what I found to be notable.

I conclude the post with some thoughts and speculation on the industry and the future of labor.


Coffee & Power
New company started by Philip Rosedale that feels similar to an Elance/Guru/oDesk. The marketplace was supposedly built using crowdsourcing only. The founders also created an iPhone app quickly after launching Coffee & Power, using only the “platform.” I use platform loosely because Rosedale refers to his site as a platform but it feels more like a marketplace than an extensible platform…
I like the fact that it has its own built-in currency even though there are a ton of complications related to digital currencies–the proliferation of them makes me think the headaches are worth it though…

Creating a job, posting a skill or bidding on work appears super easy but there’s only a light layer of organization. There is a form of credibility system but it feels pretty lightweight. (“so-and-so is trusted by x number of people”)

Ultimately, I feel like I’m looking at a nice version Craigslist.

These guys are an old guard outsourcing company founded back in the 60s as a back office record keeping service. They’re supposedly really big now–from what I can find they’re a multi-billion dollar per year business. Not bad.

What’s interesting to me is that they’re not even approaching the problem from a crowdsourcing perspective. The way they look at it, they’ve been solving complex, distributed workflows for years–and doing it in highly regulated markets (financials, mutual funds, etc).

These guys are launching what I would call a platform for getting work done. It’s the logical step for a business already working with distributed labor channels.

The product is suppose to be flexible enough to plug into companies such that the company can design a workflow with their software and then tap into on-demand labor when needed.

There was no real demo of this software, so I’m a little skeptical of the depth of execution: Is it all talk and lots of hand-waving about the future of labor organization or do they have something that could plug in at Coca-Cola?

From a strategy perspective, these guys don’t strike me as making a strong technology play but I do think they have the distribution channels in place (i.e customers paying top-dollar and sustained budgeting for expanding lean-labor needs).

These guys setup a site for crowdsourcing startups/businesses. The founders were saying “platform” but it also felt more like a website than a platform.

What they showed must have been mockups or screenshots from a private section of the site because I haven’t been able to locate a login button on the site (but there is a sign out button that doesn’t work!)

From what I can tell, they’re looking to blend the needs of different participants in a business venture (investors, idea-peeps and expertise-providers) in hopes that people on the internet can carry out the necessary steps to have successful businesses. They essentially posit that you can run a completely distributed business and that to get the services you require, you either need to provide capital, skills or ideas and UFOStart is a site for channeling each of those needs/offers. Supposedly everyone gets compensated through shares or money or both.

I definitely think there is room for collective ownership but it’s all about the details and without a login, there are basically no details. :(

This was a long sales pitch in my opinion. The speaker didn’t seem particularly prepared to talk about anything but LingoTech. It may have just been a case of thinking “buyers, buyers, buyers” and then that not being the audience (or maybe it was just me).

LingoTech is integrated with Oracle and Jive (and probably others I didn’t see) which I thought was good because it positions them to strong distribution channels. (i.e the localization manager can say, “well, they’re integrated with the corporate infrastructure, so we should use {this platform}.”)

WRT to Jing he said, “companies need translation in order to understand what they’re customers are saying on their own websites.”
LingoTech also have providing Translation Memory/Glossary Support, Fuzzy Match.

LingoTech uses the following MTs: Google, Bing, ProMT.

Ultimately, I saw a company really focusing in on the translation industry with tools that are needed to remain competitive. It should be interesting to see if LingoTech can challenge LionBridge. (My money is against LingoTech because nothing “oooo, ahhhh, that’s a winner” came out of that breakout and that’s what will be needed to unseat the market gorilla).

My friend Jason left Servio to become the first employee at this company, so first it is cool to see something public of the work he’s been doing. :)

Humanoid is a platform for doing simple tasks. They supposedly have a UI for building projects. I wasn’t able to get a demo. What I like is that “humanoid” grew out of SpeakerText’s need to have a better platform option than Amazon Mechanical Turk. Basically, what AMT was providing was total crap in terms of quality (this is what the founder Matt said). To circumvent this, they built their own platform. Cool.

This is a play taken right out of ReWork.

What I don’t like is that they claim it’s a platform that “anyone can use” but they don’t have any exposed UI for it–or at least one that I could access. This makes me feel like it’s all talk and no walk, but I know my friend Jason is good so there’s got to be something to see under the hood!

They also claim that hopefully through machine learning, they’ll be able to continually reduce the need for human labor because the “system will know how to answer the question once it’s been trained by the crowd.” That seems like a mighty claim that actually has some feasibility to it in my opinion. It’s all about the type and scope of work.

For example, there are some great synergies between their parent company SpeakerText and some of the alternative offerings they have (translation, OCR, etc). It’s likely that type of work they’re referring to and not “video creation” or something complex like that.

“Industry Champions” (Panel Discussion)
Panel consisting of CEO of Trada Niel Robertson, VP of Amazon Mechanical Turk Sharon Chiarella, and CMO of uTest Matt Johnson, LiveOps EVP of Sales Matt Fisher.  

Two notable take-aways:

1) The established, focused, skill-based crowdsourcing companies (uTest, LiveOps and Trada) start thinking about attrition and have solved the attrition problem before trying to tackle the enterprise market. uTest thinks they’ve got this down–and I will say from talking with their Director of Public Relations (reports to CMO who I’ve heard speak a few times), I believe they’ve put some real effort into building a sustainable crowd of testers.

2) The VP of Amazon Mechanical Turk basically said, “I don’t think there’s much merit to crowdsourcing companies claiming to ‘test’ people before accessing projects and using that as a quality control measure.” For some reason a company that has absolutely no quality control process in place making that claim smacks of denial and makes Ms. Chiarella sound like she has no idea what’s going on. It just may be that I can say this because I’ve seen the results of testing. I’m such a fan that crowd-focused quality control measures were a focus of my talk!

Their Advice for Companies:

  • Crowdsorucing is about humans (so treat them like that)
  • Crowds are constantly evolving and creating massive data problems (so solve the data problems)
  • Never lose site of the fact that you have two customers: workers and vendors
  • Iterate into Existence: Test, tweak, test, tweak, solution and then scale

Common Myths about Crowdsourcing:

  • It’s easy (it takes a lot of money, a real solution to a real problem and time to develop)
  • Crowdsouring is a “silver bullet” for all problems
  • Crowdsourcing means bad quality
These guys had a bunch of pretty cool graphics put together on the crowdsourcing industry. I really liked how they breakdown the different implementation types into easy-to-digest images.

I say “they” because I have no idea how many people work with or if it’s just Carl Esposti doing all the work.
These guys look like they do a lot of crowdsourcing consultation work–and mostly on the buy side. Esposti’s breakout session was a focus on designing crowdsourcing solutions for companies. We need to be a bit more cosy with him and clarify that our work platform can solve all of the complex workflows he showed (because he said, “Only some of these can be solved with crowdsourcing” which implies that his perception of the technologies is in some way skewed or biased).

Their GM/VP of Sales and Marketing Mark Allen claims they are “crossing the chasm” right now. It still seems like they’re struggling with low-end work and a lack of notable customers to carry the company across the chasm.

Reflections and Speculations

Overall I had a great time getting to spy on my industry cohorts. I certainly made some good contacts that I look forward to fostering in the future. So far as how people are solving real problems, my impression is that crowdsourcing as a business methodology is still all over the place. People are just scratching at the surface of what is the future of labor.

My current attitude is “wow, we are just lightyears ahead of everyone else.” I felt like “crowdsourcing” is still not thinking about labor in the future–or if the company is thinking about labor distribution, they’re seeing themselves as simple market places for labor. A market place for labor feels more like something we’d find back in a Renaissance-Era village than in a post-internet society.

I didn’t see anyone solving the really complex workflows. Most companies seem to skirt this by creating a one-to-one, task-creator to labor-provider model.

This means companies are either making Craigslist 2.0s or creating ill-defined platform plays designed to solve the same damn simple projects we had figured out two years ago–and quickly realized there is NO money in!

What I didn’t see was a company saying, “We built a platform that any company can use to mimic their own organizational structure and tap into other on-demand workflows.” DST was the closest to this vision and he didn’t provide a demo; he showed a picture of their existing workflows (for clients) that meld in-house and crowdsourced labor. Pretty lame but at least they get the trajectory of where labor structures will ultimately arrive.

I can also tell that companies in this space are still struggling. LiveOps and uTest struck me as the most up-beat. That isn’t to say that the other companies aren’t getting deals; I’m sure they are. I just don’t think we’re seeing a group of companies that are expanding quickly–i.e. there’s no run-away company starting to take the lion’s share of the current market (and so likely the lion’s share of the future market).

I’m encouraged that we’re getting a strong base in content creation but it’s clear that we still have a lot of experimenting to do before we’ll arrive at the billion-dollar business. I’m also convicted that when we do arrive there, we’ll have one of the most adaptable, 20th century-resembling labor forces that isn’t just a hodgepodge of “chaotic market place activities” but a truly on-demand labor force ranging up and down the skill spectrum.

Building Crowds: Part 2 of 3

This is the second part of a three-part series on building crowds for crowdsourcing projects.

In Part One, I discussed what I see as the first of three basics steps for starting a crowdsourcing project, focusing on quality control processes and avoiding ambiguity. (Click here to read part-one.) Part Two focuses on interacting with the crowd, how to go about doing it, setting rules and ultimately getting the crowd to the point where it is relatively self-managed. In my personal opinion, getting a crowd to become self-reliant and self-managed should be the goal for any company setting to crowdsource.

Part Two: Create Forums for Discussion

Once you have your quality control processes well defined, it’s time to focus on how you will communicate with the crowd—and how the crowd will communicate with itself.

Email is certainly an option but it doesn’t scale well. There was a recent WSJ article featuring the story of Amazon and Jeff Bezos’ early days that highlights this issue: hundreds of hours spent interacting on a one-to-one basis with customers. This won’t work for crowdsourcing so you shouldn’t even bother approaching crowd management as an “oh, we’ll use email until we figure something better out.”

In crowdsourcing projects, you likely have two sets of customers: (1) the people paying for the work (or contest) and (2) the crowd.

It’s good to spend a lot of personalized time interacting with the customers from set (1) but not quite as important or effective for the group from subset (2).

What you need is a forum, literally, where announcements, answers to important questions and social interactions can take place and project from one person’s inquiry to the rest of the crowd. That is, the responses need to scale and reach the rest of the crowd. Even if you use email to send out one-directional communications (updates you don’t really expect responses to), you will get responses so if it’s possible I would create the forum as the only place where pertinent updates, instructions, etc are communicated to the crowd. It’s just easier.

This seems obvious when said but Servio didn’t use a forum at first. Servio’s crowdsourcing platform is called CloudCrowd, a Facebook application, and so we used Facebook’s built in “wall comments” feature to handle customer (worker) support inquiries. It was massively inefficient and ineffective.

Our VP of Marketing Mark Chatow was largely responsible for crowd management at this stage of the company (I think we were a total of 4 people on the “business side” at this time), and he did his best to keep answers flowing to the crowd. The problem, unfortunately, is that the wall post approach doesn’t work as a knowledge base. It’s too transient. Forums have the built-in archiving so all information is readily available via searches—and building a knowledge repository that persists across weeks and months is super important for building a well-informed, like-minded people (vis a vis project expectations at least).

After using what Facebook provided for about 6 or 9 months, we moved to forums. Servio currently uses Get Satisfaction as our worker support forum. There are a lot of great tools in Get Satisfaction, which is basically a “second generation” community forums tool. Get Satisfaction’s current marketing positioning is primarily focused on customer support for products or services. Not really designed for “crowdsourcing worker support,” there are some minor downsides to Get Satisfaction but nothing to stop me from recommending it as a great forum tool for managing “crowds.”

Notably, Get Satisfaction appears to have a winning market position as identified in its case studies section as well as knowing that companies are starting to use customer feedback to help guide product design&emdash;a trend seeing a renaissance in recent times and with good reason! (If you click through there, note that 70 to 80% of products fail due to product relevance and differentiation, something “crowdsourcing” can help resolve.)

One of the reasons I like Get Satisfaction is that the UI is nice&emdash;much nicer than phpBB for example—but that’s only a small consideration for me.

One of the more challenging questions I’ve faced for the last two years has been, “How can we get the crowd to self-manage itself?” What does the crowd need so that it can become self-sustaining and not suffer a drop in quality (i.e. fall victim to the tyranny, depravity and mediocrity of the masses)?

This is a question the crowdsourcing ‘insiders’ (i.e. the people running businesses based on crowdsourcing) ask themselves frequently. For example, at a Crowdsortium meeting Matt Johnson, CMO of UTest looked at self-managed crowds as a “late stage development” for crowdsourcing models. I’m not sure I entirely agree (it may be a “word choice” dispute) but I see his point: If you’re still struggling to manage the crowd with your own internal resources (likely due to process problems, see part one), how can you expect the crowd to manage itself?

Servio is solving this problem in two ways:

  1. The creation of forums with anointed “Champions,” people who we’ve identified in our crowd as being extremely active in working with us to resolve issues with projects (almost always ambiguity&emdash;again, see part one!), provide quality work and are always respectful to others. I won’t spend too much time on this point but it is absolutely necessary that you squelch troll-activity. Here is an early post showing how we tried to set expectations for forum activity. Note that I was very straight-forward and honest about how I viewed the forum activity lately. I used plain language, was not overly formal and did not call any particular person out. I was general but human (note that if you think you can use “20th Century Marketing-Speak” to manage your crowd you are dead wrong. You need to interact with the crowd in a personalized, friendly, open manner (Or, you will be flanked and the forums will unwind to chaos: see Godwin’s Law). Note
    here that this approach works. When we set the expectations and then interact with the crowd and also promote people to higher levels of “crowd management capabilities,” that the crowd will quickly start self-managing. You can go through Servio’s worker support forums and see the crowd self-managing itself all the time. This is only possible because we’ve been consistent about using the forums to help resolve issues and promote good, reliable people from within the crowd to help us manage everything.
  2. I cannot go into too much detail without divulging trade secrets but Servio uses internal tools to help setup “tiers” of workers. If you look at our project listing page, you’ll see a “Tests” tab where we have credential tests available. When someone takes the “Level One English Editor” test, that test is reviewed by our crowd. It will show up on the “Available Work” tab where other Level One English Editors are able to review the test. If they reject the test, we admins will not likely review the test. It’s been deemed poor quality by our exiting editors who we’ve all personally vetted (and conduct on-going reviews to monitor sustained quality expectations), so why bother looking any further into the “rejected test takers test”? It’s a waste of time. That was our first “breakthrough” for self-policing the crowd. It was a major win, too. After having a forum in place for communicating our expectations (and monitoring and ensuring those standards through our quality control processes), we needed to have a way to scale the crowd without increasing our admin overhead. We’ve expanded on the success of this program by enabling even higher level workers (say “Level Two English Editors”) to conduct a second or third review of tests approved by the Level One Editors. (You won’t be able to see this tab until you are anointed a higher-than-level-one status for any credential but the higher-level credential test reviews takes place on an “Admin” tab also located on our Workspace Platform).

In summary, if you’re working through your checklist of must-haves for building a crowd (for whatever purpose), you now know you need to have clear processes in place, account for quality control before anything else (Part 1) and create an infrastructure through which the crowd can communicate with itself and eventually begin self-monitoring itself (Part 2).

If you’re interested in more information on this topic, I recommend taking a look at one of my earlier posts, “Crowdsourcing (Done Right) is Like an Orchestra.” I will also be speaking on this topic at CrowdConf 2011.

If you have any questions, please leave a comment. I look forward to your feedback. :)

Why I Think About CrowdSourcing at All Times

Some times I wonder if I’m dreadfully boring to others. I find myself staring off into the distance thinking about crowdsoucring, its potential and how I’m using it at work—that and how I should be using crowdsourcing at work. :)

I currently work at Servio, a crowdsourcing company in San Francisco, CA.

Servio provides business services delivered through an online work platform called CloudCrowd. The flexibility of the work platform amazes me daily. We can basically do any kind of online work through CloudCrowd. It’s a blank slate. Sometimes we talk about CloudCrowd in terms of factories and work stations. And in a certain sense, all modern work operates similarly to a factory: Bob does marketing, Jane does sales, work goes from one “hub” to the next, etc, etc. At least I can say I’m lucky enough to have a comfortable chair, desk and general work environment at my factory (not everyone gets to make that claim).

CrowdSourced Labor

CloudCrowd laborers, aka ‘workers,’ work remotely, whether from home, an office or a cafe, which is a nice perk of crowdsourced labor. Ironically enough, I work from an office like many workers do across the world. Proof enough that some things are easier done in person and not online.

One of the key challenges I face daily is how to solve a scaling problem. The contemporary world is built on scale: XYZ-corporation doesn’t just need 10 widgets, it needs 10 million. I help companies reach this goal by designing projects (or workflows) using CloudCrowd. In addition to custom project design, I also help design “stand-by systems” that will provide specific services like, document editing, translation, and content writing. As you can probably imagine, in a world driven by search results, content is the “ring that rules them all” and business large to small are craving massive amounts of content.

So, to answer the question I pose in the title of this post, I think about crowdsourcing non-stop because it’s what I do for a living. Luckily for me, I find it completely riveting. There’s never a dull moment when you’re dealing with factories producing gagillions of lines of content in real time. Some of the major challenges we face at Servio are workflow design, instruction writing and quality control. I plan to spend considerable time writing about those subjects in the future.

Until then, happy crowdsourcing!