Crowdsourced QA services: benefit or detriment?
Remember the chilling chant in the Batman film “The Dark Knight Rises”? Did you know that the chant was a product of a crowdsourcing exercise by the composer and musician, Hans Zimmer? Thousands of Batman fans in 107 countries around the world uploaded the chant in their own voices, which Zimmer then mixed together to build his massive worldwide choir for the final movie score. Nothing like movie bragging rights to get a fandom involved in crowdsourcing! Thus starts the topic: crowdsourced QA services…
However, crowdsourcing has become something more than tapping into an eager group of fans and it’s been around longer than you may realize. According to an eYeka study, 85% of the best global brands have been engaging in crowdsourcing activities in the last ten years. Gartner, Inc. anticipates that by 2018, 75% of the world’s high performing enterprises will be using crowdsourcing. It goes without saying that crowdsourcing has been expanded into the software industry as well, particularly on the quality assurance (QA) front.
Crowdsourced testing provides a unique way to have real world users test your software under real-world conditions. The practice has matured a lot and, with a lot of tech innovations in this field, it’s growing into a good business offering as well. Crowdsourcing has come a long way from being a bunch of enthusiastic folks with certain profile to test a software. It is now an elaborate and scalable setup that provides real world environments, spanning across devices, operating systems, browsers, etc., and human testers to run test cases written by you. Anytime, anywhere.
Myth Vs Truth
Myth: Crowdsourced testers are just another group of (motivated??) monkey testers
Truth: There are a number of reputable and organized crowdsourced QA platforms/organisations such as: RainforestQA, Usertesting, MyCrowd, PassBrains, Qualitrix. These platforms generally have a large groups of trained, experienced, and well rated testers. Some of them have machine learning algorithms to weed out erroneous test results as well. These organizations are serious players, investing strong tech and human resources, and they’re good enough to add value to the QA process for any software development process.
Myth: Crowdsourced testing is panacea for all quality blues
Truth: No single solution is going to solve all your quality issues. Leveraging a crowdsourcing platform for repetitive QA tasks can help teams use their testing resources more efficiently. Another benefit to couroudsourced testing is for teams that have irregular testing needs. By crowdsourcing their testing, teams can easily scale their testing capacity before big releases or other significant events. This allows them to maintain the minimum in-house QA resources full-time, while having the assurance that their testing force can expand when the need arises. In short, assess your needs, and quality test appropriately.
Myth: Crowdsourcing only provides exploratory testing
Truth: Crowdsourced testing groups are not just limited to exploratory testing services. Most of them enable you to create entire regression suites, integrated with a continuous integration setup.
Why use Crowdsourced QA?
- Human Element: Software is becoming more and more interactive with the user, so it makes sense to let a human test something that was designed for another human being.
- Access to Range of Devices: – On some of the crowdsourcing platforms, you can just select the browsers or devices you need to run yours tests on.
- Budget: Startups and growing firms run on a frugal budget. Maintaining a large test engineering infrastructure can turn out to be resource intensive.
- Workload Relief: By outsourcing the management of the test case to a crowdsourcing platform, you can focus on writing quality test cases, rather than how to run them.
- Availability: With users around the world, many platforms have 24/7 availability to execute tests.
- Local testing: When launching a new product for different locales, crowdsourcing can give the ability to reach all those varied users.
- Test cases/instructions have to be very concise: Since these instructions may be consumed by a crowd on a different corner of the globe, nothing should be left to interpretation. This calls for writing objective test cases/instructions/requirements and revising them where required so that they are easily understood by anyone with basic English comprehension.
- Trust issues and confidentiality: Even with security controls, there are risks that come with exposing sensitive intellectual property and high value content to crowdsourcing platforms. Be sure to outline the confidentiality expectations for your users and also consider whether it is worth the risk.
- Debugging and bug reproduction: Some of the environment and load specific defects may prove harder to reproduce, so good audit and debug logging provisions should be coded in and enabled.
- Revisions: After your crowdsourced exploratory testing of a software product, you may end up with lots of reports of small issues. This can lead to a lengthy triage process afterwards and needs to be accounted for in project planning.
- Migration: Not all platforms are alike, so it can be challenging and time-consuming to migrate from one crowdsourced platform to another.
- Dedicated QA person: When there isn’t a dedicated champion for the project, it can be difficult to refine future runs based on new learnings. A dedicated quality assurance individual is needed to oversee the whole exercise.
- Accountability and motivation: An in-house test engineering team may be [more/less accountable and/or motivated than] outsourcing partner.
How to evaluate a crowdsourcing testing partner/platform
- Quality, capabilities (device inventory, environments on offer etc), scale and ease of use of the platform.
- Platform integration with any test/bug/project management tool used within your team.
- Turnaround time for test execution cycles.
- Confidentiality clauses
- Availability of a quick response support team
- Global outreach to ensure a steady number of testers available
- The organization is full scale company not just an intermediary between the client and the testers
- Flexible pricing model
“Synerzip team is very responsive & quick to adopt new technologies. Team naturally follows best practices, does peer reviews and delivers quality output, thus exceeding client expectations.”
“Synerzip’s agile processes & daily scrums were very valuable, made communication & time zone issues work out successfully.”
“Synerzip’s flexible and responsible team grew to be an extension to the StepOne team. Typical concerns of time zone issues did not exist with Synerzip team.”
“Synerzip worked in perfect textbook Agile fashion – releasing working demos every two weeks. Though aggressive schedules, Synerzip was able to deliver a working product in 90 days, which helped Zimbra stand by their commitment to their customers.”
“Outstanding product delivery and exceptional project management, comes from DNA of Synerzip.”
“Studer product has practically taken a 180% turn from what it was, before Synerzip came in. Synerzip cost is very reasonable as compared to the work they do.”
“Synerzip makes the timezone differences work FOR the customer, enabling a positive experience for us. ‘Seeing is believing’, so we decided to give it a shot and the project was very successful.”
“The Synerzip team seamlessly integrates with our team. We started seeing results within the first sprint. And due to the team’s responsiveness, we were able to get our product to the sales cycle within 7 months.”
“Product management team from Synerzip is exceptional and has a clear understanding of Studer’s needs. Synerzip team gives consistent performance and never misses a deadline.”
“Synerzip is different because of the quality of their leadership, efficient team and clearly set methodologies. Studer gets high level of confidence from Synerzip along with significant cost advantage of almost 50%”
“Synerzip’s hiring approach and practices are worth applauding. Working with Synerzip is like
“What you see is what you get”.”
“Synerzip has dedicated experts for every area. Synerzip helped Tangoe save a lot of cost, still giving a very high quality product.”
“Synerzip gives tremendous cost advantage in terms of hiring and growing the team to be productive verses a readymade team. Synerzip is one company that delivers “co –development” to the core!”
“Synerzip is a great company to work with. Good leadership and a warm, welcoming attitude of the team are additional plus points.”
“Our relationship with Synerzip is very collaborative, and they are our true partners as our values match with theirs.”
“Synerzip has proven to be a great software product co-development partner. It is a leader because of its great culture, its history, and its employee retention policies. ExamSoft’s clients are happy with the product, and that’s how ExamSoft measures that all is going well.”
“They possess a great technical acumen with a burning desire to solve problems. The team always takes the initiative and ownership in all the processes they follow. Synerzip has played a vital role in our scaling up and was a perfect partner in cost, efficiency, and schedules.”
“As we are a startup, things change on a weekly basis, but Synerzip team has been flexible in adapting the same”
“Synerzip team has been very proactive in building the best quality software, bringing in best practices, and cutting edge innovation for our company.”
“We’ve been working for more than six years with Synerzip and its one of the better, if not the best, experience I’ve had working with an outsourcing company.”
“My experience with Synerzip is that they have the talent. You throw a problem at them, and someone from that team helps to solve the issue.”
“The breadth and depth of technical abilities that Synerzip brings on the table and the UX work done by them for this project exceeded my expectations!”
“Synerzip UX designers very closely represent their counterparts in the US in terms of their practice, how they tackle problems, and how they evangelize the value of UX.”
“Synerzip team understood the requirements well and documented them to make sure they understood them rightly.”
“Synerzip is definitely not a typical offshore company. Synerzip team is incredibly communicative, agile, and delivers on its commitments.”
“Working with Synerzip helped us accelerate our roadmap in ways we never thought possible!”
“While working with Synerzip, I get a feeling of working with a huge community of resources, who can jump in with the skills as needed.”