Posted: June 26, 2013
I won’t go into how testing concepts prior to investment or full-blown execution is important. You probably know this already or else you would not land here.
What strikes me as odd though is that i have not come across much material about how to go about testing. So here is a short-list that i have to realize summarizes the most common pitfalls:
1. Test the product’s core statement
Focus on the essence of the service. Strip it down from all possible bells and whistles and try to confirm or reject the basic hypothesis about the project.
Essentially try to answer two questions:
a. which problem is the product solving?
b. is this problem important enough for the target group? (to look for it, to return to it, to pay for it)
If either (a) or (b) are not a 100% YES, then you are sure to fail. No doubt about it.
2. Get more educated on the subject
Assuming that (1) is covered then you need to accept that you don’t know as much as you should about the problem you are trying to solve.
For example, let’s say you want to develop a video aggregation service for sports lovers as there is so much sport in the world. And you go through step 1 and the need for a solution is indeed validated. So let’s say that you develop a vision to create an elaborate algorithm to gather data from various sources and incorporate also machine-learning to personalize the service. Stop. First find out what matters most to your target group. What is it that they are lacking now? You should not answer this based on yourself as this would be the equivalent of running a survey with a focus group of one person.
The biggest risk here is to focus on the wrong features, waste time and energy on developing, let’s say, machine learning technology to optimize personalized ranking of videos to find out that users care about something so specific that could be easily pinpointed without the need of developing a self-improving mechanism. Once you research a bit, then you will be in a position to design and build an MVP.
3. Build a prototype (MVP)
In the example above, just do the service manually. Build a tool to facilitate a person to manually place videos in a playlist. You don’t have the time to do it, hire someone. It will cost you far less than building a machine to do it. Also chances are that the person will do the work better than the algorithm will anyway. Do a small ad campaign and direct traffic to a site showing the editor’s playlist of sports videos. Then you study visitors’ behavior: average time on site, number of pageviews per visit, how many visitors actually return, etc. If the concept is interesting it will show some positive signs. In case you believe that personalization is key to the service and the one-editor-for-all approach does not do justice to the concept, then segment users really finely. In the spots video aggregation example, target fans of a particular team in a specific sport. Then the editor should focus on that team, and safely assume that the playlist would be as good as a personalized one for the particular target group.
4. Design a test for maximum knowledge
The test’s goal is to gather knowledge. Aim for that. Let’s assume you want to launch a new service to a userbase you have access to. Don’t send the invite to users randomly. Segment users based on every attribute you have. And do the segmentation separately. One for each attribute. Separate the userbase in three groups based on each of the attributes: low 50%, medium 30% and top 20%. Whatever the results come out to be, you will be able to draw some conclusions about the userbase and the service’s appeal.
5. Change one item at a time
Otherwise you won’t know where to attribute the difference. So, for example, if you are testing how different usergroups behave, you need to make sure that you keep everything the same. Send the same email, on the same day and time, keep the same landing page, etc. If you want to test two different email subject lines, the usergroups need to be the same, etc.
6. Segment before time = 0
Segment users before the time of the test, not after. What not to do: run the test to a random sample and then see who participates and analyze that group. This could lead to a number of problems. For example, the test could alter the segmentation, e.g. trigger visits to the site, and hence shift users from one usergroup to the other. Or high users will most probably end up being under-represented if the sample is randomly selected, and according to the Pareto rule those aree the most important people that one should focus on.
7. Define what you are looking at
Carefully define the metrics you will be looking at after the done is done. Make sure you will get the information you will need. Write it down in a table. Confirm that all placeholders will be filled in with numbers after the test runs.
8. Put down actual values before hand
Take the time to predict what you are expecting to get. This is necessary also to budget and time the test but it will also help you to evaluate the end result. It will show show you where to focus your attention to: where your assumption proved to be far off. Do not put down too much data to look at as you risk losing focus. Keep your eyes on what confirms/ reject the primary hypothesis at the heart of the product’s value.
Harold Camping (pictured in December 2002), predicted doomsday to arrive on the following day, May 21, 2011 (http://goo.gl/QVVqW)
9. Be fast.
The faster you design, execute & evaluate a test, the faster you will move on with the next one. The more tests you do, the more educated you become and the better the decisions you make. By working fast you are not being sloppy, you are maximizing the knowledge per time you collect. And it is all about knowledge at this point when you are researching.
10. Look out for interferences
Watch out for outside factors affecting the test. For example, if the testing is done via email then make sure that on that day you don’t also send the weekly newsletter. This is something that could affect behavior in a random manner.
Please share any thoughts or personal experiences of new idea typical pitfalls in the comments section. I would be really interested to find out about them now, before hand.