Posted: June 26, 2013 Filed under: start up | Tags: A-B testing, innovation, Start-up, testing new idea, user segmentation
I won’t go into how testing concepts prior to investment or full-blown execution is important. You probably know this already or else you would not land here.
What strikes me as odd though is that i have not come across much material about how to go about testing. So here is a short-list that i have to realize summarizes the most common pitfalls:
1. Test the product’s core statement
Focus on the essence of the service. Strip it down from all possible bells and whistles and try to confirm or reject the basic hypothesis about the project.
Essentially try to answer two questions:
a. which problem is the product solving?
b. is this problem important enough for the target group? (to look for it, to return to it, to pay for it)
If either (a) or (b) are not a 100% YES, then you are sure to fail. No doubt about it.
2. Get more educated on the subject
Assuming that (1) is covered then you need to accept that you don’t know as much as you should about the problem you are trying to solve.
For example, let’s say you want to develop a video aggregation service for sports lovers as there is so much sport in the world. And you go through step 1 and the need for a solution is indeed validated. So let’s say that you develop a vision to create an elaborate algorithm to gather data from various sources and incorporate also machine-learning to personalize the service. Stop. First find out what matters most to your target group. What is it that they are lacking now? You should not answer this based on yourself as this would be the equivalent of running a survey with a focus group of one person.
The biggest risk here is to focus on the wrong features, waste time and energy on developing, let’s say, machine learning technology to optimize personalized ranking of videos to find out that users care about something so specific that could be easily pinpointed without the need of developing a self-improving mechanism. Once you research a bit, then you will be in a position to design and build an MVP.
3. Build a prototype (MVP)
In the example above, just do the service manually. Build a tool to facilitate a person to manually place videos in a playlist. You don’t have the time to do it, hire someone. It will cost you far less than building a machine to do it. Also chances are that the person will do the work better than the algorithm will anyway. Do a small ad campaign and direct traffic to a site showing the editor’s playlist of sports videos. Then you study visitors’ behavior: average time on site, number of pageviews per visit, how many visitors actually return, etc. If the concept is interesting it will show some positive signs. In case you believe that personalization is key to the service and the one-editor-for-all approach does not do justice to the concept, then segment users really finely. In the spots video aggregation example, target fans of a particular team in a specific sport. Then the editor should focus on that team, and safely assume that the playlist would be as good as a personalized one for the particular target group.
4. Design a test for maximum knowledge
The test’s goal is to gather knowledge. Aim for that. Let’s assume you want to launch a new service to a userbase you have access to. Don’t send the invite to users randomly. Segment users based on every attribute you have. And do the segmentation separately. One for each attribute. Separate the userbase in three groups based on each of the attributes: low 50%, medium 30% and top 20%. Whatever the results come out to be, you will be able to draw some conclusions about the userbase and the service’s appeal.
5. Change one item at a time
Otherwise you won’t know where to attribute the difference. So, for example, if you are testing how different usergroups behave, you need to make sure that you keep everything the same. Send the same email, on the same day and time, keep the same landing page, etc. If you want to test two different email subject lines, the usergroups need to be the same, etc.
6. Segment before time = 0
Segment users before the time of the test, not after. What not to do: run the test to a random sample and then see who participates and analyze that group. This could lead to a number of problems. For example, the test could alter the segmentation, e.g. trigger visits to the site, and hence shift users from one usergroup to the other. Or high users will most probably end up being under-represented if the sample is randomly selected, and according to the Pareto rule those aree the most important people that one should focus on.
7. Define what you are looking at
Carefully define the metrics you will be looking at after the done is done. Make sure you will get the information you will need. Write it down in a table. Confirm that all placeholders will be filled in with numbers after the test runs.
8. Put down actual values before hand
Take the time to predict what you are expecting to get. This is necessary also to budget and time the test but it will also help you to evaluate the end result. It will show show you where to focus your attention to: where your assumption proved to be far off. Do not put down too much data to look at as you risk losing focus. Keep your eyes on what confirms/ reject the primary hypothesis at the heart of the product’s value.
Harold Camping (pictured in December 2002), predicted doomsday to arrive on the following day, May 21, 2011 (http://goo.gl/QVVqW
9. Be fast.
The faster you design, execute & evaluate a test, the faster you will move on with the next one. The more tests you do, the more educated you become and the better the decisions you make. By working fast you are not being sloppy, you are maximizing the knowledge per time you collect. And it is all about knowledge at this point when you are researching.
10. Look out for interferences
Watch out for outside factors affecting the test. For example, if the testing is done via email then make sure that on that day you don’t also send the weekly newsletter. This is something that could affect behavior in a random manner.
Please share any thoughts or personal experiences of new idea typical pitfalls in the comments section. I would be really interested to find out about them now, before hand.
Posted: February 25, 2013 Filed under: ecommerce, start up
Gus Hansen is maybe the nr1 poker player in the world. But it should not take Gus Hansen to know the first rule of poker: don’t give away the cards you are holding. Keep people guessing instead. Hence the expression poker face. Even better, mislead your opponents to think that you are holding something other to force a wrong decision from their part. This could read like a business book tip. Trick your competition and so on. But when it comes to eBusiness then this is exactly the opposite of what you should be doing. Better lay all your cards on the table and pray for someone to have a look and share an opinion.
To prove my point, let’s take the case of start-ups. Say that it is the year 2005, Facebook does not yet exist and you find yourself in a coffee place and your friend mentions that he is thinking of building a site where visitors could subscribe to create their personal page displaying their picture and a very brief bio. Would you realize they would be talking about the biggest internet phenomenon to date? Would you even grant it as a good idea to start with? Would you feel envy that you did not think of it first? Would you wish he would get you involved? Would you actually expect that person to go ahead and build it in the end? Say there is also another person following the conversation. Would you expect, following this discussion, that few months later there would be two competing Facebook-like sites in the market, your friend’s and the third person’s listening in? What if your friend would go around town and mention that idea to everyone he met? Would you then expect to get a number of Facebook-like sites emerging triggered by your friend pitching idea left and right?
Hell no. One thing is certain to happen, your friend would get great feedback and build on his idea listening to comments and criticism. For example, some person might mention that it would be cool to allow for users to invite friends and get alerts for whenever their friend does anything to his profile. This idea would be gold. Even if 200 Facebook-like sites launched that year, this insight would set it apart. Another person might propose to allow users to upload pictures. Would people really be willing to upload pictures of their personal moments to the wide public? Can you know the answer to this? How could you unless you asked around and got feedback. If all humans had my taste in cuisine, there would only be Italian restaurants and chocolate ice-cream. Tough to predict what people will like and do. Some feedack will be gold, some will be crap, some will just need to be processed. For example, the comment about privacy is valid and could lead you to coming up with limiting access to profiles based on whether they are friends or not. If you don’t do this, you will most certainly get the product wrong and leave room for the next person with a similar idea to capture it. And be certain out of the international online population, that person won’t be too long.
You might be thinking that all this argumentation stands for start-ups and new ideas. How does it apply to running businesses that are already out there? You will be amazed at how much eBusinessmen prefer to keep in the shadows. How many eCommerce businesses share mistakes they have done with social media? Or best practices? How many of them still don’t know how much is right when it comes to investing in Facebook? Quora? Are they even aware of this opportunity? What about mCommerce? Are they correct to not have invested in this or did they waste their money developing a mobile friendly website? Are they really as SE optimized as they should? Are they leveraging email marketing as they should? Are they even segmenting their userbase or is everyone receiving the same email? Still that same message are they doing A-B testing? Is it worth it? Whoever does it won’t admit to it. The rest are to find out about this whole thing on their own.
In the past, a store would open in the high street and would compete with the rest. Now the high street has expanded to include the whole world. The best practice would be for businessmen to group in teams, share all their cards and try to beat the enemies at the gate to make sure they are ahead in the global game as much as possible.
The reason I am making this point is because I am tired of reading articles or going to presentations where businessmen get on stage with the sole intent to sell their company and its products. They don’t aim in sharing the knowledge, letting you in in any secret. This is bad practice that will drag the whole ecosystem behind. Some other group of businesses will outgrow you and you will be left behind. The other alternative is that your ecosystem will outgrow the rest and gradually expand its borders. This is something to aim for.
So you have an idea, you have a good practice or a bad practice to share. Then just share it. Discuss it. Be open about it. It will only benefit everyone around you in the long term. Yourself included.