Posted: June 26, 2013 Filed under: start up | Tags: A-B testing, innovation, Start-up, testing new idea, user segmentation
I won’t go into how testing concepts prior to investment or full-blown execution is important. You probably know this already or else you would not land here.
What strikes me as odd though is that i have not come across much material about how to go about testing. So here is a short-list that i have to realize summarizes the most common pitfalls:
1. Test the product’s core statement
Focus on the essence of the service. Strip it down from all possible bells and whistles and try to confirm or reject the basic hypothesis about the project.
Essentially try to answer two questions:
a. which problem is the product solving?
b. is this problem important enough for the target group? (to look for it, to return to it, to pay for it)
If either (a) or (b) are not a 100% YES, then you are sure to fail. No doubt about it.
2. Get more educated on the subject
Assuming that (1) is covered then you need to accept that you don’t know as much as you should about the problem you are trying to solve.
For example, let’s say you want to develop a video aggregation service for sports lovers as there is so much sport in the world. And you go through step 1 and the need for a solution is indeed validated. So let’s say that you develop a vision to create an elaborate algorithm to gather data from various sources and incorporate also machine-learning to personalize the service. Stop. First find out what matters most to your target group. What is it that they are lacking now? You should not answer this based on yourself as this would be the equivalent of running a survey with a focus group of one person.
The biggest risk here is to focus on the wrong features, waste time and energy on developing, let’s say, machine learning technology to optimize personalized ranking of videos to find out that users care about something so specific that could be easily pinpointed without the need of developing a self-improving mechanism. Once you research a bit, then you will be in a position to design and build an MVP.
3. Build a prototype (MVP)
In the example above, just do the service manually. Build a tool to facilitate a person to manually place videos in a playlist. You don’t have the time to do it, hire someone. It will cost you far less than building a machine to do it. Also chances are that the person will do the work better than the algorithm will anyway. Do a small ad campaign and direct traffic to a site showing the editor’s playlist of sports videos. Then you study visitors’ behavior: average time on site, number of pageviews per visit, how many visitors actually return, etc. If the concept is interesting it will show some positive signs. In case you believe that personalization is key to the service and the one-editor-for-all approach does not do justice to the concept, then segment users really finely. In the spots video aggregation example, target fans of a particular team in a specific sport. Then the editor should focus on that team, and safely assume that the playlist would be as good as a personalized one for the particular target group.
4. Design a test for maximum knowledge
The test’s goal is to gather knowledge. Aim for that. Let’s assume you want to launch a new service to a userbase you have access to. Don’t send the invite to users randomly. Segment users based on every attribute you have. And do the segmentation separately. One for each attribute. Separate the userbase in three groups based on each of the attributes: low 50%, medium 30% and top 20%. Whatever the results come out to be, you will be able to draw some conclusions about the userbase and the service’s appeal.
5. Change one item at a time
Otherwise you won’t know where to attribute the difference. So, for example, if you are testing how different usergroups behave, you need to make sure that you keep everything the same. Send the same email, on the same day and time, keep the same landing page, etc. If you want to test two different email subject lines, the usergroups need to be the same, etc.
6. Segment before time = 0
Segment users before the time of the test, not after. What not to do: run the test to a random sample and then see who participates and analyze that group. This could lead to a number of problems. For example, the test could alter the segmentation, e.g. trigger visits to the site, and hence shift users from one usergroup to the other. Or high users will most probably end up being under-represented if the sample is randomly selected, and according to the Pareto rule those aree the most important people that one should focus on.
7. Define what you are looking at
Carefully define the metrics you will be looking at after the done is done. Make sure you will get the information you will need. Write it down in a table. Confirm that all placeholders will be filled in with numbers after the test runs.
8. Put down actual values before hand
Take the time to predict what you are expecting to get. This is necessary also to budget and time the test but it will also help you to evaluate the end result. It will show show you where to focus your attention to: where your assumption proved to be far off. Do not put down too much data to look at as you risk losing focus. Keep your eyes on what confirms/ reject the primary hypothesis at the heart of the product’s value.
Harold Camping (pictured in December 2002), predicted doomsday to arrive on the following day, May 21, 2011 (http://goo.gl/QVVqW
9. Be fast.
The faster you design, execute & evaluate a test, the faster you will move on with the next one. The more tests you do, the more educated you become and the better the decisions you make. By working fast you are not being sloppy, you are maximizing the knowledge per time you collect. And it is all about knowledge at this point when you are researching.
10. Look out for interferences
Watch out for outside factors affecting the test. For example, if the testing is done via email then make sure that on that day you don’t also send the weekly newsletter. This is something that could affect behavior in a random manner.
Please share any thoughts or personal experiences of new idea typical pitfalls in the comments section. I would be really interested to find out about them now, before hand.
Posted: May 24, 2013 Filed under: Uncategorized | Tags: email marketing, personalization, user segmentation
If you search for 1 to 1 marketing you will get results such as personalization, CRM technologies, userbase segmentation and so on. Before you go there and invest in a software or recruit an agency, first identify the principle behind all of those buzzwords and try applying it in your email marketing.
The juice: make the recipient feel like this message is intended to him personally rather than give him the i-am-bulking-everyone-i-have-access-to impression. The more relevant the message, the higher the probability that the recipient will become engaged upon receiving it. That’s the juice.
How do you get recipients to get this feeling? Here a few examples:
-make the content of the message as relevant to the recipient as possible. One way to achieve this is through userbase segmentation. For example, segment in a list users that purchased a specific product and prepare a custom message for them only. Or make a list of customers that spent more than 100 Euros the past 15 days.
-be explicit in what you are doing. If you did segment your users and you are now addressing a specific segment of those, tell them so. And do so as early in the message a possible (see benefits of cutting to the chase). For example, “You have spent more than 100 Euros the past 15 days and that places you in our Golden Club and eligible for a unique offer…”.
-make copy personalized. The trick here is to make all copy seem as if it is written for this recipient only. This does not mean you need to change your brand’s tone of voice, no just imitate how a company representative would address a customer. To go a step further, you can make the message more personalized by addressing the recipient by his or her name (most email marketing software allow this functionality), explicitly explain why the message is relevant to the recipient, or even better make evident how this message is exclusive for the particular recipient.
-be polite. In 1-to-1 relationships manners matter more. Some examples: don’t try to trick them, don’t make unsubscribing difficult, don’t enroll them in the mailing list without their consent, don’t lie, etc.
-do not treat all customers equal. When I walk in a store and I am a regular, I expect to be treated as one. Similarly if I am not a regular, too much intimacy might be annoying. How would that translate in email marketing world? Segment your users in terms of worth, say gold, silver and bronze, and then treat each group differently. What could differ? The contact frequency first of all, the tone of voice, the quality of the offers, anything that signifies preferential treatment in your business
I believe the main pillars on how to make recipients feel special are here. Do all of the above and you will get 90% of the benefits a 5,000 Euro/ month CRM system could offer you. Have you found another trick in order to make recipients feel special? Please write a comment about it.