Will people like my new product idea? What do customers like or dislike about their current experience? How can I improve on my existing product? These are all great questions, which are hard to come by through secondary research.
In my previous article, I discussed cheap and inexpensive ways to conduct secondary research (research that someone has already collected for you). Unfortunately, many times you can’t get the exact questions answered that you need, so you have to get the feedback yourself. In the discipline of consumer behavior this is called primary research.
If My Answers Frighten You…
As any attorney will tell you, getting good answers requires asking good questions. In my first article on this topic, I talked about the importance asking the right business question. In this article I will discuss the necessity of asking the right survey questions.
Writing survey questions seems like a straightforward endeavor, but there are head-smacking mistakes you can make that can’t be corrected after the data is collected. By way of illustration I tried to write the World’s Worst Survey question as an example. Let’s take a look.
Dissecting the World’s Worst Survey Question
While somewhat farcical, I have seen questions not far off from this. So let’s dissect this atrocity one piece at a time.
The first problem is — it’s a loaded question. Much like a lawyer asking Colonel Mustard, “…so where were you after you killed Professor Plumb with a wrench in the study?” on the witness stand, it would be bounced out of court before the good Colonel could open his mouth. It’s a loaded question. The question assumes that the respondent buys suntan oil. In the northern latitudes that is a rare purchase, and for many, not a purchase at all!
Next, I have conflated price and value for the money. They are related but distinct concepts. You must tease the concepts apart, otherwise neither you nor the respondent will have a clear idea of what you are evaluating.
In addition to mixing price and value together, we have no reference point for price. How much is it? In addition, offering up a price with no reference point challenges the consumer to make any judgment. So when looking at price specifically, it should have other price points for customers to evaluate. I don’t know about you, but I don’t regularly keep track of suntan oil prices.
The larger issue, in general, is that asking pricing questions is tricky business. No one likes to spend money, so there is always an inherent bias around the importance of price that drives imprecise results. When possible, it is best to get at the pricing issue through indirect methods such a choice studies. People are just not honest when it comes to “how much would you pay” and other direct pricing-type questions. It has low predictive validity.
Next we turn to the scaling. First, the scale is reversed with the good rating (exceeds my expectations) on the left while the poor rating (below my expectation) is on the right. While a subject of some debate, most researchers would agree that a low to high order is preferred. Why? Almost everything in the Western world is righty-tighty, lefty-loosey. That is: higher is always on the right, and lower is always on the left. It makes it easier for the respondent and will ensure more accurate measurements.
Which brings me to the issue of scaling. Expectations scales are notoriously poor in the measurement of evaluative attitudes. The reason being is that the starting point is a mystery. For example, say you expected this article to stink, but you feel that it is merely mediocre. I have exceeded your expectations! Does that mean it’s good? No, sadly it does not.
Finally, I can fill a four drawer metal file cabinet with articles about what the right number of points on a scale might be. Let me save you some time and summarize the outcome of this corpus of research. The answer is: it depends. The considerations are: the level of involvement the respondent has with the issue in question, the levels of differentiation (also known as just noticeable differences) that can be detected, and the goal of the research in the first place.
If I am conducting a compliance study and want to know the cleanliness of the bathrooms, I might ask “Were the restrooms clean?” with the response scale being “☐ no ☐ yes.”
In the case of taste testing a new Central Coast Pinot Noir with sommeliers, I might ask “How dry would you evaluate this Central Coast Pinot Noir?” and might provide a 10-hedonic-point scale anchored by “Extremely Dry” to “Extremely Sweet” with several other anchor points in between. Experts who are highly engaged in a product category will see shades of distinction other might not.
Reconstructing the World’s Worst Survey Question
So let’s fix my World’s Worst Survey question. Let’s see if we can resurrect the intent in a more accurate form..
As I mentioned, the pricing question is a whole different kettle of fish where we can use a within or between experimental design. Alternatively, there is a class of analytics called Choice Models or Discrete Choice models, which utilize techniques such a Conjoint Analysis to dampen unwanted bias in the data when it comes to understanding price elasticities. Also, MaxDiff can be adopted for this use which is easier to deploy and analyze. However, all that is a topic for another day.
So as you can see, writing a survey can seem simple — but you will want to avoid common mistakes. There are several good books on the topics, with alluring titles such as Improving Survey Questions by Floyd Fowler and Internal, Mail, and Mixed-Mode Surveys, which is a classic by Don Dillman and Jolene Smyth. Both are excellent sources. Of course you always contact me. I will be sure to exceed your expectations.
