Episode 006: This season’s most harmful survey practice

You’ve seen others use it and wonder if you should use it yourself: the technique where you segment or bucket prospects. This practice violates several psychological principles and most are methodologically unsound so that you drive people away, gather misinformation and make some people think poorly of you. Avoid it. Here’s why.

Takeaways:

• As it is popularly used, this practice suffers from methodological errors and violates principles of psychological measurement with the predictable result of lower response rates and uninterpretable results.

• We like to think we are complex and have many layers to us, so don’t then assume others can be simply described by one and only one category. Use surveys to extract the full depth and complexity of those you want to pull into your orbit.

• As I’ve preached before, treat prospects as you would a first date. Would you ask your date a single question (“Are you adventurous, creative, reserved or intelligent? You can only pick one!”) and then design the evening around that bucket? Not unless you want your date to run away, and that’s what your customers are doing.

Transcript

If you’re trying to find customers and fans and clients and people like you, you might be using a technique that is really harming your effort. It’s probably the most harmful, popular practice I’ve seen since the Net Promoter Score.

The Net Promoter Score is that question “Would you recommend our products and services?” and they will give you a 0-10 score. You’ve heard me rant and tell you how harmful that practice is, but this technique is almost as bad. It gives you misinformation, it drives people away, and it just makes you look bad. This techniques is some variation off buckets or segmentation or just asking people. You see this a lot. You go to a web page or somebody sent you a landing page or an opt-in page and there’s a questionnaire. They’ll ask you a question like “Which of these describes you?” or “Which of these is most like you?” and you have to choose one. The idea behind it is when you choose one of these options then we’ll give you a follow up question or it’ll present a certain video or a certain angle based on your response. This has a dozen problems with it and I will mention two today –one methodological problem and one psychological problem why you would never want to use this technique.

I’ve been sent this many times. I’ve just had a colleague send it to me again this past week. He sent me a survey he was going to do. He built a page and clients would go to it and it would say “Which of the following describes you?” and you have to fill out these questions. Hi question was:

Which of the following describes you?

I have a business
I want to have a business but don’t yet
I’m happy with my day job

There are many problems with this. From a methodological perspective: you have overlapping choices and you never want that in any survey question. That’s what happens in this technique. Somebody might read that question, “Which describes you?” and you might say “Well, I am happy with my day job, but I’d like to have a business.” Doesn’t that sound like anybody you know? Of course it does! But that’s too bad because you have to pick one. You either have to have a business or you have to be happy with your day job. When you’re faced with this situation, when you can choose between A or C and just pick one, it’s not accurate. This is misinformation. You don’t have a full picture of what’s going on. The whole point of doing a survey is to get into the heads of your people. To get what they want and need and to get accurate, meaningful information from them; so why would you ask a poorly worded question that would give you inaccurate or incomplete results?

Other people are going to read that question: “Which of the following describes you?” and they’re going to say “Well, none of this sound like me.” And that is a totally logical, appropriate answer and they leave, they don’t respond. This is exactly what we’re finding in the data. It’s turning people off. You’re trying to put people into a bucket, but if they don’t fit the bucket, then they leave; or they choose one of two possible choices. You’re not asking the right questions in the right way and this is how this technique is seen every time I’ve seen it. It’s set up in this way –overlapping categories or categories that are lacking.

People can have different shades of meaning. You might read these answers and think to yourselves it’s totally logical and say “Well, I did start a business but it’s more like a job. You know, it’s systematized. I don’t really have a business. If I ever left, it would vanish, so it’s not really a business. It’s more like a job”. Well, is that choice A or B? It could be both, it could be neither, or some people might read it and say “I have some ideas. I am a consultant. That’s what I do on my day job”. Well, is that A or B or C? It could be all of them, it could be none of them. The point is that you only gave them three choices and they have to pick one so you’re either getting inaccurate information by forcing them to choose or they’re going to leave and that’s the problem. You don’t know. When you get the data, you simply don’t know. You're going to have people that chose C and you’re going to think “Oh they chose C because they’re happy with their day job”, but that’s not what that meant to everybody. Yes, they are happy with their day job, but they’re also starting a new business; but you forced them into one of these buckets. That’s the methodological problem --one of them.

From a psychological problem, we know from identity that we think of ourselves as being very complex, right? Each of us is very complex. There’s many layers to us. We can hold conflicting thoughts in our heads simultaneously, but many people look at others very simply. We put them into categories. We say “Oh that person is a Democrat or a Republican”. “That person is in this group or that group”. Well, it’s one thing to have that bad mindset in your daily life to see yourself as very complex, yet other people are simple categories. It’s another thing to take that bad mindset and put it into practice. You’re trying to bring people into your orbit. You’re trying to find people, students, members, clients and customers to sell your products and services; to get them into your orbit, to be around them. Why would you then use this bucket technique to drive them away; to put them into simple categories? They’re going to recognize that, so from a Psychology perspective, this technique fails.

People don’t want to be put into a bucket. Some of these buckets aren’t going to match. They’re going to leave. That’s what the theory says should happen. That’s what the practice says should happen and by golly, that’s what happens; and that’s what we’re seeing when people are doing launches.

People have done launches for multiple years. This year, they’re using this technique –to ask people this prompt to segment them and they find the response rate has plummeted. Why is that? You’re using this technique; trying to put people into buckets and they’re either being driven away or even you’re getting a response, you can’t say with any confidence that it’s accurate because they might have fit into two categories.

You wouldn’t do this on a date right? You’re on a first date, you’re trying to make your first impression. You wouldn’t pick up your date, bring your date in the car, and then the first question you ask is “Hey! Before we start, would you describe yourself adventurous, creative, reserved or intelligent? You know, I just want to make sure I know who you are, which bucket you fit in. I wouldn’t want to do intelligent activities if you’re really the reserved type. We don’t want to go that path”. That would be a bad move on a date. Your date might think “Gee, I’m intelligent and I’m adventurous. Why can’t I choose both?” More likely, your date is just going to open the door and just run away, which is what your people are doing. They’re running away. They’re faced with this choice. That’s what a bad survey does. It drives people away or it gives misinformation.

Why would you do this when you’re trying to bring people into your orbit and then put them into a bucket? People are more complex than that. They want to share their complexity.

I’ve been conducting surveys and gathering customer feedback for 27 years now. I have reviewed about 7,000 surveys. I have gathered over 8 million data points and I have learned a few things over that time. I have learned that people want to tell you about themselves. They want to describe the fullness and complexity of themselves. They want to give you all the shades of meaning. You’ve got to listen to that. You can’t just ask them to answer a simple bucket question and say “Hey are you A, B or C?” You’ve just sealed off all the information. You’ve proven to them that you don’t care for them to express who they truly are.

Avoid that method. Quit putting people into buckets. It gives you misinformation. It drives your people away. It makes you look bad. Other than that, it’s probably a pretty good technique.

There’s more customer feedback tough love to come. Until then, this is Doc Champagne, helping you to keep your customers forever.

Subscribe to the show: Go to the Apple podcast directory. Be sure to rate the show and give a review. Include your name and website and I will promote it on an upcoming show!

Love to Connect!

LinkedIn: https://www.linkedin.com/in/mattchampagne
Twitter: @Evaluation Guy
Go to: www.matthewchampagne.com/podcasts for more