Bias can sneak up in the most unexpected ways. Everyone has their individual preferences and opinions, and as much as we might not want them to, those can leak into our professional lives. Poorly constructed survey questions are an example of how personal bias can leak into our work, causing us to create questions that reflect a bias in some way.
What we’re talking about here is leading and loaded questions. If you’re not aware of your biases when you’re creating forms, these questions could sneak into your surveys and influence your users into filling out answers that don’t accurately reflect their belief. Tainted, half-true data isn’t helpful for you or the people you’re sharing it with.
Leading vs. Loaded Questions
What exactly are leading and loaded questions? There are several definitions and different understandings of these questions, but here’s one way to understand the difference between the two.
Leading questions are intended to lead people to answer questions in a specific way based on how they’re phrased. Often they contain information that they want confirmed rather than a question that tries to get at the true answer.
Here’s a real-life example of a leading question. In a 1970 Virginia Slims poll, people were surveyed on their thoughts about women in leadership, including this question:
“There won’t be a woman President of the United States for a long time and that’s probably just as well.”
It’s very clear how the survey creators intended this question to be answered. In addition to being leading, this question is flawed in other ways, too. This question really contains two questions, each of which could have a different answer from the same person. For example, someone could agree that there wouldn’t be a female president for a while, but not agree that it was for the best. The construction of the question forces respondents to answer yes to both statements or no to both statements.
Another example of a leading question could be asking form respondents something like “Do you love our amazing support team?” With this kind of phrasing, it feels harsh to answer anything other than yes. Instead, ask something along the lines of “How would you rate the performance of our support team on a scale of 1 to 10?”
Loaded questions are similar to leading questions in that they subtly (or not so subtly) push the user toward a particular response. The defining feature of this question type is the assumption about the respondent that is included implicitly in the question.
Loaded questions can seem pretty benign at a first glance. Robin Pearl of Estee Lauder noted in a 2008 Marketing Research Association conference talk that questions beginning with the query “What do you love about…” can be problematic. (Note: The source article for this question calls it leading question, but we would call it a loaded question because of the assumptions it makes about the person you’re asking it of.)
This question could be asked about a lot of things: a product, a person, a business. The problem with it is that it assumes that your user loves whatever you’re asking them about. Maybe all you’re looking for in these cases is positive answers, but if you want honest feedback, this might not be the best way to phrase this question.
Leading and loaded questions have small differences, but it’s important to remember that they are both ways of confusing, misleading, or trying to influence users into making a particular selection. Sometimes they’re created deliberately, other times they’re unconscious. In nearly all cases, it’s possible to modify them to present better options to users and get more accurate results in return.
Words to Avoid
Sometimes the problem with a question is that it contains loaded words: words overcharged with negative or positive emotion or words that imply a bias toward the question. Using absolute words, according to FluidSurveys, puts your users in a difficult position, forcing them to think in black and white terms. Likewise, using strong, emotionally charged verbs and adjectives can influence the way your users think about an issue.
How to Avoid Flawed, Biased Questions in Your Forms
When you’re creating forms of any type, take these steps to make sure you’re not using leading or loaded questions and that you’re respecting the intelligence of your users.
1. Look at your questions and ask yourself if there’s a particular way you want a question to be answered or if there’s a certain type of response you’re expecting.
Solution: Re-word questions to focus on all options; don’t just ask readers to confirm something you believe to be true.
2. Look at the words you’re using. Are you describing something in a biased way?
Solution: Remove biased language and describe options using clear, to-the-point phrasing. Don’t suggest in any way that one response is better than another.
3. Examine whether the questions you’re asking require users to give an answer that doesn’t completely represent their response.
Solution: Separate out any grouped questions and clarify any user characteristics before you make an assumption within a question.
There are lots of ethical issues related to form building and we’re aiming to explore as many as we can in our form ethics series. Read the previous post in this series, or connect with us on Twitter to let us know what other topics you’d like to learn more about!