Max Weylandt.

← Back

Getting Longer Responses to Open-Ended Questions

7 September, 2024

This blog summarizes a paper I wrote during the course of my PhD. You can see a version of that paper here

All treatments, seen at once. Most respondents saw only some of these elements.

When people take surveys, they usually have to squeeze their opinions into one of the given options of a multiple-choice setup, which is often a bad fit. Open-ended survey questions are seriously interesting: as researchers, we get much more detail, and even answers we didn’t anticipate at all.

Methods like topic models or embedding regression make it feasible to analyze open-ended surveys at a much larger scale than before, but they do require lots of data — a two-word answer doesn’t give a topic model much information to use when deciding what topic the answer is about, for example.

But people who take surveys don’t want to give long answers, as a rule — they are busy, and thinking about something in detail and typing it out takes a lot of effort. Satisficing theory says that three factors affect how much effort people make when answering a question: how difficult it is, how able the respondent is, and how motivated the respondent is.

I ran an experiment to see whether I could motivate people to write more. The image above captures all of the things I tried:

  1. I promised them more entries in a lottery for real money if they wrote more
  2. I asked them to hit a certain target and gave them a word counter
  3. A progress bar that changes color or
  4. one in the form of a dog running toward a bowl of food.

In reality, people saw one of any possible combination of those four elements. In the end, 1. and 2. really worked: they both doubled the amount that people wrote! People who didn’t see any treatment wrote 20 words on average — those who learned about the lottery or were asked for certain word counts wrote 25 and 19 words more than that, respectively.

Responses visualized. Note people often saw several treatments at once, which is why some dots (observations) appear more than once
Treatment Estimate SE p 25% 75%
color -1.96 3.01 0.51 -7.85 3.93
dog -1.83 3.01 0.54 -7.72 4.06
lottery 25.84 3.18 0.00 19.61 32.07
word 19.12 2.96 0.00 13.32 24.92

Average Marginal Effects of Treaments from regression model that regressed length of response on all treatments and all their interactions

If you ask me, I think the lottery approach is inferior, even if it gets us a few words more. Incentivizing more responses with money can be ethically dubious, and asking people for a word count gave similar-enough results without having to worry about that. Either way though, that’s a pretty easy way to get more responses!

But when people write these longer answers, do they actually give us more content or do they just waffle on to hit the target? As far as I can tell, there are no serious drawbacks. Yes, there were one or two overly long off-topic ones. But several measures of quality returned no statistically significant differences between those incentivized to write more and those who weren’t. And responses in the lottery and word count treatments contained just under one topic more, by my count.

Still, I would like someone to do a more thorough look at what we gain from these longer responses — do they shift the substantive results we get? Do they reveal a different set of preferences, or a different intensity of beliefs?


← Back ↑ Top