Max Weylandt.

A null result: induced threat from immigration doesn’t seem to affect efficacy (a fair bit later in the survey)

TL;DR

Ford and Mellon (2020) implemented a survey experiment in the European Social survey that stimulates threat in respondents (their outcome is support for immigration). While their experiment was at the very end of the survey, some respondents did see some other questions as part of a test, responses to which could have presumably been affected by this treatment as well. While a bunch of questions measuring efficacy are statistically significant, the coefficients are basically zero in a practical sense.

Ford and Mellon’s paper

At the end of the European Social Survey in 2014/15, Ford and Mellon (2020) give people one of four immigrant profiles and ask them how many people of that group should be allowed to immigrate to the respondent’s country. The profiles differ on two axes: skill level (‘professionals’ vs ‘unskilled laborers’) and cultural proximity (these depended on country, but were often European vs non-European).

They frame this as the interplay between a skills premium and an ethnic premium. In a related corner of the literature, on group threat, we might talk about realistic threat (perceived threats to people’s economic or physical wellbeing) and symbolic threat (perceived threats to their cultural superiority) (Stephan, Ybarra, and Rios 2015).

Whatever you call it, they find that people prefer skilled migrants, and often (but not always) prefer Europeans over non-Europeans.

I learned about this paper from Claassen and McLaren (2021), which kind of puts forward the idea of correlating other attitudes with this experiment, which I thought was quite clever.

I then learned that this experiment was right at the end of the questionnaire. This makes sense – if seeing these profiles affects how people answer about immigration, it might affect other answers too, and so it’s best not to mess up other questions. Luckily for our purposes here the survey researchers did ask some questions afterwards, because they were testing some question wordings for inclusion in a future round. Not all of them are relevant, but I thought there might be a plausible story where raised level of threat affects people’s efficacy.

For example Enos (2016) shows a link between threat and political participation, and efficacy questions often ask people about how much of a difference their participation makes.

Measures

I had some choosing to do. Because these are test questions,they are asked a bunch of different ways, with respondents assigned one kind of question randomly. I wanted to choose one formulation each from the get go, to avoid (subconsciously) picking the most favourable one as data analysis went on.

I basically decided to use the question format they ended up using in round 8 for the survey – after all, they spent a lot of energy on deciding that one was the optimal way to ask for this attitude.

Data Preparation

I quickly tried to replicate their results, just to make sure that I had done the recoding right.

 data <- read_dta("ESS7e02_2.stata/ESS7e02_2.dta")

data <- data %>% mutate(threat_econ = case_when(
  admaimg == 3 | admaimg == 4 ~ "low-skilled",
  admaimg == 1 | admaimg == 2 ~ "Professional",
  TRUE ~ NA_character_
  ),
  threat_cultural = case_when(
  admaimg == 1 | admaimg == 3 ~ "EU",
  admaimg == 4 | admaimg == 2 ~ ".Non-EU",
  TRUE ~ NA_character_
  )

  )

data <- data %>% mutate(Y = coalesce(alpfpe, alpfpne, allbpe, allbpne))

data <- data %>% mutate(Y_flipped = case_when(
  Y == 4 ~ 1,
  Y == 3 ~ 2,
  Y == 2 ~ 3,
  Y == 1 ~ 4,
  TRUE ~ NA_real_
))

 # Run Replication


mod_pooled_fl <- lm(Y_flipped~  threat_cultural* threat_econ, data=data)

# Make Table

get_estimates(mod_pooled_fl)[,c(1:3,7,9)] %>%
  slice(2, 3:4, 1) %>%
  mutate(term = str_replace(term, "threat_culturalEU", "Ethnic Premium"),
         term = str_replace(term, "threat_econProfessional", "High skill premium"),
         term = str_replace(term, "Ethnic Premium:High skill premium", "Ethnic x high Skill")) %>%  kable(digits=c(0,2,3,2,3,3))

Pooled models: efficacy ~ threat

So I joined in the supplemental data,and estimated five models: for each outcome, a simple linear regression of efficacy on the threat indicators, pooled across all countries.

 # load in supplementary data as data2
data2 <- read_dta("ESS7MTMMe02_1.stata/ESS7MTMMe2_1.dta")
data2 <- essurvey::recode_missings(data2)

#join it to the main survey
joined <- left_join(data, data2, by=c("cntry", "idno"))

# clone response columns to ones with legible names
joined <- joined %>%
  mutate(say_in_gov = testf13,
         influence_pol = testf14,
         pol_care = testf15,
         able_groups = testf16,
         able_particip = testf17)

 mod1 <- lm(say_in_gov ~ threat_cultural* threat_econ, data=joined)

mod2 <- lm(influence_pol ~ threat_cultural* threat_econ, data=joined)

mod3 <- lm(pol_care ~ threat_cultural* threat_econ, data=joined)

mod4 <- lm(able_groups ~ threat_cultural* threat_econ, data=joined)

mod5 <- lm(able_particip ~ threat_cultural* threat_econ, data=joined)



models <- list("System lets us have say"= mod1,
               "Can Influence Politics"= mod2,
               "Politicians Care" = mod3,
               "Able to take active role" = mod4,
               "Able to participate in politics" = mod5)
table_pooled <- sjPlot::tab_model(models,
                  show.ci = FALSE,
                  p.style = "stars")

so, not much going at all. This is the same models with country fixed effects:

 mod1 <- lm(say_in_gov ~ threat_cultural* threat_econ * factor(cntry) , data=joined)

mod2 <- lm(influence_pol ~ threat_cultural* threat_econ * factor(cntry), data=joined)

mod3 <- lm(pol_care ~ threat_cultural* threat_econ * factor(cntry), data=joined)

mod4 <- lm(able_groups ~ threat_cultural* threat_econ * factor(cntry), data=joined)

mod5 <- lm(able_particip ~ threat_cultural* threat_econ * factor(cntry), data=joined)


models <- list("System lets us have say"= mod1,
               "Can Influence Politics"= mod2,
               "Politicians Care" = mod3,
               "Able to take active role" = mod4,
               "Able to participate in politics" = mod5)
table_fe <-sjPlot::tab_model(models,
                  show.ci = FALSE,
                  p.style = "stars")

Individual Country Regressions

Ok, so there are a few results significant at the .05 level – still not much to write home about, in my opinion. Here are some regressions of the efficacy outcomes on the interaction of the two threats for each country, only showing the coefficients for culture.

 countries <- unique(joined$cntry)

variables <- c("say_in_gov", "influence_pol", "pol_care", "able_groups", "able_particip")

outcomes <- data.frame(country_test = countries,
                       country=NA,
                       coef = NA,
                       se = NA,
                       ci_low = NA,
                       ci_high = NA )

# remove SI and FI
# those who got EU in SI or Professional in FI didn't get
# asked the outcome variables (or they are NA for another reason)

countries <- countries[-c(9,21)]

graphs <- list()

for (v in 1:length(variables)) {

  for (i in 1:length(countries)) {
    data_filtered <- joined %>% filter(cntry==countries[i])

    mod <- lm(as.formula(paste(variables[v], "~ threat_cultural * threat_econ")) , data=data_filtered)

    estimates <- get_estimates(mod)

    outcomes$country[i] = countries[i]
    outcomes$coef[i]    = estimates$estimate[2]
    outcomes$se[i]      = estimates$std.error[2]
    outcomes$ci_low[i]  = estimates$conf.low[2]
    outcomes$ci_high[i] = estimates$conf.high[2]


    p <- ggplot(outcomes) +
      aes(x=reorder(country,coef), y=coef) +
      geom_point() +
      geom_segment(aes(x=country,xend=country, y=ci_low, yend=ci_high   ))+
      geom_hline(yintercept=0, color="grey")+
      labs(x = "Country", y="Effect of Cultural Threat", title = variables[v])+
      theme_bw()+
      theme(text = element_text(size=8))+
      ylim(c(-0.5, 1))

    graphs[[v]] <- p

  }
}

graphs[[1]] + graphs[[2]] + graphs[[3]]
graphs[[5]] + graphs[[4]]

As far as I’m concerned there’s nothing here. Yes, some countries are sometimes significant on some measures, but these are a lot of regressions, and we’d expect a few things to be significant just by chance. There is no pattern here I can spot that makes me want to dig deeper.

Placebo Analysis

Because I had planned to do so anyways, I ran a placebo test where I regressed other survey questions that should be unrelated to the treatment, because they came before the treatment. Indeed the effects are null.

Conclusion

So, I didn’t find any effects of this threat treatment on a bunch of efficacy measures. These efficacy measures did not resoundingly indicate null results like the placebos did, but there was nothing of practical significance in the results that seemed noteworthy. Again, testing several measures on this many countries, with a large sample of respondents, is bound to produce some tables with stars — but that doesn’t meet the kind of threshold we should have.

This was a very brief analysis and there are lots of other things you could do, including looking at the people who were asked other versions of the efficacy questions. After all, I only looked at the subset that was randomly assigned to see these particular formulations. I’m sure there’s also lots you could do on the modeling side. And, for sure, there was quite the gap between the treatment and when they were asked about efficacy (there was lots of demographic info in the middle). So maybe this relationship does exist in real life. But in this dataset? I doubt it. I tend to think that if the effect is there, a simple analysis should find it.

Cited