Description
we discussed the importance of intergroup contact in settings like neighborhoods, public transportation, and sports leagues. All of these studies ensured or inferred that intergroup contact actually took place. However, some researchers have studied the possibility that even imagined contact with a member of an outgroup can change people’s perception of that group. In this Data Exploration assignment we will explore two datasets derived from imagined intergroup contact experiments. In part one, you will look at data from a recent study conducted by Dong Wang, Iain Johnston (a professor in the Harvard Government Department) and Baoyu Wang. Wang et al. (2021) conducted an experiment on a group of Chinese students to determine if imagined social contact could reduce antipathy toward Japanese people. In part two, you will look at the results of our in-class survey, which tested whether imagined social contact with a member of one’s less-preferred political party would change attitudes toward members of that party. You can do either part first, but you will probably find the exercise most valuable if you do some of each part.
If you have a question about any part of this assignment, please ask! Note that the actionable part of each question is bolded.
Part One: Chinese Students and Perception of Japanese People
Data Details:
- File Name: ChinaJapanData.csv
- Source: These data are from (Wang et al. (2021))[https://drive.google.com/open?id=111pbDphCslbMXbmPBwKNQeQ authuser=renos%40g.harvard.edu&usp=drive_fs]. Please take some time to skim this paper in order to get a feel for the population they studied, their key hypotheses, and their experimental procedure. Subjects were asked to imagine a bus ride, either one in which they talked to a Japanese person (treatment) or just enjoyed the scenery (control). They were then asked a series of questions to assess their affective feelings toward Japanese and Chinese people, their perceptions of the characteristics of Japanese and Chinese identity, and demographic, policy, and pschological questions to serve as control variables.
| Variable Name | Variable Description |
| subject | Anonymized identifier for each experimental subject |
| treated | Binary variable equal to TRUE if the subject was told to imagine a bus ride with a Japanese person (the treatment) and FALSE if the subject was told to imagine the scenery on a bus ride (control) |
| JapanPos | Affective feeling about Japanese people ranging from 1 (negative) to 7 (positive) |
| JapanWarm | Affective feeling about Japanese people ranging from 1 (cool) to 7 (warm) |
| Variable Name | Variable Description |
| JapanAdmire | Affective feeling about Japanese people ranging from 1 (loathing) to 7 (admiration) |
| JapanRespect | Affective feeling about Japanese people ranging from 1 (contempt) to 7 (respect) |
| ChinaPos | Affective feeling about Chinese people ranging from 1 (negative) to 7 (positive) |
| ChinaWarm | Affective feeling about Chinese people ranging from 1 (cool) to 7 (warm) |
| ChinaAdmire | Affective feeling about Chinese people ranging from 1 (loathing) to 7 (admiration) |
| ChinaRespect | Affective feeling about Chinese people ranging from 1 (contempt) to 7 (respect) |
| PosDiff | Difference between the Chinese and Japanese positivity score |
| WarmDiff | Difference between the Chinese and Japanese warmth score |
| AdmireDiff | Difference between the Chinese and Japanese admiration score |
| RespectDiff | Difference between the Chinese and Japanese respect score |
| JapanID_avg | Average of 30 ratings of Japanese people on identity trait pairs, coded from 1 to 7 where higher numbers are less favorable; see p. 12 of Wang et al. (2021) for details |
| ChinaID_avg | Average of the same 30 identity ratings of Chinese people |
| ID_diff_avg | Difference between ChinaID_avg and JapanID_avg |
| age | Age in years |
| gender | Gender, coded 1 for male and 0 for female |
| jpfriend | Indicator variable for if subject has a Japanese friend (1) or does not (0) |
| MediaInd | Attitude toward media independence from the government ranging from 1 (strongly oppose) to 5 (strongly support) |
| freetrade | Indicator variable for if subject supports free trade (1) or does not
(0) |
| school_major | Categorical variable denoting major in school; 1 = social sciences, 2 = humanities, 3 = sciences and engineering, 4 = law |
| PrejControl | Motivation to Control Prejudice index; an average of 17 items rated from 1 to 7 in which higher scores denote a greater motivation to control the expression of prejudice |
# load the data ChinaJapan <- read_csv(‘ChinaJapanData.csv’)
Question 1
Part a
When surveys ask a number of questions to try and measure the same underlying concept, it is common to make a summary index by taking the average of all these items. Create new variables for the average affective feeling toward Japanese people, the average affective feeling toward Chinese people, and the average difference between the two.
ChinaJapan <- ChinaJapan %>% mutate(avg_aff_Japanese = (JapanPos + JapanWarm + JapanAdmire + JapanRespect)/4, avg_aff_Chinese = (ChinaPos + ChinaWarm + ChinaAdmire + ChinaRespect)/4) %>%
mutate(avg_diff_aff = avg_aff_Chinese – avg_aff_Japanese)
Part b
For at least one of the individual affect items and all three affect averages you created in part a (China, Japan, and the difference between them), report mean values for the treatment and control groups, an estimate of the difference between those groups, and the results of a test for statistical significance. Did imagined social contact change subjects’ affect toward Japanese people? Chinese people? What about their affective polarization?
| ChinaJapan %>% group_by(treated) %>%
summarize(mean(avg_aff_Chinese), mean(avg_aff_Japanese), mean(avg_diff_aff), mean(PosDiff)) |
## ‘summarise()‘ ungrouping output (override with ‘.groups‘ argument)
## # A tibble: 2 x 5
## treated ‘mean(avg_aff_Chi~ ‘mean(avg_aff_Jap~ ‘mean(avg_diff_~ ‘mean(PosDiff)‘
## <lgl> <dbl> <dbl> <dbl> <dbl>
## 1 FALSE 5.00 3.72 1.28 1.18 ## 2 TRUE 4.89 4.25 0.638 0.417
t.test(ChinaJapan$avg_diff_aff, ChinaJapan$treated)
##
## Welch Two Sample t-test
##
## data: ChinaJapan$avg_diff_aff and ChinaJapan$treated
## t = 3.0069, df = 142.43, p-value = 0.003121
## alternative hypothesis: true difference in means is not equal to 0 ## 95 percent confidence interval: ## 0.1570171 0.7596496 ## sample estimates:
## mean of x mean of y
## 0.9583333 0.5000000
Part c
Researchers often present the size of an experimental effect in terms of Cohen’s D, which calculates the ratio of the treatment effect to the standard deviation. This is a way to understand if a treatment effect is substantively large, in addition to statistically significant. A common “rule of thumb” is that a Cohen’s D score of 0.2 is small, 0.5 is medium, and 0.8 is large. Wang et al. (2021) present Cohen’s D scores in the tables throughout their article.
Here is a useful interactive visualization of Cohen’s D: https://rpsychologist.com/cohend/
Using the cohen.d() function, for each of the variables you used in Part b above, calculate a Cohen’s D and interpret whether it is small, medium, or large. Do these “rule of thumb” interpretations match your intuitive interpretation? Are any of the differences of means stastically significant, yet substantively small according to the Cohen’s D score?
Question 2
Part a
Wang et al. (2021) also investigate whether imagined social contact changes Chinese students’ perception of the characteristics associated with Japanese identiy, as well as the difference in perception of the semantic content associated with Chinese and Japanese identity. They use 30 pairs of opposite phrases (like “frank/hypocritical,” “civilized/barbaric,” and “peace-loving/belligerent”) on scales from 1 to 7 to measure these identity traits (higher scores are less favorable). We’ve provided you with the averages of these thirty items for perception of Chinese and Japanese identites, as well as the average difference between them.
For all three of these indices, report average values for the treatment and control groups, an estimate of the difference between those groups, and the results of a test for statistical significance. How do the experimental effects of imagined contact on identity traits compare to the effects on affect?
Part b
A useful way to visualize these effects of the experimental treatment is with a treatment effects plot. Below, we provide some sample code for how to make a treatment effects plot based on the t-tests you conducted
in part a of this question. Please work through the code to make sure you understand it and interpret the findings depicted in the resulting plot. See p. 2 of Mousa (2020) from this week’s readings for a good example of plots like this.
) # for this v
| # First, store the t-tests as new objects in R
JapanIDdiff <- difference_in_means(JapanID_avg ~ treated, data = ChinaJapan) ChinaIDdiff <- difference_in_means(ChinaID_avg ~ treated, data = ChinaJapan) Diff_in_ID_diff <- difference_in_means(ID_diff_avg ~ treated, data = ChinaJapan) # These objects are called lists; they store values like point estimates and confidence intervals of sta # Next, we extract information from these lists and save them as vectors that you will use to make the d outcomes <- c(‘Japanese Identity Rating’, ‘Chinese Identity Rating’, ‘Identity Difference’ pointests <- c(JapanIDdiff$coefficients, ChinaIDdiff$coefficients, Diff_in_ID_diff$coefficients) lowbounds <- c(JapanIDdiff$conf.low, ChinaIDdiff$conf.low, Diff_in_ID_diff$conf.low) upbounds <- c(JapanIDdiff$conf.high, ChinaIDdiff$conf.high, Diff_in_ID_diff$conf.high) # Combine the vectors into a data frame treatment_effect_plot_data <- tibble(outcomes,pointests,lowbounds,upbounds) # Make the plot ggplot(treatment_effect_plot_data, mapping=aes(x=factor(outcomes, levels = outcomes), y= |
pointests, ymin=
# Why did we use the factor() function for defining the x-axis? If you take the factor() function with t
Part c
Make your own treatment effects plot to visualize the effect of treatment on all four items associated with affect toward Japanese people and the index averaging those items. Interpret your results.
Question 3: Data Science Question
Part a
Pick one of the two average difference variables (AffectDiff_avg or ID_diff_avg) to use as a dependent variable for this question. Then pick at least two of the control variables and write hypotheses about how they would affect your chosen dependent variable. Do this before you do part b of this question.
Part b
Use multiple regression to test your hypotheses. Be sure to include your selected control variables and the treatment variable at a minimum, bu the exact form of the model is up to you. Report and interpret the regression coefficients in the context of your hypotheses.
| mod = lm(avg_diff_aff ~ freetrade + school_major + treated, data = ChinaJapan %>% mutate(school_major = ifelse(school_major %in% c(1, 2), 1,
stargazer(mod, type = “text”) |
0)))
##
## ===============================================
## Dependent variable:
## —————————
## avg_diff_aff
## ———————————————–
| ## freetrade -0.179 | |
| ## ## | (0.260) |
| ## school_major | -0.145 |
| ## ## | (0.292) |
| ## treated | -0.624** |
| ## ## | (0.289) |
| ## Constant | 1.511*** |
| ## | (0.339) |
##
## ———————————————–
## Observations 120
## R2 0.047
## Adjusted R2 0.023
## Residual Std. Error 1.574 (df = 116)
## F Statistic 1.920 (df = 3; 116)
## ===============================================
## Note: *p<0.1; **p<0.05; ***p<0.01
mod = lm(avg_diff_aff ~ gender + jpfriend + treated, data = ChinaJapan) stargazer(mod, type = “text”)
##
## ===============================================
## Dependent variable:
## —————————
## avg_diff_aff
## ———————————————–
## gender -0.051
## (0.288)
##
## jpfriend 0.188
## (0.298)
##
| ## treated | -0.649** |
| ## ## | (0.288) |
| ## Constant | 1.239*** |
| ## | (0.273) |
##
## ———————————————–
## Observations 120
## R2 0.045
## Adjusted R2 0.020
## Residual Std. Error 1.577 (df = 116)
## F Statistic 1.803 (df = 3; 116)
## ===============================================
## Note: *p<0.1; **p<0.05; ***p<0.01
Part c
Coefficient plots can be a good way to display main results from a regression in a way that is visually easier to interpret than a table. They are very similar to the treatment effect plots we made in the previous question.
Make a coefficient plot with the point estimates and 95% confidence intervals for all of your regression coefficients other than the intercept. Be sure to include some kind of line at zero to aid interpretation of statistical significance. Take a look at p. 5 of Brown et al. (2021) from this week’s reading for an example of a coefficient plot (although note that that is plotting one regression coefficient across multiple regression specifications; we’re asking you to plot multiple regression coefficients from one regression model).
| mod = lm(avg_diff_aff ~ freetrade + school_major + treated, data = ChinaJapan %>% mutate(school_major = ifelse(school_major %in% c(1, 2), 1,
dwplot(mod, vline = geom_vline( xintercept = 0, colour = “grey60”, linetype = 2 )) |
0)))
ANSWER: I used the dotwhisker package to do this.
Part Two: Harvard Students and Perception of Members of the Other Party
Data Details:
- File Name: Oct28ClassData.csv
- Source: These data are from the Qualtrics survey you all took last week. About half of you were in the control condition and were simply asked to imagine a bus ride with beautiful scenery; the other half were in the treatment condition and were asked to imagine a bus ride next to a member of your nonpreferred US political party. All students were then asked a series of questions to assess their affective feelings toward Democrats and Republicans and their perceptions of the characteristics of Democratic and Republican identity. Additional control variables were merged from the class background survey.
| Variable Name | Variable Description |
| Treated | Binary variable equal to TRUE if the subject was told to imagine a bus ride with a member of the opposing political party (the treatment) and FALSE if the subject was told to imagine the scenery on a bus ride (control) |
| ClosestParty | Which of the two major US political parties the subject feels closes to |
| Variable Name | Variable Description |
| strongPARTISAN | Binary variable coded as TRUE if subject self-identifies as a strong partisan and FALSE otherwise |
| ControlScenario_1 | First text-based reflection of the imagined bus ride for those in the control condition |
| ControlScenario_2 | Second text-based reflection of the imagined bus ride for those in the control condition |
| ControlScenario_3 | Third text-based reflection of the imagined bus ride for those in the control condition |
| TreatmentScenario_1 | First text-based reflection of the imagined bus ride for those in the treatment condition |
| TreatmentScenario_2 | Second text-based reflection of the imagined bus ride for those in the treatment condition |
| TreatmentScenario_3 | Third text-based reflection of the imagined bus ride for those in the treatment condition |
| RepublicanAffect_1 | Affective feeling about Republicans ranging from 1 (negative) to 7 (positive) |
| RepublicanAffect_2 | Affective feeling about Republicans ranging from 1 (cool) to 7 (warm) |
| RepublicanAffect_3 | Affective feeling about Republicans ranging from 1 (loathing) to 7 (admiration) |
| RepublicanAffect_4 | Affective feeling about Republicans ranging from 1 (contempt) to 7 (respect) |
| DemocraticAffect_1 | Affective feeling about Democrats ranging from 1 (negative) to 7
(positive) |
| DemocraticAffect_2 | Affective feeling about Democrats ranging from 1 (cool) to 7 (warm) |
| DemocraticAffect_3 | Affective feeling about Democrats ranging from 1 (loathing) to 7 (admiration) |
| DemocraticAffect_4 | Affective feeling about Democrats ranging from 1 (contempt) to 7 (respect) |
| RepublicanIdentity_1 | Rating of Republican identity trait from 1 (obstinate) to 7 (open-minded) (note that identity favorability is associated with the higher number, unlike in the data from Part One) |
| RepublicanIdentity_2 | Rating of Republican identity trait from 1 (evil) to 7 (moral) |
| RepublicanIdentity_3 | Rating of Republican identity trait from 1 (arrogant) to 7 (humble) |
| RepublicanIdentity_4 | Rating of Republican identity trait from 1 (cruel) to 7 (kind) |
| DemocraticIdentity_1 | Rating of Democratic identity trait from 1 (obstinate) to 7 (open-minded) |
| DemocraticIdentity_2 | Rating of Democratic identity trait from 1 (evil) to 7 (moral) |
| DemocraticIdentity_3 | Rating of Democratic identity trait from 1 (arrogant) to 7 (humble) |
| DemocraticIdentity_4 | Rating of Democratic identity trait from 1 (cruel) to 7 (kind) |
| gender | Character variable reflecting self-identified gender |
| college_stats | Binary variable coded as TRUE if subject self-identifies as having taken college-level statistics and FALSE otherwise |
| year | Year in college from 1 to 4 |
| US | Binary variable coded as TRUE if subject self-identifies as having been born in the United States and FALSE otherwise |
| InPartyAffect_(1-4) | Affective feelings about the in-party using the same numbering scheme as above |
| InPartyAffect_avg | Average affective feelings about the in-party |
| Variable Name | Variable Description |
| OutPartyAffect_(1-4) | Affective feelings about the out-party using the same numbering scheme as above |
| OutPartyAffect_avg | Average affective feelings about the out-party |
| InPartyIdentity_(1-4) | Identity ratings about the in-party using the same numbering scheme as above |
| InPartyIdentity_avg | Average identity ratings about the in-party |
| OutPartyIdentity_(1-4) | Identity ratings about the out-party using the same numbering scheme as above |
| OutPartyAffect_avg | Average identity ratings about the out-party |
| AffectDiff_(1-4) | Difference in affective feelings between in-party and out-party using the same numbering scheme as above |
| AffectDiff_avg | Average difference in affective feelings between in-party and out-party |
| IdentityDiff_(1-4) | Difference in identity ratings between in-party and out-party using the same numbering scheme as above |
| IdentityDiff_avg | Average difference in identity ratings between in-party and out-party |
# load the data
ClassExperiment <- read_csv(‘Oct28ClassData.csv’)
Question 4
Part a
For at least one of the individual affect items, the in-party and out-party affect averages, and the average affect difference, report average values for the treatment and control groups, an estimate of the difference between those groups, and the results of a test for statistical significance. Did imagined social contact change your classmates’ affect toward members of the opposing party? Their own party? What about their affective polarization?
| ClassExperiment <- ClassExperiment %>% mutate(avg_aff_in_party = ifelse(ClosestParty == “the Republican party”,
(RepublicanAffect_1 + RepublicanAffect_2 + RepublicanAffect_3 + RepublicanAffect_4)/4, (DemocraticAffect_1 + DemocraticAffect_2 + DemocraticAffect_3 + DemocraticAffect_4)/4), avg_aff_out_party = ifelse(ClosestParty == “the Republican party”, (DemocraticAffect_1 + DemocraticAffect_2 + DemocraticAffect_3 + DemocraticAffect_4)/4, (RepublicanAffect_1 + RepublicanAffect_2 + RepublicanAffect_3 + RepublicanAffect_4)/4)) %>% filter(!is.na(avg_aff_in_party), !is.na(avg_aff_out_party)) %>% mutate(avg_diff_aff = avg_aff_in_party – avg_aff_out_party) ClassExperiment %>% group_by(Treated) %>% summarize(mean(avg_aff_in_party), mean(avg_aff_out_party), mean(avg_diff_aff)) |
## # A tibble: 2 x 4
## Treated ‘mean(avg_aff_in_party)‘ ‘mean(avg_aff_out_party)‘ ‘mean(avg_diff_aff~
## <lgl> <dbl> <dbl> <dbl> ## 1 FALSE 5.05 3.11 1.94
## 2 TRUE 4.84 2.98 1.86
t.test(ClassExperiment$avg_diff_aff, ClassExperiment$Treated)
##
## Welch Two Sample t-test
##
## data: ClassExperiment$avg_diff_aff and ClassExperiment$Treated
## t = 6.4763, df = 83.945, p-value = 6.041e-09
## alternative hypothesis: true difference in means is not equal to 0 ## 95 percent confidence interval: ## 0.9516018 1.7949735 ## sample estimates:
## mean of x mean of y
## 1.8938356 0.5205479
Part b
For at least one individual identity trait, the in-party and out-party identity averages, and the average identity difference, report mean values for the treatment and control groups, an estimate of the difference between those groups, and the results of a test for statistical significance. How do the experimental effects of imagined contact on identity traits compare to the effects on affect in the class sample?
Question 5
Compare the results from the class experiment to the results from Wang et al. (2021). What do you hypothesize accounts for similarities or differences in the results?
Question 6: Data Science Question
We have not yet asked you to use the free response data, which takes the form of unstructured text. With this question, we challenge you to find a creative way to use this data. To structure your work, first suggest a hypothesis that could be investigated using the text data and the other data from the experiment. Second, implement a method to use the text data to test this hypothesis. Your method can involve automated or manual processing of the text. You might consider using a function like nchar to characterize the length of response or a package like (stm)[https://cran.rproject.org/web/packages/stm/index.html] to do more sophisticated content analysis.
library(tidytext) library(janeaustenr) austen_books()
## # A tibble: 73,422 x 2
## text book
## * <chr> <fct>
## 1 “SENSE AND SENSIBILITY” Sense & Sensibility
## 2 “” Sense & Sensibility
## 3 “by Jane Austen” Sense & Sensibility
## 4 “” Sense & Sensibility
## 5 “(1811)” Sense & Sensibility ## 6 “” Sense & Sensibility ## 7 “” Sense & Sensibility ## 8 “” Sense & Sensibility ## 9 “” Sense & Sensibility
## 10 “CHAPTER 1” Sense & Sensibility
## # … with 73,412 more rows
| nrc_positive <- get_sentiments(“nrc”) %>% filter(sentiment == “positive”) %>% mutate(positive = 1)
nrc_negative <- get_sentiments(“nrc”) %>% filter(sentiment == “negative”) %>% mutate(positive = 0) sentiments <- nrc_positive %>% rbind(nrc_negative) ClassExperiment.sentiments <- ClassExperiment %>% mutate(participant = row_number()) %>% unite(“text”, ControlScenario_1:`TreatmentScenario _3`, sep = ” “, na.rm = TRUE) %>% unnest_tokens(word, text) %>% inner_join(sentiments, by = “word”) %>% group_by(participant) %>% summarize(treated = mean(Treated), avg_aff_in_party = mean(avg_aff_in_party), avg_aff_out_party = mean(avg_aff_out_party), avg_diff_aff = mean(avg_diff_aff), avg_sentiment = mean(positive)) %>% ungroup() mod = lm(avg_diff_aff ~ avg_sentiment + treated, data = ClassExperiment.sentiments) stargazer(mod, type = “text”) |
##
## ===============================================
## Dependent variable:
## —————————
## avg_diff_aff
## ———————————————–
| ## avg_sentiment 0.816 | |
| ## ## | (0.742) |
| ## treated | -0.245 |
| ## ## | (0.576) |
| ## Constant | 1.395** |
| ## ## | (0.542) |
## ———————————————–
## Observations 47
## R2 0.027 ## Adjusted R2 -0.017
## Residual Std. Error 1.844 (df = 44)
## F Statistic 0.605 (df = 2; 44)
## ===============================================
## Note: *p<0.1; **p<0.05; ***p<0.01
Question 7
Can you glean any additional insights by using the control variables included with the class experiment data? For example, are the results different if you subset to strong partisans or people born in the United States? Be creative.
