Hypothesis Testing 2

Associations between Variables

April 23, 2024

Hypotheses


  • Null hypothesis: there is no relationship between treatment and outcome, the difference is due to chance
  • Alternative hypothesis: there is a relationship, the difference is not due to chance

Approach


  • Under the null hypothesis, treatment has NO impact on y (the outcome)
  • This means that if we were to change the values of the treatment variable, the values on y would stay the same

Approach

  • So…we can simulate the null distribution by:
    • Reshuffling the treatment variable
    • Calculating the treatment effect
    • Repeating many times
  • Then we can ask: how likely would we be to observe the treatment effect in our data, if there is no effect of the treatment?

Résumé Experiment Example

Bertrand and Mullainathan studied racial discrimination in responses to job applications in Chicago and Boston. They sent 4,870 résumés, randomly assigning names associated with different racial groups. - Data are in openintro package as an object called resume - I will save as myDat

library(openintro)
myDat <- resume

Callbacks by Race


Remember, race of applicant is randomly assigned.

library(tidyverse)

mns <- myDat |>
  group_by(race) |> 
  summarize(calls = mean(received_callback))
mns
# A tibble: 2 × 2
  race   calls
  <chr>  <dbl>
1 black 0.0645
2 white 0.0965


Let’s save the means for white and black applicants.


mean_white = mns$calls[2]
mean_black = mns$calls[1]


And calculate the treatment effect. The treatment effect is the difference in means.


teffect <- mean_white - mean_black
teffect
[1] 0.03203285

Before formal tests, let’s look at the data–the estimates and the confidence intervals…


First, let’s make the CIs for the white applicants.


library(tidymodels)
boot_df_white <- myDat |>
  filter(race == "white") |> 
  specify(response = received_callback) |>  
  generate(reps = 15000, type = "bootstrap") |> 
  calculate(stat = "mean")
lower_bound_white <- boot_df_white |> summarize(lower_bound_white = quantile(stat, 0.025)) |> pull() 
upper_bound_white <- boot_df_white |> summarize(upper_bound_white = quantile(stat, 0.975)) |> pull()


Now, let’s create the CIs for black applicants.


boot_df_black <- myDat |>
  filter(race == "black") |> 
  specify(response = received_callback) |>  
  generate(reps = 15000, type = "bootstrap") |> 
  calculate(stat = "mean")
lower_bound_black <- boot_df_black |> summarize(lower_bound_black = quantile(stat, 0.025)) |> pull() 
upper_bound_black <- boot_df_black |> summarize(upper_bound_black = quantile(stat, 0.975)) |> pull()


Now, let’s tidy the data for plotting.


plotData <- tibble(
  race = c("Black", "White"),
  meanCalls = c(mean_black, mean_white),
  lower95 = c(lower_bound_black, lower_bound_white),
  upper95 = c(upper_bound_black, upper_bound_white)
)
plotData
# A tibble: 2 × 4
  race  meanCalls lower95 upper95
  <chr>     <dbl>   <dbl>   <dbl>
1 Black    0.0645  0.0550  0.0743
2 White    0.0965  0.0850  0.108

Plot

Plot


ggplot(plotData, aes(y = meanCalls, x = race, ymin = lower95, ymax = upper95)) +
  geom_col(fill = "steelblue4") +
  geom_errorbar(width = .05) +
  theme_bw()  +
  ylim(0, .15) +
  labs(x = "Race of Applicant",
       y = "Call Back Rate")

Is this evidence of racial discrimination?


  • What is the null hypothesis?
  • What is the alternative hypothesis?
  • How can we formally test the null hypothesis to decide whether to reject it?

Formal Hypothesis Test


  • Calculate the difference in means (White - Black)
  • Shuffle the race variable
  • Calculate the difference in means for the shuffled data
  • Repeat many times
  • Simulates the null distribution of differences in callbacks

Hypothetical Original Data


Applicant Race Callback
A Black Yes
B Black No
C Black No
D White Yes
E White No
F White No

Step 1: Calculate Original Difference in Callback Rates


  • Objective: Understand initial association between race and callback rates

Step 2: Shuffle (Permute) the Race Variable


  • Method: Randomly reassign race labels, keeping callback outcomes fixed

Hypothetical Shuffled Data


Applicant Race (Shuffled) Callback
A White Yes
B Black No
C White No
D White Yes
E Black No
F Black No

Step 3: Calculate Difference in Callback Rates Again


  • After Shuffling: Calculate the difference in callback rates between Black and White groups
  • Purpose: Determine if observed difference is due to chance

Repeat Many Times


  • Repeat shuffling 5000 times to generate a distribution of differences by chance
  • Test: Compare observed difference to null distribution to assess effect of race on callbacks
  • If observed difference is extreme (p-value is low), reject the null hypothesis

Simulating with tidymodels


In real life we are going to use the tidymodels package to do the simulation for us.

null_dist <- myDat |>
  specify(response = received_callback, explanatory = race) |>
  hypothesize(null = "independence") |>
  generate(5000, type = "permute") |>
  calculate(stat = "diff in means", 
            order = c("white", "black")) #

Visualize

Visualize


ggplot(data = null_dist, mapping = aes(x = stat)) +
  geom_histogram(binwidth = 0.01, fill = "steelblue4") +
  labs(title = "Null distribution + Estimate",
       x = "Estimated Difference under the Null") +     
  geom_vline(xintercept = teffect, linetype="dotted", 
                color = "black", size=1) + theme_bw()

Calculate the p-value


null_dist |>
  filter(stat > teffect) |>
  summarise(p_value = n()/nrow(null_dist)) 
# A tibble: 1 × 1
  p_value
    <dbl>
1       0

What should we conclude?


  • The p-value is very small (below .05 threshold)
  • Therefore, we reject the null hypothesis: the racial gap is extremely unlikely to have occurred due to chance alone
  • This is evidence of racial discrimination

Your Turn!


  • Use the gender variable in the resume data to assess whether there is gender discrimination in call backs
  • Plot means and 95% confidence intervals for the call back rate for men and women
  • Write the null and alternative hypotheses
  • Simulate the null distribution
  • Visualize the null distribution and the gender gap
  • Calculate the p-value
  • What do you conclude from your test?

IAFF 6501 Website