Grant: Prize: IHSPR Rising Star Award

Difference-in-differences (DiD) analysis is one of the most popular approaches for measuring the impact of health policies and programs. This method involves comparing the change in a treatment group exposed to a policy to a control group not exposed to the policy, before and after the policy change. DiD assumes certain similarities between the treatment and control groups before the policy change. Researchers often try to further improve the comparability of the two groups by matching individuals from each group with similar observable characteristics and then evaluating the effects of the policy. We performed a simulation experiment to demonstrate that matching in DiD can lead to incorrect estimates of policy impacts. We generated many datasets for a hypothetical outcome for treatment and control units under different assumptions about how similar or different the units were prior to a hypothetical policy intervention. We then used DiD to estimate the impact of the intervention with and without matching the treatment and control units. We found that under a specific set of circumstances, which are likely common in health services and policy analysis, matching led to incorrect estimates of the policy effect. Based on our findings, we provide guidance on how and when to appropriately incorporate matching into the DiD study design. This study will directly inform the use and application of an increasingly common method to measure the impact of health policies and programs.