Upcoming talk/data workshop on Monday, 6/30: Leah Clark

Please join us Monday, June 30 at 10:00 a.m. in 232M Baker Hall for this talk and workshop led by Leah Clark (CMU Economics and Public Policy)

Title: Patterns of Student Enrollment and Teacher Staffing in Allegheny County Schools since 1997.

**This will be a data analysis workshop. Leah will present her research problem, data resources and questions of interest, and we will talk through some of the data analysis issues as a group.**

Abstract: Enrollment in Pittsburgh Public Schools has declined every year since 1997. Meanwhile, some public schools have closed, some charter schools have opened, and the school-age population in Pittsburgh has declined. I am investigating whether school- and district-level data can provide insight into the choices parents make about where to enroll their children, and the choices teachers make about where to work. The data will permit analyses of other similarly-situated U.S. cities.

Upcoming Talk: Adam Sales

Please join us Monday, June 16 at 10:00 a.m. in 232M Baker Hall for this talk given by Adam Sales (CMART post-doc, CMU/RAND)

Title: Using Covariate Balance to Choose a Regression Discontinuity Bandwidth

Abstract: Regression discontinuity designs (RDDs) are scenarios in which subjects’ treatment assignments are a function of a “running variable” R: treatment is assigned when R is greater (or less) than a known threshold c. These data setups are particularly conducive to causal inference, since the treatment assignment process is known. However, in order to estimate effects, researchers need to model the function relating R, treatment assignment, and the outcome. This modeling step becomes more robust as researchers restrict their data to subjects with R close to the cutoff. That is, researchers can choose a bandwidth around the cutoff, and only use data from within that bandwidth to estimate effects. Recently, Cattaneo et al. (2014) and Sales and Hansen (2014) have suggested using covariate balance tests to choose this bandwidth. Bandwidths within which the analysis fails to detect a significant “treatment effect” on pre-treatment covariates are deemed acceptable. This bandwidth selection technique, applied naively, has several statistical issues, including multiple comparisons, and requires researchers to choose an acceptable level of statistical significance for the covariate balance tests. In this talk, I will suggest a method that resolves these difficulties. The new method is derived from change-point estimation in time-series analysis. The data example for the talk will be Lindo, Sanders, and Oreopoulos (2009)’s study of the effect of academic probation in college on students’ subsequent GPAs.

Upcoming Talk: Ally Thomas

Please join us Monday, May 12 at 10:00 a.m. in 232M Baker Hall for this talk given by Ally Thomas, a Ph.D. student at the the City University of New York Graduate Center.

Title: Evaluating a New York City-Wide STEM Initiative using Genetic Matching

Abstract: Recently, we have been developing evidence (after one year of implementation) of the effects of the Math Science Partnership in New York City (MSPinNYC2) on high school students’ achievement—in terms of scores on end-of-course tests in two key STEM disciplines: Integrated Algebra and Living Environment. The MSPinNYC2 program restructures early high school STEM courses to include 6-8 Teaching Assistant Scholars (TAS) who, along with the teachers, facilitate in-classroom group work on a daily basis. Genetic Matching (Diamond & Sekhon, 2013), a multivariate matching method that uses a search algorithm developed to maximize the balance of observed covariates, was used to create matches of MSP students with similar NYC students. Results from the first and second year suggest the MSPinNYC2 was not effective in raising academic achievement for PERC students in the 9th grade mathematics course (Integrated Algebra), but was effective for PERC students in the 9th grade biology (Living Environment) course. Furthermore the study provided evidence that Genetic Matching is valuable and effective in monitoring the efficacy of a large multi-site instructional intervention.

Upcoming Presentation: The Spencer Foundation and Educational Research

Funding Your Research: An Information Session with the Spencer Foundation

Friday, May 9, 2014
10:00 a.m.
232M Baker Hall
Carnegie Mellon University

In this informational presentation, which will include a question and answer session, Spencer Foundation Associate Program Officer Robert Ream will provide an overview of the Foundation’s three main funding streams including field-initiated grants, fellowships, and the Foundation’s proactive initiatives.

The Spencer Foundation seeks to investigate ways in which education, broadly conceived, can be improved around the world. The Foundation supports the high-quality investigation of education through its research programs and the strengthening and renewal of the educational research community through its fellowship and training programs.

Sponsored by:
CMART: Carnegie Mellon and RAND Traineeship
in Methodology and Interdisciplinary Education Research,
with funding from the Institute for Education Sciences

CMART Lecture and Workshop Series 2013-2014: Robert K. Ream, Thursday May 8th

CMART Lecture and Workshop Series 2013-2014

Robert K. Ream, Ph.D.

Associate Professor of Education, University of California Riverside and
Associate Program Officer, Spencer Foundation

Household Wealth and Adolescents’ Socioemotional Functioning in Schools

DATE and TIME: Thursday, May 8, 2014, 12:00-1:15pm

LOCATION: RAND Corporation (4570 Fifth Ave.), Room 6202

In this talk, Robert Ream will address his work estimating the part played by household wealth in explaining the relationship between family resources and adolescents’ socioemotional skills. The analyses draw from a large-scale nationally-representative sample of adolescents who were in eighth grade in the 2006-7 school year. The findings foster insights regarding whether advantages in wealth correspond to advantages in adolescents’ socioemotional competencies. Findings also identify variation in the magnitude of association between wealth and socioemotional skills according to one’s position in the hierarchy of wealth.

ROBERT K. REAM is Associate Professor of Education at the University of California, Riverside. He joined the UC Riverside faculty in 2004 after postdoctoral fellowships at Princeton University and the RAND Corporation. He is currently on leave from UC Riverside, having taken up responsibilities as an Associate Program Officer at the Spencer Foundation in Chicago. The social dynamics of educational inequality have been the focus of his agenda for research designed to advance understanding of the relation between education and social opportunity. His work appears in a variety of scholarly journals including American Educational Research Journal, Sociology of Education, and Teachers College Record. His book, Uprooting Children: Mobility, Social Capital, and Mexican American Achievement, was published in 2005 by LFB Scholarly Publishing, New York, in the book series, “The New Americans: Recent Immigration and American Society.” Before embarking on a career in research, Dr.Ream served as a legislative aide to former California State Senator Gary K. Hart.

The CMART speaker series is sponsored by the Carnegie Mellon and RAND Traineeships (CMART) in Methodology and Interdisciplinary Research, with funding from the Institute of Education Sciences.

Paper Workshop: Guanglei Hong (with Pizza!)

Please join us on Friday, May 2, 232M Baker Hall for an informal talk with CMART visitor Guanglei Hong, PhD, Professor of Comparative Human Development at the University of Chicago.

Guanglei will present work in progress: “Investigating variation in causal mediation mechanisms across experimental sites.”

Pizza will be provided.

Anyone interested in meeting with Dr. Hong on Thursday or Friday May 1 or 2, please contact acsales@cmu.edu

Upcoming Seminar: Guanglei Hong

Please join us Thursday, May 1 at 5:00PM. in 135A Baker Hall (Adamson Wing) for a seminar given by Guanglei Hong, Professor of Comparative Human Development at the University of Chicago.
Please Note: this is not the usual SERG day, time, or place.

Guanglei is visiting as part of this year’s CMART Lecture and Workshop Series. She is an Associate Professor in the Department of Comparative Human Development and a member of the Committee on Education at the University of Chicago. Her research focuses on developing causal inference theories and methods for evaluating educational policies and instructional programs in multi-level, longitudinal settings.

Title: How Interventions Work: New Weighting Methods for Causal Mediation Analysis

Dr. Hong will discuss her recent work in developing methods for causal mediation analysis, or investigating the mechanisms through which a treatment impacts an outcome. Conventional methods for mediation analysis generate biased results when the mediator-outcome relationship depends on the treatment condition. She introduces ratio-of-mediator- probability weighting (RMPW) for decomposing total effects into natural direct and indirect effects in the presence of treatment-by-mediator interactions. Dr. Hong will discuss an application of the technique to investigate whether employment mediates the relationship between an experimental welfare-to-work program and maternal depression.

The paper on which the talk is based is accessible at http://www.andrew.cmu.edu/user/acsales/HongPaper.pdf

Upcoming Talk: John Engberg

Please join us Monday, April 21 at 10:00 a.m. in 232M Baker Hall for this talk given by John Engberg, a senior economist at the RAND Corporation.*

Title: Creating a Common Scale for Teacher Value Added


Many districts and states are calculating teacher value added (TVA) measures using student assessment results. These measures are intended to capture teachers’ contributions to student achievement growth and increasingly are used as an indicator of teaching effectiveness. The typical TVA measure provides information regarding the relative value added of teachers within a district or state during a particular year, but is not scaled in a way that allows for comparisons over time or between districts and states. We present a method for transforming district TVA measures to a common scale using publicly-available descriptive statistics for state-level distributions of state assessment test scores and National Assessment of Educational Progress (NAEP) scores. This method will allow administrators and researchers to use a common scale to assess the performance of individual teachers and groups of teachers. We conclude with a discussion of the limitations of this method, including the difficultly of calculating the precision of the resulting TVA measures.

* This is joint work with Italo Gutierrez, an associate economist at the RAND Corporation.

Upcoming Talk: Sarah Ryan

Please join us Monday, March 31 at 10:00 a.m. in 232M Baker Hall for this talk given by Sarah Ryan, a postdoctoral fellow with Carnegie Mellon University and RAND.*


Understanding the Effectiveness of Open Learning Initiative Online Courses among Community College Students: Findings and Challenges for Future Research


A recent evaluation study examined learning gains among community college students enrolled in courses where faculty agreed to use Open Learning Initiative (OLI) course materials with gains among students of faculty who did not use OLI resources. Participants were enrolled in introductory “gatekeeper” courses critical to graduation success: Anatomy & Physiology, Biology, Psychology and Statistics. The results of a quasi-experimental analysis revealed positive but non-significant effects with respect to learning gains among OLI students.These results, along with potential explanations for the null effect and recommendations for future work, are discussed. Specifically, it is asserted that details of implementation and fidelity have important implications for research that attempts to isolate causal associations between online learning and student outcomes in postsecondary settings.

* This is joint work with Julia Kaufman (RAND), Candace Thille (Stanford University) and Norman Bier (Carnegie Mellon University)

Upcoming Talk: Dan McCaffrey

Please join us Monday, March 24 at 10:00 a.m. in 232M Baker Hall for this talk given by Dan McCaffrey a Principal Research Scientist at ETS.

Title: SIMEX: Is it the solution to modeling with error-prone covariates?


In 1994, Cook and Stefanski proposed Simulation-Extrapolation (SIMEX) estimation to remove bias in parameter estimation when fitting linear and nonlinear models with error prone covariates. The approach is alluringly simple: add even more noise to your covariates, fit naïve models that ignore measurement error many times, and project to a case with no error. Like the bootstrap or jackknife, because the method relies on additional computations using the naïve approach rather than developing complex models and computational solutions, SIMEX can be applied to almost any problem. However, computational feasibility does not necessarily mean good statistical properties of the results. In this talk, we provide a brief introduction to SIMEX and then discuss the its use in three applications: linear models when covariates have heteroskedastic measurement error that depends on the latent error free measure, inverse probability of treatment weighting for incomplete data or causal modeling, and the use of Student Growth Percentiles to evaluate teachers or schools. In the linear models, SIMEX combined with nonparameteric maximum likelihood estimation of the density of the error free variable performs very well. In the other applications the projection step proves tricky and unless samples are very large SIMEX at best reduces the bias due to measurement error.
Joint work with J. R. Lockwood