Please join us Monday, April 21 at 10:00 a.m. in 232M Baker Hall for this talk given by John Engberg, a senior economist at the RAND Corporation.*
Title: Creating a Common Scale for Teacher Value Added
Many districts and states are calculating teacher value added (TVA) measures using student assessment results. These measures are intended to capture teachers’ contributions to student achievement growth and increasingly are used as an indicator of teaching effectiveness. The typical TVA measure provides information regarding the relative value added of teachers within a district or state during a particular year, but is not scaled in a way that allows for comparisons over time or between districts and states. We present a method for transforming district TVA measures to a common scale using publicly-available descriptive statistics for state-level distributions of state assessment test scores and National Assessment of Educational Progress (NAEP) scores. This method will allow administrators and researchers to use a common scale to assess the performance of individual teachers and groups of teachers. We conclude with a discussion of the limitations of this method, including the difficultly of calculating the precision of the resulting TVA measures.
* This is joint work with Italo Gutierrez, an associate economist at the RAND Corporation.
Please join us Monday, March 31 at 10:00 a.m. in 232M Baker Hall for this talk given by Sarah Ryan, a postdoctoral fellow with Carnegie Mellon University and RAND.*
Understanding the Effectiveness of Open Learning Initiative Online Courses among Community College Students: Findings and Challenges for Future Research
A recent evaluation study examined learning gains among community college students enrolled in courses where faculty agreed to use Open Learning Initiative (OLI) course materials with gains among students of faculty who did not use OLI resources. Participants were enrolled in introductory “gatekeeper” courses critical to graduation success: Anatomy & Physiology, Biology, Psychology and Statistics. The results of a quasi-experimental analysis revealed positive but non-significant effects with respect to learning gains among OLI students.These results, along with potential explanations for the null effect and recommendations for future work, are discussed. Specifically, it is asserted that details of implementation and fidelity have important implications for research that attempts to isolate causal associations between online learning and student outcomes in postsecondary settings.
* This is joint work with Julia Kaufman (RAND), Candace Thille (Stanford University) and Norman Bier (Carnegie Mellon University)
Please join us Monday, March 24 at 10:00 a.m. in 232M Baker Hall for this talk given by Dan McCaffrey a Principal Research Scientist at ETS.
Title: SIMEX: Is it the solution to modeling with error-prone covariates?
In 1994, Cook and Stefanski proposed Simulation-Extrapolation (SIMEX) estimation to remove bias in parameter estimation when fitting linear and nonlinear models with error prone covariates. The approach is alluringly simple: add even more noise to your covariates, fit naïve models that ignore measurement error many times, and project to a case with no error. Like the bootstrap or jackknife, because the method relies on additional computations using the naïve approach rather than developing complex models and computational solutions, SIMEX can be applied to almost any problem. However, computational feasibility does not necessarily mean good statistical properties of the results. In this talk, we provide a brief introduction to SIMEX and then discuss the its use in three applications: linear models when covariates have heteroskedastic measurement error that depends on the latent error free measure, inverse probability of treatment weighting for incomplete data or causal modeling, and the use of Student Growth Percentiles to evaluate teachers or schools. In the linear models, SIMEX combined with nonparameteric maximum likelihood estimation of the density of the error free variable performs very well. In the other applications the projection step proves tricky and unless samples are very large SIMEX at best reduces the bias due to measurement error.
Joint work with J. R. Lockwood
Please join us Monday, February 17 at 10:00 a.m. in 232M Baker Hall for this talk given by Elizabeth Hirshorn, a post-doctoral fellow in the Learning Research and Development Center at the Pittsburgh Science of Learning Center.
Title: Reading off the beaten pathway: Investigating alternate pathways to literacy
Being able to read and comprehend text is an essential skill in our society, from filling out tax forms, to understanding contracts, to reading the news. Researchers have made great gains in understanding how typical readers achieve literacy, but there are still many unanswered questions about whether alternate routes to literacy exist, and if they do, how they work. Research has demonstrated that certain skills are very important for and predictive of reading ability in the typical pathway to literacy. One such skill is phonological awareness, or knowledge of smaller units of speech sounds, which facilitates the mapping between speech and written letters. Phonological knowledge is thought to support high quality lexical representations that facilitate comprehension. However, not all learners are created with an equal ability to attain these foundational skills. I will present some of my past, current, and future research that examines important individual differences in reading ability and potential atypical pathways to literacy through two main strands of investigation: 1) internal factors due to an individual’s cognitive abilities, like phonological awareness, and 2) external factors due to the writing system or language itself.
Please join us Monday, February 10 at 10:00 a.m. in 232M Baker Hall for this talk given by Jason Imbrogno, a Ph.D. candidate in Economics in the Tepper School of Business at CMU.
The Efficacy of a Pre-Algebra Cognitive Tutor in Chile and Mexico
A math cognitive tutor (MCT) system widely used throughout the U.S. was adapted for use in Chilean and Mexican public middle schools. The curriculum requires large changes in pedagogy, including the use of computers for individual students to progress through an extended pre-algebra program. The study was conducted over a 6-month time period. Using a hierarchical linear model (HLM), we show that students enrolled in schools which were randomly assigned to adopt the MCT significantly improved their standardized math test scores as compared to control group peers. However, the implementation of the changes in the schools and classrooms was not perfect. Those schools which were better prepared to make changes, especially those with sufficient computers and technical support services, saw their students master more of the software part of the curriculum. Students and teachers generally viewed the MCT positively. The results on math performance and attitudes are promising for further propagation of the MCT curriculum. Knowledge from this study regarding the structure and implementation required for schools to successfully exploit the unique teaching capabilities of the MCT should guide the future diffusion of this specific technology.
Please join us on Monday, January 27 at 10:00 a.m. in 232M Baker Hall for this talk given by J.R. Lockwood, a researcher at the Educational Testing Service.
Title: Recent Results on Weighting and Matching Methods for Estimating Causal Effects Using Error-Prone Covariates
Abstract: Weighting and matching estimators using observed covariates to adjust for differences among non-equivalent groups are commonly used to estimate causal effects with observational data. The theorems supporting these approaches require that covariates necessary to adjust for group differences be measured without error. However, covariate measurement error is common in many applications, including education research where test scores are typically important covariates but measure latent achievement constructs with error. This presentation will review recent work on approaches that can be used to correct weighting and matching estimators for covariate measurement error. It argues that a general strategy exists to correct weighting estimators, but matching estimators cannot generally be fixed.
This is joint work with Daniel F. McCaffrey
Please join us on Monday, December 16 at 10:00 a.m. in 232M Baker Hall for this talk given by Lindsay Page, a professor at the University of Pittsburgh School of Education.
Title: Summer Nudging: Can Personalized Text Messages and Peer Mentor Outreach Increase College Going Among Low-Income High School Graduates?
Abstract: Despite decades of policy intervention to increase college entry among low-income students, substantial disparities in college participation by family income persist. Policymakers have largely overlooked the summer after high school as an important time period in students’ transition to college, yet recent research documents summer attrition rates ranging from 10 – 40 percent among students who had been accepted to college and declared an intention to enroll in college as of high school graduation. Encouragingly, several experimental interventions demonstrate that students’ postsecondary plans are quite responsive to additional outreach during the summer months. Questions nonetheless remain about how to maximize the impact and cost-effectiveness of summer support. Text messaging and peer mentor outreach programs are two promising approaches both to inform students of college-related summer tasks and to connect them to professional support when they need help. In this paper, we report on two randomized trials to investigate the role of technology and peer mentor outreach in mitigating summer attrition and helping students enroll and succeed in college. We find that an automated and personalized text messaging campaign to remind students of required pre-matriculation tasks substantially increased college enrollment among students who had less access to college planning supports. A peer mentor intervention increased four-year college enrollment. At a cost of $7 per participant for the text message campaign and $80 per participant for the peer mentor campaign, both strategies—and particularly the text outreach—are promising and cost-effective approaches to increase college entry among populations traditionally underrepresented in higher education. This is joint work with Benjamin Castleman (University of Virginia).
Please join us on Monday, December 2 at 10:00 a.m. in 232M Baker Hall for this talk given by Layla Unger, a doctoral student in Psychology at CMU.
Title: Tracking the Development of Semantic Organization
Abstract: Over the course of development, children learn a substantial amount of information about things in the world around them. However, they must also learn about relationships between things in order to convert the individual collections of attributes known about each thing into a structured, interconnected web of semantic knowledge. Prior research has revealed that children’s ability to recognize different types of relationships expands considerably over the course of development. In contrast, research on the way that semantic knowledge is organized on the basis of these relationships has revealed a static developmental trajectory, in which the growing body of information that children learn with increasing age does not appear to be accompanied by any changes in the way that this knowledge is organized. This discrepancy may be due to limitations of the methodology and analyses used in semantic organization research. The aim of my current project is to explore ways of overcoming these limitations in order to illuminate the development of semantic organization. During this talk, I will focus on the analyses that I am attempting to apply to this question, and the challenges that I have encountered along the way.
Please join us on Monday, November 18 at 10:00 a.m. in 232M Baker Hall for this talk given by Adam Sales, a Post-doc at CMU Statistics and RAND.
Title: Adjusting for Many Covariates in a Matching-Based Educational Program Evaluation
Authors: Adam Sales, Ben Hansen
Abstract: When experiments are impossible, education researchers often attempt to estimate the effect of an intervention by adjusting for available pre-treatment covariates. One popular adjustment method is to estimate propensity scores: a subject’s propensity score is its probability of being assigned to treatment, conditional on its covariates. However, propensity scores are difficult or impossible to estimate when the number of available covariates is large compared to the number of subjects. For instance, many states provide hundreds or thousands of school-level variables, many of which can be used as covariates in a propensity score model. We propose a solution to this problem that combines traditional propensity scoring with high-dimensional predictive models of the outcome of interest. We will apply our method to evaluate a school-level intervention on six Kentucky high schools, for which over 700 school-level variables are available.
Please join us on Monday, November 4 at 10:00 a.m. in 232M Baker Hall for this talk given by Sean Kelly, an assistant professor in the Department of Administrative and Policy Studies at the University of Pittsburgh.
Title: Accounting for the Relationship between Initial Status and Growth in Regression Models
Authors: Sean Kelly, Feifei Ye
Abstract: Educational analysts studying achievement and other educational outcomes frequently encounter an association between initial status and growth, which has important implications for the analysis of covariate effects, including group differences in growth. As explicated by Allison (1990), where only two time points of data are available, it can be difficult or impossible to identify a preferred model. In this paper we extend Allison’s inquiry by considering multiple sources of the association between initial status and growth simultaneously, including measurement error but also intrinsic associations between initial status and growth. We illustrate the potential tradeoffs between the change-score model specifications (models without a control for initial status) and regressor-variable specifications (with a control for initial status) using simulated data.