Upcoming Talk: JR Lockwood (ETS) Dec 1 2014

Please join us Monday, December 1 at 10:00 a.m. in 232M Baker Hall for this talk given by J.R. Lockwood, PhD, of The Educational Testing Service

Title: Inferring Constructs of Effective Teaching from Classroom Observations: An Application of Bayesian Exploratory Factor Analysis Without Restrictions

Abstract: he dramatic public policy shifts toward increasing teacher ac-
countability have generated numerous sets of student outcome, teach-
ing process and teacher knowledge-based instruments all seeking to
measure the quality of teaching. These instruments are being used
to assess and pay teachers without a clear understanding of what as-
pects of teaching are being assessed, and how the many dimensions
that comprise any one measure relate to those of other measures. We
use data from multiple instruments collected from approximately 450
middle school mathematics and English language arts teachers and
their students to inform research and practice on teacher performance
measurement by modeling the underlying constructs of high-quality
teaching. We make inferences about these constructs using a novel
approach to Bayesian exploratory factor analysis (EFA) that, unlike
commonly-used approaches for identifying factor loadings in Bayesian
EFA, is invariant to how the data dimensions are ordered. Using this
approach with our data reveals two distinct teaching constructs in
both mathematics and English language arts: 1) Practices used by
teachers to instruct and engage students; and 2) Teacher management
of classrooms.We demonstrate the relationships of these constructs to
other indicators of teaching quality including teacher content knowl-
edge and student performance on standardized tests.

Upcoming Talk Monday, November 17: Ilya Goldin

Please join us Monday, November 17 at 10:00 a.m. in 232M Baker Hall for this talk given by Ilya Goldin, PhD, of Pearson Education

Title: Individual differences in identifying sources of science knowledge
Joint work with Maggie Renken, April Galyardt, and Ellen Litkowski

Abstract. We have developed an instrument to assess students’ proficiencies in identifying sources of science knowledge (SoK) in text passages. We describe the new web-based instrument and our evaluation of the instrument with a sample (n = 338) of children grades 2-8. By creating and validating this tool, we aim to establish a learning progression, inform science teaching, and tailor instruction to individual differences. Our findings suggest that students demonstrate differential ability in identifying SoK and thus imply the need for instruction to accommodate individual student perspectives on SoK. We expect that highlighting student ability in identifying SoK as a distinct skill will enable differentiated, adaptive instruction. We further expect this instrument to make explicit a component of what it means to think like a scientist, and in doing so facilitate conversations among teachers and students about the practice of science.

Upcoming Talk: Dan McCaffrey

Please join us Monday, October 20 at 10:00 a.m. in 232M Baker Hall for this talk given by Dan McCaffrey of the Educational Testing Service

Title: Uncovering Multivariate Structure in Classroom Observations in the Presence of Rater Errors


We examine the factor structure of scores from the CLASS-S protocol obtained from observations of middle school classroom teaching. Factor analysis has been used to support both interpretations of scores from classroom observation protocols, like CLASS-S, and the theories about teaching that underlie them. However, classroom observations contain multiple sources of error, most predominately rater errors. We demonstrate that errors in scores made by two raters on same lesson have a factor structure that is distinct from the factor structure at the teacher level. Consequently, the ‘standard’ approach of analyzing on teacher-level average dimension scores can yield incorrect inferences about the factor structure at the teacher level and possibly misleading evidence about the validity of scores and theories of teaching. We consider alternative hierarchical estimation approaches designed to prevent the contamination of estimated teacher-level factor. These alternative approaches find a teacher-level factor structure for CLASS-S that consists of strongly correlated support and classroom management factors. Our results have implications for future studies using factor analysis on classroom observation data to develop validity evidence and test theories of teaching and for practitioners who rely on the results of such studies to support their use and interpretation of the classroom observation scores.

Upcoming Talk: Brian Junker

Please join us Monday, October 6 at 10:00 a.m. in 232M Baker Hall for this talk given by Brian Junker (CMU)

Title: Predictive Inference Using Latent Variables with Covariates

Joint with Dan A Black (University of Chicago), Lynne Steuerle Scho efield (Swarthmore), and Lowell J Taylor (Carnegie Mellon)

Abstract: Plausible Values (PVs) have been a standard multiple imputation tool for latent proficiency variables in large scale education survey data since their implementation in the National Assessment of Educational Progress (NAEP) in the 1980′s. Today PVs are used widely in many national and international education surveys. When latent proficiency is the dependent variable in an analysis, well-constructed PVs provide guarantees of unbiasedness for inferences about latent proficiency. We review the well-known results that provide these guarantees, and try to extend them to the case in which latent proficiency is one of the independent variables in an analysis. We show that the same guarantees are impossible in the latter case, and provide an alternative approach, based on Schofield’s (2008) mixed effects structural equations (MESE) model. An example using data from the 1992 National Adult Literacy Survey (NALS) illustrates our results.

Upcoming Talk: Adam Sales

Please join us Monday, Sept 22 at 10:00 a.m. in 232M Baker Hall for this talk given by Adam Sales (CMART post-doc, CMU/RAND)

Title: A Useful Model? Using Covariates to Test the Usefulness of the Randomization Assumption.

Social scientists often look for “natural experiments” to estimate causal effects. They claim, based on quirks in known part of the data generating process, that a treatment was assigned haphazardly. It is good practice in these analyses to test covariate balance: significantly different distributions of baseline covariates between treatment and control groups will falsify the random treatment model. But as we know from Stats 101, failing to reject the hypothesis that covariates are balanced does not imply that they are. In this talk, I will propose a modified procedure where instead of testing the hypothesis that covariates are balanced, researchers can test whether they are imbalanced enough to invalidate the study.
I will use as an example data from Demming (2009) that investigated the effects of head start.

UPDATE: To see the slides, click here

Upcoming Talk: David Choi

After a bit of a break,
please join us Monday, August 11 at 10:00 a.m. in 232M Baker Hall for this talk given by David Choi, a Professor of Statistics and Information Systems at Heinz College, CMU.

Title: Estimation of monotone treatment effects in network experiments

Abstract: Randomized experiments in social network settings are a trending research topic. In addition to the logistical difficulties of running a large social experiment, there may also be statistical challenges in analyzing the data. We discuss the statistical challenge of analyzing experimental data in social networks when the network cannot be divided into smaller non-interacting subgroups, so that interference between units must be taken into account. We present work in progress on how to rigorously analyze such data, assuming that the treatment effect is nonnegative but otherwise making no further assumptions on the flow of influence between units.

Upcoming talk/data workshop on Monday, 6/30: Leah Clark

Please join us Monday, June 30 at 10:00 a.m. in 232M Baker Hall for this talk and workshop led by Leah Clark (CMU Economics and Public Policy)

Title: Patterns of Student Enrollment and Teacher Staffing in Allegheny County Schools since 1997.

**This will be a data analysis workshop. Leah will present her research problem, data resources and questions of interest, and we will talk through some of the data analysis issues as a group.**

Abstract: Enrollment in Pittsburgh Public Schools has declined every year since 1997. Meanwhile, some public schools have closed, some charter schools have opened, and the school-age population in Pittsburgh has declined. I am investigating whether school- and district-level data can provide insight into the choices parents make about where to enroll their children, and the choices teachers make about where to work. The data will permit analyses of other similarly-situated U.S. cities.

Upcoming Talk: Adam Sales

Please join us Monday, June 16 at 10:00 a.m. in 232M Baker Hall for this talk given by Adam Sales (CMART post-doc, CMU/RAND)

Title: Using Covariate Balance to Choose a Regression Discontinuity Bandwidth

Abstract: Regression discontinuity designs (RDDs) are scenarios in which subjects’ treatment assignments are a function of a “running variable” R: treatment is assigned when R is greater (or less) than a known threshold c. These data setups are particularly conducive to causal inference, since the treatment assignment process is known. However, in order to estimate effects, researchers need to model the function relating R, treatment assignment, and the outcome. This modeling step becomes more robust as researchers restrict their data to subjects with R close to the cutoff. That is, researchers can choose a bandwidth around the cutoff, and only use data from within that bandwidth to estimate effects. Recently, Cattaneo et al. (2014) and Sales and Hansen (2014) have suggested using covariate balance tests to choose this bandwidth. Bandwidths within which the analysis fails to detect a significant “treatment effect” on pre-treatment covariates are deemed acceptable. This bandwidth selection technique, applied naively, has several statistical issues, including multiple comparisons, and requires researchers to choose an acceptable level of statistical significance for the covariate balance tests. In this talk, I will suggest a method that resolves these difficulties. The new method is derived from change-point estimation in time-series analysis. The data example for the talk will be Lindo, Sanders, and Oreopoulos (2009)’s study of the effect of academic probation in college on students’ subsequent GPAs.

Upcoming Talk: Ally Thomas

Please join us Monday, May 12 at 10:00 a.m. in 232M Baker Hall for this talk given by Ally Thomas, a Ph.D. student at the the City University of New York Graduate Center.

Title: Evaluating a New York City-Wide STEM Initiative using Genetic Matching

Abstract: Recently, we have been developing evidence (after one year of implementation) of the effects of the Math Science Partnership in New York City (MSPinNYC2) on high school students’ achievement—in terms of scores on end-of-course tests in two key STEM disciplines: Integrated Algebra and Living Environment. The MSPinNYC2 program restructures early high school STEM courses to include 6-8 Teaching Assistant Scholars (TAS) who, along with the teachers, facilitate in-classroom group work on a daily basis. Genetic Matching (Diamond & Sekhon, 2013), a multivariate matching method that uses a search algorithm developed to maximize the balance of observed covariates, was used to create matches of MSP students with similar NYC students. Results from the first and second year suggest the MSPinNYC2 was not effective in raising academic achievement for PERC students in the 9th grade mathematics course (Integrated Algebra), but was effective for PERC students in the 9th grade biology (Living Environment) course. Furthermore the study provided evidence that Genetic Matching is valuable and effective in monitoring the efficacy of a large multi-site instructional intervention.