Teach to the tested
Small Differences that Matter: Mistakes in Applying to College
Amanda Pallais
NBER Working Paper, September 2013
Abstract:
This paper estimates the sensitivity of students' college application decisions to a small change in the cost of sending standardized test scores to colleges. Using confidential ACT micro data, I find that when the ACT increased from three to four the number of free score reports that ACT-takers could send, the fraction of test-takers sending four reports rose substantially while the fraction sending three fell by an offsetting amount. Students simultaneously sent their scores to a wider range of colleges. Using micro data from the American Freshman Survey, two identification strategies show that ACT-takers sent more college applications and low-income ACT-takers attended more selective colleges after the cost change. The first strategy compares ACT-takers before and after the cost change, controlling for time trends and covariates, and the second estimates difference-in-difference regressions using SAT-takers as a control group. Back-of-the-envelope calculations suggest that by inducing low-income students to attend more selective colleges, the policy change significantly increased their expected earnings. Because the cost of sending an additional (non-free) ACT score was merely $6 throughout, this sizable behavioral change is surprising and suggests that students may use simple heuristics in making their application decisions. In such a setting, small policy perturbations can have large effects on welfare.
----------------------
Are Catholic Primary Schools More Effective Than Public Primary Schools?
Todd Elder & Christopher Jepsen
Journal of Urban Economics, forthcoming
Abstract:
This paper assesses the causal effects of Catholic primary schooling on student outcomes such as test scores, grade retention, and behavior. Catholic school students have substantially better average outcomes than do public school students throughout the primary years, but we present evidence that selection bias is entirely responsible for these advantages. Estimates based on several empirical strategies, including an approach developed by Altonji et al. (2005a) to use selection on observables to assess the bias arising from selection on unobservables, imply that Catholic schools do not appreciably boost test scores. All of the empirical strategies point to sizeable negative effects of Catholic schooling on mathematics achievement. Similarly, we find very little evidence that Catholic schooling improves behavioral and other non-cognitive outcomes once we account for selection on unobservables.
----------------------
Is There Empirical Evidence That Charter Schools “Push Out” Low-Performing Students?
Ron Zimmer & Cassandra Guarino
Educational Evaluation and Policy Analysis, December 2013, Pages 461-480
Abstract:
A major concern among opponents to charter schools is whether these schools will serve all students. Some have raised concerns that charter schools will “push out” low-achieving students in hopes of improving the schools’ academic profile while minimizing costs by educating fewer challenging students. In this article, we use data from an anonymous major urban school district to examine whether we see exit patterns consistent with the claim that charter schools are more likely to push out low-achieving students than are traditional public schools (TPSs). Overall, we find no empirical evidence to support the notion of push-out.
----------------------
The Medium-Term Impacts of High-Achieving Charter Schools on Non-Test Score Outcomes
Will Dobbie & Roland Fryer
NBER Working Paper, October 2013
Abstract:
High-performing charter schools can significantly increase the test scores of poor urban students. It is unclear whether these test score gains translate into improved outcomes later in life. We estimate the effects of high-performing charter schools on human capital, risky behaviors, and health outcomes using survey data from the Promise Academy in the Harlem Children's Zone. Six years after the random admissions lottery, youth offered admission to the Promise Academy middle school score 0.283 standard deviations higher on a nationally-normed math achievement test and are 14.1 percentage points more likely to enroll in college. Admitted females are 12.1 percentage points less likely to be pregnant in their teens, and males are 4.3 percentage points less likely to be incarcerated. We find little impact of the Promise Academy on self-reported health. We conclude with speculative evidence that high-performing schools may be sufficient to significantly improve human capital and reduce certain risky behaviors among the poor.
----------------------
Incentives, Selection, and Teacher Performance: Evidence from IMPACT
Thomas Dee & James Wyckoff
NBER Working Paper, October 2013
Abstract:
Teachers in the United States are compensated largely on the basis of fixed schedules that reward experience and credentials. However, there is a growing interest in whether performance-based incentives based on rigorous teacher evaluations can improve teacher retention and performance. The evidence available to date has been mixed at best. This study presents novel evidence on this topic based on IMPACT, the controversial teacher-evaluation system introduced in the District of Columbia Public Schools by then-Chancellor Michelle Rhee. IMPACT implemented uniquely high-powered incentives linked to multiple measures of teacher performance (i.e., several structured observational measures as well as test performance). We present regression-discontinuity (RD) estimates that compare the retention and performance outcomes among low-performing teachers whose ratings placed them near the threshold that implied a strong dismissal threat. We also compare outcomes among high-performing teachers whose rating placed them near a threshold that implied an unusually large financial incentive. Our RD results indicate that dismissal threats increased the voluntary attrition of low-performing teachers by 11 percentage points (i.e., more than 50 percent) and improved the performance of teachers who remained by 0.27 of a teacher-level standard deviation. We also find evidence that financial incentives further improved the performance of high-performing teachers (effect size = 0.24).
----------------------
Small High Schools and Student Achievement: Lottery-Based Evidence from New York City
Atila Abdulkadiroğlu, Weiwei Hu & Parag Pathak
NBER Working Paper, October 2013
Abstract:
One of the most wide-ranging reforms in public education in the last decade has been the reorganization of large comprehensive high schools into small schools with roughly 100 students per grade. We use assignment lotteries embedded in New York City's high school match to estimate the effects of attendance at a new small high school on student achievement. More than 150 unselective small high schools created between 2002 and 2008 have enhanced autonomy, but operate within-district with traditional public school teachers, principals, and collectively-bargained work rules. Lottery estimates show positive score gains in Mathematics, English, Science, and History, more credit accumulation, and higher graduation rates. Small school attendance causes a substantial increase in college enrollment, with a marked shift to CUNY institutions. Students are also less likely to require remediation in reading and writing when at college. Detailed school surveys indicate that students at small schools are more engaged and closely monitored, despite fewer course offerings and activities. Teachers report greater feedback, increased safety, and improved collaboration. The results show that school size is an important factor in education production and highlight the potential for within-district reform strategies to substantially improve student achievement.
----------------------
Tin-Chun Lin & Amanda Couch
Applied Economics Letters, Winter 2013, Pages 220-223
Abstract:
In this article, data from 286 Indiana school districts during the 2009–2010 school year were used in a case study to investigate whether fiscal funding has a direct effect on education output (i.e. student achievement). Two results were found: (1) while federal taxes are not significantly associated with student achievement in public schools, both state and local taxes are strongly related to student achievement in public schools, implying a direct effect on education output; and (2) state tax funding has a greater impact on education output than local tax funding.
----------------------
Interactive Learning Online at Public Universities: Evidence from a Six-Campus Randomized Trial
William Bowen et al.
Journal of Policy Analysis and Management, forthcoming
Abstract:
Online instruction is quickly gaining in importance in U.S. higher education, but little rigorous evidence exists as to its effect on student learning. We measure the effect on learning outcomes of a prototypical interactive learning online statistics course by randomly assigning students on six public university campuses to take the course in a hybrid format (with machine-guided instruction accompanied by one hour of face-to-face instruction each week) or a traditional format (as it is usually offered by their campus, typically with about three hours of face-to-face instruction each week). We find that learning outcomes are essentially the same — that students in the hybrid format are not harmed by this mode of instruction in terms of pass rates, final exam scores, and performance on a standardized assessment of statistical literacy. We also conduct speculative cost simulations and find that adopting hybrid models of instruction in large introductory courses has the potential to significantly reduce instructor compensation costs in the long run.
----------------------
The Association Between School-to-Work Programs and School Performance
Erin Welsh et al.
Journal of Adolescent Health, forthcoming
Purpose: The School-to-Work (STW) Opportunities Act was passed to aid students in transitioning from education to employment by offering work-based learning opportunities. In the United States, 72% of high schools offer work-based learning opportunities for credit. This is the first study to describe school performance and school-based behaviors among students enrolled in STW programs and compare them with nonworking and other-working students.
Methods: In 2003, a questionnaire was administered to five school districts and one large urban school in Wisconsin. Between 2008 and 2010, analyses were completed to characterize STW students and compare them with other students.
Results: Of the 6,519 students aged 14–18 years included in the analyses, 461 were involved in an STW program (7%), 3,108 were non-working (48%), and 2,950 were other-working students (45%). Compared with other students, STW students were less likely to have a grade point average >2.0, more likely to have three or more unexcused absences from school, and more likely to spend <1 hour in school-sponsored activities. Holding multiple jobs also negatively affected a student's academic performance.
Conclusions: School-to-Work students reported poorer academic performance and more unhealthy school-related behaviors compared with nonworking students and other-working students. Whereas many factors have a role in why students perform poorly in school, more research on students enrolled in STW programs is needed to understand whether participating has a negative impact on students' academic achievement.
----------------------
Teacher Behavior under Performance Pay Incentives
Michael Jones
Economics of Education Review, December 2013, Pages 148–164
Abstract:
Over the last decade many districts implemented performance pay incentives to reward teachers for improving student achievement. Economic theory suggests that these programs could alter teacher work effort, cooperation, and retention. Because teachers can choose to work in a performance pay district that has characteristics correlated with teacher behavior, I use the distance between a teacher's undergraduate institution and the nearest performance pay district as an instrumental variable. Using data from the 2003 and 2007 waves of the Schools and Staffing Survey, I find that teachers respond to performance pay incentives by working fewer hours per week. Performance pay also decreases participation in unpaid cooperative school activities, while there is suggestive evidence that teacher turnover decreases. The treatment effects are heterogeneous; male teachers respond more positively than female teachers. In Florida, which restricts state performance pay funding to individual teachers, I find that work effort and teacher turnover increase.
----------------------
Voluntary Disclosure and the Strategic Behavior of Colleges
Michael Conlin, Stacy Dickert-Conlin & Gabrielle Chapman
Journal of Economic Behavior & Organization, December 2013, Pages 48–64
Abstract:
This paper investigates how outside ranking organizations such as U.S. News and World Report affect colleges’ admission decisions. To do this, we focus on a policy that has received criticism for being motivated by ranking concerns: optional reporting of SAT I scores. This policy allows colleges to report an average SAT I score based on those applicants who chose to submit their scores which may not be reflective of actual student body quality. We use proprietary data from two liberal arts colleges to address how the optional reporting policy affects the colleges’ admission decisions as well as how applicants’ SAT I scores influence their decision to submit these scores to the colleges. The data suggest that college admission departments are behaving strategically by rewarding applicants who do submit their SAT I scores when their scores will raise the college's average SAT I score reported to U.S. News and World Report and rewarding applicants who do not submit when their SAT I scores will lower the college's reported score. The data also suggest that applicants are behaving strategically by choosing not to reveal their SAT I scores if they are below a value one might predict based on their other observable characteristics.
----------------------
Merit Aid, Student Mobility, and the Role of College Selectivity
Rajashri Chakrabarti & Joydeep Roy
Federal Reserve Working Paper, September 2013
Abstract:
In this paper, we investigate the role of college selectivity in mobility decisions (both in-state and out-of-state) of freshmen students following Georgia’s HOPE scholarship program. How did HOPE affect the selectivity of colleges attended by Georgia’s freshmen students? Did it induce Georgia’s freshmen students who would have otherwise attended more selective out-of-state colleges to instead attend less selective in-state ones? Or was there movement to more selective ones, both in-state and out-of-state? Using student residency and enrollment data from IPEDS and selectivity data from Barron’s and Peterson’s, we find that in the aftermath of HOPE, Georgia freshmen attended relatively more selective colleges overall. Disaggregating further, we find that Georgia freshmen attending in-state colleges attended more selective ones. Georgia freshmen attending out-of-state colleges were also more likely to attend more selective colleges, most likely due to an increase in the reservation price to go to out-of-state colleges following HOPE. Our results are robust to a variety of sensitivity checks and have important policy implications. In particular, Peltzman had observed in his classic 1973 paper that in-kind subsidies can induce individuals to invest in less quality-adjusted human capital than they might otherwise. The fact that Georgia freshmen attended relatively more selective colleges in the post-HOPE period allays, to some extent, the concern that state merit aid programs can adversely affect long-term outcomes and human capital formation.
----------------------
Evaluation of the College Possible Program: Results from a Randomized Controlled Trial
Christopher Avery
NBER Working Paper, October 2013
Abstract:
This paper reports the results of a randomized trial of the College Possible program, which provides two years of college preparatory work for high school juniors and seniors in Minneapolis and St. Paul. The trial involved 238 students, including 134 who were randomly selected for admission to the program. The results indicate that the College Possible program significantly increased both applications and enrollment to both four-year colleges and selective four-year colleges; we estimate that initial enrollment at four-year colleges increased by more than 15 percentage points for program participants, but find little evidence of any effect of the program on ACT performance or college enrollment overall.
----------------------
Leslie Stratton & James Wetzel
B.E. Journal of Economic Analysis & Policy, October 2013, Pages 1121–1142
Abstract:
Standard analyses of college outcomes look at six-year graduation rates, treating all non-graduates alike as “failures”. However, we find that 36% of non-graduates are still enrolled. Using micro-level data with rich information on demographic and academic background, we employ a multinomial logit model to distinguish among graduates, persisters, and dropouts six years following matriculation. We find that there are significant differences across these populations. Separate evidence indicates that as many as half of those persisting at the six-year mark will graduate within a few years. Thus, six-year graduation rates understate “success,” but future success is not the same for all groups. Holding academic background constant, reported graduation rates are lower for Hispanics because they are taking longer to graduate and lower for first-generation college students because they are dropping out. The most important factor is academic background, suggesting that increased financial aid is unlikely to substantially increase graduation rates.
----------------------
Merit Aid and Post-College Retention in the State
David Sjoquist & John Winters
Journal of Urban Economics, forthcoming
Abstract:
One goal of state merit-based financial aid programs is to increase the stock of college-educated labor in the state by retaining college-educated persons in the state after college. However, there has been surprisingly little research on whether state merit aid programs are effective at this goal. This paper investigates the effect of state merit aid programs on the post-college location of 24-30 year olds. We use decennial census and American Community Survey microdata to consider post-college retention effects in the 25 states that implemented merit aid programs between 1991 and 2004. Our preferred specification implies that strong state merit aid programs on average increase the probability that a college attendee lives in his or her birth state during ages 24-30 by 2.76 percentage points. We also estimate the effect for individual states and find meaningful differences across states in the effect of merit aid programs on in-state post-college retention and explore explanations for these differences.
----------------------
The Aggregate Effect of School Choice: Evidence from a Two-stage Experiment in India
Karthik Muralidharan & Venkatesh Sundararaman
NBER Working Paper, September 2013
Abstract:
We present experimental evidence on the impact of a school choice program in the Indian state of Andhra Pradesh (AP) that featured a unique two-stage lottery-based allocation of school vouchers that created both a student-level and a market-level experiment. This design allows us to study both the individual and the aggregate effects of school choice (including spillovers). We find that private-school teachers have lower levels of formal education and training than public-school teachers, and are paid much lower salaries. On the other hand, private schools have a longer school day, a longer school year, smaller class sizes, lower teacher absence, higher teaching activity, and better school hygiene. After two and four years of the program, we find no difference between the test scores of lottery winners and losers on math and Telugu (native language). However, private schools spend significantly less instructional time on these subjects, and use the extra time to teach more English, Science, Social Studies, and Hindi. Averaged across all subjects, lottery winners score 0.13σ higher, and students who attend private schools score 0.23σ higher. We find no evidence of spillovers on public-school students who do not apply for the voucher, or on students who start out in private schools to begin with, suggesting that the program had no adverse effects on these groups. Finally, the mean cost per student in the private schools in our sample is less than a third of the cost in public schools. Our results suggest that private schools in this setting deliver (slightly) better test score gains than their public counterparts, and do so at substantially lower costs per student. More generally, our results highlight that ignoring heterogeneity among schools' instructional programs and patterns of time use may lead to incorrect inference on the impact of school choice on learning outcomes.
----------------------
Contract Teachers: Experimental Evidence from India
Karthik Muralidharan & Venkatesh Sundararaman
NBER Working Paper, September 2013
Abstract:
The large-scale expansion of primary schooling in developing countries has led to the increasing use of non-civil-service contract teachers who are locally-hired from the same village as the school, are not professionally trained, have fixed-term renewable contracts, and are paid much lower salaries than regular civil-service teachers. This has been a controversial policy, but there is limited evidence on the effectiveness of contract teachers in improving student learning. We present experimental evidence on the impact of contract teachers using data from an ‘as is’ expansion of contract-teacher hiring across a representative sample of 100 randomly-selected government-run rural primary schools in the Indian state of Andhra Pradesh. At the end of two years, students in schools with an extra contract teacher performed significantly better than those in comparison schools by 0.16σ and 0.15σ, in math and language tests respectively. Contract teachers were also much less likely to be absent from school than civil-service teachers (18% vs. 27%). Using the experimental variation in school-level pupil-teacher ratio (PTR) induced by the provision of an extra contract teacher, we estimate that reducing PTR by 10% using a contract teacher would increase test scores by 0.03σ/year. Using high-quality panel data over five years we estimate that the corresponding gain to reducing PTR by 10% using a regular civil-service teacher would be 0.02σ/year. Thus, in addition to finding that contract teachers are effective at improving student learning outcomes, we find that they are no less effective than regular civil-service teachers who are more qualified, better trained, and paid five times higher salaries.
----------------------
The Impact of Pre-school on Adolescents’ Outcomes: Evidence from a Recent English Cohort
Patricia Apps, Silvia Mendolia & Ian Walker
Economics of Education Review, December 2013, Pages 183–199
Abstract:
This paper investigates the relationship between attendance at pre-school school and children's outcomes into early adulthood. In particular, we are interested in: child cognitive development at ages 11, 14 and 16; intentions towards tertiary education; economic activity in early adulthood; and a group of non-cognitive outcomes such as risky health behaviour; and personality traits. Using matching methods to control for a very rich set of child and family characteristics, we find evidence that pre-school childcare moderately improves results in cognitive tests at age 11 and 14 and 16. Positive effects are especially noticeable for girls and children from disadvantaged socio-economic backgrounds. Results for non-cognitive outcomes are weaker: we do not find any significant evidence of improvement in psychological well-being, petty crime involvement, or on almost all health behaviours. While the cognitive effects may well serve to reduce lifecycle inequalities there is no support here for other important social benefits.
----------------------
Kirabo Jackson
Journal of Public Economics, forthcoming
Abstract:
Using exogenous secondary school assignments to remove self-selection bias to schools and peers within schools, I credibly estimate both (1) the effect of attending schools with higher-achieving peers, and (2) the direct effect of short-run peer quality improvements within schools, on the same population. While students at schools with higher-achieving peers have better academic achievement, within-school short-run increases in peer achievement improve outcomes only at high-achievement schools. Short-run (direct) peer quality accounts for only one tenth of school value-added on average, but at least one-third among the most selective schools. There are large and important differences by gender.
----------------------
When Does Inter-School Competition Matter? Evidence from the Chilean “Voucher” System
Francisco Gallego
B.E. Journal of Economic Analysis & Policy, October 2013, Pages 525–562
Abstract:
I investigate the effects of voucher-school competition on educational outcomes. I test whether voucher-school competition (1) improves student outcomes and (2) has stronger effects when public schools face a hard-budget constraint. Since both voucher-school competition and the degree of hardness of the budget constraint for public schools are endogenous to public school quality, I exploit (i) the interaction of the number of Catholic priests in 1950 and the institution of the voucher system in Chile in 1981 as a potentially exogenous determinant of the supply of voucher schools and (ii) a particular feature of the electoral system that affects the identity of the mayors of different counties (who manage public schools) as a source of exogenous variation in the degree of hardness of the public schools’ budget constraints. Using this information, I find that (1) an increase of one standard deviation of the ratio of voucher-to-public schools increases test scores by just around 0.10 standard deviations; and (2) the effects are significantly bigger for public schools facing more binding minimum enrollment levels.
----------------------
After-School Tutoring and the Distribution of Student Performance
Min-Hsiung Huang
Comparative Education Review, November 2013, Pages 689-710
Abstract:
As more primary and secondary students worldwide seek after-school tutoring in academic subjects, concerns are being raised about whether after-school tutoring can raise average test scores without widening the variability in student performance, and whether students of certain ability levels may benefit more than others from after-school tutoring. To address these questions, I compared the distributions of student performance across countries with differing levels of participation in after-school tutoring, while controlling for country-level unobserved heterogeneity using a fixed-effects model. Participating in either mathematics or science tutoring after school is found to raise national average performance without widening the dispersion in student performance. In science, low-performing students benefit more from tutoring than do high-performing students. In mathematics, high-performing students benefit more from tutoring than do low-performing students.