In
this post, which he wrote for this blog, he deconstructs a recent study by
prominent economists about school reform. The idea of projecting how many
trillions might be saved if the schools adopted certain test-based reforms rang
a bell.
I
checked my copy of Reign of Error and
found that Eric Hanushek had predicted in 2011 that if the U.S. replaced the
lowest-performing teachers with average teachers, we would match the test
scores of Canada and Finland and generate an additional $112 trillion in
economic output over our lifetimes. (Eric A. Hanushek, “Valuing Teachers: How
Much is a Good Teacher Worth?” Education Next (Summer 2011).
The
following article under review says the gains produced by raising NAEP scores
would generate “only” $76 trillion in new economic output. Not sure why the
future gains dropped from $112 trillion to $76 trillion. The article reviewed
here can be found online at educationnext.org and will appear in the summer
2016 issue of Education Next (http://educationnext.org/pays-improve-school-quality-student-achievement-economic-gain/).
The Cargo Cult Educational and
Economic Reform Theory
By
William J. Mathis
As
U.S. Forces island-hopped across the Pacific during World War II, Melanesians
noticed that the Yankees would land, immediately bull-doze huge landing strips,
put up rows of lights and build a control tower. Great metal birds would then
be attracted, land, and off-load tons of valuable cargo.
Being quick learners
and believing that if they built it, manna would come; the islanders dug
landing strips out of the jungle, placed torches along the sides and built a
bamboo tower to attract these birds. Thus was born a new version of the
economic theory of the “cargo cult.”
With
rigorous application of just such impeccable reasoning, Erik Hanushek, Jens
Ruhose and Ludger Woessman have published their latest re-write, It Pays to Improve School Quality, as the feature story
in the summer 2016 issue of Education Next.
With
a carefully selected data set, you can do amazing things with statistics.
Since
the common school movement of the mid nineteenth century, we have known that
investments in education provide great returns to society and the economy.
Contemporary funding reformers have thus called for equality in investments in
education. The inconsistency in this case is that a veteran opponent of
adequate school funding is the lead proponent of the cargo cult education myth.
As
attractive as the myth is, there are four major faults; (1) they over-simplify
and misread the economic development literature, (2) they wrongly argue
causation from correlation, (3) they incorporate fatal statistical errors in
their analysis, and (4) they frustrate the reader with unexplained mystery
methods.
Economic
Development –
It is puzzling to see economists interpret the economic development literature
so narrowly. For instance, the World Economic Forum’s twelve pillars
of growth mentions education as part of only three of these pillars; early
education, training, and post-graduate research.
Unfortunately, NAEP math
scores measure none of these relevant education pillars particularly well.
Transportation, infrastructure, macroeconomic support, and other vital
necessities for economic development are not even part of the equation.
Presumably, the invisible hand of middle school NAEP math scores will provide
the missing meta-flux which will parachute trillions of dollars onto a needy
society.
It
would be good to see the economic development theory that supports this overly
constricted model, but the discerning reader will be disappointed. There is no
review of the literature — even though there is an entire discipline devoted to
these questions. But the reader does not have to rely simply on the World
Economic Forum.
The United States, with modest performance on international
math scores (PISA), has won the distinction of being number one in the 2014
World Global Entrepreneurship and Development Index. And on another OECD
designed assessment, the national Innovation Index, the U.S. was essentially
tied for fourth place.
A
more realistic and comprehensive model would surely include other relevant
factors. Variables like the decline in carbon based extractive industries, the
graying of the population, the effect of health care costs, and the reported
oversupply of STEM-qualified job seekers might have greater economic relevance
than how kids performed on an eighth-grade math test some years earlier. But
broader issues, such as these, are not addressed.
Correlation
and Causation –
The authors strongly contend that the relationship between math scores and a
stronger economy is causal (pp. 21-22). That is, high NAEP math test scores cause economic growth. This may seem a bold
over-reach, but the authors flatly state “extensive analysis of the
cross-country evidence has shown that a causal interpretation of the
relationships is credible.”
They garnish this statement with phrases such as
the “strong relationship between test scores and economic growth” and advise
“Any state political leader of vision would do well to make school quality a
high priority.”
States should make a “sustained commitment” and “large
economic benefits should accrue.” The overt necessity is an (unspecified but
major) investment in education and particularly in math education.
Strangely,
they then justify making this commitment by citing “the very weak correlation
between increased spending on schools and higher levels of student
achievement.” The reader is then faced with making sense of the authors’ urging
to invest more in test scores while simultaneously saying their overall investment
mechanism doesn’t work. This self-contradiction seems lost on the authors.
Perhaps,
they would re-purpose some unspecified amount of money from some unknown
source. But this is never addressed and is left to conjecture.
The
Fatal Flaw –
With correlation, it matters a great deal what variables are entered in the
analysis and how they are measured. For instance, if the number of predictors
is limited to a small set of highly interrelated measures, then the importance
of these variables will be inflated.
This leads to uncertainty due to a wide
list of omitted “third factor” explanations — which is particularly problematic
in this paper. Likewise, when using the states as units of the analysis, the
variance is collapsed for both test scores and for spending.
Elementary
statistics shows that this reduction in score intervals jacks up the
correlation and exaggerates the resulting findings. This is how you generate
trillions and trillions of theoretical dollars.
This
leads to the question of whether you can claim causality when third or lurking
factors are not examined and when the ecological correlation fallacy is in
play. The obvious answer, of course, is that this causal claim cannot be
supported.
Mystery
Methods –
The methods used to support their argument remain a mystery. While the
central rationale hinges on the “strong correlation” between test scores and
economic growth, the reader will search in vain to find the simple yet key
statistic, the correlation between wealth and math scores. In reviewing earlier
work, David Berliner noted this same omission in the 2014 version but it has
not been fixed.
The
authors do say test scores account for 20-35% of the growth but, again, this is
not explained nor are other factors (except migration) considered. The reader
is simply told to behold the “strong relationship” as illustrated by a
scatterplot (Table 4) which is about as symmetrical as a shotgun blast.
It
must be noted that earlier work (2012, for example) by these authors was often
characterized by far greater methodological detail and lengthier discussion of
omitted variables, poor measures, units of analyses, etc. The latest version
provides none of this detail and the differences in findings between the
reports is simply unexplained.
The
results are extrapolated to a lifetime (defined as 80 years of age) of
earnings. Considering the volatility of the economy over long periods of time,
and the notoriously weak track record of economic predictions, the arithmetic
that gets us to $76 trillion puts a lot of weight on those middle school math
scores. One gets the feeling that one is listening to stock broker speculations
of pork belly futures instead of policy analysts.
******
Ultimately,
the reader is left to puzzle over the purpose of the paper. If it is to
encourage investments in education, the authors would find themselves faced by
their own contradictions.
If it is to encourage the use of test-based reforms,
they would have to overlook three decades of test-based reform which have
produced no convincing narrowing of the achievement gap.
If they wish to
demonstrate that money matters, they inadvertently succeed. Yet, they don’t
explain how this mechanism would work. When this current paper is considered as
part of the ten year set, the reader is left with the impression of statistical
exotica run amok displayed more for the appearance of intellectual elegance or
as a numerological fantasy rather than as a disciplined or useful exploration
of either education or economics.
Finally,
assuming we built the runway and lit the torches, would the $76 trillion of
manna descend from the heavens? Not likely. The underpinning economic theory is
not developed, complete or costed. The curricular improvements are not defined,
costed or planned. The fiscal gains are speculative.
For
a nation that has yet to restore education expenditures to pre-2008 levels, the
promise is a chimera. There is little in the current political landscape that points
to sufficient investments. A more realistic scenario is that greater
disparities between low and high spending states can be expected with ESSA. The
probability of this proposal’s strengthened investment in middle school math
being enacted is further diminished by the primary author’s tour of the
nation’s courthouses