The answer: when it comes to science, U.S. 4th graders rank #8 of 36 participating countries; 8th graders rank #11 of 48 countries. Top FOUR nations are from East Asia. (And for all the money Saudi Arabia has, their students rank waaaay near the bottom of the list.)
I invited Bart Leahy, a technical writer from Huntsville, Alabama, and blogger at Bartacus to sift through the complicated charts and figures recently released in the Trends in International Mathematics and Science Study and boil down the basics for us.
His analysis closes with a bonus: suggestions on how we can improve the situation.  Should we do away with the failing “self-esteem building” curriculum? Perhaps follow Singapore’s lead and bring back corporal punishment? 😉
Read Bart’s full report, below,  and let us know if you have other recommendations on ways we can improve science and math education.

Analyzing the Trends in International Mathematics and Science Study (TIMMS), Part I

By Bart D. Leahy

 

At Darlene’s invitation, I was given the opportunity to review and discuss the 2007 Trends in International Mathematics and Science Study (TIMMS). This is a large study of math and science scores among 4^th and 8^th grade students, with a lot of data to be crunched, which might explain why the 2007 study is being released in late 2008. Given the sheer amount of data, I will concentrate on just some of the highlights and lowlights about American students’ test scores. Darlene also provided a link to a news story from the Philadelphia perspective.

 

In the first part of my analysis, I will concentrate on the United States’ ranking compared with other nations of the world, and the federal policy implications. In part two, I will address some of the specific results in the study, and what and how we should teach our children in the future.

 

First, a little background on the study:

 

How TIMSS was Conducted

In the United States, TIMSS was administered in spring 2007. The U.S. sample is representative of both public and private school students at 4th and 8th grades nationally. In total, 257 schools and 10,350 students participated at grade four, and 239 schools and 9,723 students participated at grade eight. More information about how the assessment was developed and conducted is included in the technical notes of the U.S. report on TIMSS (http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2009001).

How TIMSS Results are Reported

Like other large-scale assessments, TIMSS was not designed to provide individual student scores, but rather national and group estimates of performance. Achievement results from TIMSS are reported on a scale from 0 to 1,000. In order to compare performance over time, each TIMSS administration is placed on the same scale, which has a mean of 500 and standard deviation of 100. The TIMSS scale average (500) is the mean score of the original TIMSS 1995 countries (including the United States). Countries can compare their scores over time to this standardized TIMSS scale average, as well as compare their scores directly with other countries.

All differences described using TIMSS data are statistically significant at the .05 level. Differences that are not statistically significant are either not discussed or referred to as “not measurably different” or “not statistically significant.”

In addition to numerical scale results, TIMSS includes international benchmarks at four points on the mathematics and science scales—advanced international benchmark (625), high international benchmark (550), intermediate international benchmark (475), and low international benchmark (400).

 

One thing to bear in mind, which the background document (linked above) makes clear: the 4^th and 8^th grade results are not exactly apples-to-apples comparisons, as different standards were used, as well as different categories of knowledge. For example, the test used to grade 8^th graders the Program for International Student Assessment (PISA), breaks out physical sciences and chemistry, something the 4^th grade test—the TIMMS—does not. Another challenge for the statisticians is that some nations participated in one survey but not the other, while some nations participated in both.

What the Numbers Say

Below are the stats for the participating countries’ math scores and where U.S. students fit in at the 4^th and 8^th grade levels.

 

Average mathematics scores of fourth- and eighth-grade students, by country: 2007

TIMSS scale average 500

Black Text = Nations participating in only one survey

Red Text = Nations participating in both

Blue Text = United States scores

Grade Four (TIMMS)		Grade Eight (PISA)
Country	Average Score	Country	Average Score
Hong Kong SAR	607	Chinese Taipei	598
Singapore	599	Korea, Rep. of	597
Chinese Taipei	576	Singapore	593
Japan	568	Hong Kong SAR	572
Kazakhstan	549	Japan	570
Russian Federation	544	Hungary	517
England	541	England	513
Latvia	537	Russian Federation	512
Netherlands	535	United States	508
Lithuania	530	Lithuania	506
United States	529	Czech Republic	504
Germany	525	Slovenia	501
Denmark	523	Armenia	499
Australia	516	Australia	496
Hungary	510	Sweden	491
Italy	507	Malta	488
Austria	505	Scotland	487
Sweden	503	Serbia	486
Slovenia	502	Italy	480
Armenia	500	Malaysia	474
Slovak Republic	496	Norway	469
Scotland	494	Cyprus	465
New Zealand	492	Bulgaria	464
Czech Republic	486	Israel	463
Norway	473	Ukraine	462
Ukraine	469	Romania	461
Georgia	438	Bosnia and Herzegovina	456
Iran, Islamic Rep. of	402	Lebanon	449
Algeria	378	Thailand	441
Colombia	355	Turkey	432
Morocco	341	Jordan	427
El Salvador	330	Tunisia	420
Tunisia	327	Georgia	410
Kuwait	316	Iran, Islamic Rep. of	403
Qatar	296	Bahrain	398
Yemen	224	Indonesia	397
		Syrian Arab Republic	395
		Egypt	391
		Algeria	387
		Colombia	380
		Oman	372
		Palestinian Nat’l Auth.	367
		Botswana	364
		Kuwait	354
		El Salvador	340
		Saudi Arabia	329
		Ghana	309
		Qatar	307

 

Now you can slice and dice these numbers any way you like. There’s plenty to work with in just these two data sets. Looking at the tables, one can make the following statements without being too controversial:

 

• The top five nations’ students are all from Asia; the top four are from East Asian nations.
• U.S. fourth grade students are ranked 11^th out of 36 nations participating.
• The U.S. students’ rank goes up to 9^th out of 48 nations participating among eighth graders, but the average score is lower by 21 points.
• U.S. students trail behind several of our economic competitors in math scores, such as Hong Kong, Japan, and Russia, but also behind developing nations like Kazakhstan.
• It is difficult to determine the rankings of U.S. students compared to our largest economic competitors, the People’s Republic of China and India, as they did not participate in the survey. Of the nations ahead of the U.S., all of them have smaller populations.

 

Average science scores of fourth and eighth-grade students, by country: 2007

TIMSS scale average 500

Black Text = Nations participating in only one survey

Red Text = Nations participating in both

Blue Text = United States scores

Grade Four (TIMMS)		Grade Eight (PISA)
Country	Average Score	Country	Average Score
Singapore	587	Singapore	567
Chinese Taipei	557	Chinese Taipei	561
Hong Kong SAR	554	Japan	554
Japan	548	Korea, Rep. of	553
Russian Federation	546	England	542
Latvia	542	Hungary	539
England	542	Czech Republic	539
United States	539	Slovenia	538
Hungary	536	Hong Kong SAR	530
Italy	535	Russian Federation	530
Kazakhstan	533	United States	520
Germany	528	Lithuania	519
Australia	527	Australia	515
Slovak Republic	526	Sweden	511
Austria	526	Scotland	496
Sweden	525	Italy	495
Netherlands	523	Armenia	488
Slovenia	518	Norway	487
Denmark	517	Ukraine	485
Czech Republic	515	Jordan	482
Lithuania	514	Malaysia	471
New Zealand	504	Thailand	471
Scotland	500	Serbia	470
Armenia	484	Bulgaria	470
Norway	477	Israel	468
Ukraine	474	Bahrain	467
Iran, Islamic Rep. of	436	Bosnia and Herzegovina	466
Georgia	418	Romania	462
Colombia	400	Iran, Islamic Rep. of	459
El Salvador	390	Malta	457
Algeria	354	Turkey	454
Kuwait	348	Syrian Arab Republic	452
Tunisia	318	Cyprus	452
Morocco	297	Tunisia	445
Qatar	294	Indonesia	427
Yemen	197	Oman	423
		Georgia	421
		Kuwait	418
		Colombia	417
		Lebanon	414
		Egypt	408
		Algeria	408
		Palestinian Nat’l Auth.	404
		Saudi Arabia	403
		El Salvador	387
		Botswana	355
		Qatar	319
		Ghana	303

 

Again, from a high level, here are the facts as writ in the science category:

• The top four nations are again from East Asia.
• U.S. fourth graders come out 8^th out of 36 in science, while U.S. eighth graders 11^th out of 48 participating nations.
• U.S. students’ scores and ranking dropped between fourth and eighth grades.
• The top two nations’ students remained the top two in the four-year gap. All of the participating nations ahead of the U.S. in fourth grade remained ahead of the U.S. in eighth grade.
• The average grades for the sciences are all a few points lower than the average math scores.

 

Now I will step into the mine field.

What to Do About It All

The numbers are what they are. The controversy starts when we have to decide what to do about them. Let’s assume the U.S. has a vested interest in maintaining the largest, most advanced, and most prosperous economy in the world. Based on that assumption, one would also presume that we can and should do something.

 

As a purely empirical matter, we need to look at what the nations at the top are doing right, and determine what actions we can duplicate or translate to an American educational environment. More painfully, we must face realistically what things we’re doing wrong and either fix or stop doing them.

 

Singapore, for example, has an educational tradition rooted in Chinese Imperial examination practices, which emphasized long study and high expectations. That seems simple enough, and ideally all parents would have the same expectations for their children. However, are American parents prepared to accept schools that expect long hours of studying or administer corporal punishment (e.g. paddling, spanking)? Recall the outrage over the caning of an 18-year-old American for vandalism. Or, are educators prepared to focus on “readin’, writin’, and ‘rithmetic” and dispense with a “self-esteem building” curriculum? Would the best teachers be prepared to work at the worst schools with an expectation of improving things and being rewarded according accordingly? Is the federal government prepared to give up centralized control of educational practices to allow the states or local communities determine how best to meet the needs of their students?

 

It depends on how badly we want to improve and what prices we are willing to pay to remain competitive in the world.

 

But rather than look to Singapore or Japan, we can simply focus on the two best-performing American states, Massachusetts and Minnesota. What lessons might we learn from those states’ practices, and how far can other states go to duplicate their success, if feasible?

 

First, both are Northern, liberal (“blue”) states, where higher taxes and “big good government” are expected. These practices might go over fine in some areas, but would be quite a shift for anti-tax, limited-government “red states.” But the bottom-line translation might just be “You get what you pay for.”

 

What else might one investigate? Their curricula? The respective education departments can be found here for MA and here for MN. I leave that exercise to the reader. Their diversity? Unlikely—Massachusetts is 89% white, Minnesota 92%. Class size? Money spent per student? The numbers vary: $11,981 per student in Massachusetts, $9,138 per student in Minnesota (Alabama, where I live, spends $7,646 per student and had 18% of its students performing above average, as opposed to Massachusetts, which had 51% of its students performing above average). Physical infrastructure? More importantly, even if all other external factors were somehow magically made equal, could the results all be made equal nationwide? I’ll try to answer some of that last question in part two.

 

This seems to me like a great opportunity for Obama to demonstrate the strength of America’s diversity and creativity by decentralizing our educational establishment. After all, if we have one, centralized education system via No Child Left Behind and that system is causing our students to fall behind other nations of the world, then that system needs to be fixed. Or diversified. Or replaced.

 

Devolution would offer each state the opportunity to perform its own local experiments in educational achievement. Federal funding could continue, but control and “strings” would exist at the state or local levels. The internet would allow local school districts to observe or share teaching ideas and copy them (or not) elsewhere. Local control of curricula would also open the door to vouchers, allowing for more competition and fewer perverse disincentives to achievement, like cutting funding to struggling schools or good teachers not sharing their ideas for fear of losing merit pay to other teachers. If you think I’m kidding about the latter item, here’s what the NEA says about merit pay:

 

Merit pay systems force teachers to compete, rather than cooperate. They create a disincentive for teachers to share information and teaching techniques. This is especially true because there is always a limited pool of money for merit pay. Thus, the number-one way teachers learn their craft –learning from their colleagues — is effectively shut down. If you think we have turnover problems in teaching now, wait until new teachers have no one to turn to.

 

(Mind you, there are probably ways to measure “merit” based on training and cooperating with others, and every profession works within a “limited pool of money” for merit pay, but we’ll leave those arguments aside for now.)

 

Will all of the local educational experiments succeed? Probably not. However, parallel, diversified approaches would have to be better than one system failing everybody at once. Will the U.S. be willing to fix what needs to be fixed? Again, it depends on how badly we want to.

 

Bart Leahy is a technical writer living in Huntsville, Alabama. He blogs regularly at http://bartacus.blogspot.com.

 

Keywords: TIMMS, Trends in International Mathematics and Science Study, 2007 Report, U.S. test scores, academic achievement, science, technology, engineering, math, STEM, National Center for Education Statistics, ranking, 4^th grade, 8^th grade, merit pay, local curriculum control, decentralization, No Child Left Behind, NEA, National Education Association, Massachusetts, Minnesota