Political pollsters are pretending they know what's happening. They don't.

This week I have to explain a little statistics, because the political news media have failed so badly.

If you’re like me, you’re probably exhausted with all the so-called news stories about the latest U.S. presidential polls. After each poll, which seems to be a daily occurrence as the election draws near, news outlets report the results on their front pages with breathless excitement. The race is changing! Harris is pulling ahead! Trump is catching up! Swing states are swingier than ever!

To cite just one example, CNN just reported that “Polls show Harris’ numbers in Pennsylvania have shifted over the past month.” This story included a video featuring polling expert Harry Enten, who stated “there has never been a race this close in the polling since 1972!” OMG! (The “OMG” is from me, not Enten.)

Okay, now let’s get to reality (with a small dose of statistics).

A poll is nothing more than a tiny sample of voters’ opinions. A poll might ask, say, 1000 people who they plan to vote for, and then report the results. If you ask a truly representative sample of people, a poll can give you a pretty good idea of how the candidates stand.

The problem is, this is really hard to do accurately, and it’s become much harder since everyone switched to cell phones. Decades ago, pollsters would phone people, and those people would answer the phone. Not any more: many people won’t answer a call from a number they don’t know, and people who do answer might have a bias towards one party or the other.

So pollsters compile some numbers and then adjust them (more on that below), and then report the results with a “margin of error,” which goes something like this. Suppose that a poll finds Trump leading in Nebraska by 18%, with a margin of error of 4%. That means that he might be leading by anywhere from 14% to 22% – and that’s just one pollster’s guess. He might be leading by 30%, or even losing, if you ask a different pollster.

But a 4% margin means that if you run the poll again and again, the results will swing back and forth, randomly, in a pretty wide range.

That’s exactly what’s happening in Pennsylvania. The voters are split, but every poll shows a slightly different result, because that’s what random sampling does. It’s not news!

So let me give you some actual data. Let’s look at Pennsylvania because it seems to be the closest Presidential race, according to the polls. Here are the actual margins of victory from the past 9 elections - real numbers, not polls:

• 2020: Biden (D) won by 1.2%
• 2016: Trump (R) by 0.7%
• 2012: Obama (D) by 5.4%
• 2008: Obama (D) by 10.3%
• 2004: GW Bush (R) by 2.5%
• 2000: Gore (D) by 4.2%
• 1996: Clinton (D) by 9.2%
• 1992: Clinton (D) by 9.1%
• 1988: GHW Bush (R) by 2.3%

Clearly, Pennsylvania has been closely divided in recent years. Suppose the race this year will eventually be won, by either candidate, by less than 2%, as happened in 2016 and 2020. Then what would we expect polls to show us? Given their typical 4% margin of error, we’d expect polls to show a race that flips back and forth from one poll to another, even if no one is changing their mind.

And that, I argue, is just what we’re seeing. The polls aren’t accurate enough, statistically speaking, to tell us anything other than “we don’t know who will win Pennsylvania.” But the media reports each one as if it’s a revelation.

Now back to those “adjustments” that pollsters make. In 2016 and 2020, the polls were off by quite a lot. In 2016, as everyone knows, pollsters were highly confident that Hillary Clinton would win, and they were wrong. They were almost as confident that Joe Biden would win in 2020, and they were right–but the race was closer than they predicted, and their estimates were once again off. In the 2022 midterm elections, though, they were wrong again, but in the opposite direction, and Democrats did better than forecast.

Did they over-correct after 2020, and is that why they predicted much bigger Republican gains in 2022? And have they fixed that now, so that polls this year will be spot on? Who knows?

The thing is, it’s very hard to figure out who will actually show up to vote, and who might answer the phone when a pollster calls, and whether they’re even telling the truth. So pollsters make statistical adjustments based on past experience, weighing some voters more than others. In general, they don’t tell the public precisely how they do this.

Are these adjustments accurate? Well, here’s the kicker: we won’t know until after the election! But one thing is almost certainly true: the changing polls are overstating the number of people who are changing their mind.

So I have a suggestion to the media: stop reporting every poll as if it’s news. Instead, tell us where the candidates stand on issues that really matter: support for Ukraine, support for Israel, health care policy, immigration, respect for the rule of law, stuff like that. I know, crazy stuff, right?

I realize that my plea will fall on deaf ears. It’s so much easier for The Washington Post, CNN, The New York Times, Fox News, and others to write about and talk about polls, and pretend these are actual news. It’s also lazy.

The problem with our democracy isn't gerrymandering. It's integers

(This week, in the wake of the recent U.S. elections, I take a detour from my usual topics to apply a little math to our election system.)

As everyone knows, the U.S. Congress has grown increasingly un-representative. We have states where the population is evenly split among Democrats and Republications, but where–thanks to partisan gerrymandering–the number of House members is grossly skewed in favor of one party. Even without gerrymandering, voters for the losing side in many House districts feel, justifiably, that they have no representation in Congress.

The solution is surprisingly simple. We can save it with a little math. It's not even that complicated. The problem, as I explain below, is integers.

The New York Times just published an argument that the House of Representatives is too small. They point out that the House grew every decade until 1911, when its size was frozen at 435. The Times says that after 100 years of population growth, "America needs a bigger House."

Fair enough, but how do we fix it? The Times argues, oddly enough, that the right number is 593 representatives. Why? Because 593 is the cube root of the total U.S. population. Curiously, many other democracies follow a cube root rule, first described in this 1972 paper by Rein Taagepera. The legislature of Denmark, for instance, has 179 representatives for a population of 5.77 million, and 179 cubed is 5,375,339. Canada has 37 million people and 333 members of their House of Commons, a near-perfect example of this rule, if you ignore their 105 senators.

The first thing to point out is that the NY Times got the math a little bit wrong. The current U.S. population is 329 million, for which the cube root is 690. So if we keep the Senate at 100 members, then we need 590 Representatives in the House, not 593. But the Canadian model doesn't count their senate, so perhaps we need 690 Representatives. But that's a small quibble.

The real problem, though, is that expanding the House by 35% won't address the fundamental problems of our democracy. The Times observed, correctly, that a single representative can't stay in touch with 750,000 people. Increasing the size of the House to 593 will reduce that number to 550,000, which will hardly help. The framers of the Constitution wanted one representative for every 30,000 people, by the way, but that would yield a ridiculously large House today.

The real solution is to get rid of our reliance on integers. Let me explain.

The root of our problem is that each Congressional district elects just one person, in a winner-take-all election where you only need to win by one vote. This means that the losers end up with a Representative who simply doesn't represent them. This means that, in a close election, 49.9% of the voters can be effectively disenfranchised. Even in lopsided victories, where 70% of the voters support the winner, the remaining 30% are stuck with someone who doesn't represent them.

The solution: elect TWO representatives from each Congressional district, and award them each a fractional vote in Congress. Each of the top two vote-getters would have a Congressional vote that is proportional to the number of voters who supported them. Thus if a district elects a Democrat (D) with 55% of the vote, and the losing Republican (R) gets 45%, both of them go to Congress, and D gets 0.55 votes while R gets 0.45 votes.

This will double the size of the House, to 830 members. It will also completely fix partisan gerrymandering. Here's why: imagine a state that is 50-50 Democrat and Republican, but that has packed one district so that 80% of its voters are Republican, allowing it to create three majority-Democratic districts that are 60-40 in favor of D's. Under the current system, that state has 3 Democrats and 1 Republican in Congress. (We have many states that look just like this under our current system.)

Under my new system, our hypothetical state would send 4 D's and 4 R's to Congress. The R from the "packed" district would get 0.8 votes, and the R's from the other three districts would get 0.4 votes each. The entire state delegation would therefore have 0.8 + 0.4 + 0.4 + 0.4 = 2 Republican votes, and 0.2 + 0.6 + 0.6 + 0.6 = 2 Democratic votes, accurately reflecting the overall population of the state.

Gerrymandering is nearly impossible in this system. Packing voters into one district would simply increase the voting power of the majority member for that district, while reducing the voting power of other members of the same party by a corresponding amount.

My system is perfectly legal, and Congress could create it with a simple bill, just as they increased the size of the House in the past. No Constitutional amendment is necessary.

What if more than two people are running for a House seat, as is often the case? We could divide the single House vote proportionally among the top two vote-getters, ignoring the third parties. (States could also use ranked-choice voting to re-apportion the votes of the losing candidates.) A nice side effect is that "protest" votes for third parties wouldn't have such a devastating effect on either of the top candidates. What if only one person ran for a seat? Easy: he or she would get a full vote in Congress rather than a fractional vote. What if the top two vote-getters were from the same party? No problem there, they would both go to Congress, and their party would get a full vote from that district.

Of course, this would make counting votes in the House a bit more complicated. The majority and minority whips wouldn't be able to simply count integers; instead, they'd have to add up the fractional votes of their 435 members. But why should we limit ourselves to a voting system that only uses first grade math? In the U.S., fractions and decimals are covered by the fourth grade. I think Congress can handle that.

There. I've now fixed our democracy. Time to get back to science.

Governor Scott Walker's attack on academic freedom in Wisconsin

Wisconsin Governor Scott Walker really doesn’t like professors. He seems to have a special grudge against the University of Wisconsin, against which he has launched a two-pronged attack this year.

It’s rare for a governor to attack the flagship university in his own state. Governors usually do just the opposite, promoting their universities to the rest of the world whenever the opportunity arises. 

What's more, Walker is currently campaigning for the Republican nomination for President. Presidential candidates usually try to make new friends and influence voters. It’s rare for a candidate to single out a large non-political group (other than foreign enemies of the United States, or other obvious bad guys) and systematically go after them.

Thus it’s surprising–startling, really–to observe how Scott Walker is waging a war on academic freedom in his home state of Wisconsin. Let's look at what he's doing, and then ask why.

First, back in January he proposed an enormous $300 million cut to the University of Wisconsin’s budget, at a time when other state universities are finally recovering from the recession. Now he’s proposing to get rid of academic tenure, not only threatening faculty jobs but also destroying academic freedom for professors at the University of Wisconsin.

Walker’s attack on tenure was just endorsed by a major committee in the state legislature, which voted 12-4 to eliminate tenure from state law. The Wisconsin faculty responded that this new policy, if implemented,

“will inflict lasting damage on a highly successful institution that was built and nurtured with major investments by Wisconsin taxpayers over a period of 167 years.... It would be difficult to overstate how destructive and unnecessary the [legislature's] proposed changes to tenure and shared governance are.”

Why is Governor Scott Walker (aided by his legislature) attacking his own state’s leading university? One could hypothesize that he harbors some resentment over the fact that he himself never graduated from college: he quit school in his senior year at Marquette University. Susan Milligan at US News argues that this disqualifies him as a candidate; perhaps this criticism bothers Walker. But plenty of people succeed in demanding careers without a diploma–just look at Bill Gates and Steve Jobs.

Perhaps the reason behind Walker’s dislike of academia is that he thinks that professors are too liberal. If true, this is deeply disturbing: it means he wants to stifle speech that he disagrees with. This kind of repression of scholarship is one of the most important reasons for tenure in the first place. One doesn’t have to look far–hello, Vladimir Putin?–to find examples of how powerful politicians can suppress speech, to the detriment of their societies. As UC Irvine’s Mark Levine wrote this week,

“none of academia’s core functions could occur without tenure and the assurance of academic freedom it enables.”

I wrote to Governor Walker's office to ask the question above, and also to ask if he thinks his actions will weaken the University of Wisconsin. His press secretary, Laurel Patrick, didn't answer directly, but responded that 

"the Governor’s original budget proposal removed all references to the UW from state statute in order to allow for the proposed authority to create its own policies.  This would allow the UW Board of Regents to address the issue of tenure going forward."

The leaders of a national university governing board association disagreed, pointing out that under Walker's proposed new policy,

"decisions about a tenured faculty member's service could be based less on performance and institutional finances and more on the political or personal views of board members."

I’ve observed the benefits of tenure directly many times, both at Johns Hopkins University and at my previous academic home, the University of Maryland. While at UMD, for example, I wrote several articles highly critical of the university president for his boneheaded decisions about the football coach. Many of my colleagues expressed similar views to me in private, but the untenured ones were unwillingly to speak openly. If we want scholars to speak truthfully, they need to be free of fears of retribution.

Governor Walker’s actions make even less sense when viewed from outside the state, where the University of Wisconsin is considered one of the nation's top public universities (currently ranked 13th among public schools). With his draconian budget cuts and his assault on the tenure system, Walker is sending a message that professors at Wisconsin should sit down and shut up. Some of them–those most able to move, which likely includes some of their best talent–might now be looking for greener pastures elsewhere. Come to think of it, we are recruiting for 50 new endowed professorships at Hopkins, thanks to Michael Bloomberg; perhaps I should be thanking Governor Walker.

It’s disturbing that Wisconsin's governor is using his power not only to weaken one of the state's biggest assets, but also to attack the free expression of ideas. I can't come up with any explanation for his actions that doesn't appear vindictive and short-sighted. This isn’t the kind of behavior I want in any politician, and certainly not in someone who wants to be the most powerful politician in the nation.

Government ranking of U.S. universities: a truly bad idea

The U.S. Secretary of Education, Arne Duncan, has come up with a plan to produce official government rankings of our universities.

The plan was announced this past August, and over the past month, the Obama administration has been holding public forums around the country to get input about its plan.  But it seems like they’ve already made up their minds.

I hope not.  This is such a bad idea I don’t know where to start.  But I’ll start.

First of all, we have multiple rankings systems already, including the highly regarded U.S. News college rankings, which millions of students, faculty, and administrators use every year.  Even though everyone loves to complain about it, U.S. News is pretty darned good: they rank colleges in many categories, by region, by specialty, and more. They also rank graduate programs and professional schools such as law and medicine.

If you don’t like U.S. News, there are several other rankings, including the more recently established World Rankings from Shanghai and The Times Higher Education rankings. These are excellent rankings, well-documented using multiple criteria, and not nearly so US-centric. All these websites are chock-full of useful data about hundreds of universities.

I know, I know: “ratings aren’t rankings” as Ben Miller wrote recently at the Inside Higher Ed site. But I’m not at all confident that the proposed ratings won’t turn into a ranking system, especially with the weight of the federal government behind them.

So we don’t need a federal ranking of universities. That’s the first problem.

Second, this push to create official ratings will inevitably lead to a new bureaucracy within the Education Department, which will then create a constituency that will fight to keep itself in existence. How many staff will Secretary Duncan hire to create these rankings? Dozens? Hundreds?  And how many university employees around the country will then have to be hired to answer whatever questions the government asks?  This seems like it could quickly become a very expensive proposition, running in the tens of millions of dollars annually, or perhaps even more. I haven't seen any estimate of how much this new system would cost, but I'm betting on a lot. What will we cut from the federal budget to create this new system?

Third, a ranking system will likely spawn a host of new requirements that universities will have to satisfy. Why? Well, primarily because federal financial aid to universities will be tied to their ratings. Thus it’s pretty clear that universities will do whatever they can to keep the feds happy. And I’ve no doubt that with a full-time bureaucracy in place, the federal raters will keep moving the goalpost - coming up with new measures that in turn will spur new costs throughout academia. I’m highly skeptical that any of these government metrics will lead to better education.

We’ve already seen what government scorecards do in our public education system. Thanks to the No Child Left Behind program, we now have incessant testing of students, beginning in elementary school, and thousands of hours devoted to teaching students how to take tests rather than learn new material. Schools have not improved as a result. Do we want this trend to creep into colleges too?

I’m not the only one who thinks this a bad idea. Janet Napolitano, the president the University of California system and former Secretary of Homeland Security under President Obama, told the Washington Post that she is “deeply skeptical” of the criteria that a federal ratings bureau would develop.

“It’s not like you’re buying a car or a boat,” said Napolitano.

Secretary of Education Arne Duncan has already criticized the critics of the new rankings system, calling the criticism “premature and a little silly.”  Duncan emphasizes the need to address the alarming number of college students who default on their student loans. This is certainly a problem, but a college ranking (or rating) system is not the solution.

Perhaps the biggest problems with student debt is the rapid rise in mediocre, for-profit online colleges. If the feds want to get the loan problem under control, they should stop funneling money to these Yugos of higher education. As the PBS show Frontline pointed out in 2010, for-profit universities are

 “churning out worthless degrees that leave students with a mountain of debt.”  

And they’re not cheap, either - the GAO found that

“tuition in 14 out of 15 cases, regardless of degree, was more expensive at the for-profit college than at the closest public colleges.”

So yes, we do have a problem with student debt. One solution would be to exclude truly bad colleges, which are responsible for a disproportionate share of student debt, from federal aid. But that would mean naming the bad apples, who in turn will claim that the government is somehow being unfair. Perhaps the new ratings are an attempt to be fair, but it just makes no sense to rate everyone in order to identify the worst universities. Having a federal government agency produce college rankings is just a bad idea.

How to predict an election? Ask the math geeks.

Mark Newman's rendering of the 2012 U.S. election,
 weighted by population

It's time for a bit of gloating.  No, not for Democrats over Republicans, though I'm sure that's going on.  It's time for the math geeks to throw a bit of scorn at those insufferable, over-confident frat boys who call themselves political prognosticators, and who spent most of the past two years telling us that they knew how the election would turn out.  They bloviated endlessly on talk shows, explaining why their favored candidate would win, and how he would do it.  

Politicos behave just like promoters of quack treatments when things go wrong: they always have a ready answer, and somehow their "theories" can never be proven wrong.  It seems that the only thing these guys are really expert at is getting themselves onto talk shows. Now that the election is over, let's hope that happens a bit less often.

Instead, pay attention to the math geeks.  The statisticians and analysts who build mathematical models based on multiple polls and other data absolutely nailed this election.  Nailed it!  Nate Silver at FiveThirtyEight blog predicted the winner of the presidential election in all 50 states.  So did Sam Wang , a biophysics professor at Princeton, over at the Princeton Election Consortium blog.  And so did Simon Jackman, a political science professor at Stanford who writes for HuffPo.  Nate Silver first drew everyone's attention in the 2008 election, when he correctly predicted 49 out of 50 states.  Last week's success shows that this is not an anomaly, although it has the mathematically challenged pundits in a tizzy.

Hopkins statistics professor Jeff Leek wrote a nice explanation of how these models work over at Simply Statistics, so I won't explain it here.  Suffice it to say that mathematical models don't work by chattering with their buddies at political rallies.

Mathematics delivered the goods.  And make no mistake, this is the way of the future.

That hasn't stopped the punditocracy yet.   On election night, Republican hatchet man Karl Rove was sputtering on Fox News that Romney could still win, after Fox News itself - which is little more than a media arm of the Republican party -called Ohio and the election for Obama.  Rove, who predicted that Romney would win with 285 electoral voites, also orchestrated the spending of over $127 million on Romney, not to mention his spending on 12 Senate candidates, 10 of whom lost.  Has he admitted he did anything wrong?  Nope.

Rove wasn't the only one wrong.  As Techcrunch pointed out, 

"every single major pundit was wrong - some comically wrong."  

The Atlantic created a detailed score sheet listing all the pundits and their predictions of the overall winner, the electoral college total, and the winner in all the swing states.  And indeed, even those who predicted correctly that Obama would win got most of the swing states wrong.

Here's what needs to happen.  The television networks need to realize that political expertise is meaningless when it comes to making statistical predictions.  Let's treat political forecasting just like weather forecasting, using models that are demonstrably accurate (such as Silver's).  Television stations can hire attractive political "forecasters" (because physical appearance matters on TV, like it or not) who will describe the latest forecasts just like today's weather forecasters do.  Now that I think of it, why not let the weather forecasters do both jobs?  We already have them in place at every local TV station in the country.  Think of all the money the networks will save.

But what about all that air time they need to fill with talking heads arguing about who will win elections?  Well, this makes about as much sense as having two self-proclaimed experts arguing about whether it's going to snow this weekend.  Maybe they can find real experts who will argue about issues rather than about who's ahead in the polls.

Ha ha, just kidding!  Who wants to hear about issues?  But if you must know how the race is going, ask the math geeks.

The U.S. enters the 20th century (or: why our electrical grid is a failure)

No, the title of this article is not a typo.  We did enter the 20th century - the early 20th century - last week.  All it took was a thunderstorm.


Sometimes I am struck by how phenomenally shortsighted our national leaders are.  In just 45 minutes last week, a single storm brought the capital of the U.S. to its knees for nearly a week.  Millions of people were left in the dark, with no power, no telephones, no internet, and no air-conditioning during the worst heat wave in years.  The damaged stretched from Ohio east into Maryland and Virginia.  A national emergency?  Yes.  The response from the federal government?  Nothing.  Nada.  Zip.  The local governments and the privately-owned power companies were left scrambling, as usual, without the resources to fix things.

How is this possible?  Simple: unusually strong winds during the storm, up to 70 mph (112 kph), knocked down huge trees everywhere.   In this part of the U.S., like most of the country, all the power lines are above ground, strung between flimsy wooden poles, exactly as they were over 100 years ago.  So just like that, in the blink of an eye, the high-tech region around the nation's capital was tossed back into the early 20th century, when horse-drawn carriages ruled the day and when the only form of air-conditioning was shade.

This is pathetic.  How can it be that we haven't put our valuable infrastructure - the power and communications network - underground, where a common thunderstorm can't touch it?  This debate comes up after every storm, as it has this week (especially in the Washington and Baltimore region), and the answer is always the same: it's too expensive.  The power company immediately responded to a few tentative suggestions that the lines should be underground, lashing out that those who were making the suggestions were naive, or didn't realize how costly it was.  The power company then blamed the victims, saying that local governments and residents make it difficult for them to trim the trees properly.

Tree trimming?  Are you kidding me?  How exactly were you planning to trim the 60-foot oak that fell in my back yard, snapping the power lines feeding our neighborhood?  Or the countless other huge trees that crashed through power lines?

The problem we face will never be solved by the power company.  It costs money to put lines underground, lots of it.  Sure, the costs of these disasters are far, far greater than the cost of "undergrounding", but those costs aren't paid by the power company, so why should they care?  They don't.

So yes, it is too expensive for the power company to put lines underground, so they will never, ever do it.  Not on their own dime, that is. Local governments won't solve it either: they are just too small and too poor.  We need a national effort to put our valuable, all-too-vulnerable power and communications lines underground - everywhere.  

What, the U.S. can't afford it?  It's true, we have a massive debt that is getting worse each year.  But somehow we can afford to build roads and other infrastructure in Iraq and Afghanistan.  Our national leaders seem to think the infrastructure in those countries is more important than in our own backyard.  This is just nuts.

Here's an argument that might just reach our political leaders.  Losing power and communications is a national security issue.  The storm last week caused far greater disruption than any terrorist could ever hope to achieve with man-made devices.  Our enemies don't need to attack us: they can simply let us continue to spend our money bombing other countries and then rebuilding the infrastructure in those countries.  Ironically, this was Reagan's strategy with the Soviet Union: force them to spend themselves out of existence.  It worked.

We need a discussion on this at the national level, where priorities need to be re-set to recognize the antequated state of our own infrastructure.  Over the past decade, we've spent hundreds of billions of dollars destroying much of the infrastructure in Iraq and Afghanistan, and many more billions re-building some of that infrastructure.  Then when a storm hits our own capital, government and industry representatives, and multiple columnists in the media, say it's "too expensive" to put the lines underground.

And by the way, the investment to fix our power infrastructure might be large, but overall it will save money, but only if you account for the costs to everyone (not just the power company).  The American Society of Civil Engineers estimates that 

"service interruptions and capacity bottlenecks that … will cost households $6 billion in 2012, $71 billion in total by 2020 and $354 billion in total by 2040. Businesses will pay $10 billion in 2012, $126 billion in 2020 and $641 billion by 2040 in avoidable costs."

To avoid these losses, the ASCE estimates we need $11 billion in additional investment per year through 2020.

The Department of Homeland Security, created in the panic following the 9/11 attacks, hasn't taken any responsibility for putting our national power and communications infrastructure underground.  On the contrary: power crews from Canada who were coming to aid us down here in Maryland and Virginia after last week's storm got held up at the border by our own DHS personnel, delaying some of the crews by many hours while people sweated, food spoiled, and tempers boiled.  Let's stop all this phony security theater and start rebuilding our country.  Shut down the entire DHS if they can't even keep the lights on.

And let's bring our troops home now, today, and put them to work fixing our crumbling infrastructure.  Countless news stories over the past few years have lamented the difficulties that our veterans have finding jobs when they return home: well, there's plenty for them to do.   And fixing our infrastructure - undergrounding power lines, building new roads and bridges - is a far better use of our money than blowing things up.