The Hunt for Hidden Variables

Physicists love to talk about ‘hidden variables’, most often to save their classically trained backsides from the vicissitudes of Quantum Mechanics. Even Einstein hoped that particle physics would someday identify hitherto unknown factors that would provide a classical explanation for quantum weirdness: “God does not play dice!”

Except, of course, he does! In fact, he’s Head Boss in the Craps Pit at Bellagio. Despite his rather Puritanical facade, there’s nothing God enjoys more than a good game of chance. But how do you get to experience the rush of ‘hitting 6 (or 8) the hard way’ when you’re supposedly omniscient and omnipotent?

In the prose prologue to the epic Book of Job, Satan challenges God to a wager and God cannot resist, even though it’s guaranteed to generate a heap of trouble for everyone concerned, including God himself. If there was ever someone in need of an intervention, and a referral to GA, it’s God!

Sidebar: God bets on Job’s character…and God wins! Job’s righteousness proves to be unshakeable. But Job is so righteous that he has no qualms about shining the light of Justice back on God himself. “Didn’t see that coming!” So…not so omniscient after all? Turns out, God’s gnosis ends where Job’s praxis (and yours) begins (aka ‘free will’). 

Ultimately Job calls out God for his callous disregard and, after a trial lasting almost as long as OJ’s, God changes his plea to nolo contendere. Accordingly he is required to restore Job’s assets, pay punitive damages, and absorb court costs.

Prior to his bet on Job, which backfired big time, YHWH went all in on the Big Bang. Right now, he’s ‘white knuckling’ it. Things have not gone exactly as hoped…but we’ve used up less than 12% of our allotted game time; so ‘better days are coming’…and they better come soon! 

But back to the hunt for hidden variables. Unfortunately for Einstein & Co., no one has put forward a persuasive hidden variable model to account for quantum weirdness. Worse, John Bell, et al. have proven mathematically that any viable hidden variable theory would have to satisfy some very narrow conditions. 

One such condition: God’s omniscience and omnipotence must include Job’s (and your) praxis – there can be no room whatsoever for free will. But we just demonstrated (above) that this is apparently not how things work in the real world. So we may just have to accept Alfred E. Newman’s Conjecture: Things are weird, so what, me worry?

But while physicists have failed miserably in their search for hidden variables, political scientists might be having better luck – a ray of sunshine may be peeking through the dark clouds of chaos theory. Perhaps social scientists will be first to identify the elusive factors that drive unexpected results. 

The job is two-part: (1) demonstrate that there are unidentified factors that drive unexpected results and (2) identify those factors. For example, election outcomes are notoriously hard to predict. Don’t get me wrong: it’s often possible to forecast results based on last minute snapshots (aka polls) that measure voters’ intentions ‘on election eve’. But it is impossible to deduce those results from objective (sociological) data.

You can ask would be voters who they plan to support on election day, and you may get a straight answer. But absent such a veridical declaration, there is no way you can predict voters’ individual or aggregate behavior. No matter how much you know about a voter, you cannot reliably predict how that individual will vote. 

Take the last 7 US Presidential Elections, for example. Some proved easy to forecast based on final polling, some not so much, but in no case did voters’ actual behavior validate our predictive models. Anyone who thinks they can predict how Hispanic males will vote in the 2026 midterm elections is whistlin’ Dixie. How come? The answer may lie in hidden variables. 

Political Science is based on certain assumptions: (1) people will vote according to their ‘best interests’ (as they perceive them), (2) people will vote for candidates they think will do the best job in public office, and (3) people will vote for candidates that agree with them on the issues.

Turns out, none of these is true! So we must search for hidden variables if we hope to understand the actual behavior of electorates. Our search spans 66 years and 300 miles, from an Irish Donnybrook in Boston to a Muslim-Jewish standoff in New York City. 

It’s 1959 and Boston University Political Science professor, Murray Levin, has decided to do a deep dive into Boston’s upcoming mayoral election. It seems a poor career choice. The election is widely viewed as a mere formality. The cognoscente have already sworn-in their candidate. A native of South Boston, President of the Massachusetts State Senate, and runner-up in the previous mayoral election, John E. Powers is the mayor presumptive.

But it’s Boston 1959, so the race will not be uncontested. Registrar of Deeds, John Collins, throws his hat in the ring along with several better known denizens of the under card. No matter, a campaign is de rigueur…but proforma. A primary is held and, as expected, Powers gets 50% more votes than his closest rival, who turns out to be Collins (above). November is shaping up to be a snoozefest. 

Fast forward 6 weeks, “Collins routs Powers!” - 56% to 44%. Powers carried only 4 of Boston’s 22 Wards.  Powers doubled his primary vote total, but Collins quadrupled his. Even more shocking, 10% of those who voted for Collins in the finale had voted for Powers in the preliminary. If Powers could have just held on to those voters, the contest would have been a toss-up. 

(If you’re looking to explore ‘why and how’ Collins was able to turn things around in just 6 weeks, you may want to look at my earlier piece on this race, Political Alienation.

On the weekend after the election, Professor Levin and his team interviewed 500 Boston voters, drawn from every corner of the City. Before 1959, demographics had always played the key role in Boston politics. And after the Collins era (8 years later), they would dominate Boston politics again, eventually catapulting the City onto the front page of every American newspaper (resistance to ‘forced busing’); but not in 1959. 

Levin’s study showed that religion, income, and ethnicity played no significant role in 1959’s results. Collins won the votes of Catholics, Protestants, and Jews (there were no avowed atheists or agnostics in Boston in 1959), high, middle and low income residents, Irish, Italian, Yankee and African American voters. He ran the table. Folks who wouldn’t normally share a sidewalk with each other shared a voting booth. 

It turned out that Boston voters had something in common that was stronger than their economic, religious and ethnic conflicts. According to Murray Levin, that mysterious X factor was ‘political alienation’.

Perhaps Professor Levin’s most astonishing finding was that the majority of voters in Boston’s Mayoral election that year did not think their chosen candidate would do a better job as mayor than his opponent. This tendency was especially prominent among Collins voters! The conclusion is inescapable: people voted for Collins, not because they thought he would do a better job than Powers, but for some other reason: et viola, a ‘hidden variable’! 

Fast forward to NYC 2025. Democratic Socialist, Zohran Mamdani, upset liberal former Governor, Andrew Cuomo in the Mayoral primary. Not close; in fact, similar to the Powers-Collins first round spread, 66 years earlier and 300 miles Northeast. Now they are scheduled for a rematch this November. The result is unpredictable, but already we see evidence of a hidden variable at work: there is a significant disconnect between voters’ personal positions on issues and their policy prescriptions for the City. 

A recent NYT survey asked voters to consider 4 issues: (1) raising taxes on the wealthiest New Yorkers, (2) hiring 5,000 more police officers, (3) strengthening rent control and focusing its benefits on lower income tenants, (4) making municipal bus service free to all riders. Surprisingly, almost exactly 70% of voters favored each of the first 3 policy changes and 60% favored the 4^th.  

Any self-respecting poly sci wonk would assume that a voter who favored higher marginal tax rates, more police on the streets, more robust and/or better focused rent control, and free service for all who ride the City’s buses would also think that the City should move to enact these policies. But that wonk would be wrong! 

Only on the subject of soaking the rich did voters’ policy prescriptions match their personal preferences: c. 70% of all voters favor increasing taxes on the wealthy and 70% think that the City should actually implement this policy. What seemed like a tautology amounts to front page news (or it would if there were still any ‘front pages’).

On the other three initiatives (more police, stronger rent control, and free bus service), voters’ public policy prescriptions did not match their personal policy preferences. It turns out, that in spite of a c. 70% personal policy preference, only 63% think the City should actually hire more police officers and only 61% want the city to make proposed changes to its rent control program.

Regarding free bus service, 60% personally favor the change but only 45% think the City should actually eliminate fares on City buses. How is that possible? 

As in Boston 1959, voters in NYC 2025 have something more in mind than personal policy preferences. This unknown quantity may well influence their votes on election day. Only problem: I don’t know what it is that’s top of mind with these voters or how it might influence their votes. 

I can conclude, however, that ‘scientists’ have finally demonstrated the existence of hidden variables…but it’s social scientists, not physicists!

***

Giovanni di Paolo’s The Creation of the World and the Expulsion from Paradise (1445) is a luminous Sienese panel that unites two episodes from Genesis in a single circular vision.

Inside a mandorla of radiant gold and stars, God appears as an architect of the cosmos, surrounded by concentric bands of celestial spheres and the zodiac, signifying His supreme ordering of the universe.

At the lower right, Adam and Eve are driven from Eden by a flaming sword, underscoring the fall from perfect creation to human exile and the enduring power of divine decree over both nature and human destiny.