Facts Follow Data, But Ideas Follow Intuition

The 19th-century physicist James Maxwell devised the famous Maxwell’s equations, and his achievement deserves special note not just physically but for what it meant to the domain of logical science, in terms of process.

That’s because he was the first person in physics to say “this is how it should be.”

Physics starts from observed facts and builds hypotheses to fit them. A theory is recognized as correct when the hypothesis and the facts clearly agree.

Maxwell “predicted” part of his equations partway through, without observed facts.

Simply put, they’re relational equations among electric field, electric flux density, magnetic field, and magnetic flux density, but once he’d verified things to a certain point, he is said (or maybe not said) to have declared “the rest must be like this, because that would be beautiful.”

In any case, the correctness of the equations was never proven in his lifetime; it was verified by others after his death.

This might not seem so unusual today, but in the physics of that era, everything started from observation as a matter of course, and Maxwell is said to be the first to take this kind of “prediction” approach.

But the history of bringing “correctness” and “beauty” into scholarship is itself old. Astronomy is a typical example.

Geocentrism, completed by Ptolemy before the common era and believed for more than ten centuries afterward, was originally proposed by Pythagoras in a philosophical context, with no observational facts or grounds whatsoever. And yet there was beauty in it: all the heavenly bodies moving in uniform circular motion around the earth.

Afterward, many observed facts attacked that self-righteous beauty, but scholars responded by repeatedly patching things together with this or that sophistry, all while insisting that geocentrism must remain the premise.

The ultimate result was Ptolemy, whose logic had grown so bafflingly complex that ordinary people struggled to understand it, and which still contained contradictions with many observed facts (but since there was essentially no sharing of observational data at the time, no proper criticism arose, and Ptolemy’s theory was broadly accepted throughout the world).

A man named Aristarchus had already proposed, at that time, a theory close to today’s heliocentrism, but it was ignored for more than ten subsequent centuries. This was the result of being too caught up in a self-serving “correctness.” Heliocentrism is beautiful too.

Sometimes intuition hits the mark.

According to brain science, intuition arises when a particular synaptic connection, trained tightly and routinely, sends an electric current cleanly down that finely honed route in response to a given input. In other words, intuition means “an answer sharply derived from experience and daily accumulation,” never a mere guess.

Some baseball players answer in interviews with something like “I could see the ball’s path clearly, so I read it as high and away, and I swung as hard as I could,” but a swing, including the preparatory motion, is said to take at least 0.3 seconds. Meanwhile a 150 km/h pitch takes 0.4 seconds to travel from the mound to home plate. In terms of human time perception, someone with resolution finer than 0.1 seconds is said to have sharpness beyond ordinary people. A common training exercise for sharpening visually driven reflexes measures how fast someone can touch a lamp that lights up on a board. Even for that simplified a movement, ordinary people take more than 0.3 seconds. Even a thoroughly trained athlete apparently can’t get under 0.1 seconds. Putting this all together, “watching the ball’s path and swinging” is no longer humanly possible. In other words, the batter really can’t see the ball. There may be some final adjustment in the last instant of the swing (itself a superhuman feat), but in reality the batter has already predicted the pitch type, speed, and course before the ball even leaves the pitcher’s fingertips — even without being consciously aware of it. This too is intuition, obtained as the accumulated result of years of practice and the study of countless pitchers’ habits.

Intuition shouldn’t be underestimated.

On the other hand, observed results are not always correct. That’s because obtaining data itself involves any number of difficulties in the process.

The 17th-century astronomer Picard calculated the diameter of the sun, a pending question of the time, but 300 years later, when his achievement was reviewed, the precision of the measuring instruments of that era was called into question. Fortunately, it was proven that the micrometer built by a researcher named Auzout, who was closely acquainted with Picard at the time, had extremely good precision, and the matter was resolved without incident. Still, this had been a serious crisis: had any fault been found there, Picard’s great achievement itself would have been blown away.

Have you ever seen a “Stevenson screen” at an elementary school? (They apparently still exist in some places even now.) Inside those boxes are instruments measuring temperature, humidity, and so on, keeping records. They’re placed at elementary schools because, for measurements from a Stevenson screen to maintain a certain level of accuracy, the surroundings need to be free of obstacles and have moderately good airflow, and the schoolyard was an easy candidate for “a broad open space that’s usually available within a given area.”

Global warming is much discussed these days, and according to the IPCC, global mean temperature has risen by 1°C over the past 150 years. Stevenson screens ultimately have accuracy issues, apparently producing errors of more than ±1°C. They’re also installed at roughly human eye level, which tends to read high unless corrections are made for the heat island effect, said to have raised temperatures by 3°C in just the last half century.

In the end, today’s final temperature figures come from integrating results across various methods, including satellite measurements. In other words, there’s little doubt that room for doubting recent temperature figures has shrunk considerably. But even granting that, if you look at observed results limited to just the last 15 years or so, warming doesn’t appear to have progressed much, which presents a bit of a dilemma (though this is more a matter of politics than science).

Careful attention also needs to go to how data is presented and how it’s read.

The famous “hockey stick curve” from the IPCC’s Third Assessment Report showed a shocking result: combining paleoclimatology research with modern measurements made it look as though, from the latter half of the 20th century, dramatic warming unprecedented in human history had suddenly set in. But the paleoclimatology in question estimates past temperatures from analysis of isotopes such as carbon contained in particular sediments or tree rings from various locations, and compared with modern measurement methods that comprehensively measure the temperature of the entire earth’s surface from satellites and produce an average, it’s inferior in accuracy and naturally contains a certain degree of error (estimating the past, for which there are no accurate records, requires piling up inferences at a level resembling “if the wind blows, the bucket maker profits,” and is, needless to say, extremely difficult work).

Michael Mann, who presented the data, presented it fairly as a scientist, margin of error included. But simply drawing a somewhat thick line smoothly through around the center of that margin of error on the graph instantly produced a terrifying curve.

Mann was subsequently accused from various quarters of having “deliberately manipulated the impression of warming,” but he sued for defamation and won. Of course he did. The graphs shown by later research indeed weren’t hockey sticks, but they “generally fell within the upper and lower bounds of the margin of error that Mann had shown.” What Mann presented may have been misleading, but he hadn’t lied. And the margin of error, too, had been properly shown.

In other words, a formal ruling held that if there had been excessive incitement, the responsibility for that lay with the interpretation of those who received the information.

Afterward, in its Fifth Assessment Report, the IPCC began showing multiple overlaid estimated datasets for past temperatures, which led to recognition of the Medieval Warm Period, warmer than today. It’s also become known that a small-scale cold period preceded the accelerated warming.

Rewinding a bit, even accounting for these multiple estimated datasets, it remained undeniably true that warming had progressed over the past 100 to 150 years or so. But even so, the deep rift caused by that one instance of a mismatch between how data was presented and how it was read still runs strong and can hardly be said to have been fully resolved (the emergence of President Trump in America is not entirely unrelated to this story).

Problems have also been pointed out with Mann and his colleagues’ analytical methods themselves, but this is strictly a discussion within the domain of science or mathematics. In fact, the IPCC has faced criticism on points such as the unclear background and certainty of the data it adopts, or cited papers being limited to a small number of authors, and it’s gradually moving toward improvement.

Believing without limit in the omnipotence of data is, one could say, far too naive and dangerous.

Data is what best represents facts. That much is certain. But at the same time, we mustn’t forget that all data is always “something worth doubting.”

Intuition often lacks logical grounding and is difficult to prove or reproduce. It’s extremely hard to handle.

But at the same time, we mustn’t forget that various kinds of intuition are built on the accumulated experience of oneself and others up to that point.

What should we do when something we recognized as “fact” turns out to be wrong? Thinking rationally, we should promptly acknowledge it, fix what needs fixing, and move on. Nothing is more wasteful than continuing to keep company with something known to be wrong.

But what should we do when something we held as “belief” turns out to be wrong?

Take the earlier example of geocentrism: many geocentrists grasped it not as “mere fact” but bound up with philosophical questions about how the world came to be, truth, or what a human being is, what we are. The despair when that was overturned is beyond measure.

Belief and data should not be tied together. Doing so risks damaging belief for the sake of data, and damaging data for the sake of belief.

Picard’s achievement made accurate measurement of longitude possible, and by enabling measurement of the distances between planets, it became the foundation for elucidating the workings of the solar system; Newton, too, is said to have referred to Picard’s measured data in deriving the law of universal gravitation. But depending on the precision of the micrometer Auzout made, all of that could have been ruined.

On the other hand, of course, intuition and fact should not be tied together either. There’s nothing particular to add on that point.

When do these gaps arise? They seem to occur when perceptions of “settled fact” are misaligned, whether between people or between society and the individual.

To put “settled fact” another way, more simply: something that ought to be obvious.

Suppose, for example, that the prejudice (and I do think it’s a prejudice) that “the rich are happier than the poor” is presented as something that ought to be obvious. Or suppose that kind of atmosphere pervades society. Someone presents data: “when you quantify life satisfaction for the bottom 10% and top 10% by income, the top 10% clearly show higher satisfaction.” This is data stating a perfectly accurate fact. But the problem with this data, as you’ll quickly notice, is that all we have here is data on the extremely rich and the extremely poor. In other words, it hasn’t proven that “there is a positive correlation between income and happiness.” In a sense, it’s data that merely restates the proposition itself. So the proposition above tells most people next to nothing.

And yet it is a fact.

If, because of that weight, it ends up exerting some influence on belief, that would undoubtedly be unfortunate. A fact is a fact. It has nothing to do with belief.

Let’s bring in a different fact for comparison. According to a famous experiment in behavioral economics, once annual income exceeds a certain level, happiness apparently loses its correlation with income. This is the so-called limit of monetary incentives. And this “certain level” threshold is by no means something like an annual income of hundreds of millions of dollars — in the experiment at the time, it was $120,000 a year. Factoring in the income gap between America (the West Coast) and Japan at the time, that would come to somewhere around $64,000 to $73,000 (though the gap has widened further since then). Given that the average annual income for a typical salaried worker is around $41,000, and about $55,000 on average at listed companies, $64,000 to $73,000 is indeed high. But considering that this average includes brand-new hires just joining a company, and that the average income limited to college-educated salaried workers in their 50s exceeds $55,000, it’s arguably not such an unreachably high income after all. So this stagnation, or even reversal, generalizes to some degree. This is why it’s commonly said that “a life worked to the bone at an annual income of around $91,000 is the toughest of all.” The tax system, too, is cleverly designed, so that once you account for taxes and benefits, take-home pay doesn’t differ much between an annual income of around $91,000 and around $64,000. What this means is that it’s entirely possible for there to be almost no monetary benefit to offset the pressure of “having a high income” and the social responsibility that comes with the burden it imposes. So in this range, the hypothesis that income and happiness actually have a negative correlation even becomes tenable.

This too is a fact.

Fact like this, and another fact, and yet another fact. Even after piling up fact upon fact this way, does it actually get us any closer to belief? If anything, the substance only seems to grow blurrier, like layering on grimy sheets of cellophane.

Belief, meanwhile, would be the answer to a question like this: Can raising our income actually make us happy? Are we made happy by economic growth based on neoliberal market principles?

For this, human intuition seems more useful, if anything. It might also help to recall someone close to you who’s no longer of this world. How did your grandfather’s life actually turn out? How about your mentor’s life?

Also, an average is nothing more than an average across the whole, and doesn’t necessarily mean you fit into it. If anything, more people fall outside it than within it. Statistics, too, only show an overall tendency, and there’s naturally a chance — more than enough of a chance — that you’re among the minority. Statistical data only tells you something about the “whole”; it tells you nothing about “you.” You can see where “your” data falls statistically, but that’s of no help whatsoever in deciding where you’re headed, what you want to achieve, how you want to live, how you want to die.

“When in doubt, follow your heart” turns out to be surprisingly apt wisdom.

Originally published in Japanese at https://clazytech.com/2021/06/463/. Translated with LLM assistance and reviewed before publication.