Clay Tech

"clay-works make things real"

translated from clazytech.com

Nobita and the Lovely Valley Between VR and AR

This article has been migrated from a post originally written in September 2016 on my old blog, mixture-art.net. The writing and information are preserved as they were at the time.

Something I dug up from a few years back.

Some time ago I had the chance to meet an AR specialist. “It wouldn’t do to say something stupid and be rude,” I thought, so I studied up a bit on “what is this AR thing anyway” beforehand.

I may look like a lone wolf who does his own thing, but I’m actually surprisingly considerate, see

But it’s an engineer’s nature to get stuck on some small detail in the middle of studying. “I wonder if AR will eventually be used in combination with VR…” Once that thought crossed my mind, that was it. The long piece below was the result. Lol.

“Can AR and VR coexist? And what would that even mean?”

So begins a labyrinth of speculation.

—————————————————————-

VR (virtual reality) has always been a field close to me personally. I dabbled in it a bit back in graduate school, and later, while working in audio, it was a word that kept flickering at the edge of my vision. It’s also a concept that matters a great deal in making things. I wouldn’t call myself a specialist with decades in the field, but I know a fair amount.

AR (Augmented Reality), on the other hand, I don’t know as well. Part of the reason is that it’s a technology that’s only drawn attention in recent years, but honestly, the bigger reason is that I just haven’t consciously engaged with it that much. When I actually looked into it, the concept itself apparently dates back to the early 20th century. (Quite a thing to come up with, back in the era of sending Morse code over telegraph machines.)

Let’s start by thinking about the definitions of these two and the essential difference between them.

Everyone probably already knows the concept of VR, but let’s get it straight anyway: “A technology, and the body of methods built around it, that uses engineering means to stimulate a user’s senses and thereby create an environment that may not physically exist, or may take a different form, but whose functional essence is the same as something real” (quoting the ever-convenient Wikipedia). In short, it’s the whole set of technologies for artificially creating something that feels like reality.

AR is generally described as a technology that branched off from VR. For example, it’s about displaying additional information with some added value layered on top of a real image.

An example any Japanese person would immediately recognize is the Scouter from Dragon Ball.

Beyond that, it’s a concept that science fiction and anime have used constantly and enthusiastically. Star Trek, Ghost in the Shell, Denno Coil — the examples are endless.

In recent years, AR technology has started being used at the consumer level, with Sekai Camera as its flagship example, and more recently, Pokemon GO.

And although things haven’t gone well for it, there’s also Google Glass as an example of an actual product that made it to market.

At a glance, it might seem that VR carries a strict, almost ascetic goal of “reproducing reality,” while AR has a much looser feel to it — something more like “let’s just add something useful, yeah!”

But when you think about which of the two is actually more rigorous in terms of its definition and scope of application, it turns out AR is the one with more constraints.

VR covers things like “sense-swapping,” which is itself a kind of light VR —

[GIGAZINE] Gender Swap: A virtual experience letting you feel what it’s like when male and female bodies are switched

— all the way to something completely different, like, to use an example from a bit back, “Sword Art Online,” where you dive from an ordinary household into a fantasy world. That too counts as VR by definition.

AR, by contrast, takes the approach of adding some kind of value on top of reality, so it has to have “reality” as its starting line, or else nothing makes sense. What’s needed isn’t something “reality-like,” as in VR — it’s reality itself.

This is why I find myself doubting the framing that AR is simply derivative of VR. It feels to me like the two might be built on fundamentally different premises from the start.

Let’s continue thinking about rigor here. From the standpoint of “the precision being pursued,” VR can still be said to be the more rigorous of the two. What VR aims for is “something that feels like reality,” and naturally, the closer it gets to actual reality, the better.

But—

however—

there’s a famous concept in robotics called the “uncanny valley.” Looking into it, I found that it was actually proposed by an emeritus professor at my own alma mater.

The “uncanny valley,” simply put, describes a pattern in human cognition: as you make a robot look more and more human, up to a certain point, the more human-like it looks, the more “familiarity” increases — but then at some point, it suddenly starts to look “strange” instead. That is, it describes a habit of human cognition in which thresholds and priorities shift, and the term “valley” refers to that dramatic drop in familiarity.

For a Japanese person, probably the most accessible, easy-to-grasp proof of the uncanny valley’s existence is the “Final Fantasy” movie.

Huh? Are the examples skewing toward games and anime? Well, let’s let that slide.

That film was based on the world of a video game series that had been massively, explosively popular at the time, and it used the CG technology of the highly acclaimed FF series to make a movie — and it bombed dramatically. Various reasons have been given, but at bottom they all run along one line: audiences couldn’t feel familiarity with the characters → couldn’t immerse themselves in the story → found it unenjoyable. I think that’s the core of it, and it’s basically correct.

In other words, the visuals in the FF games had reached a nice level of “wow, that’s incredibly realistic for a game!” But when they pushed a notch further toward realism to make the movie, they fell straight into this valley.

People pointed out unnatural object movement, strange skin texture, subtle discrepancies from how real humans actually move, and so on — the kind of nitpicking contest that must have brought the creators to tears, thinking “none of this was ever pointed out in the FF games!”

But apparently, “once something crosses a certain threshold of quality, people start nitpicking the details” — that seems to be how human psychology works.

I’m sure everyone can relate to this at least a little. The closer something gets to perfection, the more people want to point out tiny flaws just to vent.

I believe this was from an actual study done at a university in Belgium, but there was an attempt to define “the conditions for beauty” by digitizing information like the position, size, and shape of facial features and running various experiments on it. Years of research found that “the average of all the faces a person has seen over their lifetime” comes extremely close to that person’s ideal of a beautiful woman. Average = well-proportioned. That finding by itself is a remarkable discovery.

↓ This is the average face of 60 high school girls.

On top of that, researchers created: A. The average of randomly sampled ordinary women’s faces B. The same face as A, but with slightly fuller lips

and asked men which one they found more to their “taste.”

B was overwhelmingly more popular.

Everyone uniformly answered that A was “beautiful,” but in terms of personal “preference,” its favorability was extremely low — people said it looked “artificial,” “a bit scary,” “hard to read an expression on.” B, by contrast, got positive reactions across the board — “the lips look sexy,” “it looks cute.” A face too close to perfection feels creepy; a face that’s slightly off, it turns out, feels more real and more approachable. Humans can detect the creepiness of something artificial with remarkable sensitivity.

Needless to say, in communication and commercial contexts, “approachability” and “likability” are crucial factors. I think this is also a good example that lets us reconfirm the problem standing in the way of VR.

There’s also an interesting story about human perception in the world of music.

Rhythm, needless to say, is supposed to be constant. But apparently, if it’s truly, strictly “constant,” humans find it unsettling. As one aspect of our biological movement, humans are subtly, constantly vibrating. Even when we think we’re holding still, we can’t actually hold perfectly still. Try standing still in front of a mirror and see. Try to line up, say, the position of your shoulder exactly with a piece of furniture in the background — you’ll see that you’re actually swaying slightly. Maybe you still won’t be able to tell even after trying that. But that’s fine — it really is an extremely tiny motion. Still, in the CG rendering of human figures mentioned earlier, this turns out to be an extremely important point. Without this tiny trembling, a figure instantly reads as “artificial.” This is an example of the body’s internal auto-balancing mechanism working to maintain bipedal balance, but fluctuations like this exist throughout various parts of the human body — and that’s precisely what feels pleasant. In other words, “fixed” feels uncomfortable to humans, while “wavering slightly” actually feels, surprisingly, pleasant.

Also, a skilled drummer is often said to have “evenly matched notes.” Sound has not just amplitude — loudness — but various envelope parameters that shape its contour. When those are the same every single time, people say the notes are “evenly matched.” When this is artificially produced — as with drum machines and other mechanical sounds — the sound is strictly identical every single time. And humans pick up on that sensitively. It feels unsettling.

I think quite a lot of people say “I just can’t get into programmed/sequenced music.” I’m one of them. Human perception is extremely sharp, and fooling it completely is not an easy thing to do at all.

So, there’s a bottomless valley in front of us.

The simplest solution is “don’t head toward the valley.” In the case of human faces mentioned earlier, this means “deformation,” “cartoonification.” I think one of the reasons Japanese manga culture is so highly regarded is precisely that it never headed toward “realistic depiction.” Even in a gritty, realistic-style work, a character’s eyes are usually drawn rounder than normal, and the body’s lines aren’t faithful to realistic proportions — some kind of stylization is always applied. The reason is simple: readers’ affection for the characters goes up that way.

Even with 2D idols, there’s not a single success story that came from aiming for realistic modeling.

On the other hand, naturally, there’s also the “ascetic answer” — the challenge known as “leaping over the valley,” or “the far side of the valley.”

For people aiming to build the perfect humanoid, and more broadly, for people who’ve positioned themselves on the “vector pushing VR forward,” this is an extremely important guiding principle.

To get to the far side of the valley, you have to set that as your goal from the very start. “From the very start” is an extremely important point.

Take the FF example from earlier — even if the engineers had worked twice as hard, even if the editors had spent ten times as long, they still wouldn’t have been able to clear the valley in the end. To avoid making the same mistake again, you need to “retool the method itself.” One major technique being incorporated into recent films, for example, is motion capture. You attach multiple pointers to an actor’s body (things whose 3D position can be read via a special camera) and have them perform. Then you apply CG processing to that captured data. Because the original “movement” is genuinely human, no “strangeness” ever enters into it. This drew a lot of attention with the film “Avatar” (since the whole thing was done that way), but it’s actually a technique that’s been around for a very long time. If my understanding is correct, this kind of pointer technology was originally developed for sports science, to study form and muscle movement.

For another example of attempts to cross the valley, it’s going to get a little nerdy, but let me bring up an example from audio.

In the field of acoustics, there was a period when “transfer functions” started being talked about heavily. I’ll spare the detailed explanation and point you to a separate post below, but simply put, it’s a way of turning the phenomenon of “how a sound is perceived” into a function of its characteristic parameters.

What on earth is a transfer function?

By using that function to run numerical calculations and generate audio signals, you can achieve things like: “It feels like there are more speakers than the two actually playing,” or “Even though you’re listening in a small room, it feels like you’re in a huge concert hall” — classic realizations of VR.

An evolved form of this is “home theater,” which spread explosively for a while.

A reasonably high-quality theater system doesn’t just play back audio in 5.1ch or 7.2ch — it also applies processing based on this “transfer function” to create “a more glamorous soundstage.”

But this transfer function is a genuinely tricky thing. One manufacturer takes the austere approach of “using years of hall acoustic measurement data and know-how,” while another instead relies on “the results of acoustic simulation,” and yet another’s approach is basically “well, an engineer just kind of made it up… actually, no comment.”

This sits right along the uphill path of the slope toward VR, climbing a staircase of increasing excellence in the order: engineer’s own creation < simulation << measured data.

And that valley everyone keeps mentioning exists, categorically speaking, within the realm of “measured data.” The other two categories are at a level of “ah yes, nice try, doing your best to get there,” and still need quite a bit more effort before they can even claim the name “VR.” Meanwhile, the ones using actual measured data are, naturally, of higher quality precisely because they put in the money, time, and effort. But even when the data is good, in the subsequent layer of audio synthesis processing, you can end up with something that’s “extremely close, but precisely because of that, feels unsettling.”

For example, a well-designed concert hall reflects the sound played on stage in every direction. That means, from the listener’s perspective too, reflected sound arrives from every direction — front, sides, behind, ceiling, floor, and so on. But it’s commonly said that “reflections along the vertical axis aren’t that important,” or “floor reflections can basically be ignored” — various compromises get built in for the sake of practicality.

You need to cut processing load to improve response time; placing speakers on the ceiling isn’t realistic. I get the reasoning.

But human perception is remarkably sharp, and hearing is perhaps its sharpest faculty (I think). Human senses don’t let compromises like that slide. They immediately send back a signal saying, “something’s off here.” It’s genuinely impressive that humans can apparently detect a difference of about one degree in the direction a sound is coming from.

As an example of “something’s off here,” a friend of mine once raised the issue that “when you talk over Skype, your eyes never actually meet.” That’s a very fair point — it’s the loss of “realness” hiding behind the “convenience.” Human perception doesn’t let that kind of thing slip by either.

<The Compatibility of VR and AR — A Thought Experiment>

But I don’t think this valley is even worth a glance for AR.

The key point for AR is how to provide supplementary information layered onto reality in a way that’s both efficient and genuinely useful, and that also feels good.

So there’s no role here for the valley to play.

A part of reality that’s blatantly “artificial” — how does that control people’s “familiarity” and “favorability”?

That raises the question: “Can advancing VR and increasing the value of AR coexist?”

Let’s try a thought experiment.

Suppose you’re wearing glasses like Google Glass that can display supplementary information. Imagine a scene where you’re talking with a friend over Skype. Suppose the other person is displayed not in a window, but blended seamlessly into the background — “as if they were really there.” Suppose this is achieved with the full refinement of VR technology, at a quality so high it’s genuinely indistinguishable from reality. Now let’s imagine some use cases for AR in this situation.

Case A: “Let’s go through tomorrow’s presentation materials together.” → Both people’s movements are detected, and a document is displayed that lets you flip pages, highlight, and search.

Case B: “I’m at the bookstore right now trying to decide which one to buy.” → The other person, displayed as if standing right next to you, points at books and explains them. At the same time, information they’re holding, or information pulled from the internet, is displayed in a pop-up-style window.

Case C:

…Honestly, I’m starting to lose track of what I’m even looking for. What’s so great about this? What feels good about it? What exactly has value here?

In Case A, we had the VR of “a meeting partner right in front of you as if they were truly there,” plus the VR of “virtually flipping through pages of meeting materials,” and on top of that we layered in an augmented-reality element of “highlighting and search.”

In Case B, we had the VR of “the reassuring feeling of having a friend right there beside you,” mixed with the augmented-reality element of “pulling information from the internet and displaying it in a pop-up.”

At this point, it feels like it can no longer be discussed along a single axis.

The answer is probably “No.” The ultimate form of VR lies on the far side of the valley, and reaching it means reaching into the human unconscious and stirring it up like fermenting miso paste, pushing relentlessly forward. Thoroughly ascetic.

In contrast — though I’m not sure if it’s rude or arrogant to put it this way — the ultimate form of AR always sits on the axis of practical usefulness. Reality is neither more nor less than reality; what constitutes added value is everything outside of it — “unreality,” “augmented reality.” The user’s attention instead shifts toward that added layer, and if VR happened to be present there at that moment, its value would unquestionably be diminished. It would retreat to the near side of the valley. It would roll back down the slope.

<Does AR Complement Reality? Does VR Complement Reality?>

Summing up the differences between the two discussed so far, I think it can be put this way:

AR = practicality + a bit of art (ideation)

VR = natural science + a bit of fantasy (imagination)

Certainly there’s a lot of overlap, but I still feel there’s a gap between them that can’t simply be explained away as “derivation” or “an advanced form.”

So trying to make the two compatible ultimately feels like wasted effort that ends up falling into some valley or other. If you’re going to move in one direction, you need to clearly commit to a concept and push forward with it. And depending on the situation, that includes making the hard call to prioritize one and sacrifice the other.

<And Also, Doraemon>

Lately I’ve taken to thinking about product design in terms of Doraemon’s secret gadgets for just about everything.

Almost every “boyhood dream” is packed somewhere into Doraemon. (I’ll say it flatly.)

Pretty much every need and want that currently exists can be expressed through some secret gadget. (I’ll say that flatly too.)

And yet, when it comes to the VR and AR discussed in this piece, I realize these are things that never appear in Doraemon at all. (At least as far as I know, which is an extremely loose bar.)

Surprising.

Surprisingly surprising.

Unexpectedly surprising.

But the answer is simple.

In a world where you can go anywhere instantly, travel back and forth through time, swap the insides of people, make things bigger or smaller freely at will, and fly through the sky just by strapping a bamboo-copter to your head, neither VR nor AR is needed.

In other words, the wants and ideals that humans are truly meant to satisfy lie even further beyond VR and AR.

If so, the road ahead is long.

Valley after valley, a valley here, a valley there too. It’s dizzying just thinking about it.

But if that’s where we’re ultimately headed, then we have to keep our sights set high and go at it prepared to leap over every single valley in one breath — hop, step, jump, clearing them all in a single great bound. Actually, no — if we don’t approach it that way, where’s the fun in it?

And with that, I think the reason behind the title has landed, so that’s where I’ll end this.


Originally published in Japanese at https://clazytech.com/2016/09/1995/. Translated with LLM assistance and reviewed before publication.