Vague goals and fuzzy metrics are why your PoC fails
Actually, this is a strange question.
PoC, or Proof of Concept, is sometimes described as “proving a hypothesis,” but I think the term is mostly used to describe the first experiment conducted after coming up with an idea for a product or service. Developing a product or service takes both time and money.
Put yourself in the decision-maker’s position and you’ll understand that a Go/No-Go judgment on a project costing hundreds of millions of yen can’t be made from a few dozen slides of a well-crafted presentation. “Prove to me that what’s written here is actually true.” That’s what the PoC is for. So even if the result of running a PoC is “this idea is bad, let’s not do it,” that’s still an achievement in itself. It means you avoided directing hundreds of millions of yen in expenses toward a project that would have failed.
So what does “no results” actually mean? I suspect it comes down to problems with “goal setting” and “measuring effectiveness.”
<Goal Setting>
The Go/No-Go judgment should be made rigorously. That’s why it’s extremely important to decide in advance on criteria like “if we can achieve this, it’s a GO (the project moves forward)” and “if we can’t meet this, it’s a No-Go (the project ends).”
<Measuring Effectiveness>
The judgment criteria need to be clear. It’s preferable to set them numerically as much as possible. But if you don’t choose appropriate parameters when using performance as a criterion, the project will immediately get bogged down. That’s because the prototype built for a PoC generally hasn’t gone through several stages that a real product would. QA and certification are often examples of this. QA, or quality assurance, involves running various tests to evaluate objective functional behavior and stability. Certification evaluates legal compliance and conformance to standards. Most products get sent back for revision repeatedly during this process, and quality improves with each round of refinement. In most cases, a PoC doesn’t have enough time or money to do this. So these steps either don’t happen or get simplified. I often see evaluation judgments turn shaky as a result.
Suppose you’re running a PoC for a product worn on the body. It might be used in sports, so waterproofing is required. Even if you set a final product target like “IPX8 equivalent (tentative),” evaluating that in a PoC is difficult. The parts aren’t molded components, and unless you build it on a mass production line, evaluating waterproof quality doesn’t mean much anyway. Suppose the real focus of the PoC was the algorithm built into the product — a product that detects something about the human body and sends a notification. The goal of the PoC would then be “confirming the usefulness of the algorithm,” and the effectiveness criterion would be set as “error within ±1% against reference data.” You test it in the field and measure its effectiveness. You get the comment “this is unusable,” because the waterproofing performance was inadequate and you had to redo the PoC multiple times. As a comment, this is of course correct, but it has nothing to do with the judgment criteria. Yet the result of the PoC ends up in a wishy-washy place, something like “the performance was perfect, but there’s an opinion that we can’t move the project forward as it stands.”
If goal setting and risk assessment had been done properly at the outset, you should be able to conclude instead: “waterproofing performance and operational stability were anticipated issues, so we’ll treat them as a challenge for the next step. For now, accuracy has been proven sufficient, so there are no concerns about the algorithm. Let’s move the project to the next step.”
That said, in reality there seem to be many cases where setting numerical criteria is difficult. I imagine that’s where things often fall apart.
Under the limited budget of a PoC, for example, it’s difficult to run large-scale marketing research using the prototype you’ve built. You don’t have enough devices, you can’t pay compensation to participants, and to begin with, it takes too much time to do it on a small scale. “We tried it, it works fine, is this okay?” — that kind of thing makes it hard to decide to move to the next step. But it’s also hard to create clear numerical criteria.
One approach you can take in such a situation is to make clear who the decision-maker is and when the decision date is. Put a bit crudely, this means locking in an agreement like “so-and-so will decide Go/No-Go on such-and-such date.” This might sound like a rough approach, but it isn’t. The decision-maker is normally the person with authority to determine or execute the budget, so they’ll be some kind of department head, an officer, or in some cases the president. Once you’ve decided who the decision-maker is and set the decision date, you start communicating with that person. You talk about the aims of the project and the goals you’re pursuing, and in return, you lock in what will be used as the judgment criteria. If it’s a boss who’s done things properly and is trustworthy, they won’t go back on that decision. Of course, interference from other quarters is entirely possible, but at the very least, you should be able to make clear the line the project needs to clear through the PoC.
If, despite clearing that line, and despite having clearly proven it, no Go/No-Go decision comes down for the project, you can assume that some negative change in circumstances or some kind of trouble has occurred, shaking the value of the project itself. In such a case, it’s wise to treat it as effectively a No-Go decision, cleanly halt the project without clinging to the sunk cost, and reconsider from a different angle.
The content of this post is an excerpt (original text) from the following book. If you’re interested, please consider purchasing it.
The Shape of a Happy IoT Startup
The Shape of a Happy IoT Startup
Originally published in Japanese at https://clazytech.com/2022/09/1152/. Translated with LLM assistance and reviewed before publication.