Failed PoCs Usually Trace Back To Bad Goals And Metrics

2022-09-06 original Engineering

Actually, this is a strange question.

PoC (Proof of Concept) is sometimes described as “proving a hypothesis,” but I think it’s a term most often used to describe an experiment carried out after coming up with an idea for a product or service. Developing a product or service takes both time and money.

Picture the position of a decision-maker, and you’ll understand that a Go/No-Go decision on a project costing hundreds of millions of yen can’t possibly rest on a few dozen slides of a well-crafted presentation. “Prove to me that what’s written here is actually true.” That’s what PoC is for. So even if the result of running a PoC is “This idea is bad. Let’s stop here,” that in itself is an achievement. It meant not pouring hundreds of millions of yen into a project that was going to fail.

So what does it mean when “no results come out”? I suspect this is a problem of “goal setting” and “measuring effectiveness.”

Go/No-Go decisions should be made with severity. That’s why it’s extremely important to decide in advance on criteria such as “if this is achieved, it’s a GO (the project moves forward)” and “if this isn’t met, it’s a No-Go (the project ends).”

The criteria for a decision need to be clear. They should be set numerically as much as possible. But if you don’t choose the right parameters when using performance as your criteria, the project will immediately get stuck in the mud. The prototype made for a PoC generally hasn’t gone through several stages that a product built for actual production would go through. QA and certification are typical examples. QA, or quality assurance, involves running various tests to evaluate objective functional behavior and its stability. Certification evaluates legal compliance and conformance to standards. Most products go through repeated rejections and revisions at this stage, and each time, improvements are made to raise the quality of the product. In most cases, there isn’t enough time or money to do this within a PoC. So these steps end up being skipped or simplified. I often see evaluation judgments become shaky as a result.

Say you’re running a PoC for a product worn on the body. It might be used in sports, so waterproofing is required. Even if you set a final product target of, say, “equivalent to IPX8 (tentative),” it’s difficult to evaluate that within a PoC. The parts aren’t molded components, and unless you’re manufacturing on a mass-production line, evaluating waterproof quality doesn’t mean much anyway. Suppose the focus of the PoC was actually the algorithm mounted in the product—a product that detects something about the human body and sends a notification. In that case, the goal of the PoC is “confirming the usefulness of the algorithm,” and the criteria for measuring effectiveness might be set as “error within ±1% compared to reference data.” Now suppose you actually try it out in the field and measure its effectiveness. A comment comes in: “This is useless.” That’s because the waterproofing was insufficient, forcing repeated redos during the PoC. As an impression, this is naturally correct, but it has nothing to do with the decision criteria. Yet the outcome of the PoC ends up in a wishy-washy atmosphere, something like “the performance was perfect, but there’s an opinion that we can’t move the project forward as it stands.”

But if goal setting and the identification of concerns (a risk assessment of the project) had been done properly at the outset, the conclusion should have been something like: “Waterproof performance and operational stability were anticipated issues, so those will be addressed as a task for the next step. For now, we’ve proven that the accuracy is sufficient, so there’s no concern about the algorithm. Let’s move the project to the next step.”

That said, in reality, there seem to be an extremely large number of cases where setting numerical criteria is difficult. I imagine this is where many projects stumble.

For example, in a situation with limited budget like a PoC, it’s difficult to conduct large-scale marketing research using the prototype you’ve built. There aren’t enough devices, and you can’t pay compensation to participants. And to begin with, it takes too much time to do it compactly. “We tried it out for now, and it seems to be working fine—does this look okay?” It’s hard to make a decision to move to the next step based on that. But it’s also difficult to create clear numerical criteria.

In such a case, one approach you can take is to “clearly designate the decision-maker and the decision date.” Put a bit crudely, this means locking in something like “so-and-so will decide Go/No-Go on such-and-such date.” This might sound like a rough approach, but it’s really not. The decision-maker is normally the person with authority to determine or execute the budget, so this would be some kind of department head, an executive, or in some cases the president. Once you’ve decided on the decision-maker and the decision date, you begin communicating with that person. You talk about the aims of the project and the goals you’re pursuing, and in response, you lock in what will be used as the criteria for judgment. If it’s a boss worthy of trust who has handled things properly, they won’t reverse that decision. Outside interference is entirely possible, of course, but at the very least, you should be able to make clear the line the project needs to clear through the PoC.

If, despite clearing the line set in this way, and despite having clearly proven it, no Go/No-Go decision comes out for the project, then some non-positive change in circumstances or some kind of trouble has likely occurred, shaking the value of the project itself. In such a case, it’s wiser to treat it as effectively a No-Go decision, and to boldly halt the project without clinging to the current sunk costs, then reconsider from a different angle.

The content of this post is an excerpt (original text) from the following book. If you’re interested, please consider purchasing the book.

The Shape of a Happy IoT Startup

Originally published in Japanese at https://clazytech.com/2022/09/1152/. Translated with LLM assistance and reviewed before publication.