How to benchmark your product against competitors without lying to yourself

Every competitive comparison chart I have ever seen has the same remarkable property: the company that made it wins.

Row after row of green checkmarks on their side, tasteful grey crosses on the other. Sometimes there is one row where the competitor wins, tucked near the bottom, and it is always something nobody cares about, included for the specific purpose of looking fair.

That is not benchmarking. That is a mirror with a spreadsheet border.

The problem with your criteria

Here is how the fake version happens, and it is not malicious, it is just human.

You pick the criteria. Naturally, you pick the things you are good at, because those are the things you have thought about most and are proudest of. Then you score everyone on your criteria and, astonishingly, you win.

But the buyer did not use your criteria. They used theirs. And theirs were assembled from a demo, a review site, a colleague's opinion, and a gut feeling about whether your onboarding looked painful.

Benchmarking against criteria you chose is like grading your own exam after writing the questions.

Where the real criteria come from

From the people who evaluated you and chose someone else.

This is uncomfortable, which is why almost nobody does it, and why the ones who do it learn things nobody else knows. Find three prospects who went with a competitor. Ask them what they compared. Not what they liked. What they compared, and in what order.

You will discover that the criteria you agonised over did not appear on the list. And that the thing that lost you the deal was something you have never once put on a chart, like the fact that setup took three weeks and the competitor was running in an afternoon.

That is your benchmark. Not the one you invented.

The rows where you lose

This is the entire point of the exercise and the part people skip.

If your benchmark does not produce at least two rows where a competitor genuinely beats you, you did not benchmark. You did therapy.

Write the losing rows down. Show them to your team. It will be unpleasant and it will be the most useful hour anyone spends that month, because those rows are the only ones that tell you what to do. The winning rows tell you to keep doing what you are already doing, which is not information.

The parity trap

And then, having found the rows where you lose, do not automatically go fix them.

This is the trap. You find a feature gap, you close the feature gap, they ship something else, you close that gap. Two years later you have built a worse version of their product, arriving late to every fight, and the strange specific thing that made anyone choose you originally has quietly rotted from neglect.

Some gaps you close because they are genuinely losing you deals. Most gaps you look at, understand, and deliberately leave open, because closing them would cost you the thing that makes you different.

Knowing which is which is the actual job. The chart is just how you find out.