What 31 premium accounts taught me about what users actually value

By September 2026, premium users were buying the product but, from the outside, seemed not to use it. The team had a comfortable explanation: maybe users buy and forget, or maybe something is technically broken.

I had a stronger belief: global search across questions and answers was the core value of the product. The roadmap was being shaped around that belief.

Problem

If premium users do not use the product, retention and revenue are at risk. But the bigger risk was different: we might be investing in the wrong value proposition.

The question was: what do premium users actually use — and what does that mean for the roadmap?

Evidence

I audited 31 premium accounts with subscriptions expiring in less than 15 days. I checked what each account actually did.

Key findings:

• 61% (19/31) did not use global search more than 10 times.
• 28 of 31 were actively using the product. The “technical problem” hypothesis was wrong.
• Users were not rejecting the product. They were using test pages, not search.
• Only 2 of 31 (6%) bought and never logged in — a small edge case, not a trend.

Options considered

1. Keep investing in global search and AI. It was our belief; maybe users just needed better onboarding.
2. Improve onboarding and communication. Explain the search feature better.
3. Redesign the test pages. Users already preferred them; the design was old.
4. Contact the non-users and ask why. A small qualitative check.
5. Do nothing. The product was working; leave it alone.

Decision

I stopped further development of global search and made test-page redesign the #1 priority.

I also kept global search as a supporting feature, not the core value. I asked the team to contact the two non-users to understand whether they needed help or a refund.

Why:

• The evidence was behavioral, not opinion-based.
• 19 of 31 users were actively using test pages; that was the real product.
• Continuing to invest in search would have been investing in my own belief, not in user value.
• The test pages had an old design, no upsell, and no modern results screen. That was a clear product opportunity.

Execution

• Shared the findings with the owner and team. The owner trusted the analysis; there was no resistance.
• Paused further search development and deprioritized other initiatives, including Marlins expansion and AI search.
• Moved the team to the test-page redesign:
  • test start button on the first screen;
  • clearer test structure;
  • upsell to buy the test as a PDF;
  • certificate after passing (premium required);
  • better results screen.
• The redesign shipped in October 2026. I tracked bounce rate as a diagnostic (baseline 9%), but the commercial metrics were test starts, premium conversion, and PDF upsell.

Result

The result was a roadmap change and a measurable lift:

• Global search was no longer the center of the roadmap.
• The team redesigned the test experience around what users actually used.
• After the redesign, organic-to-premium conversion rose 79%, organic-to-PDF conversion rose 47%, and sales rose 23%.
• The lift held for two weeks. This is a before/after result, not a controlled A/B, so I do not claim pure causality; seasonality and other changes were not isolated.

The important result is that the decision changed before more money was spent on the wrong feature — and the commercial metrics moved in the right direction after the change.

What I learned

• The most dangerous assumption is the one you do not test.
• Users’ behavior is more honest than their words — and more honest than my own beliefs.
• A PM’s job is to kill their own ideas with evidence, not to defend them.
• A small sample can be a signal, not statistics. I treated it as a signal and kept the decision reversible.
• A before/after lift is enough to act, but not enough to claim causality. The next step would be a controlled test.
• What I would do differently: instrument behavioral analytics earlier and define the success metric for the redesign before launch, not after.