Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Choosing pilot users inside the customer's organisation before full go-live

Technical article
Hosting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

Direct answer: Nominate a pilot cohort that mirrors production complexity in miniature—critical roles, edge workflows, one difficult site and a sceptical power user—give them named success metrics and a time-boxed calendar, then gate full go-live on pilot evidence signed by the customer’s executive sponsor, not on “IT says the server is up.”

Why it matters

ISVs lose renewals when the first month is organisation-wide chaos. A customer-side adoption pilot proves that real staff can complete paid workflows on your hosted tenant before you flip the whole payroll, clinical or logistics population.

This differs from hosting migration pilots (moving bytes between old servers and a hoster) and from provider dress rehearsals. Here the subject is who inside the customer should try the software first and what they must prove. Your CS and implementation team owns sampling; the customer owns nominating credible people who will tell the truth when screens confuse them.

How it works

Sizing the cohort

Hypothetical example: for a mid-market rollout you might nominate on the order of 15–40 users—enough to surface role conflicts for your workflow count, small enough to coach daily. Adjust to your customer; this is one planning scenario, not observed industry statistics. Include:

  • Process owner for each critical workflow (approver, dispatcher, clinician).
  • Data entry heavy role and read-only report consumer.
  • Remote or low-bandwidth site if field access matters.
  • Vocal sceptic invited early—better than a surprise at town hall.

Exclude: executives who will not click buttons, and IT-only testers who never run month-end.

Success metrics (examples—not universal SLAs)

  • Task completion rate for ten scripted scenarios (time and error count).
  • Support tickets per pilot user per week trending down.
  • Data quality checks (exceptions queue emptying).
  • Subjective System Usability Scale (SUS) score or simple 1–5 “would recommend to colleague” after week two.

Tailor metrics to what the contract promises. Agree measurable acceptance criteria with the customer sponsor before go/no-go—for example task completion rates, ticket trends and usability scores you will actually collect—rather than relying on unspecified availability percentages.

Calendar

Week Focus
1 Core happy path + office hours
2 Exceptions, integrations, reports
3 Load from parallel legacy (if any)
4 Go/no-go readout and training plan for wave 2

Governance

Weekly pilot stand-up: customer sponsor, ISV CS, customer super-user. Decisions logged: config change vs training gap vs defect.

Hosted platform angle

Pilot tenants may share production code version but should use controlled data (scrubbed or synthetic per training-data guidance). If pilot runs against production data, tighten partner and trainer visibility.

What to choose and why

Customer shape Pilot emphasis
Single site Depth on all roles
Multi-site franchise One flagship + one troubled site
Regulated Audit trail review in metrics
Replacement of incumbent Side-by-side task timing

Fail forward: extend pilot with scope cut, not silent organisation-wide launch.

Risks and limits

  • Pilots that only include the vendor’s friends produce false confidence.
  • Unpaid overtime for super-users burns goodwill—budget incentives or relief time.
  • Pilot customisations that will not scale to wave 2 become technical debt; track them.
  • Not statistically rigorous sampling theory—pragmatic product management only.
  • No lab validation.

Sources

Related reading

Learning notes

  1. Pilot users should look like real staff, not only IT.
  2. Metrics must match what the customer bought.
  3. Time-box pilots or they become permanent parallel run.
  4. Sceptics early save executive surprises later.
  5. Pilot evidence is a gate, not a demo checkbox.