Skip to main content
    Back to Insights

    Transformation in practice

    AI Adoption

    How to evaluate an AI pilot and decide what to do next

    Transformery·3 min read

    The demo drafts a reply in seconds. In customer service, someone checks every sentence, looks up exceptions and corrects the greeting. Whether the AI pilot helps depends on the whole request. How much work does it take to get from a new request to an approved reply? Do customers still receive a reliable answer?

    Before deciding whether to expand, compare the same workflow with and without AI. Include time spent reviewing and correcting the output. Agree on quality requirements and make the decision with the people who are responsible for the work.

    Examine a complete workflow

    Consider an illustrative example from a Swiss business. The service team wants to spend less time writing replies to common requests. The pilot covers a small set of standard cases. Staff review each draft and approve it before sending anything.

    Compare requests of a similar level of difficulty. Record which cases belong in the comparison and how you handle missing information or exceptions. Use data approved for the test. If you use examples drawn from daily work, include typical exceptions on purpose. They help you see where the workflow reaches its limits.

    Bring three observations to the review

    • Time across the workflow. How long do people spend on the request, including looking up information, entering it, checking the output and correcting it? If response time matters, record the waiting time as well.
    • Quality of the reply. Use the same criteria to review the replies. Are the details correct, the next steps clear and the promised deadlines ones the team can actually offer? Record mistakes caught before sending and errors that make a reply unsuitable for approval.
    • Work for the team. Who deals with poor drafts? What happens when a request falls outside the agreed scope? Ask staff when they would choose to use the workflow and why.

    Before the test, agree on what improvement would matter and which errors would rule out further use. A handful of cases can show you what to investigate next. A wider rollout needs enough comparable observations and a way to handle exceptions in everyday work.

    Continue, adjust or end the pilot

    Continue. The team sees a benefit across comparable workflows, the replies meet the agreed quality requirements and someone can take responsibility for running it. Name the next bounded use and the date you will review it.

    Adjust. The benefit is limited to certain requests. Narrow the scope or change one specific step. Then check how the change affects effort and quality.

    End. Review time absorbs the savings, a critical error remains possible or the conditions for running the workflow are missing. Record what you learned and how the team will handle the requests from here.

    A template for recording the decision

    We compared [workflow and cases]. The time and work including review were [observation]. For quality, we observed [findings and exceptions]. We will [continue, adjust or end] because [reason]. [Person or role] will take the next step by [date]. We will review the decision using [evidence].

    A useful pilot ends with a decision the team can explain. People should be able to say which requests the workflow applies to and who they can turn to when they have a problem.

    Read the case study on AI in insurance workflows

    If you are planning a first pilot: Transformery's AI consulting and adoption support.