Skip to content
    CC
    Chetan Chhangani

    Improving trust in AI-assisted discovery

    Defined and evaluated product behaviour for AI-assisted discovery, turning broad expectations of relevance and correctness into repeatable product-quality criteria.

    Owned workstreamShared productAI ProductSearch & Evaluation
    ~90%CorrectnessInternal evaluation

    The problem

    Users needed a simpler way to discover relevant information across a complex product environment without manually constructing rigid search or filter combinations, or knowing exactly where information was stored.

    Returning an answer was not enough. The product also needed to interpret ambiguous intent, return useful results, provide enough context for users to judge relevance and behave predictably when it could not provide a useful result.

    I owned a product workstream covering user and problem discovery, supported behaviour, product-quality criteria, evaluation design, result presentation, explainability principles, failure behaviour and iteration priorities. Engineering and data partners owned architecture, model and implementation decisions.

    The product decision

    "Good AI search" is not a usable product requirement.

    Terms such as accurate, relevant, useful and good search are too subjective to guide product development unless they are translated into explicit, repeatable criteria.

    Relevance

    Does the result address the user's actual intent?

    Evaluate whether the result answers the need behind the request, not only whether matching information exists.

    Correctness

    Is the returned information accurate within the evaluation context?

    Check whether the result is supported by the available product information used for evaluation.

    Context

    Can the user understand why the result is useful or relevant?

    Make result presentation accountable enough for users to judge relevance instead of asking them to trust the output blindly.

    Failure behaviour

    Does the product behave predictably when it cannot provide a useful result?

    Define how the experience should respond when the system cannot confidently support a useful answer.

    1. Subjective expectation
    2. Defined product-quality criteria
    3. Repeatable evaluation
    4. Failure patterns
    5. Prioritised iteration

    How I approached it

    1. Understand

      Identified how users expected discovery to behave and where existing search or discovery behaviour created friction.

    2. Define

      Translated broad expectations such as relevance, correctness and useful results into explicit product-quality criteria.

    3. Evaluate

      Created a repeatable evaluation approach for testing expected behaviour and identifying failure patterns.

    4. Iterate + Validate

      Used evaluation findings to prioritise changes, then re-tested behaviour against the same quality framework.

    What I learned

    AI product quality has to be defined before it can be improved. Words such as accurate, relevant, useful or good search are too ambiguous to guide product decisions unless they are converted into repeatable criteria.

    Trust is influenced not only by whether the product returns a correct result, but also by whether users can understand that result and whether the product behaves predictably when it cannot provide a useful answer.

    Failure behaviour is itself a product experience. In this work, defining how the product should respond when it could not provide a useful result was part of building trust, not a secondary edge case.