Five questions that move an AI vendor conversation from promise to evidence

A polished demonstration is not an operating answer

AI vendors are often strongest when showing what their technology can generate. Credit union leaders need to understand something different: how the capability will operate inside their institution.

These five questions move the conversation toward that evidence.

1. What institutional decision does this improve?

Ask the vendor to connect the capability to a specific decision, workflow, or member outcome. If the answer remains at the level of “efficiency,” “personalization,” or “better insights,” the operating value is not yet defined.

The vendor should be able to explain who acts differently, what changes, and how leadership will know.

2. What context can it use? What is it prevented from using?

Understand how the product accesses institutional knowledge, member context, policy, and strategic priorities. Then examine the boundary. Which sources are approved? How is access controlled? What information is excluded? How are changes to source material reflected?

This is not only a technical integration question. It determines the quality and appropriateness of the guidance.

3. Where does human judgment remain visible?

Ask which outputs are informational, which are recommendations, and which can trigger action. Determine whether the rationale is visible to the person responsible for acting and whether uncertainty or missing information is surfaced.

A useful answer should define authority, review, and escalation. Simply stating that a “human is in the loop” is not enough.

4. How will we connect activity to outcomes?

Request a clear evidence chain from eligible signal to generated guidance, frontline action, and institutional result. Usage analytics alone do not show whether the capability advances growth or service.

The vendor should explain what the institution can observe, export, segment, and review at each decision gate.

5. What evidence earns broader exposure?

Ask the vendor to help define a controlled first scope and the conditions for expanding, refining, or stopping. A credible answer should acknowledge where the product may not fit and what evidence would change the recommendation.

The goal is not to force certainty before learning. It is to make the learning governable.

Document the judgment

Capture the vendor response, the supporting evidence, and the institution's own assessment separately. That distinction prevents presentation from becoming proof and gives leadership a clear basis for the next decision.

Use the full AI vendor evidence guide to structure the evaluation across institutional requirements, vendor responses, evidence, and judgment.