How do I know if someone is an AI expert and not just using ChatGPT the way I could?

These are the nine questions we at Criterian would ask a provider, with the answer that would make us wary and the one that would convince us.

1. Does this need AI or classic software?

If the process is a set of specific rules that repeat, classic software solves it. AI comes in when there is a reasoning layer: reading a text or a document, understanding the context and deciding what to do. They almost always go together. In customer support, reading a message and deciding which rule applies is AI; applying the rule is code.

That is why a good starting point is processes with fairly simple reasoning followed by programmatic rules.

Red flag: "AI can automate everything". Automation is programmatic; what AI adds is reasoning, so those automations do not always depend on a person.

2. Where does AI really add value, and where does it not?

It adds value in tasks that are not very complex but are very varied and share a purpose: reading and sorting emails, summarising, following up or assistant tasks.

It is not usually the right choice for high-volume programmatic processes or for those that put security at risk. And where a mistake has serious consequences, AI does not decide on its own.

Red flag: a provider who proposes AI for those cases too.

3. Which model would you use, and why?

There is no single best model for everything. An expert has a rule for choosing by task, backed by benchmarks, and explains it along three axes: cost, security and quality. A complex task that needs a plan and several passes is not the same as a creative task or a very specific, well-bounded one.

Red flag: "the latest one".

4. How do you know the AI is doing a good job?

It needs a structure: what goes in and what comes out, what reasoning it is asked for, which guardrails apply, how it gets the context it needs to decide well and at what point a person approves. Hallucinations are reduced by controlling the context and the guardrails.

Red flag: they cannot explain any of those pieces, or they trust the model not to make mistakes.

5. Where do you put a person to approve?

Where a mistake could have serious consequences, a person has to approve.

And there is a risk in the other direction: that the person approving stops using their own judgement. Using AI as a "second brain" creates dependency, and that should be taken into account when designing the system.

6. How does the system improve over time without getting more expensive?

What surrounds the model (the harness) affects whether the system improves as it grows: how it keeps memory, how it learns from decisions, where the person comes in, the guardrails and the traceability needed to audit it. The goal is for it to improve without the cost going up.

Red flag: proposing to add more and more skills as the main improvement.

7. What are the risks of using skills built by others?

We tend to assume a third-party skill is well built and install it as if it were safe code. It may hide capabilities or processes you do not expect. An expert looks inside it before using it and does not treat it as a black box.

Red flag: they install third-party skills without checking what they do.

8. Is my data ready for AI?

To be ready, it has to be structured, clean and normalised, with backups and a recovery plan. And before touching it you need to be clear about what it will be used for, what transformations it needs and what the end goal is.

Red flag: they start talking about models without asking about your data.

9. What happens when the models change?

Models change often. It makes sense to design in a model-agnostic way, so you can switch models without rebuilding the system. If something works, there is no need to change it just because a new model is out.

Red flag: they propose switching models every time a new one comes out.

The nine questions in a table

What an expert gives you that ChatGPT does not

A company's systems are complex in people, processes and technology, and some situations tend to repeat. An expert has lived through them and knows which variables matter most in each case. They use AI to analyse those variables in more or less detail and widen what they take into account when deciding. The decision is still theirs.

One difference is in how learning happens. People learn mainly through repetition or through impact. In our experience, AI tends to treat a serious mistake the same as a minor one: when you correct it, it creates a rule, often too rigid, and applies it from then on. It is useful for carrying out tasks and widening the variables to consider, but it does not replace the judgement of whoever decides.

How we apply it

A pattern repeats across Criterian's projects: AI is used where you need to interpret the intent behind a user, a message or a report in order to classify it, and from there the code takes over.