Designing AI solutions

How we scope, choose and evaluate a model before any infrastructure is provisioned.

This is the process we run before provisioning anything. It is deliberately slow at the start, because almost every expensive mistake in an AI project is made in the first week: picking a model before the task is defined, or building a pipeline for a problem that did not need one.

Start with the decision, not the model

Write down what a human currently does, and what they would do differently if the system worked. If you cannot describe the decision that changes, there is no product yet. There is a demo.

Question	Why it decides the architecture
What is the output, exactly?	A label, a number, a span of text and a document it came from: each implies a different model class and a different way to evaluate it.
Who acts on it, and can they be wrong?	A clinician reviewing a suggestion tolerates a wrong answer. An automated action does not. This sets your confidence threshold and whether you need a human in the loop at all.
Where does the ground truth live?	If nobody can say what the right answer was, you cannot evaluate, and you will be shipping on impressions.
How current must the knowledge be?	Static knowledge fits in a fine-tuned or prompted model. Knowledge that changes weekly needs retrieval.
What may not leave the country?	Data residency constrains your region, and sometimes your entire platform choice, before anything else does.

Pick the simplest pattern that fits

Pattern	Use when	Cost of getting it wrong
Prompted foundation model	The task is general language work and the knowledge is either common or fits in the prompt.	Low. Change the prompt.
Retrieval-augmented generation	Answers must be grounded in a corpus you control, and cite it. This is what Seerah AI is.	Medium. Re-chunking means re-indexing.
Fine-tuned model	Format or domain register matters more than facts, and you have thousands of labelled examples.	High. Retraining is slow and the data work dominates.
Purpose-built model	The input is not language: images, signals, tabular clinical features. This is what the Optical Diagnostics models are.	Highest. This is a research project with an engineering wrapper.

Do not reach for RAG reflexively

Retrieval adds an index, a chunking strategy, an embedding model and a whole class of failure where the right passage was never retrieved. If the knowledge fits in a prompt and rarely changes, put it in the prompt.

Compare models on evidence

Both platforms give you a catalogue with published benchmarks and a playground. Use them in that order: narrow on benchmarks, then test the shortlist on your inputs, because a benchmark tells you about the average task and you do not have the average task.

Shortlist from the catalogue

In Azure AI Foundry, the model catalogue exposes a Benchmarks tab per model and a Compare models view that charts quality against cost. Add two or three candidates and read the chart. A smaller model at a fraction of the price is very often within noise of the flagship on a narrow task.

Test in the playground on real inputs

Set the system prompt you actually intend to ship, then paste in ten real cases, including the awkward ones you would rather not think about. Ask a follow-up like “explain your reasoning” to see whether the answer holds up or was a lucky guess.

Hold out a real evaluation set

Fifty genuine cases with known answers beat five hundred synthetic ones. Take them from the failures your current process produced. Those encode the actual distribution of your users.

Decide, and write down why

Record the model, the version, the score it reached and the threshold you accepted. In twelve months someone will ask why this model, and “it seemed good” is not an answer that survives an audit.

Set quota before you build

On Azure, a deployment carries a tokens-per-minute rate limit that you set at deploy time. Set it deliberately: too high and one runaway loop drains a subscription quota shared with every other project; too low and you get rate-limit errors that look like bugs.

Regional quota is a real constraint

Model availability and quota vary by region, and a region that has capacity today may not next month. Check regional availability for the specific model you chose before you commit an architecture to it: a model you cannot deploy in your residency region is a model you cannot use.

Evaluate before and after every change

An evaluation set is what converts “this prompt feels better” into a number. Run it on every prompt change, every model version bump, and on a weekly sample of live traffic so drift shows up before complaints do.

Segment the results. An aggregate score hides the case that matters: a change can lift the mean two points while breaking every input in your second-largest language.

# The shape of the thing, whatever framework you run it in.
results = []

for case in evaluation_set:
    output = run_pipeline(case["input"])
    results.append({
        "id": case["id"],
        "segment": case["language"],       # segment on every axis you serve
        "expected": case["expected"],
        "actual": output,
        "correct": score(case["expected"], output),
    })

by_segment = group_scores(results, key="segment")

# Gate on the worst segment, never the mean.
assert min(by_segment.values()) >= 0.80, by_segment

Before it takes real traffic

Pin the model version. Aliases move under you; a pinned version does not.
Log the version with every result. Without it you cannot reproduce a past output.
Cap output length and set a spend alert. Unbounded generation is unbounded cost.
Decide the failure behaviour. What the product does when the model is unavailable is a product decision, not an afterthought.
Keep a kill switch. A way to disable the AI path in under a minute, without a deploy.
Tell users it is AI, and give them a route to a human.

Next: how this maps onto Azure and AWS.