Azure AI

Building on Azure AI Foundry: language, speech, vision, document and knowledge-mining services.

Azure AI Foundry is the front door: a portal where you browse models, deploy one into a project, and get an endpoint to call. Around it sit the task-specific services (speech, vision, document intelligence, search) that you reach for when a foundation model is the wrong tool.

The service map

Need	Service	Notes
Text generation and reasoning	Azure AI Foundry model catalogue	GPT, Phi and open-weight families behind one deployment and endpoint model.
Speech to text, text to speech	Azure AI Speech	Real-time from a microphone or batch from a file; SSML for prosody control.
Image captioning, tagging, object and people detection	Azure AI Vision	Prebuilt analysis without training anything.
Reading text in images	Azure AI Vision (OCR)	Line and word level results, each with a bounding polygon.
Face detection and analysis	Azure AI Face	Gated. Access to some capabilities requires an approved application.
Custom image classification or detection	Azure AI Custom Vision	Tag, train, evaluate and publish an iteration from the portal.
Structured extraction from forms	Azure AI Document Intelligence	Prebuilt models for invoices and receipts, or a custom model trained on your own layouts.
Multimodal extraction with your own schema	Azure AI Content Understanding	Define fields once, apply across documents, images, audio and video.
Search over your own corpus	Azure AI Search	Indexers, skillsets and a knowledge store. The retrieval half of RAG.

Deploying a model in AI Foundry

Choose the model from the catalogue

Open the Foundry portal, find the model, and read the Details and Benchmarks tabs before committing. Use Compare models to chart your shortlist on quality against cost.

Create a project

Use this model prompts you to create a project. Expand advanced options and set the Foundry resource, subscription, resource group and region explicitly rather than accepting defaults. The region is the part you cannot change later.

Set the rate limit deliberately

Deploy with the Global Standard deployment type and set a tokens-per-minute limit you actually intend. Quota is shared across the subscription, so an unbounded deployment can starve every other project in it.

Test in the playground

Set the real system prompt, then try genuinely hard inputs. Ask the model to explain its reasoning; the explanation is usually where a plausible-looking wrong answer gives itself away.

Take the endpoint into code

The project Overview page carries the endpoint you connect to. Authenticate with Entra ID rather than a key wherever you can.

Calling a Foundry project

The project client resolves your deployment and hands back an OpenAI-compatible chat client, so the call itself looks familiar.

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

project_client = AIProjectClient(
    credential=DefaultAzureCredential(),
    endpoint=PROJECT_ENDPOINT,
)

client = project_client.get_openai_client(api_version="2024-10-21")

response = client.chat.completions.create(
    model=MODEL_DEPLOYMENT_NAME,
    messages=[
        {"role": "system", "content": "You are an AI assistant that helps solve problems."},
        {"role": "user", "content": "Summarise this clinic note."},
    ],
)

print(response.choices[0].message.content)

Prefer Entra ID over keys

DefaultAzureCredential works from your laptop via az login and from a deployed app via managed identity, with no code change and no key to leak. Reach for a key only where identity genuinely is not available.

Audio and image prompts

Multimodal models take a content-part array rather than a plain string. Local files are base64 data URLs; remote ones are passed by URL.

import base64

with open("note.mp3", "rb") as f:
    encoded = base64.b64encode(f.read()).decode("utf-8")

response = client.chat.completions.create(
    model=MODEL_DEPLOYMENT_NAME,
    messages=[
        {"role": "system", "content": "You transcribe and summarise clinical audio."},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Summarise the key findings."},
                {
                    "type": "input_audio",
                    "input_audio": {"data": encoded, "format": "mp3"},
                },
            ],
        },
    ],
)

Speech

Azure AI Speech is a separate resource with its own key and region. Use it rather than a multimodal model when you need real-time transcription, word-level timings, or controllable synthesis.

import azure.cognitiveservices.speech as speech_sdk

config = speech_sdk.SpeechConfig(SPEECH_KEY, SPEECH_REGION)

# Recognise from a file; use AudioConfig(use_default_microphone=True) for live.
audio = speech_sdk.AudioConfig(filename="question.wav")
recognizer = speech_sdk.SpeechRecognizer(config, audio)
result = recognizer.recognize_once_async().get()

if result.reason == speech_sdk.ResultReason.RecognizedSpeech:
    print(result.text)

For synthesis, SSML gives you control that a plain string cannot: voice, rate, pitch, and explicit pauses:

<speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xml:lang="en-US">
  <voice name="en-GB-LibbyNeural">
    <prosody rate="0.9">
      Your appointment is confirmed for
      <break strength="weak"/>
      Thursday the fourteenth.
    </prosody>
  </voice>
</speak>

Vision and OCR

Image analysis returns several features from one call: caption, tags, objects and people, each with a bounding box and a confidence.

from azure.ai.vision.imageanalysis import ImageAnalysisClient
from azure.ai.vision.imageanalysis.models import VisualFeatures
from azure.core.credentials import AzureKeyCredential

client = ImageAnalysisClient(endpoint=AI_ENDPOINT, credential=AzureKeyCredential(AI_KEY))

with open("street.jpg", "rb") as f:
    result = client.analyze(
        image_data=f.read(),
        visual_features=[
            VisualFeatures.CAPTION,
            VisualFeatures.TAGS,
            VisualFeatures.OBJECTS,
            VisualFeatures.PEOPLE,
            VisualFeatures.READ,      # OCR in the same call
        ],
    )

print(result.caption.text, result.caption.confidence)

for block in result.read.blocks:
    for line in block.lines:
        print(line.text, line.bounding_polygon)

Face is a gated service

Azure AI Face restricts several capabilities behind an approved access application, and identification and verification are the ones most products want. Confirm you can get approval before designing around it. This is exactly the constraint shaping the Attender timeline.

Document Intelligence

Start with a prebuilt model. prebuilt-invoice and prebuilt-receipt already know what a total and a vendor are, and they return fields with a confidence each. Train a custom model only when your layout is genuinely not covered, since that needs a handful of labelled samples in blob storage and a training run.

from azure.ai.formrecognizer import DocumentAnalysisClient
from azure.core.credentials import AzureKeyCredential

client = DocumentAnalysisClient(ENDPOINT, AzureKeyCredential(KEY))

with open("invoice-1234.pdf", "rb") as f:
    poller = client.begin_analyze_document("prebuilt-invoice", document=f)

for invoice in poller.result().documents:
    for name in ("VendorName", "InvoiceTotal", "DueDate"):
        field = invoice.fields.get(name)
        if field:
            # Route anything low-confidence to a human rather than trusting it.
            print(name, field.value, round(field.confidence, 2))

Content Understanding

Where Document Intelligence is document-shaped, Content Understanding is schema-shaped: you define the fields you want once, then apply that analyser across documents, images, audio and video. In practice this is the fastest route from “we have a pile of mixed media” to structured rows.

Create a custom task in the Foundry portal and upload a sample.
Start from a template where one fits (invoice extraction, for instance), then edit the field list.
For each field choose the method: extract a value present in the content, classify into a fixed set, or generate a value the model infers.
Build the analyser, test it, then call it by id from code.

Knowledge mining with AI Search

AI Search is the retrieval half of a RAG system. An indexer pulls from a data source such as Blob Storage, a skillset enriches each document on the way through (OCR, key phrases, language detection, entity recognition), and the results land in an index, optionally projected into a knowledge store for downstream analytics.

Land the documents

Upload the corpus into a Blob Storage container.

Import and enrich

From the Search resource, use Import data to connect the data source, attach cognitive skills, and map enriched fields into the index.

Mark fields correctly

Retrievable, filterable, sortable, facetable and searchable are separate flags, and each one you enable costs index size. Turn on what the query actually needs.

Query it

Use the search client from your application, then pass the retrieved passages to a model as grounding, always keeping the source reference so the answer can cite it.

from azure.search.documents import SearchClient
from azure.core.credentials import AzureKeyCredential

search = SearchClient(SEARCH_ENDPOINT, INDEX_NAME, AzureKeyCredential(SEARCH_KEY))

results = search.search(
    search_text="retinopathy screening interval",
    select=["title", "content", "url"],
    top=5,
)

grounding = [
    {"source": r["url"], "text": r["content"][:1500]}
    for r in results
]

Clean up when you are done

Search resources, Speech resources and model deployments bill while they exist, not while they are used. Delete the resource group when an experiment ends. This is the single most common source of surprise Azure spend.

The same build on the other platform: AWS AI.