EasyCompilers

Vocabulary for this course

The terms the chapters assume, in plain words. You can open this from any chapter.

backoff
Waiting longer between each retry, so a service that is struggling gets room to recover.
context window
The most text, counted in tokens, that a model can consider in a single request.
cost
The money a request is billed for, usually based on how many input and output tokens it uses.
idempotency
The property that repeating the same request has the same effect as sending it once.
jitter
A small random amount added to a wait, so many clients do not retry at the same moment.
latency
The time from sending a request to receiving the full answer, including time spent waiting in a queue.
prompt caching
Reusing the stored copy a service keeps of unchanged leading text so later requests cost less and start faster.
rate limit
A cap on how many requests you may send in a period, after which the service refuses more.
schema
A written description of the fields a response must contain and the type of each value.
streaming
Receiving an answer in pieces as it is produced instead of waiting for the complete response.
structured output
A response constrained to a schema, so every field and type can be checked and used by code.
timeout
A limit on how long a request may wait before your code gives up and treats it as failed.
token
A small unit of text, such as part of a word, a whole word or punctuation, that a model reads and bills for.
tool call
A request from a model for your code to run a named function with arguments, with the result sent back.

LLM APIs from the ground up

Provider portability

“OpenAI-compatible” covers the common request shape and stops before the features you actually need. Own the types, normalise the errors and choose providers by capability and evaluation, not by brand.

Part 9 of 10

Intermediate15 min readBack to Courses

The compatible subset is smaller than the name

Every provider advertises an OpenAI-compatible endpoint, and the compatibility stops somewhere different each time. Messages, roles and choices are close enough to share a client; tool calls, streaming events, JSON schema, stop reasons and usage fields are where the differences live. A client that works against the first provider it was written for is a prototype, not a portable one.

A field that is optional in one API is required in another, and a name that means the same thing is spelled differently. max_tokens and max_completion_tokens, a string stop sequence and a list, a finish reason of length and one of max_tokens: small differences that a shared type hides until the first edge case.

The only honest test is the subset you actually use, run against each provider. Write that subset down as a contract, pin the API version, and re-run the contract when a provider announces a change. “Compatible” is a claim, not a guarantee.

An interface of your own

Put the provider behind an interface you own. Your code asks for a completion with messages and options and receives a reply with text, tool calls, usage and a finish reason; the adapter translates in both directions and no SDK object crosses the boundary.

Normalise the failures as carefully as the successes. Retryable, authentication, invalid request, content filtered and context overflow are five different decisions, and every provider expresses them differently. A shared error enum is what lets the retry policy stay provider-agnostic. The mapping table is small, and it is the difference between a retry policy and a pile of special cases.

Keep the escape hatch small and explicit. A provider-specific option bag is fine for a feature you have deliberately adopted; a branch on the model name inside the business logic is not. Log the provider and model on every call so a quality change can be attributed.

One call, a fallback chain

from litellm import completion

response = completion(
    model="anthropic/claude-sonnet-4-5",
    messages=messages,
    fallbacks=["openai/gpt-4.1-mini", "gemini/gemini-2.5-flash"],
)

print(response.choices[0].message.content)
print(response.usage.prompt_tokens, response.usage.completion_tokens)

Two shapes, one reply

Run both adapters and compare the normalised objects. Then add a third provider whose finish reason is “max_tokens” and map it to the same “length” value.

Output
Click "Run" to execute your code

Capabilities, not model names

Choose by capability, not by brand. Make a table of what the product needs — tool calls, JSON schema, vision, streaming, a context of a hundred thousand tokens, a maximum output — and mark which providers satisfy each. A model that is excellent at prose but cannot call a tool is not a candidate for the agent feature.

When a capability is missing, degrade on purpose. A provider without native tool calls can still answer a JSON protocol you validate yourself; a provider without schema enforcement can be repaired and re-asked. What must not happen is a silent fallback to plain text that the caller then parses hopefully. A degraded path should be visible in the response metadata, so the caller knows which mode it got.

Capability detection belongs in configuration, not in a string match on the model name. Names change, aliases move and a new version can add a feature without renaming anything. A table maintained next to the adapter is a smaller liability than a regular expression.

Write one recorded response per provider into the adapter tests, then assert that all of them normalise to the same reply object. The fixture is the contract.

A fallback is a second product

A fallback is a second product, not a retry. Switch only on errors that another provider might not have — a timeout, a 5xx, a rate limit — and never on an invalid request or a content-policy refusal, which will fail the same way everywhere.

Side effects make fallback dangerous. If the first provider may have charged a card or sent an email before timing out, the second attempt needs the same idempotency key and the same deduplication record, or the fallback becomes a double charge. The idempotency record belongs in a store both attempts can see, not in the process that happened to make the first call.

Quality, price, latency, data residency and retention all change when the provider changes. Keep an allow-list per tenant and per region, evaluate the fallback on the same fixed set as the primary, and record which provider answered so a complaint can be traced to the right model.

Choose by capability, then health

Run the chain, then add a fourth provider with tools, JSON and a healthy status. Confirm it is chosen first and that the unhealthy one is still skipped.

Output
Click "Run" to execute your code

Check what you learned

One question at a time. A wrong answer explains itself — take another go.

Part 1 of 5

What does an OpenAI-compatible endpoint guarantee?