wai.OpenAI("gpt-4.1-mini", api_key="sk-..."), wai.Anthropic(...), wai.Fireworks(...),
wai.models.Bedrock(model_id, region=), wai.Endpoint(model, url=),
wai.Ollama(...), wai.Hosted(). Its repr says where the call goes and
which key it uses, wai.configure(agent=, judge=, api_key=) sets it once
for the process, and print(wai.settings) shows what each role resolves
to. That page is Connect your agent.
This page is the string form underneath, for configs and command lines.
The agent is the first argument of simulate(). Pass the model as a
string, and the key comes from that provider’s usual environment
variable. Every request goes straight to that provider. The situation
writer runs on the same model, so no While key is involved.
The provider and the model are separated by a colon. DSPy and LiteLLM
use a slash (
dspy.LM("openai/gpt-4o-mini")), so that spelling is the
common typo; wai.configure and simulate both refuse it on the line you
typed it and name the colon form to use instead:
"vllm:Qwen/Qwen3-4B@http://localhost:8000/v1". A bare model
name ("gpt-4.1-mini") and a provider that is not in the table above are
refused the same way, naming the seven forms.
The window, and the key a reasoning model renames
A prompt is cut to fit a context window only where whileai knows the window: a While-hosted endpoint, or one you name yourself withZP_CONTEXT_TOKENS. Everywhere else the prompt goes out as you wrote it,
and the server’s own 400 is what shrinks it. Before this the openai:
path inherited the hosted pool’s 4,096 tokens, so a 15,000-character user
turn to a 272k-context model arrived as 7,500 characters with max_tokens
lowered from 4096 to 1522, and nothing in the return value said so (#755).
Where the squeeze does run, it is on the record rather than in the margin:
- the reply carries
_prompt_truncated: the window, the characters asked and sent, the reply budget asked and sent; - the rollout step carries
prompt_truncatedwith the same numbers; data.degradedgainsprompt_truncated, anddata.warningsthe line that names the fix, once per endpoint.
o3 and o4-mini, rejects max_tokens and
asks for max_completion_tokens. whileai sends max_tokens first, since
every vLLM, Ollama and pre-GPT-5 endpoint takes it, and renames the key
once on a 400 that names the other spelling. There is no table of model
ids to keep current, so the model released next month works too.
A host that is gone
A 404 is not a slow server. An endpoint that was stopped or renamed answers the same way forever, so the run counts permanent failures (404, 405, 410, 501) per host and stops after three against one host with nothing succeeding in between, naming the host and the way out. Three rather than one because a host swapping its route mid-deploy answers a single 404 and then serves (DEAD_HOST_STRIKES: convention, untested). A
5xx, a 429 and a timeout are not counted: those are cold containers and
rate limits, and they keep every retry they had. Before this a run against
a stopped app sat at 0/1000 rollouts for 7m14s with no error and no
mention of a 404 anywhere (#816).
Set the key the way you already do for that provider:
Amazon Bedrock
bedrock:<model-id> reaches any chat model Bedrock hosts on your AWS
account through its Converse API: a foundation model
(anthropic.claude-haiku-4-5-20251001-v1:0), a cross-region inference
profile (us.anthropic.claude-sonnet-5), or the ARN of a model you
imported yourself. Pin the region with @<region>; without it the SDK
reads AWS_REGION, then AWS_DEFAULT_REGION, then uses us-east-1.
Two ways in, both your own:
A model that refuses
temperature (Claude Sonnet 5 answers 400 “temperature is deprecated for this model”) is called once more without the field and samples at its own default.
wai.models.Bedrock(model_id, region=, api_key=), is the
same spec with a place for a key; it sits one dot down so the front door
stays small.
A model you trained and imported into Bedrock is the same spec with its
ARN as the model id, so the held-out measurement that scored the base
scores the served weights unchanged:
InvokeModel with the OpenAI chat body instead, so the SDK picks that
route from the ARN and you change nothing; and an idle import is
unloaded, so the first call after a pause took 96 s and returns
ModelNotReadyException until it is back, which the SDK waits through
(ten tries, fifteen seconds apart) before saying so in one sentence.
Neither route returns log-probabilities, so logprobs= has no effect.
Bedrock honors tool calling on imports only for GPT-OSS models.
Serve it on While
https://models.while.ai/v1 is one OpenAI-compatible endpoint in
front of every model your account registers, wherever it runs: a Bedrock
import in While’s account or in yours, or any /v1 server you host. Your
While key picks the account, model picks the row, and any OpenAI client
works, so a served model needs no SDK change at all:
A subdomain of your own:
PUT /domain {"subdomain": "acme"} makes
https://acme.models.while.ai/v1 answer for your account and no other;
a call there with another account’s key is refused. GET /v1/models lists
what you registered, DELETE /models/{name} removes one. Streaming works. An import that sat idle is restored on the
first call; until it is back the endpoint answers 503 with
code: model_starting and a Retry-After, which the OpenAI SDKs honor.
What While keeps: per model per day, the count of calls, errors and tokens,
and which five-minute windows the model answered in, and nothing else.
Never a prompt, a completion or a request log; Bedrock invocation logging
is off and the endpoint logs errors without content. Each account gets 600
requests a minute across its keys, answered with a 429 and a Retry-After
past that.
What it costs, for a model While hosts (one you handed over with
publish): $0.12 per unit-minute while it answers, counted in the same
five-minute windows Bedrock bills, $5 per unit per month while it is kept,
and $10 per import. A unit is a Bedrock Custom Model Unit; an 8B model is
2. A model in your own AWS account, or a server you run, is routed for
free. Add a card once under Account on while.ai; without one, publish
and calls to a hosted model answer 402 with code: billing_required.
Tools are functions
@wai.tool turns a typed function into the tool: the signature is the
schema, the docstring is the description, Annotated[str, "note"] or a
Google-style Args: block gives a parameter its note, and a parameter
with a default is optional. The mock world answers the calls, faults
first. To have the bodies answer instead, pass
execute=wai.Tool.dispatch([get_order, ...]). Raw OpenAI schema dicts
still work in the same list.
No tools at all yet
draft_tools writes plausible schemas from one sentence about the agent,
on the same key:
drafted, so you can tell it from a
declared tool. Replace them with your real schemas when you have them.
The judge
Same shape as the agent: a callable over a row, a verifier (wai.verify.MathEqual(), wai.verify.CodeExec(tests=...)), or a model
string on its own key. The judge is never the model it is judging; the
evals guide shows how to check it against people before you
trust it.
Three models, three arguments
Three models can take part in a run. A model string can name each one, and each goes in a different place.
Set them once for a machine with
WHILEAI_AGENT, WHILEAI_SURROGATE
(the writer) and WHILEAI_JUDGE instead. If agent and judge end up the
same model, data.degraded carries same_model and the warning says so.
The While key
Only the hosted parts need it. The SDK looks in this order and stops at the first it finds:
Keys start with
zp_.
What reaches While
Three things, and only when you ask for them:
Everything else runs on your machine.
wai status prints which key
the SDK will use and where it came from; wai login or
wai signup --email you@example.com gets one when you want the hosted
parts. The platform reference covers that side.