# Workers Ai
# Agents
import { LinkButton } from "~/components"
Build AI assistants that can perform complex tasks on behalf of your users using Cloudflare Workers AI and Agents.
Go to Agents documentation
# Changelog
import { ProductReleaseNotes } from "~/components";
{/* */}
# Cloudflare Workers AI
import {
CardGrid,
Description,
Feature,
LinkTitleCard,
Plan,
RelatedProduct,
Render,
LinkButton,
Flex,
} from "~/components";
Run machine learning models, powered by serverless GPUs, on Cloudflare's global network.
Workers AI allows you to run AI models in a serverless way, without having to worry about scaling, maintaining, or paying for unused infrastructure. You can invoke models running on GPUs on Cloudflare's network from your own code — from [Workers](/workers/), [Pages](/pages/), or anywhere via [the Cloudflare API](/api/resources/ai/methods/run/).
Workers AI gives you access to:
- **50+ [open-source models](/workers-ai/models/)**, available as a part of our model catalog
- Serverless, **pay-for-what-you-use** [pricing model](/workers-ai/platform/pricing/)
- All as part of a **fully-featured developer platform**, including [AI Gateway](/ai-gateway/), [Vectorize](/vectorize/), [Workers](/workers/) and more...
Get started
Watch a Workers AI demo
---
## Features
Workers AI comes with a curated set of popular open-source models that enable you to do tasks such as image classification, text generation, object detection and more.
---
## Related products
Observe and control your AI applications with caching, rate limiting, request retries, model fallback, and more.
Build full-stack AI applications with Vectorize, Cloudflare’s vector database. Adding Vectorize enables you to perform tasks such as semantic search, recommendations, anomaly detection or can be used to provide context and memory to an LLM.
Build serverless applications and deploy instantly across the globe for exceptional performance, reliability, and scale.
Create full-stack applications that are instantly deployed to the Cloudflare global network.
Store large amounts of unstructured data without the costly egress bandwidth fees associated with typical cloud storage services.
Create new serverless SQL databases to query from your Workers and Pages projects.
A globally distributed coordination API with strongly consistent storage.
Create a global, low-latency, key-value data storage.
---
## More resources
Build and deploy your first Workers AI application.
Learn about Free and Paid plans.
Learn about Workers AI limits.
Learn how you can build and deploy ambitious AI applications to Cloudflare's
global network.
Learn which storage option is best for your project.
Connect with the Workers community on Discord to ask questions, share what you
are building, and discuss the platform with other developers.
Follow @CloudflareDev on Twitter to learn about product announcements, and
what is new in Cloudflare Workers.
# Extraction models
Extraction models turn information into specific values your application can use, such as an invoice total, a due date, or an estimated payment amount. They return typed values rather than choosing from a fixed set of options or producing open-ended content. This helps you connect AI predictions to application logic.
You provide an input state and define the values you want the model to return, including each value's type and any additional constraints. The model uses these definitions and the input state to produce a corresponding typed result for each value. Typed results provide a predictable structure but do not guarantee correctness.
## When to use extraction models
Use an extraction model when your application needs to extract, compute, estimate, or forecast a value, such as an amount, date, or duration. Alternatively, use a decision model to select from known options, or a generative model to produce open-ended content.
The following table compares these model types using invoice examples:
| Model type | Use when you need | Invoice example |
| ---------- | ------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| Extraction | Extract, compute, estimate, or forecast values, such as amounts, dates, durations, or text. | Read the total and due date from an invoice. |
| Decision | Select from a predefined set of options. | Determine if an invoice requires approval, manual review, or rejection. |
| Generative | Produce open-ended content. | Summarize discrepancies between the invoice, purchase order, and delivery receipt. |
Your application can use returned values in application logic or pass them as inputs to another model.
## How extraction models work
An extraction request combines an input state with a set of named field definitions. The input state supplies the information the model evaluates. Each field definition describes a value to return, specifies its type, and may include constraints. The model uses both to produce a corresponding result for each field.
For example, you provide invoice text that states a total of $1,080 and a due date of 2026-11-01. Separately, you define an invoice total field as a number in USD and a due date field as a date.
The model uses these definitions to identify the requested values in the invoice. It returns `1080` and `2026-11-01` as the corresponding typed results.
Your application can then use the returned values. For example, it can compare the due date with the current date to flag an overdue invoice.
## Extraction model capabilities
Extraction models support extraction, computation, estimation, and forecasting.
### Extraction
Return a value explicitly present in the input, such as the total printed on an invoice. Extraction identifies an existing value rather than calculating or estimating one.
### Computation
Derive a value from facts in the input. For example, calculate how many days an invoice is overdue using its due date and a current date you provide. Unlike extraction, the result does not need to appear explicitly in the input.
### Estimation
Predict a value that the input does not determine, such as when an unpaid invoice will be paid. Unlike computation, estimation addresses an unknown outcome rather than deriving a result from supplied facts.
### Forecasting
Predict a sequence of future values over an interval, such as an invoice's outstanding balance each week for the next month. Unlike a single estimate, a forecast returns a value for each step in the requested period.
## Interpret extraction results
Types and constraints define the form a result can take. They do not guarantee that it answers your question correctly.
### Missing results
For fields that extract or compute a single value, a `null` result indicates that the model considers the input insufficient to determine the answer. The information may be missing, or the model may have failed to identify it.
Lists return an empty list when the model finds no matching values. Estimates and forecasts return predictions even when the input provides little information.
### Uncertainty
The field type determines how uncertainty is represented and how it can be interpreted. Fields that extract and compute a single value include confidence scores for the returned answer. Estimates and forecasts include distributions over possible values.
For example, an estimated payment date might have a 10th percentile of November 5 and a 90th percentile of November 20. These dates mark the middle 80% of the predicted distribution. The actual payment date can still fall outside that interval.
### Constraints
Constraints restrict the values the model can return. They do not guarantee that the answer is correct.
For example, restricting an estimated payment date to November ensures that the returned date falls in November. The invoice could still be paid in December.
### Source spans
Extracted text includes a source span, which identifies its start and end positions in the input. You can use these positions to find the returned text in its original context.
## Design extraction tasks
### Define clear fields
Give each field a descriptive name and use its instructions to explain what value to return. For example, a field named `invoice_total` could have the instructions "Total amount due, including tax and shipping, after discounts." Specify its type and unit separately as a number in USD.
### Choose appropriate types and constraints
Set each field's type according to how your application uses the result. For example, use a date for a payment deadline and a duration for days overdue.
Use the available field settings to specify precision and value bounds. For example, request two decimal places for an invoice amount. When bounding an estimated payment date, choose a window that reflects plausible payment dates, not just your preferred deadline.
### Include task-relevant context
Include the information each task depends on in the input state. For example, calculating days overdue requires the invoice's due date and a current date you provide. State which date to use rather than assuming the model knows when the request runs.
### Match safeguards to risk
Choose safeguards based on the consequences of an incorrect result. Displaying an estimated payment date may require fewer checks than using an extracted amount to authorize a payment.
Decide how your application handles missing or uncertain results. Before consequential or irreversible actions, check results against other evidence or require human review.
## Limitations
- Typed output does not guarantee correctness. A result can satisfy its type and constraints while still answering the question incorrectly.
- Results depend on the available information and field definitions. Returning an estimate does not mean the input provides enough information to predict the outcome accurately.
- Uncertainty estimates are not guarantees. Confidence scores and prediction intervals do not replace application-level checks. An interval's actual coverage can vary across tasks and datasets.
- Capabilities vary by model. Consult the [Clairic model documentation](/workers-ai/models/clairic/) for supported inputs, field types, metadata, context window, and model-specific limitations.
## Next steps
- Review the [Clairic model reference](/workers-ai/models/clairic/) for supported inputs, field types, model-specific limits, request and response schemas, and invoice examples.
- Consult the [Workers AI REST API reference](/api/resources/ai/methods/run/) for the inference endpoint.
- [Configure a Workers AI binding](/workers-ai/configuration/bindings/) to access Workers AI from a Worker.
- [Use the Workers AI REST API](/workers-ai/get-started/rest-api/) to authenticate and send requests over HTTP.
# Concepts
import { DirectoryListing } from "~/components";
# Vercel AI SDK
import {
PackageManagers,
TypeScriptExample,
WranglerConfig,
} from "~/components";
To use Workers AI with the [AI SDK](https://ai-sdk.dev/), use `createAI` from `agents/models/ai-sdk`. This page covers Workers AI models. For AI Gateway catalog models, fallback, gateway options, and the full option reference, refer to [AI SDK](/agents/models/ai-sdk/).
The `workers-ai-provider` package is deprecated. Use `createAI` instead.
## Setup
Install the Agents SDK and the AI SDK. `createAI` needs `ai@^7` and `@ai-sdk/provider@^4`:
Add the `AI` binding to your Wrangler configuration:
```jsonc
{
"ai": {
"binding": "AI",
},
}
```
## Choose a model
Pass the id of any [Workers AI model](/workers-ai/models/). Model ids start with `@cf/` and autocomplete from `@cloudflare/workers-types`:
```ts
import { createAI } from "agents/models/ai-sdk";
const ai = createAI({ binding: env.AI });
const model = ai("@cf/zai-org/glm-4.7-flash");
```
## Generate text
```ts
import { createAI } from "agents/models/ai-sdk";
import { generateText } from "ai";
export default {
async fetch(request: Request, env: Env) {
const ai = createAI({ binding: env.AI });
const { text } = await generateText({
model: ai("@cf/zai-org/glm-4.7-flash"),
prompt: "Write a 50-word essay about hello world.",
});
return new Response(text);
},
};
```
## Stream text
For longer responses, stream the output as the model generates it:
```ts
import { createAI } from "agents/models/ai-sdk";
import { streamText } from "ai";
export default {
async fetch(request: Request, env: Env) {
const ai = createAI({ binding: env.AI });
const result = streamText({
model: ai("@cf/zai-org/glm-4.7-flash"),
prompt: "Write a 50-word essay about hello world.",
});
return result.toTextStreamResponse();
},
};
```
## Generate structured output
Pass a Zod schema to get a typed JSON response:
```ts
import { createAI } from "agents/models/ai-sdk";
import { generateText, Output } from "ai";
import { z } from "zod";
export default {
async fetch(request: Request, env: Env) {
const ai = createAI({ binding: env.AI });
const { output } = await generateText({
model: ai("@cf/zai-org/glm-4.7-flash"),
prompt: "Generate a lasagna recipe.",
output: Output.object({
schema: z.object({
recipe: z.object({
ingredients: z.array(z.string()),
description: z.string(),
}),
}),
}),
});
return Response.json(output);
},
};
```
## Use other model types
Embeddings, images, speech, transcription, and reranking come from the same provider:
```ts
import {
embedMany,
generateImage,
generateSpeech,
rerank,
transcribe,
} from "ai";
const { embeddings } = await embedMany({
model: ai.embedding("@cf/baai/bge-base-en-v1.5"),
values: ["hello", "world"],
});
const { image } = await generateImage({
model: ai.image("@cf/black-forest-labs/flux-1-schnell"),
prompt: "A cyberpunk lizard",
});
const { audio } = await generateSpeech({
model: ai.speech("@cf/deepgram/aura-1"),
text: "Hello from Cloudflare.",
});
const { text } = await transcribe({
model: ai.transcription("@cf/openai/whisper-large-v3-turbo"),
audio: audio.uint8Array,
});
const { rerankedDocuments } = await rerank({
model: ai.reranking("@cf/baai/bge-reranker-base"),
documents: ["a cyberpunk lizard", "a rainy Tuesday"],
query: "Which one is cooler?",
});
```
For model-specific settings, refer to [Use other modalities](/agents/models/ai-sdk/#use-other-modalities).
## Use AI Gateway
Requests go through [AI Gateway](/ai-gateway/). The gateway defaults to `default`. To use another gateway, set its `id`:
```ts
const ai = createAI({ binding: env.AI, id: "my-gateway" });
```
The same provider also calls models from the [AI Gateway catalog](/ai/models/), such as `anthropic/claude-opus-5.5`. Refer to [AI SDK](/agents/models/ai-sdk/) for catalog models, caching, fallback, and the other gateway options.
## Related resources
- [AI SDK provider reference](/agents/models/ai-sdk/)
- [Workers AI models](/workers-ai/models/)
- [AI SDK documentation](https://ai-sdk.dev/docs/introduction)
# Workers Bindings
import { Type, MetaInfo, WranglerConfig } from "~/components";
## Workers
[Workers](/workers/) provides a serverless execution environment that allows you to create new applications or augment existing ones.
To use Workers AI with Workers, you must create a Workers AI [binding](/workers/runtime-apis/bindings/). Bindings allow your Workers to interact with resources, like Workers AI, on the Cloudflare Developer Platform. You create bindings on the Cloudflare dashboard or by updating your [Wrangler file](/workers/wrangler/configuration/).
To bind Workers AI to your Worker, add the following to the end of your Wrangler file:
```jsonc
{
"ai": {
"binding": "AI" // i.e. available in your Worker on env.AI
}
}
```
## Pages Functions
[Pages Functions](/pages/functions/) allow you to build full-stack applications with Cloudflare Pages by executing code on the Cloudflare network. Functions are Workers under the hood.
To configure a Workers AI binding in your Pages Function, you must use the Cloudflare dashboard. Refer to [Workers AI bindings](/pages/functions/bindings/#workers-ai) for instructions.
## Methods
### async env.AI.run()
`async env.AI.run()` runs a model. Takes a model as the first parameter, and an object as the second parameter.
```javascript
const answer = await env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
prompt: "What is the origin of the phrase 'Hello, World'"
});
```
**Parameters**
* `model`
* The model to run.
**Supported options**
* `stream`
* Returns a stream of results as they are available.
```javascript
const answer = await env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
prompt: "What is the origin of the phrase 'Hello, World'",
stream: true
});
return new Response(answer, {
headers: { "content-type": "text/event-stream" }
});
```
# Hugging Face Chat UI
Use Workers AI with [Chat UI](https://github.com/huggingface/chat-ui?tab=readme-ov-file#text-embedding-models), an open-source chat interface offered by Hugging Face.
## Prerequisites
You will need the following:
* A [Cloudflare account](https://dash.cloudflare.com)
* Your [Account ID](/fundamentals/account/find-account-and-zone-ids/)
* An [API token](/workers-ai/get-started/rest-api/#1-get-api-token-and-account-id) for Workers AI
## Setup
First, decide how to reference your Account ID and API token (either directly in your `.env.local` using the `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` variables or in the endpoint configuration).
Then, follow the rest of the setup instructions in the [Chat UI GitHub repository](https://github.com/huggingface/chat-ui?tab=readme-ov-file#text-embedding-models).
When setting up your models, specify the `cloudflare` endpoint.
```json
{
"name" : "nousresearch/hermes-2-pro-mistral-7b",
"tokenizer": "nousresearch/hermes-2-pro-mistral-7b",
"parameters": {
"stop": ["<|im_end|>"]
},
"endpoints" : [
{
"type": "cloudflare",
// optionally specify these if not included in .env.local
"accountId": "your-account-id",
"apiToken": "your-api-token"
//
}
]
}
```
## Supported models
This template works with any [text generation models](/workers-ai/models/) that begin with the `@hf` parameter.
# Configuration
import { DirectoryListing } from "~/components";
# OpenAI compatible API endpoints
import { Render } from "~/components";
## Usage
### Chat Completions and embeddings
Most Workers AI text generation models support the OpenAI Chat Completions API. Embedding models support the OpenAI Embeddings API.
Use the [OpenAI JavaScript SDK](https://github.com/openai/openai-node) by setting the Workers AI base URL, API token, and model name.
```js title="OpenAI SDK example"
import OpenAI from "openai";
const openai = new OpenAI({
apiKey: env.CLOUDFLARE_API_KEY,
baseURL: `https://api.cloudflare.com/client/v4/accounts/${env.CLOUDFLARE_ACCOUNT_ID}/ai/v1`,
});
const chatCompletion = await openai.chat.completions.create({
messages: [{ role: "user", content: "Make some robot noises" }],
model: "@cf/meta/llama-3.1-8b-instruct",
});
const embeddings = await openai.embeddings.create({
model: "@cf/baai/bge-large-en-v1.5",
input: "I love matcha",
});
```
```bash title="cURL example"
curl --request POST \
--url https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/v1/chat/completions \
--header "Authorization: Bearer {api_token}" \
--header "Content-Type: application/json" \
--data '
{
"model": "@cf/meta/llama-3.1-8b-instruct",
"messages": [
{
"role": "user",
"content": "how to build a wooden spoon in 3 short steps? give as short as answer as possible"
}
]
}
'
```
### Fail fast when capacity is unavailable
For synchronous Chat Completions, set `options.rejectIfBusy` in the top-level request body. This makes the request fail instead of waiting in a capacity queue.
OpenAI clients that preserve custom fields can send this option. Clients that remove unknown fields do not apply it, so requests proceed normally.
Refer to [Reject busy requests](/workers-ai/features/reject-if-busy/) for examples and error behavior.
### Responses API for GPT-OSS
The Responses API is supported only by the [`@cf/openai/gpt-oss-120b`](/workers-ai/models/gpt-oss-120b/) and [`@cf/openai/gpt-oss-20b`](/workers-ai/models/gpt-oss-20b/) models. Responses requests must be non-streaming, only `stream: false` is supported.
```js title="GPT-OSS Responses API example"
import OpenAI from "openai";
const openai = new OpenAI({
apiKey: env.CLOUDFLARE_API_KEY,
baseURL: `https://api.cloudflare.com/client/v4/accounts/${env.CLOUDFLARE_ACCOUNT_ID}/ai/v1`,
});
const response = await openai.responses.create({
model: "@cf/openai/gpt-oss-120b",
input: "Talk to me about open source",
});
```
### AI Gateway
These endpoints are also compatible with [AI Gateway](/ai-gateway/usage/providers/workersai/#openai-compatible-endpoints).
# Pi AI
import {
PackageManagers,
TypeScriptExample,
WranglerConfig,
} from "~/components";
To use Workers AI with [Pi AI](https://github.com/earendil-works/pi/tree/main/packages/ai), use `createAI` from `agents/models/pi-ai`. It returns Pi AI `Model` objects, so Workers AI models work with Pi AI's `stream` and `complete`, with [Pi Durable](https://github.com/earendil-works/pi/tree/main/packages/durable), and with the [Pi harness](/agents/harnesses/pi/). This page covers Workers AI models. For AI Gateway catalog models, fallback, gateway options, and the full reference, refer to [Pi AI](/agents/models/pi-ai/).
## Setup
Install the Agents SDK and Pi AI. `createAI` needs `@earendil-works/pi-ai@~1.4.0`:
Add the `AI` binding to your Wrangler configuration:
```jsonc
{
"ai": {
"binding": "AI",
},
}
```
## Generate text
Name a Workers AI model by its `@cf/` id. The provider carries Pi AI's entry points, so nothing is registered globally:
```ts
import { createAI } from "agents/models/pi-ai";
export default {
async fetch(request: Request, env: Env) {
const ai = createAI({ binding: env.AI });
const reply = await ai.complete(ai("@cf/zai-org/glm-4.7-flash"), {
messages: [
{ role: "user", content: "What is Cloudflare?", timestamp: Date.now() },
],
});
// `content` holds text, thinking, and tool call parts.
const text = reply.content
.filter((part) => part.type === "text")
.map((part) => part.text)
.join("");
return Response.json({ text });
},
};
```
## Stream text
`ai.stream()` returns Pi AI's event stream. `ai.streamSimple()` and `ai.completeSimple()` take Pi AI's simple options, such as a `reasoning` level and `maxTokens`:
```ts
const model = ai("@cf/zai-org/glm-4.7-flash");
const context = {
messages: [{ role: "user", content: "Hi", timestamp: Date.now() }],
};
for await (const event of ai.stream(model, context)) {
// text_delta, thinking_delta, toolcall_*, done, error
}
```
A compatibility layer turns each Workers AI model's response into the shape Pi AI's parsers read, so tool calls and reasoning work the same across model families.
## Set model details
A Workers AI id that Pi AI does not list resolves with a 128,000 token context window, 8,192 output tokens, and zero cost. Pass the real values when you know them:
```ts
const model = ai("@cf/zai-org/glm-4.7-flash", {
contextWindow: 131_072,
maxTokens: 16_384,
});
```
## Generate images
```ts
const images = ai.images("@cf/black-forest-labs/flux-1-schnell", { steps: 8 });
const result = await ai.generateImages(images, {
input: [{ type: "text", text: "A red bicycle" }],
});
```
## Use AI Gateway
Every request goes through AI Gateway. Set the gateway with `id`, which defaults to `default`:
```ts
const ai = createAI({ binding: env.AI, id: "my-gateway" });
```
To call from outside a Worker, pass `accountId` and `apiToken` instead of `binding`. For gateway options, catalog models, and fallback, refer to [Pi AI](/agents/models/pi-ai/).
## Related resources
- [Pi AI provider reference](/agents/models/pi-ai/)
- [Pi harness](/agents/harnesses/pi/)
- [Workers AI models](/workers-ai/models/)
- [Vercel AI SDK](/workers-ai/configuration/ai-sdk/)
# Features
import { DirectoryListing } from "~/components";
# JSON Mode
import { Code } from "~/components";
export const jsonModeSchema = `{
response_format: {
title: "JSON Mode",
type: "object",
properties: {
type: {
type: "string",
enum: ["json_object", "json_schema"],
},
json_schema: {},
}
}
}`;
export const jsonModeRequestExample = `{
"messages": [
{
"role": "system",
"content": "Extract data about a country."
},
{
"role": "user",
"content": "Tell me about India."
}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"capital": {
"type": "string"
},
"languages": {
"type": "array",
"items": {
"type": "string"
}
}
},
"required": [
"name",
"capital",
"languages"
]
}
}
}`;
export const jsonModeResponseExample = `{
"response": {
"name": "India",
"capital": "New Delhi",
"languages": [
"Hindi",
"English",
"Bengali",
"Telugu",
"Marathi",
"Tamil",
"Gujarati",
"Urdu",
"Kannada",
"Odia",
"Malayalam",
"Punjabi",
"Sanskrit"
]
}
}`;
When we want text-generation AI models to interact with databases, services, and external systems programmatically, typically when using tool calling or building AI agents, we must have structured response formats rather than natural language.
Workers AI supports JSON Mode, enabling applications to request a structured output response when interacting with AI models.
## Schema
JSON Mode is compatible with OpenAI’s implementation; to enable add the `response_format` property to the request object using the following convention:
Where `json_schema` must be a valid [JSON Schema](https://json-schema.org/) declaration.
## JSON Mode example
When using JSON Format, pass the schema as in the example below as part of the request you send to the LLM.
The LLM will follow the schema, and return a response such as below:
As you can see, the model is complying with the JSON schema definition in the request and responding with a validated JSON object.
## Supported Models
This is the list of models that now support JSON Mode:
- [@cf/meta/llama-3.3-70b-instruct-fp8-fast](/workers-ai/models/llama-3.3-70b-instruct-fp8-fast/)
- [@cf/meta/llama-3-8b-instruct](/workers-ai/models/llama-3-8b-instruct/)
- [@cf/meta/llama-3.1-8b-instruct](/workers-ai/models/llama-3.1-8b-instruct/)
- [@hf/nousresearch/hermes-2-pro-mistral-7b](/workers-ai/models/hermes-2-pro-mistral-7b/)
- [@hf/thebloke/deepseek-coder-6.7b-instruct-awq](/workers-ai/models/deepseek-coder-6.7b-instruct-awq/)
- [@cf/deepseek-ai/deepseek-r1-distill-qwen-32b](/workers-ai/models/deepseek-r1-distill-qwen-32b/)
We will continue extending this list to keep up with new, and requested models.
Note that Workers AI can't guarantee that the model responds according to the requested JSON Schema. Depending on the complexity of the task and adequacy of the JSON Schema, the model may not be able to satisfy the request in extreme situations. If that's the case, then an error `JSON Mode couldn't be met` is returned and must be handled.
JSON Mode currently doesn't support streaming.
# Prompt caching
Prompt caching (also called prefix caching) is a performance optimization that allows Workers AI to respond faster to requests with prompts that share common inputs. It reduces Time to First Token (TTFT) and increases Tokens Per Second (TPS) throughput by reusing previously computed input tensors instead of reprocessing them from scratch.
Cached input tokens are billed at a discounted rate compared to regular input tokens. Workers AI enables prefix caching by default for select models. Compatibility and pricing details are listed on each [model page](/workers-ai/models/).
## How it works
When an LLM processes a request, it goes through two stages:
1. **Prefill stage** — processes input tokens (system prompts, tool definitions, conversation history).
2. **Output stage** — generates output tokens.
With prefix caching, Workers AI stores the computed input tensors from the prefill stage. On subsequent requests that share the same prefix, the model skips prefill for the cached portion and only processes the new input tokens. This saves significant compute time, especially for agentic workloads where consecutive requests share large amounts of context.
For example, when a coding agent sends a new prompt, it typically resends all previous prompts, tool definitions, and conversation history. The delta between consecutive requests is often just a few new lines. Prefix caching avoids redundant prefill on all the shared context.
## Session affinity header
Prefix caching only works when a request routes to the same model instance that holds the cached tensors. To maximize cache hit rates, send the `x-session-affinity` header with a unique identifier for your session or agent. This routes requests with the same identifier to the same model instance, increasing the likelihood of a prefix cache hit.
### REST API
```bash
curl -X POST \
"https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/run/@cf/moonshotai/kimi-k2.5" \
-H "Authorization: Bearer {api_token}" \
-H "Content-Type: application/json" \
-H "x-session-affinity: ses_12345678" \
-d '{
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "What is prefix caching and why does it matter?"
}
],
"max_tokens": 2400,
"stream": true
}'
```
### Workers AI binding
```js
const response = await env.AI.run(
"@cf/moonshotai/kimi-k2.5",
{
messages: [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: "Explain prefix caching." },
],
},
{
extraHeaders: {
"x-session-affinity": "ses_12345678",
},
},
);
```
## Structuring prompts for caching
Prefix caching matches the exact token sequence from the start of the prompt. A single token difference invalidates the cache from that point onward.
To maximize cache hits:
- **Place static content first.** System prompts, tool definitions, and shared instructions should appear at the beginning of the prompt. Put user-specific or dynamic content (timestamps, user queries) at the end.
- **Avoid timestamps in system prompts.** Including a timestamp at the start of a system prompt changes the prefix on every request, defeating the cache entirely. If time context is required, add it to the user message instead.
- **Reuse tool definitions across requests.** For function-calling agents, tools are part of the prompt prefix. Keeping tool definitions consistent across requests in the same session increases cache reuse.
## Monitoring cached tokens
Workers AI surfaces cached token counts in the response `usage` object. Use this to verify that prefix caching is working and to track cost savings. The first request will usually be cold, so it is expected that cached tokens are not returned on the first hit. Inputs need to be sufficiently large enough in order to be cached due to block size.
Cached tokens are billed at a lower rate than regular input tokens, which get totalled into your neuron count.
# Prompting
import { Code } from "~/components";
export const scopedExampleOne = `{
messages: [
{ role: "system", content: "you are a very funny comedian and you like emojis" },
{ role: "user", content: "tell me a joke about cloudflare" },
],
};`;
export const scopedExampleTwo = `{
messages: [
{ role: "system", content: "you are a professional computer science assistant" },
{ role: "user", content: "what is WASM?" },
{ role: "assistant", content: "WASM (WebAssembly) is a binary instruction format that is designed to be a platform-agnostic" },
{ role: "user", content: "does Python compile to WASM?" },
{ role: "assistant", content: "No, Python does not directly compile to WebAssembly" },
{ role: "user", content: "what about Rust?" },
],
};`;
export const unscopedExampleOne = `{
prompt: "tell me a joke about cloudflare";
}`;
export const unscopedExampleTwo = `{
prompt: "[INST]comedian[/INST]\n[INST]tell me a joke about cloudflare[/INST]",
raw: true
};`;
Part of getting good results from text generation models is asking questions correctly. LLMs are usually trained with specific predefined templates, which should then be used with the model's tokenizer for better results when doing inference tasks.
There are two ways to prompt text generation models with Workers AI:
:::note[Important]
We recommend using unscoped prompts for inference with LoRA.
:::
### Scoped Prompts
This is the recommended method. With scoped prompts, Workers AI takes the burden of knowing and using different chat templates for different models and provides a unified interface to developers when building prompts and creating text generation tasks.
Scoped prompts are a list of messages. Each message defines two keys: the role and the content.
Typically, the role can be one of three options:
- system - System messages define the AI's personality. You can
use them to set rules and how you expect the AI to behave.
- user - User messages are where you actually query the AI by
providing a question or a conversation.
- assistant - Assistant messages hint to the AI about the
desired output format. Not all models support this role.
OpenAI has a [good explanation](https://platform.openai.com/docs/guides/text-generation#messages-and-roles) of how they use these roles with their GPT models. Even though chat templates are flexible, other text generation models tend to follow the same conventions.
Here's an input example of a scoped prompt using system and user roles:
Here's a better example of a chat session using multiple iterations between the user and the assistant.
Note that different LLMs are trained with different templates for different use cases. While Workers AI tries its best to abstract the specifics of each LLM template from the developer through a unified API, you should always refer to the model documentation for details. For example, instruct models like Codellama are fine-tuned to respond to a user-provided instruction, while chat models expect fragments of dialogs as input.
### Unscoped Prompts
You can use unscoped prompts to send a single question to the model without worrying about providing any context. Workers AI will automatically convert your `prompt` input to a reasonable default scoped prompt internally so that you get the best possible prediction.
You can also use unscoped prompts to construct the model chat template manually. In this case, you can use the raw parameter. Here's an input example of a [Mistral](https://docs.mistral.ai/models/#chat-template) chat template prompt:
# Reject busy requests
import { TypeScriptExample } from "~/components";
Set `rejectIfBusy` when your application should not wait in a capacity queue. Workers AI rejects the synchronous inference request if capacity is unavailable.
## Send a REST request
For the native REST API, add `rejectIfBusy` to the request `options` object:
```bash
curl --request POST \
--url "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai/run/@cf/google/gemma-4-26b-a4b-it" \
--header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
--header "Content-Type: application/json" \
--data '{
"messages": [
{
"role": "user",
"content": "Explain what a capacity queue is."
}
],
"options": {
"rejectIfBusy": true
}
}'
```
## Use the Workers binding
For the Workers AI binding, pass `rejectIfBusy` in the third argument to `env.AI.run()`:
```ts
const response = await env.AI.run(
"@cf/google/gemma-4-26b-a4b-it",
{
messages: [
{
role: "user",
content: "Explain what a capacity queue is.",
},
],
},
{ rejectIfBusy: true },
);
```
Do not add `rejectIfBusy` to the model input object. The binding only applies this option from the third argument.
## Call Chat Completions
For OpenAI-compatible Chat Completions, add `options` at the top level of the request body:
```bash
curl --request POST \
--url "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai/v1/chat/completions" \
--header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
--header "Content-Type: application/json" \
--data '{
"model": "@cf/google/gemma-4-26b-a4b-it",
"messages": [
{
"role": "user",
"content": "Explain what a capacity queue is."
}
],
"options": {
"rejectIfBusy": true
}
}'
```
OpenAI clients that preserve custom fields can send this option. Clients that remove unknown fields do not apply it, so requests proceed normally.
## Handle capacity errors
Rejected requests return HTTP status `429` and internal error code `3040`. The error message is `Capacity temporarily exceeded, please try again.`
Refer to [Workers AI errors](/workers-ai/platform/errors/) for error details.
# Asynchronous Batch API
import { Render, PackageManagers, WranglerConfig, CURL } from "~/components";
Asynchronous batch processing lets you send a collection (batch) of inference requests in a single call. Instead of expecting immediate responses for every request, the system queues them for processing and returns the results later.
Batch processing is useful for large workloads such as summarization or embeddings when there is no human interaction. Using the batch API will guarantee that your requests are fulfilled eventually, rather than erroring out if Cloudflare does not have enough capacity at a given time.
When you send a batch request, the API immediately acknowledges receipt with a status like `queued` and provides a unique `request_id`. This ID is later used to poll for the final responses once the processing is complete.
You can use the Batch API by either creating and deploying a Cloudflare Worker that leverages the [Batch API with the AI binding](/workers-ai/features/batch-api/workers-binding/), using the [REST API](/workers-ai/features/batch-api/rest-api/) directly or by starting from a [template](https://github.com/craigsdennis/batch-please-workers-ai).
:::note[Note]
Ensure that the total payload is under 10 MB.
:::
## Demo application
If you want to get started quickly, click the button below:
[](https://deploy.workers.cloudflare.com/?url=https://github.com/craigsdennis/batch-please-workers-ai)
This will create a repository in your GitHub account and deploy a ready-to-use Worker that demonstrates how to use Cloudflare's Asynchronous Batch API. The template includes preconfigured AI bindings, and examples for sending and retrieving batch requests with and without external references. Once deployed, you can visit the live Worker and start experimenting with the Batch API immediately.
## Supported Models
Refer to our [model catalog](/workers-ai/models/?capabilities=Batch) for supported models.
# REST API
If you prefer to work directly with the REST API instead of a [Cloudflare Worker](/workers-ai/features/batch-api/workers-binding/), below are the steps on how to do it:
## 1. Sending a Batch Request
Make a POST request using the following pattern. You can pass `external_reference` as a unique ID per-request that will be returned in the response.
```bash title="Sending a batch request"
curl "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai/run/@cf/baai/bge-m3?queueRequest=true" \
--header "Authorization: Bearer $API_TOKEN" \
--header 'Content-Type: application/json' \
--json '{
"requests": [
{
"query": "This is a story about Cloudflare",
"contexts": [
{
"text": "This is a story about an orange cloud"
},
{
"text": "This is a story about a llama"
},
{
"text": "This is a story about a hugging emoji"
}
],
"external_reference": "reference-1"
}
]
}'
```
```json output {4}
{
"result": {
"status": "queued",
"request_id": "768f15b7-4fd6-4498-906e-ad94ffc7f8d2",
"model": "@cf/baai/bge-m3"
},
"success": true,
"errors": [],
"messages": []
}
```
## 2. Retrieving the Batch Response
After receiving a `request_id` from your initial POST, you can poll for or retrieve the results with another POST request:
```bash title="Retrieving a response"
curl "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai/run/@cf/baai/bge-m3?queueRequest=true" \
--header "Authorization: Bearer $API_TOKEN" \
--header 'Content-Type: application/json' \
--json '{
"request_id": ""
}'
```
```json output
{
"result": {
"responses": [
{
"id": 0,
"result": {
"response": [
{ "id": 0, "score": 0.73974609375 },
{ "id": 1, "score": 0.642578125 },
{ "id": 2, "score": 0.6220703125 }
]
},
"success": true,
"external_reference": "reference-1"
}
],
"usage": { "prompt_tokens": 12, "completion_tokens": 0, "total_tokens": 12 }
},
"success": true,
"errors": [],
"messages": []
}
```
# Workers Binding
import {
Render,
PackageManagers,
TypeScriptExample,
WranglerConfig,
CURL,
} from "~/components";
You can use Workers Bindings to interact with the Batch API.
## Send a Batch request
Send your initial batch inference request by composing a JSON payload containing an array of individual inference requests and the `queueRequest: true` property (which is what controls queueing behavior).
:::note[Note]
Ensure that the total payload is under 10 MB.
:::
```ts {26} title="src/index.ts"
export interface Env {
AI: Ai;
}
export default {
async fetch(request, env): Promise {
const embeddings = await env.AI.run(
"@cf/baai/bge-m3",
{
requests: [
{
query: "This is a story about Cloudflare",
contexts: [
{
text: "This is a story about an orange cloud",
},
{
text: "This is a story about a llama",
},
{
text: "This is a story about a hugging emoji",
},
],
},
],
},
{ queueRequest: true },
);
return Response.json(embeddings);
},
} satisfies ExportedHandler;
```
```json output {4}
{
"status": "queued",
"model": "@cf/baai/bge-m3",
"request_id": "000-000-000"
}
```
You will get a response with the following values:
- **`status`**: Indicates that your request is queued.
- **`request_id`**: A unique identifier for the batch request.
- **`model`**: The model used for the batch inference.
Of these, the `request_id` is important for when you need to [poll the batch status](#poll-batch-status).
### Poll batch status
Once your batch request is queued, use the `request_id` to poll for its status. During processing, the API returns a status `queued` or `running` indicating that the request is still in the queue or being processed.
```typescript title=src/index.ts
export interface Env {
AI: Ai;
}
export default {
async fetch(request, env): Promise {
const status = await env.AI.run("@cf/baai/bge-m3", {
request_id: "000-000-000",
});
return Response.json(status);
},
} satisfies ExportedHandler;
```
```json output
{
"responses": [
{
"id": 0,
"result": {
"response": [
{ "id": 0, "score": 0.73974609375 },
{ "id": 1, "score": 0.642578125 },
{ "id": 2, "score": 0.6220703125 }
]
},
"success": true,
"external_reference": "reference-1"
}
],
"usage": { "prompt_tokens": 12, "completion_tokens": 0, "total_tokens": 12 }
}
```
When the inference is complete, the API returns a final HTTP status code of `200` along with an array of responses. Each response object corresponds to an individual input prompt, identified by an `id` that maps to the index of the prompt in your original request.
# Fine-tunes
import { Feature } from "~/components";
Learn how to use Workers AI to get fine-tuned inference.
Upload a LoRA adapter and run fine-tuned inference with one of our base models.
---
## What is fine-tuning?
Fine-tuning is a general term for modifying an AI model by continuing to train it with additional data. The goal of fine-tuning is to increase the probability that a generation is similar to your dataset. Training a model from scratch is not practical for many use cases given how expensive and time consuming they can be to train. By fine-tuning an existing pre-trained model, you benefit from its capabilities while also accomplishing your desired task.
[Low-Rank Adaptation](https://arxiv.org/abs/2106.09685) (LoRA) is a specific fine-tuning method that can be applied to various model architectures, not just LLMs. It is common that the pre-trained model weights are directly modified or fused with additional fine-tune weights in traditional fine-tuning methods. LoRA, on the other hand, allows for the fine-tune weights and pre-trained model to remain separate, and for the pre-trained model to remain unchanged. The end result is that you can train models to be more accurate at specific tasks, such as generating code, having a specific personality, or generating images in a specific style.
# Using LoRA adapters
import { APIRequest, TabItem, Tabs } from "~/components";
Workers AI supports fine-tuned inference with adapters trained with [Low-Rank Adaptation](https://blog.cloudflare.com/fine-tuned-inference-with-loras). This feature is in open beta and free during this period.
## Limitations
- We only support LoRAs for a [variety of models](/workers-ai/models/?capabilities=LoRA) (must not be quantized)
- Adapter must be trained with rank `r <=8` as well as larger ranks if up to 32. You can check the rank of a pre-trained LoRA adapter through the adapter's `config.json` file
- LoRA adapter file must be < 300MB
- LoRA adapter files must be named `adapter_config.json` and `adapter_model.safetensors` exactly
- You can test up to 100 LoRA adapters per account
---
## Choosing compatible LoRA adapters
### Finding open-source LoRA adapters
We have started a [Hugging Face Collection](https://huggingface.co/collections/Cloudflare/workers-ai-compatible-loras-6608dd9f8d305a46e355746e) that lists a few LoRA adapters that are compatible with Workers AI. Generally, any LoRA adapter that fits our limitations above should work.
### Training your own LoRA adapters
To train your own LoRA adapter, follow the [tutorial](/workers-ai/guides/tutorials/fine-tune-models-with-autotrain/).
---
## Uploading LoRA adapters
In order to run inference with LoRAs on Workers AI, you'll need to create a new fine tune on your account and upload your adapter files. You should have a `adapter_model.safetensors` file with model weights and `adapter_config.json` with your config information. _Note that we only accept adapter files in these types._
Right now, you can't edit a fine tune's asset files after you upload it. We will support this soon, but for now you will need to create a new fine tune and upload files again if you would like to use a new LoRA.
Before you upload your LoRA adapter, you'll need to edit your `adapter_config.json` file to include `model_type` as one of `mistral`, `gemma` or `llama` like below.
```json null {10}
{
"alpha_pattern": {},
"auto_mapping": null,
...
"target_modules": [
"q_proj",
"v_proj"
],
"task_type": "CAUSAL_LM",
"model_type": "mistral",
}
```
### Wrangler
You can create a finetune and upload your LoRA adapter via wrangler with the following commands:
```bash title="wrangler CLI" {1,7}
npx wrangler ai finetune create
#🌀 Creating new finetune "test-lora" for model "@cf/mistral/mistral-7b-instruct-v0.2-lora"...
#🌀 Uploading file "/Users/abcd/Downloads/adapter_config.json" to "test-lora"...
#🌀 Uploading file "/Users/abcd/Downloads/adapter_model.safetensors" to "test-lora"...
#✅ Assets uploaded, finetune "test-lora" is ready to use.
npx wrangler ai finetune list
┌──────────────────────────────────────┬─────────────────┬─────────────┐
│ finetune_id │ name │ description │
├──────────────────────────────────────┼─────────────────┼─────────────┤
│ 00000000-0000-0000-0000-000000000000 │ test-lora │ │
└──────────────────────────────────────┴─────────────────┴─────────────┘
```
### REST API
Alternatively, you can use our REST API to create a finetune and upload your adapter files. You will need a Cloudflare API Token with `Workers AI: Edit` permissions to make calls to our REST API, which you can generate via the Cloudflare Dashboard.
#### Creating a fine-tune on your account
#### Uploading your adapter weights and config
You have to call the upload endpoint each time you want to upload a new file, so you usually run this once for `adapter_model.safetensors` and once for `adapter_config.json`. Make sure you include the `@` before your path to files.
You can either use the finetune `name` or `id` that you used when you created the fine tune.
```bash title="cURL"
## Input: finetune_id, adapter_model.safetensors, then adapter_config.json
## Output: success true/false
curl -X POST https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai/finetunes/{FINETUNE_ID}/finetune-assets/ \
-H 'Authorization: Bearer {API_TOKEN}' \
-H 'Content-Type: multipart/form-data' \
-F 'file_name=adapter_model.safetensors' \
-F 'file=@{PATH/TO/adapter_model.safetensors}'
```
#### List fine-tunes in your account
You can call this method to confirm what fine-tunes you have created in your account
```json output
{
"success": true,
"result": [
[
{
"id": "00000000-0000-0000-0000-000000000",
"model": "@cf/meta-llama/llama-2-7b-chat-hf-lora",
"name": "llama2-finetune",
"description": "test"
},
{
"id": "00000000-0000-0000-0000-000000000",
"model": "@cf/mistralai/mistral-7b-instruct-v0.2-lora",
"name": "mistral-finetune",
"description": "test"
}
]
]
}
```
---
## Running inference with LoRAs
To make inference requests and apply the LoRA adapter, you will need your model and finetune `name` or `id`. You should use the chat template that your LoRA was trained on, but you can try running it with `raw: true` and the messages template like below.
```javascript null {5-6}
const response = await env.AI.run(
"@cf/mistralai/mistral-7b-instruct-v0.2-lora", //the model supporting LoRAs
{
messages: [{ role: "user", content: "Hello world" }],
raw: true, //skip applying the default chat template
lora: "00000000-0000-0000-0000-000000000", //the finetune id OR name
},
);
```
```bash null {5-6}
curl https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai/run/@cf/mistral/mistral-7b-instruct-v0.2-lora \
-H 'Authorization: Bearer {API_TOKEN}' \
-d '{
"messages": [{"role": "user", "content": "Hello world"}],
"raw": "true",
"lora": "00000000-0000-0000-0000-000000000"
}'
```
# Public LoRA adapters
import { APIRequest } from "~/components";
Cloudflare offers a few public LoRA adapters that can immediately be used for fine-tuned inference. You can try them out immediately via our [playground](https://playground.ai.cloudflare.com).
Public LoRAs will have the name `cf-public-x`, and the prefix will be reserved for Cloudflare.
:::note
Have more LoRAs you would like to see? Let us know on [Discord](https://discord.cloudflare.com).
:::
| Name | Description | Compatible with |
| -------------------------------------------------------------------------- | ---------------------------------- | ----------------------------------------------------------------------------------- |
| [cf-public-magicoder](https://huggingface.co/predibase/magicoder) | Coding tasks in multiple languages | `@cf/mistral/mistral-7b-instruct-v0.1` `@hf/mistral/mistral-7b-instruct-v0.2` |
| [cf-public-jigsaw-classification](https://huggingface.co/predibase/jigsaw) | Toxic comment classification | `@cf/mistral/mistral-7b-instruct-v0.1` `@hf/mistral/mistral-7b-instruct-v0.2` |
| [cf-public-cnn-summarization](https://huggingface.co/predibase/cnn) | Article summarization | `@cf/mistral/mistral-7b-instruct-v0.1` `@hf/mistral/mistral-7b-instruct-v0.2` |
You can also list these public LoRAs with an API call:
## Running inference with public LoRAs
To run inference with public LoRAs, you just need to define the LoRA name in the request.
We recommend that you use the prompt template that the LoRA was trained on. You can find this in the HuggingFace repos linked above for each adapter.
### cURL
```bash null {10}
curl https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/run/@cf/mistral/mistral-7b-instruct-v0.1 \
--header 'Authorization: Bearer {cf_token}' \
--data '{
"messages": [
{
"role": "user",
"content": "Write a python program to check if a number is even or odd."
}
],
"lora": "cf-public-magicoder"
}'
```
### JavaScript
```js null {11}
const answer = await env.AI.run("@cf/mistral/mistral-7b-instruct-v0.1", {
stream: true,
raw: true,
messages: [
{
role: "user",
content:
"Summarize the following: Some newspapers, TV channels and well-known companies publish false news stories to fool people on 1 April. One of the earliest examples of this was in 1957 when a programme on the BBC, the UKs national TV channel, broadcast a report on how spaghetti grew on trees. The film showed a family in Switzerland collecting spaghetti from trees and many people were fooled into believing it, as in the 1950s British people didn't eat much pasta and many didn't know how it was made! Most British people wouldnt fall for the spaghetti trick today, but in 2008 the BBC managed to fool their audience again with their Miracles of Evolution trailer, which appeared to show some special penguins that had regained the ability to fly. Two major UK newspapers, The Daily Telegraph and the Daily Mirror, published the important story on their front pages.",
},
],
lora: "cf-public-cnn-summarization",
});
```
# Function calling
import { YouTube, TabItem, Tabs } from "~/components";
Function calling enables people to take Large Language Models (LLMs) and use the model response to execute functions or interact with external APIs. The developer usually defines a set of functions and the required input schema for each function, which we call `tools`. The model then intelligently understands when it needs to do a tool call, and it returns a JSON output which the user needs to feed to another function or API.
In essence, function calling allows you to perform actions with LLMs by executing code or making additional API calls.
## How can I use function calling?
Workers AI has [embedded function calling](/workers-ai/features/function-calling/embedded/) which allows you to execute function code alongside your inference calls. We have a package called [`@cloudflare/ai-utils`](https://www.npmjs.com/package/@cloudflare/ai-utils) to help facilitate this, which we have open-sourced on [Github](https://github.com/cloudflare/ai-utils).
For industry-standard function calling, take a look at the documentation on [Traditional Function Calling](/workers-ai/features/function-calling/traditional/).
To show you the value of embedded function calling, take a look at the example below that compares traditional function calling with embedded function calling. Embedded function calling allowed us to cut down the lines of code from 77 to 31.
```sh
# The ai-utils package enables embedded function calling
npm i @cloudflare/ai-utils
```
```js title="Embedded function calling example"
import {
createToolsFromOpenAPISpec,
runWithTools,
autoTrimTools,
} from "@cloudflare/ai-utils";
export default {
async fetch(request, env, ctx) {
const response = await runWithTools(
env.AI,
"@hf/nousresearch/hermes-2-pro-mistral-7b",
{
messages: [{ role: "user", content: "Who is Cloudflare on github?" }],
tools: [
// You can pass the OpenAPI spec link or contents directly
...(await createToolsFromOpenAPISpec(
"https://gist.githubusercontent.com/mchenco/fd8f20c8f06d50af40b94b0671273dc1/raw/f9d4b5cd5944cc32d6b34cad0406d96fd3acaca6/partial_api.github.com.json",
{
overrides: [
{
// for all requests on *.github.com, we'll need to add a User-Agent.
matcher: ({ url, method }) => {
return url.hostname === "api.github.com";
},
values: {
headers: {
"User-Agent":
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36",
},
},
},
],
},
)),
],
},
).then((response) => {
return response;
});
return new Response(JSON.stringify(response));
},
};
```
```js title="Traditional function calling example"
export default {
async fetch(request, env, ctx) {
const response = await env.AI.run(
"@hf/nousresearch/hermes-2-pro-mistral-7b",
{
messages: [{ role: "user", content: "Who is Cloudflare on GitHub?" }],
tools: [
{
name: "getGithubUser",
description:
"Provides publicly available information about someone with a GitHub account.",
parameters: {
type: "object",
properties: {
username: {
type: "string",
description: "The handle for the GitHub user account.",
},
},
required: ["username"],
},
},
],
},
);
const selected_tool = response.tool_calls[0];
let res;
if (selected_tool.name == "getGithubUser") {
try {
const username = selected_tool.arguments.username;
const url = `https://api.github.com/users/${username}`;
res = await fetch(url, {
headers: {
// Github API requires a User-Agent header
"User-Agent":
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36",
},
}).then((res) => res.json());
} catch (error) {
return error;
}
}
const finalResponse = await env.AI.run(
"@hf/nousresearch/hermes-2-pro-mistral-7b",
{
messages: [
{
role: "user",
content: "Who is Cloudflare on GitHub?",
},
{
role: "assistant",
content: JSON.stringify(selected_tool),
},
{
role: "tool",
content: JSON.stringify(res),
},
],
tools: [
{
name: "getGithubUser",
description:
"Provides publicly available information about someone with a GitHub account.",
parameters: {
type: "object",
properties: {
username: {
type: "string",
description: "The handle for the GitHub user account.",
},
},
required: ["username"],
},
},
],
},
);
return new Response(JSON.stringify(finalResponse));
},
};
```
## What models support function calling?
There are open-source models which have been fine-tuned to do function calling. When browsing our [model catalog](/workers-ai/models/), look for models with the function calling property beside it. For example, [@hf/nousresearch/hermes-2-pro-mistral-7b](/workers-ai/models/hermes-2-pro-mistral-7b/) is a fine-tuned variant of Mistral 7B that you can use for function calling.
# Traditional
This page shows how you can do traditional function calling, as defined by industry standards. Workers AI also offers [embedded function calling](/workers-ai/features/function-calling/embedded/), which is drastically easier than traditional function calling.
With traditional function calling, you define an array of tools with the name, description, and tool arguments. The example below shows how you would pass a tool called `getWeather` in an inference request to a model.
```js title="Traditional function calling example"
const response = await env.AI.run("@hf/nousresearch/hermes-2-pro-mistral-7b", {
messages: [
{
role: "user",
content: "what is the weather in london?",
},
],
tools: [
{
name: "getWeather",
description: "Return the weather for a latitude and longitude",
parameters: {
type: "object",
properties: {
latitude: {
type: "string",
description: "The latitude for the given location",
},
longitude: {
type: "string",
description: "The longitude for the given location",
},
},
required: ["latitude", "longitude"],
},
},
],
});
return new Response(JSON.stringify(response.tool_calls));
```
The LLM will then return a JSON object with the required arguments and the name of the tool that was called. You can then pass this JSON object to make an API call.
```json
[
{
"arguments": { "latitude": "51.5074", "longitude": "-0.1278" },
"name": "getWeather"
}
]
```
For a working example on how to do function calling, take a look at our [demo app](https://github.com/craigsdennis/lightbulb-moment-tool-calling/blob/main/src/index.ts).
# API Reference
Learn more about the API reference for [embedded function calling](/workers-ai/features/function-calling/embedded).
## runWithTools
This wrapper method enables you to do embedded function calling. You pass it the AI binding, model, inputs (`messages` array and `tools` array), and optional configurations.
- `AI Binding`Ai
- The AI binding, such as `env.AI`.
- `model`BaseAiTextGenerationModels
- The ID of the model that supports function calling. For example, `@hf/nousresearch/hermes-2-pro-mistral-7b`.
- `input`Object
- `messages`RoleScopedChatInput\[]
- `tools`AiTextGenerationToolInputWithFunction\[]
- `config`Object
- `streamFinalResponse`boolean optional
- `maxRecursiveToolRuns`number optional
- `strictValidation`boolean optional
- `verbose`boolean optional
- `trimFunction`boolean optional - For the `trimFunction`, you can pass it `autoTrimTools`, which is another helper method we've devised to automatically choose the correct tools (using an LLM) before sending it off for inference. This means that your final inference call will have fewer input tokens.
## createToolsFromOpenAPISpec
This method lets you automatically create tool schemas based on OpenAPI specs, so you don't have to manually write or hardcode the tool schemas. You can pass the OpenAPI spec for any API in JSON or YAML format.
`createToolsFromOpenAPISpec` has a config input that allows you to perform overrides if you need to provide headers like Authentication or User-Agent.
- `spec`string
- The OpenAPI specification in either JSON or YAML format, or a URL to a remote OpenAPI specification.
- `config`Config optional - Configuration options for the createToolsFromOpenAPISpec function
- `overrides`ConfigRule\[] optional
- `matchPatterns`RegExp\[] optional
- `options` Object optional \{
`verbose` boolean optional
\}
# Get Started
import { TypeScriptExample, PackageManagers } from "~/components";
This guide will instruct you through setting up and deploying your first Workers AI project with embedded function calling. You will use Workers, a Workers AI binding, the [`ai-utils package`](https://github.com/cloudflare/ai-utils), and a large language model (LLM) to deploy your first AI-powered application on the Cloudflare global network with embedded function calling.
## 1. Create a Worker project with Workers AI
Follow the [Workers AI Get Started Guide](/workers-ai/get-started/workers-wrangler/) until step 2.
## 2. Install additional npm package
Next, run the following command in your project repository to install the Worker AI utilities package.
## 3. Add Workers AI Embedded function calling
Update the `index.ts` file in your application directory with the following code:
```ts
import { runWithTools } from "@cloudflare/ai-utils";
type Env = {
AI: Ai;
};
export default {
async fetch(request, env, ctx) {
// Define function
const sum = (args: { a: number; b: number }): Promise => {
const { a, b } = args;
return Promise.resolve((a + b).toString());
};
// Run AI inference with function calling
const response = await runWithTools(
env.AI,
// Model with function calling support
"@hf/nousresearch/hermes-2-pro-mistral-7b",
{
// Messages
messages: [
{
role: "user",
content: "What the result of 123123123 + 10343030?",
},
],
// Definition of available tools the AI model can leverage
tools: [
{
name: "sum",
description: "Sum up two numbers and returns the result",
parameters: {
type: "object",
properties: {
a: { type: "number", description: "the first number" },
b: { type: "number", description: "the second number" },
},
required: ["a", "b"],
},
// reference to previously defined function
function: sum,
},
],
},
);
return new Response(JSON.stringify(response));
},
} satisfies ExportedHandler;
```
This example imports the utils with `import { runWithTools} from "@cloudflare/ai-utils"` and follows the API reference below.
Moreover, in this example we define and describe a list of tools that the LLM can leverage to respond to the user query. Here, the list contains of only one tool, the `sum` function.
Abstracted by the `runWithTools` function, the following steps occur:
```mermaid
sequenceDiagram
participant Worker as Worker
participant WorkersAI as Workers AI
Worker->>+WorkersAI: Send messages, function calling prompt, and available tools
WorkersAI->>+Worker: Select tools and arguments for function calling
Worker-->>-Worker: Execute function
Worker-->>+WorkersAI: Send messages, function calling prompt and function result
WorkersAI-->>-Worker: Send response incorporating function output
```
The `ai-utils package` is also open-sourced on [Github](https://github.com/cloudflare/ai-utils).
## 4. Local development & deployment
Follow steps 4 and 5 of the [Workers AI Get Started Guide](/workers-ai/get-started/workers-wrangler/) for local development and deployment.
:::note[Workers AI Embedded Function Calling charges]
Embedded function calling runs Workers AI inference requests. Standard charges for inference (e.g. tokens) usage will be charged.
Resources consumed (e.g. CPU time) during embedded functions' code execution will be charged just as any other Worker's code execution.
:::
## API reference
For more details, refer to [API reference](/workers-ai/features/function-calling/embedded/api-reference/).
# Embedded
import { DirectoryListing } from "~/components"
Cloudflare has a unique [embedded function calling](https://blog.cloudflare.com/embedded-function-calling) feature that allows you to execute function code alongside your tool call inference. Our npm package [`@cloudflare/ai-utils`](https://www.npmjs.com/package/@cloudflare/ai-utils) is the developer toolkit to get started.
Embedded function calling can be used to easily make complex agents that interact with websites and APIs, like using natural language to create meetings on Google Calendar, saving data to Notion, automatically routing requests to other APIs, saving data to an R2 bucket - or all of this at the same time. All you need is a prompt and an OpenAPI spec to get started.
:::caution[REST API support]
Embedded function calling depends on features native to the Workers platform. This means that embedded function calling is only supported via [Cloudflare Workers](/workers-ai/get-started/workers-wrangler/), not via the [REST API](/workers-ai/get-started/rest-api/).
:::
## Resources
# Troubleshooting
This section will describe tools for troubleshooting and address common errors.
## Logging
General [logging](/workers/observability/logs/) capabilities for Workers also apply to embedded function calling.
### Function invocations
The invocations of tools can be logged as in any Worker using `console.log()`:
```ts title="Logging tool invocations" {6}
export default {
async fetch(request, env, ctx) {
const sum = (args: { a: number; b: number }): Promise => {
const { a, b } = args;
// Logging from within embedded function invocations
console.log(`The sum function has been invoked with the arguments a: ${a} and b: ${b}`)
return Promise.resolve((a + b).toString());
};
...
}
}
```
### Logging within `runWithTools`
The `runWithTools` function has a `verbose` mode that emits helpful logs for debugging of function calls as well input and output statistics.
```ts title="Enabled verbose mode" {13}
const response = await runWithTools(
env.AI,
'@hf/nousresearch/hermes-2-pro-mistral-7b',
{
messages: [
...
],
tools: [
...
],
},
// Enable verbose mode
{ verbose: true }
);
```
## Performance
To respond to a LLM prompt with embedded function, potentially multiple AI inference requests and function invocations are needed, which can have an impact on user experience.
Consider the following to improve performance:
* Shorten prompts (to reduce time for input processing)
* Reduce number of tools provided
* Stream the final response to the end user (to minimize the time to interaction). See example below:
```ts title="Streamed response example" {15}
async fetch(request, env, ctx) {
const response = (await runWithTools(
env.AI,
'@hf/nousresearch/hermes-2-pro-mistral-7b',
{
messages: [
...
],
tools: [
...
],
},
{
// Enable response streaming
streamFinalResponse: true,
}
)) as ReadableStream;
// Set response headers for streaming
return new Response(response, {
headers: {
'content-type': 'text/event-stream',
},
});
}
```
## Common Errors
If you are getting a `BadInput` error, your inputs may exceed our current context window for our models. Try reducing input tokens to resolve this error.
# Use fetch() handler
A very common use case is to provide the LLM with the ability to perform API calls via function calling.
In this example the LLM will retrieve the weather forecast for the next 5 days.
To do so a `getWeather` function is defined that is passed to the LLM as tool.
The `getWeather`function extracts the user's location from the request and calls the external weather API via the Workers' [`Fetch API`](/workers/runtime-apis/fetch/) and returns the result.
```ts title="Embedded function calling example with fetch()"
import { runWithTools } from '@cloudflare/ai-utils';
type Env = {
AI: Ai;
};
export default {
async fetch(request, env, ctx) {
// Define function
const getWeather = async (args: { numDays: number }) => {
const { numDays } = args;
// Location is extracted from request based on
// https://developers.cloudflare.com/workers/runtime-apis/request/#incomingrequestcfproperties
const lat = request.cf?.latitude
const long = request.cf?.longitude
// Interpolate values for external API call
const response = await fetch(
`https://api.open-meteo.com/v1/forecast?latitude=${lat}&longitude=${long}&daily=temperature_2m_max,precipitation_sum&timezone=GMT&forecast_days=${numDays}`
);
return response.text();
};
// Run AI inference with function calling
const response = await runWithTools(
env.AI,
// Model with function calling support
'@hf/nousresearch/hermes-2-pro-mistral-7b',
{
// Messages
messages: [
{
role: 'user',
content: 'What the weather like the next 5 days? Respond as text',
},
],
// Definition of available tools the AI model can leverage
tools: [
{
name: 'getWeather',
description: 'Get the weather for the next [numDays] days',
parameters: {
type: 'object',
properties: {
numDays: { type: 'numDays', description: 'number of days for the weather forecast' },
},
required: ['numDays'],
},
// reference to previously defined function
function: getWeather,
},
],
}
);
return new Response(JSON.stringify(response));
},
} satisfies ExportedHandler;
```
# Examples
import { DirectoryListing } from "~/components";
# Use KV API
Interact with persistent storage to retrieve or store information enables for powerful use cases.
In this example we show how embedded function calling can interact with other resources on the Cloudflare Developer Platform with a few lines of code.
## Pre-Requisites
For this example to work, you need to provision a [KV](/kv/) namespace first. To do so, follow the [KV - Get started ](/kv/get-started/) guide.
Importantly, your Wrangler file must be updated to include the `KV` binding definition to your respective namespace.
## Worker code
```ts title="Embedded function calling example with KV API"
import { runWithTools } from "@cloudflare/ai-utils";
type Env = {
AI: Ai;
KV: KVNamespace;
};
export default {
async fetch(request, env, ctx) {
// Define function
const updateKvValue = async ({
key,
value,
}: {
key: string;
value: string;
}) => {
const response = await env.KV.put(key, value);
return `Successfully updated key-value pair in database: ${response}`;
};
// Run AI inference with function calling
const response = await runWithTools(
env.AI,
"@hf/nousresearch/hermes-2-pro-mistral-7b",
{
messages: [
{ role: "system", content: "Put user given values in KV" },
{ role: "user", content: "Set the value of banana to yellow." },
],
tools: [
{
name: "KV update",
description: "Update a key-value pair in the database",
parameters: {
type: "object",
properties: {
key: {
type: "string",
description: "The key to update",
},
value: {
type: "string",
description: "The value to update",
},
},
required: ["key", "value"],
},
function: updateKvValue,
},
],
},
);
return new Response(JSON.stringify(response));
},
} satisfies ExportedHandler;
```
## Verify results
To verify the results, run the following command
```sh
npx wrangler kv key get banana --binding KV --local
```
# Tools based on OpenAPI Spec
Oftentimes APIs are defined and documented via [OpenAPI specification](https://swagger.io/specification/). The Cloudflare `ai-utils` package's `createToolsFromOpenAPISpec` function creates tools from the OpenAPI spec, which the LLM can then leverage to fulfill the prompt.
In this example the LLM will describe the a Github user, based Github's API and its OpenAPI spec.
```ts title="Embedded function calling example from OpenAPI Spec"
import { createToolsFromOpenAPISpec, runWithTools } from '@cloudflare/ai-utils';
type Env = {
AI: Ai;
};
const APP_NAME = 'cf-fn-calling-example-app';
export default {
async fetch(request, env, ctx) {
const toolsFromOpenAPISpec = [
// You can pass the OpenAPI spec link or contents directly
...(await createToolsFromOpenAPISpec(
'https://gist.githubusercontent.com/mchenco/fd8f20c8f06d50af40b94b0671273dc1/raw/f9d4b5cd5944cc32d6b34cad0406d96fd3acaca6/partial_api.github.com.json',
{
overrides: [
{
matcher: ({ url }) => {
return url.hostname === 'api.github.com';
},
// for all requests on *.github.com, we'll need to add a User-Agent.
values: {
headers: {
'User-Agent': APP_NAME,
},
},
},
],
}
)),
];
const response = await runWithTools(
env.AI,
'@hf/nousresearch/hermes-2-pro-mistral-7b',
{
messages: [
{
role: 'user',
content: 'Who is cloudflare on Github and how many repos does the organization have?',
},
],
tools: toolsFromOpenAPISpec,
}
);
return new Response(JSON.stringify(response));
},
} satisfies ExportedHandler;
```
# Conversion Options
By default, the `toMarkdown` service extracts text content from your files. To further extend the capabilities of the conversion process, you can pass options to the service to control how specific file types are converted.
Options are organized by file type and are all optional.
## Available options
### Output
```typescript
{
output?: {
format?: 'markdown' | 'text';
}
}
```
- `format`: controls the format of the converted content. Defaults to `markdown`. Set to `text` to receive plain text with Markdown syntax removed.
When `format` is `text`, the `format` field of the [`ConversionResult`](/workers-ai/features/markdown-conversion/usage/binding/#conversionresult-definition) is also set to `text`.
### Images
```typescript
{
image?: {
descriptionLanguage?: 'en' | 'it' | 'de' | 'es' | 'fr' | 'pt';
}
}
```
- `descriptionLanguage`: controls the language of the AI-generated image descriptions.
:::caution
This option works on a _best-effort_ basis: it is not guaranteed that the resulting text will be in the desired language.
:::
### HTML
```typescript
{
html?: {
hostname?: string;
cssSelector?: string;
}
}
```
- `hostname`: absolute base URL used to resolve relative links in the HTML. Include the scheme and, when applicable, the page path — for example, `https://example.com/docs/page`. A bare hostname such as `example.com` is not sufficient.
- `cssSelector`: string containing a CSS selector pattern to pick specific elements from your HTML. Refer to [how HTML is processed](/workers-ai/features/markdown-conversion/how-it-works/#html) for more details.
### PDF
```typescript
{
pdf?: {
metadata?: boolean;
}
}
```
- `metadata`: Previously, all converted PDF files always included metadata information when converted. This option allows you to opt-out of this behavior.
## Examples
### Binding
To configure custom options, pass a `conversionOptions` object inside the second argument of the binding call, like this:
```typescript
await env.AI.toMarkdown(..., {
conversionOptions: {
html: { ... },
pdf: { ... },
...
}
})
```
### REST API
Since the REST API uses file uploads, the request's `Content-Type` will be `multipart/form-data`. As such, include a new form field with your stringified object as a value:
```bash
curl https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai/tomarkdown \
-X POST \
-H 'Authorization: Bearer {API_TOKEN}' \
...
-F 'conversionOptions={ "html": { ... }, ... }'
```
# How it works
When parsing files before converting them to Markdown, there are some cleanup tasks we do depending on the type of file you are trying to convert.
### HTML
When we detect an HTML file, a series of things happen to the HTML content before it is converted:
- Some elements are ignored, including `script` and `style` tags.
- Meta tags are extracted. These include `title`, `description`, `og:title`, `og:description` and `og:image`.
- [JSON-LD](https://json-ld.org/) content is extracted, if it exists. This will be appended at the end of the converted markdown.
- The base URL to use for resolving relative links is extracted from the `` element1, if it exists, according to the spec (that is, only the first instance of the base URL is counted).
- If the `cssSelector` option is:
- present, then only those elements that match the selector are kept for further processing;
- missing, then elements such as ``, `