Edge Functions

Running AI Models

Run AI models in Edge Functions using the built-in Supabase AI API.

Edge Functions have a built-in API for running AI models. You can use this API to generate embeddings, build conversational workflows, and do other AI related tasks in your Edge Functions.

This allows you to:

  • Generate text embeddings without external dependencies
  • Run Large Language Models via Ollama or Llamafile
  • Build conversational AI workflows

Setup

There are no external dependencies or packages to install to enable the API.

Create a new inference session:

1
const model = new Supabase.ai.Session('model-name')

Running a model inference

Once the session is instantiated, you can call it with inputs to perform inferences:

1
// For embeddings (gte-small model)
2
const embeddings = await model.run('Hello world', {
3
  mean_pool: true,
4
  normalize: true,
5
})
6
7
// For text generation (non-streaming)
8
const response = await model.run('Write a haiku about coding', {
9
  stream: false,
10
  timeout: 30,
11
})
12
13
// For streaming responses
14
const stream = await model.run('Tell me a story', {
15
  stream: true,
16
  mode: 'ollama',
17
})

Generate text embeddings

Generate text embeddings using the built-in gte-small model:

1
const model = new Supabase.ai.Session('gte-small')
2
3
Deno.serve(async (req: Request) => {
4
  const params = new URL(req.url).searchParams
5
  const input = params.get('input')
6
  const output = await model.run(input, { mean_pool: true, normalize: true })
7
  return new Response(JSON.stringify(output), {
8
    headers: {
9
      'Content-Type': 'application/json',
10
      Connection: 'keep-alive',
11
    },
12
  })
13
})

Using Large Language Models (LLM)

Inference via larger models is supported via Ollama and Mozilla Llamafile. In the first iteration, you can use it with a self-managed Ollama or Llamafile server.

Running locally

1

Install Ollama

Install Ollama and pull the Mistral model

1
ollama pull mistral
2

Run the Ollama server

1
ollama serve
3

Set the function secret

Set a function secret called AI_INFERENCE_API_HOST to point to the Ollama server

1
echo "AI_INFERENCE_API_HOST=http://host.docker.internal:11434" >> supabase/functions/.env
4

Create a new function

1
supabase functions new ollama-test
1
import 'jsr:@supabase/functions-js/edge-runtime.d.ts'
2
const session = new Supabase.ai.Session('mistral')
3
4
Deno.serve(async (req: Request) => {
5
  const params = new URL(req.url).searchParams
6
  const prompt = params.get('prompt') ?? ''
7
8
  // Get the output as a stream
9
  const output = await session.run(prompt, { stream: true })
10
11
  const headers = new Headers({
12
    'Content-Type': 'text/event-stream',
13
    Connection: 'keep-alive',
14
  })
15
16
  // Create a stream
17
  const stream = new ReadableStream({
18
    async start(controller) {
19
      const encoder = new TextEncoder()
20
21
      try {
22
        for await (const chunk of output) {
23
          controller.enqueue(encoder.encode(chunk.response ?? ''))
24
        }
25
      } catch (err) {
26
        console.error('Stream error:', err)
27
      } finally {
28
        controller.close()
29
      }
30
    },
31
  })
32
33
  // Return the stream to the user
34
  return new Response(stream, {
35
    headers,
36
  })
37
})
5

Serve the function

1
supabase functions serve --env-file supabase/functions/.env
6

Execute the function

1
curl --get "http://localhost:54321/functions/v1/ollama-test" \
2
--data-urlencode "prompt=write a short rap song about Supabase, the Postgres Developer platform, as sung by Nicki Minaj" \
3
-H "Authorization: $ANON_KEY"

Deploying to production

Once the function is working locally, it's time to deploy to production.

1

Deploy an Ollama or Llamafile server

Deploy an Ollama or Llamafile server and set a function secret called AI_INFERENCE_API_HOST to point to the deployed server:

1
supabase secrets set AI_INFERENCE_API_HOST=https://path-to-your-llm-server/
2

Deploy the function

1
supabase functions deploy
3

Execute the function

1
curl --get "https://project-ref.supabase.co/functions/v1/ollama-test" \
2
--data-urlencode "prompt=write a short rap song about Supabase, the Postgres Developer platform, as sung by Nicki Minaj" \
3
-H "Authorization: $ANON_KEY"