AI & Vectors

Video Search with Mixpeek Multimodal Embeddings

Implement video search with the Mixpeek Multimodal Embed API and Supabase Vector.

The Mixpeek Embed API allows you to generate embeddings for various types of content, including videos and text. You can use these embeddings for:

  • Text-to-Video / Video-To-Text / Video-to-Video / Text-to-Text Search
  • Fine-tuning on your own video and text data

This guide demonstrates how to implement video search using Mixpeek Embed for video processing and embedding, and Supabase Vector for storing and querying embeddings.

You can find the full application code as a Python Poetry project on GitHub.

Create a new Python project with Poetry

Poetry provides packaging and dependency management for Python. If you haven't already, install poetry via pip:


_10
pip install poetry

Then initialize a new project:


_10
poetry new video-search

Setup Supabase project

If you haven't already, install the Supabase CLI, then initialize Supabase in the root of your newly created poetry project:


_10
supabase init

Next, start your local Supabase stack:


_10
supabase start

This will start up the Supabase stack locally and print out a bunch of environment details, including your local DB URL. Make a note of that for later use.

Install the dependencies

Add the following dependencies to your project:

  • supabase: Supabase Python Client
  • mixpeek: Mixpeek Python Client for embedding generation

_10
poetry add supabase mixpeek

Import the necessary dependencies

At the top of your main Python script, import the dependencies and store your environment variables:


_10
from supabase import create_client, Client
_10
from mixpeek import Mixpeek
_10
import os
_10
_10
SUPABASE_URL = os.getenv("SUPABASE_URL")
_10
SUPABASE_KEY = os.getenv("SUPABASE_API_KEY")
_10
MIXPEEK_API_KEY = os.getenv("MIXPEEK_API_KEY")

Create embeddings for your videos

Next, create a seed method, which will create a new Supabase table, generate embeddings for your video chunks, and insert the embeddings into your database:


_46
def seed():
_46
# Initialize Supabase and Mixpeek clients
_46
supabase: Client = create_client(SUPABASE_URL, SUPABASE_KEY)
_46
mixpeek = Mixpeek(MIXPEEK_API_KEY)
_46
_46
# Create a table for storing video chunk embeddings
_46
supabase.table("video_chunks").create({
_46
"id": "text",
_46
"start_time": "float8",
_46
"end_time": "float8",
_46
"embedding": "vector(768)",
_46
"metadata": "jsonb"
_46
})
_46
_46
# Process and embed video
_46
video_url = "https://example.com/your_video.mp4"
_46
processed_chunks = mixpeek.tools.video.process(
_46
video_source=video_url,
_46
chunk_interval=1, # 1 second intervals
_46
resolution=[720, 1280]
_46
)
_46
_46
for chunk in processed_chunks:
_46
print(f"Processing video chunk: {chunk['start_time']}")
_46
_46
# Generate embedding using Mixpeek
_46
embed_response = mixpeek.embed.video(
_46
model_id="vuse-generic-v1",
_46
input=chunk['base64_chunk'],
_46
input_type="base64"
_46
)
_46
_46
# Insert into Supabase
_46
supabase.table("video_chunks").insert({
_46
"id": f"chunk_{chunk['start_time']}",
_46
"start_time": chunk["start_time"],
_46
"end_time": chunk["end_time"],
_46
"embedding": embed_response['embedding'],
_46
"metadata": {"video_url": video_url}
_46
}).execute()
_46
_46
print("Video processed and embeddings inserted")
_46
_46
# Create index for fast search performance
_46
supabase.query("CREATE INDEX ON video_chunks USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100)").execute()
_46
print("Created index")

Add this method as a script in your pyproject.toml file:


_10
[tool.poetry.scripts]
_10
seed = "video_search.main:seed"
_10
search = "video_search.main:search"

After activating the virtual environment with poetry shell, you can now run your seed script via poetry run seed. You can inspect the generated embeddings in your local database by visiting the local Supabase dashboard at localhost:54323.

Perform a video search from a text query

With Supabase Vector, you can query your embeddings. You can use either a video clip as search input or alternatively, you can generate an embedding from a string input and use that as the query input:


_32
def search():
_32
# Initialize Supabase and Mixpeek clients
_32
supabase: Client = create_client(SUPABASE_URL, SUPABASE_KEY)
_32
mixpeek = Mixpeek(MIXPEEK_API_KEY)
_32
_32
# Generate embedding for text query
_32
query_string = "a car chase scene"
_32
text_emb = mixpeek.embed.video(
_32
model_id="vuse-generic-v1",
_32
input=query_string,
_32
input_type="text"
_32
)
_32
_32
# Query the collection
_32
results = supabase.rpc(
_32
'match_video_chunks',
_32
{
_32
'query_embedding': text_emb['embedding'],
_32
'match_threshold': 0.8,
_32
'match_count': 5
_32
}
_32
).execute()
_32
_32
# Display the results
_32
if results.data:
_32
for result in results.data:
_32
print(f"Matched chunk from {result['start_time']} to {result['end_time']} seconds")
_32
print(f"Video URL: {result['metadata']['video_url']}")
_32
print(f"Similarity: {result['similarity']}")
_32
print("---")
_32
else:
_32
print("No matching video chunks found")

This query will return the top 5 most similar video chunks from your database.

You can now test it out by running poetry run search, and you will be presented with the most relevant video chunks to the query "a car chase scene".

Conclusion

With just a couple of Python scripts, you are able to implement video search as well as reverse video search using Mixpeek Embed and Supabase Vector. This approach allows for powerful semantic search capabilities that can be integrated into various applications, enabling you to search through video content using both text and video queries.