Skip to content

Commit 2bcc4c2

Browse files
authored
feat: Add support for multimodal embeddings in vectorizers (#452)
This PR generalizes the `BaseVectorizer` to be agnostic to any modality (since it previously exclusively supported text inputs). Building from the new base, this PR then extends the implementation for some vectorizers to support multimodal embeddings (renaming them away from being specifically for text). ## `BaseVectorizer` The move away from having the `BaseVectorizer` explicitly expect text inputs means a change in the signature of the `embed` methods away from `vectorizer.embed(text="lorem ipsum...")` to `vectorizer.embed(content="lorem ipsum...")`. This is a breaking change for existing usages of the vectorizers that use the keyword argument, and the usages will need to be updated to align with the new schema. Caching for multimodal embeddings is supported for all vectorizers introduced in this PR. ## Multimodal Implementations The following vectorizers have been renamed to no longer be explicitly text vectorizers, and moved to no longer be defined in the `vectorize.text` module. Imports and usages for these vectorizers will need to be updated to avoid errors. The `CustomTextVectorizer` has also been renamed and moved to be `redisvl.utils.vectorize.custom.CustomVectorizer`. ### VoyageAI Old: `redisvl.utils.vectorize.text.voyageai.VoyageAITextVectorizer` New: `redisvl.utils.vectorize.voyageai.VoyageAIVectorizer` ```python from redisvl.utils.vectorize import VoyageAIVectorizer # --- Basic usage vectorizer = VoyageAIVectorizer( model="voyage-3-large", api_config={"api_key": "your-voyageai-api-key"} # OR set VOYAGE_API_KEY in your env ) query_embedding = vectorizer.embed( content="your input query text here", input_type="query" ) doc_embeddings = vectorizer.embed_many( contents=["your document text", "more document text"], input_type="document" ) # --- Multimodal usage - requires Pillow and voyageai>=0.3.6 (for video) from PIL import Image from voyageai.video_utils import Video vectorizer = VoyageAIVectorizer( model="voyage-multimodal-3.5", api_config={"api_key": "your-voyageai-api-key"} # OR set VOYAGE_API_KEY in your env ) # text text_embedding = vectorizer.embed( content="your input query text here", input_type="query" ) # image image_embedding = vectorizer.embed_image( "path/to/your/image.jpg", input_type="query" ) image_embedding = vectorizer.embed( Image.open("path/to/your/image.jpg"), input_type="query" # video video_embedding = vectorizer.embed_video( "path/to/your/video.mp4", input_type="document" ) video_embedding = vectorizer.embed( Video.from_path("path/to/your/video.mp4", model=vectorizer.model), input_type="document" ) ``` ### Vertex AI Old: `redisvl.utils.vectorize.text.vertexai.VertexAITextVectorizer` New: `redisvl.utils.vectorize.vertexai.VertexAIVectorizer` ```python from redisvl.utils.vectorize import VertexAIVectorizer # Basic usage vectorizer = VertexAIVectorizer( model="textembedding-gecko", api_config={ "project_id": "your_gcp_project_id", # OR set GCP_PROJECT_ID "location": "your_gcp_location", # OR set GCP_LOCATION }) embedding = vectorizer.embed("Hello, world!") # Multimodal usage from vertexai.vision_models import Image, Video vectorizer = VertexAIVectorizer( model="multimodalembedding@001", api_config={ "project_id": "your_gcp_project_id", # OR set GCP_PROJECT_ID "location": "your_gcp_location", # OR set GCP_LOCATION } ) text_embedding = vectorizer.embed("Hello, world!") image_embedding = vectorizer.embed(Image.load_from_file("path/to/your/image.jpg")) image_embedding = vectorizer.embed_image("path/to/your/image.jpg") video_embedding = vectorizer.embed(Video.load_from_file("path/to/your/video.mp4")) video_embedding = vectorizer.embed_video("path/to/your/video.mp4") ``` ### Amazon Bedrock Old: `redisvl.utils.vectorize.text.bedrock.BedrockTextVectorizer` New: `redisvl.utils.vectorize.bedrock.BedrockVectorizer` ```python from redisvl.utils.vectorize import BedrockVectorizer vectorizer = BedrockVectorizer( model="amazon.titan-embed-text-v2:0", api_config={ "aws_access_key_id": "your_access_key", "aws_secret_access_key": "your_secret_key", "aws_region": "us-east-1" } ) embedding = vectorizer.embed("Hello, world!") # Multimodal usage from pathlib import Path from PIL import Image vectorizer = BedrockVectorizer( model="amazon.titan-embed-image-v1:0", api_config={ "aws_access_key_id": "your_access_key", "aws_secret_access_key": "your_secret_key", "aws_region": "us-east-1" } ) image_embedding = vectorizer.embed(Path("path/to/your/image.jpg")) image_embedding = vectorizer.embed(Image.open("path/to/other/image.png")) image_embedding = vectorizer.embed_image("path/to/your/image.jpg") # Embedding a list of mixed modalities embeddings = vectorizer.embed_many( ["Hello", "world!", Path("path/to/your/image.jpg"), Image.open("path/to/other/image.png")], batch_size=2 ) ``` ### Hugging Face While the sentence-transformers package does not explicitly allow for multimodal usage (the package is designed for text-based use-cases), some officially supported multimodal models can be used without issue via the `SentenceTransformer` class. This PR removes strict enforcement of text inputs for the `HFTextVectorizer` to enable these use-cases. ```python from PIL import Image from redisvl.utils.vectorize import HFTextVectorizer vectorizer = HFTextVectorizer(model="sentence-transformers/clip-ViT-L-14") embeddings1 = vectorizer.embed("Hello, world!") embeddings2 = vectorizer.embed(Image.open("path/to/your/image.jpg")) ``` ## Open Topics Since this PR introduces a few breaking changes, do we want to maintain backwards compatibility (with deprecation warnings) for syntax that is changing? This includes: - `vectorizer.embed(text=...)` -> `vectorizer.embed(content=...)` - `VoyageAITextVectorizer` -> `VoyageAIVectorizer` - `VertexAITextVectorizer` -> `VertexAIVectorizer` - `BedrockTextVectorizer` -> `BedrockVectorizer` - `CustomTextVectorizer` -> `CustomVectorizer`
1 parent 5e1e95c commit 2bcc4c2

30 files changed

+3263
-2692
lines changed

docs/api/vectorizer.rst

Lines changed: 61 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,19 @@
22
Vectorizers
33
***********
44

5+
.. note::
6+
**Backwards Compatibility:** Several vectorizers have deprecated aliases
7+
available in the ``redisvl.utils.vectorize.text`` module for backwards
8+
compatibility:
9+
10+
- ``VoyageAITextVectorizer`` → Use ``VoyageAIVectorizer`` instead
11+
- ``VertexAITextVectorizer`` → Use ``VertexAIVectorizer`` instead
12+
- ``BedrockTextVectorizer`` → Use ``BedrockVectorizer`` instead
13+
- ``CustomTextVectorizer`` → Use ``CustomVectorizer`` instead
14+
15+
These aliases are deprecated as of version 0.13.0 and will be removed
16+
in a future major release.
17+
518
HFTextVectorizer
619
================
720

@@ -38,14 +51,19 @@ AzureOpenAITextVectorizer
3851
:members:
3952

4053

41-
VertexAITextVectorizer
54+
VertexAIVectorizer
4255
======================
4356

44-
.. _vertexaitextvectorizer_api:
57+
.. _vertexaivectorizer_api:
58+
59+
.. currentmodule:: redisvl.utils.vectorize.vertexai
4560

46-
.. currentmodule:: redisvl.utils.vectorize.text.vertexai
61+
.. note::
62+
For backwards compatibility, an alias ``VertexAITextVectorizer`` is available
63+
in the ``redisvl.utils.vectorize.text`` module. This alias is deprecated
64+
as of version 0.13.0 and will be removed in a future major release.
4765

48-
.. autoclass:: VertexAITextVectorizer
66+
.. autoclass:: VertexAIVectorizer
4967
:show-inheritance:
5068
:members:
5169

@@ -62,37 +80,64 @@ CohereTextVectorizer
6280
:members:
6381

6482

65-
BedrockTextVectorizer
83+
BedrockVectorizer
6684
=====================
6785

68-
.. _bedrocktextvectorizer_api:
86+
.. _bedrockvectorizer_api:
6987

70-
.. currentmodule:: redisvl.utils.vectorize.text.bedrock
88+
.. currentmodule:: redisvl.utils.vectorize.bedrock
7189

72-
.. autoclass:: BedrockTextVectorizer
90+
.. note::
91+
For backwards compatibility, an alias ``BedrockTextVectorizer`` is available
92+
in the ``redisvl.utils.vectorize.text`` module. This alias is deprecated
93+
as of version 0.13.0 and will be removed in a future major release.
94+
95+
.. autoclass:: BedrockVectorizer
7396
:show-inheritance:
7497
:members:
7598

7699

77-
CustomTextVectorizer
100+
CustomVectorizer
78101
====================
79102

80-
.. _customtextvectorizer_api:
103+
.. _customvectorizer_api:
104+
105+
.. currentmodule:: redisvl.utils.vectorize.custom
81106

82-
.. currentmodule:: redisvl.utils.vectorize.text.custom
107+
.. note::
108+
For backwards compatibility, an alias ``CustomTextVectorizer`` is available
109+
in the ``redisvl.utils.vectorize.text`` module. This alias is deprecated
110+
as of version 0.13.0 and will be removed in a future major release.
83111

84-
.. autoclass:: CustomTextVectorizer
112+
.. autoclass:: CustomVectorizer
85113
:show-inheritance:
86114
:members:
87115

88116

89-
VoyageAITextVectorizer
117+
VoyageAIVectorizer
90118
======================
91119

92-
.. _voyageaitextvectorizer_api:
120+
.. _voyageaivectorizer_api:
121+
122+
.. currentmodule:: redisvl.utils.vectorize.voyageai
123+
124+
.. note::
125+
For backwards compatibility, an alias ``VoyageAITextVectorizer`` is available
126+
in the ``redisvl.utils.vectorize.text`` module. This alias is deprecated
127+
as of version 0.13.0 and will be removed in a future major release.
128+
129+
.. autoclass:: VoyageAIVectorizer
130+
:show-inheritance:
131+
:members:
132+
133+
134+
MistralAITextVectorizer
135+
========================
136+
137+
.. _mistralaitextvectorizer_api:
93138

94-
.. currentmodule:: redisvl.utils.vectorize.text.voyageai
139+
.. currentmodule:: redisvl.utils.vectorize.text.mistral
95140

96-
.. autoclass:: VoyageAITextVectorizer
141+
.. autoclass:: MistralAITextVectorizer
97142
:show-inheritance:
98143
:members:

docs/user_guide/04_vectorizers.ipynb

Lines changed: 780 additions & 780 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)