Databricks Vector Search
Databricks Vector Search is a serverless similarity search engine that allows you to store a vector representation of your data, including metadata, in a vector database. With Vector Search, you can create auto-updating vector search indexes from Delta tables managed by Unity Catalog and query them with a simple API to return the most similar vectors.
This notebook shows how to use LangChain with Databricks Vector Search.
Install databricks-vectorsearch
and related Python packages used in this notebook.
%pip install --upgrade --quiet langchain-core databricks-vectorsearch langchain-openai tiktoken
Use OpenAIEmbeddings
for the embeddings.
import getpass
import os
os.environ["OPENAI_API_KEY"] = getpass.getpass("OpenAI API Key:")
Split documents and get embeddings.
from langchain_community.document_loaders import TextLoader
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import CharacterTextSplitter
loader = TextLoader("../../how_to/state_of_the_union.txt")
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
docs = text_splitter.split_documents(documents)
embeddings = OpenAIEmbeddings()
emb_dim = len(embeddings.embed_query("hello"))
API Reference:
Setup Databricks Vector Search client
from databricks.vector_search.client import VectorSearchClient
vsc = VectorSearchClient()
Create a Vector Search Endpoint
This endpoint is used to create and access vector search indexes.
vsc.create_endpoint(name="vector_search_demo_endpoint", endpoint_type="STANDARD")
Create Direct Vector Access Index
Direct Vector Access Index supports direct read and write of embedding vectors and metadata through a REST API or an SDK. For this index, you manage embedding vectors and index updates yourself.
vector_search_endpoint_name = "vector_search_demo_endpoint"
index_name = "ml.llm.demo_index"
index = vsc.create_direct_access_index(
endpoint_name=vector_search_endpoint_name,
index_name=index_name,
primary_key="id",
embedding_dimension=emb_dim,
embedding_vector_column="text_vector",
schema={
"id": "string",
"text": "string",
"text_vector": "array<float>",
"source": "string",
},
)
index.describe()
from langchain_community.vectorstores import DatabricksVectorSearch
dvs = DatabricksVectorSearch(
index, text_column="text", embedding=embeddings, columns=["source"]
)
API Reference:
Add docs to the index
dvs.add_documents(docs)
Similarity search
query = "What did the president say about Ketanji Brown Jackson"
dvs.similarity_search(query)
print(docs[0].page_content)
Work with Delta Sync Index
You can also use DatabricksVectorSearch
to search in a Delta Sync Index. Delta Sync Index automatically syncs from a Delta table. You don't need to call add_text
/add_documents
manually. See Databricks documentation page for more details.
dvs_delta_sync = DatabricksVectorSearch("catalog_name.schema_name.delta_sync_index")
dvs_delta_sync.similarity_search(query)