TF-IDF
TF-IDF means term-frequency times inverse document-frequency.
This notebook goes over how to use a retriever that under the hood uses TF-IDF using scikit-learn
package.
For more information on the details of TF-IDF see this blog post.
%pip install --upgrade --quiet scikit-learn
from langchain_community.retrievers import TFIDFRetriever
API Reference:
Create New Retriever with Texts
retriever = TFIDFRetriever.from_texts(["foo", "bar", "world", "hello", "foo bar"])
Create a New Retriever with Documents
You can now create a new retriever with the documents you created.
from langchain_core.documents import Document
retriever = TFIDFRetriever.from_documents(
[
Document(page_content="foo"),
Document(page_content="bar"),
Document(page_content="world"),
Document(page_content="hello"),
Document(page_content="foo bar"),
]
)
API Reference:
Use Retriever
We can now use the retriever!
result = retriever.invoke("foo")
result
[Document(page_content='foo', metadata={}),
Document(page_content='foo bar', metadata={}),
Document(page_content='hello', metadata={}),
Document(page_content='world', metadata={})]
Save and load
You can easily save and load this retriever, making it handy for local development!
retriever.save_local("testing.pkl")
retriever_copy = TFIDFRetriever.load_local("testing.pkl")
retriever_copy.invoke("foo")
[Document(page_content='foo', metadata={}),
Document(page_content='foo bar', metadata={}),
Document(page_content='hello', metadata={}),
Document(page_content='world', metadata={})]