VectorCache: A Python Library for Efficient LLM Query Caching

Overview of VectorCache

VectorCache is a Python library designed to enhance the performance of Large Language Model (LLM) queries through semantic caching. This approach allows for faster response times and reduced costs by caching responses based on semantic similarity, rather than just exact matches.

The library offers a nuanced caching mechanism, similar to Redis, but with the added capability to recognize semantically similar queries, making it particularly efficient for domains with frequent queries within specific topics or fields.

Benefits of Using VectorCache

One of the primary benefits of VectorCache is its ability to significantly reduce latency in LLM query responses, providing quicker feedback to users. Additionally, by minimizing the number of direct LLM requests, it helps in saving on usage costs.

VectorCache is versatile and can be integrated with any LLM provider, making it a flexible solution for various applications.

Key Features

VectorCache includes several key features that enhance its functionality, such as support for multiple vector stores, adaptive thresholding for cache queries, and verbose logging for debugging purposes. These features collectively contribute to its efficiency and ease of use.

The library also supports dynamic adjustment of the similarity threshold based on the target hit rate, which is particularly useful for achieving optimal cache performance.

Vector Stores Support

VectorCache supports a variety of vector stores, including Redis, Qdrant, Deeplake, ChromaDB, pgvector, Pinecone, and Milvus. This wide support ensures that users can choose the most suitable storage solution for their specific needs.

Getting Started

To get started with VectorCache, ensure you have Python 3.9 or higher installed. Install the library using pip (`pip install vector-cache`). Additional vector stores and storage databases can be installed as optional dependencies.

Initialization of VectorCache involves specifying parameters such as the embedding model, cache storage interface, vector store interface, and various optional settings for adaptive thresholding and verbose logging.

Example Usage

An example of initializing VectorCache with specific parameters is provided in the documentation. This example demonstrates how to set up the cache with an embedding model, cache storage, vector store, and adaptive thresholding settings.

Refer to the `examples` folder in the repository for more detailed sample usage scenarios.

Compatibility with LLM Providers

VectorCache is designed to be compatible with any LLM provider, making it a versatile tool for various applications. This compatibility ensures that users can leverage the benefits of semantic caching regardless of their chosen LLM provider.

The library's modular design includes components for embedding models, cache storage, vector stores, cache manager, and similarity evaluator, which together facilitate seamless integration with different LLM providers.

How to Contribute

VectorCache welcomes contributions from the community. Interested contributors can refer to the contribution guidelines provided in the repository to understand how to get involved.

The project is open to improvements and expansions, including support for additional vector stores. Contributors can help in enhancing the library's functionality and compatibility with more tools and services.