Current state: Under Discussion
...
Keywords: Vector Index
Released: TBD
Summary
We got requirement for supporting ScaNN index in Milvus. This proposal is about how to integrate a new index into knowhere and expose it to client.
Document about how knowhere works in milvus v1.x: m1_chain.pdf
Docement about how knowhere works in milvus v2.0: TODO
Motivation
Google recently released a new algorithm to find Approximate Nearest Neighbors called ScaNN. They show through benchmarks that it can perform significantly better than the existing solutions like Annoy, FAISS, hnsw. The goal is to add support for ScaNN algorithm to the list of Milvus supported indexes.
Public Interfaces
Follwing public interfaces would be updated in knowhere =>
...
Code Block | ||||
---|---|---|---|---|
| ||||
namespace IndexEnum { const char* INDEX_SCANN = "SCANN"; } |
Design Details
As part of third party library, we will have to add a new library for ScaNN. This directory here: https://github.com/google-research/google-research/tree/master/scann/scann/scann_ops contains APIs for build/search etc. Also, ScaNN uses Bazel as it's dependency management while we would need to use CMake.
We will create new wrapper classes called IndexScann.h and IndexScann.One the library has been added, cpp in knowhere.
Code Block | ||||
---|---|---|---|---|
| ||||
class IndexScann : public VecIndex {
public:
IndexScann() {
index_type_ = IndexEnum::INDEX_SCANN;
}
BinarySet
Serialize(const Config& config) override;
void
Load(const BinarySet& index_binary) override;
void
Train(const DatasetPtr& dataset_ptr, const Config& config) override;
void
AddWithoutIds(const DatasetPtr&, const Config&) override;
DatasetPtr
Query(const DatasetPtr& dataset_ptr, const Config& config, const faiss::BitsetView bitset) override;
int64_t
Count() override;
int64_t
Dim() override;
void
UpdateIndexSize() override;
} |
IndexScann.cpp would be the implementation of IndexScann.h.
Compatibility, Deprecation, and Migration Plan
...