Current state: ["Under DiscussionDeclined"]
ISSUE: #17754
Keywords: IDMAP, BinaryIDMAP, brute force search, chunk
...
Add new interface only in Knowhere
Code Block | ||
---|---|---|
| ||
/**class knowhereBruteForce wrapper{ API topublic: call faiss brute force searchstatic forDatasetPtr all metric types * * @param metric_type * @param xb training vecors, size nb * dim * @param xq query vecors, size nq * dim * @param dim * @param nb rows of training vectors * @param nq rows of query vectors * @param topk * @param labels output, memory allocated and freed by caller * @param distances output, memory allocated and freed by coller * @param bitset */ void BruteForceSearch( const knowhere::MetricType& metric_type Search(const DatasetPtr base_dataset, const void* xb, const void* xq, const int64_t dimDatasetPtr query_dataset, const int64_t nb, const int64_t nq, const int64_t topkConfig& config, int64_t* labels, float* distances, const faiss::BitsetView bitset); /** knowhere wrapper API tostatic callDatasetPtr faiss brute force range search for all metric types * * @param metric_type * @param xb training vecors, size nb * dim * @param xq query vecors, size nq * dim * @param dim * @param nb rows of training vectors * @param nq rows of query vectors * @param radius range search radius * @param labels output, memory allocated inside and freed by caller * @param distances output, memory allocated inside and freed by coller * @param lims output, memory allocated inside and freed by coller * @param bitset */ void BruteForceRangeSearch( const knowhere::MetricType& metric_type RangeSearch(const DatasetPtr base_dataset, const void* xb, const void* xq, const DatasetPtr const int64_t dimquery_dataset, const int64_t nb, const int64_t nq, const float radius, int64_t*Config& labelsconfig, float*& distances, size_t*& lims, const faiss::BitsetView bitset); }; |
BruteForceSearchBruteForce::Search() is wrapper API to call Faiss brute force search for all metric types. The output parameters "labels" and "distances" are pointers to the memory allocated by caller. Caller will also free these memory after use.BruteForceRangeSearch
BruteForce::RangeSearch() is wrapper API to call Faiss brute force range search for all metric types. The output parameters "labels", "distances" and "lims" are reference of pointers to the memory allocated in Faiss. This is because the length of range search result is unknown, caller cannot allocate memory in advance. Caller need free these memory after use.
Design Details(required)
For growing segment in segcore, all brute force search related code can be removed, call knowhere's BruteForceSearch() and BruteForceRangeSearch() instead.
...
- This MEP will be transparent for users, and will not introduce any compatibility issue.
Test Plan(required)
Knowhere need add some unittests to test new interface `AddExWithoutIds()`.New testcases are added in Knowhere to test "BruteForceSearch()" and "BruteForceRangeSearch()".
- in knowhere/unittest/test_bruteforce.cpp
No extra testcases need be added in Milvus because current growing segment search testcases can cover this change.
...