...
- no CPU time, memory and disk consumption when creating index
- resource saving, only query node is involved in
- no code duplication for growing segment search
- unified search result for sealed segment and growing segment
Proposal 1 (only take IDMAP as an example)
Reuse index IDMAP, add new interface in both knowhere and faiss.
- Faiss adds new field "codes_ex" and new interface "add_ex" for structure IndexFlat. In IndexFlat, "codes" and "codes_ex" are mutual exclusive, user cannot set both of them.
- Knowhere adds a new interface `AddExWithoutIds()` for IDMAP.
- In Milvus, re-write API "FloatSearchBruteForce()" and "BinarySearchBruteForce()", let they it use enhanced IDMAP to search instead of calling Faiss interfaces.
Advantages: Little code change
Cons: Need add new interfaces in both Faiss and Knowhere
Proposal 2 (only take IDMAP as an example)
Add new index type IDMAP_EX, add new interface in faiss.
- Faiss adds new field "codes_ex" and new interface "add_ex" for structure IndexFlat. In IndexFlat, "codes" and "codes_ex" are mutual exclusive, user cannot set both of them.
- Knowhere adds a new index type IDMAP_EX, which use "add_ex" to insert vector data.
- In Milvus, re-write API "FloatSearchBruteForce()", let it use new IDMAP_EX to search.
Advantages: No new interface in Knowhere
Cons: Need add new index type in Knowhere, most part of code of this index is duplicate with IDMAP
Proposal 3 (only take IDMAP as an example)
Add a new wrapper API in knowhere to call faiss brute force search for all metric types
Advantages: No code change in faiss, only one new interface in Knowhere
Cons: This change dis-obey knowhere's design concept. By now all operations in knowhere is for an index, but this API is for all metric types, not for an index.
Public Interfaces(optional)
...