Feature plans
Features
| estimated deliver release | Urgency | Importance | Workload(month*person) | Details |
|---|---|---|---|---|---|
SQL Support | 2.4 Beta, 3.0 release | 4 | 5 | 12 | Support mysql connector, with insert, delete, search, aggregate, ddl support |
Velox execution engine | 2.3/2.4 | 4 | 4 | 6 | Use velox to execute TableScan, Predicate, aggregation operators |
MMap data management | 2.4 | 3 | 4 | 3+ | Load data into disk and mmap for searching. Let Milvus to serve data large than memory |
Hybrid search with BM25 and vector | 3.0 or later | 2 | 4 | 6+ | Search jointedly with bm25 score and vector distance score |
Dynamic schema change | 3.0 | 4 | 5 | 6+ | Add, remove column |
Distributed Log store | 3.0 or later | 2 | 3 | 6+ | Implement distributed log device to replace kafka/pulsar for faster speed and recovery |
Add Log Node and remove datanode | 3.0 | 2 | 3 | 3+ | Add log node to handle write/flush, datanode will merge with indexnode and only handle stateless jobs |
Dynamic shard change | 3.0 or later | 2 | 2 | 3+ | Change collection shard number in flight |
Change data capture | 2.3/2.4 | 3 | 3 | 3+ | export inserted data to kafka and datawarehouse
|
Cluster level replication | 2.4 | 4 | 4 | 3+ | replicate data between two clusters for cross datacenter failure recovery |
PITR | 3.0 or later | 1 | 2 | 3+ | replay backup at any time |
New persistent format | 2.3 | 4 | 5 | 3+ | Change bin log data format to improve search and recovery speed. |
Ranking Support | 3.0 or later | 1 | 2 | 3+ | Support complex ranking between scalar and vector score with machine learning model |
Primary key dedup | 3.0 | 4 | 4 | 3+ | Dedup or overwrite when user write same primary key |
Aggregation | 2.3 | 5 | 4 | 3 | Support count/groupby with where condition |
Complex data type | 3.0 | 2 | 4 | 3 | Support list, set, json datatype and there queries such as IN |
GPU | 2.4/3.0 | 3 | 5 | 3 | Support GPU based faiss and graph index |
Multi vector support | 3.0 or later | 1 | 1 | 3 | Need more user scenario |
Condition delete | 3.0 | 1 | 4 | 3 | Delete from xxx where nonPK = ?? |
Fp16/Bf16 support | 3.0 | 2 | 4 | 1+ | Support BF16 and Fp16 could improve search latency and throught to 2X |
Snapshot/Rollback | 3.0 or later | 1 | 1 | 3+ | Snapshot is cool, but it's not as urgent for now |
Support Quantization for graph index | 2.4 | 4 | 4 | 1+ | HNSW + PQ/SQ, NGT-PG |
Auto Index 2.0 | 3.0 | 1 | 3 | 3+ | Smart index parameter tuning |
Support Models in Milvus | 3.0 or later | 1 | 4 | 6+ | Support onnx models to do ranking and other models such as PCA |
Data iterator | 2.4 | 5 | 4 | 3 | Iterate through all data with condition in the collection |
Spark Connector | 3.0 | 3 | 3 | 3 | Combine spark to work with milvus together on offline processing |
ScaNN Support | 2.3 | 4 | 4 | 1+ | Support scaNN in knowhere |
Hedged Read | 2.4 | 4 | 3 | 1+ | when collection enable multiple replicas, hedged read helps to improve availability and reduce tail latency |
Binary vector support | 3.0 | 2 | 4 | 1+ | Support binary vector in graph index |
Support null data | 2.4 | 2 | 4 | 3+ | Support data to be null |
Knowhere/Segcore metrics | 2.4 | 5 | 5 | 3 | Support prometheus based metrics collection |
Vector as output field | 2.4 | 3 | 4 | 1+ | Support to retrieve vector field when search |
Bulkload with clustering data | 2.4 | 2 | 4 | 3 | Support clustering data into segment before bulkload |
Multi Vector | 3.0 | 3 | 3 | 3 | Support multiple vector field in single entity |
Tools
|
|
|
|
|
|
|---|---|---|---|---|---|
Tracing | 2.3 | 3 | 3 | 3 | Dynamic tracing search/query request |
WebUI | 2.4 | 4 | 4 | 1+ | Webui to show segment/channel distribution, index and collection stats |
Milvus CLI | 2.4 | 4 | 3 | 1 | Help on triggering load balancing， compaction, flush and other operations |
Milvus system check | 2.4 | 4 | 4 | 0.5 | Check the consistency between etcd, S3 and memory |
Backup | 2.3 | 2 | 3 | 1 | Back and restore data |
performance diag tool | 3.0 or later | 1 | 1 | 1 | diagnose performance , including cpu usage, memory usage and more |
Health check | 2.4 | 3 | 3 | 1 | Check cluster health status |
Other Enhancement
|
|
|
|
|
|
|---|---|---|---|---|---|
Hybrid search performance | 2.3 | 3 | 5 | 3+ | Improve search with filtering performance, especially for strict filtering condition such as PK=1 |
Streaming data search performance | 2.3 | 5 | 5 | 3+ | Improve search performance with concurrent write with read |
Loadbalancing on large cluster | 2.3 | 3 | 3 | 1+ | Change current load balancing strategy |
Failure recovery speed | 2.3/2.4 | 4 | 4 | 3 | Milvus can be fully recovered in 1 minuted under single machine crash, and zero down time with multiple replicas |
Compaction optimization | 2.4 | 4 | 4 | 3 |
|
Error code | 2.4 | 5 | 4 | 3 | Refine all error code and ensure each error returned has a correct error |
access log | 3.0 or later | 1 | 1 | 1 | record all the access log |
Scalability | 3.0 | 2 | 4 | 3 | each shard can hold 1B data, test on 5B data set |
LLM + Milvus DEMO | 2.4 | 5 |