Feature plans

Features 


estimated deliver releaseUrgencyImportanceWorkload(month*person)Details
SQL Support2.4 Beta, 3.0 release4512Support mysql connector, with insert, delete, search, aggregate, ddl support
Velox execution engine2.3/2.4446Use velox to execute TableScan, Predicate, aggregation operators
MMap data management2.4343+Load data into disk and mmap for searching. Let Milvus to serve data large than memory 
Hybrid search with BM25 and vector3.0 or later246+Search jointedly with bm25 score and vector distance score
Dynamic schema change3.0456+Add, remove column
Distributed Log store3.0 or later236+Implement distributed log device to replace kafka/pulsar for faster speed and recovery
Add Log Node and remove datanode3.0233+Add log node to handle write/flush, datanode will merge with indexnode and only handle stateless jobs
Dynamic shard change3.0 or later223+Change collection shard number in flight
Change data capture2.3/2.4333+export inserted data to kafka and datawarehouse


Cluster level replication2.4443+replicate data between two clusters for cross datacenter failure recovery
PITR3.0 or later123+replay backup at any time
New persistent format2.3453+Change bin log data format to improve search and recovery speed.
Ranking Support3.0 or later123+Support complex ranking between scalar and vector score with machine learning model
Primary key dedup3.0443+ Dedup or overwrite when user write same primary key
Aggregation 2.3543Support count/groupby with where condition
Complex data type3.0243Support list, set, json datatype and there queries such as IN
GPU2.4/3.0353Support GPU based faiss and graph index
Multi vector support3.0 or later113Need more user scenario 
Condition delete3.0143Delete from xxx where nonPK = ??
Fp16/Bf16 support3.0241+Support BF16 and Fp16 could improve search latency and throught to 2X
Snapshot/Rollback3.0 or later113+Snapshot is cool, but it's not as urgent for now 
Support Quantization for graph index2.4 441+HNSW + PQ/SQ, NGT-PG
Auto Index 2.03.0 133+Smart index parameter tuning
Support Models in Milvus3.0 or later146+Support onnx models to do ranking and other models such as PCA
Data iterator2.4543Iterate through all data with condition in the collection
Spark Connector 3.0 333Combine spark to work with milvus together on offline processing
ScaNN Support2.3 441+Support scaNN in knowhere
Hedged Read2.4 431+ when collection enable multiple replicas, hedged read helps to improve availability and reduce tail latency
Binary vector support3.0241+Support binary vector in graph index
Support null data2.4 243+ Support data to be null

Knowhere/Segcore metrics

2.4 553Support prometheus based metrics collection
Vector as output field2.4 341+Support to retrieve vector field when search
Bulkload with clustering data2.4 243Support clustering data into segment before bulkload
Multi Vector3.0333Support multiple vector field in single entity

Tools







Tracing2.3333Dynamic tracing search/query request 
WebUI2.4441+Webui to show segment/channel distribution, index and collection stats
Milvus CLI 2.4431Help on triggering load balancing, compaction, flush and other operations
Milvus system check2.4440.5Check the consistency between etcd, S3 and memory
Backup2.3231Back and restore data 
performance diag tool3.0 or later111diagnose performance , including cpu usage, memory usage and more
Health check2.4331Check cluster health status

Other Enhancement







Hybrid search performance2.3353+Improve search with filtering performance, especially for strict filtering condition such as PK=1
Streaming data search performance2.3553+Improve search performance with concurrent write with read
Loadbalancing on large cluster2.3331+Change current load balancing strategy
Failure recovery speed2.3/2.4443Milvus can be fully recovered in 1 minuted under single machine crash, and zero down time with multiple replicas
Compaction optimization2.4443
  1. Introduce major compaction to repartition data 2. refine minor compaction to handle frequent delete
Error code 2.4543Refine all error code and ensure each error returned has a correct error
access log3.0 or later111record all the access log
Scalability3.0243each shard can hold 1B data, test on 5B data set
LLM + Milvus DEMO2.4531+A demo to show how Milvus can work together with openAI and huggingface
Memory control for flush, compaction and index building2.4343ensure the memory utilization is stable when compaction and flush triggered.
Go, Java, Python, Cpp, NodeJs, Restful SDK refinement2.3541+refine all sdk api and syncup,fully tested all the sdk listed
Build optimization2.2/2.3221+ Increase build speed, remove useless dependency, use conan as dependency management