Milvus Project Roadmap and Time Schedule
Time | Note | |
---|---|---|
2.2 | 2022.10.28 | Stable |
2.2.3 | 2023.2.8 | Stable |
2.3 | 2023.2.28 | Experiment |
2.4 | 2023.6.30 | Stable |
3.0-RC1 | 2023.8.30 | Experiment |
3.0-GA | 2023.10.30 | Stable |
detailed features
Roadmap features
2.2
- Refactor Segment Assignment in Query coord
- Support Disk ANN Index
- Support RBAC
- Support BulkInsert Data
- Support Rate Limiter → Memory protection, Insertion/Search throughput
- Support collection level data TTL
- Primary/Backup coordinator
- Refine and fully tested pymilvus, java and go SDK.
- Release birdwatcher, milvus meta inspect and hacking tool
- Better Monitoring, collection level DML metrics supported
2.2.3
1. Rolling Upgrade
2. Resource group for querynode physical isolation
3. BulkInsert any sized data.
4. Performance optimization
5. Dynamic config change
2.3
- Refactor On Disk Data format
- Performance Improving on both memory/disk index
- Faster failure recovery speed.
- Merge Index coordinator/ Data coordinator into 1 role
- Integrate with new vector search engine, Knowhere 2.0
- Support Range Search
- Refine Alias functionality
- Backup support
- Dynamic Partition load/release
- Incremental data subscription
- Milvus benchmark tool
- Fully tested Restful API
- SQL support - count
- Intergated with Huggingface, OpenAI, Paddle paddle
2.4
- Support GPU index
- Support ScaNN Index
- Refactor Scalar Execution engine with Velox
- Better Hybrid search speed
- Better streaming data search performance
- Better Query pagination
- Improve multi memory replica availability
- Integrate with more inference generation system
- Tracing
- Support offline deployment
- SQL support - search,insert, delete, query
- Segment mmap load for memory index
- Index parameter optimization tool
- Compaction optimization
3.0
- Full SQL Support
- Modify collection schema dynamically - add column, remove column
- Support more datatype,map,list...
- Remove Datanode, all compaction/bulkload functionality moved into indexnode, streaming functionalities moved into log node
- Primary/Backup milvus cluster
- Support entity Update
- GPU index build
- Fully tested Cpp/Rust API
- New Milvus bootcamp
Version | Feature | Owner | Status | Comment |
---|---|---|---|---|
2.1 | Supports string data types | xi-ge16,dragondriver | in progress | https://wiki.lfaidata.foundation/display/MIL/MEP+11+--+Support+String+DataType?src=contextnavpagetreemode |
2.1 | Username/Password | awekjiang | in progress | MEP 27 -- Support Basic Authentication |
2.2 | Supports Scalar bitmap/inverted Index for string and numeric data types | pending | ||
2.2 | Collection alias atomic switch | pending | ||
2.2 | Supports alter collection | pending | ||
2.2 | Support auto data paritition | pending | ||
3.0 | Support list,set,json data type | pending |
DML
Version | Feature | Owner | Status | Comment |
---|---|---|---|---|
2.1 | support query/search result iteration | delayed to 2.2 | ||
2.1 | Support auto-expiration | Writtic | in progress | Partly finished. TBD next step. |
2.2 | Supports search by id | pending | TBD | |
2.2 | Vector range search | pending | TBD | |
2.2 | Update support | pending | ||
2.2 | Delete performance optimization | pending | ||
3.0 | Support complicated scalar operation with vectorization(DuckDB, Data fusion, Gandiva) | pending | ||
Long term | Multi vector support | pending | ||
Long term | Supports upsert/primary key deduplication | pending |
Features
Version | Feature | Owner | Status | Comment |
---|---|---|---|---|
2.1 | Multi tenant support and access control/Support ACL | Ke Jiang | in progress | MEP 29 -- Support Role-Based Access Control |
2.1 | Data compression | yah01 | done | |
2.2 | query pagination | pending | ||
2.2 | Segment LRU load | pending | ||
2.2 | Change data capture | pending | ||
3.0 | UDF support | pending | ||
3.0 | User specified reranking and preprocessing in container | pending | ||
3.0 | Adopts incremental backup | pending | ||
Long Term | Supports static data encryption | pending |
Performance/Cost
Version | Feature | Owner | Status | Comment |
---|---|---|---|---|
2.1 | performance benchmark | wangting | done | |
2.1 | performance optimization and monitoring | czs007 | in progress | |
2.1 | Multi memory replica | sunby | in progress | |
2.1 | Data bulkload | yhmo | done | MEP 24 -- Support bulk load |
2.2 | Supports GPU Index building and embedding retrieval | pending | ||
2.2 | Load balance optimization | congqi xia | pending | |
2.2 | Supports ScaNN Index | walmart labs | in progress | |
3.0 | Adopts cost-based query optimization algorithm to improve hybrid search efficiency | pending | ||
Long Term | Supports on-disk vector indexing | pending | ||
Long Term | Automatic index optimization | pending |
Stability
Version | Feature | Owner | Status | Comment |
---|---|---|---|---|
2.1 | Chaos test | zhuwenxing | done | basically works |
2.x | Pressure test | yanliang567 | pending | |
2.1 | Supports segment in memory replicas | sunby | in progress | |
2.2 | Coordinator Primary/Backup support | in progress | MEP30 | |
2.2 | Flow control && back pressure support | jaime | pending | |
3.0 | Master/Slave cluster | pending | ||
3.0 | Query node resource isolation | pending |
Ease Of Use
Version | Feature | Owner | Status | Comment |
---|---|---|---|---|
2.1 | Embedded Milvus that runs on laptops | soothing-rain | in progress | |
2.1 | windows support | matrix ji | in progress | |
2.1 | arm support | yusheng ma | in progress | |
2.2 | Multi datacenter deployment and multi-cloud integration | pending | ON AWS | |
2.2 | Cluster auto scaling | LoveEachDay | pending | |
2.2 | Milvus deploy with ansible | zwd1208 | in progress | |
Long Term | Offline vector computation | pending |
SDK
Version | Feature | Owner | Status | Comment |
---|---|---|---|---|
2.1 | Supports Restful APIs | haorenfsa, Anosh | in progress | coding |
2.1 | Supports C++ SDK | yhmo,matrixji,ArkaprabhaChakraborty | in progress | almost done |
3.0 | Milvus Query Language(SQL like, with ANN,Range support) | pending | ||
Long Term | Support GraphQL | pending | ||
Long Term | Support Rust SDK | pending |
Integration
Version | Feature | Owner | Status | Comment |
---|---|---|---|---|
2.1 | Integrates Kafka | Jaime | in progress | |
2.2 | Support meta storage interface (Meta in etcd, zookeeper and other user implemented logic) | Jaime | pending | |
2.2 | Refine milvus bootcamp with Towhee and Hugging face | hou jie | pending | |
2.2 | Data stored over local/distributed filesystems | pending | ||
Long Term | Integrates distributed KV stores such as HBase/TiKV/FoundationDB | pending |