Current state: Accepted
...
To reduce network transmission and skip Plusar management, the new interface will allow users to input the path of some data files(json, numpy, etc.) on MinIO/S3 storage, and let the data nodes directly read these files and parse them into segments. The internal logic of the process becomes:
1. client calls import() to pass some file paths to Milvus proxy node
2. proxy node passes the file paths to data coordinator node
3. data coordinator node picks a data node or multiple data nodes (according to the sharding number) to parse files, each file can be parsed into a segment or multiple segments.
SDK Interfaces
The python API declaration:
...
Result segments availability
By definition, the result segments shall be available altogether. Which means there shall be no intermediate state for loading.
To achieve this property, the segments shall be marked as "Loading" state and be invisible before the whole loading procedure completes.
Bulk Load with Delete
Constraint: The segments generated by Bulk Load shall not be affected by delete operations before the whole procedure is finished.
...
- Allow adding segments without removing one
- Bring target segments online atomically.
Bulk Load and Index Building
The current behavior of query cluster is that if there is an index built for the collection, the segments will not be loaded(as sealed segment) before the index is built.
This constraint shall remain in first implementation of Bulk Load:
Constraint: The bulk load procedure shall include the period of index building of the result segments
Bulk Load as a tool
The bulk load logic can be extracted into a tool to run outside of Milvus process. It shall be implemented in the next release.