...
In some scenarios, users want to delete some entities from a collection that will no longer be searched out. Currently, users can only manually filter out unwanted entities from search results. We hope to implement a new function that allows users to delete entities from a collection.
Public Interfaces
...
| language | py |
|---|
...
& Behavior
Delete API can be used to delete entities in the collection, and the deleted entities will no longer appear in the results of the Query and Search request.
`collection_name` is the name of the collection to delete entities from.
`expr` is an expression indicated whether an entity should be deleted in the collection. Only the `in` operator is supported in the Delete API. Document of expression: https://github.com/milvus-io/milvus/blob/master/docs/design_docs/query_boolean_expr.md
...
`partition_name`
...
is
...
the
...
name
...
of
...
the partition
...
to
...
delete
...
entities
...
from,
...
`None`
...
means
...
all
...
partition.
...
`Delete`
...
returns
...
after
...
being
...
written
...
into
...
the
...
insert
...
channel,
...
which
...
means
...
the
...
delete
...
request
...
has
...
been
...
reliably
...
saved
...
and
...
will
...
be
...
applied
...
in
...
search/query
...
requests.
...
The
...
type
...
of
...
return
...
value
...
is
...
MutationResult,
...
which
...
contains
...
several
...
properties,
...
and
...
only
...
`_primary_keys`
...
will
...
be
...
filled.
...
Same
...
as
...
Insert
...
API,
...
Milvus
...
only
...
guarantee
...
the
...
visibility
...
of
...
operations
...
with
...
one client.
...
This
...
means
...
that,
...
within
...
the
...
sequence
...
of
...
the
...
operations
...
"delete(),
...
search()",
...
the
...
result
...
of
...
the
...
search
...
will
...
not
...
contains
...
the
...
entities
...
deleted.
...
Since
...
different clients
...
connect
...
to
...
different
...
Proxy,
...
the
...
time
...
between
...
different
...
Proxy
...
is
...
not
...
exactly
...
the same.
...
So
...
even
...
if
...
you
...
manually
...
call
...
the
...
delete
...
method
...
and
...
the
...
search
...
method
...
on
...
two
...
clients
...
sequentially,
...
it
...
is
...
uncertain
...
whether
...
the
...
search
...
request
...
returns
...
the
...
deleted
...
entities.
...
Currently,
...
Milvus
...
does
...
not
...
support
...
dedup
...
in
...
inserting,
...
so
...
the
...
delete
...
operation
...
will
...
delete
...
all
...
satisfied
...
entities.
...
Delete
...
a
...
non-existent
...
entity
...
is
...
not
...
an
...
error,
...
so
...
delete()
...
will
...
not
...
raise
...
an
...
Error.
...
| Code Block | ||
|---|---|---|
| ||
def delete(self, collection_name, expr, partition_name=None, timeout=None, **kwargs)->MutationResult:
"""
Delete entities with an expression condition.
And return results to show which primary key is deleted successfully
:param collection_name: Name of the collection to delete entities from
:type collection_name: str
:param expr: The query expression
:type expr: str
:param partition_name: Name of partitions that contain entities
:type partition_name: str
:param timeout: An optional duration of time in seconds to allow for the RPC. When timeout
is set to None, client waits until server response or error occur
:type timeout: float
:return: delete request executed results.
:rtype: MutationResult
:raises:
RpcError: If gRPC encounter an error
ParamError: If parameters are invalid
BaseException: If the return result from server is not ok
""" |
...
- Insert channel, handle following msg types:
- DDL msg (CreateCollection/DropCollection/CreatePartition/DropPartition)
- InsertMsg
- DeleteMsg
- Search Query channel, handle following msg types:
- SearchMsg
- RetrieveMsg
...
QueryNode consumes messages from both Insert channel and Search Query channel.
To support delete, we will send DeleteMsg into Insert channel also.
...
Since Milvus's storage is an append-only, `delete` function is implemented using soft delete, setting a flag on entity to indicate this entity has been deleted.
...
Proposal delete operation persistent
- DataNode subscribe the insert channel
- Proxy receives a delete request
- parse `expr` to get primary keys, then split DeleteMsg into insert channels by primary keys (done)
- DataNode receives a delete request from the insert channel, save it in buffer, and write it into the delta channel
- save deleted id into structure map<segID, list(id, ts)>
- receive DeleteMsg and persist delete data
- when DataNode start up, load all segments info from Minio into memory (done)
- update datanode flowgraph
- when DDNode receive DeleteMsg, save it into FlowGraphMsg structure, and send FlowGraphMsg to next node InsertBufferNode (done)
- when InsertBufferNode receive FlowGraphMsg, process InsertMsg in it, then wrapper DeleteMsg into another FlowGraphMsg and sent it to next node DeleteNode (done)
- when DeleteNode receive FlowGraphMsg, process DeleteMsg in it, save deleted ids and timestamps into delBuf (done)
- update DeleteNode to handle flushMsg
- add another flushChan for DeleteNode (done)
- when DeleteNode receives flushMsg, save all data in delBuf into MinIO
- deleted ids and timestamps are saved into two separated Minio file with name "/by-dev/deltalog/collectionID/partitionID/segmentID/xxx"
- DataNode notifies IndexNode to building indexes
- (done)
- DataNode forwards ttMsg and DeleteMsg to delta channel (doing)
Proposal delete operation serving search(sealed+growing)
...