milvus-io/milvus

[Feature]: Adding rate limiting to the Flush API

Open

#24,346 opened on May 24, 2023

View on GitHub
 (10 comments) (0 reactions) (1 assignee)Go (4,000 forks)batch import
good first issuekind/feature

Repository metrics

Stars
 (44,298 stars)
PR merge metrics
 (Avg merge 7d) (314 merged PRs in 30d)

Description

Is there an existing issue for this?

  • I have searched the existing issues

Is your feature request related to a problem? Please describe.

The Flush API provides users with the ability to persist data in the stream, which can save time waiting for indexing. However, frequent calls to Flush by many users in order to persist data as quickly as possible, even after inserting a single piece of data, is very unfriendly to Milvus' storage mechanism. Every time Flush is called in Milvus' storage system, a new set of files is generated. Ideally, the system should batch persistence based on runtime conditions rather than manually calling Flush too frequently and generating massive amounts of small files that greatly affect subsequent processes such as Load and Compaction. This can lead to a series of stability and performance issues.

There's no doubt about the role of the Flush API. To prevent misuse by users, call restrictions need to be added such as only allowing one call per minute or making this parameter configurable.

Describe the solution you'd like.

No response

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

Contributor guide