Skip to content

[Stream Load][Optimize] Filter partitions by where header when generate stream load plan #5666

Description

@liutang123

When doing stream load, user can set partitions or temporary_partitions to specifies which partitions to import data into.

Sometimes, user don't known the partition names. FE will send all partitions info to BE for starting stream load. If some partitions(not the load partition) has bad tablet, FE will throw tablet xxx has few replicas: 1 and the load will fail.

proposal

Filter the partition names by where header.

As shown below:
Client send a stream load http request witch where header.
curl -XPUT -H 'where:dt=1' http://FE:port/api/db_name/table_name/_stream_load
If there are 3 partitions
p1 [0, 1)
p2 [1, 2)
p3 [2,3) bad partition
FE should only sends partition p2's info to BE.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions