[fix](meta) add sync new image timeout by CanGuan · Pull Request #25768 · apache/doris

CanGuan · 2023-10-23T07:58:30Z

Add timeout config for sync

Proposed changes

Issue Number: close#25764

Add sync_new_image_timeout config, for adjusting synchronize image timeout from master

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Add timeout config for sync

morningman · 2023-10-23T10:46:32Z

Have you try it? can it really solve the timeout issue?

morningman · 2023-10-23T10:46:45Z

+ "adjust by the size of image file in the ${meta_dir}/image and the network environment between nodes. " +
+ "The default values is 3000 ms."
+ })
+ public static int sync_new_image_timeout = 30000;


Suggested change
publicstaticintsync_new_image_timeout = 30000;
publicstaticintsync_new_image_timeout_s = 300;

morningman · 2023-10-23T10:46:56Z

 String filename = Storage.IMAGE + "." + version;
 File dir = new File(this.imageDir);
- MetaHelper.getRemoteFile(url, HTTP_TIMEOUT_SECOND * 1000, MetaHelper.getFile(filename, dir));
+ MetaHelper.getRemoteFile(url, Config.sync_new_image_timeout, MetaHelper.getFile(filename, dir));


Suggested change
MetaHelper.getRemoteFile(url, Config.sync_new_image_timeout, MetaHelper.getFile(filename, dir));
MetaHelper.getRemoteFile(url, Config.sync_new_image_timeout_s * 1000, MetaHelper.getFile(filename, dir));

Add timeout config for sync

CanGuan · 2023-10-23T11:15:55Z

Have you try it? can it really solve the timeout issue?

We found this bug by adding new follower to our doris cluster. By tracing this bug, we found the sync image timeout is only 5s and no way to modify it. After fixing and deploying to our cluster, it fix the problem. Here are the success log

In our case (image file is 1G), Follower takes 22s to synchronize the file from FE master.

refer to issue #25764

Add timeout config for sync

dataroaring · 2023-10-24T07:06:51Z

+ + "adjust by the size of image file in the ${meta_dir}/image and the network environment between nodes. "
+ + "The default values is 300s."
+ })
+ public static int sync_new_image_timeout_s = 300;


sync_image_timeout_second is a better name.

morningman

LGTM

morningman · 2023-10-24T10:42:50Z

run buildall

fix checkstyle

CanGuan · 2023-10-24T13:05:43Z

LGTM

Please review again, chore code for checkstyle

morningman

LGTM

morningman · 2023-10-25T07:17:03Z

run buildall

dataroaring

LGTM

github-actions · 2023-10-25T07:19:11Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-10-25T07:19:14Z

PR approved by anyone and no changes requested.

doris-robot · 2023-10-25T08:01:45Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.56 seconds
stream load tsv: 572 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162517629 Bytes

The image file of our cluster reaches 2.3G. After the checkpoint, Followers synchronize the image timeout, resulting in the continuous increase of the bdb directory. related pr: #25768

Add timeout config for sync

…#26003) The image file of our cluster reaches 2.3G. After the checkpoint, Followers synchronize the image timeout, resulting in the continuous increase of the bdb directory. related pr: apache#25768

Add timeout config for sync

…#26003) The image file of our cluster reaches 2.3G. After the checkpoint, Followers synchronize the image timeout, resulting in the continuous increase of the bdb directory. related pr: apache#25768

[fix](meta) add sync new image timeout (apache#25764)
c860d2d
Add timeout config for sync

CanGuan changed the title ~~[fix](meta) add sync new image timeout (#25764)~~[fix](meta) add sync new image timeoutOct 23, 2023

morningman reviewed Oct 23, 2023
View reviewed changes

[fix](meta) change config name (apache#25764)
ee3ba71
Add timeout config for sync

guanzhaoyun added 2 commits October 23, 2023 19:34

[fix](meta) change config name (apache#25764)
58eab3b
Add timeout config for sync

[fix](meta) fix checkstyle problems (apache#25764)
da024b3
Add timeout config for sync

dataroaring requested changes Oct 24, 2023
View reviewed changes

guanzhaoyunand others added 2 commits October 24, 2023 15:50

[fix](meta) change config name (apache#25764)
e38d5b4

Merge branch 'master' into master
e8dcefa

morningman added the dev/2.0.3 label Oct 24, 2023

morningman previously approved these changes Oct 24, 2023
View reviewed changes

guanzhaoyunand others added 3 commits October 24, 2023 18:59

[fix](meta) add sync new image timeout (apache#25764)
1a078be
fix checkstyle

Merge branch 'master' into master
ba19683

Merge remote-tracking branch 'origin/master'
570226c

CanGuan dismissed morningman’s stale review via 570226cOctober 24, 2023 11:02

CanGuanand others added 3 commits October 24, 2023 20:22

Merge branch 'master' into master
3c99d36

[fix](meta) add sync new image timeout (apache#25764)
5f1ec32
fix checkstyle

Merge remote-tracking branch 'origin/master'
e68c081

morningman approved these changes Oct 25, 2023
View reviewed changes

morningman self-assigned this Oct 25, 2023

dataroaring approved these changes Oct 25, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 25, 2023

github-actionsBot added the reviewed label Oct 25, 2023

morningman added the usercase Important user case type label label Oct 25, 2023

morningman merged commit 0fb57a6 into apache:masterOct 25, 2023

luozenglin mentioned this pull request Oct 27, 2023
[fix](meta) fix follower sync image timeout after checkpoint. #26003
Merged

dutyu pushed a commit to dutyu/doris that referenced this pull request Oct 28, 2023
[fix](meta) add sync new image timeout (apache#25768)
abc3a3e
Add timeout config for sync

xiaokang pushed a commit to xiaokang/doris that referenced this pull request Oct 31, 2023
[fix](meta) add sync new image timeout (apache#25768)
b8582a4
Add timeout config for sync

xiaokang pushed a commit to xiaokang/doris that referenced this pull request Nov 1, 2023
[fix](meta) add sync new image timeout (apache#25768)
dbe002f
Add timeout config for sync

xiaokang added dev/2.0.3-merged and removed dev/2.0.3 labels Nov 1, 2023

luozenglin mentioned this pull request Nov 2, 2023
[cherry-pick](meta) fix follower sync image timeout after checkpoint. (#26003) #26276
Merged

gnehil pushed a commit to gnehil/doris that referenced this pull request Dec 4, 2023
[fix](meta) add sync new image timeout (apache#25768)
09dbe4b
Add timeout config for sync

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
[fix](meta) add sync new image timeout (apache#25768)
8f372ed
Add timeout config for sync

nsnhuang mentioned this pull request Nov 14, 2024
[fix](config) fe config sync_image_timeout_second should not be masterOnly #43954
Merged

	publicstaticintsync_new_image_timeout = 30000;
	publicstaticintsync_new_image_timeout_s = 300;

	MetaHelper.getRemoteFile(url, Config.sync_new_image_timeout, MetaHelper.getFile(filename, dir));
	MetaHelper.getRemoteFile(url, Config.sync_new_image_timeout_s * 1000, MetaHelper.getFile(filename, dir));

Uh oh!

Conversation

CanGuan commented Oct 23, 2023

Proposed changes

Further comments

Uh oh!

morningman commented Oct 23, 2023

Uh oh!

morningmanOct 23, 2023

Choose a reason for hiding this comment

Uh oh!

CanGuanOct 23, 2023

Choose a reason for hiding this comment

Uh oh!

morningmanOct 23, 2023

Choose a reason for hiding this comment

Uh oh!

CanGuanOct 23, 2023

Choose a reason for hiding this comment

Uh oh!

CanGuan commented Oct 23, 2023

Uh oh!

dataroaringOct 24, 2023

Choose a reason for hiding this comment

Uh oh!

CanGuanOct 24, 2023

Choose a reason for hiding this comment

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

morningman commented Oct 24, 2023

Uh oh!

CanGuan commented Oct 24, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

morningman commented Oct 25, 2023

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Oct 25, 2023

Uh oh!

github-actionsBot commented Oct 25, 2023

Uh oh!

doris-robot commented Oct 25, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants