Skip to content

[bug](GC)the issue of incorrect disk usage - #18397

Merged
morningman merged 5 commits into
apache:masterfrom
chenlinzhong:fix_disk
Apr 8, 2023
Merged

[bug](GC)the issue of incorrect disk usage#18397
morningman merged 5 commits into
apache:masterfrom
chenlinzhong:fix_disk

Conversation

@chenlinzhong

@chenlinzhongchenlinzhong commented Apr 5, 2023

Copy link
Copy Markdown
Contributor

Proposed changes

disk used capacity is incorrect：
DataUsedCapacity+AvailCapacity !=TotalCapacity
image

Problem summary

some invalid rowset rowset not be deleted for ever due to the bug:
image
when key_may_exists() return true not means it is real exists

Checklist(Required)

  • Does it affect the original behavior
  • Has unit tests been added
  • Has document been added or modified
  • Does it need to update dependencies
  • Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@chenlinzhongchenlinzhong changed the title [bugfix] Fix the issue of incorrect disk usage[bug] (fix) the issue of incorrect disk usageApr 5, 2023
@github-actions

github-actionsBot commented Apr 5, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@chenlinzhongchenlinzhong changed the title [bug] (fix) the issue of incorrect disk usage[bug] (GC) the issue of incorrect disk usageApr 5, 2023
@chenlinzhongchenlinzhong changed the title [bug] (GC) the issue of incorrect disk usage[bug](GC)the issue of incorrect disk usageApr 5, 2023
@chenlinzhong

chenlinzhong commented Apr 5, 2023

Copy link
Copy Markdown
ContributorAuthor

run all

@morningmanmorningman self-assigned this Apr 6, 2023
@chenlinzhong

chenlinzhong commented Apr 6, 2023

Copy link
Copy Markdown
ContributorAuthor

run p0

1 similar comment
@chenlinzhong

chenlinzhong commented Apr 6, 2023

Copy link
Copy Markdown
ContributorAuthor

run p0

@chenlinzhong

chenlinzhong commented Apr 6, 2023

Copy link
Copy Markdown
ContributorAuthor

run beut

@chenlinzhong

chenlinzhong commented Apr 6, 2023

Copy link
Copy Markdown
ContributorAuthor

run feut

@chenlinzhong

chenlinzhong commented Apr 6, 2023

Copy link
Copy Markdown
ContributorAuthor

run p0

@chenlinzhong

chenlinzhong commented Apr 6, 2023

Copy link
Copy Markdown
ContributorAuthor

ru beut

@chenlinzhong

chenlinzhong commented Apr 6, 2023

Copy link
Copy Markdown
ContributorAuthor

run clickbench

morningman
morningman reviewed Apr 6, 2023
View reviewed changes
Comment threadbe/src/io/fs/file_system.cpp Outdated
Comment threadbe/src/olap/rowset/rowset_meta_manager.cpp
Comment threadbe/src/olap/tablet.cpp Outdated
@hello-stephen

hello-stephen commented Apr 6, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 36.12 seconds
stream load tsv: 443 seconds loaded 74807831229 Bytes, about 161 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 72 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230407160549_clickbench_pr_126080.html

@github-actions

github-actionsBot commented Apr 7, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

3 similar comments
@github-actions

github-actionsBot commented Apr 7, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions

github-actionsBot commented Apr 7, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions

github-actionsBot commented Apr 7, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@chenlinzhong

chenlinzhong commented Apr 7, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

morningman
morningman approved these changes Apr 8, 2023
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman
morningman merged commit 1616783 into apache:masterApr 8, 2023
@morningmanmorningman mentioned this pull request Apr 8, 2023
Merged
5 tasks
morningman added a commit that referenced this pull request Apr 10, 2023
@morningman
[fix](hdfs) not setting hadoop username when kerberos enabled (#18485)
ea47a6a
1. If we set hadoop user property along with kerberos info, the authentication will fail.
2. fix some minor issue of local fs, follow up #18397
3. Add KW_HOSTNAME to keywords region, follow up #17329
4. Fix tvf not working with pipeline engine, follow up #18376
morningman added a commit that referenced this pull request Apr 11, 2023
@morningman
[fix](hdfs) not setting hadoop username when kerberos enabled (#18485)
62eb640
1. If we set hadoop user property along with kerberos info, the authentication will fail.
2. fix some minor issue of local fs, follow up #18397
3. Add KW_HOSTNAME to keywords region, follow up #17329
4. Fix tvf not working with pipeline engine, follow up #18376
morningman pushed a commit that referenced this pull request Apr 12, 2023
@chenlinzhong@morningman
[bug](GC)the issue of incorrect disk usage (#18397)
1af97cd
morningman pushed a commit that referenced this pull request Apr 13, 2023
@chenlinzhong@morningman
[bug](GC)the issue of incorrect disk usage (#18397)
c54dacf
gnehil pushed a commit to gnehil/doris that referenced this pull request Apr 21, 2023
@chenlinzhong@gnehil
[bug](GC)the issue of incorrect disk usage (apache#18397)
13318ca
gnehil pushed a commit to gnehil/doris that referenced this pull request Apr 21, 2023
@morningman@gnehil
[fix](hdfs) not setting hadoop username when kerberos enabled (apache…
1e73966
…#18485)
1. If we set hadoop user property along with kerberos info, the authentication will fail.
2. fix some minor issue of local fs, follow up apache#18397
3. Add KW_HOSTNAME to keywords region, follow up apache#17329
4. Fix tvf not working with pipeline engine, follow up apache#18376
mongo360 pushed a commit to mongo360/doris that referenced this pull request Jul 12, 2023
@morningmancaiconghui1
[fix](hdfs) not setting hadoop username when kerberos enabled (apache…
b54bba0
…#18485)
1. If we set hadoop user property along with kerberos info, the authentication will fail.
2. fix some minor issue of local fs, follow up apache#18397
3. Add KW_HOSTNAME to keywords region, follow up apache#17329
4. Fix tvf not working with pipeline engine, follow up apache#18376
mongo360 pushed a commit to mongo360/doris that referenced this pull request Jul 12, 2023
@chenlinzhongcaiconghui1
[bug](GC)the issue of incorrect disk usage (apache#18397)
4ef2c7c
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningmanmorningmanmorningman approved these changes

Assignees

@morningmanmorningman

Labels

dev/1.2.4-merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@chenlinzhong@hello-stephen@morningman@yangzhg