Skip to content

[feature-wip] (memory tracker) (step6, End) Fix some details - #9301

Merged
morningman merged 6 commits into
apache:masterfrom
xinyiZzz:switch_tls_tracker6_fix_datils
May 10, 2022
Merged

[feature-wip] (memory tracker) (step6, End) Fix some details #9301
morningman merged 6 commits into
apache:masterfrom
xinyiZzz:switch_tls_tracker6_fix_datils

Conversation

@xinyiZzz

@xinyiZzzxinyiZzz commented Apr 28, 2022•
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: close#7196

Problem Summary:

  1. Fix LoadTask, ChunkAllocator, TabletMeta, Brpc, the accuracy of memory track.
  2. Modified some MemTracker names, deleted some unnecessary trackers, and improved readability.
  3. More powerful MemTracker debugging capabilities.
  4. Avoid creating TabletColumn temporary objects and improve BE startup time by 8%.
  5. Fix some other details.

Checklist(Required)

  1. Does it affect the original behavior: (Yes/No/I Don't know)
  2. Has unit tests been added: (Yes/No/No Need)
  3. Has document been added or modified: (Yes/No/No Need)
  4. Does it need to update dependencies: (Yes/No)
  5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

For related tests of impact and accuracy, can refer to the previous pr: #9145

In future:

  1. More tests, more accurate statistics.
  2. Faster tracking, there is now a 2% performance penalty.
  3. Use MemTracker raw pointers to improve performance by 1%.
  4. Optimized to switch existing Tracker and Task Tracker in TLS.
  5. Supports memory leak monitoring.

@xinyiZzz
xinyiZzzforce-pushed the switch_tls_tracker6_fix_datils branch 2 times, most recently from a04f145 to f2a8708CompareApril 29, 2022 10:39
morningman
morningman reviewed Apr 30, 2022
View reviewed changes
Comment threadbe/src/exec/tablet_sink.cpp Outdated
Comment threadbe/src/http/action/compaction_action.h Outdated
Comment threadbe/src/olap/tablet_manager.cpp Outdated
Comment threadbe/src/olap/tablet_manager.h Outdated
Comment threadbe/src/olap/tablet_schema.cpp Outdated
Comment threadbe/src/runtime/mem_pool.cpp Outdated
Comment threadbe/src/runtime/mem_tracker.h
Comment threadbe/src/runtime/mem_tracker.cpp
Comment threadbe/src/runtime/tcmalloc_hook.h Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
@xinyiZzz
xinyiZzzforce-pushed the switch_tls_tracker6_fix_datils branch 2 times, most recently from 22d4988 to 80f58dfCompareMay 6, 2022 03:13
@morningmanmorningman added kind/improvement area/memory-consumption dev/backlog waiting to be merged in future dev branch labels May 7, 2022
morningman
morningman previously approved these changes May 7, 2022
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label May 7, 2022
@github-actions

github-actionsBot commented May 7, 2022

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented May 7, 2022

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@xinyiZzz
xinyiZzzforce-pushed the switch_tls_tracker6_fix_datils branch from 28e4be0 to 5b56690CompareMay 9, 2022 02:04
@github-actionsgithub-actionsBot removed the approved Indicates a PR has been approved by one committer. label May 9, 2022
@morningman
morningman merged commit b34ed43 into apache:masterMay 10, 2022
zhengshiJ pushed a commit to zhengshiJ/incubator-doris that referenced this pull request May 13, 2022
@xinyiZzzjianghaochen
[feature-wip] (memory tracker) (step6, End) Fix some details (apache#…
3b3e65c
…9301)
1. Fix LoadTask, ChunkAllocator, TabletMeta, Brpc, the accuracy of memory track.
2. Modified some MemTracker names, deleted some unnecessary trackers, and improved readability.
3. More powerful MemTracker debugging capabilities.
4. Avoid creating TabletColumn temporary objects and improve BE startup time by 8%.
5. Fix some other details.
starocean999 pushed a commit to starocean999/incubator-doris that referenced this pull request May 19, 2022
@xinyiZzz@starocean999
[feature-wip] (memory tracker) (step6, End) Fix some details (apache#…
0a5f18f
…9301)
1. Fix LoadTask, ChunkAllocator, TabletMeta, Brpc, the accuracy of memory track.
2. Modified some MemTracker names, deleted some unnecessary trackers, and improved readability.
3. More powerful MemTracker debugging capabilities.
4. Avoid creating TabletColumn temporary objects and improve BE startup time by 8%.
5. Fix some other details.
englefly pushed a commit to englefly/incubator-doris that referenced this pull request May 23, 2022
@xinyiZzzminghong.zhou
[feature-wip] (memory tracker) (step6, End) Fix some details (apache#…
d129d51
…9301)
1. Fix LoadTask, ChunkAllocator, TabletMeta, Brpc, the accuracy of memory track.
2. Modified some MemTracker names, deleted some unnecessary trackers, and improved readability.
3. More powerful MemTracker debugging capabilities.
4. Avoid creating TabletColumn temporary objects and improve BE startup time by 8%.
5. Fix some other details.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningmanmorningmanmorningman approved these changes

Assignees

No one assigned

Labels

area/memory-consumptionarea/vectorizationdev/backlogwaiting to be merged in future dev branchkind/improvementreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature] Refactored memory statistics framework MemTracker

2 participants

@xinyiZzz@morningman