[Feature](multi-catalog) Add memory tracker for orc reader/writer and arrow parquet writer。 by kaka11chen · Pull Request #37257 · apache/doris

kaka11chen · 2024-07-04T02:22:10Z

Proposed changes

backport #37234

doris-robot · 2024-07-04T02:22:15Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions

clang-tidy made some suggestions

github-actions · 2024-07-04T02:28:53Z

+
+#pragma once
+
+#include "orc/MemoryPool.hh"


warning: 'orc/MemoryPool.hh' file not found [clang-diagnostic-error]
#include"orc/MemoryPool.hh" ^

github-actions · 2024-07-04T02:28:54Z

+#endif // #if defined(USE_JEMALLOC) && defined(USE_MEM_TRACKER)
+ }
+
+ void free(char* p) override {


warning: pointer parameter 'p' can be pointer to const [readability-non-const-parameter]
Suggested change
voidfree(char*p) override {
voidfree(constchar*p) override {

github-actions · 2024-07-04T02:28:54Z

 set_compression_type(compress_type);
 }

+VOrcTransformer::~VOrcTransformer() {


warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
VOrcTransformer::~VOrcTransformer() { ^

github-actions · 2024-07-04T02:28:54Z

 const iceberg::Schema* iceberg_schema = nullptr);

- ~VOrcTransformer() = default;
+ ~VOrcTransformer();


warning: annotate this function with 'override' or (rarely) 'final' [modernize-use-override]
Suggested change
~VOrcTransformer();
~VOrcTransformer()override;

kaka11chen · 2024-07-04T05:49:13Z

run buildall

doris-robot · 2024-07-04T06:53:21Z

TeamCity be ut coverage result:
Function Coverage: 36.32% (9150/25194)
Line Coverage: 27.88% (74698/267971)
Region Coverage: 26.77% (38518/143874)
Branch Coverage: 23.47% (19520/83168)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9d35789ab994b03d9c4e892ca6ab5befaa2d0260_9d35789ab994b03d9c4e892ca6ab5befaa2d0260/report/index.html

zgxme · 2024-07-05T03:54:29Z

run buildall

doris-robot · 2024-07-05T04:58:56Z

TeamCity be ut coverage result:
Function Coverage: 36.31% (9151/25199)
Line Coverage: 27.87% (74696/268060)
Region Coverage: 26.76% (38517/143945)
Branch Coverage: 23.45% (19518/83230)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bb95ae7145370a6f5f37949e561d0a51e23dfe73_bb95ae7145370a6f5f37949e561d0a51e23dfe73/report/index.html

github-actions

clang-tidy made some suggestions

kaka11chen · 2024-07-09T07:22:05Z

run buildall

kaka11chen · 2024-07-09T08:10:58Z

run buildall

github-actions

clang-tidy made some suggestions

kaka11chen · 2024-07-09T08:32:14Z

run buildall

kaka11chen · 2024-07-09T08:59:59Z

run buildall

doris-robot · 2024-07-09T10:24:07Z

TeamCity be ut coverage result:
Function Coverage: 36.31% (9158/25219)
Line Coverage: 27.87% (74767/268237)
Region Coverage: 26.77% (38577/144101)
Branch Coverage: 23.47% (19548/83298)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a879062fdfbff971bd9172f6d62c09943acb09c6_a879062fdfbff971bd9172f6d62c09943acb09c6/report/index.html

github-actions

clang-tidy made some suggestions

kaka11chen · 2024-07-11T07:49:40Z

run buildall

kaka11chen · 2024-07-11T08:49:22Z

run buildall

kaka11chen · 2024-07-19T10:11:22Z

run buildall

doris-robot · 2024-07-19T11:51:48Z

TeamCity be ut coverage result:
Function Coverage: 36.41% (9245/25393)
Line Coverage: 27.98% (75548/270023)
Region Coverage: 26.80% (38835/144923)
Branch Coverage: 23.55% (19717/83734)
Coverage Report: http://coverage.selectdb-in.cc/coverage/608f36c6fc49b4c63eccc19ac4ba38acccfb2bc3_608f36c6fc49b4c63eccc19ac4ba38acccfb2bc3/report/index.html

kaka11chen · 2024-07-23T17:27:45Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-07-23T17:29:42Z

+ return arrow::Status::OK();
+}
+
+void ArrowAllocator::deallocate_aligned(uint8_t* ptr, int64_t size, int64_t alignment) {


warning: pointer parameter 'ptr' can be pointer to const [readability-non-const-parameter]
Suggested change
voidArrowAllocator::deallocate_aligned(uint8_t* ptr, int64_t size, int64_t alignment) {
voidArrowAllocator::deallocate_aligned(constuint8_t* ptr, int64_t size, int64_t alignment) {
be/src/vec/exec/format/parquet/arrow_memory_pool.h:47:
- void deallocate_aligned(uint8_t* ptr, int64_t size, int64_t alignment);+ void deallocate_aligned(const uint8_t* ptr, int64_t size, int64_t alignment);

kaka11chen · 2024-07-23T17:57:25Z

run buildall

github-actions

clang-tidy made some suggestions

doris-robot · 2024-07-23T19:27:19Z

TeamCity be ut coverage result:
Function Coverage: 36.39% (9242/25396)
Line Coverage: 27.97% (75531/270063)
Region Coverage: 26.79% (38835/144955)
Branch Coverage: 23.54% (19715/83762)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bf1a1a9416d265e469de8b9021ee6d004adbe8ef_bf1a1a9416d265e469de8b9021ee6d004adbe8ef/report/index.html

xinyiZzz

LGTM

github-actions · 2024-07-24T03:42:56Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-24T03:42:58Z

PR approved by anyone and no changes requested.

kaka11chen · 2024-07-24T06:23:48Z

run buildall

doris-robot · 2024-07-24T08:01:32Z

TeamCity be ut coverage result:
Function Coverage: 36.40% (9243/25396)
Line Coverage: 27.97% (75534/270071)
Region Coverage: 26.79% (38830/144960)
Branch Coverage: 23.53% (19712/83764)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8f3272d573f85eac5bf4ac38c04a24caaa92e5bf_8f3272d573f85eac5bf4ac38c04a24caaa92e5bf/report/index.html

kaka11chen · 2024-07-24T09:47:42Z

run buildall

doris-robot · 2024-07-24T11:29:09Z

TeamCity be ut coverage result:
Function Coverage: 36.40% (9243/25396)
Line Coverage: 27.97% (75533/270094)
Region Coverage: 26.78% (38821/144981)
Branch Coverage: 23.53% (19711/83784)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d746ddc44603b8c12ecf6a09c79c8f1681a95dbf_d746ddc44603b8c12ecf6a09c79c8f1681a95dbf/report/index.html

Backport #38562. Fix allocator.h compiling failed on mac which introduced by #37257.

github-actionsBot reviewed Jul 4, 2024
View reviewed changes

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch 3 times, most recently from cf824f8 to 9d35789Compare July 4, 2024 05:48

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from 9d35789 to bb95ae7Compare July 5, 2024 03:36

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from bb95ae7 to 4ebdd75Compare July 9, 2024 07:13

github-actionsBot reviewed Jul 9, 2024
View reviewed changes

Comment threadbe/src/vec/common/allocator.cpp
Comment threadbe/src/vec/common/allocator.h

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch 2 times, most recently from 37921e7 to 7df36ceCompare July 9, 2024 08:08

github-actionsBot reviewed Jul 9, 2024
View reviewed changes

Comment threadbe/src/vec/common/allocator.cpp

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from 7df36ce to fa2511bCompare July 9, 2024 08:32

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from fa2511b to a879062Compare July 9, 2024 08:57

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from a879062 to 8d619bfCompare July 11, 2024 07:31

github-actionsBot reviewed Jul 11, 2024
View reviewed changes

Comment threadbe/src/vec/common/allocator.cpp

[Feature](multi-catalog) Add memory tracker for orc reader and writer.
c28752d

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from 8d619bf to 78d16deCompare July 11, 2024 07:49

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from 78d16de to 9b362e4Compare July 11, 2024 08:48

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from 9b362e4 to ed40639Compare July 11, 2024 09:24

xinyiZzz reviewed Jul 23, 2024
View reviewed changes

Comment threadbe/src/vec/common/allocator.h Outdated

xinyiZzz reviewed Jul 23, 2024
View reviewed changes

Comment threadbe/src/vec/common/allocator.h Outdated

xinyiZzz reviewed Jul 23, 2024
View reviewed changes

Comment threadbe/src/vec/common/allocator.h Outdated

xinyiZzz reviewed Jul 23, 2024
View reviewed changes

Comment threadbe/src/vec/exec/format/parquet/arrow_memory_pool.h

github-actionsBot reviewed Jul 23, 2024
View reviewed changes

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch 2 times, most recently from 64a4a7e to bf1a1a9Compare July 23, 2024 17:56

github-actionsBot reviewed Jul 23, 2024
View reviewed changes

Comment threadbe/src/vec/common/allocator.h
Comment threadbe/src/vec/common/allocator.h

xinyiZzz approved these changes Jul 24, 2024
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 24, 2024

yiguolei reviewed Jul 24, 2024
View reviewed changes

Comment threadbe/src/vec/common/allocator.h

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from bf1a1a9 to 8f3272dCompare July 24, 2024 06:19

Fix review comments.
d746ddc

kaka11chen force-pushed the orc_parquet_memory_tracking_2.1 branch from 8f3272d to d746ddcCompare July 24, 2024 08:29

morningman merged commit a751372 into apache:branch-2.1Jul 25, 2024

kaka11chen mentioned this pull request Aug 1, 2024
[Fix](memory) Fix allocator.h compiling failed on mac. #38646
Merged

yiguolei pushed a commit that referenced this pull request Aug 1, 2024
[Fix](memory) Fix allocator.h compiling failed on mac. (#38646)
4042cdf
Backport #38562. Fix allocator.h compiling failed on mac which introduced by #37257.

yiguolei mentioned this pull request Sep 5, 2024
Release Note 2.1.6 #40443
Closed
3 tasks

	voidArrowAllocator::deallocate_aligned(uint8_t* ptr, int64_t size, int64_t alignment) {
	voidArrowAllocator::deallocate_aligned(constuint8_t* ptr, int64_t size, int64_t alignment) {


		#pragma once

		#include "orc/MemoryPool.hh"

Uh oh!

Conversation

kaka11chen commented Jul 4, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotJul 4, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBotJul 4, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBotJul 4, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBotJul 4, 2024

Choose a reason for hiding this comment

Uh oh!

kaka11chen commented Jul 4, 2024

Uh oh!

doris-robot commented Jul 4, 2024

Uh oh!

zgxme commented Jul 5, 2024

Uh oh!

doris-robot commented Jul 5, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kaka11chen commented Jul 9, 2024

Uh oh!

kaka11chen commented Jul 9, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kaka11chen commented Jul 9, 2024

Uh oh!

kaka11chen commented Jul 9, 2024

Uh oh!

doris-robot commented Jul 9, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kaka11chen commented Jul 11, 2024

Uh oh!

kaka11chen commented Jul 11, 2024

Uh oh!

kaka11chen commented Jul 19, 2024

Uh oh!

doris-robot commented Jul 19, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaka11chen commented Jul 23, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotJul 23, 2024

Choose a reason for hiding this comment

Uh oh!

kaka11chen commented Jul 23, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kaka11chen commented Jul 4, 2024•
edited
Loading