Skip to content

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in … - #2833

Merged
szetszwo merged 1 commit into
apache:HDDS-4454from
guohao-rosicky:HDDS-4454-stream-channel
Nov 17, 2021
Merged

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …#2833
szetszwo merged 1 commit into
apache:HDDS-4454from
guohao-rosicky:HDDS-4454-stream-channel

Conversation

@guohao-rosicky

@guohao-rosickyguohao-rosicky commented Nov 12, 2021

Copy link
Copy Markdown
Contributor

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in the stream write

jira: https://issues.apache.org/jira/browse/HDDS-5961

@guohao-rosicky

guohao-rosicky commented Nov 12, 2021

Copy link
Copy Markdown
ContributorAuthor

hi @captainzmc , can you help review this patch?

@guohao-rosicky

guohao-rosicky commented Nov 12, 2021

Copy link
Copy Markdown
ContributorAuthor

The old StreamDataChannel implementation did not update the usedBytes in the Container metadata at write time.
There will be multiple Container types in the future, I put the implementation of DataChannel into the Container package.

@guohao-rosicky

guohao-rosicky commented Nov 12, 2021

Copy link
Copy Markdown
ContributorAuthor

hi @szetszwo , can you help review this patch?

@szetszwo

szetszwo commented Nov 15, 2021

Copy link
Copy Markdown
Contributor

@guohao-rosicky , I just have sync'ed the HDDS-4454 branch with master. Could you update this pull request?

@guohao-rosicky

guohao-rosicky commented Nov 15, 2021

Copy link
Copy Markdown
ContributorAuthor

@szetszwo@captainzmc , I have completed the update.

captainzmc
captainzmc reviewed Nov 15, 2021
View reviewed changes

@captainzmccaptainzmc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall, Failed UT may not be caused by this pr, it can be triggered again.

Comment thread...src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java Outdated

@captainzmccaptainzmcNov 15, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now we had updae metrics.incContainerBytesStats and containerData.updateWriteStats. Can you also add some test case? Just make sure that future changes do not affect this logic.

@guohao-rosickyguohao-rosickyNov 17, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@captainzmc ,Has been modified.

szetszwo
szetszwo requested changes Nov 16, 2021
View reviewed changes

@szetszwoszetszwo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@guohao-rosicky , thanks a lot for working on this. Some comments inlined; see also https://issues.apache.org/jira/secure/attachment/13036136/2833_review.patch .

BTW, we use 4 spaces for continuation indent in Ozone. Please change the continuation indent to 4 spaces (instead of 8). Thanks.

Comment thread...src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java Outdated

@szetszwoszetszwoNov 16, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use a new Type, say StreamWrite?

@guohao-rosickyguohao-rosickyNov 16, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@szetszwo is container type.

Comment thread...src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java Outdated

@szetszwoszetszwoNov 16, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove public.

Comment thread...src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java Outdated
Comment on lines 40 to 41

@szetszwoszetszwoNov 16, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove LOG since it is not used.

Comment thread...src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java Outdated

@szetszwoszetszwoNov 16, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove public.

Comment thread...src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java Outdated

@szetszwoszetszwoNov 16, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use ContainerData.

Comment thread...src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java Outdated

@szetszwoszetszwoNov 16, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use ContainerData so that we do not need to cast it when calling the constructor.

Comment thread...iner-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java Outdated

@szetszwoszetszwoNov 16, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add @Override.

Comment thread...e/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/ContainerDispatcher.java Outdated

@szetszwoszetszwoNov 16, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add default to throw UnsupportedOperationException so that the tests won't have to override it.

Comment thread...a/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java Outdated
Comment on lines 513 to 518

@szetszwoszetszwoNov 16, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not try-catch and throws StorageContainerException in the getStreamDataChannel(..) declaration.

Comment thread...tainer-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java Outdated

@szetszwoszetszwoNov 16, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add @Override.

@guohao-rosicky

guohao-rosicky commented Nov 17, 2021

Copy link
Copy Markdown
ContributorAuthor

Has been modified, @szetszwo@captainzmc .Please Take A Look.

szetszwo
szetszwo approved these changes Nov 17, 2021
View reviewed changes

@szetszwoszetszwo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 the latest change looks good.

@szetszwo
szetszwo merged commit 869f943 into apache:HDDS-4454Nov 17, 2021
szetszwo pushed a commit that referenced this pull request Nov 30, 2021
@guohao-rosicky@szetszwo
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
153ef3e
…the stream write (#2833)
captainzmc pushed a commit that referenced this pull request Dec 20, 2021
@guohao-rosicky@captainzmc
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
b1753e6
…the stream write (#2833)
szetszwo pushed a commit that referenced this pull request Dec 30, 2021
@guohao-rosicky@szetszwo
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
c659513
…the stream write (#2833)
szetszwo pushed a commit that referenced this pull request Jan 20, 2022
@guohao-rosicky@szetszwo
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
8d13099
…the stream write (#2833)
captainzmc pushed a commit that referenced this pull request Feb 9, 2022
@guohao-rosicky@captainzmc
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
a25e50c
…the stream write (#2833)
szetszwo pushed a commit that referenced this pull request Feb 16, 2022
@guohao-rosicky@szetszwo
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
6e01261
…the stream write (#2833)
szetszwo pushed a commit that referenced this pull request Mar 15, 2022
@guohao-rosicky@szetszwo
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
06dbec5
…the stream write (#2833)
szetszwo pushed a commit that referenced this pull request Mar 24, 2022
@guohao-rosicky@szetszwo
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
6bf06d1
…the stream write (#2833)
szetszwo pushed a commit to szetszwo/ozone that referenced this pull request May 6, 2022
@guohao-rosicky@szetszwo
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
bb41750
…the stream write (apache#2833)
szetszwo pushed a commit that referenced this pull request May 13, 2022
@guohao-rosicky@szetszwo
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
628a7a4
…the stream write (#2833)
szetszwo pushed a commit that referenced this pull request May 24, 2022
@guohao-rosicky@szetszwo
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
4871da7
…the stream write (#2833)
szetszwo pushed a commit that referenced this pull request Jun 9, 2022
@guohao-rosicky@szetszwo
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
e350e01
…the stream write (#2833)
captainzmc pushed a commit to captainzmc/hadoop-ozone that referenced this pull request Jul 4, 2022
@guohao-rosicky@captainzmc
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
ddc8a2c
…the stream write (apache#2833)
szetszwo pushed a commit that referenced this pull request Oct 25, 2022
@guohao-rosicky@szetszwo
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
8cbadbc
…the stream write (#2833)
(cherry picked from commit ddc8a2c)
(cherry picked from commit 65ec5817e22a666a4fa4fe76ac5c0c1c3263865a)
szetszwo pushed a commit that referenced this pull request Nov 7, 2022
@guohao-rosicky@szetszwo
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
7290ff9
…the stream write (#2833)
(cherry picked from commit ddc8a2c)
(cherry picked from commit 65ec5817e22a666a4fa4fe76ac5c0c1c3263865a)
(cherry picked from commit 8cbadbc)
szetszwo pushed a commit that referenced this pull request Dec 1, 2022
@guohao-rosicky@szetszwo
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
ad2fcba
…the stream write (#2833)
(cherry picked from commit 7290ff9)
szetszwo pushed a commit that referenced this pull request Dec 16, 2022
@guohao-rosicky@szetszwo
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
9a7edb0
…the stream write (#2833)
(cherry picked from commit 7290ff9)
nishitpatira pushed a commit to nishitpatira/ozone that referenced this pull request Dec 16, 2022
@guohao-rosickyNishit Patira
HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …
2cc3a78
…the stream write (apache#2833)
(cherry picked from commit 7290ff9)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@captainzmccaptainzmccaptainzmc left review comments
@szetszwoszetszwoszetszwo approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@guohao-rosicky@szetszwo@captainzmc