[fix](auditlog) add missing audit log fields and duplicate audit log error by morningman · Pull Request #42262 · apache/doris

morningman · 2024-10-22T10:25:13Z

What problem does this PR solve?

Problem Summary:

Issue 1

There are some fields that is missing in audit log table.
This PR add them all:

shuffle_send_rows
shuffle_send_bytes
scan_bytes_from_local_storage
scan_bytes_from_remote_storage
is_nereids
compute_group

Notice that compute_group is previously name cloudClusterName in fe.audit.log,
which is incorrect, so I change it to the right name.

After this PR, all these fields will be saved in both audit log table and fe.audit.log

Issue 2

The AuditEventBuilder need to be reset at each run, the there will be duplicate audit log.

Issue 3

Add a new statement call flush_audit_log(). It will flush the audit log immediately to audit_log table.
This is useful in test case, so that we don't need to wait 1min to flush the audit log data.

Check List (For Committer)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No colde files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes. cloudClusterName in fe.audit.log change to computeGroup. And some fields are added to the audit log table.
Does this need documentation?
- No.
- Yes. I will add this later
Release note
fix add missing audit log fields and duplicate audit log error

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-10-22T10:25:18Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

morningman · 2024-10-31T06:41:26Z

run buildall

github-actions · 2024-10-31T06:57:12Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-10-31T06:57:14Z

PR approved by anyone and no changes requested.

morningman · 2024-10-31T07:11:12Z

run buildall

morningman · 2024-10-31T09:58:51Z

run buildall

morningman · 2024-10-31T13:23:30Z

run buildall

…error #42262 (#43015) bp #42262

morningman · 2024-11-01T10:08:03Z

run buildall

morningman · 2024-11-01T13:13:58Z

run buildall

morningman · 2024-11-02T00:53:27Z

run buildall

github-actions · 2024-11-02T03:19:54Z

PR approved by at least one committer and no changes requested.

…error (#42262) ### What problem does this PR solve? Problem Summary: #### Issue 1 There are some fields that is missing in audit log table. This PR add them all: - shuffle_send_rows - shuffle_send_bytes - scan_bytes_from_local_storage - scan_bytes_from_remote_storage - is_nereids - compute_group Notice that `compute_group` is previously name `cloudClusterName` in fe.audit.log, which is incorrect, so I change it to the right name. After this PR, all these fields will be saved in both audit log table and fe.audit.log #### Issue 2 The `AuditEventBuilder` need to be reset at each run, the there will be duplicate audit log. #### Issue 3 Add a new statement `call flush_audit_log()`. It will flush the audit log immediately to audit_log table. This is useful in test case, so that we don't need to wait 1min to flush the audit log data. ### Release note [fix](auditlog) add missing audit log fields and duplicate audit log error

…error (apache#42262) ### What problem does this PR solve? Problem Summary: #### Issue 1 There are some fields that is missing in audit log table. This PR add them all: - shuffle_send_rows - shuffle_send_bytes - scan_bytes_from_local_storage - scan_bytes_from_remote_storage - is_nereids - compute_group Notice that `compute_group` is previously name `cloudClusterName` in fe.audit.log, which is incorrect, so I change it to the right name. After this PR, all these fields will be saved in both audit log table and fe.audit.log #### Issue 2 The `AuditEventBuilder` need to be reset at each run, the there will be duplicate audit log. #### Issue 3 Add a new statement `call flush_audit_log()`. It will flush the audit log immediately to audit_log table. This is useful in test case, so that we don't need to wait 1min to flush the audit log data. ### Release note [fix](auditlog) add missing audit log fields and duplicate audit log error

### What problem does this PR solve? Problem Summary: 1. add some missing output blocks for `test_audit_log_behavior.out` 2. `call flush_audit_log()` does not support old planner, add check for it. followup #42262

…error #42262 (#43303) cherry pick from #42262

### What problem does this PR solve? Problem Summary: 1. add some missing output blocks for `test_audit_log_behavior.out` 2. `call flush_audit_log()` does not support old planner, add check for it. followup apache#42262

## Versions apache/doris#42262 this pr remove trace_id in audit_log - [x] dev - [x] 3.0 - [x] 2.1 - [ ] 2.0 ## Languages - [ ] Chinese - [ ] English ## Docs Checklist - [ ] Checked by AI - [ ] Test Cases Built

…error (apache#42262) ### What problem does this PR solve? Problem Summary: #### Issue 1 There are some fields that is missing in audit log table. This PR add them all: - shuffle_send_rows - shuffle_send_bytes - scan_bytes_from_local_storage - scan_bytes_from_remote_storage - is_nereids - compute_group Notice that `compute_group` is previously name `cloudClusterName` in fe.audit.log, which is incorrect, so I change it to the right name. After this PR, all these fields will be saved in both audit log table and fe.audit.log #### Issue 2 The `AuditEventBuilder` need to be reset at each run, the there will be duplicate audit log. #### Issue 3 Add a new statement `call flush_audit_log()`. It will flush the audit log immediately to audit_log table. This is useful in test case, so that we don't need to wait 1min to flush the audit log data. ### Release note [fix](auditlog) add missing audit log fields and duplicate audit log error

### What problem does this PR solve? Problem Summary: 1. add some missing output blocks for `test_audit_log_behavior.out` 2. `call flush_audit_log()` does not support old planner, add check for it. followup apache#42262

morningman force-pushed the audit_log_fix branch 3 times, most recently from d8dfde3 to 88f6f3eCompare October 31, 2024 04:02

morningman changed the title ~~[fix](auditlog) add missing audit log fields~~[fix](auditlog) add missing audit log fields and duplicate audit log errorOct 31, 2024

morningman marked this pull request as ready for review October 31, 2024 06:21

morningman force-pushed the audit_log_fix branch from b0811f1 to a3326feCompare October 31, 2024 06:21

morningman added dev/2.1.x dev/3.0.x labels Oct 31, 2024

yiguolei previously approved these changes Oct 31, 2024
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Oct 31, 2024

morningman dismissed yiguolei’s stale review via a15944eOctober 31, 2024 07:11

morningman force-pushed the audit_log_fix branch from 8cd874e to a15944eCompare October 31, 2024 07:11

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Oct 31, 2024

morningman added the kind/behavior-changed label Oct 31, 2024

morningman mentioned this pull request Oct 31, 2024
[fix](auditlog) add missing audit log fields and duplicate audit log error #42262 #43015
Merged

morningman added a commit that referenced this pull request Nov 1, 2024
[fix](auditlog) add missing audit log fields and duplicate audit log …
e85e6c8
…error #42262 (#43015) bp #42262

wm1581066 added dev/2.1.7-merged and removed dev/2.1.x labels Nov 1, 2024

morningman force-pushed the audit_log_fix branch from e6090a9 to c267ab4Compare November 1, 2024 10:07

morningman added the dev/2.0.x label Nov 1, 2024

morningman added 6 commits November 2, 2024 08:52

1
136f17c

2
0ad1ccc

3
33c74b8

3
ac7f748

5
b7d4d8f

1
0fb215d

morningman force-pushed the audit_log_fix branch from d7705a1 to 0fb215dCompare November 2, 2024 00:53

wuwenchi approved these changes Nov 2, 2024
View reviewed changes

CalvinKirs approved these changes Nov 2, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Nov 2, 2024

morningman merged commit c354351 into apache:masterNov 2, 2024

github-actionsBot mentioned this pull request Nov 2, 2024
branch-3.0: [fix](auditlog) add missing audit log fields and duplicate audit log error #43129
Closed
16 tasks

morningman mentioned this pull request Nov 4, 2024
[fix](test) fix some test cases #43217
Merged
16 tasks

morningman mentioned this pull request Nov 6, 2024
[fix](auditlog) add missing audit log fields and duplicate audit log error #42262 #43303
Merged

yiguolei mentioned this pull request Nov 6, 2024
Release Note 2.1.7 #43319
Closed

morningman added a commit that referenced this pull request Nov 6, 2024
[fix](auditlog) add missing audit log fields and duplicate audit log …
1e93d71
…error #42262 (#43303) cherry pick from #42262

morningman added dev/3.0.3-merged and removed dev/3.0.x labels Nov 6, 2024

gavinchou added the cir label Nov 11, 2024

yiguolei mentioned this pull request Jan 19, 2025
Release Note 2.1.8 #47198
Closed

zhangstar333 mentioned this pull request May 13, 2025
[doc](audit_log) update audit_log about trace_id apache/doris-website#2375
Merged
8 tasks

Navigation Menu

Uh oh!

[fix](auditlog) add missing audit log fields and duplicate audit log error - #42262

[fix](auditlog) add missing audit log fields and duplicate audit log error#42262
morningman merged 8 commits into
apache:masterfrom
morningman:audit_log_fix

morningman commented Oct 22, 2024•
edited
Loading

Uh oh!

doris-robot commented Oct 22, 2024

Uh oh!

morningman commented Oct 31, 2024

Uh oh!

github-actionsBot commented Oct 31, 2024

Uh oh!

github-actionsBot commented Oct 31, 2024

Uh oh!

morningman commented Oct 31, 2024

Uh oh!

morningman commented Oct 31, 2024

Uh oh!

morningman commented Oct 31, 2024

Uh oh!

morningman commented Nov 1, 2024

Uh oh!

morningman commented Nov 1, 2024

Uh oh!

morningman commented Nov 2, 2024

Uh oh!

github-actionsBot commented Nov 2, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

morningman commented Oct 22, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Issue 1

Issue 2

Issue 3

Check List (For Committer)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Oct 22, 2024

Uh oh!

morningman commented Oct 31, 2024

Uh oh!

github-actionsBot commented Oct 31, 2024

Uh oh!

github-actionsBot commented Oct 31, 2024

Uh oh!

morningman commented Oct 31, 2024

Uh oh!

morningman commented Oct 31, 2024

Uh oh!

morningman commented Oct 31, 2024

Uh oh!

morningman commented Nov 1, 2024

Uh oh!

morningman commented Nov 1, 2024

Uh oh!

morningman commented Nov 2, 2024

Uh oh!

github-actionsBot commented Nov 2, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

morningman commented Oct 22, 2024•
edited
Loading