Skip to content

[Enhancement] Convert metric to Json format - #3635

Merged
chaoyli merged 5 commits into
apache:masterfrom
chaoyli:master
May 27, 2020
Merged

[Enhancement] Convert metric to Json format#3635
chaoyli merged 5 commits into
apache:masterfrom
chaoyli:master

Conversation

@chaoyli

@chaoylichaoyli commented May 19, 2020

Copy link
Copy Markdown
Contributor

Add a JSON format for existing metrics like this.

{
"tags":
{
"metric":"thread_pool",
"name":"thrift-server-pool",
"type":"active_thread_num"
},
"unit":"number",
"value":3
}

I add a new JsonMetricVisitor to handle the convertion.
It's not to modify existing PrometheusMetricVisitor and SimpleCoreMetricVisitor.
Also I add a unit item to indicate the metric better.

@chaoylichaoyli self-assigned this May 19, 2020
@chaoylichaoyli changed the title [Feature] Convert metric to Json format[Enhancement] Convert metric to Json formatMay 19, 2020
@chaoyli

chaoyli commented May 19, 2020

Copy link
Copy Markdown
ContributorAuthor

#3636

morningman
morningman requested changes May 19, 2020
View reviewed changes
Comment threadbe/src/http/action/metrics_action.cpp Outdated
Comment threadbe/src/http/action/metrics_action.cpp
Comment threadfe/src/main/java/org/apache/doris/metric/JsonMetricVisitor.java Outdated
Comment threadfe/src/main/java/org/apache/doris/metric/JsonMetricVisitor.java Outdated
Comment threadfe/src/main/java/org/apache/doris/metric/JsonMetricVisitor.java
Comment threadfe/src/main/java/org/apache/doris/metric/SimpleCoreMetricVisitor.java Outdated
Comment threadfe/src/main/java/org/apache/doris/metric/PrometheusMetricVisitor.java Outdated
Comment threadfe/src/main/java/org/apache/doris/metric/PrometheusMetricVisitor.java Outdated
Comment threadfe/src/main/java/org/apache/doris/qe/StmtExecutor.java Outdated
Comment threadfe/src/main/java/org/apache/doris/service/FrontendServiceImpl.java Outdated
@morningman

morningman commented May 19, 2020

Copy link
Copy Markdown
Contributor

Why we need a json format metric output?

@chaoyli

chaoyli commented May 21, 2020•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Why we need a json format metric output?

It's for incorporting another Monitor Tools easily. Json will be a good choice for this demand.

morningman
morningman reviewed May 21, 2020
View reviewed changes
Comment threadfe/src/main/java/org/apache/doris/task/AgentTaskQueue.java Outdated
Comment threadfe/src/main/java/org/apache/doris/task/AgentTaskQueue.java Outdated
@morningman

morningman commented May 21, 2020

Copy link
Copy Markdown
Contributor

Why we need a json format metric output?

It's for incorporting another Monitor Tools easily. Json will be a good choice for this demand.

If we adopt this json form of expansion each new line, will it result in a very large result set?
Which monitoring system uses Json?

@chaoyli

chaoyli commented May 25, 2020•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Why we need a json format metric output?

It's for incorporating another Monitor Tools easily. Json will be a good choice for this demand.

If we adopt this json form of expansion each new line, will it result in a very large result set?
Which monitoring system uses Json?

I have test it, It will not larger than a query profile.
open-falcon opensourced by Xiaomi is use JSON to collect metric.
https://github.com/open-falcon/falcon-plus

morningman
morningman reviewed May 25, 2020
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, except for some little problem.
And please fix UT.

Comment threadfe/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
chaoyli added 4 commits May 26, 2020 20:37
@chaoyli
[Feature] Convert metric to Json format
8befd9b
Add a JSON format for existing metrics like this.
```
{
"tags":
{
"metric":"thread_pool",
"name":"thrift-server-pool",
"type":"active_thread_num"
},
"unit":"number",
"value":3
}
```
I add a new JsonMetricVisitor to handle the convertion.
It's not to modify existing PrometheusMetricVisitor and SimpleCoreMetricVisitor.
Also I add a unit item to indicate the metric better.
@chaoyli
[Feature] Convert metric to Json format
b5d2f75
Add a JSON format for existing metrics like this.
```
{
"tags":
{
"metric":"thread_pool",
"name":"thrift-server-pool",
"type":"active_thread_num"
},
"unit":"number",
"value":3
}
```
I add a new JsonMetricVisitor to handle the convertion.
It's not to modify existing PrometheusMetricVisitor and SimpleCoreMetricVisitor.
Also I add a unit item to indicate the metric better.
I also add cloning tablet into statistic divided by database.
@chaoyli
[Feature] Change some function name to better understanding
f178894
And also use white space to replace newline in audit.log
@chaoyli
Use rapidjson instead
cf25ffb
@chaoyli

chaoyli commented May 26, 2020•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

LGTM, except for some little problem.
And please fix UT.

I have fixed the UT

morningman
morningman approved these changes May 26, 2020
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningmanmorningman added the approved Indicates a PR has been approved by one committer. label May 26, 2020
@chaoyli
chaoyli merged commit 1cc78fe into apache:masterMay 27, 2020
@morningmanmorningman mentioned this pull request May 27, 2020
Closed
baidu-doris pushed a commit to baidu-doris/incubator-doris that referenced this pull request Jun 17, 2020
@chaoyli
[Enhancement] Convert metric to Json format (apache#3635)
b1d6b8f
Add a JSON format for existing metrics like this.
```
{
"tags":
{
"metric":"thread_pool",
"name":"thrift-server-pool",
"type":"active_thread_num"
},
"unit":"number",
"value":3
}
```
I add a new JsonMetricVisitor to handle the transformation.
It's not to modify existing PrometheusMetricVisitor and SimpleCoreMetricVisitor.
Also I add
1. A unit item to indicate the metric better
2. Cloning tablet statistics divided by database.
3. Use white space to replace newline in audit.log
acelyc111 pushed a commit to acelyc111/incubator-doris that referenced this pull request Jan 20, 2021
@chaoyli@acelyc111
[Enhancement] Convert metric to Json format (apache#3635)
6f7bf60
Add a JSON format for existing metrics like this.
```
{
"tags":
{
"metric":"thread_pool",
"name":"thrift-server-pool",
"type":"active_thread_num"
},
"unit":"number",
"value":3
}
```
I add a new JsonMetricVisitor to handle the transformation.
It's not to modify existing PrometheusMetricVisitor and SimpleCoreMetricVisitor.
Also I add
1. A unit item to indicate the metric better 2. Cloning tablet statistics divided by database.
3. Use white space to replace newline in audit.log
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningmanmorningmanmorningman approved these changes

Assignees

@chaoylichaoyli

Labels

approvedIndicates a PR has been approved by one committer.kind/improvement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@chaoyli@morningman