Reorganize development docs by dai-chen · Pull Request #1200 · opensearch-project/sql

dai-chen · 2022-12-20T21:33:38Z

Signed-off-by: Chen Dai daichen@amazon.com

Description

This PR is the first one aimed to improve our dev documentation. Currently there are only a few docs in docs/dev/ folder. Actually there are many valuable info in our old docs or PR description. As initial effort, this PR added an index page reorganize existing docs to categories. Meanwhile I extracted a few PR description for main feature to new docs. Will upload more docs in separate PR.

Please see ~~https://github.com/dai-chen/sql-1/blob/refactor-dev-docs-rebased/docs/dev/index.md~~ -> https://github.com/opensearch-project/sql/blob/main/docs/dev/index.md

Issues Resolved

#1219

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov-commenter · 2022-12-23T22:16:08Z

Codecov Report

Merging #1200 (c7e306f) into main (1b58f7d) will increase coverage by 0.03%.
The diff coverage is n/a.

@@ Coverage Diff @@## main #1200 +/- ##
============================================
+ Coverage 98.32% 98.35% +0.03% - Complexity 3550 3609 +59 
============================================
Files 344 344 Lines 8762 8946 +184 Branches 554 569 +15 ============================================
+ Hits 8615 8799 +184 
Misses 142 142 Partials 5 5

Flag	Coverage Δ
sql-engine	`98.35% <ø> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...c/main/java/org/opensearch/sql/expression/DSL.java	`100.00% <0.00%> (ø)`
...n/java/org/opensearch/sql/utils/DateTimeUtils.java	`100.00% <0.00%> (ø)`
...a/org/opensearch/sql/data/model/ExprDateValue.java	`100.00% <0.00%> (ø)`
...a/org/opensearch/sql/data/model/ExprTimeValue.java	`100.00% <0.00%> (ø)`
.../org/opensearch/sql/data/model/ExprValueUtils.java	`100.00% <0.00%> (ø)`
...g/opensearch/sql/data/model/ExprDatetimeValue.java	`100.00% <0.00%> (ø)`
...pensearch/sql/sql/parser/AstExpressionBuilder.java	`100.00% <0.00%> (ø)`
...pensearch/sql/expression/function/FunctionDSL.java	`100.00% <0.00%> (ø)`
...arch/sql/expression/datetime/DateTimeFunction.java	`100.00% <0.00%> (ø)`
...h/sql/expression/function/BuiltinFunctionName.java	`100.00% <0.00%> (ø)`
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Chen Dai <daichen@amazon.com>

YANG-DB

This is an excellent direction +1

YANG-DB · 2023-01-24T17:37:30Z

+ **CLI**
+ **JDBC Driver**
+ **ODBC Driver**
+ **Query Workbench**


Isn’t this a UI feature ?
can we also have a UI section associated to the SQL ?

yes, I was planning to add some brief intro or link to their repo. Added here just to give users a complete picture of our eco system.

YANG-DB · 2023-01-24T17:39:14Z

+
+# OpenSearch SQL/PPL Engine Development Manual
+
+## Introduction


can we also have tutorial section for playground like exploring

sure, I'm still thinking what's the best way to do this. I have a sub task here for adding tutorial / quick start section: #1219.
Many other projects provide sample data and executable script/jar. User can play around with sample data files via CLI and their file connector. Because our CLI is written in Python, probably we can put some quick start example for playground website.

dai-chen · 2023-01-24T17:50:43Z

This is an excellent direction +1

@YANG-DB I've merged this PR right before your comments. Feel free to comment on #1219 if any idea. I will do it in the future PRs. Thanks!

Yury-Fridlyand · 2023-01-24T03:52:33Z

+* *Observability :* The ability to understand whats happening inside your business/application using logs, traces, metrics and other data emitted from the application.
+
+
+## 3.Tenets


Suggested change
## 3.Tenets
## 3.Tenets

Yury-Fridlyand · 2023-01-24T03:52:40Z

+* Changes to PPL Query Language grammar should be simple and easy to onboard.
+* Component design should be extensible for supporting new data stores.
+
+## 4.Out of Scope for the Design.


Suggested change
## 4.Out of Scope for the Design.
## 4.Out of Scope for the Design

Yury-Fridlyand · 2023-01-24T03:54:48Z

+
+
+


Suggested change

Yury-Fridlyand · 2023-01-24T03:54:55Z

+
+
+
+### 6.1 Data source representation.


Suggested change
### 6.1 Data source representation.
### 6.1 Data source representation

Yury-Fridlyand · 2023-01-24T03:55:09Z

+* *Catalog :* we can describe a catalog as the union of a connector and the underlying instance of the datasource being referred. This gives us flexbility to refer different instances of a similar datastore with the same connector i.e. we can refer data from multiple prometheus instances using prometheus connector. The name for each catalog should be different.
+
+Example Prometheus Catalog Definition
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:50:32Z

+
+* JSON Format


Suggested change
* JSON Format
JSON Format

Yury-Fridlyand · 2023-01-24T17:50:58Z

+[
+{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
+{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
+{"account_number": ExprIntegerValue(3)}
+]


Suggested change
[
{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
{"account_number": ExprIntegerValue(3)}
]
```json
[
{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
{"account_number": ExprIntegerValue(3)}
]
``

Yury-Fridlyand · 2023-01-24T17:51:13Z

+##### Include NULL and MISSING value in the QueryResult (Issue 1, 2)
+The SELECT operator will be translated to PhysicalOpeartor with a list of expression to resolve ExprValue from input data. With the above example, when handling NULL and MISSING value, the expected output data should be as follows.
+
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:53:02Z

+
+An additional list of Schema is also required to when protocol is JDBC.
+
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:58:07Z

+1. Each data source adds special logical operator and explode the optimizer rule space. For example, Prometheus also has `PrometheusLogicalMetricAgg` and `PrometheusLogicalMetricScan` accordingly. They have the exactly same pattern to match query plan tree as OpenSearch.
+2. A bigger problem is the difficulty of transforming from logical to physical when there are 2 `Table`s in query plan. Because only 1 of them has the chance to do the `implement()`. This is a blocker for supporting `INSERT ... SELECT ...` statement or JOIN query. See code below.
+
+```


Suggested change
```
```java

dai-chen · 2023-01-24T19:25:06Z

This is not back port to other branch intentionally. Because I think we may need single dedicated branch for Github pages in future.

dai-chen · 2022-12-20T21:33:38Z

Signed-off-by: Chen Dai daichen@amazon.com

Description

This PR is the first one aimed to improve our dev documentation. Currently there are only a few docs in docs/dev/ folder. Actually there are many valuable info in our old docs or PR description. As initial effort, this PR added an index page reorganize existing docs to categories. Meanwhile I extracted a few PR description for main feature to new docs. Will upload more docs in separate PR.

Please see ~~https://github.com/dai-chen/sql-1/blob/refactor-dev-docs-rebased/docs/dev/index.md~~ -> https://github.com/opensearch-project/sql/blob/main/docs/dev/index.md

Issues Resolved

#1219

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov-commenter · 2022-12-23T22:16:08Z

Codecov Report

Merging #1200 (c7e306f) into main (1b58f7d) will increase coverage by 0.03%.
The diff coverage is n/a.

@@ Coverage Diff @@## main #1200 +/- ##
============================================
+ Coverage 98.32% 98.35% +0.03% - Complexity 3550 3609 +59 
============================================
Files 344 344 Lines 8762 8946 +184 Branches 554 569 +15 ============================================
+ Hits 8615 8799 +184 
Misses 142 142 Partials 5 5

Flag	Coverage Δ
sql-engine	`98.35% <ø> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...c/main/java/org/opensearch/sql/expression/DSL.java	`100.00% <0.00%> (ø)`
...n/java/org/opensearch/sql/utils/DateTimeUtils.java	`100.00% <0.00%> (ø)`
...a/org/opensearch/sql/data/model/ExprDateValue.java	`100.00% <0.00%> (ø)`
...a/org/opensearch/sql/data/model/ExprTimeValue.java	`100.00% <0.00%> (ø)`
.../org/opensearch/sql/data/model/ExprValueUtils.java	`100.00% <0.00%> (ø)`
...g/opensearch/sql/data/model/ExprDatetimeValue.java	`100.00% <0.00%> (ø)`
...pensearch/sql/sql/parser/AstExpressionBuilder.java	`100.00% <0.00%> (ø)`
...pensearch/sql/expression/function/FunctionDSL.java	`100.00% <0.00%> (ø)`
...arch/sql/expression/datetime/DateTimeFunction.java	`100.00% <0.00%> (ø)`
...h/sql/expression/function/BuiltinFunctionName.java	`100.00% <0.00%> (ø)`
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Chen Dai <daichen@amazon.com>

YANG-DB

This is an excellent direction +1

YANG-DB · 2023-01-24T17:37:30Z

+ **CLI**
+ **JDBC Driver**
+ **ODBC Driver**
+ **Query Workbench**


Isn’t this a UI feature ?
can we also have a UI section associated to the SQL ?

yes, I was planning to add some brief intro or link to their repo. Added here just to give users a complete picture of our eco system.

YANG-DB · 2023-01-24T17:39:14Z

+
+# OpenSearch SQL/PPL Engine Development Manual
+
+## Introduction


can we also have tutorial section for playground like exploring

sure, I'm still thinking what's the best way to do this. I have a sub task here for adding tutorial / quick start section: #1219.
Many other projects provide sample data and executable script/jar. User can play around with sample data files via CLI and their file connector. Because our CLI is written in Python, probably we can put some quick start example for playground website.

dai-chen · 2023-01-24T17:50:43Z

This is an excellent direction +1

@YANG-DB I've merged this PR right before your comments. Feel free to comment on #1219 if any idea. I will do it in the future PRs. Thanks!

Yury-Fridlyand · 2023-01-24T03:52:33Z

+* *Observability :* The ability to understand whats happening inside your business/application using logs, traces, metrics and other data emitted from the application.
+
+
+## 3.Tenets


Suggested change
## 3.Tenets
## 3.Tenets

Yury-Fridlyand · 2023-01-24T03:52:40Z

+* Changes to PPL Query Language grammar should be simple and easy to onboard.
+* Component design should be extensible for supporting new data stores.
+
+## 4.Out of Scope for the Design.


Suggested change
## 4.Out of Scope for the Design.
## 4.Out of Scope for the Design

Yury-Fridlyand · 2023-01-24T03:54:48Z

+
+
+


Suggested change

Yury-Fridlyand · 2023-01-24T03:54:55Z

+
+
+
+### 6.1 Data source representation.


Suggested change
### 6.1 Data source representation.
### 6.1 Data source representation

Yury-Fridlyand · 2023-01-24T03:55:09Z

+* *Catalog :* we can describe a catalog as the union of a connector and the underlying instance of the datasource being referred. This gives us flexbility to refer different instances of a similar datastore with the same connector i.e. we can refer data from multiple prometheus instances using prometheus connector. The name for each catalog should be different.
+
+Example Prometheus Catalog Definition
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:50:32Z

+
+* JSON Format


Suggested change
* JSON Format
JSON Format

Yury-Fridlyand · 2023-01-24T17:50:58Z

+[
+{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
+{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
+{"account_number": ExprIntegerValue(3)}
+]


Suggested change
[
{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
{"account_number": ExprIntegerValue(3)}
]
```json
[
{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
{"account_number": ExprIntegerValue(3)}
]
``

Yury-Fridlyand · 2023-01-24T17:51:13Z

+##### Include NULL and MISSING value in the QueryResult (Issue 1, 2)
+The SELECT operator will be translated to PhysicalOpeartor with a list of expression to resolve ExprValue from input data. With the above example, when handling NULL and MISSING value, the expected output data should be as follows.
+
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:53:02Z

+
+An additional list of Schema is also required to when protocol is JDBC.
+
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:58:07Z

+1. Each data source adds special logical operator and explode the optimizer rule space. For example, Prometheus also has `PrometheusLogicalMetricAgg` and `PrometheusLogicalMetricScan` accordingly. They have the exactly same pattern to match query plan tree as OpenSearch.
+2. A bigger problem is the difficulty of transforming from logical to physical when there are 2 `Table`s in query plan. Because only 1 of them has the chance to do the `implement()`. This is a blocker for supporting `INSERT ... SELECT ...` statement or JOIN query. See code below.
+
+```


Suggested change
```
```java

dai-chen · 2023-01-24T19:25:06Z

This is not back port to other branch intentionally. Because I think we may need single dedicated branch for Github pages in future.

dai-chen · 2022-12-20T21:33:38Z

Signed-off-by: Chen Dai daichen@amazon.com

Description

This PR is the first one aimed to improve our dev documentation. Currently there are only a few docs in docs/dev/ folder. Actually there are many valuable info in our old docs or PR description. As initial effort, this PR added an index page reorganize existing docs to categories. Meanwhile I extracted a few PR description for main feature to new docs. Will upload more docs in separate PR.

Please see ~~https://github.com/dai-chen/sql-1/blob/refactor-dev-docs-rebased/docs/dev/index.md~~ -> https://github.com/opensearch-project/sql/blob/main/docs/dev/index.md

Issues Resolved

#1219

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov-commenter · 2022-12-23T22:16:08Z

Codecov Report

Merging #1200 (c7e306f) into main (1b58f7d) will increase coverage by 0.03%.
The diff coverage is n/a.

@@ Coverage Diff @@## main #1200 +/- ##
============================================
+ Coverage 98.32% 98.35% +0.03% - Complexity 3550 3609 +59 
============================================
Files 344 344 Lines 8762 8946 +184 Branches 554 569 +15 ============================================
+ Hits 8615 8799 +184 
Misses 142 142 Partials 5 5

Flag	Coverage Δ
sql-engine	`98.35% <ø> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...c/main/java/org/opensearch/sql/expression/DSL.java	`100.00% <0.00%> (ø)`
...n/java/org/opensearch/sql/utils/DateTimeUtils.java	`100.00% <0.00%> (ø)`
...a/org/opensearch/sql/data/model/ExprDateValue.java	`100.00% <0.00%> (ø)`
...a/org/opensearch/sql/data/model/ExprTimeValue.java	`100.00% <0.00%> (ø)`
.../org/opensearch/sql/data/model/ExprValueUtils.java	`100.00% <0.00%> (ø)`
...g/opensearch/sql/data/model/ExprDatetimeValue.java	`100.00% <0.00%> (ø)`
...pensearch/sql/sql/parser/AstExpressionBuilder.java	`100.00% <0.00%> (ø)`
...pensearch/sql/expression/function/FunctionDSL.java	`100.00% <0.00%> (ø)`
...arch/sql/expression/datetime/DateTimeFunction.java	`100.00% <0.00%> (ø)`
...h/sql/expression/function/BuiltinFunctionName.java	`100.00% <0.00%> (ø)`
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Chen Dai <daichen@amazon.com>

YANG-DB

This is an excellent direction +1

YANG-DB · 2023-01-24T17:37:30Z

+ **CLI**
+ **JDBC Driver**
+ **ODBC Driver**
+ **Query Workbench**


Isn’t this a UI feature ?
can we also have a UI section associated to the SQL ?

yes, I was planning to add some brief intro or link to their repo. Added here just to give users a complete picture of our eco system.

YANG-DB · 2023-01-24T17:39:14Z

+
+# OpenSearch SQL/PPL Engine Development Manual
+
+## Introduction


can we also have tutorial section for playground like exploring

sure, I'm still thinking what's the best way to do this. I have a sub task here for adding tutorial / quick start section: #1219.
Many other projects provide sample data and executable script/jar. User can play around with sample data files via CLI and their file connector. Because our CLI is written in Python, probably we can put some quick start example for playground website.

dai-chen · 2023-01-24T17:50:43Z

This is an excellent direction +1

@YANG-DB I've merged this PR right before your comments. Feel free to comment on #1219 if any idea. I will do it in the future PRs. Thanks!

Yury-Fridlyand · 2023-01-24T03:52:33Z

+* *Observability :* The ability to understand whats happening inside your business/application using logs, traces, metrics and other data emitted from the application.
+
+
+## 3.Tenets


Suggested change
## 3.Tenets
## 3.Tenets

Yury-Fridlyand · 2023-01-24T03:52:40Z

+* Changes to PPL Query Language grammar should be simple and easy to onboard.
+* Component design should be extensible for supporting new data stores.
+
+## 4.Out of Scope for the Design.


Suggested change
## 4.Out of Scope for the Design.
## 4.Out of Scope for the Design

Yury-Fridlyand · 2023-01-24T03:54:48Z

+
+
+


Suggested change

Yury-Fridlyand · 2023-01-24T03:54:55Z

+
+
+
+### 6.1 Data source representation.


Suggested change
### 6.1 Data source representation.
### 6.1 Data source representation

Yury-Fridlyand · 2023-01-24T03:55:09Z

+* *Catalog :* we can describe a catalog as the union of a connector and the underlying instance of the datasource being referred. This gives us flexbility to refer different instances of a similar datastore with the same connector i.e. we can refer data from multiple prometheus instances using prometheus connector. The name for each catalog should be different.
+
+Example Prometheus Catalog Definition
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:50:32Z

+
+* JSON Format


Suggested change
* JSON Format
JSON Format

Yury-Fridlyand · 2023-01-24T17:50:58Z

+[
+{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
+{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
+{"account_number": ExprIntegerValue(3)}
+]


Suggested change
[
{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
{"account_number": ExprIntegerValue(3)}
]
```json
[
{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
{"account_number": ExprIntegerValue(3)}
]
``

Yury-Fridlyand · 2023-01-24T17:51:13Z

+##### Include NULL and MISSING value in the QueryResult (Issue 1, 2)
+The SELECT operator will be translated to PhysicalOpeartor with a list of expression to resolve ExprValue from input data. With the above example, when handling NULL and MISSING value, the expected output data should be as follows.
+
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:53:02Z

+
+An additional list of Schema is also required to when protocol is JDBC.
+
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:58:07Z

+1. Each data source adds special logical operator and explode the optimizer rule space. For example, Prometheus also has `PrometheusLogicalMetricAgg` and `PrometheusLogicalMetricScan` accordingly. They have the exactly same pattern to match query plan tree as OpenSearch.
+2. A bigger problem is the difficulty of transforming from logical to physical when there are 2 `Table`s in query plan. Because only 1 of them has the chance to do the `implement()`. This is a blocker for supporting `INSERT ... SELECT ...` statement or JOIN query. See code below.
+
+```


Suggested change
```
```java

dai-chen · 2023-01-24T19:25:06Z

This is not back port to other branch intentionally. Because I think we may need single dedicated branch for Github pages in future.

dai-chen · 2022-12-20T21:33:38Z

Signed-off-by: Chen Dai daichen@amazon.com

Description

This PR is the first one aimed to improve our dev documentation. Currently there are only a few docs in docs/dev/ folder. Actually there are many valuable info in our old docs or PR description. As initial effort, this PR added an index page reorganize existing docs to categories. Meanwhile I extracted a few PR description for main feature to new docs. Will upload more docs in separate PR.

Please see ~~https://github.com/dai-chen/sql-1/blob/refactor-dev-docs-rebased/docs/dev/index.md~~ -> https://github.com/opensearch-project/sql/blob/main/docs/dev/index.md

Issues Resolved

#1219

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov-commenter · 2022-12-23T22:16:08Z

Codecov Report

Merging #1200 (c7e306f) into main (1b58f7d) will increase coverage by 0.03%.
The diff coverage is n/a.

@@ Coverage Diff @@## main #1200 +/- ##
============================================
+ Coverage 98.32% 98.35% +0.03% - Complexity 3550 3609 +59 
============================================
Files 344 344 Lines 8762 8946 +184 Branches 554 569 +15 ============================================
+ Hits 8615 8799 +184 
Misses 142 142 Partials 5 5

Flag	Coverage Δ
sql-engine	`98.35% <ø> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...c/main/java/org/opensearch/sql/expression/DSL.java	`100.00% <0.00%> (ø)`
...n/java/org/opensearch/sql/utils/DateTimeUtils.java	`100.00% <0.00%> (ø)`
...a/org/opensearch/sql/data/model/ExprDateValue.java	`100.00% <0.00%> (ø)`
...a/org/opensearch/sql/data/model/ExprTimeValue.java	`100.00% <0.00%> (ø)`
.../org/opensearch/sql/data/model/ExprValueUtils.java	`100.00% <0.00%> (ø)`
...g/opensearch/sql/data/model/ExprDatetimeValue.java	`100.00% <0.00%> (ø)`
...pensearch/sql/sql/parser/AstExpressionBuilder.java	`100.00% <0.00%> (ø)`
...pensearch/sql/expression/function/FunctionDSL.java	`100.00% <0.00%> (ø)`
...arch/sql/expression/datetime/DateTimeFunction.java	`100.00% <0.00%> (ø)`
...h/sql/expression/function/BuiltinFunctionName.java	`100.00% <0.00%> (ø)`
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Chen Dai <daichen@amazon.com>

YANG-DB

This is an excellent direction +1

YANG-DB · 2023-01-24T17:37:30Z

+ **CLI**
+ **JDBC Driver**
+ **ODBC Driver**
+ **Query Workbench**


Isn’t this a UI feature ?
can we also have a UI section associated to the SQL ?

yes, I was planning to add some brief intro or link to their repo. Added here just to give users a complete picture of our eco system.

YANG-DB · 2023-01-24T17:39:14Z

+
+# OpenSearch SQL/PPL Engine Development Manual
+
+## Introduction


can we also have tutorial section for playground like exploring

sure, I'm still thinking what's the best way to do this. I have a sub task here for adding tutorial / quick start section: #1219.
Many other projects provide sample data and executable script/jar. User can play around with sample data files via CLI and their file connector. Because our CLI is written in Python, probably we can put some quick start example for playground website.

dai-chen · 2023-01-24T17:50:43Z

This is an excellent direction +1

@YANG-DB I've merged this PR right before your comments. Feel free to comment on #1219 if any idea. I will do it in the future PRs. Thanks!

Yury-Fridlyand · 2023-01-24T03:52:33Z

+* *Observability :* The ability to understand whats happening inside your business/application using logs, traces, metrics and other data emitted from the application.
+
+
+## 3.Tenets


Suggested change
## 3.Tenets
## 3.Tenets

Yury-Fridlyand · 2023-01-24T03:52:40Z

+* Changes to PPL Query Language grammar should be simple and easy to onboard.
+* Component design should be extensible for supporting new data stores.
+
+## 4.Out of Scope for the Design.


Suggested change
## 4.Out of Scope for the Design.
## 4.Out of Scope for the Design

Yury-Fridlyand · 2023-01-24T03:54:48Z

+
+
+


Suggested change

Yury-Fridlyand · 2023-01-24T03:54:55Z

+
+
+
+### 6.1 Data source representation.


Suggested change
### 6.1 Data source representation.
### 6.1 Data source representation

Yury-Fridlyand · 2023-01-24T03:55:09Z

+* *Catalog :* we can describe a catalog as the union of a connector and the underlying instance of the datasource being referred. This gives us flexbility to refer different instances of a similar datastore with the same connector i.e. we can refer data from multiple prometheus instances using prometheus connector. The name for each catalog should be different.
+
+Example Prometheus Catalog Definition
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:50:32Z

+
+* JSON Format


Suggested change
* JSON Format
JSON Format

Yury-Fridlyand · 2023-01-24T17:50:58Z

+[
+{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
+{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
+{"account_number": ExprIntegerValue(3)}
+]


Suggested change
[
{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
{"account_number": ExprIntegerValue(3)}
]
```json
[
{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
{"account_number": ExprIntegerValue(3)}
]
``

Yury-Fridlyand · 2023-01-24T17:51:13Z

+##### Include NULL and MISSING value in the QueryResult (Issue 1, 2)
+The SELECT operator will be translated to PhysicalOpeartor with a list of expression to resolve ExprValue from input data. With the above example, when handling NULL and MISSING value, the expected output data should be as follows.
+
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:53:02Z

+
+An additional list of Schema is also required to when protocol is JDBC.
+
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:58:07Z

+1. Each data source adds special logical operator and explode the optimizer rule space. For example, Prometheus also has `PrometheusLogicalMetricAgg` and `PrometheusLogicalMetricScan` accordingly. They have the exactly same pattern to match query plan tree as OpenSearch.
+2. A bigger problem is the difficulty of transforming from logical to physical when there are 2 `Table`s in query plan. Because only 1 of them has the chance to do the `implement()`. This is a blocker for supporting `INSERT ... SELECT ...` statement or JOIN query. See code below.
+
+```


Suggested change
```
```java

dai-chen · 2023-01-24T19:25:06Z

This is not back port to other branch intentionally. Because I think we may need single dedicated branch for Github pages in future.

dai-chen · 2022-12-20T21:33:38Z

Signed-off-by: Chen Dai daichen@amazon.com

Description

This PR is the first one aimed to improve our dev documentation. Currently there are only a few docs in docs/dev/ folder. Actually there are many valuable info in our old docs or PR description. As initial effort, this PR added an index page reorganize existing docs to categories. Meanwhile I extracted a few PR description for main feature to new docs. Will upload more docs in separate PR.

Please see ~~https://github.com/dai-chen/sql-1/blob/refactor-dev-docs-rebased/docs/dev/index.md~~ -> https://github.com/opensearch-project/sql/blob/main/docs/dev/index.md

Issues Resolved

#1219

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov-commenter · 2022-12-23T22:16:08Z

Codecov Report

Merging #1200 (c7e306f) into main (1b58f7d) will increase coverage by 0.03%.
The diff coverage is n/a.

@@ Coverage Diff @@## main #1200 +/- ##
============================================
+ Coverage 98.32% 98.35% +0.03% - Complexity 3550 3609 +59 
============================================
Files 344 344 Lines 8762 8946 +184 Branches 554 569 +15 ============================================
+ Hits 8615 8799 +184 
Misses 142 142 Partials 5 5

Flag	Coverage Δ
sql-engine	`98.35% <ø> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...c/main/java/org/opensearch/sql/expression/DSL.java	`100.00% <0.00%> (ø)`
...n/java/org/opensearch/sql/utils/DateTimeUtils.java	`100.00% <0.00%> (ø)`
...a/org/opensearch/sql/data/model/ExprDateValue.java	`100.00% <0.00%> (ø)`
...a/org/opensearch/sql/data/model/ExprTimeValue.java	`100.00% <0.00%> (ø)`
.../org/opensearch/sql/data/model/ExprValueUtils.java	`100.00% <0.00%> (ø)`
...g/opensearch/sql/data/model/ExprDatetimeValue.java	`100.00% <0.00%> (ø)`
...pensearch/sql/sql/parser/AstExpressionBuilder.java	`100.00% <0.00%> (ø)`
...pensearch/sql/expression/function/FunctionDSL.java	`100.00% <0.00%> (ø)`
...arch/sql/expression/datetime/DateTimeFunction.java	`100.00% <0.00%> (ø)`
...h/sql/expression/function/BuiltinFunctionName.java	`100.00% <0.00%> (ø)`
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Chen Dai <daichen@amazon.com>

YANG-DB

This is an excellent direction +1

YANG-DB · 2023-01-24T17:37:30Z

+ **CLI**
+ **JDBC Driver**
+ **ODBC Driver**
+ **Query Workbench**


Isn’t this a UI feature ?
can we also have a UI section associated to the SQL ?

yes, I was planning to add some brief intro or link to their repo. Added here just to give users a complete picture of our eco system.

YANG-DB · 2023-01-24T17:39:14Z

+
+# OpenSearch SQL/PPL Engine Development Manual
+
+## Introduction


can we also have tutorial section for playground like exploring

sure, I'm still thinking what's the best way to do this. I have a sub task here for adding tutorial / quick start section: #1219.
Many other projects provide sample data and executable script/jar. User can play around with sample data files via CLI and their file connector. Because our CLI is written in Python, probably we can put some quick start example for playground website.

dai-chen · 2023-01-24T17:50:43Z

This is an excellent direction +1

@YANG-DB I've merged this PR right before your comments. Feel free to comment on #1219 if any idea. I will do it in the future PRs. Thanks!

Yury-Fridlyand · 2023-01-24T03:52:33Z

+* *Observability :* The ability to understand whats happening inside your business/application using logs, traces, metrics and other data emitted from the application.
+
+
+## 3.Tenets


Suggested change
## 3.Tenets
## 3.Tenets

Yury-Fridlyand · 2023-01-24T03:52:40Z

+* Changes to PPL Query Language grammar should be simple and easy to onboard.
+* Component design should be extensible for supporting new data stores.
+
+## 4.Out of Scope for the Design.


Suggested change
## 4.Out of Scope for the Design.
## 4.Out of Scope for the Design

Yury-Fridlyand · 2023-01-24T03:54:48Z

+
+
+


Suggested change

Yury-Fridlyand · 2023-01-24T03:54:55Z

+
+
+
+### 6.1 Data source representation.


Suggested change
### 6.1 Data source representation.
### 6.1 Data source representation

Yury-Fridlyand · 2023-01-24T03:55:09Z

+* *Catalog :* we can describe a catalog as the union of a connector and the underlying instance of the datasource being referred. This gives us flexbility to refer different instances of a similar datastore with the same connector i.e. we can refer data from multiple prometheus instances using prometheus connector. The name for each catalog should be different.
+
+Example Prometheus Catalog Definition
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:50:32Z

+
+* JSON Format


Suggested change
* JSON Format
JSON Format

Yury-Fridlyand · 2023-01-24T17:50:58Z

+[
+{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
+{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
+{"account_number": ExprIntegerValue(3)}
+]


Suggested change
[
{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
{"account_number": ExprIntegerValue(3)}
]
```json
[
{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
{"account_number": ExprIntegerValue(3)}
]
``

Yury-Fridlyand · 2023-01-24T17:51:13Z

+##### Include NULL and MISSING value in the QueryResult (Issue 1, 2)
+The SELECT operator will be translated to PhysicalOpeartor with a list of expression to resolve ExprValue from input data. With the above example, when handling NULL and MISSING value, the expected output data should be as follows.
+
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:53:02Z

+
+An additional list of Schema is also required to when protocol is JDBC.
+
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:58:07Z

+1. Each data source adds special logical operator and explode the optimizer rule space. For example, Prometheus also has `PrometheusLogicalMetricAgg` and `PrometheusLogicalMetricScan` accordingly. They have the exactly same pattern to match query plan tree as OpenSearch.
+2. A bigger problem is the difficulty of transforming from logical to physical when there are 2 `Table`s in query plan. Because only 1 of them has the chance to do the `implement()`. This is a blocker for supporting `INSERT ... SELECT ...` statement or JOIN query. See code below.
+
+```


Suggested change
```
```java

dai-chen · 2023-01-24T19:25:06Z

This is not back port to other branch intentionally. Because I think we may need single dedicated branch for Github pages in future.

dai-chen · 2022-12-20T21:33:38Z

Signed-off-by: Chen Dai daichen@amazon.com

Description

This PR is the first one aimed to improve our dev documentation. Currently there are only a few docs in docs/dev/ folder. Actually there are many valuable info in our old docs or PR description. As initial effort, this PR added an index page reorganize existing docs to categories. Meanwhile I extracted a few PR description for main feature to new docs. Will upload more docs in separate PR.

Please see ~~https://github.com/dai-chen/sql-1/blob/refactor-dev-docs-rebased/docs/dev/index.md~~ -> https://github.com/opensearch-project/sql/blob/main/docs/dev/index.md

Issues Resolved

#1219

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov-commenter · 2022-12-23T22:16:08Z

Codecov Report

Merging #1200 (c7e306f) into main (1b58f7d) will increase coverage by 0.03%.
The diff coverage is n/a.

@@ Coverage Diff @@## main #1200 +/- ##
============================================
+ Coverage 98.32% 98.35% +0.03% - Complexity 3550 3609 +59 
============================================
Files 344 344 Lines 8762 8946 +184 Branches 554 569 +15 ============================================
+ Hits 8615 8799 +184 
Misses 142 142 Partials 5 5

Flag	Coverage Δ
sql-engine	`98.35% <ø> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...c/main/java/org/opensearch/sql/expression/DSL.java	`100.00% <0.00%> (ø)`
...n/java/org/opensearch/sql/utils/DateTimeUtils.java	`100.00% <0.00%> (ø)`
...a/org/opensearch/sql/data/model/ExprDateValue.java	`100.00% <0.00%> (ø)`
...a/org/opensearch/sql/data/model/ExprTimeValue.java	`100.00% <0.00%> (ø)`
.../org/opensearch/sql/data/model/ExprValueUtils.java	`100.00% <0.00%> (ø)`
...g/opensearch/sql/data/model/ExprDatetimeValue.java	`100.00% <0.00%> (ø)`
...pensearch/sql/sql/parser/AstExpressionBuilder.java	`100.00% <0.00%> (ø)`
...pensearch/sql/expression/function/FunctionDSL.java	`100.00% <0.00%> (ø)`
...arch/sql/expression/datetime/DateTimeFunction.java	`100.00% <0.00%> (ø)`
...h/sql/expression/function/BuiltinFunctionName.java	`100.00% <0.00%> (ø)`
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Chen Dai <daichen@amazon.com>

YANG-DB

This is an excellent direction +1

YANG-DB · 2023-01-24T17:37:30Z

+ **CLI**
+ **JDBC Driver**
+ **ODBC Driver**
+ **Query Workbench**


Isn’t this a UI feature ?
can we also have a UI section associated to the SQL ?

yes, I was planning to add some brief intro or link to their repo. Added here just to give users a complete picture of our eco system.

YANG-DB · 2023-01-24T17:39:14Z

+
+# OpenSearch SQL/PPL Engine Development Manual
+
+## Introduction


can we also have tutorial section for playground like exploring

sure, I'm still thinking what's the best way to do this. I have a sub task here for adding tutorial / quick start section: #1219.
Many other projects provide sample data and executable script/jar. User can play around with sample data files via CLI and their file connector. Because our CLI is written in Python, probably we can put some quick start example for playground website.

dai-chen · 2023-01-24T17:50:43Z

This is an excellent direction +1

@YANG-DB I've merged this PR right before your comments. Feel free to comment on #1219 if any idea. I will do it in the future PRs. Thanks!

Yury-Fridlyand · 2023-01-24T03:52:33Z

+* *Observability :* The ability to understand whats happening inside your business/application using logs, traces, metrics and other data emitted from the application.
+
+
+## 3.Tenets


Suggested change
## 3.Tenets
## 3.Tenets

Yury-Fridlyand · 2023-01-24T03:52:40Z

+* Changes to PPL Query Language grammar should be simple and easy to onboard.
+* Component design should be extensible for supporting new data stores.
+
+## 4.Out of Scope for the Design.


Suggested change
## 4.Out of Scope for the Design.
## 4.Out of Scope for the Design

Yury-Fridlyand · 2023-01-24T03:54:48Z

+
+
+


Suggested change

Yury-Fridlyand · 2023-01-24T03:54:55Z

+
+
+
+### 6.1 Data source representation.


Suggested change
### 6.1 Data source representation.
### 6.1 Data source representation

Yury-Fridlyand · 2023-01-24T03:55:09Z

+* *Catalog :* we can describe a catalog as the union of a connector and the underlying instance of the datasource being referred. This gives us flexbility to refer different instances of a similar datastore with the same connector i.e. we can refer data from multiple prometheus instances using prometheus connector. The name for each catalog should be different.
+
+Example Prometheus Catalog Definition
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:50:32Z

+
+* JSON Format


Suggested change
* JSON Format
JSON Format

Yury-Fridlyand · 2023-01-24T17:50:58Z

+[
+{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
+{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
+{"account_number": ExprIntegerValue(3)}
+]


Suggested change
[
{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
{"account_number": ExprIntegerValue(3)}
]
```json
[
{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
{"account_number": ExprIntegerValue(3)}
]
``

Yury-Fridlyand · 2023-01-24T17:51:13Z

+##### Include NULL and MISSING value in the QueryResult (Issue 1, 2)
+The SELECT operator will be translated to PhysicalOpeartor with a list of expression to resolve ExprValue from input data. With the above example, when handling NULL and MISSING value, the expected output data should be as follows.
+
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:53:02Z

+
+An additional list of Schema is also required to when protocol is JDBC.
+
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:58:07Z

+1. Each data source adds special logical operator and explode the optimizer rule space. For example, Prometheus also has `PrometheusLogicalMetricAgg` and `PrometheusLogicalMetricScan` accordingly. They have the exactly same pattern to match query plan tree as OpenSearch.
+2. A bigger problem is the difficulty of transforming from logical to physical when there are 2 `Table`s in query plan. Because only 1 of them has the chance to do the `implement()`. This is a blocker for supporting `INSERT ... SELECT ...` statement or JOIN query. See code below.
+
+```


Suggested change
```
```java

dai-chen · 2023-01-24T19:25:06Z

This is not back port to other branch intentionally. Because I think we may need single dedicated branch for Github pages in future.

dai-chen · 2022-12-20T21:33:38Z

Signed-off-by: Chen Dai daichen@amazon.com

Description

This PR is the first one aimed to improve our dev documentation. Currently there are only a few docs in docs/dev/ folder. Actually there are many valuable info in our old docs or PR description. As initial effort, this PR added an index page reorganize existing docs to categories. Meanwhile I extracted a few PR description for main feature to new docs. Will upload more docs in separate PR.

Please see ~~https://github.com/dai-chen/sql-1/blob/refactor-dev-docs-rebased/docs/dev/index.md~~ -> https://github.com/opensearch-project/sql/blob/main/docs/dev/index.md

Issues Resolved

#1219

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov-commenter · 2022-12-23T22:16:08Z

Codecov Report

Merging #1200 (c7e306f) into main (1b58f7d) will increase coverage by 0.03%.
The diff coverage is n/a.

@@ Coverage Diff @@## main #1200 +/- ##
============================================
+ Coverage 98.32% 98.35% +0.03% - Complexity 3550 3609 +59 
============================================
Files 344 344 Lines 8762 8946 +184 Branches 554 569 +15 ============================================
+ Hits 8615 8799 +184 
Misses 142 142 Partials 5 5

Flag	Coverage Δ
sql-engine	`98.35% <ø> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...c/main/java/org/opensearch/sql/expression/DSL.java	`100.00% <0.00%> (ø)`
...n/java/org/opensearch/sql/utils/DateTimeUtils.java	`100.00% <0.00%> (ø)`
...a/org/opensearch/sql/data/model/ExprDateValue.java	`100.00% <0.00%> (ø)`
...a/org/opensearch/sql/data/model/ExprTimeValue.java	`100.00% <0.00%> (ø)`
.../org/opensearch/sql/data/model/ExprValueUtils.java	`100.00% <0.00%> (ø)`
...g/opensearch/sql/data/model/ExprDatetimeValue.java	`100.00% <0.00%> (ø)`
...pensearch/sql/sql/parser/AstExpressionBuilder.java	`100.00% <0.00%> (ø)`
...pensearch/sql/expression/function/FunctionDSL.java	`100.00% <0.00%> (ø)`
...arch/sql/expression/datetime/DateTimeFunction.java	`100.00% <0.00%> (ø)`
...h/sql/expression/function/BuiltinFunctionName.java	`100.00% <0.00%> (ø)`
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Chen Dai <daichen@amazon.com>

YANG-DB

This is an excellent direction +1

YANG-DB · 2023-01-24T17:37:30Z

+ **CLI**
+ **JDBC Driver**
+ **ODBC Driver**
+ **Query Workbench**


Isn’t this a UI feature ?
can we also have a UI section associated to the SQL ?

yes, I was planning to add some brief intro or link to their repo. Added here just to give users a complete picture of our eco system.

YANG-DB · 2023-01-24T17:39:14Z

+
+# OpenSearch SQL/PPL Engine Development Manual
+
+## Introduction


can we also have tutorial section for playground like exploring

sure, I'm still thinking what's the best way to do this. I have a sub task here for adding tutorial / quick start section: #1219.
Many other projects provide sample data and executable script/jar. User can play around with sample data files via CLI and their file connector. Because our CLI is written in Python, probably we can put some quick start example for playground website.

dai-chen · 2023-01-24T17:50:43Z

This is an excellent direction +1

@YANG-DB I've merged this PR right before your comments. Feel free to comment on #1219 if any idea. I will do it in the future PRs. Thanks!

Yury-Fridlyand · 2023-01-24T03:52:33Z

+* *Observability :* The ability to understand whats happening inside your business/application using logs, traces, metrics and other data emitted from the application.
+
+
+## 3.Tenets


Suggested change
## 3.Tenets
## 3.Tenets

Yury-Fridlyand · 2023-01-24T03:52:40Z

+* Changes to PPL Query Language grammar should be simple and easy to onboard.
+* Component design should be extensible for supporting new data stores.
+
+## 4.Out of Scope for the Design.


Suggested change
## 4.Out of Scope for the Design.
## 4.Out of Scope for the Design

Yury-Fridlyand · 2023-01-24T03:54:48Z

+
+
+


Suggested change

Yury-Fridlyand · 2023-01-24T03:54:55Z

+
+
+
+### 6.1 Data source representation.


Suggested change
### 6.1 Data source representation.
### 6.1 Data source representation

Yury-Fridlyand · 2023-01-24T03:55:09Z

+* *Catalog :* we can describe a catalog as the union of a connector and the underlying instance of the datasource being referred. This gives us flexbility to refer different instances of a similar datastore with the same connector i.e. we can refer data from multiple prometheus instances using prometheus connector. The name for each catalog should be different.
+
+Example Prometheus Catalog Definition
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:50:32Z

+
+* JSON Format


Suggested change
* JSON Format
JSON Format

Yury-Fridlyand · 2023-01-24T17:50:58Z

+[
+{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
+{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
+{"account_number": ExprIntegerValue(3)}
+]


Suggested change
[
{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
{"account_number": ExprIntegerValue(3)}
]
```json
[
{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
{"account_number": ExprIntegerValue(3)}
]
``

Yury-Fridlyand · 2023-01-24T17:51:13Z

+##### Include NULL and MISSING value in the QueryResult (Issue 1, 2)
+The SELECT operator will be translated to PhysicalOpeartor with a list of expression to resolve ExprValue from input data. With the above example, when handling NULL and MISSING value, the expected output data should be as follows.
+
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:53:02Z

+
+An additional list of Schema is also required to when protocol is JDBC.
+
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:58:07Z

+1. Each data source adds special logical operator and explode the optimizer rule space. For example, Prometheus also has `PrometheusLogicalMetricAgg` and `PrometheusLogicalMetricScan` accordingly. They have the exactly same pattern to match query plan tree as OpenSearch.
+2. A bigger problem is the difficulty of transforming from logical to physical when there are 2 `Table`s in query plan. Because only 1 of them has the chance to do the `implement()`. This is a blocker for supporting `INSERT ... SELECT ...` statement or JOIN query. See code below.
+
+```


Suggested change
```
```java

dai-chen · 2023-01-24T19:25:06Z

This is not back port to other branch intentionally. Because I think we may need single dedicated branch for Github pages in future.

dai-chen · 2022-12-20T21:33:38Z

Signed-off-by: Chen Dai daichen@amazon.com

Description

This PR is the first one aimed to improve our dev documentation. Currently there are only a few docs in docs/dev/ folder. Actually there are many valuable info in our old docs or PR description. As initial effort, this PR added an index page reorganize existing docs to categories. Meanwhile I extracted a few PR description for main feature to new docs. Will upload more docs in separate PR.

Please see ~~https://github.com/dai-chen/sql-1/blob/refactor-dev-docs-rebased/docs/dev/index.md~~ -> https://github.com/opensearch-project/sql/blob/main/docs/dev/index.md

Issues Resolved

#1219

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov-commenter · 2022-12-23T22:16:08Z

Codecov Report

Merging #1200 (c7e306f) into main (1b58f7d) will increase coverage by 0.03%.
The diff coverage is n/a.

@@ Coverage Diff @@## main #1200 +/- ##
============================================
+ Coverage 98.32% 98.35% +0.03% - Complexity 3550 3609 +59 
============================================
Files 344 344 Lines 8762 8946 +184 Branches 554 569 +15 ============================================
+ Hits 8615 8799 +184 
Misses 142 142 Partials 5 5

Flag	Coverage Δ
sql-engine	`98.35% <ø> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...c/main/java/org/opensearch/sql/expression/DSL.java	`100.00% <0.00%> (ø)`
...n/java/org/opensearch/sql/utils/DateTimeUtils.java	`100.00% <0.00%> (ø)`
...a/org/opensearch/sql/data/model/ExprDateValue.java	`100.00% <0.00%> (ø)`
...a/org/opensearch/sql/data/model/ExprTimeValue.java	`100.00% <0.00%> (ø)`
.../org/opensearch/sql/data/model/ExprValueUtils.java	`100.00% <0.00%> (ø)`
...g/opensearch/sql/data/model/ExprDatetimeValue.java	`100.00% <0.00%> (ø)`
...pensearch/sql/sql/parser/AstExpressionBuilder.java	`100.00% <0.00%> (ø)`
...pensearch/sql/expression/function/FunctionDSL.java	`100.00% <0.00%> (ø)`
...arch/sql/expression/datetime/DateTimeFunction.java	`100.00% <0.00%> (ø)`
...h/sql/expression/function/BuiltinFunctionName.java	`100.00% <0.00%> (ø)`
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Chen Dai <daichen@amazon.com>

YANG-DB

This is an excellent direction +1

YANG-DB · 2023-01-24T17:37:30Z

+ **CLI**
+ **JDBC Driver**
+ **ODBC Driver**
+ **Query Workbench**


Isn’t this a UI feature ?
can we also have a UI section associated to the SQL ?

yes, I was planning to add some brief intro or link to their repo. Added here just to give users a complete picture of our eco system.

YANG-DB · 2023-01-24T17:39:14Z

+
+# OpenSearch SQL/PPL Engine Development Manual
+
+## Introduction


can we also have tutorial section for playground like exploring

sure, I'm still thinking what's the best way to do this. I have a sub task here for adding tutorial / quick start section: #1219.
Many other projects provide sample data and executable script/jar. User can play around with sample data files via CLI and their file connector. Because our CLI is written in Python, probably we can put some quick start example for playground website.

dai-chen · 2023-01-24T17:50:43Z

This is an excellent direction +1

@YANG-DB I've merged this PR right before your comments. Feel free to comment on #1219 if any idea. I will do it in the future PRs. Thanks!

Yury-Fridlyand · 2023-01-24T03:52:33Z

+* *Observability :* The ability to understand whats happening inside your business/application using logs, traces, metrics and other data emitted from the application.
+
+
+## 3.Tenets


Suggested change
## 3.Tenets
## 3.Tenets

Yury-Fridlyand · 2023-01-24T03:52:40Z

+* Changes to PPL Query Language grammar should be simple and easy to onboard.
+* Component design should be extensible for supporting new data stores.
+
+## 4.Out of Scope for the Design.


Suggested change
## 4.Out of Scope for the Design.
## 4.Out of Scope for the Design

Yury-Fridlyand · 2023-01-24T03:54:48Z

+
+
+


Suggested change

Yury-Fridlyand · 2023-01-24T03:54:55Z

+
+
+
+### 6.1 Data source representation.


Suggested change
### 6.1 Data source representation.
### 6.1 Data source representation

Yury-Fridlyand · 2023-01-24T03:55:09Z

+* *Catalog :* we can describe a catalog as the union of a connector and the underlying instance of the datasource being referred. This gives us flexbility to refer different instances of a similar datastore with the same connector i.e. we can refer data from multiple prometheus instances using prometheus connector. The name for each catalog should be different.
+
+Example Prometheus Catalog Definition
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:50:32Z

+
+* JSON Format


Suggested change
* JSON Format
JSON Format

Yury-Fridlyand · 2023-01-24T17:50:58Z

+[
+{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
+{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
+{"account_number": ExprIntegerValue(3)}
+]


Suggested change
[
{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
{"account_number": ExprIntegerValue(3)}
]
```json
[
{"age": ExprIntegerValue(1), "account_number": ExprIntegerValue(1)},
{"age": ExprNullValue(), "account_number": ExprIntegerValue(2)},
{"account_number": ExprIntegerValue(3)}
]
``

Yury-Fridlyand · 2023-01-24T17:51:13Z

+##### Include NULL and MISSING value in the QueryResult (Issue 1, 2)
+The SELECT operator will be translated to PhysicalOpeartor with a list of expression to resolve ExprValue from input data. With the above example, when handling NULL and MISSING value, the expected output data should be as follows.
+
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:53:02Z

+
+An additional list of Schema is also required to when protocol is JDBC.
+
+```


Suggested change
```
```json

Yury-Fridlyand · 2023-01-24T17:58:07Z

+1. Each data source adds special logical operator and explode the optimizer rule space. For example, Prometheus also has `PrometheusLogicalMetricAgg` and `PrometheusLogicalMetricScan` accordingly. They have the exactly same pattern to match query plan tree as OpenSearch.
+2. A bigger problem is the difficulty of transforming from logical to physical when there are 2 `Table`s in query plan. Because only 1 of them has the chance to do the `implement()`. This is a blocker for supporting `INSERT ... SELECT ...` statement or JOIN query. See code below.
+
+```


Suggested change
```
```java

dai-chen · 2023-01-24T19:25:06Z

This is not back port to other branch intentionally. Because I think we may need single dedicated branch for Github pages in future.

dai-chen added the documentation Improvements or additions to documentation label Dec 20, 2022

dai-chen self-assigned this Dec 20, 2022

dai-chen force-pushed the refactor-dev-docs-rebased branch from e31b2c0 to 3ad530dCompare January 3, 2023 19:50

dai-chen changed the title ~~Reorganize development documentation~~Add table of content page and reorganize dev docsJan 3, 2023

dai-chen mentioned this pull request Jan 3, 2023
[FEATURE] Improve development documentation #1219
Open
5 tasks

Add new dev doc index page
7b038c4
Signed-off-by: Chen Dai <daichen@amazon.com>

dai-chen force-pushed the refactor-dev-docs-rebased branch from 3ad530d to 7b038c4Compare January 13, 2023 22:35

dai-chen added 3 commits January 13, 2023 14:43

Rename all docs in consistent naming convention
0ecab8e
Signed-off-by: Chen Dai <daichen@amazon.com>

Convert feature PR to new doc
bf835de
Signed-off-by: Chen Dai <daichen@amazon.com>

Update index doc
7ce4ea2
Signed-off-by: Chen Dai <daichen@amazon.com>

dai-chen changed the title ~~Add table of content page and reorganize dev docs~~Add index page and reorganize dev docsJan 13, 2023

Add link in root readme
c7e306f
Signed-off-by: Chen Dai <daichen@amazon.com>

dai-chen marked this pull request as ready for review January 13, 2023 23:47

dai-chen requested a review from a team as a code owner January 13, 2023 23:47

dai-chen changed the title ~~Add index page and reorganize dev docs~~Reorganize development docsJan 14, 2023

penghuo approved these changes Jan 17, 2023
View reviewed changes

dai-chen requested review from anirudha and joshuali925 January 20, 2023 18:36

anirudha approved these changes Jan 24, 2023
View reviewed changes

dai-chen added backport 2.x and removed backport 2.x labels Jan 24, 2023

dai-chen merged commit 1ed9701 into opensearch-project:mainJan 24, 2023

dai-chen deleted the refactor-dev-docs-rebased branch January 24, 2023 17:38

YANG-DB reviewed Jan 24, 2023
View reviewed changes

Yury-Fridlyand reviewed Jan 24, 2023
View reviewed changes