Dynamo to S3 Sample DAG and Docs by ferruzzi · Pull Request #21920 · apache/airflow

ferruzzi · 2022-03-01T22:16:38Z

Adds a sample DAG and docs for the existing Dynamo to S3 Transfer operator.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

ferruzzi · 2022-03-03T00:53:23Z

@eladkal - Another short one in the DAG/Docs improvement project if you get time, since you reviewed the previous ones.

ferruzzi · 2022-03-03T18:55:23Z

Failing image build; rebased

eladkal · 2022-03-06T10:31:39Z

+ backup_db = DynamoDBToS3Operator(
+ task_id='backup_db',
+ dynamodb_table_name=TABLE_NAME,
+ s3_bucket_name=BUCKET_NAME,
+ # Max output file size in bytes. If the Table is too large, multiple files will be created.
+ file_size=1000,


Can you please extend the example/docs to include also the segment explanation?
airflow/airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py
Lines 65 to 87 in 602abe8
To parallelize the replication, users can create multiple tasks of DynamoDBToS3Operator.
For instance to replicate with parallelism of 2, create two tasks like:
.. code-block:: python
op1 = DynamoDBToS3Operator(
task_id="replicator-1",
dynamodb_table_name="hello",
dynamodb_scan_kwargs={
"TotalSegments": 2,
"Segment": 0,
},
...,
)
op2 = DynamoDBToS3Operator(
task_id="replicator-2",
dynamodb_table_name="hello",
dynamodb_scan_kwargs={
"TotalSegments": 2,
"Segment": 1,
},
...,
(and remove the segment example from the comments)

These changed should be in b0278a1

github-actions · 2022-03-08T10:53:17Z

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

ferruzzi · 2022-03-01T22:16:38Z

Adds a sample DAG and docs for the existing Dynamo to S3 Transfer operator.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

ferruzzi · 2022-03-03T00:53:23Z

@eladkal - Another short one in the DAG/Docs improvement project if you get time, since you reviewed the previous ones.

ferruzzi · 2022-03-03T18:55:23Z

Failing image build; rebased

eladkal · 2022-03-06T10:31:39Z

+ backup_db = DynamoDBToS3Operator(
+ task_id='backup_db',
+ dynamodb_table_name=TABLE_NAME,
+ s3_bucket_name=BUCKET_NAME,
+ # Max output file size in bytes. If the Table is too large, multiple files will be created.
+ file_size=1000,


Can you please extend the example/docs to include also the segment explanation?
airflow/airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py
Lines 65 to 87 in 602abe8
To parallelize the replication, users can create multiple tasks of DynamoDBToS3Operator.
For instance to replicate with parallelism of 2, create two tasks like:
.. code-block:: python
op1 = DynamoDBToS3Operator(
task_id="replicator-1",
dynamodb_table_name="hello",
dynamodb_scan_kwargs={
"TotalSegments": 2,
"Segment": 0,
},
...,
)
op2 = DynamoDBToS3Operator(
task_id="replicator-2",
dynamodb_table_name="hello",
dynamodb_scan_kwargs={
"TotalSegments": 2,
"Segment": 1,
},
...,
(and remove the segment example from the comments)

These changed should be in b0278a1

github-actions · 2022-03-08T10:53:17Z

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

ferruzzi · 2022-03-01T22:16:38Z

Adds a sample DAG and docs for the existing Dynamo to S3 Transfer operator.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

ferruzzi · 2022-03-03T00:53:23Z

@eladkal - Another short one in the DAG/Docs improvement project if you get time, since you reviewed the previous ones.

ferruzzi · 2022-03-03T18:55:23Z

Failing image build; rebased

eladkal · 2022-03-06T10:31:39Z

+ backup_db = DynamoDBToS3Operator(
+ task_id='backup_db',
+ dynamodb_table_name=TABLE_NAME,
+ s3_bucket_name=BUCKET_NAME,
+ # Max output file size in bytes. If the Table is too large, multiple files will be created.
+ file_size=1000,


Can you please extend the example/docs to include also the segment explanation?
airflow/airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py
Lines 65 to 87 in 602abe8
To parallelize the replication, users can create multiple tasks of DynamoDBToS3Operator.
For instance to replicate with parallelism of 2, create two tasks like:
.. code-block:: python
op1 = DynamoDBToS3Operator(
task_id="replicator-1",
dynamodb_table_name="hello",
dynamodb_scan_kwargs={
"TotalSegments": 2,
"Segment": 0,
},
...,
)
op2 = DynamoDBToS3Operator(
task_id="replicator-2",
dynamodb_table_name="hello",
dynamodb_scan_kwargs={
"TotalSegments": 2,
"Segment": 1,
},
...,
(and remove the segment example from the comments)

These changed should be in b0278a1

github-actions · 2022-03-08T10:53:17Z

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

ferruzzi · 2022-03-01T22:16:38Z

Adds a sample DAG and docs for the existing Dynamo to S3 Transfer operator.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

ferruzzi · 2022-03-03T00:53:23Z

@eladkal - Another short one in the DAG/Docs improvement project if you get time, since you reviewed the previous ones.

ferruzzi · 2022-03-03T18:55:23Z

Failing image build; rebased

eladkal · 2022-03-06T10:31:39Z

+ backup_db = DynamoDBToS3Operator(
+ task_id='backup_db',
+ dynamodb_table_name=TABLE_NAME,
+ s3_bucket_name=BUCKET_NAME,
+ # Max output file size in bytes. If the Table is too large, multiple files will be created.
+ file_size=1000,


Can you please extend the example/docs to include also the segment explanation?
airflow/airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py
Lines 65 to 87 in 602abe8
To parallelize the replication, users can create multiple tasks of DynamoDBToS3Operator.
For instance to replicate with parallelism of 2, create two tasks like:
.. code-block:: python
op1 = DynamoDBToS3Operator(
task_id="replicator-1",
dynamodb_table_name="hello",
dynamodb_scan_kwargs={
"TotalSegments": 2,
"Segment": 0,
},
...,
)
op2 = DynamoDBToS3Operator(
task_id="replicator-2",
dynamodb_table_name="hello",
dynamodb_scan_kwargs={
"TotalSegments": 2,
"Segment": 1,
},
...,
(and remove the segment example from the comments)

These changed should be in b0278a1

github-actions · 2022-03-08T10:53:17Z

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

ferruzzi · 2022-03-01T22:16:38Z

Adds a sample DAG and docs for the existing Dynamo to S3 Transfer operator.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

ferruzzi · 2022-03-03T00:53:23Z

@eladkal - Another short one in the DAG/Docs improvement project if you get time, since you reviewed the previous ones.

ferruzzi · 2022-03-03T18:55:23Z

Failing image build; rebased

eladkal · 2022-03-06T10:31:39Z

+ backup_db = DynamoDBToS3Operator(
+ task_id='backup_db',
+ dynamodb_table_name=TABLE_NAME,
+ s3_bucket_name=BUCKET_NAME,
+ # Max output file size in bytes. If the Table is too large, multiple files will be created.
+ file_size=1000,


Can you please extend the example/docs to include also the segment explanation?
airflow/airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py
Lines 65 to 87 in 602abe8
To parallelize the replication, users can create multiple tasks of DynamoDBToS3Operator.
For instance to replicate with parallelism of 2, create two tasks like:
.. code-block:: python
op1 = DynamoDBToS3Operator(
task_id="replicator-1",
dynamodb_table_name="hello",
dynamodb_scan_kwargs={
"TotalSegments": 2,
"Segment": 0,
},
...,
)
op2 = DynamoDBToS3Operator(
task_id="replicator-2",
dynamodb_table_name="hello",
dynamodb_scan_kwargs={
"TotalSegments": 2,
"Segment": 1,
},
...,
(and remove the segment example from the comments)

These changed should be in b0278a1

github-actions · 2022-03-08T10:53:17Z

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

ferruzzi · 2022-03-01T22:16:38Z

Adds a sample DAG and docs for the existing Dynamo to S3 Transfer operator.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

ferruzzi · 2022-03-03T00:53:23Z

@eladkal - Another short one in the DAG/Docs improvement project if you get time, since you reviewed the previous ones.

ferruzzi · 2022-03-03T18:55:23Z

Failing image build; rebased

eladkal · 2022-03-06T10:31:39Z

+ backup_db = DynamoDBToS3Operator(
+ task_id='backup_db',
+ dynamodb_table_name=TABLE_NAME,
+ s3_bucket_name=BUCKET_NAME,
+ # Max output file size in bytes. If the Table is too large, multiple files will be created.
+ file_size=1000,


Can you please extend the example/docs to include also the segment explanation?
airflow/airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py
Lines 65 to 87 in 602abe8
To parallelize the replication, users can create multiple tasks of DynamoDBToS3Operator.
For instance to replicate with parallelism of 2, create two tasks like:
.. code-block:: python
op1 = DynamoDBToS3Operator(
task_id="replicator-1",
dynamodb_table_name="hello",
dynamodb_scan_kwargs={
"TotalSegments": 2,
"Segment": 0,
},
...,
)
op2 = DynamoDBToS3Operator(
task_id="replicator-2",
dynamodb_table_name="hello",
dynamodb_scan_kwargs={
"TotalSegments": 2,
"Segment": 1,
},
...,
(and remove the segment example from the comments)

These changed should be in b0278a1

github-actions · 2022-03-08T10:53:17Z

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

ferruzzi · 2022-03-01T22:16:38Z

Adds a sample DAG and docs for the existing Dynamo to S3 Transfer operator.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

ferruzzi · 2022-03-03T00:53:23Z

@eladkal - Another short one in the DAG/Docs improvement project if you get time, since you reviewed the previous ones.

ferruzzi · 2022-03-03T18:55:23Z

Failing image build; rebased

eladkal · 2022-03-06T10:31:39Z

+ backup_db = DynamoDBToS3Operator(
+ task_id='backup_db',
+ dynamodb_table_name=TABLE_NAME,
+ s3_bucket_name=BUCKET_NAME,
+ # Max output file size in bytes. If the Table is too large, multiple files will be created.
+ file_size=1000,


Can you please extend the example/docs to include also the segment explanation?
airflow/airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py
Lines 65 to 87 in 602abe8
To parallelize the replication, users can create multiple tasks of DynamoDBToS3Operator.
For instance to replicate with parallelism of 2, create two tasks like:
.. code-block:: python
op1 = DynamoDBToS3Operator(
task_id="replicator-1",
dynamodb_table_name="hello",
dynamodb_scan_kwargs={
"TotalSegments": 2,
"Segment": 0,
},
...,
)
op2 = DynamoDBToS3Operator(
task_id="replicator-2",
dynamodb_table_name="hello",
dynamodb_scan_kwargs={
"TotalSegments": 2,
"Segment": 1,
},
...,
(and remove the segment example from the comments)

These changed should be in b0278a1

github-actions · 2022-03-08T10:53:17Z

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

ferruzzi · 2022-03-01T22:16:38Z

Adds a sample DAG and docs for the existing Dynamo to S3 Transfer operator.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

ferruzzi · 2022-03-03T00:53:23Z

@eladkal - Another short one in the DAG/Docs improvement project if you get time, since you reviewed the previous ones.

ferruzzi · 2022-03-03T18:55:23Z

Failing image build; rebased

eladkal · 2022-03-06T10:31:39Z

+ backup_db = DynamoDBToS3Operator(
+ task_id='backup_db',
+ dynamodb_table_name=TABLE_NAME,
+ s3_bucket_name=BUCKET_NAME,
+ # Max output file size in bytes. If the Table is too large, multiple files will be created.
+ file_size=1000,


Can you please extend the example/docs to include also the segment explanation?
airflow/airflow/providers/amazon/aws/transfers/dynamodb_to_s3.py
Lines 65 to 87 in 602abe8
To parallelize the replication, users can create multiple tasks of DynamoDBToS3Operator.
For instance to replicate with parallelism of 2, create two tasks like:
.. code-block:: python
op1 = DynamoDBToS3Operator(
task_id="replicator-1",
dynamodb_table_name="hello",
dynamodb_scan_kwargs={
"TotalSegments": 2,
"Segment": 0,
},
...,
)
op2 = DynamoDBToS3Operator(
task_id="replicator-2",
dynamodb_table_name="hello",
dynamodb_scan_kwargs={
"TotalSegments": 2,
"Segment": 1,
},
...,
(and remove the segment example from the comments)

These changed should be in b0278a1

github-actions · 2022-03-08T10:53:17Z

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

ferruzzi requested a review from mik-laj as a code owner March 1, 2022 22:16

boring-cyborgBot added area:providers kind:documentation provider:amazon AWS/Amazon - related issues labels Mar 1, 2022

ferruzzi closed this Mar 3, 2022

ferruzzi reopened this Mar 3, 2022

ferruzzi added 3 commits March 3, 2022 10:54

Dynamo to S3 Sample DAG and Docs
abe8086

Add Prerequisites section to the doc file
10edeb7

Updated provider.yaml
0ac66c6

ferruzzi force-pushed the ferruzzi/docs-update/dynamo-to-s3 branch from 26e5bbf to 0ac66c6Compare March 3, 2022 18:54

o-nikolas mentioned this pull request Mar 5, 2022
Add docs and sample dags for AWS Batch Operator/Sensor #22010
Merged

eladkal requested changes Mar 6, 2022
View reviewed changes

PR Fixes
b0278a1

eladkal approved these changes Mar 8, 2022
View reviewed changes

github-actionsBot added the okay to merge It's ok to merge this PR as it does not require more tests label Mar 8, 2022

potiuk approved these changes Mar 8, 2022
View reviewed changes

potiuk merged commit 374354d into apache:mainMar 8, 2022

ferruzzi deleted the ferruzzi/docs-update/dynamo-to-s3 branch March 8, 2022 18:10

This was referenced Mar 8, 2022
EMR on EKS Sample DAG and Docs Update #22095
Merged
EMR Sample DAG and Docs Update #22189
Merged

o-nikolas mentioned this pull request Mar 15, 2022
Add docs and example dag for AWS Glue #22295
Merged

ferruzzi mentioned this pull request Mar 24, 2022
Adds HiveToDynamoDB Transfer Sample DAG and Docs #22517
Merged

This was referenced Apr 26, 2022
Amazon Sagemaker Sample DAG and docs update #23256
Merged
Update the Athena Sample DAG and Docs #23428
Merged

ferruzzi mentioned this pull request May 12, 2022
Update the DMS Sample DAG and Docs #23681
Merged