omit config file requirement by aaronsteers · Pull Request #27 · dataops-tk/tapdance

aaronsteers · 2020-10-20T22:42:36Z

This update removes the requirement to have the tap installed or to have any config.json file setup - assuming:

There's a catalog.json file in the expectation location
The use does not specify '--rescan'

Essentially, in those conditions where we just want to update the plan and we already have the cached catalog metadata, we can skip connecting to the remote source and also skip loading the config file.

Note:

The config file will still be loaded if it exists, but an error will no longer be thrown if the config doesn't exist, or if the tap in not installed.

aaronsteers · 2020-10-20T22:44:57Z

 "REPLICATION_STRATEGY", "INCREMENTAL"
 )
 config.validate_replication_strategy(replication_strategy)
- catalog_dir = config.get_tap_output_dir(tap_name, taps_dir)


Moved these lines up higher so they are able to be used in evaluating the catalog file existance.

aaronsteers · 2020-10-20T22:45:34Z

 taps_dir=taps_dir,
 config_dir=config_dir,
 config_file=config_file,
+ config_file_required=config_file_required,


This allows us to say "not required" if we have enough info to run the plan without rerunning the tap.

ahardisty

looks like an elegant solution tucked into a complex system. Well done.

aaronsteers · 2020-10-20T22:42:36Z

This update removes the requirement to have the tap installed or to have any config.json file setup - assuming:

There's a catalog.json file in the expectation location
The use does not specify '--rescan'

Essentially, in those conditions where we just want to update the plan and we already have the cached catalog metadata, we can skip connecting to the remote source and also skip loading the config file.

Note:

The config file will still be loaded if it exists, but an error will no longer be thrown if the config doesn't exist, or if the tap in not installed.

aaronsteers · 2020-10-20T22:44:57Z

 "REPLICATION_STRATEGY", "INCREMENTAL"
 )
 config.validate_replication_strategy(replication_strategy)
- catalog_dir = config.get_tap_output_dir(tap_name, taps_dir)


Moved these lines up higher so they are able to be used in evaluating the catalog file existance.

aaronsteers · 2020-10-20T22:45:34Z

 taps_dir=taps_dir,
 config_dir=config_dir,
 config_file=config_file,
+ config_file_required=config_file_required,


This allows us to say "not required" if we have enough info to run the plan without rerunning the tap.

ahardisty

looks like an elegant solution tucked into a complex system. Well done.

aaronsteers · 2020-10-20T22:42:36Z

This update removes the requirement to have the tap installed or to have any config.json file setup - assuming:

There's a catalog.json file in the expectation location
The use does not specify '--rescan'

Essentially, in those conditions where we just want to update the plan and we already have the cached catalog metadata, we can skip connecting to the remote source and also skip loading the config file.

Note:

The config file will still be loaded if it exists, but an error will no longer be thrown if the config doesn't exist, or if the tap in not installed.

aaronsteers · 2020-10-20T22:44:57Z

 "REPLICATION_STRATEGY", "INCREMENTAL"
 )
 config.validate_replication_strategy(replication_strategy)
- catalog_dir = config.get_tap_output_dir(tap_name, taps_dir)


Moved these lines up higher so they are able to be used in evaluating the catalog file existance.

aaronsteers · 2020-10-20T22:45:34Z

 taps_dir=taps_dir,
 config_dir=config_dir,
 config_file=config_file,
+ config_file_required=config_file_required,


This allows us to say "not required" if we have enough info to run the plan without rerunning the tap.

ahardisty

looks like an elegant solution tucked into a complex system. Well done.

aaronsteers · 2020-10-20T22:42:36Z

This update removes the requirement to have the tap installed or to have any config.json file setup - assuming:

There's a catalog.json file in the expectation location
The use does not specify '--rescan'

Essentially, in those conditions where we just want to update the plan and we already have the cached catalog metadata, we can skip connecting to the remote source and also skip loading the config file.

Note:

The config file will still be loaded if it exists, but an error will no longer be thrown if the config doesn't exist, or if the tap in not installed.

aaronsteers · 2020-10-20T22:44:57Z

 "REPLICATION_STRATEGY", "INCREMENTAL"
 )
 config.validate_replication_strategy(replication_strategy)
- catalog_dir = config.get_tap_output_dir(tap_name, taps_dir)


Moved these lines up higher so they are able to be used in evaluating the catalog file existance.

aaronsteers · 2020-10-20T22:45:34Z

 taps_dir=taps_dir,
 config_dir=config_dir,
 config_file=config_file,
+ config_file_required=config_file_required,


This allows us to say "not required" if we have enough info to run the plan without rerunning the tap.

ahardisty

looks like an elegant solution tucked into a complex system. Well done.

aaronsteers · 2020-10-20T22:42:36Z

This update removes the requirement to have the tap installed or to have any config.json file setup - assuming:

There's a catalog.json file in the expectation location
The use does not specify '--rescan'

Essentially, in those conditions where we just want to update the plan and we already have the cached catalog metadata, we can skip connecting to the remote source and also skip loading the config file.

Note:

The config file will still be loaded if it exists, but an error will no longer be thrown if the config doesn't exist, or if the tap in not installed.

aaronsteers · 2020-10-20T22:44:57Z

 "REPLICATION_STRATEGY", "INCREMENTAL"
 )
 config.validate_replication_strategy(replication_strategy)
- catalog_dir = config.get_tap_output_dir(tap_name, taps_dir)


Moved these lines up higher so they are able to be used in evaluating the catalog file existance.

aaronsteers · 2020-10-20T22:45:34Z

 taps_dir=taps_dir,
 config_dir=config_dir,
 config_file=config_file,
+ config_file_required=config_file_required,


This allows us to say "not required" if we have enough info to run the plan without rerunning the tap.

ahardisty

looks like an elegant solution tucked into a complex system. Well done.

aaronsteers · 2020-10-20T22:42:36Z

This update removes the requirement to have the tap installed or to have any config.json file setup - assuming:

There's a catalog.json file in the expectation location
The use does not specify '--rescan'

Essentially, in those conditions where we just want to update the plan and we already have the cached catalog metadata, we can skip connecting to the remote source and also skip loading the config file.

Note:

The config file will still be loaded if it exists, but an error will no longer be thrown if the config doesn't exist, or if the tap in not installed.

aaronsteers · 2020-10-20T22:44:57Z

 "REPLICATION_STRATEGY", "INCREMENTAL"
 )
 config.validate_replication_strategy(replication_strategy)
- catalog_dir = config.get_tap_output_dir(tap_name, taps_dir)


Moved these lines up higher so they are able to be used in evaluating the catalog file existance.

aaronsteers · 2020-10-20T22:45:34Z

 taps_dir=taps_dir,
 config_dir=config_dir,
 config_file=config_file,
+ config_file_required=config_file_required,


This allows us to say "not required" if we have enough info to run the plan without rerunning the tap.

ahardisty

looks like an elegant solution tucked into a complex system. Well done.

aaronsteers · 2020-10-20T22:42:36Z

This update removes the requirement to have the tap installed or to have any config.json file setup - assuming:

There's a catalog.json file in the expectation location
The use does not specify '--rescan'

Essentially, in those conditions where we just want to update the plan and we already have the cached catalog metadata, we can skip connecting to the remote source and also skip loading the config file.

Note:

The config file will still be loaded if it exists, but an error will no longer be thrown if the config doesn't exist, or if the tap in not installed.

aaronsteers · 2020-10-20T22:44:57Z

 "REPLICATION_STRATEGY", "INCREMENTAL"
 )
 config.validate_replication_strategy(replication_strategy)
- catalog_dir = config.get_tap_output_dir(tap_name, taps_dir)


Moved these lines up higher so they are able to be used in evaluating the catalog file existance.

aaronsteers · 2020-10-20T22:45:34Z

 taps_dir=taps_dir,
 config_dir=config_dir,
 config_file=config_file,
+ config_file_required=config_file_required,


This allows us to say "not required" if we have enough info to run the plan without rerunning the tap.

ahardisty

looks like an elegant solution tucked into a complex system. Well done.

aaronsteers · 2020-10-20T22:42:36Z

This update removes the requirement to have the tap installed or to have any config.json file setup - assuming:

There's a catalog.json file in the expectation location
The use does not specify '--rescan'

Essentially, in those conditions where we just want to update the plan and we already have the cached catalog metadata, we can skip connecting to the remote source and also skip loading the config file.

Note:

The config file will still be loaded if it exists, but an error will no longer be thrown if the config doesn't exist, or if the tap in not installed.

aaronsteers · 2020-10-20T22:44:57Z

 "REPLICATION_STRATEGY", "INCREMENTAL"
 )
 config.validate_replication_strategy(replication_strategy)
- catalog_dir = config.get_tap_output_dir(tap_name, taps_dir)


Moved these lines up higher so they are able to be used in evaluating the catalog file existance.

aaronsteers · 2020-10-20T22:45:34Z

 taps_dir=taps_dir,
 config_dir=config_dir,
 config_file=config_file,
+ config_file_required=config_file_required,


This allows us to say "not required" if we have enough info to run the plan without rerunning the tap.

ahardisty

looks like an elegant solution tucked into a complex system. Well done.

omit config file requirement
8fa6156

aaronsteers commented Oct 20, 2020
View reviewed changes

ahardisty approved these changes Oct 21, 2020
View reviewed changes

omit config file requirement
8fa6156

aaronsteers commented Oct 20, 2020
View reviewed changes

ahardisty approved these changes Oct 21, 2020
View reviewed changes

omit config file requirement
8fa6156

aaronsteers commented Oct 20, 2020
View reviewed changes

ahardisty approved these changes Oct 21, 2020
View reviewed changes

omit config file requirement
8fa6156

aaronsteers commented Oct 20, 2020
View reviewed changes

ahardisty approved these changes Oct 21, 2020
View reviewed changes

omit config file requirement
8fa6156

aaronsteers commented Oct 20, 2020
View reviewed changes

ahardisty approved these changes Oct 21, 2020
View reviewed changes

omit config file requirement
8fa6156

aaronsteers commented Oct 20, 2020
View reviewed changes

ahardisty approved these changes Oct 21, 2020
View reviewed changes

omit config file requirement
8fa6156

aaronsteers commented Oct 20, 2020
View reviewed changes

ahardisty approved these changes Oct 21, 2020
View reviewed changes

omit config file requirement
8fa6156

aaronsteers commented Oct 20, 2020
View reviewed changes

ahardisty approved these changes Oct 21, 2020
View reviewed changes

Uh oh!

Conversation

aaronsteers commented Oct 20, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahardisty left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

aaronsteers commented Oct 20, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahardisty left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

aaronsteers commented Oct 20, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahardisty left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

aaronsteers commented Oct 20, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahardisty left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

aaronsteers commented Oct 20, 2020

Uh oh!