Convert DP+ to a CKAN extension

[jqnatividad]

DP+ was forked from DP which uses ckanserviceprovider.

ckanserviceprovider was meant to be a general library for CKAN for making web services, and was originally created in 2013.

Right now, DP and DP+ are really the only users of the library, and in the 10 years since, a lot has changed.

DP+ 0.x will continue to use ckanserviceprovider so older CKAN installations using DataPusher can still use it.

However, especially now that CKAN v2.9 uses Python 3.8+ which DP+ requires to call qsv using subprocess options only available in Python 3.7+, v1.x of DP+ will work as a CKAN extension.

This has the following benefits:

• easier to maintain as it just works as a regular extension
• no need to maintain a separate web service running in its own virtualenv
• we can drop the ckanserviceprovider dependency
• we can use the ckan_default database to store DP+ job history instead of creating our own DP+ database
• we can add additional UI elements beyond the existing Datastore tab. Here are some examples of what a DP+ extension can do:
  • Have the ability to set resource-specific DP+ job settings on the UI
  • Have the ability to see how the resource is stored in the Datastore (with datastore_info API)
  • Have a more advanced Data Dictionary interface that goes beyond setting label/description and overriding types

This is already WIP in the https://github.com/dathere/datapusher-plus/tree/dev-v1.0 branch and we hope to the make the first release summer 2023.

EDIT:

This is completed

Now main branch is the same as it was dev-v1.0 and all the contributions, bug fixes should be branched of main

In some time master branch will be renamed into legacy-datapusher-plus in case someone still uses the service version

[jqnatividad]

DP+ 1.x should also be more interactive - i.e. users have the ability to configure the DP+ job on a per resource basis.

99% of the time, the default configuration should work fine. But when the user wants to tweak DP+ configuration for that particular job (e.g ask for PII screening using a particular PII screening configuration, add summary stats, check for dupes, etc.), there should be some DP+ knobs exposed in the Datastore tab to change DP+ settings just for that resource.

[sagargg]

Hi @jqnatividad Any idea, when it will be ready for release ??

[tino097]

@sagargg its almost done, any testing on the dev-v1.0 branch would be helpfull

[jhbruhn]

What is the current state of this migration? I am considering setting up datapusher-plus, but would like to avoid to set up an external service if future updates require it to be an ckanext instead. Could I already use the dev-1.0 branch for this, and if so, how?

[tino097]

@jhbruhn dev-1.0 can be used to set the DP+ as extension and you are more than welcome to give it a try

To use it, just checkout to that branch and follow same steps as any other extension for installation

[jhbruhn]

Thanks. I installed the extension in my ckan 2.10 test instance and included it as the "datapusher" plugin (according to this

datapusher-plus/setup.py

Line 85 in 1cfba1c

datapusher=ckanext.datapusher_plus.plugin:DatapusherPlusPlugin

). I then defined CKAN_DATAPUSHER_URL as ckan would not start if not defined to the value I was using with the original datapusher, but stopped that service.

But when I upload a file and try to push it to the datastore (did happen automatically as well), I get this error in the Web Interface: Could not connect to DataPusher. without a more specific error in the ckan logs.

[tino097]

You should start a job worker
ckan jobs worker

[jhbruhn]

I did (now), but the result is still the same. Could the original, ckan internal datapusher be interfering with datapusher-plus here? Should I open another issue?