Skip to content

Setting Up DRUF on CKAN 2.11.3 datapusher 3.1.0a #331

Open
Open
Setting Up DRUF on CKAN 2.11.3 datapusher 3.1.0a#331

Description

@a5dur
a5dur
opened on Jun 18, 2026

Setting Up DRUF on CKAN 2.11.3 — Issues & Workarounds

Notes from setting up the DRUF (Dataset Resource Upload First) workflow with DataPusher+ on a CKAN 2.11.3 instance.
Posted here so other developers hit the same wall less hard.

Environment

Component Version
CKAN 2.11.3 (stable release)
DataPusher+ latest (main / 0e46e10)
Python 3.10
Prefect 3.x (as required by DP+)
ckanext-scheming latest

Issue 1: Prefect Worker Fails to Start — BooleanOptionalAction ImportError

What happened

After installing DataPusher+ and its requirements, attempting to start the Prefect worker immediately crashed with:

ImportError: cannot import name 'BooleanOptionalAction' from 'argparse'

Full traceback:

Traceback (most recent call last):
  File ".../pydantic_settings/env_settings.py", line ..., in ...
    from argparse import BooleanOptionalAction
ImportError: cannot import name 'BooleanOptionalAction' from 'argparse'
  (/path/to/venv/lib/python3.10/site-packages/argparse.py)

Why it happens

pydantic-settings (a Prefect dependency) imports BooleanOptionalAction, which was added to the Python standard library in Python 3.9. On Python 3.10 this should be available without issue.

The problem is that some older pip packages distribute a standalone argparse module and install argparse.py directly into site-packages. Because site-packages is earlier in Python's module search path than the standard library, this legacy file shadows the stdlib argparse, causing the import to fail even on Python 3.10.

You can confirm this is your issue:

python3 -c "import argparse; print(argparse.__file__)"
# If it prints a site-packages path instead of a stdlib path, you have the problem.
# Normal (correct) output looks like: /usr/lib/python3.10/argparse.py
# Problem output looks like: /path/to/venv/lib/python3.10/site-packages/argparse.py

Fix

Remove (or rename) the shadowing argparse.py from site-packages:

# Find it
find /path/to/your/venv -name "argparse.py" -path "*/site-packages/*"

# Remove it (or rename to .bak if you want to be cautious)
rm /path/to/your/venv/lib/python3.10/site-packages/argparse.py

After removing, verify the stdlib version is now loaded:

python3 -c "import argparse; print(argparse.__file__)"
# Should now print something like: /usr/lib/python3.10/argparse.py
python3 -c "from argparse import BooleanOptionalAction; print('OK')"
# Should print: OK

Preventive fix for Docker / CI builds

If you are building in Docker, add a cleanup step after all pip install commands to remove any stray argparse.py that may have been deposited by a transitive dependency:

# Defensive cleanup: remove any legacy argparse.py that shadows the stdlib.
# pydantic-settings (Prefect dep) needs BooleanOptionalAction (stdlib 3.9+).
RUN find /usr/local/lib /usr/lib -name "argparse.py" -path "*/site-packages/*" -delete || true

The || true prevents the build from failing if no such file exists, which is the normal case.

Issue 2: DRUF Does Not Redirect to Metadata Form After Resource Upload

What happened

After getting the Prefect worker running and enabling DRUF in ckan.ini:

ckanext.datapusher_plus.enable_druf = true
ckanext.datapusher_plus.enable_form_redirect = true

The workflow partially works — clicking "Add Dataset" correctly skips the metadata form and goes straight to the resource upload page. But after uploading the file and clicking "Add", the user lands on the dataset read page (/dataset/{name}) instead of the metadata edit form (/dataset/{id}/edit).

CKAN request log:

302 POST /dataset/{id}/resource/new
302 GET  /dataset/{id}
200 GET  /dataset/{dataset-name}

Expected:

302 POST /dataset/{id}/resource/new
302 GET  /dataset/{id}/edit
200 GET  /dataset/{id}/edit

The dataset ends up published with only the auto-generated placeholder name (e.g. dataset97689) and no real metadata filled in.

Root cause: IFormRedirect does not exist in CKAN 2.11.3

DataPusher+ implements CKAN's IFormRedirect plugin interface to intercept the post-save redirect and point it at the metadata edit form. In plugin.py:

try:
    p.implements(p.IFormRedirect)
except (ImportError, AttributeError):
    # IFormRedirect not available in this CKAN version
    pass

The resource_save_redirect method (the one that would return the dataset.edit URL) is only ever called if IFormRedirect is registered — which requires the interface to exist in the CKAN version being used.

IFormRedirect does not exist in any stable CKAN release as of 2.11.3. It was introduced on the unreleased branch 7778-iformredirect. The DataPusher+ README links to this branch under the "Enabling DRUF" section:

"To enable DRUF you need DRUF compatible ckan version" → https://github.com/ckan/ckan/tree/7778-iformredirect

Because the try/except silently swallows the AttributeError, CKAN starts without error and no warning is logged. The redirect hook simply never fires. CKAN's default post-resource-save redirect logic takes over — which sends the user to the read page.

How CKAN's redirect logic works (relevant context)

In ckan/views/resource.py, after a resource is saved, the redirect destination is determined by the value of a save hidden field posted with the form:

save field value What CKAN does
go-metadata Activates the package (state=active), redirects to dataset read page
go-dataset Redirects to dataset edit page (the metadata form)
go-dataset-complete Redirects to dataset read page
again Redirects to resource new page (add another resource)

CKAN's standard "Add" button in package/snippets/resource_form.html uses value="go-dataset-complete" — which goes to the read page. That is what fires when DRUF's IFormRedirect hook does not exist to intercept it.

Additionally, the DRUF-created package has state='active' (set in druf_view.py). Because the package is not in draft state, CKAN renders the new_resource_not_draft.html template rather than the wizard-stage version — so there is no "Finish" button, only "Add" (go-dataset-complete).

Workaround for CKAN 2.11.3

Since IFormRedirect is unavailable, the workaround is to:

  1. Have druf_view.py pass a ?druf=1 query parameter on its redirect to the resource upload page.
  2. Override package/snippets/resource_form.html in your CKAN extension to detect ?druf=1 and swap the "Add" button (go-dataset-complete) for a "Save & Fill Metadata" button (go-dataset) — which CKAN already routes to dataset.edit.

Step 1 — druf_view.py patch:

# In ckanext/datapusher_plus/druf_view.py, change line 46 from:
return h.redirect_to('dataset_resource.new', id=pkg['id'])

# to:
return h.redirect_to('dataset_resource.new', id=pkg['id'], druf='1')

In Flask/CKAN, url_for kwargs that are not part of the URL rule are appended as query string parameters. This produces /dataset/{id}/resource/new?druf=1.

Step 2 — Template override (in your CKAN extension):

Create templates/package/snippets/resource_form.html inside your CKAN extension:

{% ckan_extends %}

{% block add_button %}
  {% if request.args.get('druf') %}
    <button class="btn btn-primary" name="save" value="go-dataset" type="submit">{{ _('Save & Fill Metadata') }}</button>
  {% else %}
    {{ super() }}
  {% endif %}
{% endblock %}

{% ckan_extends %} resolves to CKAN's own resource_form.html snippet. request.args is the Flask query string dict and is available in all CKAN Jinja2 templates.

How the full flow works after the workaround:

  1. User clicks "Add Dataset" → DRUF view (POST /resource-first/new) creates placeholder package
  2. druf_view.py redirects to GET /dataset/{id}/resource/new?druf=1
  3. CKAN renders new_resource_not_draft.html → loads resource_form_snippet
  4. Your extension's resource_form.html override is served; request.args.get('druf') is '1'
  5. "Save & Fill Metadata" button (value="go-dataset") rendered instead of "Add"
  6. User uploads file and clicks "Save & Fill Metadata"
  7. Form POSTs to /dataset/{id}/resource/new with save=go-dataset
  8. CKAN saves the resource, hits elif save_action == 'go-dataset': → redirects to GET /dataset/{id}/edit
  9. User lands on the metadata edit form

Note: ?druf=1 does not need to be in the form's action URL. The save field value in the POST body is what drives the redirect, not the URL. The query parameter is only needed during the GET (template render) to know which button to show.

Proper fix (awaiting CKAN merge)

The proper solution is for IFormRedirect to be merged into a stable CKAN release. Track the PR here: https://github.com/ckan/ckan/tree/7778-iformredirect. Once that branch lands in a stable CKAN release, the try/except in plugin.py will succeed, resource_save_redirect will be called, and the workaround above will no longer be needed (though it is harmless to leave in place — on CKAN versions that do have IFormRedirect, the ?druf=1 query parameter will still be present and request.args.get('druf') will still return '1', but the redirect will already have been handled correctly by IFormRedirect before the template is even rendered for a second time).

Summary

# Issue Root Cause Workaround
1 Prefect worker crashes — BooleanOptionalAction ImportError Legacy argparse.py in site-packages shadows stdlib Remove site-packages/argparse.py; add defensive find ... -delete to Docker build
2 DRUF sends user to read page instead of metadata form IFormRedirect not in CKAN 2.11.3 (only on unreleased branch 7778-iformredirect) Pass ?druf=1 in druf_view.py redirect; override resource_form.html to swap button to go-dataset

Both workarounds are non-destructive and compatible with future CKAN versions that do implement IFormRedirect.

Activity

  1. a5dur commented on Jun 24, 2026

    @a5dur
    a5dur
    on Jun 24, 2026
    ContributorAuthor

    Continued Setup Notes — CKAN 2.11.3 + DPP 3.0.0a0 on ARM64 (aarch64)

    Expanding on the original findings above with a full second-run report. Some issues from the first run no longer apply; new issues surfaced around suggestion buttons and template priority.

    Environment

    Component Version / Detail
    OS Ubuntu 22.04.5 LTS
    Architecture aarch64 (Apple Silicon Ubuntu VM)
    Kernel Linux 6.8.0-124-generic
    Python 3.10.12
    CKAN 2.11.3
    DataPusher+ 3.0.0a0 (installed from pip, site-packages)
    Prefect 3.7.5
    qsv (qsvdp) 20.1.0-jemalloc (aarch64 binary)
    ckanext-scheming 3.1.0
    ckanext-spatial 2.3.1 (dathere/ckanext-spatial@gztr fork)
    ckanext-envvars 0.0.6
    ckanext-gztr 0.0.1 (dev install, geoconnex branch)
    dathere_theme 0.0.1 (dev install, sandbox branch)
    pyproj 3.7.1
    shapely 2.1.0

    CKAN plugins (in .ini order):

    envvars activity datastore scheming_datasets spatial_metadata spatial_query
dathere_theme dathere_custom_css dathere_custom_homepage dathere_custom_header
dathere_custom_footer gztr datapusher_plus

    Issues From First Run — Status in This Run

    Issue Status
    BooleanOptionalAction ImportError Not encountered — clean venv, no legacy argparse.py
    DRUF sends user to read page instead of metadata form Resolved with different workaround (see below)

    Issue 3 (New): DRUF Redirect — Different Workaround Applied

    The ?druf=1 query-parameter approach described above works cleanly. In this run a slightly different approach was used instead — detecting DRUF state from pkg_dict.resources rather than a query parameter — but both achieve the same result.

    What was done

    Because IFormRedirect still does not exist in CKAN 2.11.3, a two-file template override was added inside ckanext-gztr:

    templates/package/new_resource_not_draft.html — passes pkg_dict to the resource form snippet (it is not passed by default, so the button condition cannot evaluate):

    {% ckan_extends %}

{% block form %}
  {% snippet resource_form_snippet, data=data, errors=errors,
     error_summary=error_summary, include_metadata=false,
     pkg_name=pkg_name, stage=stage, dataset_type=dataset_type,
     pkg_dict=pkg_dict %}
{% endblock %}

    templates/package/snippets/resource_form.html — swaps the "Add" button for "Save & Fill Metadata" when DRUF is enabled and the package has no resources yet:

    {% ckan_extends %}

{% block add_button %}
  {% set enable_druf = h.asbool(config.get('ckanext.datapusher_plus.enable_druf', 'false')) %}
  {% if enable_druf and pkg_dict is defined and pkg_dict and not pkg_dict.resources %}
    <button class="btn btn-primary" name="save" value="go-dataset" type="submit">
      {{ _('Save & Fill Metadata') }}
    </button>
  {% else %}
    {{ super() }}
  {% endif %}
{% endblock %}

    Why new_resource_not_draft.html matters: DRUF creates packages with state=active (not draft), so CKAN renders new_resource_not_draft.html instead of the wizard template. That template does not forward pkg_dict to the snippet, so pkg_dict is always undefined inside resource_form.html. The first file above fixes that.

    Result: After upload, clicking "Save & Fill Metadata" sends save=go-dataset, and ckan/views/resource.py routes that to h.redirect_to('{}.edit'.format(package_type), id=id) — the metadata edit form. ✅

    Issue 4 (New): FormulaProcessing Stage Skipped — ckan.plugins Is a List in CKAN 2.11.3

    What happened

    After DRUF redirect was fixed, the metadata form opened but suggestion buttons and auto-filled fields were absent. Worker logs showed:

    AttributeError: 'list' object has no attribute 'split'

    in ckanext/datapusher_plus/jobs/stages/formula.py inside should_skip().

    Root cause

    In CKAN 2.11.3, tk.config.get('ckan.plugins', '') returns a Python list, not a string. The should_skip() method assumed a string and called .split() on it.

    Fix applied (in-place patch to formula.py)

    # In FormulaStage.should_skip():
plugins_config = tk.config.get('ckan.plugins', '')
if isinstance(plugins_config, list):
    enabled_plugins = plugins_config
else:
    enabled_plugins = [p.strip() for p in plugins_config.split()]

    After this patch, FormulaProcessing runs correctly and dpp_suggestions is populated with STATUS: "DONE":

    {
  "STATUS": "DONE",
  "package": {
    "description": "Dataset contains 41 records with the following columns: station_id, station_name, latitude, longitude, date, parameter, value, unit, quality_flag.",
    "spatial_description": "Dataset contains geographic point data. Latitude column: \"latitude\", Longitude column: \"longitude\".",
    "coordinate_system": "WGS84",
    "update_frequency": "other",
    "primary_key_field": null
  }
}

    The suggestion_formula fields in the schema (description, spatial_description, coordinate_system, update_frequency, primary_key_field) are being evaluated correctly by Jinja2 and the results saved to dpp_suggestions.package. ✅

    Issue 5 (New — Current Blocker): DPP Scheming Form Snippet Templates Not Loading

    What's happening

    Even with FormulaProcessing completing successfully and dpp_suggestions populated, the metadata edit form (/dataset/{id}/edit) shows no suggestion buttons next to fields. The rendered HTML uses ckanext-scheming's standard markdown.html template:

    <!-- scheming's version -->
<label class="form-label" for="field-description">Description</label>
<textarea id="field-description" ...></textarea>

    DPP's override at scheming/form_snippets/markdown.html should render:

    <!-- DPP's version -->
<span class="form-label">Description:</span>
<button class="suggestion-btn" data-module="scheming-suggestions" style="display:none;">...</button>
<textarea id="field-description" ...></textarea>

    DPP's version is not being used. Same for select.html (coordinate_system, update_frequency fields) and text.html (primary_key_field).

    Root cause: IConfigurer._reverse_iteration_order = True

    CKAN's PluginImplementations(IConfigurer) iterates plugins in reverse order of their position in ckan.plugins. From ckan/plugins/interfaces.py:

    class IConfigurer(Interface):
    # plugins from the beginning of the plugin list can alter configuration
    # of plugins from the end of the list
    _reverse_iteration_order = True

    And in ckan/plugins/core.py:

    if self.interface._reverse_iteration_order:
    ordered_plugins = list(reversed(ordered_plugins))

    Each call to tk.add_template_directory() prepends to config['plugin_template_paths']:

    config_[config_var] = [absolute_path] + config_[config_var]

    Combined effect: plugins listed earliest in ckan.plugins have their update_config() called last (reversed), so they prepend last → end up first in plugin_template_paths → highest template priority.

    With scheming_datasets listed before datapusher_plus:

    • datapusher_plus.update_config() is called first (reversed order, DPP is last in list)
    • scheming_datasets.update_config() is called later
    • scheming prepends its path after DPP → scheming ends up before DPP in the template search path
    • scheming's scheming/form_snippets/markdown.html wins over DPP's

    Confirmed by adding debug logging to jinja_extensions.get_jinja_env_options():

    DEBUG_TEMPLATE_PATHS: [
  '/usr/.../ckanext/scheming/templates',        # ← scheming FIRST (highest priority)
  '/usr/.../ckanext/datapusher_plus/templates/druf',
  '/usr/.../ckanext/datapusher_plus/templates'   # ← DPP last (lowest priority)
]

    What was tried

    Attempt 1: Move datapusher_plus to the END of ckan.plugins.
    → Made it worse. DPP's update_config() is now called first (reversed), so DPP prepends first → ends up last in the list (lowest priority). Scheming still wins.

    Attempt 2: Copy DPP's scheming/form_snippets/*.html into ckanext-gztr/templates/ (gztr has highest priority being latest in the reversed order after scheming).
    → Works at the file level but is a maintenance burden (DPP updates don't propagate).

    Attempt 3 (current): Use extra_template_paths in ckan.ini — from ckan/config/environment.py:

    if extra_template_paths:
    # must be first for them to override defaults
    template_paths = extra_template_paths.split(',') + template_paths
    extra_template_paths=/usr/lib/ckan/default/lib/python3.10/site-packages/ckanext/datapusher_plus/templates

    → Still testing. Even with this set, the DPP template marker comment added to markdown.html does not appear in the rendered page. Investigating whether update_config() is running before or after the Jinja2 environment is built with these paths, or whether a separate Jinja2 cache (in-memory Environment._cache) is serving stale compiled templates.

    Additional complication: Missing webassets.yaml

    DPP 3.0.0a0's assets/ directory has js/ and styles/ subdirectories but no webassets.yaml. On startup:

    WARNING [ckan.lib.webassets_tools] Cannot create library datapusher_plus at
  .../ckanext/datapusher_plus/assets because webassets.yaml is missing

    This means {% asset 'datapusher_plus/suggestions' %} (called from DPP's scheming/snippets/suggestions_asset.html, which DPP's markdown.html includes) silently fails. Even if the template were loaded, the scheming-suggestions JavaScript module would not be served — so the suggestion buttons would remain hidden (style="display:none;" in suggestion_button.html) with no JS to show them.

    The JS polling mechanism in scheming-suggestions.js is designed to:

    1. On page load: poll package_show API every 2.5 seconds for dpp_suggestions
    2. When dpp_suggestions.STATUS == 'DONE' is found: reveal suggestion buttons for fields that have values in dpp_suggestions.package
    3. When user clicks a button: populate the form field with the suggested value

    This is solid architecture — but none of it fires if the asset isn't served.

    Prefect / qsv Status

    Prefect server: Running in Docker container, healthy at http://localhost:4200/api. PostgreSQL backend.

    Prefect worker: Running as a background process connecting to pool datapusher-plus.

    PREFECT_API_URL=http://localhost:4200/api CKAN_INI=/etc/ckan/default/ckan.ini \
  prefect worker start --pool datapusher-plus

    qsvdp performance: Solid. qsvdp 20.1.0-jemalloc compiled for aarch64-unknown-linux-gnu. A 41-row, 9-column test CSV processes through the full pipeline in ~20 seconds end-to-end (most of that is Prefect task orchestration overhead, not qsv). The stats, frequency, count, and geocode commands all run without issue.

    Pipeline timing: Full job from upload to STATUS: DONE in dpp_suggestions takes ~22 seconds. Since DRUF redirects to the edit form immediately after upload (~1-2 seconds), the form loads ~20 seconds before dpp_suggestions is ready. The JS polling loop handles this correctly in principle (polls up to 40×2.5s = 100 seconds), but since the JS itself isn't loading (webassets issue), it hasn't been observable in practice.

    Summary

    # Issue Status
    1 Prefect BooleanOptionalAction ImportError ✅ Not present (clean venv)
    2 DRUF redirects to read page ✅ Fixed via ckanext-gztr template override (swap add_button block)
    3 new_resource_not_draft.html doesn't pass pkg_dict ✅ Fixed via second gztr template override
    4 FormulaProcessing skipped — ckan.plugins returns list ✅ Fixed via in-place patch to formula.py (isinstance check)
    5 Missing webassets.yaml — DPP JS assets not served ⚠️ {% asset 'datapusher_plus/suggestions' %} silently fails
    6 DPP scheming form_snippets not overriding ckanext-scheming's 🔴 Current blocker — template priority inversion due to IConfigurer._reverse_iteration_order

    What would unblock this

    For Issue 5 (webassets): Adding a webassets.yaml to DPP's assets/ directory that registers the suggestions and suggestions_css bundles referenced in suggestions_asset.html.

    For Issue 6 (template priority): Either:

    • A note in DPP docs that datapusher_plus must appear before scheming_datasets in ckan.plugins (counterintuitive given _reverse_iteration_order), OR
    • Using CKAN's extra_template_paths config option to explicitly pin DPP's template path ahead of all plugins, OR
    • DPP registering its scheming form_snippet overrides via a higher-priority mechanism (e.g., calling add_template_directory from IConfigurable.configure() instead of IConfigurer.update_config(), so it runs after all update_config calls are complete and can prepend last)

    Setup: CKAN 2.11.3 on Ubuntu 22.04 aarch64, DPP 3.0.0a0, Prefect 3.7.5, qsvdp 20.1.0

  2. a5dur commented on Jun 25, 2026

    @a5dur
    a5dur
    on Jun 25, 2026
    ContributorAuthor

    ✅ DRUF + Suggestion Buttons — All Issues Resolved (CKAN 2.11.3 / DPP 3.0.0a0)

    Following up on the previous status comment. Everything is now working. Documenting every fix, including three issues that require upstream changes in this repo.

    🔴 Upstream DPP Fixes Required

    1. Missing assets/webassets.yaml — Suggestion buttons never load

    Severity: High — blocks the entire suggestion workflow

    DPP 3.0.0a0 ships:

    • assets/js/scheming-suggestions.js
    • assets/styles/suggestions.css
    • assets/js/scheming-ai-suggestions.js

    But there is no assets/webassets.yaml. CKAN's webassets system calls create_library("datapusher_plus", assets_path) which looks for webassets.yaml and silently logs a warning and exits if missing. The result: {% asset 'datapusher_plus/suggestions' %} in form snippet templates silently fails, the JS never loads, and suggestion buttons never appear — even when dpp_suggestions data is correctly populated.

    Fix — create ckanext/datapusher_plus/assets/webassets.yaml:

    suggestions:
  contents:
    - js/scheming-suggestions.js

suggestions_css:
  contents:
    - styles/suggestions.css

ai-suggestions:
  contents:
    - js/scheming-ai-suggestions.js

    2. formula.py crashes on CKAN 2.11.3 — FormulaProcessing always skipped

    Severity: High — silently skips suggestion formula evaluation

    FormulaStage.should_skip() in jobs/stages/formula.py does:

    plugins_config = tk.config.get('ckan.plugins', '')
enabled_plugins = [p.strip() for p in plugins_config.split()]

    In CKAN 2.11.3, tk.config.get('ckan.plugins') returns a Python list, not a string. Calling .split() on a list raises AttributeError: 'list' object has no attribute 'split', which is caught by the broad except Exception, logging a warning and returning True (skip). FormulaProcessing is silently skipped on every job.

    Fix:

    plugins_config = tk.config.get('ckan.plugins', '')
if isinstance(plugins_config, list):
    enabled_plugins = plugins_config
else:
    enabled_plugins = [p.strip() for p in plugins_config.split()]

    3. Template priority conflict with ckanext-scheming

    Severity: Medium — suggestion buttons render wrong template (scheming's instead of DPP's)

    DPP provides override templates at templates/scheming/form_snippets/markdown.html, select.html, text.html that inject suggestion buttons and AI suggestion markup. But CKAN's IConfigurer._reverse_iteration_order = True means plugins listed earlier in ckan.plugins get higher template priority.

    If scheming_datasets appears before datapusher_plus in ckan.plugins (which is the natural order since scheming should typically load before DPP), scheming's form_snippet templates win and DPP's overrides are never served.

    Required plugin order for DPP form_snippets to work:

    ckan.plugins = ... datapusher_plus scheming_datasets ...

    DPP's documentation and README should explicitly warn about this ordering requirement.

    ✅ What Was Fixed (Our Side)

    Problem Root Cause Fix Applied
    DRUF no redirect after resource upload No go-dataset button in resource form snippet Created gztr template overrides for package/new_resource_not_draft.html and package/snippets/resource_form.html
    FormulaProcessing stage always skipped ckan.plugins is a list in CKAN 2.11.3, .split() crashes Patched formula.py should_skip() with isinstance check
    Suggestion buttons not visible scheming's form_snippets had higher template priority than DPP's Moved datapusher_plus before scheming_datasets in ckan.plugins
    scheming-suggestions.js never loaded Missing assets/webassets.yaml Created webassets.yaml with 3 bundle definitions
    RecursionError on dataset add/edit extra_template_paths pointed to DPP's templates dir — same path appeared twice in computed_template_paths, causing infinite {% ckan_extends %} loop between DPP and datastore Removed extra_template_paths from ckan.ini

    Environment

    • CKAN 2.11.3
    • DataPusher Plus 3.0.0a0
    • Prefect 3.7.5
    • qsvdp 20.1.0
    • Ubuntu 22.04.5 LTS (aarch64, Apple Silicon VM)
    • ckanext-scheming (latest)

    🤖 Generated with Claude Code

  3. a5dur commented on Jul 28, 2026

    @a5dur
    a5dur
    on Jul 28, 2026
    ContributorAuthor

    DRUF on CKAN 2.12 (dev-v2.12, 2.12.0b0) — three more upstream fixes

    Follow-up to the 2.11.3 write-up above. Same stack brought up on CKAN 2.12 beta with ckan-devstaller; DPP 3.0.0a0 (a5dur/datapusher-plus@main), Prefect 3.7, qsvdp 20.1.0, Ubuntu 22.04 aarch64. DPP imports and ckan datapusher_plus prefect-deploy both work unchanged. Three DRUF-specific breakages, all in this repo.

    🔴 1. "Add Dataset" button disappears on organization pages

    Severity: High — DRUF entry point is unreachable from an organization

    templates/druf/snippets/add_dataset.html gates the whole button on a group variable:

    {% if h.check_config_permission('ckan.auth.create_unowned_dataset') or group %}

    CKAN 2.12 stopped passing it. The call site moved:

    2.11.3 2.12
    caller organization/read.html package/snippets/search_results.html:6
    args group=group_dict.id dataset_type=dataset_type only

    With the default ckan.auth.create_unowned_dataset = false, both sides of the or are false and the snippet renders nothing. Core's own snippets/add_dataset.html degrades gracefully to an unowned link; the DRUF override renders empty.

    g.group_dict is still set by views/group.py:433, so the value is recoverable:

    {% set owner_org = group %}
{% if not owner_org and c.group_dict and c.group_dict.get('is_organization') %}
  {% set owner_org = c.group_dict.get('id') %}
{% endif %}

{% macro add_button(dataset_type, owner_org) %}
  ...
  {% if h.check_config_permission('ckan.auth.create_unowned_dataset') or owner_org %}

    Passing owner_org as an explicit macro argument also removes the reliance on Jinja macro context lookup for group.

    🔴 2. DRUF button form has no CSRF token — 400 on click

    Severity: High — blocks the flow even once the button is visible

    CKAN 2.12 removed ckan.csrf_protection.ignore_extensions, so extension forms must carry a token. The DRUF button is a POST to resource_first.new with none:

    <form action="{% url_for 'resource_first.new' %}" method=post>
  {{ h.csrf_input() }}          {# <- required from 2.12 #}
  ...

    🔴 3. enable_druf unreadable from templates — DRUF save button silently reverts to core behaviour

    Severity: High — resource upload finishes the dataset instead of going to the metadata form

    Symptom: after uploading the first resource, the request redirects to <type>.read (the dataset page) instead of <type>.edit (the metadata form). The server log shows:

    302 /dataset/<id>/resource/new
302 /dataset/<id>
200 /dataset/dataset44316

    which is save_action == 'go-dataset-complete', i.e. core's "Add" button, not DRUF's "Save & Fill Metadata" (value="go-dataset").

    Cause is not template priority — that resolves correctly (verified: scheming/package/snippets/resource_form.html → DPP templates/druf/package/snippets/resource_form.html → the site override → core). The gating condition evaluates false:

    {% set enable_druf = h.asbool(config.get('ckanext.datapusher_plus.enable_druf', 'false')) %}

    From the 2.12 changelog:

    CKAN config options are not shared with Flask application by default. To pass option into Flask application (for example, when configuring Flask extension), add flask: true to the declaration of the config option.

    Jinja's config is Flask's config. Before 2.12 CKAN copied its whole config in, so this worked. Now:

    CKAN  config[ckanext.datapusher_plus.enable_druf] = 'true'
FLASK config[ckanext.datapusher_plus.enable_druf] = <<MISSING>>
rendered in template: 'DEFAULT_MISS'

    Compounding it, DPP never declared these options at all — hence WARNI [ckan.common] Option ckanext.datapusher_plus.enable_druf is not declared on every startup — and an undeclared option cannot be exported to Flask.

    Fix — add to ckanext/datapusher_plus/config_declaration.yaml:

          - key: ckanext.datapusher_plus.enable_druf
        type: bool
        flask: true
        editable: true
        default: false
        description: |
          Enable the DRUF (Dataset Resource Upload First) workflow.

      - key: ckanext.datapusher_plus.enable_form_redirect
        type: bool
        flask: true
        editable: true
        default: false
        description: |
          Redirect through the dataset metadata form after a resource is
          uploaded, rather than straight to the dataset page.

    Verified after the change:

    ckanext.datapusher_plus.enable_druf -> ckan=True flask=True
gztr template condition now evaluates: True

    This is strictly better than patching the consuming template: it repairs the option for every template that reads it, and it silences the undeclared-option warnings.

    Not a DPP bug, but it breaks the DRUF flow — ckanext-scheming version gate

    On a 2.12 pre-release the resource form shows only "Link", no "Upload". ckan.__version__ is 2.12.0b0, and under PEP 440 2.12.0b0 < 2.12, so scheming's gate in scheming/form_snippets/upload.html:

    is_upload_enabled=h.uploads_enabled("resource") if h.check_ckan_version("2.12") else h.uploads_enabled(),

    takes the legacy branch and calls h.uploads_enabled() with no object_type. 2.12's helper logs h.uploads_enabled call without object_type is non supported and returns False. Using h.check_ckan_version("2.12.0a0") fixes it and keeps 2.11 on the legacy branch. Filing separately with scheming.

    Environment

    • CKAN 2.12.0b0 (dev-v2.12)
    • DataPusher+ 3.0.0a0 (a5dur/datapusher-plus@main)
    • ckanext-scheming (latest), ckanext-spatial (dathere@gztr)
    • Prefect 3.7, qsvdp 20.1.0
    • Ubuntu 22.04 aarch64

  4. a5dur commented on Jul 29, 2026

    @a5dur
    a5dur
    on Jul 29, 2026
    ContributorAuthor

    🔴 4. Every push fails on CKAN 2.12: Error sending data to datastore (None - {'user_id': ['User not found']})

    Severity: High — no resource can reach the datastore

    Upload log from a clean push (CKAN 2.12.0b0, DPP 3.0.0a0):

    Starting stage: Download
Fetching from: http://localhost:5000/dataset/.../download/average-latitude-longitude-countries.csv...
File format: CSV
Downloading 0.01MB file...
Fetched 0.01MB file in 0.06 seconds.
Completed stage: Download
AI suggestions disabled (ckanext.datapusher_plus.enable_ai_suggestions=False); skipping AISuggestionsStage
Skipping stage: AISuggestionsGeneration
Starting stage: Database
COPYING 0 rows to Datastore...
DataPusher Plus error: Error sending data to datastore (None - {'user_id': ['User not found']}).
End of log

    Cause

    ckanext/datapusher_plus/datastore_utils.py, send_resource_to_datastore:

        try:
        resource_dict = tk.get_action("datastore_create")(
            {"ignore_auth": True}, request
        )

    ignore_auth: True skips the auth check but leaves context["user"] unset. datastore_create flips the datastore_active flag, which updates the resource and therefore creates an activity — and ckanext-activity's create schema (ckanext/activity/logic/schema.py:25) validates:

    "user_id": [
    not_missing,
    not_empty,
    unicode_safe,
    convert_user_name_or_id_to_id,
],

    With no user in context, convert_user_name_or_id_to_id raises Invalid(_('User not found')) (ckan/logic/validators.py:347), which surfaces as the ValidationError above. Nothing DPP does is wrong auth-wise; the context is simply missing the user that the downstream activity insert requires.

    Every other function in this module already resolves the site user for exactly this kind of call (upload_resource, patch_resource, and others all start with site_user = tk.get_action("get_site_user")(...)) — these three writes were just missed.

    Fix

        # ignore_auth skips the auth check but leaves context["user"] empty. datastore_create
    # flips the datastore_active flag, which updates the resource and creates an activity,
    # and the activity schema validates user_id -> ValidationError {'user_id': ['User not
    # found']}. Supply the site user, as the rest of this module already does.
    site_user = tk.get_action("get_site_user")({"ignore_auth": True}, {})
    try:
        resource_dict = tk.get_action("datastore_create")(
            {"ignore_auth": True, "user": site_user["name"]}, request
        )

    The same one-line omission exists in two sibling functions, which fail identically on any re-push (delete-then-recreate path):

    • delete_datastore_resource → datastore_delete
    • delete_resource → resource_delete

    Both patched the same way.

    Read-only calls in the module (datastore_search, datastore_info, package_show, resource_show) are unaffected — no activity is created, so the bare {"ignore_auth": True} context is fine there.

    Note

    COPYING 0 rows to Datastore... immediately before the error is not a second bug — the create-table call intentionally passes an empty records list, with the real data following via a separate COPY that this exception aborted before reaching.

    Fix is applied locally; posting the analysis now rather than waiting on a full green run, since the root cause and the code path are unambiguous. Will follow up if the re-push surfaces anything further downstream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions