Skip to content

Binary/bytes columns serialized as Python bytes repr (b'\x..') in result_row #1037

Description

@tderk

Describe the bug

On T-SQL adapters (dbt-sqlserver / dbt-fabric / dbt-synapse), varbinary/binary columns are serialized into result_row (in test_result_rows, and result_rows in elementary_test_results) as a Python bytes repr string rather than a usable value.

Example stored value for a 16-byte binary column (e.g. a Data Vault hash key):

b'\x1a_,:\xed\x08\x9dAL\xdadM\x83`\x8e\xad'

This is unusable downstream: it isn't the original binary, isn't hex, and can't be reliably reconstructed in SQL because the bytes repr mixes \xNN escapes with literal printable ASCII bytes.

Root cause

macros/utils/run_queries/agate_to_dicts.sql:

{% macro agate_val_serialize(val) %}
    {% if val.year is defined %} {% do return(val.isoformat()) %} {% endif %}
    {% if elementary.edr_is_decimal(val) %}
        {% do return(elementary.edr_serialize_decimal(val)) %}
    {% endif %}
    {% do return(val) %}
{% endmacro %}

bytes values fall through to return(val), and the subsequent tojson renders them via their Python repr (b'...').

Expected behavior

Binary values should be serialized to a stable, adapter-neutral string — hex is the natural choice (matches T-SQL CONVERT(varchar, col, 1)0x...).

Suggested fix

Handle bytes/bytearray explicitly in agate_val_serialize:

{% macro agate_val_serialize(val) %}
    {# binary (e.g. hash keys) -> 0x hex, instead of a Python bytes repr #}
    {% if val is not none and val.hex is defined and val.decode is defined %}
        {% do return('0x' ~ (val.hex() | upper)) %}
    {% endif %}
    {% if val.year is defined %} {% do return(val.isoformat()) %} {% endif %}
    {% if elementary.edr_is_decimal(val) %}
        {% do return(elementary.edr_serialize_decimal(val)) %}
    {% endif %}
    {% do return(val) %}
{% endmacro %}

The val.hex is defined and val.decode is defined guard matches only bytes/bytearray (avoids float, which also has .hex(); str has no .decode in py3).

Can't be worked around by consumers

agate_val_serialize is called everywhere as elementary.agate_val_serialize(...) (package-qualified) and is not adapter.dispatch-ed, so it cannot be overridden from a downstream project. The only current workaround is editing the package file directly (lost on dbt deps). Either applying the fix above, or routing this through adapter.dispatch so downstream projects can override it, would resolve this.

Environment

  • elementary package version: 0.25.0
  • Adapter: T-SQL family (dbt-sqlserver / dbt-fabric / dbt-synapse)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions