Skip to content

Fix Iceberg REST catalog SigV4 parameter for AWS authentication #23669

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Fix Iceberg REST catalog SigV4 parameter for AWS authentication #23669

wants to merge 3 commits into from

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Oct 1, 2025 •
edited
Loading

Description

Fixes the parameter name used for enabling AWS SigV4 authentication in Iceberg REST catalogs. This resolves authentication failures when connecting to AWS S3 Tables and other SigV4-enabled REST catalogs.

Problem

When configuring an Iceberg REST catalog with SigV4 authentication, users encountered the following error:

pyiceberg.exceptions.ForbiddenError: RESTError 403: Received unexpected JSON Payload: 
{"message":"Missing Authentication Token"}, errors: Field required

Root Cause

The parameter passed to PyIceberg's load_rest() function was incorrect:

  • Incorrect: "rest.sigv4": True (boolean)
  • Correct: "rest.sigv4-enabled": "true" (string)

According to PyIceberg's REST catalog documentation, the parameter name should be rest.sigv4-enabled with a string value of "true".

Changes

  • Updated ingestion/src/metadata/ingestion/source/database/iceberg/catalog/rest.py to use the correct parameter name and value type
  • Added unit test test_iceberg_rest_catalog.py to verify the correct parameter generation and prevent regression

Testing

The new unit test verifies:

  1. The parameter "rest.sigv4-enabled" is present with value "true"
  2. Signing region and signing name are correctly propagated
  3. The old incorrect parameter "rest.sigv4" is not generated

Impact

  • ✅ Enables proper AWS SigV4 authentication for Iceberg REST catalogs
  • ✅ Allows successful connections to AWS S3 Tables
  • ✅ No breaking changes to existing functionality
  • ✅ Minimal change (1 line modified)

Fixes #[issue_number]

Original prompt

This section details on the original issue you should resolve

<issue_title>Iceberg REST catalog sigv4 has wrong parameter (fix included)</issue_title>
<issue_description>Affected module
Ingestion Framework

Describe the bug

When using an Iceberg REST catalog, in my case a s3table, this error will show up:

  File "/home/airflow/.local/lib/python3.10/site-packages/pyiceberg/catalog/rest.py", line 370, in _handle_non_200_response
    raise exception(response) from exc
pyiceberg.exceptions.ForbiddenError: RESTError 403: Received unexpected JSON Payload: {"message":"Missing Authentication Token"}, errors: Field required

To Reproduce

Create a connector to a REST catalog that requires sigv4

Expected behavior

To work (:

How to fix

In the file /ingestion/src/metadata/ingestion/source/database/iceberg/catalog/rest.py at line 80 the parameter "rest.sigv4": True should be "rest.sigv4-enabled": "true"

Version:

  • OS: al2023
  • Python version:
  • OpenMetadata version: 1.7.3
  • OpenMetadata Ingestion package version: 1.7.3

Additional context
Add any other context about the problem here.
</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes #21789

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

@Copilot @harshach
Fix Iceberg REST catalog sigv4 parameter name
920925e 
Co-authored-by: harshach <38649+harshach@users.noreply.github.com>
@Copilot Copilot AI changed the title [WIP] Iceberg REST catalog sigv4 has wrong parameter (fix included) Fix Iceberg REST catalog SigV4 parameter for AWS authentication Oct 1, 2025
@Copilot Copilot AI requested a review from harshach October 1, 2025 16:38
Copilot finished work on behalf of harshach October 1, 2025 16:38
@harshach harshach marked this pull request as ready for review October 1, 2025 23:54
@harshach harshach requested a review from a team as a code owner October 1, 2025 23:54
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 1, 2025

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@harshach harshach added the safe to test Add this label to run secure Github workflows on PRs label Oct 1, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 2, 2025

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion-base-slim:trivy (debian 12.12)

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (31)

Package Vulnerability ID Severity Installed Version Fixed Version
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.12.7 2.15.0
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.13.4 2.15.0
com.fasterxml.jackson.core:jackson-databind CVE-2022-42003 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind CVE-2022-42004 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4
com.google.code.gson:gson CVE-2022-25647 🚨 HIGH 2.2.4 2.8.9
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.3.0 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.3.0 3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.7.1 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.7.1 3.25.5, 4.27.5, 4.28.2
com.nimbusds:nimbus-jose-jwt CVE-2023-52428 🚨 HIGH 9.8.1 9.37.2
commons-beanutils:commons-beanutils CVE-2025-48734 🚨 HIGH 1.9.4 1.11.0
commons-io:commons-io CVE-2024-47554 🚨 HIGH 2.8.0 2.14.0
dnsjava:dnsjava CVE-2024-25638 🚨 HIGH 2.1.7 3.6.0
io.netty:netty-codec-http2 CVE-2025-55163 🚨 HIGH 4.1.96.Final 4.2.4.Final, 4.1.124.Final
io.netty:netty-codec-http2 GHSA-xpw8-rcwv-8f8p 🚨 HIGH 4.1.96.Final 4.1.100.Final
io.netty:netty-handler CVE-2025-24970 🚨 HIGH 4.1.96.Final 4.1.118.Final
net.minidev:json-smart CVE-2021-31684 🚨 HIGH 1.3.2 1.3.3, 2.4.4
net.minidev:json-smart CVE-2023-1370 🚨 HIGH 1.3.2 2.4.9
org.apache.avro:avro CVE-2024-47561 🔥 CRITICAL 1.7.7 1.11.4
org.apache.avro:avro CVE-2023-39410 🚨 HIGH 1.7.7 1.11.3
org.apache.derby:derby CVE-2022-46337 🔥 CRITICAL 10.14.2.0 10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
org.apache.ivy:ivy CVE-2022-46751 🚨 HIGH 2.5.1 2.5.2
org.apache.mesos:mesos CVE-2018-1330 🚨 HIGH 1.4.3 1.6.0
org.apache.thrift:libthrift CVE-2019-0205 🚨 HIGH 0.12.0 0.13.0
org.apache.thrift:libthrift CVE-2020-13949 🚨 HIGH 0.12.0 0.14.0
org.apache.zookeeper:zookeeper CVE-2023-44981 🔥 CRITICAL 3.6.3 3.7.2, 3.8.3, 3.9.1
org.eclipse.jetty:jetty-server CVE-2024-13009 🚨 HIGH 9.4.56.v20240826 9.4.57.v20241219

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (3)

Package Vulnerability ID Severity Installed Version Fixed Version
Werkzeug CVE-2024-34069 🚨 HIGH 2.2.3 3.0.3
deepdiff CVE-2025-58367 🔥 CRITICAL 7.0.1 8.6.1
setuptools CVE-2025-47273 🚨 HIGH 70.3.0 78.1.1

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/extended_sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/lineage.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.yaml

No Vulnerabilities Found

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 2, 2025

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion:trivy (debian 12.9)

Vulnerabilities (19)

Package Vulnerability ID Severity Installed Version Fixed Version
libexpat1 CVE-2023-52425 🚨 HIGH 2.5.0-1+deb12u1 2.5.0-1+deb12u2
libexpat1 CVE-2024-8176 🚨 HIGH 2.5.0-1+deb12u1 2.5.0-1+deb12u2
libgnutls30 CVE-2025-32988 🚨 HIGH 3.7.9-2+deb12u3 3.7.9-2+deb12u5
libgnutls30 CVE-2025-32990 🚨 HIGH 3.7.9-2+deb12u3 3.7.9-2+deb12u5
libicu72 CVE-2025-5222 🚨 HIGH 72.1-3 72.1-3+deb12u1
libperl5.36 CVE-2023-31484 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u3
libperl5.36 CVE-2024-56406 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u2
libsqlite3-0 CVE-2025-6965 🔥 CRITICAL 3.40.1-2+deb12u1 3.40.1-2+deb12u2
libxslt1.1 CVE-2024-55549 🚨 HIGH 1.1.35-1 1.1.35-1+deb12u1
libxslt1.1 CVE-2025-24855 🚨 HIGH 1.1.35-1 1.1.35-1+deb12u1
libxslt1.1 CVE-2025-7424 🚨 HIGH 1.1.35-1 1.1.35-1+deb12u2
perl CVE-2023-31484 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u3
perl CVE-2024-56406 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u2
perl-base CVE-2023-31484 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u3
perl-base CVE-2024-56406 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u2
perl-modules-5.36 CVE-2023-31484 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u3
perl-modules-5.36 CVE-2024-56406 🚨 HIGH 5.36.0-7+deb12u1 5.36.0-7+deb12u2
sqlite3 CVE-2025-6965 🔥 CRITICAL 3.40.1-2+deb12u1 3.40.1-2+deb12u2
sudo CVE-2025-32462 🚨 HIGH 1.9.13p3-1+deb12u1 1.9.13p3-1+deb12u2

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (31)

Package Vulnerability ID Severity Installed Version Fixed Version
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.12.7 2.15.0
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.13.4 2.15.0
com.fasterxml.jackson.core:jackson-databind CVE-2022-42003 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind CVE-2022-42004 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4
com.google.code.gson:gson CVE-2022-25647 🚨 HIGH 2.2.4 2.8.9
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.3.0 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.3.0 3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.7.1 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.7.1 3.25.5, 4.27.5, 4.28.2
com.nimbusds:nimbus-jose-jwt CVE-2023-52428 🚨 HIGH 9.8.1 9.37.2
commons-beanutils:commons-beanutils CVE-2025-48734 🚨 HIGH 1.9.4 1.11.0
commons-io:commons-io CVE-2024-47554 🚨 HIGH 2.8.0 2.14.0
dnsjava:dnsjava CVE-2024-25638 🚨 HIGH 2.1.7 3.6.0
io.netty:netty-codec-http2 CVE-2025-55163 🚨 HIGH 4.1.96.Final 4.2.4.Final, 4.1.124.Final
io.netty:netty-codec-http2 GHSA-xpw8-rcwv-8f8p 🚨 HIGH 4.1.96.Final 4.1.100.Final
io.netty:netty-handler CVE-2025-24970 🚨 HIGH 4.1.96.Final 4.1.118.Final
net.minidev:json-smart CVE-2021-31684 🚨 HIGH 1.3.2 1.3.3, 2.4.4
net.minidev:json-smart CVE-2023-1370 🚨 HIGH 1.3.2 2.4.9
org.apache.avro:avro CVE-2024-47561 🔥 CRITICAL 1.7.7 1.11.4
org.apache.avro:avro CVE-2023-39410 🚨 HIGH 1.7.7 1.11.3
org.apache.derby:derby CVE-2022-46337 🔥 CRITICAL 10.14.2.0 10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
org.apache.ivy:ivy CVE-2022-46751 🚨 HIGH 2.5.1 2.5.2
org.apache.mesos:mesos CVE-2018-1330 🚨 HIGH 1.4.3 1.6.0
org.apache.thrift:libthrift CVE-2019-0205 🚨 HIGH 0.12.0 0.13.0
org.apache.thrift:libthrift CVE-2020-13949 🚨 HIGH 0.12.0 0.14.0
org.apache.zookeeper:zookeeper CVE-2023-44981 🔥 CRITICAL 3.6.3 3.7.2, 3.8.3, 3.9.1
org.eclipse.jetty:jetty-server CVE-2024-13009 🚨 HIGH 9.4.56.v20240826 9.4.57.v20241219

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (9)

Package Vulnerability ID Severity Installed Version Fixed Version
Authlib CVE-2025-59420 🚨 HIGH 1.3.1 1.6.4
Werkzeug CVE-2024-34069 🚨 HIGH 2.2.3 3.0.3
apache-airflow-providers-common-sql CVE-2025-30473 🚨 HIGH 1.21.0 1.24.1
deepdiff CVE-2025-58367 🔥 CRITICAL 7.0.1 8.6.1
redshift-connector CVE-2025-5279 🚨 HIGH 2.1.5 2.1.7
setuptools CVE-2024-6345 🚨 HIGH 65.5.1 70.0.0
setuptools CVE-2025-47273 🚨 HIGH 65.5.1 78.1.1
setuptools CVE-2025-47273 🚨 HIGH 70.3.0 78.1.1
tornado CVE-2025-47287 🚨 HIGH 6.4.2 6.5

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /home/airflow/openmetadata-airflow-apis/openmetadata_managed_apis.egg-info/PKG-INFO

No Vulnerabilities Found

@sonarqubecloud SonarQubeCloud
Copy link

sonarqubecloud bot commented Oct 2, 2025

Quality Gate Passed Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@harshach harshach Awaiting requested review from harshach

At least 1 approving review is required to merge this pull request.

Assignees

@harshach harshach

Copilot code review Copilot

Labels
safe to test Add this label to run secure Github workflows on PRs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Iceberg REST catalog sigv4 has wrong parameter (fix included)
2 participants
@Copilot @harshach