Skip to content

Upload DatasetResource with DataFactoryManagementClient.create_or_update() adds non-wanted piece of code to dataset json #31541

Open
@Lohikus

Description

@Lohikus

Describe the bug
When trying to upload DatasetResource instance via DataFactoryManagementClient method create_or_update, there is a piece of code added to the original properties
"typeProperties": { "schema": [] }

This dataset is used in Lookup activity with query, but thanks to this non-wanted additional piece of code it tries to search in the empty schema property and ends with the error:
Operation on target Lookup to database failed: ErrorCode=UserErrorInvalidValueInPayload,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to convert the value in 'schema' property to 'System.String' type. Please make sure the payload structure and value are correct.,Source=Microsoft.DataTransfer.DataContracts,''Type=System.InvalidCastException,Message=Object must implement IConvertible.,Source=mscorlib,'

To Reproduce
Steps to reproduce the behavior:

  1. Create DatasetResource instance without
    "typeProperties": { "schema": [] }
  2. Send this DatasetResource instance to adf via DataFactoryManagementClient::create_or_update
  3. Check dataset's json that it contains the
    "typeProperties": { "schema": [] }
  4. Create lookup with query and check it doesn't work till you remove this piece of code.

Expected behavior
Dataset should be uploaded as it is - without the
"typeProperties": { "schema": [] }

Screenshots
Part of the json dataset without the code:
obrazek

What is uploaded to adf:
obrazek

Additional context
I noticed during testing the problem, that when I fill typeProperties in dataset json by myself on my local and upload it via create_or_update, there is warning message:
azure.mgmt.datafactory._serialization WARNING Ignoring extracted value '[]' from <function last_rest_key_case_insensitive_extractor at 0x7f8a3aac4160> for key 'schema_type_properties_schema' (duplicate extraction, follow extractors order)
Which seems connected, like if my local settings of typeProperties overrides some default...

Metadata

Metadata

Assignees

Labels

Data FactoryMgmtThis issue is related to a management-plane library.Service AttentionWorkflow: This issue is responsible by Azure service team.customer-reportedIssues that are reported by GitHub users external to the Azure organization.needs-team-attentionWorkflow: This issue needs attention from Azure service team or SDK teamquestionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions