Description
Describe the bug
When trying to upload DatasetResource instance via DataFactoryManagementClient method create_or_update, there is a piece of code added to the original properties
"typeProperties": { "schema": [] }
This dataset is used in Lookup activity with query, but thanks to this non-wanted additional piece of code it tries to search in the empty schema property and ends with the error:
Operation on target Lookup to database failed: ErrorCode=UserErrorInvalidValueInPayload,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to convert the value in 'schema' property to 'System.String' type. Please make sure the payload structure and value are correct.,Source=Microsoft.DataTransfer.DataContracts,''Type=System.InvalidCastException,Message=Object must implement IConvertible.,Source=mscorlib,'
To Reproduce
Steps to reproduce the behavior:
- Create DatasetResource instance without
"typeProperties": { "schema": [] }
- Send this DatasetResource instance to adf via DataFactoryManagementClient::create_or_update
- Check dataset's json that it contains the
"typeProperties": { "schema": [] }
- Create lookup with query and check it doesn't work till you remove this piece of code.
Expected behavior
Dataset should be uploaded as it is - without the
"typeProperties": { "schema": [] }
Screenshots
Part of the json dataset without the code:
Additional context
I noticed during testing the problem, that when I fill typeProperties in dataset json by myself on my local and upload it via create_or_update, there is warning message:
azure.mgmt.datafactory._serialization WARNING Ignoring extracted value '[]' from <function last_rest_key_case_insensitive_extractor at 0x7f8a3aac4160> for key 'schema_type_properties_schema' (duplicate extraction, follow extractors order)
Which seems connected, like if my local settings of typeProperties overrides some default...