Skip to content

Conversation

jboliv01
Copy link
Contributor

Description

During streaming uploads using the stream_upload function in UtilitiesClient, the DataFrame slicing was inadvertently causing truncation of rows. This was due to an extraneous comma in the DataFrame slicing logic.

Steps to Reproduce

  1. Use a large DataFrame with the stream_upload function.
  2. Observe that not all rows are uploaded as expected.

Fix

Removed the extraneous comma in the DataFrame slicing logic in the stream_upload function to ensure correct slicing of rows.

Testing

I've tested the fix with several large DataFrames and confirmed that all rows are now correctly uploaded.

Additional Notes

This fix ensures that data integrity is maintained during the streaming upload process.

Removing typo from df_up slice causing df_create and df_update to break.
@samrands
Copy link
Collaborator

@jboliv01 I was not able to reproduce this bug. Could you add a test that reproduces it?

@jboliv01
Copy link
Contributor Author

Hi @samrands I've added a script you can use to reproduce the issue. Please let me know if you have any questions.

@samrands
Copy link
Collaborator

@jboliv01 I ran that test and the datasource using the old method was identical to the one using the new method. However, we'll still make this change because removing the comma fixes your issue and is more in line with the expected usage of the method from the documentation.

Removing test file
@thattolleyguy
Copy link
Collaborator

@jboliv01 We've been unable to reproduce the error even with your test. However, we also don't see any new issues. I'm going to go ahead and merge the fix in as it seems to fix the error on your side. If you have any further issues, let us know.

@thattolleyguy thattolleyguy merged commit bb93b4a into domoinc:master Sep 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants