Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNOW-1163211: Refactor StringType to allow variable max size #1871

Merged
merged 6 commits into from
Jul 19, 2024

Conversation

sfc-gh-jrose
Copy link
Contributor

@sfc-gh-jrose sfc-gh-jrose commented Jul 2, 2024

  1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

    Fixes SNOW-1163211

  2. Fill out the following pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
      • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
    • I am adding new logging messages
    • I am adding a new telemetry message
    • I am adding new credentials
    • I am adding a new dependency
    • If this is a new feature/behavior, I'm adding the Local Testing parity changes.
  3. Please describe how your code solves the related issue.

    This PR prepares snowpark for a future change to handling large objects. This mainly affects binary and string columns. Our current snowpark binary type columns do not have a concept for size so will be unaffected by this change. String type columns on the other had need to be made aware of the new max size. To implement support for this I have done the following:

    • Added a new property to the server connection that provides the max string size when connecting. This is provided by the VARCHAR_AND_BINARY_MAX_SIZE_IN_RESULT parameter that is returned on successful connection of a session. This parameter is not always set so if it is missing we will default to the previous max string size of 16mb. Local testing connections will remain on the old max size for now.
    • Added a property to StringType that can be set to say that the type is of max size. This is set by default when a string length is not specified. It can also be set manually when both the length and the max size are known.
    • When parsing schema metadata the max size from the connection is used to determine if StringType columns are of max size and if so the type flag is set.
    • Normalized the repr for StringType so that max size strings always are displayed without length. This was inconsistent before. User defined max size strings would be empty while server side strings would always have the max size in the repr.
    • For now I've updated the tests that are affected by this change and added a unit test to ensure the new functionality works as expected for StringType comparison and repr. LOB support is an account level parameter which makes integration tests difficult for now.

@github-actions github-actions bot added the local testing Local Testing issues/PRs label Jul 2, 2024
@sfc-gh-jrose sfc-gh-jrose force-pushed the jrose_snow_1163211_enable_lob branch from 50ebf8e to c96d367 Compare July 3, 2024 00:08
@sfc-gh-jrose sfc-gh-jrose marked this pull request as ready for review July 8, 2024 20:03
@sfc-gh-jrose sfc-gh-jrose requested a review from a team as a code owner July 8, 2024 20:03
@@ -155,6 +156,17 @@ def __init__(
self._lower_case_parameters = {k.lower(): v for k, v in options.items()}
self._add_application_parameters()
self._conn = conn if conn else connect(**self._lower_case_parameters)
self._max_string_size = DEFAULT_STRING_SIZE
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can directly have self.max_string_size here?

@sfc-gh-jrose sfc-gh-jrose merged commit 038e05b into main Jul 19, 2024
34 checks passed
@sfc-gh-jrose sfc-gh-jrose deleted the jrose_snow_1163211_enable_lob branch July 19, 2024 16:14
@github-actions github-actions bot locked and limited conversation to collaborators Jul 19, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
local testing Local Testing issues/PRs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants