feat: Stream on table resource #3109

sfc-gh-jmichalak · 2024-10-01T09:56:48Z

add snowflake_stream_on_table resource
add LastQueryId to the context client
add a helper to insert data in a table - this is needed to track data in streams
extract common code to be reused in other stream resources
add a new function to parse a list of schema-level object identifiers
use enums in streams sdk
add a note to CHANGES_BEFORE_V1 explaining support of COPY GRANTS
minor docs fixes

Test Plan

acceptance tests

References

https://docs.snowflake.com/en/sql-reference/sql/create-stream

TODO

Add remaining resources (external table, stage, view). Rework data source.

github-actions · 2024-10-01T11:56:41Z

Integration tests failure for 66bd22b103e4ff516a40f9da7ce174abf8a78629

github-actions · 2024-10-01T14:21:58Z

Integration tests failure for c67968c20a4f56b0c6e02df2909021f83c8e5041

pkg/acceptance/bettertestspoc/assert/objectassert/stream_snowflake_ext.go

pkg/acceptance/bettertestspoc/assert/resourceshowoutputassert/stream_show_output_ext.go

pkg/resources/stream_on_table.go

sfc-gh-jcieslak · 2024-10-02T08:26:44Z

pkg/resources/stream_on_table.go

+		}
+
+		if v := d.Get("copy_grants"); v.(bool) {
+			req.WithCopyGrants(true).WithOrReplace(true)


is OrReplace needed here? It allows for possible overriding of already existing stream, without it the task will use OrReplace only when modified attribute cannot be altered.

It's needed because otherwise when you have COPY GRANTS without OR REPLACE, Snowflake fails.

You are right about the need for orReplace needed with copy grants but I am with @sfc-gh-jcieslak here: we should steer the copy grants logic, and use it only when we need to replace the stream and not by default in create

We don't use OR REPLACE by default. It's used when the stream is already in the state, and detect changes on fields that can't be updated, because ForceNew drops the grants. In the second case, when it's not in the state, and the user wants COPY GRANTS to preserve old grants. In other cases, we don't use OR REPLACE.

I disagree with the second case. IMO a new create should be done without copy grants and without or replace. Thge assumption should be, that if we are creating a new object, the copy grants do not matter. Copy grants should be used only in the first scenario you mentioned (so the object is already managed and we discover changes).

Changed again.

pkg/resources/stream_on_table_acceptance_test.go

pkg/schemas/stream.go

pkg/schemas/stream_gen.go

pkg/sdk/parsers.go

pkg/sdk/parsers_test.go

pkg/sdk/streams_def.go

github-actions · 2024-10-02T09:57:30Z

Integration tests cancelled for d61356336d2253d924befc0e47db4006421c81b4

github-actions · 2024-10-02T17:46:57Z

Integration tests failure for c1f7c234fcd8962531b2672c8f10191f471b0ae3

MIGRATION_GUIDE.md

pkg/acceptance/bettertestspoc/assert/resourceshowoutputassert/stream_show_output_ext.go

sfc-gh-asawicki · 2024-10-02T20:22:27Z

pkg/resources/stream_on_table.go

+		}
+
+		if v := d.Get("copy_grants"); v.(bool) {
+			req.WithCopyGrants(true).WithOrReplace(true)


You are right about the need for orReplace needed with copy grants but I am with @sfc-gh-jcieslak here: we should steer the copy grants logic, and use it only when we need to replace the stream and not by default in create

pkg/resources/stream_on_table.go

pkg/schemas/stream_gen.go

pkg/sdk/streams_def.go

sfc-gh-asawicki · 2024-10-02T20:30:41Z

pkg/resources/stream_on_table_acceptance_test.go

+	table, cleanupTable := acc.TestClient().Table.CreateWithChangeTracking(t)
+	t.Cleanup(cleanupTable)
+
+	baseModel := func() *model.StreamOnTableModel {


nit: this could be added as Base(id, table_id) func in _ext.go file for model

Yes, but I'll leave this for now.

why? For consistency we should have a way of defining a default/common base for model.

Ok, I'll do this in the next pr.

github-actions · 2024-10-03T12:02:20Z

Integration tests failure for bb56ac99ba21c82519f5a5a17a498893f2acc4cf

github-actions · 2024-10-03T14:55:52Z

Integration tests failure for 5ba98dd58fa3be86e55c8e4bfcb4061d2eaded4a

pkg/resources/stream_common.go

pkg/resources/stream_on_table.go

pkg/schemas/stream.go

github-actions · 2024-10-04T09:44:31Z

Integration tests failure for 72e29d3f49ebc019e54bac1384b9e203210b15d9

github-actions · 2024-10-04T09:46:30Z

Integration tests failure for 7cb18a96e580feb3bd49e1734ade447c8f735ae3

github-actions · 2024-10-04T11:42:31Z

Integration tests failure for 08c9c852fe81b9a76f4d76583ed4ea66cf195f10

github-actions · 2024-10-04T11:55:53Z

Integration tests failure for 81dec99a87740624ce17a68afb3914dc5f43a9a3

github-actions · 2024-10-04T16:14:53Z

Integration tests failure for 5a16dd94cdd87b082fcb66aede700b9bed79e0da

sfc-gh-jcieslak · 2024-10-07T13:04:18Z

docs/resources/stream_on_table.md

@@ -63,7 +63,7 @@ resource "snowflake_stream_on_table" "stream" {
 - `at` (Block List, Max: 1) This field specifies that the request is inclusive of any changes made by a statement or transaction with a timestamp equal to the specified parameter. Due to Snowflake limitations, the provider does not detect external changes on this field. External changes for this field won't be detected. In case you want to apply external changes, you can re-create the resource manually using "terraform taint". (see [below for nested schema](#nestedblock--at))
 - `before` (Block List, Max: 1) This field specifies that the request refers to a point immediately preceding the specified parameter. This point in time is just before the statement, identified by its query ID, is completed.  Due to Snowflake limitations, the provider does not detect external changes on this field. External changes for this field won't be detected. In case you want to apply external changes, you can re-create the resource manually using "terraform taint". (see [below for nested schema](#nestedblock--before))
 - `comment` (String) Specifies a comment for the stream.
- `copy_grants` (Boolean) Retains the access permissions from the original stream when a new stream is created using the OR REPLACE clause. Use only if the resource is already managed by Terraform.
+- `copy_grants` (Boolean) Retains the access permissions from the original stream when a new stream is created using the OR REPLACE clause. Use only if the resource is already managed by Terraform. Otherwise, this field is skipped.


I would change the description of this field to something similar to:
Retains the access permissions from the original stream when a new stream is created using the OR REPLACE clause that is sometimes used when updating the stream. The value won't have any effect when creating a new stream.

Ok, I'll rephrase this in the next pr.

🤖 I have created a release *beep* *boop* --- ## [0.97.0](v0.96.0...v0.97.0) (2024-10-10) ### 🎉 **What's new:** * Add secret to sdk ([#3091](#3091)) ([7430aee](7430aee)) * Add service user and legacy service user resources ([#3119](#3119)) ([0e88e08](0e88e08)) * Handle all Task parameters in SDK ([#3103](#3103)) ([08ae072](08ae072)) * Stream on external table resource ([#3122](#3122)) ([d837341](d837341)) * Stream on table resource ([#3109](#3109)) ([97fa9b4](97fa9b4)) * Tasks v1 readiness - SDK part ([#3086](#3086)) ([0a77383](0a77383)) * Upgrade stream sdk ([#3105](#3105)) ([ad5fa11](ad5fa11)) ### 🔧 **Misc** * Add pre check to each datasource ([#3065](#3065)) ([560ab6b](560ab6b)) * Bump golang-ci lint to 1.61 ([#3112](#3112)) ([f23e085](f23e085)) * Secret Validation change ([#3111](#3111)) ([666630e](666630e)) ### 🐛 **Bug fixes:** * Fix parsing text in view, check parenthesis in ParseSchemaObjectIdentifierWithArguments ([#3102](#3102)) ([b0a67e6](b0a67e6)) * Try to reproduce 2679 and 3117 ([#3124](#3124)) ([ccdbc30](ccdbc30)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: snowflake-release-please[bot] <105954990+snowflake-release-please[bot]@users.noreply.github.com>

Stream on table

66bd22b

Cleanup

c67968c

sfc-gh-jmichalak marked this pull request as ready for review October 1, 2024 12:44

sfc-gh-jmichalak requested review from sfc-gh-jcieslak and sfc-gh-asawicki and removed request for sfc-gh-jcieslak October 1, 2024 12:44

sfc-gh-jcieslak reviewed Oct 2, 2024

View reviewed changes

sfc-gh-jmichalak added 2 commits October 2, 2024 11:52

Review

d613563

Fix tests

c1f7c23

sfc-gh-asawicki requested changes Oct 2, 2024

View reviewed changes

sfc-gh-jmichalak added 2 commits October 3, 2024 10:44

Merge remote-tracking branch 'origin/main' into stream-v1

93a3d3a

Fixes

bb56ac9

Review

5ba98dd

sfc-gh-jcieslak reviewed Oct 3, 2024

View reviewed changes

pkg/resources/stream_common.go Show resolved Hide resolved

pkg/resources/stream_on_table.go Outdated Show resolved Hide resolved

pkg/schemas/stream.go Outdated Show resolved Hide resolved

sfc-gh-jmichalak added 2 commits October 4, 2024 09:43

Cleanup

72e29d3

pre push

7cb18a9

sfc-gh-jmichalak added 2 commits October 4, 2024 11:50

skip copy grants for empty state

81dec99

pre push

08c9c85

Fix tests

5a16dd9

sfc-gh-jcieslak reviewed Oct 7, 2024

View reviewed changes

sfc-gh-jcieslak approved these changes Oct 8, 2024

View reviewed changes

sfc-gh-asawicki approved these changes Oct 9, 2024

View reviewed changes

sfc-gh-jmichalak merged commit 97fa9b4 into main Oct 9, 2024
8 of 9 checks passed

sfc-gh-jmichalak deleted the stream-v1 branch October 9, 2024 10:52

snowflake-release-please bot mentioned this pull request Oct 9, 2024

chore(main): release 0.97.0 #3100

Merged

sfc-gh-jmichalak linked an issue Oct 9, 2024 that may be closed by this pull request

[Bug]: snowflake_stream show_initial_rows = true is reset at every apply #2975

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Stream on table resource #3109

feat: Stream on table resource #3109

sfc-gh-jmichalak commented Oct 1, 2024 •

edited

Loading

github-actions bot commented Oct 1, 2024

github-actions bot commented Oct 1, 2024

sfc-gh-jcieslak Oct 2, 2024

sfc-gh-jmichalak Oct 2, 2024

sfc-gh-asawicki Oct 2, 2024

sfc-gh-jmichalak Oct 3, 2024 •

edited

Loading

sfc-gh-asawicki Oct 3, 2024

sfc-gh-jmichalak Oct 3, 2024

sfc-gh-jmichalak Oct 4, 2024

github-actions bot commented Oct 2, 2024

github-actions bot commented Oct 2, 2024

sfc-gh-asawicki Oct 2, 2024

sfc-gh-asawicki Oct 2, 2024

sfc-gh-jmichalak Oct 3, 2024

sfc-gh-asawicki Oct 9, 2024

sfc-gh-asawicki Oct 9, 2024

sfc-gh-jmichalak Oct 9, 2024

github-actions bot commented Oct 3, 2024

github-actions bot commented Oct 3, 2024

github-actions bot commented Oct 4, 2024

github-actions bot commented Oct 4, 2024

github-actions bot commented Oct 4, 2024

github-actions bot commented Oct 4, 2024

github-actions bot commented Oct 4, 2024

sfc-gh-jcieslak Oct 7, 2024

sfc-gh-jmichalak Oct 7, 2024

feat: Stream on table resource #3109

feat: Stream on table resource #3109

Conversation

sfc-gh-jmichalak commented Oct 1, 2024 • edited Loading

Test Plan

References

TODO

github-actions bot commented Oct 1, 2024

github-actions bot commented Oct 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-jmichalak Oct 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 2, 2024

github-actions bot commented Oct 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 3, 2024

github-actions bot commented Oct 3, 2024

github-actions bot commented Oct 4, 2024

github-actions bot commented Oct 4, 2024

github-actions bot commented Oct 4, 2024

github-actions bot commented Oct 4, 2024

github-actions bot commented Oct 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sfc-gh-jmichalak commented Oct 1, 2024 •

edited

Loading

sfc-gh-jmichalak Oct 3, 2024 •

edited

Loading