feat: Smoke Tests V2 #841

sichanyoo · 2024-10-16T18:58:25Z

Companion aws-sdk-swift PR: awslabs/aws-sdk-swift#1791

Issue #

1779

Description of changes

Adds the smoke test code generator for generic Smithy spec, whose behaviors can be customized via method overrides in a child class.

Scope

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…ints via open functions.

…le errors in generated code.

…. Narrow down imports to specific class used.

…r test ID or their test tag.

sichanyoo

Comments for reviewers.

sichanyoo · 2024-10-16T21:44:40Z

...y-swift-codegen/src/main/kotlin/software/amazon/smithy/swift/codegen/DirectedSwiftCodegen.kt

+            LOGGER.info("[${service.id}] Generating smoke tests for service")
+            generateSmokeTests(ctx)


This is the place where smoke test generation gets triggered

sichanyoo · 2024-10-16T21:45:46Z

...main/kotlin/software/amazon/smithy/swift/codegen/integration/HTTPBindingProtocolGenerator.kt

+    override fun generateSmokeTests(ctx: ProtocolGenerator.GenerationContext) {
+        return SmokeTestGenerator(ctx).generateSmokeTests()
+    }


This concrete implementation generates smoke tests according to generic Smithy spec.

sichanyoo · 2024-10-16T21:47:23Z

...odegen/src/main/kotlin/software/amazon/smithy/swift/codegen/integration/ProtocolGenerator.kt

+    /**
+     * Generate smoke tests defined on the service model
+     */
+    fun generateSmokeTests(ctx: GenerationContext)


This gets implemented both in smithy-swift and aws-sdk-swift, each generating tests in generic Smithy way vs. AWS way, respectively.

sichanyoo · 2024-10-16T21:48:25Z

...degen/src/main/kotlin/software/amazon/smithy/swift/codegen/integration/SmokeTestGenerator.kt

@@ -0,0 +1,267 @@
+package software.amazon.smithy.swift.codegen.integration


Generates smoke tests according to generic Smithy spec. A child class in aws-sdk-swift extends this class to customize AWS specific logic by overriding vendor specific methods.

sichanyoo · 2024-10-16T21:50:32Z

...degen/src/main/kotlin/software/amazon/smithy/swift/codegen/integration/SmokeTestGenerator.kt

+    private fun renderPrintTestResult(
+        writer: SwiftWriter,
+        isSuccess: Boolean,
+        serviceName: String,
+        operationName: String,
+        errorExpected: Boolean,
+        isSkipped: Boolean = false,
+        printCaughtError: Boolean = false
+    ) {
+        val result = if (isSuccess) "ok" else "not ok"
+        val error = if (errorExpected) "error expected from service" else "no error expected from service"
+        val skipped = if (isSkipped) " # skip" else ""
+        writer.write("print(\$S)", "$result $serviceName $operationName - $error$skipped")
+        if (printCaughtError) {
+            writer.write("print(\"# Caught unexpected error: \\(error)\")")
+        }
+        writer.write("return $isSuccess")
+    }


Test result line produced here follows the TAP protocol as well as additional restrictions defined in SEPs.

sichanyoo · 2024-10-16T21:51:39Z

...degen/src/main/kotlin/software/amazon/smithy/swift/codegen/integration/SmokeTestGenerator.kt

+    private fun renderDoCatchBlock(operationShapeId: ShapeId, testCase: SmokeTestCase, serviceName: String, writer: SwiftWriter) {
+        val operationName = operationShapeId.name
+        val errorExpected = testCase.expectation.isFailure
+        val specificErrorExpected = errorExpected && testCase.expectation.failure.get().errorId.isPresent
+
+        writer.write("do {")
+        writer.indent()
+        // Construct input struct with params from trait.
+        val inputShape = ctx.model.expectShape(ctx.model.expectShape(operationShapeId).asOperationShape().get().inputShape)
+        if (testCase.params?.get()?.size() == 0) {
+            writer.write("let input = ${inputShape.id.name}()")
+        } else {
+            writer.writeInline("let input = ")
+                .call {
+                    ShapeValueGenerator(ctx.model, ctx.symbolProvider).writeShapeValueInline(
+                        writer,
+                        inputShape,
+                        testCase.params.orElse(ObjectNode.builder().build())
+                    )
+                }
+                .write("")
+        }
+        // Create empty config
+        val clientName = getClientName()
+        writer.write("let config = try await $clientName.${clientName}Configuration()")
+        // Set any vendor-specific values into config.
+        handleVendorParams(testCase.vendorParams.orElse(null), writer)
+        // Construct client with the config
+        writer.write("let client = $clientName(config: config)")
+        // Call the operation with client and input
+        writer.write("_ = try await client.${operationName.toLowerCamelCase()}(input: input)")
+        // Writer after client call:
+        if (errorExpected) {
+            // If error was expected, print failure line and return false.
+            renderPrintTestResult(writer, false, serviceName, operationName, true)
+        } else {
+            // If expected success, print success line and return true.
+            renderPrintTestResult(writer, true, serviceName, operationName, false)
+        }
+        writer.dedent()
+
+        // Catch specific error only if it is expected
+        if (specificErrorExpected) {
+            val expectedErrorName = testCase.expectation.failure.get().errorId.get().name
+            renderSpecificErrorCatchBlock(expectedErrorName, operationName, serviceName, writer)
+        }
+
+        // Catch generic error
+        writer.write("} catch {")
+        writer.indent()
+        if (specificErrorExpected) {
+            // Specific error was expected but some other error got caught; print failure & the unexpected error and return false.
+            renderPrintTestResult(writer, false, serviceName, operationName, true, printCaughtError = true)
+        } else if (errorExpected) {
+            // If generic error was expected, print success and return true.
+            renderPrintTestResult(writer, true, serviceName, operationName, true)
+        } else {
+            // If expected success, print failure, the unexpected error, and return false.
+            renderPrintTestResult(writer, false, serviceName, operationName, false, printCaughtError = true)
+        }
+        writer.dedent()
+        writer.write("}")
+    }


renderDoCatchBlock() contains the logic for test result & error handling codegen.

sichanyoo · 2024-10-16T21:54:19Z

...degen/src/main/kotlin/software/amazon/smithy/swift/codegen/integration/SmokeTestGenerator.kt

+    open fun handleVendorParams(vendorParams: ObjectNode, writer: SwiftWriter) {
+        // Pseudo-code example:
+        //      writer.write("config.value1 = ${value1-extracted-from-vendorParams}")
+    }


The vendor params are custom client config fields included for a given service client. In our case, all the config fields added to service clients in aws-sdk-swift.

sichanyoo · 2024-10-16T21:55:09Z

...egen/src/main/kotlin/software/amazon/smithy/swift/codegen/swiftmodules/ClientRuntimeTypes.kt

@@ -91,6 +91,7 @@ object ClientRuntimeTypes {
        val InterceptorProvider = runtimeSymbol("InterceptorProvider", SwiftDeclaration.PROTOCOL)


This and below change are for test files that need to import these.

jbelkins

All of this is good.

I'm tempted to include a codegen switch to turn smoke test generation on/off, but as long as the tests are .gitignored and they remain reasonably small, I'm okay with generating them every time for now.

jbelkins

one minor code cleanup

jbelkins · 2024-10-29T15:34:33Z

...degen/src/main/kotlin/software/amazon/smithy/swift/codegen/integration/SmokeTestGenerator.kt

+    //        exit(0)
+    //    } else {
+    //        exit(1)
+    //    }


remove commented code

Comment was to show what the method generates

author Josh Elkins <jbelkins@amazon.com> 1725075023 -0500 committer Josh Elkins <jbelkins@amazon.com> 1730244524 -0500 parent 0de1791 author Josh Elkins <jbelkins@amazon.com> 1725075023 -0500 committer Josh Elkins <jbelkins@amazon.com> 1730244464 -0500 parent 0de1791 author Josh Elkins <jbelkins@amazon.com> 1725075023 -0500 committer Josh Elkins <jbelkins@amazon.com> 1730244451 -0500 parent 0de1791 author Josh Elkins <jbelkins@amazon.com> 1725075023 -0500 committer Josh Elkins <jbelkins@amazon.com> 1730244433 -0500 parent 0de1791 author Josh Elkins <jbelkins@amazon.com> 1725075023 -0500 committer Josh Elkins <jbelkins@amazon.com> 1730244416 -0500 parent 0de1791 author Josh Elkins <jbelkins@amazon.com> 1725075023 -0500 committer Josh Elkins <jbelkins@amazon.com> 1730244397 -0500 parent 0de1791 author Josh Elkins <jbelkins@amazon.com> 1725075023 -0500 committer Josh Elkins <jbelkins@amazon.com> 1730244381 -0500 parent 0de1791 author Josh Elkins <jbelkins@amazon.com> 1725075023 -0500 committer Josh Elkins <jbelkins@amazon.com> 1730244297 -0500 parent 0de1791 author Josh Elkins <jbelkins@amazon.com> 1725075023 -0500 committer Josh Elkins <jbelkins@amazon.com> 1730244277 -0500 WIP chore: Organize codegen tests & Kotlin unit tests (#830) * Organize codegen tests & refactor setup tests for XML codegen tests to reduce duplicate code. * ktlint --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> feat: Make models Sendable (#829) chore: Specify Xcode_16 application name same as on Github runner (#831) chore: Move HTTP status code test out of SmithyTestUtil (#836) chore: Protocol tests set their own AWS credentials (#837) Don't import Darwin on Windows. (#835) chore: Run macOS 15 with Xcode 16 on CI (#838) fix: Don't unwrap unneeded value during event marshal (#839) fix: Don't supply a default value when reading a non-optional value (#840) feat: Enable sigv4a for SESv2 service (#842) feat!: Bump CRT version to 0.37.0 & add CRC64NVME checksum algorithm wrapper (#845) * Bump CRT version to 0.37.0 and add wrapper for new checksum algorithm. * Add doc comment re: checksum algorithms may expand in future. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> chore: Use Smithy 1.52.1 (#846) feat: Smoke Tests V2 (#841) * Basic scaffold. * Add SmokeTestGenerator for generic Smithy spec, with customization points via open functions. * Refactor both codegen logic and generated code to fix runtime / compile errors in generated code. * Generate test runner only if there's >=1 test case(s) for the service. Narrow down imports to specific class used. * Add manual ignore lists for disabling codegen for smoke tests by their test ID or their test tag. * Add generic codegen for smoke tests * Rename variables to be more specific. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> chore: Updates version to 0.86.0

Sichan Yoo added 5 commits October 8, 2024 11:38

Basic scaffold.

73094d4

Add SmokeTestGenerator for generic Smithy spec, with customization po…

659e4eb

…ints via open functions.

Refactor both codegen logic and generated code to fix runtime / compi…

c5a03af

…le errors in generated code.

Generate test runner only if there's >=1 test case(s) for the service…

4451a38

…. Narrow down imports to specific class used.

Add manual ignore lists for disabling codegen for smoke tests by thei…

884955b

…r test ID or their test tag.

sichanyoo mentioned this pull request Oct 16, 2024

feat: Smoke Tests V2 awslabs/aws-sdk-swift#1791

Merged

Sichan Yoo added 2 commits October 16, 2024 12:02

Merge branch 'main' into feat/smoke-tests-v2

c087928

Add generic codegen for smoke tests

4bad1ef

sichanyoo requested a review from jbelkins October 16, 2024 21:43

sichanyoo commented Oct 16, 2024

View reviewed changes

Sichan Yoo added 2 commits October 16, 2024 16:26

Rename variables to be more specific.

5292ed7

Merge branch 'main' into feat/smoke-tests-v2

ee1faab

jbelkins approved these changes Oct 28, 2024

View reviewed changes

Merge branch 'main' into feat/smoke-tests-v2

b3dbce7

jbelkins requested changes Oct 29, 2024

View reviewed changes

sichanyoo requested a review from jbelkins October 29, 2024 16:38

jbelkins approved these changes Oct 29, 2024

View reviewed changes

sichanyoo merged commit 2e19981 into main Oct 29, 2024
27 checks passed

sichanyoo deleted the feat/smoke-tests-v2 branch October 29, 2024 16:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Smoke Tests V2 #841

feat: Smoke Tests V2 #841

sichanyoo commented Oct 16, 2024 •

edited

Loading

sichanyoo left a comment

sichanyoo Oct 16, 2024

sichanyoo Oct 16, 2024

sichanyoo Oct 16, 2024

sichanyoo Oct 16, 2024

sichanyoo Oct 16, 2024

sichanyoo Oct 16, 2024

sichanyoo Oct 16, 2024

sichanyoo Oct 16, 2024

jbelkins left a comment

jbelkins left a comment

jbelkins Oct 29, 2024

sichanyoo Oct 29, 2024

		LOGGER.info("[${service.id}] Generating smoke tests for service")
		generateSmokeTests(ctx)

		@@ -0,0 +1,267 @@
		package software.amazon.smithy.swift.codegen.integration

		@@ -91,6 +91,7 @@ object ClientRuntimeTypes {
		val InterceptorProvider = runtimeSymbol("InterceptorProvider", SwiftDeclaration.PROTOCOL)

feat: Smoke Tests V2 #841

feat: Smoke Tests V2 #841

Conversation

sichanyoo commented Oct 16, 2024 • edited Loading

Issue #

Description of changes

Scope

sichanyoo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbelkins left a comment

Choose a reason for hiding this comment

jbelkins left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sichanyoo commented Oct 16, 2024 •

edited

Loading