REDO: Generate seed connector specs on build (airbytehq#7613)

* add specs module with logic to fetch specs on build

* format + build and add gradle dependency for new script

* check seed file for existing specs + refactor

* add tests + a bit more refactoring

* run gw format

* update yaml config persistence to merge specs into definitions

* add comment

* add dep

* add tests for GcsBucketSpecFetcher

* get rid of static block + format

* DRY up parse call

* add GCS details to comment

* formatting + fix test

* update comment

* do not format seed specs files

* change signature of run to allow cloud to reuse this script

* run gw format

* revert commits that change signature of run

* fix comment typo

Co-authored-by: Davin Chia <davinchia@gmail.com>

* rename enum to be distinct from the enum in cloud

* add missing dependencies between modules

* add readme for seed connector spec generator

* reword

* reference readme in comment

* ignore 'spec' field in newFields logic

* rearrange dependencies so that CONNECTORS_BASE build does not depend on SeedConnectorSpecGenerator

* run format

* add some more helpful info to the GCS fetch failure message

* add more info

* get rid of unnecessary static block

* Fix publishing docs (airbytehq#7589)

* Fix publishing docs

* Reorder steps and add a comment about rebuilding the platform

* Update README.md

Co-authored-by: Lake Mossman <lake@airbyte.io>

* add dependency and rebuild

* update PR template with seed connector generation steps

* revert formatting changes to PR template

* Update build.gradle

* Remove unnecessary dep

Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>

.github/pull_request_template.md

@@ -29,7 +29,6 @@ Expand the relevant checklist and delete the others.
     - [ ] `docs/integrations/README.md`
     - [ ] `airbyte-integrations/builds.md`
 - [ ] PR name follows [PR naming conventions](https://docs.airbyte.io/contributing-to-airbyte/updating-documentation#issues-and-pull-requests)
-- [ ] Connector added to connector index like described [here](https://docs.airbyte.io/connector-development#publishing-a-connector)
 
 #### Airbyter
 
@@ -40,6 +39,8 @@ If this is a community PR, the Airbyte engineer reviewing this PR is responsible
 - [ ] Credentials added to Github CI. [Instructions](https://docs.airbyte.io/connector-development#using-credentials-in-ci). 
 - [ ] [`/test connector=connectors/<name>` command](https://docs.airbyte.io/connector-development#updating-an-existing-connector) is passing. 
 - [ ] New Connector version released on Dockerhub by running the `/publish` command described [here](https://docs.airbyte.io/connector-development#updating-an-existing-connector)
+- [ ] After the connector is published, connector added to connector index as described [here](https://docs.airbyte.io/connector-development#publishing-a-connector)
+- [ ] Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described [here](https://docs.airbyte.io/connector-development#publishing-a-connector)
 
 </p>
 </details>
@@ -59,7 +60,6 @@ If this is a community PR, the Airbyte engineer reviewing this PR is responsible
     - [ ] Connector's `bootstrap.md`. See [description and examples](https://docs.google.com/document/d/1ypdgmwmEHWv-TrO4_YOQ7pAJGVrMp5BOkEVh831N260/edit?usp=sharing)
     - [ ] Changelog updated in `docs/integrations/<source or destination>/<name>.md` including changelog. See changelog [example](https://docs.airbyte.io/integrations/sources/stripe#changelog)
 - [ ] PR name follows [PR naming conventions](https://docs.airbyte.io/contributing-to-airbyte/updating-documentation#issues-and-pull-requests)
-- [ ] Connector version bumped like described [here](https://docs.airbyte.io/connector-development#publishing-a-connector)
 
 #### Airbyter
 
@@ -70,6 +70,8 @@ If this is a community PR, the Airbyte engineer reviewing this PR is responsible
 - [ ] Credentials added to Github CI. [Instructions](https://docs.airbyte.io/connector-development#using-credentials-in-ci). 
 - [ ] [`/test connector=connectors/<name>` command](https://docs.airbyte.io/connector-development#updating-an-existing-connector) is passing. 
 - [ ] New Connector version released on Dockerhub by running the `/publish` command described [here](https://docs.airbyte.io/connector-development#updating-an-existing-connector)
+- [ ] After the new connector version is published, connector version bumped in the seed directory as described [here](https://docs.airbyte.io/connector-development#publishing-a-connector)
+- [ ] Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described [here](https://docs.airbyte.io/connector-development#publishing-a-connector)
 
 </p>
 </details>

airbyte-config/init/build.gradle

@@ -6,4 +6,8 @@ dependencies {
     implementation 'commons-cli:commons-cli:1.4'
 
     implementation project(':airbyte-config:models')
+    implementation project(':airbyte-config:persistence')
+    implementation project(':airbyte-protocol:models')
+    implementation project(':airbyte-commons-docker')
+    implementation project(':airbyte-json-validation')
 }

airbyte-config/init/src/main/java/io/airbyte/config/init/SeedType.java

@@ -7,7 +7,9 @@
 public enum SeedType {
 
   STANDARD_SOURCE_DEFINITION("/seed/source_definitions.yaml", "sourceDefinitionId"),
-  STANDARD_DESTINATION_DEFINITION("/seed/destination_definitions.yaml", "destinationDefinitionId");
+  STANDARD_DESTINATION_DEFINITION("/seed/destination_definitions.yaml", "destinationDefinitionId"),
+  SOURCE_SPEC("/seed/source_specs.yaml", "dockerImage"),
+  DESTINATION_SPEC("/seed/destination_specs.yaml", "dockerImage");
 
   final String resourcePath;
   // ID field name

airbyte-config/persistence/src/main/java/io/airbyte/config/persistence/YamlSeedConfigPersistence.java → airbyte-config/init/src/main/java/io/airbyte/config/init/YamlSeedConfigPersistence.java

@@ -2,23 +2,27 @@
  * Copyright (c) 2021 Airbyte, Inc., all rights reserved.
  */
 
-package io.airbyte.config.persistence;
+package io.airbyte.config.init;
 
 import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.io.Resources;
+import io.airbyte.commons.docker.DockerUtils;
 import io.airbyte.commons.json.Jsons;
 import io.airbyte.commons.util.MoreIterators;
 import io.airbyte.commons.yaml.Yamls;
 import io.airbyte.config.AirbyteConfig;
 import io.airbyte.config.ConfigSchema;
-import io.airbyte.config.init.SeedType;
+import io.airbyte.config.persistence.ConfigNotFoundException;
+import io.airbyte.config.persistence.ConfigPersistence;
 import io.airbyte.validation.json.JsonValidationException;
 import java.io.IOException;
 import java.net.URL;
 import java.nio.charset.StandardCharsets;
 import java.util.List;
 import java.util.Map;
+import java.util.Map.Entry;
 import java.util.stream.Collectors;
 import java.util.stream.Stream;
 
@@ -45,11 +49,40 @@ public static YamlSeedConfigPersistence get(final Class<?> seedDefinitionsResour
     return new YamlSeedConfigPersistence(seedDefinitionsResourceClass);
   }
 
-  private YamlSeedConfigPersistence(final Class<?> seedDefinitionsResourceClass) throws IOException {
+  private YamlSeedConfigPersistence(final Class<?> seedResourceClass) throws IOException {
+    final Map<String, JsonNode> sourceDefinitionConfigs = getConfigs(seedResourceClass, SeedType.STANDARD_SOURCE_DEFINITION);
+    final Map<String, JsonNode> sourceSpecConfigs = getConfigs(seedResourceClass, SeedType.SOURCE_SPEC);
+    final Map<String, JsonNode> fullSourceDefinitionConfigs = sourceDefinitionConfigs.entrySet().stream()
+        .collect(Collectors.toMap(Entry::getKey, e -> mergeSpecIntoDefinition(e.getValue(), sourceSpecConfigs)));
+
+    final Map<String, JsonNode> destinationDefinitionConfigs = getConfigs(seedResourceClass, SeedType.STANDARD_DESTINATION_DEFINITION);
+    final Map<String, JsonNode> destinationSpecConfigs = getConfigs(seedResourceClass, SeedType.DESTINATION_SPEC);
+    final Map<String, JsonNode> fullDestinationDefinitionConfigs = destinationDefinitionConfigs.entrySet().stream()
+        .collect(Collectors.toMap(Entry::getKey, e -> mergeSpecIntoDefinition(e.getValue(), destinationSpecConfigs)));
+
     this.allSeedConfigs = ImmutableMap.<SeedType, Map<String, JsonNode>>builder()
-        .put(SeedType.STANDARD_SOURCE_DEFINITION, getConfigs(seedDefinitionsResourceClass, SeedType.STANDARD_SOURCE_DEFINITION))
-        .put(SeedType.STANDARD_DESTINATION_DEFINITION, getConfigs(seedDefinitionsResourceClass, SeedType.STANDARD_DESTINATION_DEFINITION))
-        .build();
+        .put(SeedType.STANDARD_SOURCE_DEFINITION, fullSourceDefinitionConfigs)
+        .put(SeedType.STANDARD_DESTINATION_DEFINITION, fullDestinationDefinitionConfigs).build();
+  }
+
+  /**
+   * Merges the corresponding spec JSON into the definition JSON. This is necessary because specs are
+   * stored in a separate resource file from definitions.
+   *
+   * @param definitionJson JSON of connector definition that is missing a spec
+   * @param specConfigs map of docker image to JSON of docker image/connector spec pair
+   * @return JSON of connector definition including the connector spec
+   */
+  private JsonNode mergeSpecIntoDefinition(final JsonNode definitionJson, final Map<String, JsonNode> specConfigs) {
+    final String dockerImage = DockerUtils.getTaggedImageName(
+        definitionJson.get("dockerRepository").asText(),
+        definitionJson.get("dockerImageTag").asText());
+    final JsonNode specConfigJson = specConfigs.get(dockerImage);
+    if (specConfigJson == null || specConfigJson.get("spec") == null) {
+      throw new UnsupportedOperationException(String.format("There is no seed spec for docker image %s", dockerImage));
+    }
+    ((ObjectNode) definitionJson).set("spec", specConfigJson.get("spec"));
+    return definitionJson;
   }
 
   @SuppressWarnings("UnstableApiUsage")