Merge develop to app framework feature branch (#578)

* Merge master1.0.1 (#370) 1.0.1 Release * release/1.0.2 (#473) * release/1.0.2 - Addressing doc references to release 1.0.2 * - doc change * - Fixing the olm-catalog CRDs for the versions 1.0.0-RC, 1.0.0, and 1.0.1 (#475) * Fix role yaml * CSPL:1217 Added new cases for app version downgrade (#452) Co-authored-by: Sirish Mohan <68884189+smohan-splunk@users.noreply.github.com> * move m4 test to integration to clear smoke run (#487) * Feature circleci migration (#490) * Migration of Unit Test and Smoke Test pipelines from CircleCi to Github Actions (#441) * Update cron schedule * Fix a bug where standalone with replicas>1 won't come up (#489) * Added int test workflow and nightly workflows. (#493) * Modify monitoring console selection name to avoid eks cluster creation failure (#494) * Update the name of int test step (#496) * CSPL-1219 (#470) * [CSPL-1283] Fix AWS & minio S3 client code to support App framework on GCS (#498) * Fix minio S3 client code & incorrect minio initContainer ut Add changes to minio client code to handle generic S3 compatable remote stores, namely GCS. * CSPL-1301: Trigger app install for modified app pkgs (#503) * Trigger app install for modified app pkgs While an app package modification does trigger rewritteing the configmap. There are cases where this rewrite will result in the same data section as the previous version of the configmap (for example, when a single app is installed initially then modified.) When this is applied no change is detected and the Pod does not reset or install the new app. By adding a label to the configmap metadata, when an app package change is detected, we can increment this label. This will not affect the data in the configmap since the label is in the metadata section, however the label change will force a new ResourceVersion of the configmap and restart the pod, triggering the modified app install. [UPDATE] Instead of using a label in the ConfigMap metadata, reset the data in the app listing ConfigMap to nothing prior to setting it, forcing a new resourceVerison. * CSPL-1302: Bias-language removal Phase 1 [Comments & Docs] (#497) * Bias-language removal Phase 1 * CSPL-1316 : Avoid app framework flow from re-entrancy (#506) * Automated pre release workflow (#508) * CSPL-1230 Remove need for Secret keys in IAM env (#505) * CSPL-1230 Remove need for Secret keys in IAM env The operator code can be changed to allow no secretRef to be a valid volume config. If no secretRef is configured for a SmartStore or AppFramework volume, then assume the credentials are available on the env itself, possibly through a tool such as kube2iam. * Moved action file to right location (#510) * Update the URL on relese RULE (#512) * CSPL-1339: Fix make_bundle.sh to check for version change (#517) * Splunk Operator 1.0.3 release (#511) * Cspl 1335 official release workflow (#523) * Automated Release Workflow * Added automated workflow to merge develop to master * Fix typo in kubectl create secret command (#524) * Update README.md (#526) * Fix image push jobs from master branch (#530) * CSPL-1298 (#521) * CSPL-1307: Bias-language removal Phase 1 (#504) Implement Github Actions workflow for Bias Language * Update spunk-operator image tag in release directory (#535) * CSPL-1327 App framework: App installation isn't triggered if 1 appsource is empty. (#519) * CSPL-1327 App install isn't triggered if 1 appsource is empty An empty `appSource` is not necessarily an error. Treat it as a no-op and throw a log message. * CSPL-1305 - Remove bias language from Paths & URLs (#509) * Bias-Lang Removal P1 - Handling Exceptions Consolidate Paths & URLs from external sources Include exceptions to GithubActions workflow * CSPL-1305 - Resolved Nightly builds regression (#547) Fixing Bias Language URLs to use correct tokens * CSPL-1277: Check for status of bundle push on CM (#536) * CSPL-1310 Merge MC feature branch to develop (#548) * CSPL-1306 - Remove bias language from Functions & Local variables (#543) * Bias-Lang P1 - Removal from Functions & Vars Renamed Functions and variables Renamed non-CRD files Added local script for bias language * Generating CRDs after changes * Removing CRD changes for olm legacy versions * Added new image push to int test (#566) * CSPL:1387 Modify app upload logic to selectively upload apps within test case (#557) Co-authored-by: Sirish Mohan <68884189+smohan-splunk@users.noreply.github.com> * CSPL-1379 IDXC fails to scale up when MC CR is deployed with pre-existing IDXC (#562) * IDXC fails to scale up when MC CR is deployed in the namespace with pre existing IDXC * Review comments * CSPL-1438: Add missing mc test develop (#565) * Add Missing MC test in custom resource test * Add missing MC Test in License Manger * Add missing MC test in secret test cases * Cleanup for MC Missing test * CSPL-1412/CSPL-1413 Break down app framework tests (#567) * CSPL-1412 * Update appframework_test.go * CSPL-1447-updated-version-for-find-replace-action (#576) * Updated go version to 1.17.3 (#581) * Cspl 1456 fix nightly test failures by changing testenv length and cleanup env (#579) * CSPL-1443 increase length of random string in appframework test to avoid reuse of same namespace * CSPL-1443 change cleanup logic to delete env every time unless DEBUG is set to true * disabling failing smoke tests in S1 and C3
splunk · Nov 10, 2021 · 400bf23 · 400bf23
1 parent 77bb843
commit 400bf23
Show file tree

Hide file tree

Showing 140 changed files with 10,587 additions and 2,798 deletions.
diff --git a/.biased_lang_exclude b/.biased_lang_exclude
@@ -0,0 +1,14 @@
+pkg/splunk/common/urls.go
+pkg/splunk/common/urls_testingcode.go
+pkg/splunk/common/paths.go
+deploy/olm-catalog/
+go.sum
+.github/
+.circleci/
+#Below should be reviewed once CRDs have no bias lang
+pkg/
+deploy/
+test/
+docs/
+README.md
+build/make_bundle.sh
diff --git a/.env b/.env
@@ -1,2 +1,3 @@
 OPERATOR_SDK_VERSION=v0.18.2
 REVIEWERS=pdhanoyasplunk,smohan-splunk,kashok-splunk,akondur,sgontla,vebken-splunk,gaurav-splunk,mgaldino-splunk,jambrosiano,tpereirasplunk,jryb
+GO_VERSION=1.17.3
diff --git a/.github/workflows/automated-release-workflow.yml b/.github/workflows/automated-release-workflow.yml
@@ -47,7 +47,7 @@ jobs:
     - name: Setup Go
       uses: actions/setup-go@v2
       with:
-        go-version: '1.13.6'
+        go-version: ${{ steps.dotenv.outputs.GO_VERSION }}
 
     - name: Install golint
       run: |
@@ -73,6 +73,13 @@ jobs:
       run: |
         build/package.sh
         docker images
+
+    - name: Update Operator Image name in Release Folder
+      uses: jacobtomlinson/gha-find-replace@master
+      with:
+        find: "splunk/splunk-operator:${{ env.SHORT_SHA }}"
+        replace: "splunk/splunk-operator:${{ github.event.inputs.operator_image_tag }}"
+        include: "release-${{ env.SHORT_SHA }}/**"
 
     - name: Upload Release Artifacts
       uses: actions/upload-artifact@v2
@@ -105,4 +112,4 @@ jobs:
 
     - name: Push Release Image
       run: |
-        docker push splunk/splunk-operator:${{ github.event.inputs.operator_image_tag }}
+        docker push splunk/splunk-operator:${{ github.event.inputs.operator_image_tag }}
diff --git a/.github/workflows/bias-language-workflow.yml b/.github/workflows/bias-language-workflow.yml
@@ -0,0 +1,11 @@
+name: Bias Language
+on: [push]
+jobs:
+  biased_lang:
+    runs-on: ubuntu-latest
+    name: Detecting Biased Language
+    steps:
+      - uses: actions/checkout@v2
+      - id: biased-lang-linter
+        uses: splunk/biased-lang-linter@main
+        continue-on-error: false
diff --git a/.github/workflows/build-test-push-workflow.yml b/.github/workflows/build-test-push-workflow.yml
@@ -5,10 +5,13 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v2
+    - name: Dotenv Action
+      id: dotenv
+      uses: falti/dotenv-action@v0.2.7
     - name: Setup Go
       uses: actions/setup-go@v2
       with:
-        go-version: '1.13.6'
+        go-version: ${{ steps.dotenv.outputs.GO_VERSION }}
     - name: Install golint
       run: |
         go version
@@ -22,10 +25,13 @@ jobs:
     needs: check-formating
     steps:
       - uses: actions/checkout@v2
+      - name: Dotenv Action
+        id: dotenv
+        uses: falti/dotenv-action@v0.2.7
       - name: Setup Go
         uses: actions/setup-go@v2
         with:
-          go-version: '1.13.6'
+          go-version: ${{ steps.dotenv.outputs.GO_VERSION }}
       - name: Install golint
         run: |
           go version
@@ -89,6 +95,9 @@ jobs:
       S3_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
     steps:
     - uses: actions/checkout@v2
+    - name: Dotenv Action
+      id: dotenv
+      uses: falti/dotenv-action@v0.2.7
     - name: setup-docker
       uses: docker-practice/actions-setup-docker@v1
     - name: Configure AWS credentials
@@ -120,7 +129,7 @@ jobs:
     needs: vulnerability-scan
     strategy:
       matrix:
-        test: [basic, appframework, secret]
+        test: [basic, appframework, secret, monitoringconsole]
     runs-on: ubuntu-latest
     env:
       CLUSTER_NODES: 1
@@ -170,7 +179,7 @@ jobs:
       - name: Setup Go
         uses: actions/setup-go@v2
         with:
-          go-version: '1.13.6'
+          go-version: ${{ steps.dotenv.outputs.GO_VERSION }}
       - name: Install Go Lint
         run: |
           go version

diff --git a/.github/workflows/int-test-workflow.yml b/.github/workflows/int-test-workflow.yml
@@ -9,7 +9,7 @@ jobs:
   int-tests:
     strategy:
       matrix:
-        test: [appframework, secret, smartstore, monitoringconsole, scaling]
+        test: [appframework, secret, smartstore, monitoringconsole, scaling, crcrud, licensemaster]
     runs-on: ubuntu-latest
     env:
       CLUSTER_NODES: 1
@@ -59,7 +59,7 @@ jobs:
       - name: Setup Go
         uses: actions/setup-go@v2
         with:
-          go-version: '1.13.6'
+          go-version: ${{ steps.dotenv.outputs.GO_VERSION }}
       - name: Install Go Lint
         run: |
           go version
@@ -119,4 +119,39 @@ jobs:
       - name: Cleanup up EKS cluster
         if: ${{ always() }}
         run: | 
-           make cluster-down
+           make cluster-down
+  publish_operator_image: 
+    runs-on: ubuntu-latest
+    needs: int-tests
+    if: github.ref == 'refs/heads/develop'
+    env:
+      SPLUNK_OPERATOR_IMAGE_NAME: splunk/splunk-operator
+      TAG: int
+    steps:
+    - uses: actions/checkout@v2
+    - name: Dotenv Action
+      id: dotenv
+      uses: falti/dotenv-action@v0.2.7
+    - name: setup-docker
+      uses: docker-practice/actions-setup-docker@v1
+    - name: Configure AWS credentials
+      uses: aws-actions/configure-aws-credentials@v1
+      with:
+        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+        aws-region: ${{ secrets.AWS_DEFAULT_REGION }}
+    - name: Login to Amazon ECR
+      uses: aws-actions/amazon-ecr-login@v1
+    - name: Pull Splunk Operator Image Locally
+      run: |
+        docker pull ${{ secrets.ECR_REPOSITORY }}/${{ env.SPLUNK_OPERATOR_IMAGE_NAME }}:$GITHUB_SHA
+    - name: Change Operator Image Tag to edge
+      run: |
+        docker tag ${{ secrets.ECR_REPOSITORY }}/${{ env.SPLUNK_OPERATOR_IMAGE_NAME }}:$GITHUB_SHA ${{ env.SPLUNK_OPERATOR_IMAGE_NAME }}:${{ env.TAG }}
+    - name: Configure Docker Hub credentials
+      uses: docker/login-action@v1
+      with:
+        username: ${{ secrets.DOCKERHUB_USERNAME }}
+        password: ${{ secrets.DOCKERHUB_PUSH_TOKEN}}
+    - name: Push Splunk Operator Image to Docker Hub 
+      run: docker push ${{ env.SPLUNK_OPERATOR_IMAGE_NAME }}:${{ env.TAG }}
diff --git a/.github/workflows/nightly-int-test-workflow.yml b/.github/workflows/nightly-int-test-workflow.yml
@@ -6,7 +6,7 @@ jobs:
   int-tests:
     strategy:
       matrix:
-        test: [appframework, secret, smartstore, monitoringconsole, scaling]
+        test: [appframework, secret, smartstore, monitoringconsole, scaling, crcrud, licensemaster]
     runs-on: ubuntu-latest
     env:
       CLUSTER_NODES: 1
@@ -37,7 +37,9 @@ jobs:
             echo "CLUSTER_NODES=2" >> $GITHUB_ENV  
           fi
       - name: Checkcout code
-        uses: actions/checkout@v2   
+        uses: actions/checkout@v2  
+        with:
+          ref: develop
       - name: Dotenv Action
         id: dotenv
         uses: falti/dotenv-action@v0.2.7
@@ -56,7 +58,7 @@ jobs:
       - name: Setup Go
         uses: actions/setup-go@v2
         with:
-          go-version: '1.13.6'
+          go-version: ${{ steps.dotenv.outputs.GO_VERSION }}
       - name: Install Go Lint
         run: |
           go version
@@ -125,6 +127,9 @@ jobs:
       TAG: edge
     steps:
     - uses: actions/checkout@v2
+    - name: Dotenv Action
+      id: dotenv
+      uses: falti/dotenv-action@v0.2.7
     - name: setup-docker
       uses: docker-practice/actions-setup-docker@v1
     - name: Configure AWS credentials

diff --git a/.github/workflows/pre-release-workflow.yml b/.github/workflows/pre-release-workflow.yml
@@ -27,6 +27,7 @@ jobs:
   automated-release:
     name: Automated Pre Release
     runs-on: ubuntu-latest
+    if: github.ref == 'refs/heads/develop'
     steps:
     - name: Checkout code
       uses: actions/checkout@v2
@@ -64,7 +65,7 @@ jobs:
     - name: Setup Go
       uses: actions/setup-go@v2
       with:
-        go-version: '1.13.6'
+        go-version: ${{ steps.dotenv.outputs.GO_VERSION }}
 
     - name: Install golint
       run: |
@@ -87,7 +88,7 @@ jobs:
       with:
         find: "${{ github.event.inputs.old_operator_version }}"
         replace: "${{ github.event.inputs.new_operator_version }}"
-        include: "version.go"
+        include: "**version.go"
 
     - name: Update Operator Image name in DOCS
       if: github.event.inputs.old_operator_version != github.event.inputs.new_operator_version
@@ -96,7 +97,7 @@ jobs:
         find: "splunk-operator:${{ github.event.inputs.old_operator_version }}"
         replace: "splunk-operator:${{ github.event.inputs.new_operator_version }}"
         exclude: "ChangeLog.md"
-        include: ".*.md"
+        include: "**.md"
 
     - name: Update Splunk Operator Install URL in DOCS
       if: github.event.inputs.old_operator_version != github.event.inputs.new_operator_version
@@ -105,7 +106,7 @@ jobs:
         find: "/download/${{ github.event.inputs.old_operator_version }}/splunk-operator"
         replace: "/download/${{ github.event.inputs.new_operator_version }}/splunk-operator"
         exclude: "ChangeLog.md"
-        include: ".*.md"
+        include: "**.md"
 
     - name: Update Splunk Operator VERSION in DOCS
       if: github.event.inputs.old_operator_version != github.event.inputs.new_operator_version
@@ -114,15 +115,15 @@ jobs:
         find: "${{ github.event.inputs.old_operator_version }} or later"
         replace: "${{ github.event.inputs.new_operator_version }} or later"
         exclude: "ChangeLog.md"
-        include: ".*.md"
+        include: "**.md"
 
     - name: Update Splunk Enterprise Image in operator.yaml
       if: github.event.inputs.old_enterprise_version != github.event.inputs.new_enterprise_version
       uses: jacobtomlinson/gha-find-replace@master
       with:
         find: "${{ github.event.inputs.old_enterprise_version }}"
         replace: "${{ github.event.inputs.new_enterprise_version }}"
-        include: "operator.yaml"
+        include: "**operator.yaml"
 
     - name: Update Splunk Enterprise image in DOCS
       if: github.event.inputs.old_enterprise_version != github.event.inputs.new_enterprise_version
@@ -131,6 +132,7 @@ jobs:
         find: "${{ github.event.inputs.old_enterprise_version }} or later"
         replace: "${{ github.event.inputs.new_enterprise_version }} or later"
         exclude: "ChangeLog.md"
+        include: "**.md"
 
     - name: Run Make Package
       run: |
@@ -144,12 +146,11 @@ jobs:
         docker images
 
     - name: Update Operator Image name in Release Folder
-      if: github.event.inputs.old_operator_version != github.event.inputs.new_operator_version
       uses: jacobtomlinson/gha-find-replace@master
       with:
         find: "splunk/splunk-operator:${{ env.SHORT_SHA }}"
         replace: "splunk/splunk-operator-rc:${{ github.event.inputs.release_version }}"
-        include: "release-${{ env.SHORT_SHA }}/"
+        include: "release-${{ env.SHORT_SHA }}/**"
 
     - name: Upload Release Artifacts
       uses: actions/upload-artifact@v2

diff --git a/.gitignore b/.gitignore
@@ -91,3 +91,5 @@ clair-scanner-logs
 release-*
 deploy/olm-certified
 *junit.xml
+# Bias Language Linter
+.biaslanguage/
diff --git a/Makefile b/Makefile
@@ -47,6 +47,10 @@ test:
 	@echo Running unit tests for splunk-operator
 	@go test -v -covermode=count -coverprofile=coverage.out --timeout=300s github.com/splunk/splunk-operator/pkg/splunk/common github.com/splunk/splunk-operator/pkg/splunk/enterprise github.com/splunk/splunk-operator/pkg/splunk/controller github.com/splunk/splunk-operator/pkg/splunk/client github.com/splunk/splunk-operator/pkg/splunk/util
 
+lang:
+	@echo Running bias language linter
+	@build/bias_language_linter.sh
+
 stop_clair_scanner:
 	@docker stop clair_db || true
 	@docker rm clair_db || true

diff --git a/build/bias_language_linter.sh b/build/bias_language_linter.sh
@@ -0,0 +1,28 @@
+#!/bin/bash
+# Script to detect bias language
+# Reference: https://github.com/splunk/biased-lang-linter/blob/main/README.md
+
+# exit when any command fails
+set -e
+
+LINTER=https://github.com/splunk/biased-lang-linter.git
+LOCAL_DIR=./.biaslanguage
+REPO=$(pwd)
+
+#Clone bias language linter if it's the first run
+if [ ! -d "$LOCAL_DIR" ]; then
+    echo "First time running - cloning linter."
+    mkdir -p $LOCAL_DIR
+    git clone $LINTER $LOCAL_DIR
+fi
+
+#Install Ripgrep requirements
+if [ ! $(which rg) ]; then
+  echo "Ripgrep is required - installing using brew"
+  brew install ripgrep
+fi
+
+#Run the linter locally
+cd $LOCAL_DIR
+python3 run_json.py --path=$REPO
+cd -