ERROR: "http: server gave HTTP response to HTTPS client"

[awdv00]

Describe the bug
Since updating from version 1.2.3 to 1.2.4, I see the following error message logged by the artemis operator: "http: server gave HTTP response to HTTPS client"

For example:

Post "https://admin:***@artemis-broker-ss-0.artemis-broker-hdls-svc.my-cluster.svc.cluster.local:8161/console/jolokia/exec/org.apache.activemq.artemis:broker=%5C%22amq-broker%5C%22": http: server gave HTTP response to HTTPS client

github.com/go-logr/logr.Logger.Error
	/opt/app-root/src/go/pkg/mod/github.com/go-logr/logr@v1.2.4/logr.go:299
github.com/artemiscloud/activemq-artemis-operator/controllers.createAddressResource
	/opt/app-root/src/controllers/activemqartemisaddress_controller.go:249
github.com/artemiscloud/activemq-artemis-operator/controllers.(*ActiveMQArtemisAddressReconciler).createQueue
	/opt/app-root/src/controllers/activemqartemisaddress_controller.go:213
github.com/artemiscloud/activemq-artemis-operator/controllers.(*ActiveMQArtemisAddressReconciler).Reconcile
	/opt/app-root/src/controllers/activemqartemisaddress_controller.go:134
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227

The reconciliation loop seems not to work correctly. E.g. creating a new address does not work.
Interestingly, when I create a new namespace, with the broker & address resources in them, it creates the broker with the addresses.

I suspect that the issue is caused by changes in version 1.2.4, in conjunction with the -Dwebconfig.bindings.artemis.uri=http://0.0.0.0:8161 configuration I pass in my broker configuration. In version 1.2.3, the issue does not occur.

My broker config:

apiVersion: broker.amq.io/v1beta1
kind: ActiveMQArtemis
metadata:
  name: artemis-broker
spec:
  deploymentPlan:
    size: 1
    podDisruptionBudget:
      minAvailable: 1
    messageMigration: false
    enableMetricsPlugin: true
    persistenceEnabled: true
    extraMounts:
      configMaps:
        - "newlog4j-logging-config"
    podSecurityContext:
      runAsNonRoot: true
      runAsUser: 1000
      fsGroup: 1000
  acceptors:
    - name: core-stomp
      port: 61616
      protocols: core,stomp
      expose: false # default is already false, but making it explicit for more robustness
  env:
    - name: AMQ_USER
      valueFrom:
        secretKeyRef:
          name: artemis
          key: ARTEMIS_USERNAME
    - name: AMQ_PASSWORD
      valueFrom:
        secretKeyRef:
          name: artemis
          key: ARTEMIS_PASSWORD
    - name: JAVA_ARGS_APPEND
      value: -Dwebconfig.bindings.artemis.uri=http://0.0.0.0:8161 # see https://github.com/artemiscloud/activemq-artemis-operator/issues/96
  console:
    expose: false # do not create an ingress, console will still be available via port-forward
  brokerProperties:
    - 'wildcardConfiguration.delimiter=/'
  addressSettings:
    addressSetting:
      - match: '#'
        managementMessageAttributeSizeLimit: 1024
      - match: '/tmp/#'
        expiryDelay: 300000
        autoDeleteAddresses: true
        autoDeleteAddressesDelay: 30000
        autoDeleteQueues: true
        autoDeleteCreatedQueues: true
        autoDeleteQueuesDelay: 30000
        autoDeleteQueuesMessageCount: -1

My artemis operator config is the same as in the official example, with minor changes to:

• logging (-> json)
• watch all namespaces
• dynamic broker image

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    control-plane: controller-manager
  name: activemq-artemis-controller-manager
spec:
  replicas: 1
  selector:
    matchLabels:
      control-plane: controller-manager
      name: activemq-artemis-operator
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        control-plane: controller-manager
        name: activemq-artemis-operator
    spec:
      containers:
      - args:
        - --zap-log-level=info
        - --zap-encoder=json
        - --zap-time-encoding=iso8601
        - --leader-elect
        command:
        - /home/activemq-artemis-operator/bin/entrypoint
        env:
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2150
          value: quay.io/artemiscloud/activemq-artemis-broker-init:0.2.2
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2160
          value: quay.io/artemiscloud/activemq-artemis-broker-init:0.2.4
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2180
          value: quay.io/artemiscloud/activemq-artemis-broker-init:0.2.10
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2200
          value: quay.io/artemiscloud/activemq-artemis-broker-init:1.0.0
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2210
          value: quay.io/artemiscloud/activemq-artemis-broker-init:1.0.2
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2220
          value: quay.io/artemiscloud/activemq-artemis-broker-init:1.0.5
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2230
          value: quay.io/artemiscloud/activemq-artemis-broker-init:1.0.7
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2250
          value: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:e2438670831bad39bb08147bbebe67ef9465899fb7750f1a5a185569f99d5fb5
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2260
          value: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:634f3d8fd620607b56e0360a191d23a9cb8a32ba0ba1f24fe20791ec52343586
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2270
          value: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:9bbb915855f29ea16950a3fd50d0b9bac25f82cba7d197579e04d9c33a5b02c4
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2271
          value: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:fc9b9a4c7550d84fd90335dd4f9e0d8d39c09136c6034b28490a368368cb11f1
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2280
          value: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:264b33a9aad90d32a41258904f35092a141d76dcef0c115d414475444100aace
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2150
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes:0.2.0
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2160
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes:0.2.1
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2180
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes:0.2.7
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2200
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes:1.0.0
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2210
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes:1.0.2
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2220
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes:1.0.5
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2230
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes:1.0.7
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2250
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:2d155bffd246f26f516a6075ec170babc1373c72bb664cba4e3c58521ffd3558
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2260
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:f25db980c3229e66cf48690aa175f6f93e3dce3974c0e70ab23af6462e7d1893
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2270
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:a4628c2591ec06caf21a3467f6e3faeaede7fa402b4f137d943b25b4dd4ad343
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2271
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:2895b2cf6292c951a0cdde78e755514f5d7cb43ce9f7b6e7dc381d8dfeec31ab
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2280
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:42230ce9d11a1949a5bb73b6ee218b1581f585bcd553485ee438e1c5b50fcf8b
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2290
          value: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:56bae938a6b4cd87f9f5284931f440f9b9d116cda3b63b13f83a4b1c6c641959
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2290
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:4be60cfa2a300b2eef9b5f10ea756bd3f55d75fbd8890232d907f8d64080449b
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2300
          value: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:88a0098a628fe43a9b422f8ebb774086fb65ed389a195e2779482a4a63447e28
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2300
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:8fbdd1ba728058d5f437186b9055ca28596140068ebc69ecff4fa52717b19a82
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2310
          value: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:e438bcacf099a92b143793e88c00c6ac0afb31effe4b0eed49a935c9265ad943
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2310
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:4bba8828e72fd7c9d38c22ba1c2a4f7130817a4e7eac8fb63c13e86465a8e1e7
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2312
          value: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:ddaf00914e3a412d21ec3a814e24f3d4cc382b5c4d6c21e9b1b94dde16ddf194
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2312
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:81590ec33b2f00775d3c6596f610887b670de7c0b89148b09eab7584f5b0b0ec
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2320
          value: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:3bbfb6177e4839087f615a01ee54f9e06a95acaddbb083e11851b6af9315dbf0
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2320
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:52f337cd5d7d5d199eebf4fc722f8c5279f652506f009efbc0f8f6de88e18718
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2330
          value: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:93168fd8cd4fb32f041795afb8fb0e21cf81081d6cc463a44356e153dc1ad032
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2330
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:b293c1269166cc6181f68b8d803fdfcf20fdbd5df2787ed0453724b22e298fb2
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2340
          value: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:1fd99a46758cd29fce846d0d590021758b2a7a3efedb817bb205359caa7d4f67
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2340
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:6f2da9ecd8f5021964d0dfa85144830efcfbd0462d18f18f036971e5f17b3bb6
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2350
          value: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:de82344daa0caeabc6559656da0af285436a687e8f371744c0c51bdbc309167d
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2350
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:49f34122ab46ab5d7ead094b6e40817b62c75b6a809f67db07d68bce750da51e
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2360
          value: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:ffa0d7cb4068df84319776dbfb66210297da1b8466ccd5451318fc3de79736c3
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2360
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:9e1fefc9a94d72a7b5a3f4e2d53ebd23e5ab221df17a5afde42ce973277838f6
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Init_2370
          value: quay.io/artemiscloud/activemq-artemis-broker-init@sha256:e2a761725d5b029e5d33221f701438744ea0fb16a4890cd857e43aa0b5eea71c
        - name: RELATED_IMAGE_ActiveMQ_Artemis_Broker_Kubernetes_2370
          value: quay.io/artemiscloud/activemq-artemis-broker-kubernetes@sha256:cd9a01cfe791451d12144a418dbe50221c739151ecda1f22ffdbd2d10009fc82
        - name: OPERATOR_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['name']
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: WATCH_NAMESPACE
        - name: ENABLE_WEBHOOKS
          value: "false"
        image: "quay.io/artemiscloud/activemq-artemis-operator:{{ .Chart.AppVersion }}"
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8081
          initialDelaySeconds: 15
          periodSeconds: 20
        name: manager
        ports:
        - containerPort: 8383
          name: http-metrics
        readinessProbe:
          httpGet:
            path: /readyz
            port: 8081
          initialDelaySeconds: 5
          periodSeconds: 10
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      serviceAccountName: activemq-artemis-controller-manager
      terminationGracePeriodSeconds: 10

Tip

Vote this issue reacting with 👍 or 👎

[awdv00]

After removing the -Dwebconfig.bindings.artemis.uri=http://0.0.0.0:8161 configuration, the issue still persists.

[brusdev]

@awdv00 I can reproduce this issue only on 1.2.4 with -Dwebconfig.bindings.artemis.uri=http://0.0.0.0:8161 this issue is caused by artemiscloud@e99b1fe

			} else if oneVar.Name == "JAVA_ARGS_APPEND" {
				jolokiaProtocol = getProtocolFromJavaArgs(oneVar.Value)
			}
			if jolokiaUser != "" && jolokiaPassword != "" && jolokiaProtocol != "" {
				break
			}
		}
	}

	if jolokiaProtocol == "" {
		jolokiaProtocol = "http"
	} else {
		jolokiaProtocol = "https"
	}

I created the PR #1027 to fix this issue.

[awdv00]

Awesome, thank you very much for the swift reply!

[brusdev]

Resolved by artemiscloud@80a1856