-
Notifications
You must be signed in to change notification settings - Fork 440
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[chore] Fix E2E autoscale test for OpenShift #1365
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
apiVersion: kuttl.dev/v1beta1 | ||
kind: TestStep | ||
commands: | ||
- script: go run ./wait-until-hpa-ready.go --hpa simplest-collector |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
apiVersion: kuttl.dev/v1beta1 | ||
kind: TestStep | ||
commands: | ||
- script: go run ./wait-until-hpa-ready.go --hpa simplest-set-utilization-collector |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
apiVersion: kuttl.dev/v1beta1 | ||
kind: TestStep | ||
delete: | ||
- apiVersion: batch/v1 | ||
kind: Job | ||
metadata: | ||
name: tracegen |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,109 @@ | ||
// Copyright The OpenTelemetry Authors | ||
// | ||
// Licensed under the Apache License, Version 2.0 (the "License"); | ||
// you may not use this file except in compliance with the License. | ||
// You may obtain a copy of the License at | ||
// | ||
// http://www.apache.org/licenses/LICENSE-2.0 | ||
// | ||
// Unless required by applicable law or agreed to in writing, software | ||
// distributed under the License is distributed on an "AS IS" BASIS, | ||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
// See the License for the specific language governing permissions and | ||
// limitations under the License. | ||
|
||
package main | ||
|
||
import ( | ||
"context" | ||
"fmt" | ||
"os" | ||
"path/filepath" | ||
"time" | ||
|
||
"github.com/spf13/pflag" | ||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" | ||
"k8s.io/apimachinery/pkg/util/wait" | ||
"k8s.io/client-go/kubernetes" | ||
"k8s.io/client-go/tools/clientcmd" | ||
"k8s.io/client-go/util/homedir" | ||
) | ||
|
||
func main() { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. please add |
||
var hpaName string | ||
var timeout int | ||
var kubeconfigPath string | ||
|
||
defaultKubeconfigPath := filepath.Join(homedir.HomeDir(), ".kube", "config") | ||
|
||
pflag.IntVar(&timeout, "timeout", 600, "The timeout for the check.") | ||
pflag.StringVar(&hpaName, "hpa", "", "HPA to check") | ||
pflag.StringVar(&kubeconfigPath, "kubeconfig-path", defaultKubeconfigPath, "Absolute path to the KubeconfigPath file") | ||
pflag.Parse() | ||
|
||
if len(hpaName) == 0 { | ||
fmt.Println("hpa flag is mandatory") | ||
os.Exit(1) | ||
} | ||
|
||
config, err := clientcmd.BuildConfigFromFlags("", kubeconfigPath) | ||
if err != nil { | ||
fmt.Printf("Error reading the kubeconfig: %s\n", err) | ||
os.Exit(1) | ||
} | ||
|
||
client, err := kubernetes.NewForConfig(config) | ||
if err != nil { | ||
fmt.Println(err) | ||
os.Exit(1) | ||
} | ||
|
||
namespace, err := client.CoreV1().Namespaces().Get(context.Background(), os.Getenv("NAMESPACE"), metav1.GetOptions{}) | ||
if err != nil { | ||
fmt.Println(err) | ||
os.Exit(1) | ||
} | ||
|
||
hpaClientV2 := client.AutoscalingV2().HorizontalPodAutoscalers(namespace.Name) | ||
hpaClientV1 := client.AutoscalingV1().HorizontalPodAutoscalers(namespace.Name) | ||
|
||
pollInterval := time.Second | ||
|
||
// Search in v2 and v1 for an HPA with the given name | ||
err = wait.Poll(pollInterval, 0, func() (done bool, err error) { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am still not sure why actually v1 and v2 HPAs are created in a test. My understanding is that only a single HPA version should be used in a given cluster. Could you please explain why both are created? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I didn't write the test. I'm just trying to make it work on OpenShift. But this is what I found while checking for the purpose of this E2E test:
# This creates two different deployments:
# * The first one will be used to see if we scale properly
# * The second is to check the targetCPUUtilization option
When the HPAs are created, they will be created using When I ran this in OpenShift 4.11, both of them were created using So, since in KUTTL there is no way to conditionally check for one resource or another, I created the
Another thing: why the HPAs are created using different |
||
hpav2, err := hpaClientV2.Get( | ||
context.Background(), | ||
hpaName, | ||
metav1.GetOptions{}, | ||
) | ||
if err != nil { | ||
hpav1, err := hpaClientV1.Get( | ||
context.Background(), | ||
hpaName, | ||
metav1.GetOptions{}, | ||
) | ||
if err != nil { | ||
fmt.Printf("HPA %s not found\n", hpaName) | ||
return false, nil | ||
} | ||
|
||
if hpav1.Status.CurrentCPUUtilizationPercentage == nil { | ||
fmt.Printf("Current metrics are not set yet for HPA %s\n", hpaName) | ||
return false, nil | ||
} | ||
return true, nil | ||
} | ||
|
||
if hpav2.Status.CurrentMetrics == nil { | ||
fmt.Printf("Current metrics are not set yet for HPA %s\n", hpaName) | ||
return false, nil | ||
} | ||
return true, nil | ||
}) | ||
if err != nil { | ||
fmt.Println(err) | ||
os.Exit(1) | ||
} | ||
|
||
fmt.Printf("%s is ready!\n", hpaName) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this needed? The
opentelemetry-operator/tests/e2e/autoscale/01-install.yaml
Line 4 in 0232b80
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The problem is: depending on the cluster where you are running the test, the previous
duration
parameter provided totracegen
can be not enough to trigger the HPA and scale the collector.With this approach, we ran
tracegen
until the number of replicas is increased. Later, we stoptracegen
to reduce the metrics and make the HPA to scale down.