Skip to content

Conversation

@khansaad
Copy link
Contributor

@khansaad khansaad commented Feb 12, 2025

This PR adds a demo script and related files to run the demo showcasing the working of Kafka consumer.

Note: This works on openshift cluster only for now.

Signed-off-by: Saad Khan <saakhan@redhat.com>
@khansaad khansaad added the enhancement New feature or request label Feb 12, 2025
@khansaad khansaad self-assigned this Feb 12, 2025
@chandrams
Copy link
Contributor

@khansaad Is the demo working fine now? Can you please paste the output here.

Signed-off-by: Saad Khan <saakhan@redhat.com>
@khansaad
Copy link
Contributor Author

@khansaad Is the demo working fine now? Can you please paste the output here.

This demo is updated to work on openshift cluster. The ephemeral demo has been moved to autotune repo now as part of PR 1542

@khansaad
Copy link
Contributor Author

khansaad commented Mar 18, 2025

$ ./kafka_demo.sh -i quay.io/khansaad/autotune_operator:kafka

Starting the demo using:
Kruize Image: quay.io/khansaad/autotune_operator:kafka
Namespace: openshift-tuning
Cluster: openshift

#######################################
# Kafka Demo Setup on openshift
#######################################

Starting the bulk service...

#######################################
# Kruize Demo Setup on openshift
#######################################

🔄 Pulling required repositories... ✅ Done!
🔄 Installing kruize! Please wait......................✅ Installation of kruize complete!
🔄 Installing metric profile...✅ Installation of metric profile complete!
🔄 Invoking bulk service...✅ Invoked job_id 0ada2fa3-86be-47b9-aea5-ff7ab6900d98
🔔 Bulk job experiment details are currently being updated in ./kruize-bulk-demo.log.
🔄 Processing 1 experiments. Please wait....✅ Complete!
🔄 Fetching the experiments...✅ Complete!
🔄 List the recommendations for 1 experiments....✅ Complete!
📌 List of all experiments available in experiment_list.txt
📌 Recommendations for a single container in cluster can be found in recommendations_data.json.
📌 Job status is available in job_status.json


ℹ  Access kruize UI at http://kruize-ui-nginx-service-openshift-tuning.apps.kruize1.lab.upshift.rdu2.redhat.com
🔖 To explore further, access kruize UI to list and create experiments, and to view or generate recommendations!
ℹ  For kruize CLI commands, refer to the end of ./kruize-bulk-demo.log

🛠 Kruize installation took 95 seconds
🚀 Kruize experiment creation and recommendations generation took 77 seconds
🕒 Success! Kruize demo setup took 180 seconds

For detailed logs, look in kruize-bulk-demo.log


echo ✅ Bulk Started Successfully!
🔄 Exposing kruize service...✅ Route created successfully!

KRUIZE_ROUTE = kruize-openshift-tuning.apps.kruize1.lab.upshift.rdu2.redhat.com

🔄 Setting up Kafka client locally to consume recommendations...
✅ Kafka client setup completed🔄 Consuming recommendations from the recommendations-topic...
{
  "experiments": {
    "thanos-ee-2|default|msc-0|tfb-qrh-sample-0(deployment)|tfb-0": {
      "apis": {
        "recommendations": {
          "response": [
            {
              "cluster_name": "default",
              "experiment_type": "container",
              "kubernetes_objects": [
                {
                  "type": "deployment",
                  "name": "tfb-qrh-sample-0",
                  "namespace": "msc-0",
                  "containers": [
                    {
                      "container_image_name": "kruize/tfb-qrh:1.13.2.F_et17",
                      "container_name": "tfb-0",
                      "recommendations": {
                        "version": "1.0",
                        "notifications": {
                          "111000": {
                            "type": "info",
                            "message": "Recommendations Are Available",
                            "code": 111000
                          }
                        },
                        "data": {
                          "2025-03-17T18:15:00.000Z": {
                            "notifications": {
                              "111101": {
                                "type": "info",
                                "message": "Short Term Recommendations Available",
                                "code": 111101
                              },
                              "111102": {
                                "type": "info",
                                "message": "Medium Term Recommendations Available",
                                "code": 111102
                              },
                              "111103": {
                                "type": "info",
                                "message": "Long Term Recommendations Available",
                                "code": 111103
                              }
                            },
                            "monitoring_end_time": "2025-03-17T18:15:00.000Z",
                            "current": {
                              "requests": {
                                "memory": {
                                  "amount": 760190316.0899807,
                                  "format": "bytes"
                                },
                                "cpu": {
                                  "amount": 9.754809782212215,
                                  "format": "cores"
                                }
                              },
                              "limits": {
                                "memory": {
                                  "amount": 1650472818.6277752,
                                  "format": "bytes"
                                },
                                "cpu": {
                                  "amount": 22.11298585539867,
                                  "format": "cores"
                                }
                              }
                            },
                            "recommendation_terms": {
                              "short_term": {
                                "duration_in_hours": 24,
                                "notifications": {
                                  "112101": {
                                    "type": "info",
                                    "message": "Cost Recommendations Available",
                                    "code": 112101
                                  },
                                  "112102": {
                                    "type": "info",
                                    "message": "Performance Recommendations Available",
                                    "code": 112102
                                  }
                                },
                                "monitoring_start_time": "2025-03-16T18:15:00.000Z",
                                "recommendation_engines": {
                                  "cost": {
                                    "pods_count": 0,
                                    "confidence_level": 0,
                                    "config": {
                                      "requests": {
                                        "memory": {
                                          "amount": 2306396651.7195373,
                                          "format": "bytes"
                                        }
                                      },
                                      "limits": {
                                        "memory": {
                                          "amount": 2306396651.7195373,
                                          "format": "bytes"
                                        }
                                      }
                                    },
                                    "variation": {
                                      "requests": {
                                        "memory": {
                                          "amount": 1546206335.6295567,
                                          "format": "bytes"
                                        },
                                        "cpu": {
                                          "amount": -9.691928131626295,
                                          "format": ""
                                        }
                                      },
                                      "limits": {
                                        "memory": {
                                          "amount": 655923833.0917621,
                                          "format": "bytes"
                                        },
                                        "cpu": {
                                          "amount": -22.05010420481275,
                                          "format": ""
                                        }
                                      }
                                    },
                                    "notifications": {
                                      "221001": {
                                        "type": "error",
                                        "message": "Number of pods cannot be zero",
                                        "code": 221001
                                      },
                                      "223003": {
                                        "type": "error",
                                        "message": "Format field is missing in CPU Section",
                                        "code": 223003
                                      }
                                    }
                                  },
                                  "performance": {
                                    "pods_count": 0,
                                    "confidence_level": 0,
                                    "config": {
                                      "requests": {
                                        "memory": {
                                          "amount": 2306396651.7195373,
                                          "format": "bytes"
                                        }
                                      },
                                      "limits": {
                                        "memory": {
                                          "amount": 2306396651.7195373,
                                          "format": "bytes"
                                        }
                                      }
                                    },
                                    "variation": {
                                      "requests": {
                                        "memory": {
                                          "amount": 1546206335.6295567,
                                          "format": "bytes"
                                        },
                                        "cpu": {
                                          "amount": -9.691928131626295,
                                          "format": ""
                                        }
                                      },
                                      "limits": {
                                        "memory": {
                                          "amount": 655923833.0917621,
                                          "format": "bytes"
                                        },
                                        "cpu": {
                                          "amount": -22.05010420481275,
                                          "format": ""
                                        }
                                      }
                                    },
                                    "notifications": {
                                      "221001": {
                                        "type": "error",
                                        "message": "Number of pods cannot be zero",
                                        "code": 221001
                                      },
                                      "223003": {
                                        "type": "error",
                                        "message": "Format field is missing in CPU Section",
                                        "code": 223003
                                      }
                                    }
                                  }
                                },
                                "plots": {
                                  "datapoints": 4,
                                  "plots_data": {
                                    "2025-03-17T00:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0009065323730762701,
                                        "q1": 0.04897215126610477,
                                        "median": 0.05103690668043683,
                                        "q3": 0.053182463906305155,
                                        "max": 0.06098817101072385,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 132539205.39934409,
                                        "q1": 1787244414.832822,
                                        "median": 1811790346.004518,
                                        "q3": 1819515355.4609377,
                                        "max": 1828244690.823133,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-17T12:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0017374794058303757,
                                        "q1": 0.04733692788366055,
                                        "median": 0.049201329861276545,
                                        "q3": 0.05365945823283103,
                                        "max": 0.06288165058591984,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 232528147.2954956,
                                        "q1": 1840954138.702386,
                                        "median": 1844675593.5041802,
                                        "q3": 1847265854.3825698,
                                        "max": 1853152442.393028,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-17T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0,
                                        "q1": 0.0468541956181039,
                                        "median": 0.05005909505019668,
                                        "q3": 0.05129820431357265,
                                        "max": 0.05806998163260107,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 178064664.75832868,
                                        "q1": 1574511049.7073011,
                                        "median": 1839339305.326547,
                                        "q3": 1915138054.381267,
                                        "max": 1921997209.766281,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-17T06:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0024181090567153233,
                                        "q1": 0.050247220909051094,
                                        "median": 0.053415467059241155,
                                        "q3": 0.05611938029262991,
                                        "max": 0.06076955893671068,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 125366545.29705393,
                                        "q1": 1780903623.674977,
                                        "median": 1784358957.2910445,
                                        "q3": 1786332274.856986,
                                        "max": 1843172260.0079422,
                                        "format": "bytes"
                                      }
                                    }
                                  }
                                }
                              },
                              "medium_term": {
                                "duration_in_hours": 168,
                                "notifications": {
                                  "112101": {
                                    "type": "info",
                                    "message": "Cost Recommendations Available",
                                    "code": 112101
                                  },
                                  "112102": {
                                    "type": "info",
                                    "message": "Performance Recommendations Available",
                                    "code": 112102
                                  }
                                },
                                "monitoring_start_time": "2025-03-10T18:15:00.000Z",
                                "recommendation_engines": {
                                  "cost": {
                                    "pods_count": 0,
                                    "confidence_level": 0,
                                    "config": {
                                      "requests": {
                                        "memory": {
                                          "amount": 2306396651.7195373,
                                          "format": "bytes"
                                        }
                                      },
                                      "limits": {
                                        "memory": {
                                          "amount": 2306396651.7195373,
                                          "format": "bytes"
                                        }
                                      }
                                    },
                                    "variation": {
                                      "requests": {
                                        "memory": {
                                          "amount": 1546206335.6295567,
                                          "format": "bytes"
                                        },
                                        "cpu": {
                                          "amount": -9.705960545396465,
                                          "format": ""
                                        }
                                      },
                                      "limits": {
                                        "memory": {
                                          "amount": 655923833.0917621,
                                          "format": "bytes"
                                        },
                                        "cpu": {
                                          "amount": -22.06413661858292,
                                          "format": ""
                                        }
                                      }
                                    },
                                    "notifications": {
                                      "221001": {
                                        "type": "error",
                                        "message": "Number of pods cannot be zero",
                                        "code": 221001
                                      },
                                      "223003": {
                                        "type": "error",
                                        "message": "Format field is missing in CPU Section",
                                        "code": 223003
                                      }
                                    }
                                  },
                                  "performance": {
                                    "pods_count": 0,
                                    "confidence_level": 0,
                                    "config": {
                                      "requests": {
                                        "memory": {
                                          "amount": 2306396651.7195373,
                                          "format": "bytes"
                                        }
                                      },
                                      "limits": {
                                        "memory": {
                                          "amount": 2306396651.7195373,
                                          "format": "bytes"
                                        }
                                      }
                                    },
                                    "variation": {
                                      "requests": {
                                        "memory": {
                                          "amount": 1546206335.6295567,
                                          "format": "bytes"
                                        },
                                        "cpu": {
                                          "amount": -9.69353012028896,
                                          "format": ""
                                        }
                                      },
                                      "limits": {
                                        "memory": {
                                          "amount": 655923833.0917621,
                                          "format": "bytes"
                                        },
                                        "cpu": {
                                          "amount": -22.051706193475415,
                                          "format": ""
                                        }
                                      }
                                    },
                                    "notifications": {
                                      "221001": {
                                        "type": "error",
                                        "message": "Number of pods cannot be zero",
                                        "code": 221001
                                      },
                                      "223003": {
                                        "type": "error",
                                        "message": "Format field is missing in CPU Section",
                                        "code": 223003
                                      }
                                    }
                                  }
                                },
                                "plots": {
                                  "datapoints": 7,
                                  "plots_data": {
                                    "2025-03-15T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0007422926219323051,
                                        "q1": 0.04870375839064168,
                                        "median": 0.051533926749034016,
                                        "q3": 0.054475943169901495,
                                        "max": 1.6506882108800083,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 146982564.40540725,
                                        "q1": 1817068777.3104372,
                                        "median": 1823906253.2016613,
                                        "q3": 1828317269.7457411,
                                        "max": 1830703097.1156545,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-13T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0011813477461565713,
                                        "q1": 0.04095057897877644,
                                        "median": 0.04310834998848637,
                                        "q3": 0.04494440579693146,
                                        "max": 0.05039337884866435,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 191159739.82489014,
                                        "q1": 1871158363.9378545,
                                        "median": 1873319349.1481938,
                                        "q3": 1878879019.3246284,
                                        "max": 1881875235.2410026,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-12T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0011762482364348534,
                                        "q1": 0.040470261024854584,
                                        "median": 0.042396633367286406,
                                        "q3": 0.044214908645644965,
                                        "max": 3.3522171506243006,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 190217602.867619,
                                        "q1": 1867846717.2456357,
                                        "median": 1872320339.8454401,
                                        "q3": 1877418612.2739184,
                                        "max": 1883107889.141504,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-17T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0,
                                        "q1": 0.04853541993513942,
                                        "median": 0.050499658876307196,
                                        "q3": 0.05397442018923307,
                                        "max": 0.06288165058591984,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 125366545.29705393,
                                        "q1": 1784358957.2910445,
                                        "median": 1819515355.4609377,
                                        "q3": 1843172260.0079422,
                                        "max": 1921997209.766281,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-14T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0006238947307405146,
                                        "q1": 0.04683714233518385,
                                        "median": 0.05086865979779163,
                                        "q3": 0.05332863713271891,
                                        "max": 0.8340171959836855,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 146311005.24180058,
                                        "q1": 1820864859.3482418,
                                        "median": 1825904153.749459,
                                        "q3": 1829415858.7638767,
                                        "max": 1873606108.2839985,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-11T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0010493874973933309,
                                        "q1": 0.04055734380593972,
                                        "median": 0.04285581810377832,
                                        "q3": 0.04463189131238644,
                                        "max": 0.07343075861118485,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 177303127.33397546,
                                        "q1": 1869225553.2587526,
                                        "median": 1873343734.7217836,
                                        "q3": 1878862293.230583,
                                        "max": 1882682934.0278153,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-16T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0007781112365757284,
                                        "q1": 0.048158133604864886,
                                        "median": 0.051316805423000954,
                                        "q3": 0.05329124030609275,
                                        "max": 2.2058550346613925,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 146648635.69594485,
                                        "q1": 1818090470.057509,
                                        "median": 1825210754.9101815,
                                        "q3": 1829128771.6474085,
                                        "max": 1830502168.247325,
                                        "format": "bytes"
                                      }
                                    }
                                  }
                                }
                              },
                              "long_term": {
                                "duration_in_hours": 360,
                                "notifications": {
                                  "112101": {
                                    "type": "info",
                                    "message": "Cost Recommendations Available",
                                    "code": 112101
                                  },
                                  "112102": {
                                    "type": "info",
                                    "message": "Performance Recommendations Available",
                                    "code": 112102
                                  }
                                },
                                "monitoring_start_time": "2025-03-02T18:15:00.000Z",
                                "recommendation_engines": {
                                  "cost": {
                                    "pods_count": 0,
                                    "confidence_level": 0,
                                    "config": {
                                      "requests": {
                                        "memory": {
                                          "amount": 2378277162.4644713,
                                          "format": "bytes"
                                        }
                                      },
                                      "limits": {
                                        "memory": {
                                          "amount": 2378277162.4644713,
                                          "format": "bytes"
                                        }
                                      }
                                    },
                                    "variation": {
                                      "requests": {
                                        "memory": {
                                          "amount": 1618086846.3744907,
                                          "format": "bytes"
                                        },
                                        "cpu": {
                                          "amount": -9.704889679116285,
                                          "format": ""
                                        }
                                      },
                                      "limits": {
                                        "memory": {
                                          "amount": 727804343.8366961,
                                          "format": "bytes"
                                        },
                                        "cpu": {
                                          "amount": -22.06306575230274,
                                          "format": ""
                                        }
                                      }
                                    },
                                    "notifications": {
                                      "221001": {
                                        "type": "error",
                                        "message": "Number of pods cannot be zero",
                                        "code": 221001
                                      },
                                      "223003": {
                                        "type": "error",
                                        "message": "Format field is missing in CPU Section",
                                        "code": 223003
                                      }
                                    }
                                  },
                                  "performance": {
                                    "pods_count": 0,
                                    "confidence_level": 0,
                                    "config": {
                                      "requests": {
                                        "memory": {
                                          "amount": 2378277162.4644713,
                                          "format": "bytes"
                                        }
                                      },
                                      "limits": {
                                        "memory": {
                                          "amount": 2378277162.4644713,
                                          "format": "bytes"
                                        }
                                      }
                                    },
                                    "variation": {
                                      "requests": {
                                        "memory": {
                                          "amount": 1618086846.3744907,
                                          "format": "bytes"
                                        },
                                        "cpu": {
                                          "amount": -9.694016370389607,
                                          "format": ""
                                        }
                                      },
                                      "limits": {
                                        "memory": {
                                          "amount": 727804343.8366961,
                                          "format": "bytes"
                                        },
                                        "cpu": {
                                          "amount": -22.05219244357606,
                                          "format": ""
                                        }
                                      }
                                    },
                                    "notifications": {
                                      "221001": {
                                        "type": "error",
                                        "message": "Number of pods cannot be zero",
                                        "code": 221001
                                      },
                                      "223003": {
                                        "type": "error",
                                        "message": "Format field is missing in CPU Section",
                                        "code": 223003
                                      }
                                    }
                                  }
                                },
                                "plots": {
                                  "datapoints": 15,
                                  "plots_data": {
                                    "2025-03-03T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0028173151277522675,
                                        "q1": 0.050676683495522026,
                                        "median": 0.05205282462362688,
                                        "q3": 0.054756773677837406,
                                        "max": 0.057174795711428264,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 499049071.67656267,
                                        "q1": 1968986784.8332117,
                                        "median": 1968986784.8332117,
                                        "q3": 1976117503.8823333,
                                        "max": 1976117503.8823333,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-08T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0007445629205794407,
                                        "q1": 0.040796947612197355,
                                        "median": 0.04351750068506527,
                                        "q3": 0.04635302523248126,
                                        "max": 0.05769548630301326,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 173679637.241853,
                                        "q1": 1804404218.3705914,
                                        "median": 1810512609.9925804,
                                        "q3": 1813616021.186311,
                                        "max": 1867298699.294071,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-13T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0011813477461565713,
                                        "q1": 0.04095057897877644,
                                        "median": 0.04310834998848637,
                                        "q3": 0.04494440579693146,
                                        "max": 0.05039337884866435,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 191159739.82489014,
                                        "q1": 1871158363.9378545,
                                        "median": 1873319349.1481938,
                                        "q3": 1878879019.3246284,
                                        "max": 1881875235.2410026,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-07T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0019510006086271648,
                                        "q1": 0.049350115042116886,
                                        "median": 0.05265880218595607,
                                        "q3": 0.05467184466782908,
                                        "max": 0.061966008527723576,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 226292582.21388218,
                                        "q1": 1855737742.2726657,
                                        "median": 1861389934.2072368,
                                        "q3": 1863335857.5744946,
                                        "max": 1866299323.923398,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-12T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0011762482364348534,
                                        "q1": 0.040470261024854584,
                                        "median": 0.042396633367286406,
                                        "q3": 0.044214908645644965,
                                        "max": 3.3522171506243006,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 190217602.867619,
                                        "q1": 1867846717.2456357,
                                        "median": 1872320339.8454401,
                                        "q3": 1877418612.2739184,
                                        "max": 1883107889.141504,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-17T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0,
                                        "q1": 0.04853541993513942,
                                        "median": 0.050499658876307196,
                                        "q3": 0.05397442018923307,
                                        "max": 0.06288165058591984,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 125366545.29705393,
                                        "q1": 1784358957.2910445,
                                        "median": 1819515355.4609377,
                                        "q3": 1843172260.0079422,
                                        "max": 1921997209.766281,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-14T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0006238947307405146,
                                        "q1": 0.04683714233518385,
                                        "median": 0.05086865979779163,
                                        "q3": 0.05332863713271891,
                                        "max": 0.8340171959836855,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 146311005.24180058,
                                        "q1": 1820864859.3482418,
                                        "median": 1825904153.749459,
                                        "q3": 1829415858.7638767,
                                        "max": 1873606108.2839985,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-04T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0024923957720580413,
                                        "q1": 0.05060367877724651,
                                        "median": 0.05334824627742474,
                                        "q3": 0.055619358718663484,
                                        "max": 0.06408132966093161,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 495776144.84956557,
                                        "q1": 1967181859.26004,
                                        "median": 1976356253.1842134,
                                        "q3": 1978998499.297524,
                                        "max": 1981897635.3870595,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-09T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0007772273585012708,
                                        "q1": 0.04070813900273909,
                                        "median": 0.04270809930301233,
                                        "q3": 0.044420394948369954,
                                        "max": 2.0623595049649217,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 172281871.6419796,
                                        "q1": 1805780636.934293,
                                        "median": 1810296537.9805973,
                                        "q3": 1813243521.1872046,
                                        "max": 1814653017.2574277,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-05T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0018797413487326932,
                                        "q1": 0.04937222386615837,
                                        "median": 0.05181216097780582,
                                        "q3": 0.0541022917630803,
                                        "max": 0.06096913095294915,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 222628137.40674222,
                                        "q1": 1858849224.1835701,
                                        "median": 1861881418.594159,
                                        "q3": 1868788556.577288,
                                        "max": 1980164257.75793,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-10T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0008400346815746495,
                                        "q1": 0.03968240476912076,
                                        "median": 0.042257126811732264,
                                        "q3": 0.04497947533322933,
                                        "max": 3.0564175103554985,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 172748347.1654166,
                                        "q1": 1801189436.1534433,
                                        "median": 1806074724.199252,
                                        "q3": 1813250403.3857899,
                                        "max": 1816081706.6858249,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-15T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0007422926219323051,
                                        "q1": 0.04870375839064168,
                                        "median": 0.051533926749034016,
                                        "q3": 0.054475943169901495,
                                        "max": 1.6506882108800083,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 146982564.40540725,
                                        "q1": 1817068777.3104372,
                                        "median": 1823906253.2016613,
                                        "q3": 1828317269.7457411,
                                        "max": 1830703097.1156545,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-06T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0020070394711076828,
                                        "q1": 0.049082793039431784,
                                        "median": 0.05154467307391147,
                                        "q3": 0.05484354613114067,
                                        "max": 0.05891423261884128,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 223509423.41972813,
                                        "q1": 1852930618.0873852,
                                        "median": 1855594088.5513313,
                                        "q3": 1863869947.4347765,
                                        "max": 1869315831.2301807,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-11T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0010493874973933309,
                                        "q1": 0.04055734380593972,
                                        "median": 0.04285581810377832,
                                        "q3": 0.04463189131238644,
                                        "max": 0.07343075861118485,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 177303127.33397546,
                                        "q1": 1869225553.2587526,
                                        "median": 1873343734.7217836,
                                        "q3": 1878862293.230583,
                                        "max": 1882682934.0278153,
                                        "format": "bytes"
                                      }
                                    },
                                    "2025-03-16T18:15:00.000Z": {
                                      "cpuUsage": {
                                        "min": 0.0007781112365757284,
                                        "q1": 0.048158133604864886,
                                        "median": 0.051316805423000954,
                                        "q3": 0.05329124030609275,
                                        "max": 2.2058550346613925,
                                        "format": ""
                                      },
                                      "memoryUsage": {
                                        "min": 146648635.69594485,
                                        "q1": 1818090470.057509,
                                        "median": 1825210754.9101815,
                                        "q3": 1829128771.6474085,
                                        "max": 1830502168.247325,
                                        "format": "bytes"
                                      }
                                    }
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  ]
                }
              ],
              "version": "v2.0",
              "experiment_name": "thanos-ee-2|default|msc-0|tfb-qrh-sample-0(deployment)|tfb-0"
            }
          ]
        }
      }
    }
  }
}
Processed a total of 1 messages
✅ Successfully consumed one recommendation from the recommendations topic
🕒 Success! Kafka demo setup took 207 seconds

@khansaad khansaad added this to the Kruize 0.5 Release milestone Mar 20, 2025

## Demo workflow

- Reserve a namespace in ephemeral cluster
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update the readme to remove references to ephemeral/bonfire

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@chandrams
Copy link
Contributor

@khansaad Demo doesn't run and fails with this error

🔄 Pulling required repositories... ✅ Done!
🔄 Installing kruize! Please wait.................✅ Installation of kruize complete!
🔄 Installing metric profile...✅ Installation of metric profile complete!
🔄 Invoking bulk service...Traceback (most recent call last):
  File "/home/csubrama/saad/kruize-demos/monitoring/local_monitoring/bulk_demo/bulk_demo.py", line 205, in <module>
    main(sys.argv[1:])
    ~~~~^^^^^^^^^^^^^^
  File "/home/csubrama/saad/kruize-demos/monitoring/local_monitoring/bulk_demo/bulk_demo.py", line 163, in main
    job_id = job_id_json['job_id']
             ~~~~~~~~~~~^^^^^^^^^^
KeyError: 'job_id'

Kruize pod log error

2025-03-2111:03:33.683 INFO [qtp91323447-53][MetricProfileCollection.java(77)]-Trying to add the metric profile to collection: resource-optimization-local-monitoring
2025-03-2111:03:33.684 INFO [qtp91323447-53][MetricProfileCollection.java(82)]-MetricProfile added to the collection successfully: resource-optimization-local-monitoring
2025-03-2111:03:34.493 ERROR [qtp91323447-49][KruizeErrorHandler.java(71)]-{"message":"java.lang.RuntimeException: ❌ Failed to connect to Kafka at kruize-kafka-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092 : Failed to create new KafkaAdminClient","httpcode":500,"documentationLink":"","status":"ERROR"}

@khansaad
Copy link
Contributor Author

@khansaad Demo doesn't run and fails with this error

🔄 Pulling required repositories... ✅ Done!
🔄 Installing kruize! Please wait.................✅ Installation of kruize complete!
🔄 Installing metric profile...✅ Installation of metric profile complete!
🔄 Invoking bulk service...Traceback (most recent call last):
  File "/home/csubrama/saad/kruize-demos/monitoring/local_monitoring/bulk_demo/bulk_demo.py", line 205, in <module>
    main(sys.argv[1:])
    ~~~~^^^^^^^^^^^^^^
  File "/home/csubrama/saad/kruize-demos/monitoring/local_monitoring/bulk_demo/bulk_demo.py", line 163, in main
    job_id = job_id_json['job_id']
             ~~~~~~~~~~~^^^^^^^^^^
KeyError: 'job_id'

Kruize pod log error

2025-03-2111:03:33.683 INFO [qtp91323447-53][MetricProfileCollection.java(77)]-Trying to add the metric profile to collection: resource-optimization-local-monitoring
2025-03-2111:03:33.684 INFO [qtp91323447-53][MetricProfileCollection.java(82)]-MetricProfile added to the collection successfully: resource-optimization-local-monitoring
2025-03-2111:03:34.493 ERROR [qtp91323447-49][KruizeErrorHandler.java(71)]-{"message":"java.lang.RuntimeException: ❌ Failed to connect to Kafka at kruize-kafka-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092 : Failed to create new KafkaAdminClient","httpcode":500,"documentationLink":"","status":"ERROR"}

Fixed the issue as part of PR 1555

@khansaad khansaad requested a review from kusumachalasani April 2, 2025 08:35
Copy link
Contributor

@kusumachalasani kusumachalasani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, please move the console output and intermediate messages to the log.
Also, please include the consumer commands user can try to look into the messages at the end of log.

if [ ${CLUSTER_TYPE} == "kind" ]; then
sed -i 's/"isKafkaEnabled" : "false"/"isKafkaEnabled" : "true"/' ${KRUIZE_CRC_DEPLOY_MANIFEST_MINIKUBE}
# Replace the existing KAFKA_BOOTSTRAP_SERVERS value
sed -i "s|value: \"kruize-kafka-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092\"|value: \"$BOOTSTRAP_SERVER\"|g" ${KRUIZE_CRC_DEPLOY_MANIFEST_MINIKUBE}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying the demo in multiple namespaces doesn't replace the actual bootstrap server

# Perform action based on selection
if [ ${start_demo} -eq 1 ]; then
echo
echo "Starting the demo using: "
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move all these messages to the log

kafka_elapsed_time=$(time_diff "${kafka_start_time}" "${kafka_end_time}")
fi
elapsed_time=$(time_diff "${start_time}" "${end_time}")
echo "🕒 Success! Kafka Server setup took ${kafka_elapsed_time} seconds"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Throws an error when kafka setup is skipped.

}

function consume_messages() {
echo -n "🔄 Consuming recommendations from the recommendations-topic..."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please include some dots to let user know that kafka is consuming messages and is in progress.

…ver and other cosmetic changes

Signed-off-by: Saad Khan <saakhan@redhat.com>
@khansaad
Copy link
Contributor Author

khansaad commented Apr 2, 2025

Updated now. This is how it looks now:

$ ./kafka_demo.sh -b -k -n kafka1
🔄 Setting up Kafka server on openshift. Please wait.............................✅ Kafka server setup completed

🔄 Starting the bulk service...

🔄 Pulling required repositories... ✅ Done!
🔄 Installing kruize! Please wait....................✅ Installation of kruize complete!
🔄 Invoking bulk service...✅ Invoked job_id fb97a0e8-0dae-4250-8235-c555d0d5aa63
🔔 Bulk job experiment details are currently being updated in ./kruize-bulk-demo.log.
🔄 Processing 175 experiments. Please wait......✅ Complete!
🔄 Fetching the experiments...✅ Complete!
🔄 List the recommendations for 1 experiments....✅ Complete!



⏳ Setting up Kafka client locally to consume recommendations...✅ Kafka client setup completed
👀 Consuming recommendations from the recommendations-topic.......✅ Done

🕒 Success! Kafka Consumer took 20 seconds
🕒 Success! Kafka Server setup took 130 seconds
🕒 Success! Kafka demo setup took 485 seconds

For detailed logs, look in kafka-demo.log

Recommendation messages are moved to the log file and URLs are added in the log as well as in the README file for users to have easy access.

khansaad added 2 commits April 3, 2025 12:10
Signed-off-by: Saad Khan <saakhan@redhat.com>
Signed-off-by: Saad Khan <saakhan@redhat.com>
@rbadagandi1 rbadagandi1 removed this from the Kruize 0.5 Release milestone Apr 3, 2025
@rbadagandi1 rbadagandi1 added this to the Kruize 0.6 Release milestone Apr 3, 2025
@kusumachalasani
Copy link
Contributor

kusumachalasani commented Apr 14, 2025

@khansaad Can you please do the below minor changes to have all the demos look similar.

  1. Include a header before the run as part of console.

#######################################
Kafka Demo Setup on openshift
#######################################

  1. Kafka Server setup time should be displayed first and then the kafka consumer.

Success! Kafka Consumer took 20 seconds
Success! Kafka Server setup took 130 seconds

Signed-off-by: Saad Khan <saakhan@redhat.com>
@chandrams
Copy link
Contributor

@kusumachalasani - Can you please close the review of this PR.

@kusumachalasani
Copy link
Contributor

@khansaad The script abruptly ends without mentioning anything. But when I logged into log file it says Install java

Can you include the statement to look into the kafka-demo.log even when there are failures so user can know what would have happened.
It would be good to include setting up failed instead of abruptly ending the script.

./kafka_demo.sh -c openshift -i quay.io/kruize/autotune_operator:0.6

#######################################
# Kafka Demo Setup on openshift 
#######################################

⏭️ Skipping Kafka Server installation...

⏭️ Skipping bulk service initiation...

⏳ Setting up Kafka client locally to consume recommendations...

@kusumachalasani
Copy link
Contributor

@khansaad I'm trying to test it on new scalelab machine and I see the demo exits abruptly with the below issue.

#######################################
# Kafka Demo Setup on openshift 
#######################################

Creating namespace for Kafka...
Error from server (AlreadyExists): namespaces "kafka" already exists
Namespace kafka already exists.
Installing Strimzi Operator...
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator unchanged
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-namespaced unchanged
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-entity-operator-delegation unchanged
clusterrole.rbac.authorization.k8s.io/strimzi-kafka-broker unchanged
customresourcedefinition.apiextensions.k8s.io/strimzipodsets.core.strimzi.io unchanged
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-watched unchanged
customresourcedefinition.apiextensions.k8s.io/kafkaconnects.kafka.strimzi.io unchanged
clusterrole.rbac.authorization.k8s.io/strimzi-kafka-client unchanged
clusterrole.rbac.authorization.k8s.io/strimzi-entity-operator unchanged
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-broker-delegation unchanged
customresourcedefinition.apiextensions.k8s.io/kafkausers.kafka.strimzi.io unchanged
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator unchanged
customresourcedefinition.apiextensions.k8s.io/kafkatopics.kafka.strimzi.io unchanged
deployment.apps/strimzi-cluster-operator configured
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-global unchanged
customresourcedefinition.apiextensions.k8s.io/kafkabridges.kafka.strimzi.io unchanged
customresourcedefinition.apiextensions.k8s.io/kafkarebalances.kafka.strimzi.io unchanged
configmap/strimzi-cluster-operator unchanged
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-client-delegation unchanged
customresourcedefinition.apiextensions.k8s.io/kafkamirrormaker2s.kafka.strimzi.io unchanged
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-leader-election unchanged
customresourcedefinition.apiextensions.k8s.io/kafkanodepools.kafka.strimzi.io unchanged
serviceaccount/strimzi-cluster-operator unchanged
customresourcedefinition.apiextensions.k8s.io/kafkaconnectors.kafka.strimzi.io unchanged
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-watched unchanged
customresourcedefinition.apiextensions.k8s.io/kafkas.kafka.strimzi.io unchanged
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-leader-election unchanged
Waiting for Strimzi Operator to be ready...
deployment "strimzi-cluster-operator" successfully rolled out
kafka.kafka.strimzi.io/kruize-kafka-cluster unchanged
Waiting for Kafka cluster to be ready...
error: timed out waiting for the condition on kafkas/kruize-kafka-cluster

@khansaad
Copy link
Contributor Author

@kusumachalasani This issue is occuring due to the recent 0.46 release of the Strimzi Kafka operator. It no longer supports zookeeper to manage kafka clusters instead it has now moved Kraft. You can find the details here in their recent release page: https://github.com/strimzi/strimzi-kafka-operator/releases/

We can stick to 0.45 to make this work but I'm also testing the script now with this latest update. Will update here once done.

@khansaad khansaad moved this from Under Review to In Progress in Monitoring May 16, 2025
Signed-off-by: Saad Khan <saakhan@redhat.com>
@khansaad
Copy link
Contributor Author

@kusumachalasani Please try this again now. Have updated the script now to use the latest 0.46 version which uses KRaft instead of Zookeeper .
Let me know if you have any issues.

@khansaad khansaad moved this from In Progress to Under Review in Monitoring May 20, 2025
@khansaad khansaad requested a review from chandrams May 20, 2025 13:51
@kusumachalasani
Copy link
Contributor

@khansaad
The demo abruptly ends while consuming recommendations from recommendations-topic. From the log, the issue is because of Java version incompatibility issues. Provided the log below.

./kafka_demo.sh -b -k -i quay.io/kruize/autotune_operator:0.6

#######################################
# Kafka Demo Setup on openshift 
#######################################

🔄 Setting up Kafka server on openshift. Please wait....................✅ Kafka server setup completed

🔄 Starting the bulk service...

🔄 Pulling required repositories... ✅ Done!
🔄 Installing kruize! Please wait..............✅ Installation of kruize complete!
🔄 Invoking bulk service...✅ Invoked job_id f3f40b56-347d-420a-aee7-16cf01b39d39
🔔 Bulk job experiment details are currently being updated in ./kruize-bulk-demo.log.
🔄 Processing 180 experiments. Please wait......✅ Complete!
🔄 Fetching the experiments...✅ Complete!
🔄 List the recommendations for 1 experiments....✅ Complete!



⏳ Setting up Kafka client locally to consume recommendations...✅ Kafka client setup completed
👀 Consuming recommendations from the recommendations-topic....
✅ Kafka Bootstrap Server: kruize-kafka-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092
Java is already installed.
openjdk version "11.0.22" 2024-01-16 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS, mixed mode, sharing)
kafka_2.13-4.0.0.tgz does not exist. Downloading...
Extracting Kafka tgz...
Kafka setup completed!

Certificate was added to keystore
Kafka endpoint: kruize-kafka-cluster-kafka-bootstrap-kafka.apps.mno.example.com:443
Error: LinkageError occurred while loading main class org.apache.kafka.tools.consumer.ConsoleConsumer
        java.lang.UnsupportedClassVersionError: org/apache/kafka/tools/consumer/ConsoleConsumer has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0
Error: Kafka consumer command failed!

@khansaad
Copy link
Contributor Author

@khansaad The demo abruptly ends while consuming recommendations from recommendations-topic. From the log, the issue is because of Java version incompatibility issues. Provided the log below.

./kafka_demo.sh -b -k -i quay.io/kruize/autotune_operator:0.6

#######################################
# Kafka Demo Setup on openshift 
#######################################

🔄 Setting up Kafka server on openshift. Please wait....................✅ Kafka server setup completed

🔄 Starting the bulk service...

🔄 Pulling required repositories... ✅ Done!
🔄 Installing kruize! Please wait..............✅ Installation of kruize complete!
🔄 Invoking bulk service...✅ Invoked job_id f3f40b56-347d-420a-aee7-16cf01b39d39
🔔 Bulk job experiment details are currently being updated in ./kruize-bulk-demo.log.
🔄 Processing 180 experiments. Please wait......✅ Complete!
🔄 Fetching the experiments...✅ Complete!
🔄 List the recommendations for 1 experiments....✅ Complete!



⏳ Setting up Kafka client locally to consume recommendations...✅ Kafka client setup completed
👀 Consuming recommendations from the recommendations-topic....
✅ Kafka Bootstrap Server: kruize-kafka-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092
Java is already installed.
openjdk version "11.0.22" 2024-01-16 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS, mixed mode, sharing)
kafka_2.13-4.0.0.tgz does not exist. Downloading...
Extracting Kafka tgz...
Kafka setup completed!

Certificate was added to keystore
Kafka endpoint: kruize-kafka-cluster-kafka-bootstrap-kafka.apps.mno.example.com:443
Error: LinkageError occurred while loading main class org.apache.kafka.tools.consumer.ConsoleConsumer
        java.lang.UnsupportedClassVersionError: org/apache/kafka/tools/consumer/ConsoleConsumer has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0
Error: Kafka consumer command failed!

@kusumachalasani Strimzi 0.46 requires following:

for Strimzi 0.46.0:

Java Version: OpenJDK 17 or newer
Kubernetes Version: 1.25 or newer

You need to update the version. I will update the same in the docs as well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

Status: Under Review

Development

Successfully merging this pull request may close these issues.

4 participants