Skip to content

Conversation

@naufalandika
Copy link
Contributor

@naufalandika naufalandika commented Apr 8, 2025

Context

This PR contains improvement for pyfunc-ensembler-service / pyfunc-ensembler-job image building process. This improvement removes the need of supporting specific python versions and use the python version specified in user dependency instead, similar on how merlin builds pyfunc-server and batch-predictor images.

Changes in this PR includes improvement for the imagebuilder, pyfunc-ensembler-service / pyfunc-ensembler-job package, and little bit of sdk.

imagebuilder

  • Only use 1 base image for every python version used by the users.
  • Inject pyfunc-ensembler-service / pyfunc-ensembler-job dependency to user's conda.yaml.
  • Hash user conda.yaml to prevent the URL value of conda.yaml that has the same content to change and invalidating docker layer cache.

pyfunc-ensembler-service / pyfunc-ensembler-job package

  • Add new Makefile rule to build and publish the package to pypi
  • Add new github workflow to publish the package to pypi everytime sdk workflow is run successfully. pyfunc-ensembler-service / pyfunc-ensembler-job share the same version with turing-sdk package, but both of these package requires latest turing-sdk version, so we need to run the workflow only after sdk workflow completed.

sdk

  • Change the trigger to sdk workflow to python/* tag push instead of sdk/* tag push. python/* tag and version will be used by turing-sdk, pyfunc-ensembler-service and pyfunc-ensembler-job packages.

Main Modifications

imagebuilder/imagebuilder.go:

  • use 1 config for --build-arg=BASE_IMAGE=%s value
  • hash user's conda.yaml

engines/pyfunc-ensembler-*/Makefile:

  • add rule to build and publish pyfunc-ensembler-* package

engines/pyfunc-ensembler-*/app.Dockerfile:

  • injects turing-pyfunc-ensembler-* to user's dependency

.github/workflows/*

  • add publishing python/* tag as sdk workflow triggers
  • run pyfunc-ensembler-* workflow after sdk workflow is run successfully

@naufalandika naufalandika self-assigned this Apr 8, 2025
@naufalandika naufalandika force-pushed the DAT-2995_image_building_improvement branch 2 times, most recently from 611a9e2 to 2d620f7 Compare April 8, 2025 07:24
Muhammad Naufal Andika Natsir Putra added 24 commits May 23, 2025 12:44
@naufalandika naufalandika force-pushed the DAT-2995_image_building_improvement branch from e31acbe to 458bed7 Compare May 23, 2025 05:46
@naufalandika naufalandika merged commit b202317 into main May 26, 2025
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants