Better Support for AMD and ROCM via docker containers.

[jamiemoller]

Presently it is very hard to get a docker container to build with the rocm backend, some elements seem to fail independently during the build process.
There are other related projects with functional docker implementations that do work with rocm out of the box (aka llama.cpp).
I would like to work on this myself however between the speed at which things change in this project and the amount of time I have free to work on this, I am left only to ask for this.

If there are good 'stable' methods for building a docker implementation with rocm underneath already it would be very appreciated if this could be better documented. 'arch' helps nobody that wants to run on a more enterprisy os like rhel or sles.

Presently I have defaulted back to using textgen as it has a mostly functional api but its featureset is kinda woeful. (better than running llama.cpp directly imo)

[jamiemoller]

ps. love the work @mudler

[jamiemoller]

it should be noted
1 - the documentation for rocm for some reason indicates make BUILD_TYPE=hipblas GPU_TARGETS=gfx1030 ... there is no build arg
2 - stablediffusion is the hardest thing to get working in any environment ive tested. as in i have yet to actually get it to build on arch, deb, or opensuse
3 - the following dockerfile is the smoothest ive had it build so far

FROM archlinux

# Install deps
# ncnn not required as stablediffusion build is broken
RUN pacman -Syu --noconfirm
RUN pacman -S --noconfirm base-devel git rocm-hip-sdk rocm-opencl-sdk opencv clblast grpc go ffmpeg ncnn

# Configure Lib links
ENV CGO_CFLAGS="-I/usr/include/opencv4" \
    CGO_CXXFLAGS="-I/usr/include/opencv4" \
    CGO_LDFLAGS="-L/opt/rocm/hip/lib -lamdhip64 -L/opt/rocm/lib -lOpenCL -L/usr/lib -lclblast -lrocblas -lhipblas -lrocrand -lomp -O3 --rtlib=compiler-rt -unwindlib=libgcc -lhipblas -lrocblas --hip-link"

# Configure Build settings
ARG BUILD_TYPE="hipblas"
ARG GPU_TARGETS="gfx906" # selected for RadeonVII
ARG GO_TAGS="tts" # stablediffusion is broken

# Build
RUN git clone https://github.com/go-skynet/LocalAI
WORKDIR /LocalAI
RUN make BUILD_TYPE=${BUILD_TYPE} GPU_TARGETS=${GPU_TARGETS} GO_TAGS=${GO_TAGS} build

# Clean up
RUN pacman -Scc --noconfirm

[jamiemoller]

it should be noted that while i do see models load onto the card whenever there is an api call and there are computations being performed pushing the card to 200W of consumption there is never any return from the api call and the apparent inference never terminates

[mudler]

Presently it is very hard to get a docker container to build with the rocm backend, some elements seem to fail independently during the build process. There are other related projects with functional docker implementations that do work with rocm out of the box (aka llama.cpp). I would like to work on this myself however between the speed at which things change in this project and the amount of time I have free to work on this, I am left only to ask for this.

I don't have an AMD card to test, so this card is up-for-grabs.

Things are moving fast, right, but building-wise this is a good time window, there aren't plans to do changes in that code area in the short-term.

If there are good 'stable' methods for building a docker implementation with rocm underneath already it would be very appreciated if this could be better documented. 'arch' helps nobody that wants to run on a more enterprisy os like rhel or sles.

A good starting point would be in this section:

LocalAI/Dockerfile

Line 159 in 9c2d264

RUN if [ "${IMAGE_TYPE}" = "extras" ]; then \

we can pull RocM dependencies in there if the appropriate flag was passed by

[wuxxin]

@jamiemoller you could use https://github.com/wuxxin/aur-packages/blob/main/localai-git/PKGBUILD as a starting point, its a (feature limited) archlinux package of localai for CPU, CUDA and ROCM. There are binaries available via arch4edu. See #1437

[Expro]

Please do work on that. I'm trying to put any load on AMD GPU for week now. Building from source on Ubuntu for clBlast fails in so many ways it's not even funny.