Use custom ssh port for cloning ssh repository?

[enoryw]

I need to clone private repo accessible via ssh. buildx build --ssh=default --add-host=${GIT_SERVER}:${SERVER_IP} --build-arg=GIT_SERVER=$GIT_SERVER --build-arg=SSH_PORT=$SSH_PORT --file Dockerfile . works, but I don't see an option to specify a custom ssh port. Is this possible?

Currently I copy the ssh config to the container--everything in the config is already provided by the command above except the custom ssh port.