The code freezes without receiving a response or error when creating a Google Cloud Vision API client (vision.ImageAnnotatorClient)

[jorgerodriguezsj]

I have an aplication with horizontally scaled machines. And when all the machines start working together the client creation with client = vision.ImageAnnotatorClient(credentials=credentials) gets freeze.

I try the following solutions:

I have changed the credentials just in case,
I have checked the maximum request quotas and we are not even close to the maximum.
The internet connection of the machine is correct since the rest of the download and API services work correctly.

But it keeps blocking without receiving any response when the following line of code is executed: client = vision.ImageAnnotatorClient(credentials=credentials). This had not happened before.

If anyone has had the same problem, please let me know a solution.

thanks

pip freeze:

anyio==4.3.0
blis==0.7.11
boto3==1.34.16
botocore==1.34.56
cachetools==5.3.3
catalogue==2.0.10
certifi==2024.2.2
charset-normalizer==3.3.2
click==8.1.7
confection==0.1.4
cymem==2.0.8
deepmerge==1.1.1
distro==1.9.0
docopt==0.6.2
exceptiongroup==1.2.0
filelock==3.13.1
filetype==1.0.7
fsspec==2024.2.0
google-api-core==2.17.1
google-auth==2.28.1
google-cloud-vision==3.7.2
googleapis-common-protos==1.62.0
grpcio==1.62.0
grpcio-status==1.62.0
h11==0.14.0
httpcore==1.0.4
httpx==0.27.0
huggingface-hub==0.21.3
idna==3.6
Jinja2==3.1.3
jmespath==1.0.1
langcodes==3.3.0
Levenshtein==0.23.0
MarkupSafe==2.1.5
murmurhash==1.0.10
num2words==0.5.10
numpy==1.19.5
openai==1.10.0
opencv-python==4.5.5.64
packaging==23.2
pandas==1.1.5
pathlib_abc==0.1.1
pathy==0.11.0
pconf==1.7.2
pdf2image==1.16.3
pillow==10.2.0
preshed==3.0.9
proto-plus==1.23.0
protobuf==4.25.3
psycopg2==2.8.6
pyasn1==0.5.1
pyasn1-modules==0.3.0
pydantic==1.10.14
PyMuPDF==1.18.0
python-dateutil==2.9.0.post0
pytz==2023.3.post1
PyYAML==6.0.1
pyzbar==0.1.9
rapidfuzz==3.6.2
regex==2023.12.25
requests==2.31.0
rsa==4.9
s3transfer==0.10.0
safetensors==0.4.2
scipy==1.10.1
six==1.16.0
smart-open==6.4.0
sniffio==1.3.1
spacy==3.6.0
spacy-alignments==0.9.1
spacy-legacy==3.0.12
spacy-loggers==1.0.5
spacy-transformers==1.2.5
sqsworkers==1.2
srsly==2.4.8
thinc==8.1.12
tiktoken==0.6.0
tokenizers==0.13.3
torch==1.9.1+cpu
torchvision==0.10.1+cpu
tqdm==4.64.0
transformers==4.30.2
typer==0.9.0
typing_extensions==4.10.0
urllib3==1.26.18
wasabi==1.1.2

[parthea]

@jorgerodriguezsj ,

Please can you share more information about your environment? Are you running in GCE or GKE ? I found related issues opened in google-auth where the Metadata server, which is needed for credentials, may not be ready. Although vision.ImageAnnotatorClient(credentials=credentials) appears to be frozen, I wonder if it's just that the Metadata server is not ready and there are too many requests going out during that time.

googleapis/google-auth-library-python#1261
googleapis/google-auth-library-python#814

One workaround is to create a global credentials object as suggested in this bug to reduce the load on metadata server.
googleapis/google-auth-library-python#814 (comment)

Please let us know if the workaround solves the issue and whether you are using GCE or GKE.

[jorgerodriguezsj]

No, I don't use GCE or GKE. My entire infrastructure is deployed on ECS and Fargate with docker images. When the workload is high 8 or more machines wake up. Just in this moment is when the vision.ImageAnnotatorClient(credentials=credentials) appears to be frozen.

[parthea]

Please can you try the workaround in googleapis/google-auth-library-python#814 (comment) and let us know if it helps?

[parthea]

I found a similar error report where googleapis/python-vision#523 (comment) mentions that the issue doesn't occur with REST transport. Can you please try client = vision.ImageAnnotatorClient(credentials=credentials, transport="rest") instead of client = vision.ImageAnnotatorClient(credentials=credentials) to check if the issue is specific to gRPC? This will help to narrow down the issue and troubleshooting steps.

Separately, related to gRPC, googleapis/python-vision#523 (comment) also mentions that changing the version of dependencies solved the issue for one user. I'm wondering if you get different behaviour with different versions of grpcio. If the issue is gRPC specific and occurs in the latest version of grcpio, we can file a bug for the gRPC library.

[jorgerodriguezsj]

Thanks a lot for your help. In the end, I was able to fix the issue by downgrading the grpcio version to 1.57.0. Now it works perfectly without any problems.

[parthea]

@jorgerodriguezsj, If the issue can be produced with the latest version of grpcio, version 1.62.1, please file a bug in the grpcio repository here.

I'm going to close this issue as it is specific to the gRPC code rather than the client library code.

[parthea]

This is related to grpc/grpc#36265 which is similar to googleapis/python-bigtable#949 and possibly #12535. I'm going to mark this as P2 since there is a workaround mentioned in #12423 (comment)