You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've run into 2 unexpected issues/inconsistencies when downloading adapters from S3.
Issue 1:
With PREDIBASE_ADAPTERS_BUCKET=sagemaker-us-east-1-000000000000
Several prefixes with naming lorax/mistral-adapters/{id}, with id being an integer from 1 to 50. Exact same adapter files replicated into each prefix.
For most id's, adapter is download as expected. For some, such as id's 1 through 5 from my testing (e.g. adapter_id='lorax/mistral-adapters/1') I get the following error:
Traceback (most recent call last):
File "/home/ubuntu/user/invoke.py", line 8, in <module>
print(client.generate(prompt, max_new_tokens=64,adapter_id=adapter_id, adapter_source='s3').generated_text)
File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/lorax/client.py", line 157, in generate
raise parse_error(resp.status_code, payload)
lorax.errors.GenerationError: Request failed during generation: Server error: An error occurred (404) when calling the HeadObject operation: Not Found
This does not happen for adapters 6 through 50.
Issue 2:
With PREDIBASE_ADAPTERS_BUCKET=lorax-adapters
Several prefixes with naming adapter_{id}, with id being an integer from 1 to 50. Exact same adapter files replicated into each prefix.
For most id's, adapter is donwloaded as expected. For some others (e.g. adapter_id='adapter_2') I get the following error:
Traceback (most recent call last):
File "/home/ubuntu/user/invoke.py", line 8, in <module>
print(client.generate(prompt, max_new_tokens=64,adapter_id=adapter_id, adapter_source='s3').generated_text)
File "/opt/conda/envs/pytorch/lib/python3.10/site-packages/lorax/client.py", line 157, in generate
raise parse_error(resp.status_code, payload)
lorax.errors.GenerationError: Request failed during generation: Server error: Multiple models found for model_id adapter_2
Thanks for reporting this @joaopcm1996, that behavior is surprising. It should work now without needing to set the PREDIBASE_ADAPTERS_BUCKET param. I'll see if I can repro the issue.
Oh I misinterpreted your suggestion in #155 then. I just experimented not passing in PREDIBASE_ADAPTERS_BUCKET at launch and setting adapter_id as the full S3 path; still errored and didn't work for any prefix, looking for the adapter in the local cache folder instead of donwloading it. I made sure to pull the latest image (acct id is anonymized here).
from lorax import Client
client = Client("http://127.0.0.1:8080")
prompt = "[INST] Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? [/INST]"
adapter_id = "s3://sagemaker-us-east-1-000000000000/lorax/mistral-adapters/1"
print(client.generate(prompt, max_new_tokens=64,adapter_id=adapter_id, adapter_source='s3').generated_text)
Error: lorax.errors.GenerationError: Request failed during generation: Server error: Can't find 'adapter_config.json' at '/data/models--s3:----sagemaker-us-east-1-000000000000--lorax--mistral-adapters--11--/snapshots'
System Info
I've run into 2 unexpected issues/inconsistencies when downloading adapters from S3.
Issue 1:
With
PREDIBASE_ADAPTERS_BUCKET=sagemaker-us-east-1-000000000000
Several prefixes with naming
lorax/mistral-adapters/{id}
, with id being an integer from 1 to 50. Exact same adapter files replicated into each prefix.Lorax Client call:
client.generate(prompt, max_new_tokens=64,adapter_id='lorax/mistral-adapters/{id}', adapter_source='s3')
For most id's, adapter is download as expected. For some, such as id's 1 through 5 from my testing (e.g. adapter_id='lorax/mistral-adapters/1') I get the following error:
This does not happen for adapters 6 through 50.
Issue 2:
With
PREDIBASE_ADAPTERS_BUCKET=lorax-adapters
Several prefixes with naming
adapter_{id}
, with id being an integer from 1 to 50. Exact same adapter files replicated into each prefix.Lorax Client call:
client.generate(prompt, max_new_tokens=64,adapter_id=f'adapter_{id}', adapter_source='s3')
For most id's, adapter is donwloaded as expected. For some others (e.g. adapter_id='adapter_2') I get the following error:
Even though files at S3 prefix are:
Information
Tasks
Reproduction
See repro notebook here
Expected behavior
All adapter prefixes in the same bucket are loaded without error
The text was updated successfully, but these errors were encountered: