Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shelf life of data when using upload function - auto delete of data after x days #1783

Open
3 tasks
RobSch1406 opened this issue Jul 4, 2024 · 1 comment
Open
3 tasks
Labels
enhancement New feature or request

Comments

@RobSch1406
Copy link

  • bug report -> please search issues before submitting
  • [x ] feature request
  • documentation issue or request
  • regression (a behavior that used to work and stopped in a new release)
in the documentation of assistant (preview) there is the possibility to delete expired data: 

e.g. : 
`vector_store = client.beta.vector_stores.create_and_poll(
  name="Product Documentation",
  file_ids=['file_1', 'file_2', 'file_3', 'file_4', 'file_5'],
  expires_after={
	  "anchor": "last_active_at",
	  "days": 7
  }
)`

this is the link:
https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/file-search?tabs=python

have you already tried it to implement this into the current code?

Where to start?
Thanks and best regards

@RobSch1406 RobSch1406 changed the title Shelf life of data when using upload function - auto dlete of data after x days Shelf life of data when using upload function - auto delete of data after x days Jul 4, 2024
@pamelafox
Copy link
Collaborator

I have not tried to implement such a feature, no. This is the first time it's been requested.

To implement it, we need a few things:

  1. A way of marking the expiration time of a file (if it's not going to be a global expiration). We could potentially store that in the metadata of the blob, I suppose.
  2. A way of querying for expired files. Perhaps using https://learn.microsoft.com/en-us/azure/storage/blobs/storage-manage-find-blobs?tabs=azure-portal if we're using Blob metadata. Or we have to do a brute-force search over every blob and decide.
  3. A cron job for deleting expired data. Since we're hosted on AppService, that'd be via WebJobs: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-manage-find-blobs?tabs=azure-portal

So I think it is doable, but is a decent amount of work. I likely will not be implementing it anytime soon, as there are other feature requests that are more commonly requested. If you do implement it, we'd love to see it in a branch or PR.

@pamelafox pamelafox added the enhancement New feature or request label Jul 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants