Skip to content

Slim down API, focus on happy path #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 23 commits into from
Jan 15, 2025
Merged

Slim down API, focus on happy path #1

merged 23 commits into from
Jan 15, 2025

Conversation

erquhart
Copy link
Collaborator

@erquhart erquhart commented Jan 11, 2025

Focus on simplifying R2 uploads, serving, and providing reactive metadata.

  • Add useUploadFile hook to simplify signed url uploads that don't require a lot of control
  • Store metadata in the component tables and provide queries
  • Still provide serving via signed urls (just wraps the s3 sdk)
  • Drop pretty much everything else, it was all thinly wrapping the S3 sdk and not adding much value
  • Slim down and focus docs
  • Add permission checks and callbacks in the style of the TipTap Prosemirror component api

@erquhart erquhart changed the title Sync metadata Slim down API, focus on happy path Jan 11, 2025
@erquhart erquhart marked this pull request as ready for review January 12, 2025 22:57
Copy link
Contributor

@ianmacartney ianmacartney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall I like the direction, and seems easy to use! Only big thing is pagination: unifying with list & using paginator (unless pagination has since been supported over the holidays)

Comment on lines 37 to 45
onUpload: async (ctx, bucket, key) => {
// ...do something with the key
// This technically runs in the `syncMetadata` mutation, as the upload
// is performed from the client side. Will run if using the `useUploadFile`
// hook, or if `syncMetadata` function is called directly. Runs after the
// `checkUpload` callback.
await ctx.db.insert("images", {
bucket,
key,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how would you associate this with a message? I guess that would be more like using the useUploadFile callback and then taking the key and chaining another mutation? or a future version could pass args to the useUploadFile or something..

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may not be understanding your question here, but the callback provides the object key, which you can then associate with any other piece of data (such as a message). Is that what you meant?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, my question here is how you'd know which message to associate it with, since the callback only knows the bucket & key, not any context about why it was uploaded. The older approach was to provide the URL where they could pull the query arg. With this, it seems they'd need to drive it from the client (assuming the client didn't go away in the meantime).

Copy link
Collaborator Author

@erquhart erquhart Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a note indicating to do stuff like that on the client with the key returned by the useUploadFile function. Notes both here and in the client example.

// the r2 component's `deleteObject` function.
const image = await ctx.db
.query("images")
.filter((q) => q.eq(q.field("key"), key))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's make the example use an index to encourage folks to use them

args: {},
handler: async (ctx) => {
const images = await ctx.db.query("images").take(10);
const images = await ctx.db.query("images").collect();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe this could show off listing / paginating from the component's data? maybe looking up image by key, or doing the lookup the other way

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I originally had it metadata driven, but went the other way since metadata driven is less likely to be the real world scenario. I can take it that way for the sake of the example, though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this approach is fine, there just isn't an example of listing or fetching metadata to exercise that code. Doesn't need to block initial release

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Example now includes both approaches

Comment on lines +346 to +349
await ctx.scheduler.runAfter(0, this.component.lib.deleteObject, {
key: args.key,
...this.r2Config,
});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like this to be a mutation call to the component so the metadata is deleted transactionally, with the scheduler used just for the action-ey part of sending the command. So either call deleteMetadata with an extra param to also delete the object, or deleteObject doesn't delete metadata and is scheduled separately from the thick client

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just exposing deleteObject, optimistically deletes metadata and uses the action retrier for a best effort at deleting the underlying object. I suspect this is good enough for now given the metadata in Convex isn't necessarily up to date with R2 anyway, but let me know if you want stronger guarantees.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds great. i'd rather not show the metadata in Convex but have an orphan in R2 than the opposite. It's already possible to have R2 orphans if you upload to the URL but the client never calls up to sync the metadata

@erquhart
Copy link
Collaborator Author

@ianmacartney this is ready for re-review.

@erquhart erquhart requested a review from ianmacartney January 14, 2025 23:44
@ianmacartney ianmacartney merged commit 7fa5d35 into main Jan 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants