Setting cache-control && expires

[yankeeinlondon]

Is there a way to configure S3's cache-control and expires settings (ideally on a per glob basis)?

[yankeeinlondon]

Looking at the code it appears to be hard-coded to maximum length which is fine for all fingerprinted assets but for static assets like index.html, robots.txt, and cross-domain.xml this is not what you'd want. Yeah you can invalidate at the CDN level when these assets are updated but that is rather slow and cumbersome. Better to just set them to smaller values. I typically use 10, 3600, and 86400 respectively but the point is that it would be really nice to be able to configure these.

[ghedamat]

closed by #53

@ksnyde the current solution forces you to have one expiry for everything. Fell free to reopen this one or another issue if you think values should be different per file, although I'm not sure how we can achieve that easily. If you're interested an exploratory PR is very welcome!

[yankeeinlondon]

It's a showstopper for me. Surprised it isn't for more people. I will try and find the time for a PR but struggling for bandwidth and my shell scripts work ok (although I always feel like I should get on the bandwagon).

[ghedamat]

@ksnyde I think most people either serve the index.html from s3 without cloudfront OR they invalidate the whole cdn OR they don't serve those files from s3 (i.e. use redis and the normal backend).

I believe that you can also tell cloudfront to not cache specific files for what matters (which is almost identical to your setting for index.html)

A PR is definitely welcome and we're happy to provide guidance as you might need it, just ping me or the other ember-cli-deploy devs on the ember community slack!

thanks!

[yankeeinlondon]

Cloudfront will use the cache settings you set in S3 and proxy them forward. That is the only practical way to address a "lightening" approach using AWS. Directing to an index.html on S3 and then having your other assets on Cloudfront would introduce a whole host of complications that you don't want (CORS, complicated permissions rules on S3 to ensure error handling is sent to index.html but without allowing index viewing, and more); not to mention introducing unnecessary latency on the first call. Furthermore, Cloudfront's "invalidation" is a very long process and not recommended ... Certainly not "transactional" like what you'd need for a lightening build.

I haven't really dug into what's preventing this feature here (it is very easy on the AWS API) but it's hard to believe that anyone who's using "just" AWS could really use this as-is. I do believe that maybe in conjunction with the S3-index addon it would be possible but sadly it suffers from the same problem. In that case though I think there is movement afoot to correct this.

Sorry not meaning to sound passive aggressive. Just super busy and a bit surprised that others aren't knocking down doors to get this in. I will find time at some point to put in a PR but since I'm brand new to ember-cli-deploy (other than thinking about using it for months) I keep running into friction in getting started.

[yankeeinlondon]

I have always been a bit surprised by the original "lightening" architecture's appeal. I mean it sounds like originally redis was already part of the architecture which may justify it but more most sites this is simply unnecessary clutter and cost for very little (no?) benefit.

[ghedamat]

@ksnyde I'm sorry to hear :/

I agree with you that serving assets from S3 is not a good idea in general but its' still a popular strategy for index.html due to it being very simple to implement.

The AWS plugin pack uses the cloudfront invalidation strategy, even if it's slow people seem to be happy w/it, I personally don't do either and instead serve index and such from my backend while keeping cloudfront only for fingerprinted assets which is why I haven't been bitten by this particular issue.

To answer your last question, redis is pretty easy to add and it ends up offering quite a few benefits (no cors, programmatic A/B testing, manual access to old versions..) but it shouldn't be by no means the only full-feature option.

The goal of ECD is to be flexible enough to allow you to choose what fits best and what you're asking sounds reasonable :)

I wonder if the best place to tackle your use case in the short term would be https://github.com/ember-cli-deploy/ember-cli-deploy-s3-index adding the functionality to set the expiry on the index.html there.

this would still leave an open question for robots.txt and other files though so maybe the path moving forward is to support an extra option has for this plugin that allows to define different expires for files that match a certain pattern? (i.e. *.txt)

/cc @lukemelia

[lukemelia]

@ksnyde Agreed with you on all points. Redis makes sense for us because we already have it in our stack. The point of the 0.5.x rewrite of ember-cli-deploy was to create an ecosystem of plugins that allows other patterns to emerge for people in different situations. It's working (see growth of plugins and plugin packs) but we are still less than 6 months from the release of 0.5.x and a lot of the plugins are still under-featured. File- or file-pattern-level cache-control configuration in this plugin is one example.

[YoranBrondsema]

@ksnyde If I understand your problem correctly, our solution might help you. I think we were facing a similar issue, which is why I did #53. To give you some context, on the HSTRY application users can embed their content onto their own website. The embed code contains an external script. Since we obviously can't tell all webmasters (that word feels so early 2000's :-)) to update their embed code when we modify this external script, we serve this script without fingerprinting and without caching.

I think the patterns allowed by minimatch, which is what ember-cli-deploy-s3 uses for pattern matching, is kind of limiting. For instance it's very hard to exclude certain files.

So what we ended up doing is using aliases (see https://ember-cli-deploy.github.io/ember-cli-deploy/docs/v0.6.x/aliasing-plugins/) to have two passes of uploading to S3: one for everything (all scripts, stylesheets, images, audio, etc...) and a second one for just this external script for the embed. The first pass will upload the external script to S3 with default cache parameters but it will be overwritten by the second pass where the cache settings will be set to 0.

Some code to clarify:

  ENV['s3-all'] = {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    bucket: process.env.AWS_BUCKET,
    region: process.env.AWS_REGION
  };

  ENV['s3-embed-script'] = {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
    bucket: process.env.AWS_BUCKET,
    region: process.env.AWS_REGION,
    manifestPath: null,
    filePattern: '**/embed-script.js',
    cacheControl: 'no-cache, no-store, must-revalidate',
    expires: 0
  };

Long story but essentially I think you could do the same thing in your case but instead of embed-script.js you would upload index.html, robots.txt and whatever other files you want in the second pass, with cache settings that you want. You can have as many passes as you want, with different caching parameters. It's not as clean and fast as being able to configure the caching settings for each file in a single pass, but it's a workaround that works.

I hope this helps!

[yankeeinlondon]

@YoranBrondsema your suggestion is very similar to where my mind was going but I just haven't had any bandwidth to focus on this yet. Will try to get to it today. I think as a first stage solution it might be ok to have two timeouts ... one for all fingerprinted assets (aka, max time) and another for non-fingerprinted (aka, min time ... probably 10-15 seconds for me). This should keep configuration to a tolerable level and fit within the minimatch limitations you've outlined.

[achambers]

@YoranBrondsema / @ksnyde That is a perfect example of how plugin aliasing can be used. I think this is a perfectly good way to achieve what you're after.

(sorry, just catching up on this discussion)