A recommendation to apps to limit downloads to X per min?

[redimongo]

So I have no issues with apps downloading creators shows, just to be clear. The concern is it seems some apps just auto-download every single episode in the RSS feed, and with us supporting shows that have over 1000 episodes in the rss feed, I would really like the idea that we can suggest a limit for an app say 10 episodes per minutes that the app can download.

<podcast:downloadLimitPerMin>10</podcast:downloadLimitPerMin>

I get that it's an unpopular topic, but server load is a real issue which we experienced ourselves when we transferred ~30 shows at once to PodToo. We did spin up more servers and we did manage it to the best of our ability but I did not expect that applications would try and re-download every file.

What I thought industry standard was (I was wrong):

• Apps would do a HEAD request to see if the new host has the old file or a different file. (We don't rate limit this)
  • This would allow an application to verify every episode that the user has on their device without downloading the whole file. We support this, and Apple Podcasts is the only podcast player that does from our logs. More applications should support this, as you get the file size, data, and all metadata.
• 429 Too Many Request Header
  • Some of the Podcast 2.0 applications do support this (if your one of them thank you). We as a hosting provider do send back a retry-time 'Retry-After' For example return res.set('Retry-After', '10').status(429).send(checkResult.message); I understand in this day an age, people think bandwidth doesn't cost money, but when you use CDN providers they do.

The Real Maths:

16.5TB of data at 0.12c per GB costs us $1,980 and that is just one show in 7 days. Now we could do what other hosting providers are doing which is pass that costs onto the creator and make them pay it, or we could use the HTTP protocols to our advantage and one such method is 429, its a polite way to ask your app to slow down.

So what is PodToo rules?

10 files per 60 seconds, can be download - a max of 100 files per day per show(unless you verify that the user is a real listener and not someone who subscribed and hasn't been listening).

Yes BTW we do want to up our limits and work with the community but we mostly agree "A download is not a listener", it's up to us hosting providers to come up with a solution that keeps everyone happy (or we try too).

From the Creator side

"I switched to PodToo, because of the high bandwidth costs from my old provider" - Damian.

So if excessive bandwidth is a real reason that creators are switching hosting providers, then we need to think about how do players work with us hosting providers to make sure that is no longer a thing that makes creators want to switch hosting providers. We can all play apart in saving bandwidth and lowering the overall costs for hosting providers and creators.

[jamescridland]

Most CDNs charge per request, plus per byte transferred. That charging structure makes it irrelevant how quickly individual shows are downloaded. This proposal will not help the cost of bandwidth.

It seems to me that "excessive bandwidth" is not the problem here; but, instead, your systems can't deal with high concurrency. The internet can deal with high concurrency; and a standard Cloudfront + S3 bucket solution will happily deal with 16.5TB; but that'll cost you a similar amount to your current bill (about AUD $2000).

There is a way to mitigate bandwidth costs - to cut down on the bandwidth of the asset you're transferring. Buzzsprout defaults to 96kbps mono mp3, which for most podcasts is just fine. Surely, better and more appropriate audio encoding will have a better outcome.

A HEAD request is still a request, by the way - for which you get charged. To periodically poll will increase your cost.

[redimongo]

You're totally right adding a new feature won't stop that bad app. But by having a tag that suggests the download limit at least the well behaved apps will respect it.

I have emailed the app developer who has no interest in fixing their bug. Their position is that they should be allowed to download all 696 episodes at once.

Sorry but that doesn't work for me as a host, and I am tempted to ban their app.

I am currently speaking with our clients on what they would be happy with us doing, because we don't want to upset our clients. But the our clients are not the apps they are the creators who normally would pick up the excess bandwidth costs.

I don't want to go around banning apps but we need a fair standard that doesn't allow apps to request every episode in an RSS in a Millisecond.

If the user set it up to download automatically that is different. But I spoke with these users of this one app and they said they didn't not have it turn to download every episode that only had it to download 3 episodes.

So yes it's a bug with this player. Just sadly this player doesn't see it that way.

[jamescridland]

the excess bandwidth costs.

Again... where are the excess bandwidth costs? Your CDN doesn't charge you on concurrency - just on total requests/bandwidth.

Downloading 696 episodes and taking 2 hours doing it will cost the same as downloading 696 episodes all at the same time.

[redimongo]

Maybe my understanding is a little different. Excessive to me is when an app downloads all the episodes when we know that 696 hours of content.

If a person listened to that 8 hours a day that would take 87 days to listen too.

There is just no valid reason for any app to request all 696 episodes in a millisecond.

Please give me a valid reason why it's ok to do this from an app side, happy to be wrong.

[jamescridland]

"Limit downloads to X per min" is this proposal.

Downloading 696 episodes and taking 2 hours doing it will cost the same as downloading 696 episodes all at the same time.

[redimongo]

I did label it an idea. 💡 hoping for valid solutions. I don't want to force limits. But I don't want an app abusing downloads.

We all mostly agree a download is not a listener. So why should we as hosts allow bad apps to behave badly.

Honestly I would love solutions as I don't want to pass the cost on to our clients.

My title might need to be changed. But I really hope that I am clear on the issue.

I have no issue with real listeners, it's the (bad) apps I have issues with.

[francosolerio]

Personally if Castamatic did automatically download every show in a podcast at once I would consider it a (nasty) bug and would love to be alerted by the hosting company or the user. If it did happen let me know, it shouldn't absolutely do it, but bugs....

But... if the user manually requests all the episodes in a show I think HTTP 429 is the right choice, it's not the app's duty to tell the user what he can and can't download, I think it's better if the host's server establishes that. I'll look into handling 429 appropriately.

[redimongo]

It was not your app 😀

And I did email the app developer who said it they should be allowed to download all episodes.

That means all 696 episodes in one go.

X that by number of users and well you see the problem.

Now when the user DM me they said they didn't know that auto download did that. They thought it only downloaded the latest episodes.

So yeah that's why am suggesting this.

The app developer has no interest in fixing their bug.

also I have no issue with a user downloading all episodes, just please limit it to 10 files per 60 seconds and no issues in my eyes.