Started getting the SHA parameter you gave is incorrect

[mkhatib]

I've noticed recently that some of the old images gave this error. I am not entirely sure why, the secret we use for generating the sha hasn't changed. Are SHAs supposed to expire? I also noticed new generated SHAs seem to be longer than the old ones. Have the algorithm used to generate SHAs changed?

The SHA parameter you gave (1fa43656) is incorrect

Old URL (1fa43656):
http://api.manshar.com/media/W1siZiIsIjIwMTQvMTIvMTMvMTIvNTUvMTEvMzk0L1NjcmVlbl9TaG90XzIwMTRfMTJfMTNfYXRfNC41NC40MV9BTS5wbmciXV0/1fa43656/Screen%20Shot%202014-12-13%20at%204.54.41%20AM.png

New URL (9447ec52bd11b8fc):
http://api.manshar.com/media/W1siZiIsIjIwMTUvMDIvMTUvMDYvNTQvMDQvNjYwL1Jvb2ZfaGFmZXpfdG9tYi5qcGciXV0/9447ec52bd11b8fc/Roof_hafez_tomb.jpg

Any pointers to figure out what is going on? The logs don't seem to show anything.

Thanks!

[ddeyoung]

I'm also experiencing this issue, the sha changes without any changes to the application.

[markevans]

sorry for delay - I've been away. I imagine you upgraded the version? The algorithm changed for version 1.0.7.
I realise that changing the url in between patch versions can be problematic for old caches (as per the discussion here: #361) but as it was a security update I did it between 1.0.6 -> 1.0.7 so that people would update to the more secure version. In hindsight this might not have been the best idea.
Suffice to say I'll try and make sure urls don't break in future (unless specified in a major version change).

The problem should only appear if old urls are still being used for some reason. Is it possible to make sure only the new urls are used?

[ddeyoung]

I have only been using 1.0.7 and I believe it's time based not any change to the application. I can capture a working URL and let the system set overnight. When I come back the same URL the next day, with no other changes, the URL results in an invalid sha.

I have also set the secret config to make sure the sha is not randomly generated.

[markevans]

ok that's strange that's totally unexpected. the sha is generated here. https://github.com/markevans/dragonfly/blob/master/lib/dragonfly/job.rb#L159
Can you see any reason why this would change with your setup?

[ddeyoung]

Its not clear from the code. It depends on to_unique_s. What can cause variations in the output? File name only?

[bsodmike]

Hi guys, I'm seeing something strange in the context of RefineryCMS (which internally uses Dragonfly for asset processing). When a file/image link is added to a page, and persisted as say a TEXT blob in the DB, the request to this asset is always made via the persisted SHA.

It'll work as soon as its added, but after a while (and I have no metrics on this), the The SHA parameter you gave (b8f67878466d4f3c) is incorrect message starts popping up.

For images - it's a broken image placeholder that's visually seen; when downloading files by clicking a link, then the above message gets rendered.

I'm going to kill the DDOS protection for now, as this is causing headaches for users in a current app of ours.

[bblimke]

I guess there is no way to upgrade without reprocessing all images being triggered?

[bsodmike]

Well, one particular image on our staging server seems ok, but a file (PDF) in Refinery, just refuses to load. I'll try reprocessing it.

[lcowell]

I think there is something still broken here. The sha being generated appears to be correct.

a = FileAttachment.first
a.file_attachment_uid
# => "2015/07/20/13/55/39/252/Summer Youth in the Park – July 15, 22, 29,  August 5, 12, 19, and 6, 2015.pdf"
OpenSSL::HMAC.hexdigest('SHA256', Dragonfly.app.secret, a.file_attachment.job.to_unique_s)[0,16] 
# => "9d8a9a48f780d48a"

a.file_attachment.job.validate_sha!("9d8a9a48f780d48a")
# => returns job

a.file_attachment.job.validate_sha!("zzzzz")
# Dragonfly::Job::IncorrectSHA: zzzzz
# from /home/engage/gems/rails3/ruby/2.1.0/gems/dragonfly-1.0.10/lib/dragonfly/job.rb:169:in `validate_sha!'

However when I use the link generated by a.file_attachment.url I get The SHA parameter you gave (9d8a9a48f780d48a) is incorrect

[lcowell]

I may have gotten a little closer to the issue. I can see that when I decode the dragonfly URL - there appears to be a couple of unicode code points in the decoded job name and I think that's where the inconsistency between the generated and decoded urls.

url_format = '/:job/:name'
m = Dragonfly::UrlMapper.new(url_format, :basename => '[^\/]', :name => '[^\/]',  :format => '[^\.]' )
pi = "/W1siZiIsIjIwMTUvMDcvMjAvMTMvNTUvMzkvMjUyL1N1bW1lciBZb3V0aCBpbiB0aGUgUGFyayBcdTIwMTMgSnVseSAxNSwgMjIsIDI5LCAgQXVndXN0IDUsIDEyLCAxOSwgYW5kIDYsIDIwMTUucGRmIl1d/Summer%20Youth%20in%20the%20Park%20%E2%80%93%20July%2015%2C%2022%2C%2029%2C%20%20August%205%2C%2012%2C%2019%2C%20and%206%2C%202015.pdf"
q = "sha=9d8a9a48f780d48a"
params = m.params_for(pi, q)
job = Dragonfly::Job.deserialize(params['job'], Dragonfly.app)
# => => <Dragonfly::Job app=:default, steps=[fetch("2015/07/20/13/55/39/252/Summer Youth in the Park \u0005\u0000\u0000 July 15, 22, 29,  August 5, 12, 19, and 6, 2015.pdf")], content=<Dragonfly::Content temp_object=<Dragonfly::TempObject data="" >>, steps applied:0/1 >
job.validate_sha!(params['sha'])
# => Dragonfly::Job::IncorrectSHA: 9d8a9a48f780d48a
job.sha
# => "b4cbe940fdd48e9a"

[bsodmike]

Well done, good find. How are you recreating this issue locally? Just a regular dummy app with dragonfly loaded? validate_sha! calls #to_a on all the steps and uses that to create a unique identifier. You'll need to try stripping those unicode chars from the args in Job::Step#to_a,

      def to_a
        [self.class.abbreviation, *args]
      end

It may be an encoding issue in the first place in the call to Dragonfly::Job.deserialize; stripping the unicode chars is potentially a last ditch-effort. I'll try take a closer look once I'm back later on in the day.

[lcowell]

I'm just using my rails app. The steps I documented are an extraction of what's happening in dragonfly. My first comment is about how the sha is generated. The second is what dragonfly does when a request is made to the web server.

The key issue being that somewhere along the way – is becoming \u0005\u0000\u0000 (or actually \xF8\u0005\u0000 in one case??). This results in slightly different fetch step, which means different sha, which is why the shas don't match.

[bsodmike]

@lcowell Yup, that's clear; I seem to be having trouble replicating it locally now though. New Rails 4.2.3 dummy app, grabbing a fork of Dragonfly from the master branch. For it to be a serialisation issue, I thought it could be related to string encoding (as per Ruby 1.9), tried versions of 1.9.3, and Ruby 2.2.2 but locally it's behaving itself.

I was hoping to pry into this https://github.com/markevans/dragonfly/blob/master/lib/dragonfly/serializer.rb#L52

Started GET "/media/W1siZiIsIjIwMTUvMDgvMDYvNDJob2xndWc3el9TdW1tZXJfWW91dGhfaW5fdGhlX1BhcmtfSnVseV8xNV8yMl8yOV9BdWd1c3RfNV8xMl8xOV9hbmRfNl8yMDE1LmpwZyJdXQ/Summer%20Youth%20in%20the%20Park%20%E2%80%93%20July%2015%2C%2022%2C%2029%2C%20%20August%205%2C%2012%2C%2019%2C%20and%206%2C%202015.jpg?sha=81fcebc981f86349"

Let me know any further specifics on your setup please.

[bsodmike]

Hey @lcowell - Could you send me details on your version of Dragonfly & Rails please?
Here's my testbed https://github.com/bsodmike/dragonfly-sha-issue-387 let me know if you're able to replicate the issue with it. Make sure you update this https://github.com/bsodmike/dragonfly-sha-issue-387/blob/master/Gemfile#L12

[sukhchander]

@bsodmike @lcowell im experiencing this odd issue as well. the path generated by dragonfly is off. i've been trying to resolve 403 forbidden errors for a few hours now.

the app:

rails 4.2.3 + ruby 2.1.6 + refinery master.

refinery app is deployed on AWS.

i've tried with ~100 assets. the most recent asset i uploaded:

https://d32dgeeodgclxy.cloudfront.net/z/i/31d3a1d14bc02a61/W1siZiIsIjIwMTUvMDgvMjAvMjMvNDYvMzkvNzQ2NWYyOGMtNjAyNi00MTY4LWJhZDEtNDNhZjQwODI1M2Y2LzgwNDIyNTQ3MDJfZmY4ZjEzNmI5M19vLmpwZyJdLFsicCIsInRodW1iIiwiMjI1eDI1NVx1MDAzZSJdXQ/8042254702_ff8f136b93_o.jpg

this should resolve to:

https://d32dgeeodgclxy.cloudfront.net/2015/08/20/23/46/39/7465f28c-6026-4168-bad1-43af408253f6/8042254702_ff8f136b93_o.jpg

the sha:

31d3a1d14bc02a61

what should

W1siZiIsIjIwMTUvMDgvMjAvMjMvNDYvMzkvNzQ2NWYyOGMtNjAyNi00MTY4LWJhZDEtNDNhZjQwODI1M2Y2LzgwNDIyNTQ3MDJfZmY4ZjEzNmI5M19vLmpwZyJdLFsicCIsInRodW1iIiwiMjI1eDI1NVx1MDAzZSJdXQ

resolve to?

[sukhchander]

@bsodmike @lcowell i processed it with the following results

pi = "/z/i/31d3a1d14bc02a61/W1siZiIsIjIwMTUvMDgvMjAvMjMvNDYvMzkvNzQ2NWYyOGMtNjAyNi00MTY4LWJhZDEtNDNhZjQwODI1M2Y2LzgwNDIyNTQ3MDJfZmY4ZjEzNmI5M19vLmpwZyJdLFsicCIsInRodW1iIiwiMjI1eDI1NVx1MDAzZSJdXQ/8042254702_ff8f136b93_o.jpg"

params = m.params_for(pi,'job')                                                                                                               => {"job"=>
  "W1siZiIsIjIwMTUvMDgvMjAvMjMvNDYvMzkvNzQ2NWYyOGMtNjAyNi00MTY4LWJhZDEtNDNhZjQwODI1M2Y2LzgwNDIyNTQ3MDJfZmY4ZjEzNmI5M19vLmpwZyJdLFsicCIsInRodW1iIiwiMjI1eDI1NVx1MDAzZSJdXQ",
 "sha"=>"31d3a1d14bc02a61",
 "basename"=>"8042254702_ff8f136b93_o",
 "ext"=>"jpg"}

i couldn't progress further b/c

job = Dragonfly::Job.deserialize(params['job'], Dragonfly.app)

results in

Dragonfly::Register::NotFound: :thumb not registered

how would i proceed further?

[bsodmike]

@sukhchander to answer one of your earlier questions, via my testbed:

irb(main):001:0> x = "W1siZiIsIjIwMTUvMDgvMjAvMjMvNDYvMzkvNzQ2NWYyOGMtNjAyNi00MTY4LWJhZDEtNDNhZjQwODI1M2Y2LzgwNDIyNTQ3MDJfZmY4ZjEzNmI5M19vLmpwZyJdLFsicCIsInRodW1iIiwiMjI1eDI1NVx1MDAzZSJdXQ"
=> "W1siZiIsIjIwMTUvMDgvMjAvMjMvNDYvMzkvNzQ2NWYyOGMtNjAyNi00MTY4LWJhZDEtNDNhZjQwODI1M2Y2LzgwNDIyNTQ3MDJfZmY4ZjEzNmI5M19vLmpwZyJdLFsicCIsInRodW1iIiwiMjI1eDI1NVx1MDAzZSJdXQ"
irb(main):002:0> Dragonfly::Serializer.json_b64_decode x
=> [["f", "2015/08/20/23/46/39/7465f28c-6026-4168-bad1-43af408253f6/8042254702_ff8f136b93_o.jpg"], ["p", "thumb", "225x255>"]]

Trying it on @lcowell's example,

irb(main):003:0> cowell = "W1siZiIsIjIwMTUvMDcvMjAvMTMvNTUvMzkvMjUyL1N1bW1lciBZb3V0aCBpbiB0aGUgUGFyayBcdTIwMTMgSnVseSAxNSwgMjIsIDI5LCAgQXVndXN0IDUsIDEyLCAxOSwgYW5kIDYsIDIwMTUucGRmIl1d"
=> "W1siZiIsIjIwMTUvMDcvMjAvMTMvNTUvMzkvMjUyL1N1bW1lciBZb3V0aCBpbiB0aGUgUGFyayBcdTIwMTMgSnVseSAxNSwgMjIsIDI5LCAgQXVndXN0IDUsIDEyLCAxOSwgYW5kIDYsIDIwMTUucGRmIl1d"
irb(main):004:0> Dragonfly::Serializer.json_b64_decode cowell
=> [["f", "2015/07/20/13/55/39/252/Summer Youth in the Park – July 15, 22, 29,  August 5, 12, 19, and 6, 2015.pdf"]]

I want to try and get a better idea of the version of Dragonfly we're all looking at. See if you can run cat Gemfile.lock | grep dragonfly and report that along with your Rails/Refinery version.

I have so far attempted, with my testbed https://github.com/bsodmike/dragonfly-sha-issue-387

• dragonfly (1.0.10)
• Rails 4.2.3
• Ruby 2.2.2 & 1.9.3-p551

@sukhchander Can you give https://github.com/bsodmike/dragonfly-sha-issue-387 a try, which does not have Refinery. Attempt hitting /packages and /packages/new to upload a test image.

[sukhchander]

@bsodmike ill try your testbed today. i confirmed the urls are correct. the url created by dragonfly is a mask onto the actual file location a la paperclip. refinery w/ dragonfly isn't resolving to the actual path in S3. dragonfly does expose remote_url() in its API. ill see if i can patch refinery to use this. ill get back to you about your testbed later today.

[lcowell]

Hi @bsodmike, here's the info you were looking for.

dragonfly (1.0.10)
dragonfly-s3_data_store (1.2)
rails (3.2.19)

[bsodmike]

@lcowell I tried downgrading my testbed and running with Ruby 1.9.3-p551/2.2.2
https://github.com/bsodmike/dragonfly-sha-issue-387/tree/rails-3-2-19

Still no luck. I will add Refinery and see if that makes any difference.

[bsodmike]

@sukhchander here's a Rails 4.2.3 version with Refinery master https://github.com/bsodmike/dragonfly-sha-issue-387/tree/rails-4-2-3-with-refinery-master. Tested with Ruby 2.2.2.

Locally, I've copied the uploaded image link into the homepage (via the admin text-area).

And it's loading fine,

Started GET "/system/images/W1siZiIsIjIwMTUvMDgvMjcvOWtmYWxubGU2OV92bGNzbmFwXzIwMTVfMDhfMjNfMjNoMjBtMzNzMDg5LnBuZyJdXQ/vlcsnap-2015-08-23-23h20m33s089.png?sha=29d58cbca1b76499" for 127.0.0.1 at 2015-08-27 00:54:09 +0530
DRAGONFLY: GET /system/images/W1siZiIsIjIwMTUvMDgvMjcvOWtmYWxubGU2OV92bGNzbmFwXzIwMTVfMDhfMjNfMjNoMjBtMzNzMDg5LnBuZyJdXQ/vlcsnap-2015-08-23-23h20m33s089.png?sha=29d58cbca1b76499 304

[lcowell]

@bsodmike I'm wondering what you mean by 'still no luck'. Does that mean you had no luck in reproducing the issue or no luck in resolving the reproducible issue? If you can't reproduce it, it's possible that it's a bug that only presents itself when DF is configured in a certain way.

In regards to your above comment. I'm fairly certain the issue related to multibyte characters in the filename. Your URL decodes to: [[\"f\",\"2015/08/27/9kfalnle69_vlcsnap_2015_08_23_23h20m33s089.png\"]] and I don't believe that would trigger the issue. Could you try again with a filename with a multibyte char like '–'?

[bsodmike]

@lcowell although I didn't mention it I also tested a local image that I
had named the same as the file you were using the with '-' and I didn't see
any issues. I'm not at my desk at the moment, but I will follow up shortly
with examples.

Let me know how you get on with the testbed I've setup? Just remember to
update the Gemfile for Dragonfly as I'm loading the latest version locally.
Thanks.

[bsodmike]

One of my previous tests,

Started GET "/media/W1siZiIsIjIwMTUvMDgvMjEvMWtzZ3ZycXRhel9TdW1tZXJfWW91dGhfaW5fdGhlX1BhcmtfSnVseV8xNV8yMl8yOV9BdWd1c3RfNV8xMl8xOV9hbmRfNl8yMDE1LmpwZyJdXQ/Summer%20Youth%20in%20the%20Park%20%E2%80%93%20July%2015%2C%2022%2C%2029%2C%20%20August%205%2C%2012%2C%2019%2C%20and%206%2C%202015.jpg?sha=163d95c9e49ca2bd" for 127.0.0.1 at 2015-08-25 22:06:35 +0530
DRAGONFLY: GET /media/W1siZiIsIjIwMTUvMDgvMjEvMWtzZ3ZycXRhel9TdW1tZXJfWW91dGhfaW5fdGhlX1BhcmtfSnVseV8xNV8yMl8yOV9BdWd1c3RfNV8xMl8xOV9hbmRfNl8yMDE1LmpwZyJdXQ/Summer%20Youth%20in%20the%20Park%20%E2%80%93%20July%2015%2C%2022%2C%2029%2C%20%20August%205%2C%2012%2C%2019%2C%20and%206%2C%202015.jpg?sha=163d95c9e49ca2bd 304

[8] pry(main)> example = "W1siZiIsIjIwMTUvMDgvMjEvMWtzZ3ZycXRhel9TdW1tZXJfWW91dGhfaW5fdGhlX1BhcmtfSnVseV8xNV8yMl8yOV9BdWd1c3RfNV8xMl8xOV9hbmRfNl8yMDE1LmpwZyJdXQ"=> "W1siZiIsIjIwMTUvMDgvMjEvMWtzZ3ZycXRhel9TdW1tZXJfWW91dGhfaW5fdGhlX1BhcmtfSnVseV8xNV8yMl8yOV9BdWd1c3RfNV8xMl8xOV9hbmRfNl8yMDE1LmpwZyJdXQ"
[9] pry(main)> Dragonfly::Serializer.json_b64_decode(example)=> [["f", "2015/08/21/1ksgvrqtaz_Summer_Youth_in_the_Park_July_15_22_29_August_5_12_19_and_6_2015.jpg"]]

and using your approach @lcowell

url_format = '/:job/:name'
m = Dragonfly::UrlMapper.new(url_format, :basename => '[^\/]', :name => '[^\/]',  :format => '[^\.]' )
pi = "/W1siZiIsIjIwMTUvMDgvMjEvMWtzZ3ZycXRhel9TdW1tZXJfWW91dGhfaW5fdGhlX1BhcmtfSnVseV8xNV8yMl8yOV9BdWd1c3RfNV8xMl8xOV9hbmRfNl8yMDE1LmpwZyJdXQ/Summer%20Youth%20in%20the%20Park%20%E2%80%93%20July%2015%2C%2022%2C%2029%2C%20%20August%205%2C%2012%2C%2019%2C%20and%206%2C%202015.jpg"
q = "sha=163d95c9e49ca2bd"
params = m.params_for(pi, q)
=> {"sha"=>"163d95c9e49ca2bd",
 "job"=>
  "W1siZiIsIjIwMTUvMDgvMjEvMWtzZ3ZycXRhel9TdW1tZXJfWW91dGhfaW5fdGhlX1BhcmtfSnVseV8xNV8yMl8yOV9BdWd1c3RfNV8xMl8xOV9hbmRfNl8yMDE1LmpwZyJdXQ",
 "name"=>"Summer Youth in the Park – July 15, 22, 29,  August 5, 12, 19, and 6, 2015.jpg"}

[lcowell]

I've tracked the issue down. The difference appears to have been some differing implementation of the c-extensions of the json gem. When the json was decoded Dragonfly::Serializer.json_b64_decode, in linux the resulting hash was different that the resulting hash on OS X.

The solution was to upgrade the json gem. I went from json 1.5.2 to to json 1.8.3. This isn't actually a direct dependency of dragonfly. df uses multijson while allows one of many json implementations to be used.

[bsodmike]

Fantastic work @lcowell thanks for getting to the bottom of this. Explains why I simply could not replicate it locally, and only saw this on production (linux) 👍

[bsodmike]

Any thoughts on how we could prevent the 'wrong' json dependency being chosen though? Do you think this justifies a warning if we can detect the host OS as linux?

The solution was to upgrade the json gem. I went from json 1.5.2 to to json 1.8.3. This isn't actually a direct dependency of dragonfly. df uses multijson while allows one of many json implementations to be used.

[lcowell]

I doubt this is a common issue, but I wasted a lot of time on it and it's worth leaving a hint for anyone else having similar issues. Detecting the os, json version pair would definitely work, but seems a pretty involved solution. I think a note in the readme would be sufficient. Something like: if you're using the json gem, you need to use 1.8.3 and link to this issue.

Edit: you're welcome! 👍

[bsodmike]

Good idea. @markevans will you add this?

[bsodmike]

@lcowell well this certainly doesn't help https://github.com/intridea/multi_json/blob/master/Gemfile#L6
I realise its unrelated to dragonfly but if that Gemfile had ~> 1.8.0 it would take care of this issue.

[markevans]

thanks a million for tracking this down guys and well done @lcowell for finding it.
I can add a note to the docs - do you reckon just in the README?

[lcowell]

I think the README is a good place. Maybe something like:

There are known issues with json 1.5.2 which cause an "incorrect sha" error for files with non-ascii characters in the name. Please see #387 for more information.

[lcowell]

@bsodmike You're right and it wouldn't help even if it did specify that version because the Gemfile isn't included in the gem build. You can cd to your bundled version here and see: bundle show multi_json