allow setting user-agent value for fetching feeds

[andresmrm]

Possibly fix this one:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=807748

[wking]

Yeah, you should ‘git --amend’ to get down to a single Signed-off-by
at the end of your commit message. Probably just:

allow setting user-agent value for fetching feeds

Signed-off-by: Andres MRM andres@inventati.org

for the whole commit message ;).

[wking]

Feed._fetch is a wrapper around feedparser.parse, which takes an agent argument (since v2?). And Feed.load_from_config sets all the config entries as instance attributes. So instead of setting _feedparser.USER_AGENT up top and doing this little dance here, I'd probably just use:

return f(
    self.url, self.etag, modified=self.modified,
    agent=self.user_agent or _USER_AGENT,
    **kwargs)

in Feed._fetch.

And we also use _USER_AGENT to set some headers, so those should also become self.user_agent or _USER_AGENT.

[andresmrm]

What about it now? Changed the code following your comment. =)
Are you sure self.user_agent will be populated with the user-agent value from the config?
Didn't test the code for I need the Maildir support to test it here. =P

(Used git push -f to push the changes to GitHub, for I was unable to do it other
way. Hope it's not a problem in this case...)

[wking]

On Wed, Dec 16, 2015 at 10:22:27PM -0800, Andrés Martano wrote:

What about it now?

Looks good to me, but you're still missing the header updates [1,2]
and removal of the global feedparser config [3](which we don't need
now that we set the agent on each parse%28…%29 call).

Are you sure self.user_agent will be populated with the user-agent
value from the config?

Yeah, I checked that. And the assignment happens here 4.

(Used git push -f to push the changes to GitHub, for I was unable
to do it other way. Hope it's not a problem in this case...)

No, that's what you're supposed to do to push changes that
intentionally rewrite your branch history.

[andresmrm]

Sorry for the long delay.
Moved the default config to config.py. Does it makes sense?
As I said before, didn't test...

[willghatch]

+1 for the feature -- I am experiencing user-agent filtering on some blogs too.

[Ekleog]

@andresmrm I've opened rss2email/rss2email#48 to track this PR in the new https://github.com/rss2email/rss2email repository. Care to send this PR over there? :)

[andresmrm]

Sorry for the delay. Done. Feel free to close the PR.

[Ekleog]

Thank you! I can't really close this PR, but maybe @wking will come back some time and do it, if you don't want to do it yourself :)