FullText RSS issue with php links containing arguments

[sonicnkt]

Hi,
Im not sure if this is a issue with selfoss itself or the graby library but i have an issue with scraping webpages that have php links to the articles like the following:
https://webpage.com/news.php?item=news&article=test

For some reason the page that will be scraped is the base php page https://webpage.com/news.php and the arguments are not really used for processing.

One example webpage would be phoronix.com:
https://www.phoronix.com/scan.php?page=news_item&px=Steam-Survey-April-2020

looking at the debug logs shows that graby tries to scrape the following link where it replaces the & symbol with the xml escape &amp;:
https://www.phoronix.com/scan.php?page=news_item&px=Steam-Survey-April-2020

Im not sure if this "escaped" version is just displayed in the logs or if this is the url which gets processed but if it is that could be the issues im facing as trying to open this escaped link in a browser has the same results as im experiencing.

Like i said im not sure if this is a issue with selfoss or graby but i have to start investigating somwhere :)

PS: im currently not running the latest 2.19 Snapshots as i could not get this running on my system (Linux, NGINX), my current build is a few month older. I am really sorry if this issues was fixed allready.

[jtojnar]

Thanks for reporting. Can confirm it still happens on master. Apparently, SimplePie htmlencodes the links 🤦‍♀️:

1. https://github.com/SSilence/selfoss/blob/76452b0392831c6dd6249dd00c48eaaaf4da74e3/src/spouts/rss/fulltextrss.php#L58
2. https://github.com/simplepie/simplepie/blob/ae49e2201b6da9c808e5dac437aca356a11831b4/library/SimplePie/Item.php#L960
3. https://github.com/simplepie/simplepie/blob/ae49e2201b6da9c808e5dac437aca356a11831b4/library/SimplePie/Item.php#L1010
4. https://github.com/simplepie/simplepie/blob/ae49e2201b6da9c808e5dac437aca356a11831b4/library/SimplePie/Sanitize.php#L390-L393

[jtojnar]

Also please open an issue for any troubles you had when installing. I would like to release 2.19 soon so pointing out any bugs or omissions in documentation would be very appreciated.

[sonicnkt]

ah cool you could identify the issue so quickly!
I will take a look again at a newer snapshot again and report back.

[sonicnkt]

Update:
I just tried to install the latest snapshot (selfoss-2.19-76452b0.zip) and this one worked fine without any issues. Maybe i just grabbed a faulty build in the past (have not kept the file) or my server config wasnt working correctly (changed some stuff in nginx).

Another small ui thing that is still present in the latest build which should be changed imo is the counter for read items. this is a bit to close to the Text imo and looks a bit weird. Margin and size should be the same as the unread counter with the red badge.

I know this is completely offtopic and if you want i can open another issue in github.

Update 2:
Ok found several other issues with the latest snapshot, i created antoher issue for this

[jtojnar]

I am thinking about how to best fix this. Initially, I was worried some spouts return html encoded-data while other do not, which is true for the links but, fortunately, we sanitize the link before storing the entry into database so we do not need to worry about that:

https://github.com/SSilence/selfoss/blob/76452b0392831c6dd6249dd00c48eaaaf4da74e3/src/helpers/ContentLoader.php#L211

thumbnail and icon fields are also sometimes escaped but the URL is only used at fetch time so there is no BC break and hopefully URLs for static resources contain ampersand less commonly. Finally author field is affected but since the only thing we do with it is print it, it does not matter.

So now the question is, should we convert all the item links in the database to unescaped versions during migration, or mark the old fields with a version attribute and unescape them when printing.

[jtojnar]

Actually, we only need the original link during fetching so we can unescape it there as well.