Add LTO (Legal Tribune Online) publisher #799
Conversation
- Add LTOParser with support for title, body, authors, publishing_date, and topics - Configure publisher with RSS feed, NewsMap, and Sitemap sources - Generate unit tests for parser validation - Update supported publishers documentation
addie9800
left a comment
addie9800
left a comment
There was a problem hiding this comment.
Thank you so much for adding LTO 🚀. I only have a couple of comments before we are ready to merge.
| sources=[ | ||
| RSSFeed("https://www.lto.de/rss/feed.xml"), | ||
| NewsMap("https://www.lto.de/googlenews-sitemap.xml"), | ||
| Sitemap("https://www.lto.de/sitemap.xml"), |
There was a problem hiding this comment.
It seems like there are some unnecessary sitemaps in this index sitemap. We should restrict this to URLs of the form: https://www.lto.de/sitemap-type/article/page-x/sitemap.xml using the sitemap_filter argument.
|
|
||
| @attribute | ||
| def publishing_date(self) -> Optional[datetime.datetime]: | ||
| # Try to get date from meta tag or page content |
There was a problem hiding this comment.
Perhaps you can use your solution as a fallback, if self.precomputed.meta.get("date") fails. In most cases, I checked that it should be sufficient.
|
|
||
|
|
||
| class LTOParser(ParserProxy): | ||
| class V1(BaseParser): |
There was a problem hiding this comment.
It seems like the implementation of images is missing.
|
|
||
| class LTOParser(ParserProxy): | ||
| class V1(BaseParser): | ||
| _paragraph_selector = CSSSelector("div.article-text-wrapper > p") |
There was a problem hiding this comment.
In this article some bloat xp/LTO-Redaktion is also selected erroneously.
| @attribute | ||
| def topics(self) -> List[str]: | ||
| keywords = self.precomputed.meta.get("keywords") | ||
| if keywords: |
There was a problem hiding this comment.
This part is not necessary, you can use the utility function generic_topic_parsing
|
|
||
| @attribute | ||
| def topics(self) -> List[str]: | ||
| keywords = self.precomputed.meta.get("keywords") |
There was a problem hiding this comment.
I am afraid the keywords here are primarily used for SEO and don't really reflect the content of the article. It would be better to take the keywords from the section Mehr zum Thema at the bottom of the article.
3 participants
Description
This PR adds support for Legal Tribune Online (LTO), a German legal news publisher.
Changes
Testing
All unit tests pass successfully:
Checklist