Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Web crawler: language detection does not work with <html lang='xx'> #739

Closed
AlexandreToyer opened this issue Jul 1, 2014 · 0 comments
Closed

Web crawler: language detection does not work with <html lang='xx'> #739

AlexandreToyer opened this issue Jul 1, 2014 · 0 comments
Assignees
@emmanuel-keller
Labels
bug
Milestone
v1.5.4

Comments

@AlexandreToyer
Copy link
Member

When crawling pages having an attribute lang on root node html web crawler does not use it to get language of page.

For instance for some pages where this attribute is there I get "ngram recognition" as the language detection method, but it should have used the "html lang attribute".

Thanks,
Alexandre
emmanuel-keller added a commit to emmanuel-keller/opensearchserver that referenced this issue Jul 1, 2014
@emmanuel-keller
Fix jaeksoft#739. HTMLCleaner does not preserve the lang attribute on…
6574233 
… DOM

serialization.
emmanuel-keller added a commit to emmanuel-keller/opensearchserver that referenced this issue Jul 1, 2014
@emmanuel-keller
Fix jaeksoft#739. HTMLCleaner does not preserve the lang attribute on…
00ba281 
… DOM

serialization.
emmanuel-keller added a commit that referenced this issue Jul 1, 2014
@emmanuel-keller
Merge pull request #740 from emmanuel-keller/1.5
9fe6643 
Fix #739. HTMLCleaner does not preserve the lang attribute on DOM
emmanuel-keller added a commit that referenced this issue Jul 1, 2014
@emmanuel-keller
Merge pull request #741 from emmanuel-keller/1.6
4170d57 
Fix #739. HTMLCleaner does not preserve the lang attribute on DOM
emmanuel-keller added a commit that referenced this issue Jul 1, 2014
@emmanuel-keller
Merge pull request #742 from emmanuel-keller/master
8cabba9 
Fix #739. HTMLCleaner does not preserve the lang attribute on DOM
@emmanuel-keller emmanuel-keller added this to the v1.5.4 milestone Jul 1, 2014
@NicoHaase NicoHaase mentioned this issue Apr 9, 2018
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@emmanuel-keller emmanuel-keller

Labels
bug
Projects
None yet
Milestone
v1.5.4
Development

No branches or pull requests
2 participants
@emmanuel-keller @AlexandreToyer