Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

github.com line reference links result in "Non-OK status: 429" #105

Closed
kaushalmodi opened this issue Sep 28, 2018 · 3 comments
Closed

github.com line reference links result in "Non-OK status: 429" #105

kaushalmodi opened this issue Sep 28, 2018 · 3 comments
Labels
concurrency duplicate

Comments

@kaushalmodi
Copy link

kaushalmodi commented Sep 28, 2018 •
edited
Loading

Hello,

If I run htmltest on a small test file like this:

<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<!--  This file is generated by Nim. -->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
    <head>
    </head>
    <body>
        <a href="https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L611"
           class="link-seesrc" target="_blank">Source</a>
        <a href="https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L611" class="link-seesrc" target="_blank" >Edit</a>
    </body>
</html>

It works fine.

But if I run it on the actual file with a lot of github.com links, I get:

Flood of *Non-OK status: 429* errors: 
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L611
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L611
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L612
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L612
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L613
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L613
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L614
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L614
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L615
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L615
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L616
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L616
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L617
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L617
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L618
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L618
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L619
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L619
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L620
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L620
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L621
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L621
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L622
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L622
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L623
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L623
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L624
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L624
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L625
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L625
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L626
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L626
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L627
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L627
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/tree/master/lib/js/dom.nim#L628
  Non-OK status: 429 --- dom.html --> https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L628

.. truncated

So, looking up on HTTP 429 error, I find https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429.

So there's no way to check the validity of github links?

Or, this may be a feature request.. If you already check https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L628, can you skip checking of any other https://github.com/nim-lang/Nim/edit/devel/lib/js/dom.nim#L.*? That way the github server doesn't start returning 429.

  • That is, if the Github link for a file is checked to be valid once, ignore further checks to that same link, ignoring the "#Lnnn" portion.
@ssbarnea
Copy link

I encounter this even with normal urls when using concurrent mode, clearly we need a way to cache them that works in concurrent mode.

@wjdp
Copy link
Owner

wjdp commented Dec 28, 2019

Thanks for the report @kaushalmodi. As mentioned in #138 concurrent mode is has several major issues. External link checking is very broken. I would not recommend using it at present.

@wjdp
Copy link
Owner

wjdp commented Dec 28, 2019 •
edited
Loading

429's are a known issue. You have many pages with these links and concurrent mode is making far too many external link checks at the same time so GitHub is blocking you (temporarily) as it sees it as a DoS.

When concurrency is turned off, only a single link is checked at a time.

This issue is a duplicate of #5, thanks for highlighting, closing.

@wjdp wjdp added the duplicate label Dec 28, 2019
@wjdp wjdp closed this as completed Dec 28, 2019
@ChrisChinchilla ChrisChinchilla mentioned this issue Oct 15, 2020
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
concurrency duplicate
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ssbarnea @wjdp @kaushalmodi