Skip to content

wikihow does fetch article CSS properly (at least it does not uses a retry mechanism) #163

New issue
New issue
Open
Open
wikihow does fetch article CSS properly (at least it does not uses a retry mechanism)#163

Description

@benoit74

benoit74
openedon May 17, 2024

Task: https://farm.openzim.org/pipeline/3481640e-bb1e-4e47-9a82-edb4aa2d13b2/debug

Logs:

[MainThread::2024-03-21 15:03:33,763] DEBUG:17394/269089
[MainThread::2024-03-21 15:03:33,763] INFO:>> Article:Start-a-Toyota-Prius-(US)
[MainThread::2024-03-21 15:03:34,887] DEBUG:URL `https://www.wikihow.com/extensions/wikihow/IMR/assets/chat_wH.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/f/f2/CommunityAvatar3.png/-crop-104-104-104px-CommunityAvatar3.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/f/f5/CommunityAvatar4.png/-crop-104-104-104px-CommunityAvatar4.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/e/e1/Disable-Reverse-Beep-in-a-Toyota-Prius-Step-20.jpg/-crop-342-184-245px-Disable-Reverse-Beep-in-a-Toyota-Prius-Step-20.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/a/aa/Check-the-Hybrid-System-on-a-Prius-Step-6.jpg/-crop-342-184-245px-Check-the-Hybrid-System-on-a-Prius-Step-6.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/e/ec/Glovebox_filter_8.JPG/-crop-342-184-204px-Glovebox_filter_8.JPG` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/3/3e/Change-the-HID-Headlights-on-a-2007-Prius-%28Without-Removing-Bumper%29-Step-27.jpg/-crop-342-184-245px-Change-the-HID-Headlights-on-a-2007-Prius-%28Without-Removing-Bumper%29-Step-27.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/f/fb/Turbo-Charge-a-Corolla-Step-21.jpg/-crop-342-184-245px-Turbo-Charge-a-Corolla-Step-21.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/b/be/Drive-a-Prius-in-Ice-or-Snow-Step-10-Version-2.jpg/-crop-342-184-245px-Drive-a-Prius-in-Ice-or-Snow-Step-10-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/f/f5/Tom_eisenberg.png/-crop-100-100-99px-Tom_eisenberg.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/f/f5/Tom_eisenberg.png/-crop-100-100-99px-Tom_eisenberg.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/b/b0/CommunityAvatar2.png/-crop-72-72-72px-CommunityAvatar2.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/0/0a/CommunityAvatar1.png/-crop-72-72-72px-CommunityAvatar1.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/4/4e/Empath.png/-crop-127-140-127px-Empath.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/7/76/Canva-love-me-love-me-not.jpg/-crop-127-140-127px-Canva-love-me-love-me-not.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/4/45/Aesthetic-Quiz.png/-crop-127-140-127px-Aesthetic-Quiz.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/e/ed/Formal_dance.jpg/-crop-127-140-127px-Formal_dance.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/4/48/Sleep-quiz.png/-crop-127-140-127px-Sleep-quiz.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/0/06/Zodiac-Dating-Quiz2.png/-crop-127-140-127px-Zodiac-Dating-Quiz2.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/e/e1/Disable-Reverse-Beep-in-a-Toyota-Prius-Step-20.jpg/-crop-127-140-127px-Disable-Reverse-Beep-in-a-Toyota-Prius-Step-20.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/a/aa/Check-the-Hybrid-System-on-a-Prius-Step-6.jpg/-crop-127-140-127px-Check-the-Hybrid-System-on-a-Prius-Step-6.jpg` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/e/ec/Glovebox_filter_8.JPG/-crop-127-140-127px-Glovebox_filter_8.JPG` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/1/14/LR23-D-Amirra-Fidgets-Happily.png/-crop-127-140-127px-LR23-D-Amirra-Fidgets-Happily.png` already processed.
[MainThread::2024-03-21 15:03:34,888] DEBUG:URL `https://www.wikihow.com/images/thumb/6/67/Contact-Your-MP-Step-2.jpg/-crop-127-140-127px-Contact-Your-MP-Step-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/2/2c/Love-You-vs-I-Love-You-Step-13-Version-2.jpg/-crop-127-140-127px-Love-You-vs-I-Love-You-Step-13-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/7/71/Accept-Blame-when-You-Deserve-It-Step-11-Version-2.jpg/-crop-127-140-127px-Accept-Blame-when-You-Deserve-It-Step-11-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/3/38/Use-a-Moka-Pot-Step-10.jpg/-crop-127-140-127px-Use-a-Moka-Pot-Step-10.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/6/62/Clean-White-Converse-Step-35-Version-2.jpg/-crop-127-140-127px-Clean-White-Converse-Step-35-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/6/62/Be-Less-Emotional-Step-17-Version-2.jpg/-crop-127-140-127px-Be-Less-Emotional-Step-17-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/7/7b/Know-if-You%27re-Dating-a-Toxic-Person-Step-12.jpg/-crop-127-140-127px-Know-if-You%27re-Dating-a-Toxic-Person-Step-12.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/d/da/Add-Sounds-to-Discord-Soundboard-Step-16.jpg/-crop-127-140-127px-Add-Sounds-to-Discord-Soundboard-Step-16.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/a/ae/Get-Free-Clothes-from-Shein-Step-10.jpg/-crop-127-140-127px-Get-Free-Clothes-from-Shein-Step-10.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/7/76/Eye_color_quiz.png/-crop-127-140-127px-Eye_color_quiz.png` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/9/9e/Phonetically-Spell-My-Name-Step-9.jpg/-crop-127-140-127px-Phonetically-Spell-My-Name-Step-9.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/b/bf/Access-YouTube-at-School-Step-32.jpg/-crop-127-140-127px-Access-YouTube-at-School-Step-32.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/d/db/Use-Watercolor-Pencils-Step-13-Version-2.jpg/-crop-127-140-127px-Use-Watercolor-Pencils-Step-13-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/e/ec/Read-an-Egg-Cleanse-Step-1.jpg/-crop-127-140-127px-Read-an-Egg-Cleanse-Step-1.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/9/91/Make-Your-Dog-Stop-Sleeping-in-Your-Bed-Step-15-Version-4.jpg/-crop-127-140-127px-Make-Your-Dog-Stop-Sleeping-in-Your-Bed-Step-15-Version-4.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/b/b8/Be-More-Spontaneous-Step-10-Version-2.jpg/-crop-127-140-127px-Be-More-Spontaneous-Step-10-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/9/92/Cut-Hair-Straight-Step-22.jpg/-crop-127-140-127px-Cut-Hair-Straight-Step-22.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/2/23/Get-in-Shape-Step-16.jpg/-crop-127-140-127px-Get-in-Shape-Step-16.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/e/e4/Cut-Side-Swept-Bangs-Step-20-Version-2.jpg/-crop-127-140-127px-Cut-Side-Swept-Bangs-Step-20-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/f/f2/Read-Body-Language-Step-13-Version-5.jpg/-crop-127-140-127px-Read-Body-Language-Step-13-Version-5.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/2/2f/Windows-11-Mobile-Hotspot-Save-New-Hotspot-Properties.png/-crop-127-140-127px-Windows-11-Mobile-Hotspot-Save-New-Hotspot-Properties.png` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/2/23/Exercise-at-Work-Every-Day-Step-1-Version-2.jpg/-crop-127-140-127px-Exercise-at-Work-Every-Day-Step-1-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/2/2a/Play-Basketball-Step-29-Version-4.jpg/-crop-127-140-127px-Play-Basketball-Step-29-Version-4.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/d/de/Fold-Boxer-Briefs-Step-10-Version-3.jpg/-crop-127-140-127px-Fold-Boxer-Briefs-Step-10-Version-3.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/d/df/Suspend-an-Object-in-Jello-Step-13.jpg/-crop-127-140-127px-Suspend-an-Object-in-Jello-Step-13.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/c/cb/Open-a-Durian-Step-7.jpg/-crop-127-140-127px-Open-a-Durian-Step-7.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/2/2c/Make-Ginger-Tea-or-Tisane-Step-9-Version-2.jpg/-crop-127-140-127px-Make-Ginger-Tea-or-Tisane-Step-9-Version-2.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/5/50/Take-Lenses-Out-of-Glasses-Step-10.jpg/-crop-127-140-127px-Take-Lenses-Out-of-Glasses-Step-10.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/8/80/Cook-Chorizo-Step-24.jpg/-crop-127-140-127px-Cook-Chorizo-Step-24.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/5/52/Owl-Hooting-Meaning-Step-16.jpg/-crop-127-140-127px-Owl-Hooting-Meaning-Step-16.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/b/bc/Why-Do-Dogs-Sigh-Step-12-Version-3.jpg/-crop-127-140-127px-Why-Do-Dogs-Sigh-Step-12-Version-3.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/8/8a/Gryffindor-Traits-Step-17.jpg/-crop-127-140-127px-Gryffindor-Traits-Step-17.jpg` already processed.
[MainThread::2024-03-21 15:03:34,889] DEBUG:URL `https://www.wikihow.com/images/thumb/0/02/Know-when-Your-Computer-Was-Last-Used-Step-9.jpg/-crop-127-140-127px-Know-when-Your-Computer-Was-Last-Used-Step-9.jpg` already processed.
[MainThread::2024-03-21 15:03:34,890] DEBUG:URL `https://www.wikihow.com/images/thumb/3/39/Angel_Number_Quiz_image.png/-crop-127-140-127px-Angel_Number_Quiz_image.png` already processed.
[MainThread::2024-03-21 15:03:34,890] DEBUG:URL `https://www.wikihow.com/images/thumb/6/6a/Platonic-Marriage-Step-14.jpg/-crop-127-140-127px-Platonic-Marriage-Step-14.jpg` already processed.
[IMG-T-6::2024-03-21 15:03:35,062] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/f/f5/Start-a-Toyota-Prius-%28US%29-Step-3-Version-3.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-3-Version-3.jpg into ZIM::images/3139121941-v4-460px-Start-a-Toyota-Prius-(US)-Step-3-Version-3.webp
[IMG-T-0::2024-03-21 15:03:35,142] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/5/54/Start-a-Toyota-Prius-%28US%29-Step-4-Version-3.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-4-Version-3.jpg into ZIM::images/2405511860-v4-460px-Start-a-Toyota-Prius-(US)-Step-4-Version-3.webp
[IMG-T-5::2024-03-21 15:03:35,151] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/9/90/Start-a-Toyota-Prius-%28US%29-Step-5-Version-3.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-5-Version-3.jpg into ZIM::images/2441818810-v4-460px-Start-a-Toyota-Prius-(US)-Step-5-Version-3.webp
[IMG-T-2::2024-03-21 15:03:35,183] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/d/de/Start-a-Toyota-Prius-%28US%29-Step-2-Version-3.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-2-Version-3.jpg into ZIM::images/3455660863-v4-460px-Start-a-Toyota-Prius-(US)-Step-2-Version-3.webp
[IMG-T-3::2024-03-21 15:03:35,189] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/6/67/Start-a-Toyota-Prius-%28US%29-Step-6-Version-3.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-6-Version-3.jpg into ZIM::images/2454270653-v4-460px-Start-a-Toyota-Prius-(US)-Step-6-Version-3.webp
[IMG-T-9::2024-03-21 15:03:35,217] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/e/eb/Start-a-Toyota-Prius-%28US%29-Step-8-Version-2.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-8-Version-2.jpg into ZIM::images/3479384904-v4-460px-Start-a-Toyota-Prius-(US)-Step-8-Version-2.webp
[IMG-T-7::2024-03-21 15:03:35,223] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/e/e2/Start-a-Toyota-Prius-%28US%29-Step-1-Version-3.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-1-Version-3.jpg into ZIM::images/3090363148-v4-460px-Start-a-Toyota-Prius-(US)-Step-1-Version-3.webp
[IMG-T-1::2024-03-21 15:03:35,252] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/d/d2/Start-a-Toyota-Prius-%28US%29-Step-7-Version-2.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-7-Version-2.jpg into ZIM::images/3106222868-v4-460px-Start-a-Toyota-Prius-(US)-Step-7-Version-2.webp
[IMG-T-4::2024-03-21 15:03:35,277] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/b/b6/Start-a-Toyota-Prius-%28US%29-Step-10-Version-2.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-10-Version-2.jpg into ZIM::images/288633704-v4-460px-Start-a-Toyota-Prius-(US)-Step-10-Version-2.webp
[IMG-T-8::2024-03-21 15:03:35,284] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/f/f2/Start-a-Toyota-Prius-%28US%29-Step-9-Version-2.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-9-Version-2.jpg into ZIM::images/3147903772-v4-460px-Start-a-Toyota-Prius-(US)-Step-9-Version-2.webp
[IMG-T-9::2024-03-21 15:03:35,461] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/c/c1/Clean-Headlights-Step-22.jpg/-crop-127-140-127px-Clean-Headlights-Step-22.jpg into ZIM::images/3365283943--crop-127-140-127px-Clean-Headlights-Step-22.webp
[IMG-T-6::2024-03-21 15:03:35,567] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/d/df/Start-a-Toyota-Prius-%28US%29-Step-11-Version-2.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-11-Version-2.jpg into ZIM::images/683619230-v4-460px-Start-a-Toyota-Prius-(US)-Step-11-Version-2.webp
[IMG-T-2::2024-03-21 15:03:35,604] DEBUG:Attempting download of S3::https/www.wikihow.com/images/avatarOut/5/53/3565366.jpg?20160502110857 into ZIM::images/2537952873-3565366.webp
[IMG-T-5::2024-03-21 15:03:35,627] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/f/ff/Start-a-Toyota-Prius-%28US%29-Step-13-Version-2.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-13-Version-2.jpg into ZIM::images/725955494-v4-460px-Start-a-Toyota-Prius-(US)-Step-13-Version-2.webp
[IMG-T-0::2024-03-21 15:03:35,628] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/8/8b/Start-a-Toyota-Prius-%28US%29-Step-12-Version-2.jpg/v4-460px-Start-a-Toyota-Prius-%28US%29-Step-12-Version-2.jpg into ZIM::images/4284691268-v4-460px-Start-a-Toyota-Prius-(US)-Step-12-Version-2.webp
[IMG-T-7::2024-03-21 15:03:35,693] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/6/66/Get-the-Best-Gas-Mileage-from-Your-Toyota-Prius-Step-30.jpg/-crop-342-184-245px-Get-the-Best-Gas-Mileage-from-Your-Toyota-Prius-Step-30.jpg into ZIM::images/1645690422--crop-342-184-245px-Get-the-Best-Gas-Mileage-from-Your-Toyota-Prius-Step-30.webp
[IMG-T-3::2024-03-21 15:03:35,729] DEBUG:Attempting download of S3::https/www.wikihow.com/images/thumb/c/c1/Clean-Headlights-Step-22.jpg/-crop-342-184-245px-Clean-Headlights-Step-22.jpg into ZIM::images/3384551535--crop-342-184-245px-Clean-Headlights-Step-22.webp
[MainThread::2024-03-21 15:19:17,911] ERROR:Interrupting process due to error: ('Connection aborted.', TimeoutError(110, 'Connection timed out'))
[MainThread::2024-03-21 15:19:17,912] ERROR:('Connection aborted.', TimeoutError(110, 'Connection timed out'))
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 404, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1058, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.8/ssl.py", line 1073, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.8/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 404, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1058, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.8/ssl.py", line 1073, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.8/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', TimeoutError(110, 'Connection timed out'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/wikihow2zim-1.2.3-py3.8.egg/wikihow2zim/scraper.py", line 986, in run
    self.scrape_articles()
  File "/usr/local/lib/python3.8/site-packages/wikihow2zim-1.2.3-py3.8.egg/wikihow2zim/scraper.py", line 506, in scrape_articles
    if not self.scrape_article(article):
  File "/usr/local/lib/python3.8/site-packages/wikihow2zim-1.2.3-py3.8.egg/wikihow2zim/scraper.py", line 672, in scrape_article
    page_linked_styles=self.get_style_urls(soup),
  File "/usr/local/lib/python3.8/site-packages/wikihow2zim-1.2.3-py3.8.egg/wikihow2zim/scraper.py", line 88, in get_style_urls
    self.add_css(url)
  File "/usr/local/lib/python3.8/site-packages/wikihow2zim-1.2.3-py3.8.egg/wikihow2zim/scraper.py", line 226, in add_css
    source = url if inline else self.get_from_cache(url).decode("UTF-8")
  File "/usr/local/lib/python3.8/site-packages/wikihow2zim-1.2.3-py3.8.egg/wikihow2zim/scraper.py", line 206, in get_from_cache
    content = self.session.get(url).content
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', TimeoutError(110, 'Connection timed out'))

We see that the call stack is:

  • scrape_article
  • get_style_urls
  • add_css
  • get_from_cache
  • session.get

However, get_from_cache is not using any retry mechanism, plus its docstring states that it should only be used for devel/debug purposes:

wikihow/wikihow2zim/scraper.py

Lines 195 to 199 in 275bac2

def get_from_cache(self, url: str) -> bytes:
"""retrieve from local cache if present, otherwise download it first
Only useful for devel/debug because there are many resources linked
to assets and source website is quite slow"""

Either add_css is not using the proper fetch method, or the docstring + method behavior are inaccurate (we might use get_from_cache for prod but then it misses a proper backoff mechanism like the fetch method which is used for web scraping calls)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions