Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

remove deprecated urllib.request.URLopener/FancyURLopener #84850

Closed
PetterS mannequin opened this issue May 18, 2020 · 9 comments
Closed

remove deprecated urllib.request.URLopener/FancyURLopener #84850

PetterS mannequin opened this issue May 18, 2020 · 9 comments
Labels
3.11 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@PetterS
Copy link
Mannequin

PetterS mannequin commented May 18, 2020 •
edited by bedevere-app bot
Loading

BPO 40673
Nosy @vstinner, @PetterS, @remilapeyre, @idomic

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-05-18.21:30:13.142>
labels = ['type-feature', 'library', '3.11']
title = 'remove deprecated urllib.request.URLopener/FancyURLopener'
updated_at = <Date 2021-11-23.13:31:04.736>
user = 'https://github.com/PetterS'

bugs.python.org fields:

activity = <Date 2021-11-23.13:31:04.736>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-05-18.21:30:13.142>
creator = 'Petter S'
dependencies = []
files = []
hgrepos = []
issue_num = 40673
keywords = []
message_count = 7.0
messages = ['369291', '369293', '369294', '369295', '369300', '369886', '406839']
nosy_count = 4.0
nosy_names = ['vstinner', 'Petter S', 'remi.lapeyre', 'Ido Michael']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue40673'
versions = ['Python 3.11']

Linked PRs
@PetterS
Copy link
Mannequin Author

PetterS mannequin commented May 18, 2020

The following code crashes with TypeError:

URLopener().open("unknown_proxy://test")

the expected behavior is an OSError, which is normally raised for unknown protocols.

This implementation of directly calling a method called "unknown_proxy" is fragile and was recently the subject of a security issue: https://bugs.python.org/issue35907 (CVE-2019-9948)

Would be good to make this more robust.

@PetterS PetterS mannequin added 3.7 (EOL) end of life 3.8 (EOL) end of life 3.9 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels May 18, 2020
@PetterS PetterS mannequin changed the title UrlOpener raises different exceptions based on implementation detail urllib.request.URLopener raises different exceptions based on implementation detail May 18, 2020
@PetterS PetterS mannequin changed the title UrlOpener raises different exceptions based on implementation detail urllib.request.URLopener raises different exceptions based on implementation detail May 18, 2020
@PetterS
Copy link
Mannequin Author

PetterS mannequin commented May 18, 2020

I can fix this, but it does not have high priority.

@remilapeyre
Copy link
Mannequin

remilapeyre mannequin commented May 18, 2020

URLOpener has been deprecated since Python3.3 (https://github.com/python/cpython/blob/master/Lib/urllib/request.py#L1703-L1705) maybe it should just be removed from the codebase?

@PetterS
Copy link
Mannequin Author

PetterS mannequin commented May 18, 2020

I see. Perhaps that is the resolution then.

@vstinner
Copy link
Member

URLOpener has been deprecated since Python3.3 (https://github.com/python/cpython/blob/master/Lib/urllib/request.py#L1703-L1705) maybe it should just be removed from the codebase?

It emits a DeprecationWarning since at least Python 3.6, so yes, it's now fine to remove the feature. It's also documented as deprecated in:
https://docs.python.org/dev/library/urllib.request.html#urllib.request.URLopener

Maybe we kept it longer than usual because it was common in Python 2.7 to use it, and we tried to keep support for code base compatible with Python 2 and Python 3? I don't know.

I'm fine with removing the class in Python 3.10, but IMO it's too late for Python 3.9 (feature freeze is today).

@idomic
Copy link
Mannequin

idomic mannequin commented May 25, 2020

Does it includes the FancyURLopener? This class is derived from URLopener.

Also what needs to be done?
Remove the URLopener, URLopener_Tests and DummyURLopeners ?

@iritkatriel iritkatriel added 3.11 only security fixes and removed 3.7 (EOL) end of life 3.8 (EOL) end of life 3.9 only security fixes labels Nov 23, 2021
@iritkatriel iritkatriel changed the title urllib.request.URLopener raises different exceptions based on implementation detail remove deprecated urllib.request.URLopener/FancyURLopener Nov 23, 2021
@iritkatriel iritkatriel added 3.11 only security fixes and removed 3.7 (EOL) end of life 3.8 (EOL) end of life 3.9 only security fixes labels Nov 23, 2021
@iritkatriel iritkatriel changed the title urllib.request.URLopener raises different exceptions based on implementation detail remove deprecated urllib.request.URLopener/FancyURLopener Nov 23, 2021
@vstinner
Copy link
Member

I tried but failed to remove urllib deprecated functions: see bpo-45084.

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
barneygale added a commit to barneygale/cpython that referenced this issue Oct 20, 2024
@barneygale
pythonGH-84850: Remove urllib.request.URLopener and FancyURLopener
2b3f775 
These classes have been deprecated since Python 3.3.
@bedevere-app bedevere-app bot mentioned this issue Oct 20, 2024
Merged
barneygale added a commit to barneygale/cpython that referenced this issue Oct 29, 2024
@barneygale
Merge branch 'main' into pythongh-84850
ea17ffc
hugovk pushed a commit that referenced this issue Nov 19, 2024
@barneygale
GH-84850: Remove urllib.request.URLopener and FancyURLopener (#12…
4d77197 
…5739)
@hugovk
Copy link
Member

hugovk commented Nov 19, 2024

They've been removed in #125739.

TODO:

I think that we should provide some hints on how to replace code using removed APIs. Give some examples maybe?

#125739 (comment)

barneygale added a commit to barneygale/cpython that referenced this issue Nov 19, 2024
@barneygale
pythonGH-84850: Improve whatsnew entry for [Fancy]URLopener depreca…
cd4a191 
…tion
@bedevere-app bedevere-app bot mentioned this issue Nov 19, 2024
Merged
hugovk pushed a commit that referenced this issue Nov 20, 2024
@barneygale
GH-84850: Improve whatsnew entry for [Fancy]URLopener removal (#127032
9d2a879 
)
@barneygale
Copy link
Contributor

urllib.request.URLopener and FancyURLopener are removed in Python 3.14. Thanks all.

hroncok added a commit to hroncok/six that referenced this issue Nov 22, 2024
@hroncok
Python 3.14 removed the URLopener and FancyURLopener classes from url…
1fbfc29 
…lib.request

python/cpython#84850

> Remove URLopener and FancyURLopener classes from urllib.request.
> They had previously raised DeprecationWarning since Python 3.3.

The test failures were:

    __________________ test_move_items_urllib_request[URLopener] ___________________

    item_name = 'URLopener'

        @pytest.mark.parametrize("item_name",
                                  [item.name for item in six._urllib_request_moved_attributes])
        def test_move_items_urllib_request(item_name):
            """Ensure that everything loads correctly."""
            assert item_name in dir(six.moves.urllib.request)
    >       getattr(six.moves.urllib.request, item_name)
    E       AttributeError: module 'six.moves.urllib.request' has no attribute 'URLopener'

    test_six.py:170: AttributeError
    ________________ test_move_items_urllib_request[FancyURLopener] ________________

    item_name = 'FancyURLopener'

        @pytest.mark.parametrize("item_name",
                                  [item.name for item in six._urllib_request_moved_attributes])
        def test_move_items_urllib_request(item_name):
            """Ensure that everything loads correctly."""
            assert item_name in dir(six.moves.urllib.request)
    >       getattr(six.moves.urllib.request, item_name)
    E       AttributeError: module 'six.moves.urllib.request' has no attribute 'FancyURLopener'

    test_six.py:170: AttributeError
@hroncok hroncok mentioned this issue Nov 22, 2024
Merged
benjaminp pushed a commit to benjaminp/six that referenced this issue Dec 1, 2024
@hroncok
Python 3.14 removed the URLopener and FancyURLopener classes from url…
9a05aab 
…lib.request (#388)

python/cpython#84850

> Remove URLopener and FancyURLopener classes from urllib.request.
> They had previously raised DeprecationWarning since Python 3.3.

The test failures were:

    __________________ test_move_items_urllib_request[URLopener] ___________________

    item_name = 'URLopener'

        @pytest.mark.parametrize("item_name",
                                  [item.name for item in six._urllib_request_moved_attributes])
        def test_move_items_urllib_request(item_name):
            """Ensure that everything loads correctly."""
            assert item_name in dir(six.moves.urllib.request)
    >       getattr(six.moves.urllib.request, item_name)
    E       AttributeError: module 'six.moves.urllib.request' has no attribute 'URLopener'

    test_six.py:170: AttributeError
    ________________ test_move_items_urllib_request[FancyURLopener] ________________

    item_name = 'FancyURLopener'

        @pytest.mark.parametrize("item_name",
                                  [item.name for item in six._urllib_request_moved_attributes])
        def test_move_items_urllib_request(item_name):
            """Ensure that everything loads correctly."""
            assert item_name in dir(six.moves.urllib.request)
    >       getattr(six.moves.urllib.request, item_name)
    E       AttributeError: module 'six.moves.urllib.request' has no attribute 'FancyURLopener'

    test_six.py:170: AttributeError
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.11 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@vstinner @barneygale @iritkatriel @hugovk