bpo-30576 : Add HTTP compression support to http.server #2078

PierreQuentel · 2017-06-10T13:31:03Z

The Pull Request add support of HTTP compression to the built-in HTTP server. For some files (html, css, js, json...), compression reduces network load by sending the gzipped content instead of the original content. Most browsers support it.

The main change in is method send_head() of SimpleHTTPRequestHandler. If the request includes the header Accept-Encoding and its content includes "gzip", and if the content type determined by the file extension is in a list of compressed types, then the response includes the header Content-Encoding set to "gzip", the content type is set to that of the gzipped content, and the file object returned by send_head() points to the gzipped content.

If the original file is small, its content is loaded in memory and gzipped, the file object is and instance of io.BytesIO. If it is big, the gzipped content is stored in a temporary file.

The PR changes module mimetypes to add the mapping of extension ".json" to content type "application/json".

https://bugs.python.org/issue30576

# Conflicts: # Misc/NEWS

… on disk

matrixise · 2017-06-11T20:56:11Z

Doc/whatsnew/3.7.rst

@@ -219,6 +227,12 @@ math
 New :func:`~math.remainder` function, implementing the IEEE 754-style remainder
 operation. (Contributed by Mark Dickinson in :issue:`29962`.)

+mimetypes


could be in an other PR.

Same comment as below about extension .json

matrixise · 2017-06-11T20:59:06Z

Lib/http/server.py

@@ -7,6 +7,7 @@
 It does, however, optionally implement HTTP/1.1 persistent connections,
 as of version 0.3.

+


remove this line

Done in f9963f0

matrixise · 2017-06-11T21:00:48Z

Lib/mimetypes.py

@@ -439,6 +439,7 @@ def _default_mime_types():
        '.jpeg'   : 'image/jpeg',
        '.jpg'    : 'image/jpeg',
        '.js'     : 'application/javascript',
+        '.json'   : 'application/json',


in an other PR ? it's out of context

It's not totally out of context : files with extension .json are obvious candidates for compression, this is why I have included this minor change in mimetypes, thinking that it doesn't deserve a specific PR. But if you think it's better to open another one, I will.

Yes, any change to mimetypes should be a separate issue/PR.

matrixise · 2017-06-11T21:05:31Z

Lib/test/test_httpservers.py

+        self.cwd = os.getcwd()
+        basetempdir = tempfile.gettempdir()
+        os.chdir(basetempdir)
+        self.data = 10 * b'We are the knights who say Ni!'


and a test with a big file ?

Done in 586b8c3

@matrixise
Are you ok with the changes made since the first version, or do you see other things to do before the PR can be merged ?
If you think a separate PR is better for mimetypes I will submit it, it's just that I prefered to avoid the overhead of reviewing 2 PRs instead of one.

# Conflicts: # Misc/NEWS

…ck of memory

bitdancer

I already told you that a separate issue and PR are required for any change to mimetypes. Make me nosy on the issue, please (r.david.murray on the bug tracker).

bitdancer · 2017-06-12T19:28:22Z

Lib/mimetypes.py

@@ -439,6 +439,7 @@ def _default_mime_types():
        '.jpeg'   : 'image/jpeg',
        '.jpg'    : 'image/jpeg',
        '.js'     : 'application/javascript',
+        '.json'   : 'application/json',


Yes, any change to mimetypes should be a separate issue/PR.

PierreQuentel · 2017-07-01T20:40:14Z

@bitdancer
Ok, I have removed the mimetype changes from this PR and opened an issue on the tracker

PierreQuentel · 2017-07-07T07:05:52Z

@bitdancer @matrixise
With mimetypes now out of the scope, are there other things to do before the PR can be merged ?

…o {}, but allow compression in case a subclass extends it.

…s=True

…by assertNotIn(a, b)

…Chunked Transfer

… (if any) at random.

…line option "gzip" by "compressed"

PierreQuentel · 2017-10-11T12:31:02Z

@vadmium
I think I have made all the requested changes on the module, the test script and the documentation.

PierreQuentel · 2017-11-23T09:42:42Z

@rhettinger
Do you think you will have time to review this PR on time for inclusion in Python 3.7 ?

joshenders · 2018-07-28T01:24:18Z

Anyone able to review this? Ended up running against an issue today where this would have been nice to have.

jerrymarino · 2018-08-01T00:15:12Z

+1 I'd be great to see this one land 👍

brettcannon · 2018-09-11T23:05:42Z

Thanks for the PR, @PierreQuentel , but I'm closing this as me and various other core devs can't quite muster enough excitement about what this feature brings to warrant wanting to support it for a couple of decades.

joshenders · 2018-09-12T18:46:59Z

Brett, this is a capability of HTTP that has existed for decades that Python still doesn’t support. It’s shipped with every major browser and web server in the last 20 years. Part of the reason people are moving to Golang is for its superior HTTP libraries. I think this PR deserves a second look.

…

On Tue, Sep 11, 2018 at 4:06 PM Brett Cannon ***@***.***> wrote: Closed #2078 <#2078>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2078 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGrYNlZcQqpNy7_5TZC9MKPtGc49RJ_ks5uaEHpgaJpZM4N2HSc> .

sijojlouis · 2018-09-22T07:32:51Z

+1

PierreQuentel · 2018-09-22T15:22:58Z

The module is released as httpcompressionserver on PyPI.

sijojlouis · 2018-09-22T20:48:30Z

@PierreQuentel Great! thanks for this, I’ll try it out soon.

I have a suggestion for running (usage command) the module, I loved the simplicity of http.server - so was thinking if this could be so too, like http.eserver (e for encoded from html header or maybe c for compression).

Pierre Quentel added 6 commits June 9, 2017 18:46

Code and documentation for HTTP compression support in http.server

c9b880a

Merge remote-tracking branch 'upstream2/master'

481f69e

# Conflicts: # Misc/NEWS

Implementation change : if original file is big, store zipped content…

5a3693e

… on disk

Resolve conflict

bed3047

resolve conflict in NEWS

d82b7cd

For gzipped content on disk, set file pointer at position 0.

e9deb89

the-knights-who-say-ni added the CLA signed label Jun 10, 2017

Merge branch 'master' into master

39da21b

matrixise reviewed Jun 11, 2017

View reviewed changes

Pierre Quentel and others added 13 commits June 12, 2017 20:41

Merge remote-tracking branch 'upstream2/master'

f9963f0

# Conflicts: # Misc/NEWS

resolve conflict in NEWS

ace7999

Fix conflicts

128f6a6

Add HTTP compression test for big files

586b8c3

Merge branch 'master' into master

a61109e

Merge branch 'master' into master

6093cae

Support more Accept-Encoding values, eg "*", "gzip;q=0.5", "GZIP"

b27144a

Merge remote-tracking branch 'upstream2/master'

3525f56

# Conflicts: # Misc/NEWS

Merge branch 'master' of https://github.com/PierreQuentel/cpython

1ad8139

# Conflicts: # Misc/NEWS

More documentation

8a916fc

Remove trailing whitespace

6f193f9

Handle the case when storing the temporary gzip file fails, eg for la…

655f753

…ck of memory

Add comment.

2433769

bitdancer previously requested changes Jul 1, 2017

View reviewed changes

Pierre Quentel added 5 commits July 1, 2017 21:21

Remove changes to mimetypes.py

cfdcf93

Restore tests in test_httpservers, remove test with .json extension

9d8e668

Remove whitespaces

b700014

Remove whitespaces

6a31d60

Remove whitespaces

9830b89

Pierre Quentel added 8 commits October 1, 2017 14:47

Remove unused import (gzip)

1bdbb3a

If zlib is not available, set SimpleHTTPRequestHandler.compressions t…

27d02bd

…o {}, but allow compression in case a subclass extends it.

Replace sorted(...)[-1] by max(...)

ca68881

For shutil.rmtree, instead of a try/except, set argument ignore_error…

d6157d4

…s=True

Replace assertTrue(a in b) by assertIn(a, b) and assertFalse(a in b) …

a456fe6

…by assertNotIn(a, b)

Remove default_request_version

50f0a85

Merge remote-tracking branch 'upstream2/master'

87a2cc3

Restore Lib/test/bisect.py

b9f9599

bedevere-bot added the awaiting core review label Oct 1, 2017

Pierre Quentel added 9 commits October 1, 2017 16:36

Compressed data generators may send empty bytes ; adapt do_GET() for …

7881e1f

…Chunked Transfer

Remove trace

2332d91

If Accept-Encoding is set to *, use one of the supported compressions…

70f4738

… (if any) at random.

Minor changes in comments and code formatting.

d44c8bf

Adapt test with Accept-Encoding set to "*"

d844ca6

Merge remote-tracking branch 'upstream2/master'

634f572

Handle missing zlib or bz2 modules

7cfa86e

Replace command line option "--gzip" by "--compressed"

829b992

Improve documentation for attribute "compressions" ; replace command …

383ee33

…line option "gzip" by "compressed"

Merge branch 'master' into master

9d04caa

Merge branch 'master' into master

71e9090

brettcannon closed this Sep 11, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-30576 : Add HTTP compression support to http.server #2078

bpo-30576 : Add HTTP compression support to http.server #2078

PierreQuentel commented Jun 10, 2017 •

edited by bedevere-bot

Loading

matrixise Jun 11, 2017

PierreQuentel Jun 12, 2017

matrixise Jun 11, 2017

PierreQuentel Jun 12, 2017

matrixise Jun 11, 2017

PierreQuentel Jun 12, 2017

bitdancer Jun 12, 2017

matrixise Jun 11, 2017

PierreQuentel Jun 12, 2017

PierreQuentel Jul 1, 2017

bitdancer left a comment

bitdancer Jun 12, 2017

PierreQuentel commented Jul 1, 2017

PierreQuentel commented Jul 7, 2017

PierreQuentel commented Oct 11, 2017

PierreQuentel commented Nov 23, 2017 •

edited

Loading

joshenders commented Jul 28, 2018

jerrymarino commented Aug 1, 2018

brettcannon commented Sep 11, 2018

joshenders commented Sep 12, 2018 via email

sijojlouis commented Sep 22, 2018

PierreQuentel commented Sep 22, 2018

sijojlouis commented Sep 22, 2018

		@@ -7,6 +7,7 @@
		It does, however, optionally implement HTTP/1.1 persistent connections,
		as of version 0.3.

bpo-30576 : Add HTTP compression support to http.server #2078

bpo-30576 : Add HTTP compression support to http.server #2078

Conversation

PierreQuentel commented Jun 10, 2017 • edited by bedevere-bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bitdancer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PierreQuentel commented Jul 1, 2017

PierreQuentel commented Jul 7, 2017

PierreQuentel commented Oct 11, 2017

PierreQuentel commented Nov 23, 2017 • edited Loading

joshenders commented Jul 28, 2018

jerrymarino commented Aug 1, 2018

brettcannon commented Sep 11, 2018

joshenders commented Sep 12, 2018 via email

sijojlouis commented Sep 22, 2018

PierreQuentel commented Sep 22, 2018

sijojlouis commented Sep 22, 2018

PierreQuentel commented Jun 10, 2017 •

edited by bedevere-bot

Loading

PierreQuentel commented Nov 23, 2017 •

edited

Loading