do not use readline when reading the content of a part in the multipart reader

[arthurdarcet]

multipart uploads to an aiohttp server with a "large" file containing no new line fails with a ValueError('Line is too long'):

import aiohttp
import aiohttp.web
import asyncio


async def view(request):
	reader = await request.multipart()
	async for part in reader:
		v = await part.read(decode=True)
		print(part, len(v))
	return aiohttp.web.Response(text="OK")

async def server(loop):
	app = aiohttp.web.Application(loop=loop)
	app.router.add_post('/', view)
	handler = app.make_handler()
	await loop.create_server(handler, '0.0.0.0', 1111)

async def main(loop):
	srv = await server(loop)
	async with aiohttp.post('http://0.0.0.0:1111', data={'val':b'0' * (2 ** 16 - 100)}) as resp:
		print('--->', resp.status)
	print('--')
	async with aiohttp.post('http://0.0.0.0:1111', data={'val':b'0' * (2 ** 16 + 1)}) as resp:
		print('--->', resp.status)

loop = asyncio.get_event_loop()
loop.run_until_complete(main(loop))

<aiohttp.multipart.BodyPartReader object at 0x109c10f28> 65436
---> 200
--
Error handling request
Traceback (most recent call last):
  File "/Users/arthur/Documents/reaaad/lib/aiohttp/aiohttp/web_server.py", line 61, in handle_request
    resp = yield from self._handler(request)
  File "/Users/arthur/Documents/reaaad/lib/aiohttp/aiohttp/web.py", line 268, in _handle
    resp = yield from handler(request)
  File "poc.py", line 9, in view
    v = await part.read(decode=True)
  File "/Users/arthur/Documents/reaaad/lib/aiohttp/aiohttp/multipart.py", line 257, in read
    data.extend((yield from self.readline()))
  File "/Users/arthur/Documents/reaaad/lib/aiohttp/aiohttp/multipart.py", line 351, in readline
    line = yield from self._content.readline()
  File "/Users/arthur/Documents/reaaad/lib/aiohttp/aiohttp/streams.py", line 517, in wrapper
    result = yield from func(self, *args, **kw)
  File "/Users/arthur/Documents/reaaad/lib/aiohttp/aiohttp/streams.py", line 587, in readline
    return (yield from super().readline())
  File "/Users/arthur/Documents/reaaad/lib/aiohttp/aiohttp/streams.py", line 251, in readline
    raise ValueError('Line is too long')
ValueError: Line is too long
---> 500

This is because the StreamReader class only accepts lines up to 64KB.
I've updated the MultipartReader class to avoid readline when it doesn't really make sense, and to use read_chunk instead.

As far as I can see, it seems to work, but that might not be the right approach to solve this at all…

[fafhrd91]

@kxepal
you are very popular :)

[codecov-io]

Current coverage is 98.72% (diff: 100%)

Merging #1535 into master will decrease coverage by 0.20%

@@             master      #1535   diff @@
==========================================
  Files            30         30          
  Lines          6995       6989     -6   
  Methods           0          0          
  Messages          0          0          
  Branches       1169       1165     -4   
==========================================
- Hits           6920       6900    -20   
- Misses           37         53    +16   
+ Partials         38         36     -2   

Powered by Codecov. Last update 1d0750e...03101a0

[kxepal]

@arthurdarcet
Good catch!

@fafhrd91
Ahaha! Indeed. As ours multipart feature (:

[kxepal]

Nice!

[kxepal]

Can this value be picked from aiohttp.streams.DEFAULT_LIMIT to be in sync?

[arthurdarcet]

sure, i've updated the commit

[kxepal]

Well, actually test reads the boundary (: In previous case it just was placed into unread buffer.

[arthurdarcet]

I'm not entirely sure, but i think it reads the boundary then put in back in the stream (with unread_data) ; so it's as if it wasn't read

[fafhrd91]

what is status of this ticket?

[fafhrd91]

@kxepal ?

[kxepal]

@fafhrd91 Sorry, a bit busy times. I have all the tickets you ping me stared in by inbox, will check them in few hours.

[fafhrd91]

thanks!

[kxepal]

+1

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.