Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[jsinterp] Actual JS interpreter #11272

Closed
wants to merge 127 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
127 commits
Select commit Hold shift + click to select a range
d328b8c
[jsinterp] Actual parsing
sulyi Nov 23, 2016
2c85715
[jsinterp] Handling comments
sulyi Nov 23, 2016
cc895cd
[jsinterp] Parsing expr (cleanup needed)
sulyi Nov 24, 2016
8c87a18
[jsinterp] Calling field and test
sulyi Nov 24, 2016
2076b0b
[jsinterp] Clean up
sulyi Nov 25, 2016
da73cd9
[jsinterp] Quick regex fixes (thx to yan12125)
sulyi Nov 25, 2016
71a485f
[jsinterp] Complex call test (thx to yan12125)
sulyi Nov 25, 2016
8842f08
[jsinterp] String literal regex change
sulyi Nov 26, 2016
c485fe7
[jsinterp] Reject method call when name is empty (+reminder TOTOs)
sulyi Nov 26, 2016
ba5a400
[jsinterp] Simpler regex regex (+more TOTO)
sulyi Nov 26, 2016
b089388
[jsinterp] Lexer overhaul
sulyi Nov 28, 2016
9bd5dee
[jsinterp] Value parsing
sulyi Nov 28, 2016
aa7eb3d
[jsinterp] No OrderedDict
sulyi Nov 30, 2016
a0fa6bf
[jsinterp] Parser mock up
sulyi Nov 30, 2016
67d5653
[jsinterp] Minor quick fixes
sulyi Nov 30, 2016
a89d490
[jsinterp] TokenStream, expression mock up
sulyi Dec 3, 2016
f6005dc
[jsinterp] Adding _operator_expression using reversed polish notation
sulyi Dec 3, 2016
f605783
[jsinterp] Parser - take one (untested)
sulyi Dec 4, 2016
f6ad8db
[jsinterp] Refactoring and minor fixes
sulyi Dec 4, 2016
7864078
[jsinterp] Preliminary fixes after some testing of ast
sulyi Dec 5, 2016
d422aef
[jsinterp] Very basic interpreter
sulyi Dec 6, 2016
ce4a616
[jsinterp] Token class for tokens
sulyi Dec 7, 2016
c426efd
[jsinterp] More tokens
sulyi Dec 7, 2016
c2f280d
[jsinterp] Compatibility fix
sulyi Dec 7, 2016
8ff8a70
[jsinterp] Str tokens are easier to deal with
sulyi Dec 7, 2016
599b9db
[jsinterp] First parser tests
sulyi Dec 8, 2016
70a5e31
[jsinterp] Parentheses fix (test and parser)
sulyi Dec 8, 2016
4999fcc
[jsinterp] More test and str fix
sulyi Dec 8, 2016
651a1e7
[jsinterp] Coding convention fixes
sulyi Dec 8, 2016
dd6a2b5
[jsinterp] Clean up
sulyi Dec 9, 2016
c5c1273
Merge branch 'master' into jsinterp
sulyi Dec 9, 2016
6fa4eb6
[jsinterp] Fixing compatibility
sulyi Dec 9, 2016
a9c7310
[jsinterp] Adding context handling
sulyi Dec 10, 2016
e392f78
[jsinterp] Formatting code
sulyi Dec 10, 2016
88d2a4e
[jsinterp] Unittest2 in reqs
sulyi Dec 10, 2016
200903c
[jsinterp] Fixing py3 zip generator issues in parser tests
sulyi Dec 10, 2016
9d1f756
[jsinterp] Fixing deep copy zip in test_jsinterp_parse
sulyi Dec 10, 2016
f942bb3
[jsinterp] Refactoring getvalue and putvalue
sulyi Dec 10, 2016
9b5e55a
[jsinterp] Mozilla-central test first try
sulyi Dec 10, 2016
aa6e752
[jsinterp] Fixing Reference repr
sulyi Dec 10, 2016
86de1e8
[jsinterp] Adding function declaration and fixing block statement parser
sulyi Dec 10, 2016
4f55fe7
[jsinterp] Adding if parser (test needed)
sulyi Dec 11, 2016
57c8ccb
[jsinterp] Re-prioritising TODOs
sulyi Dec 11, 2016
ad49621
[jsinterp] Adding with and switch parser and fixes (tests needed)
sulyi Dec 11, 2016
c2e6ca5
[jsinterp] Adding code to if and switch test
sulyi Dec 11, 2016
ad288aa
[jsinterp] Parser test code fixes
sulyi Dec 11, 2016
48aaa41
[jsinterp] Finished parser if test
sulyi Dec 11, 2016
dedb6ee
[jsinterp] Added try parser (test needed)
sulyi Dec 11, 2016
bae3166
[jsinterp] Added debugger and throw parser (test needed)
sulyi Dec 11, 2016
96e5068
[jsinterp] Adding parser for label statement and function expression
sulyi Dec 11, 2016
f24cafe
[jsinterp] Adding parser object literal
sulyi Dec 11, 2016
a8a445f
[jsinterp] Fixing TokenStrem pop, label statement, function body
sulyi Dec 11, 2016
253e326
[jsinterp] Adding do and while parser
sulyi Dec 12, 2016
3ba28c6
[jsinterp] Adding for parser
sulyi Dec 12, 2016
cc9cb30
[jsinterp] Reprioritizing TODOs in test_jsinterp_parser.py
sulyi Dec 12, 2016
007f19e
[jsinterp] Adding code to parser tests
sulyi Dec 12, 2016
cf4c9c3
[jsinterp] Adding switch ast to parser test
sulyi Dec 12, 2016
558290d
[jsinterp] Adding object ast to parser test
sulyi Dec 12, 2016
f7993a1
[jsinterp] Refactor
sulyi Dec 12, 2016
2533dc4
[jsinterp] Adding ast to test_function_expression
sulyi Dec 12, 2016
fe141c4
[jsinterp] Refactor _object_literal
sulyi Dec 12, 2016
a2e42ed
[jsinterp] Adding ast to do parser test
sulyi Dec 12, 2016
4b8754c
[jsinterp] Adding ast to while parser test
sulyi Dec 12, 2016
b397ea2
[jsinterp] Adding ast to for parser test
sulyi Dec 12, 2016
cd0bb42
[jsinterp] Adding ast to for empty and for in parser test
sulyi Dec 12, 2016
ab37e2b
[test] Adding jstests test suite
sulyi Dec 14, 2016
c4c2aa2
[test] Adding support for signed values (hopefully)
sulyi Dec 15, 2016
e1444dd
[test] Adding support for signed values
sulyi Dec 15, 2016
0e4dd1a
[test, jsinterp] Adding sign test and refactor and fixing interpretation
sulyi Dec 15, 2016
d7443e1
[jsinterp] Adding interpreter support for pre- and postfix expressions
sulyi Dec 15, 2016
cd2bf30
[test] Adding logging to TestJSInterpreterParse
sulyi Dec 15, 2016
5238ed1
[test] Adding logging to TestJSInterpreter
sulyi Dec 15, 2016
1716801
[jsinterp] Adding interpreter support to get field
sulyi Dec 15, 2016
fce5722
[jsinterp] Adding error handling to global variable init
sulyi Dec 15, 2016
ee3dc29
[jsinterp] Adding interpreter support for set field
sulyi Dec 16, 2016
4e6f689
[jsinterp] Fixing set field
sulyi Dec 16, 2016
dca2e9e
[jsinterp] Fixing compat import
sulyi Dec 16, 2016
3b53669
[jsinterp] Adding function declaration and call
sulyi Dec 17, 2016
3f075d8
[test] jstest fixes
sulyi Dec 27, 2016
3d0252a
[jsinterp] Refactoring jsparser
sulyi Dec 28, 2016
a5e7022
[jstests] Ordering imports in __init__
sulyi Dec 29, 2016
bddf482
[jstests] Doc, dynamic import
sulyi Dec 29, 2016
41596ff
[jsbuilt-ins] jsbuilt_ins mock up
sulyi Dec 28, 2016
6f2ac27
[jsbuilt-ins] Table of content of the book of black magic
sulyi Jan 21, 2017
1725514
[jsinterp] super object in subclasses __init__
sulyi Jan 22, 2017
0eef083
[jsbuilt-ins] a riddle wrapped in mystery inside an enigma
sulyi Jan 23, 2017
484a7d2
[jsbuilt-ins] adding _type and JSObject constructor
sulyi Jan 23, 2017
65e9b0b
[jsbuilt-ins] adding Function and Array constructors
sulyi Jan 24, 2017
2dd9864
[jsbuilt-ins] minor props fix
sulyi Jan 27, 2017
a500c34
[jsbuilt-ins] major props fix
sulyi Jan 27, 2017
598f5f2
[jsbuilt-ins] String mock up Function constructor fix, to_string plac…
sulyi Jan 28, 2017
56cecdd
[jsbuilt-ins] fixing to_string
sulyi Jan 29, 2017
9ead39c
[jsbuilt-ins] fixing numerical stability of to_string
sulyi Jan 30, 2017
8733120
[jsbuilt-ins] implementing Boolean object
sulyi Feb 1, 2017
8729fe6
[jsbuilt-ins] adding type conversions (to number )
sulyi Feb 18, 2017
ec79b14
[jsbuilt-ins] adding Number class and prototype
sulyi Feb 20, 2017
dbedff2
[jsbuilt-ins] global object properties mock up
sulyi Feb 20, 2017
4d386f0
[jsbuilt-ins] major refactor
sulyi Feb 21, 2017
0136be4
[jsbuilt-ins] fixing constructors
sulyi Mar 2, 2017
49dba39
Merge branch 'master' into jsinterp
sulyi May 30, 2018
1126698
[jsinterp] Renaming `jsinterp` to jsinterp2
sulyi May 30, 2018
e44a252
[jsinterp] Using unicode literals
sulyi May 31, 2018
53f8eff
[jsbuilt_ins] Fixing circular imports
sulyi May 31, 2018
61fe8d2
[jsbuilt-ins] premerge
sulyi Jan 22, 2017
b856d55
Merge branch 'jsbuilt-ins' into jsinterp
sulyi Jun 1, 2018
70ac98a
[jsinterp] Fixing missed unicode support (yet again)
sulyi Jun 1, 2018
1f40e3e
[jsinterp] Test suit update
sulyi Jun 2, 2018
db0dc7b
[jsinterp] Fixing typos and code style
sulyi Jun 2, 2018
d977e93
[jsinterp] Fixing test skip messages
sulyi Jun 3, 2018
38b2602
[jsinterp] Complying with PEP 479
sulyi Jun 3, 2018
b9061d6
[jsinterp] Fixing TODOs and comments
sulyi Jun 4, 2018
2ce996c
[jsinterp] Unicode docstring hack
sulyi Jun 4, 2018
70d9194
[jsinterp] Multi level logging in tests
sulyi Jun 4, 2018
1b9d883
[jsinterp] Faking `Logger.getChild` for py2.6
sulyi Jun 4, 2018
327bb2d
[jsinterp] Fixing code style
sulyi Jun 4, 2018
f9f030a
[jsinterp] Implementing String split
sulyi Jun 9, 2018
db44dee
[jsinterp] Renaming tests
sulyi Jun 9, 2018
105faaf
[jsinterp] Revert `youtube-dl/youtube_dl/extractor/youtube.py`
sulyi Jun 9, 2018
b8a1742
[jsinterp] Rename `js2test` to `jstests`
sulyi Jun 10, 2018
848aa79
[jsinterp] Fixing incomplete refactor
sulyi Jun 10, 2018
bbea188
[jsinterp] revert `youtube_dl/extractor/youtube.py` (yet again)
sulyi Jun 10, 2018
37d6306
[jsinterp] Adding `JSArrayPrototype#_slice`
sulyi Jun 10, 2018
8060889
[jsinterp] TODOs in `JSStringPrototype#_split`
sulyi Jun 10, 2018
a8c640e
[jsinterp] Fixing broken Assignment Expression
sulyi Jun 10, 2018
a33b47e
[jsinterp] Adding handling lineterminator
sulyi Jun 10, 2018
93c0bb5
[jsinterp] Fixing types and operators
sulyi Jun 11, 2018
c0ef911
[jsinterp] Adding delete and void operators
sulyi Jun 11, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
File renamed without changes.
File renamed without changes.
105 changes: 105 additions & 0 deletions test/test_js2test_legacy.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
#!/usr/bin/env python

from __future__ import unicode_literals

# Allow direct execution
import os
import sys
import logging

if sys.version_info < (2, 7):
import unittest2 as unittest
else:
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

from youtube_dl.jsinterp import JSInterpreter
from .js2tests import gettestcases

__doc__ = """see: `js2tests`"""


defs = gettestcases()
# set level to logging.DEBUG to see messages about missing assertions
# set level to logging.DEBUG to see messages about code tests are running
logging.basicConfig(stream=sys.stderr, level=logging.WARNING)
log = logging.getLogger('TestJSInterpreter')


class TestJSInterpreter(unittest.TestCase):
def setUp(self):
self.defs = defs


def generator(test_case, my_log):
def test_template(self):
my_log.debug('Started...')
for test in test_case['subtests']:
excluded = test.get('exclude')
if excluded is not None and 'jsinterp' in excluded:
log_reason = 'jsinterp does not support this subtest:\n%s' % test['code']
elif 'code' not in test:
log_reason = 'No code in subtest, skipping'
elif 'asserts' not in test:
log_reason = 'No assertion in subtest, skipping'
else:
log_reason = None

if log_reason is None:
variables = test.get('globals')
code = test['code']
call = None

if variables is not None:
code = 'function f(%s){%s}' % ((''.join(variables.keys())), code)
call = ('f',) + tuple(v for v in variables.values())
my_log.debug('globals: %s' % variables)
my_log.debug(code)

jsi = JSInterpreter(code, objects=variables)
for assertion in test['asserts']:
if 'value' in assertion:
if call is None:
call = assertion['call']

if call is not None:
my_log.debug('call: %s(%s)' % (call[0], ', '.join(str(arg) for arg in call[1:])))

self.assertEqual(jsi.call_function(*call), assertion['value'])
else:
my_log.info('No value in assertion, skipping')
else:
my_log.info(log_reason)

return test_template


# And add them to TestJSInterpreter
for testcase in defs:
reason = testcase['skip'].get('jsinterp', False)
tname = 'test_' + str(testcase['name'])
i = 1
while hasattr(TestJSInterpreter, tname):
tname = 'test_%s_%d' % (testcase['name'], i)
i += 1

if reason is True:
log_reason = 'Entirely'
elif not any('asserts' in test for test in testcase['subtests']):
log_reason = '''There isn't any assertion'''
else:
log_reason = None

if log_reason is None:
test_method = generator(testcase, logging.getLogger('.'.join((log.name, tname))))
test_method.__name__ = str(tname)
if reason is not False:
test_method.__unittest_skip__ = True
test_method.__unittest_skip_why__ = reason
setattr(TestJSInterpreter, test_method.__name__, test_method)
del test_method
else:
log.info('Skipping %s:%s' % (tname, log_reason))

if __name__ == '__main__':
unittest.main()
196 changes: 104 additions & 92 deletions test/test_jsinterp.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,101 +5,113 @@
# Allow direct execution
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Direct execution is still possible. Accidental removal?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, fixed.

import os
import sys
import logging

if sys.version_info < (2, 7):
import unittest2 as unittest
else:
import unittest
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

from youtube_dl.jsinterp import JSInterpreter
from .js2tests import gettestcases

__doc__ = """see: `js2tests`"""


defs = gettestcases()
# set level to logging.DEBUG to see messages about missing assertions
# set level to logging.DEBUG to see messages about code tests are running
logging.basicConfig(stream=sys.stderr, level=logging.WARNING)
log = logging.getLogger('TestJSInterpreter')


class TestJSInterpreter(unittest.TestCase):
def setUp(self):
self.defs = defs


def generator(test_case, my_log):
def test_template(self):
my_log.debug('Started...')
for test in test_case['subtests']:
excluded = test.get('exclude')
if excluded is not None and 'jsinterp' in excluded:
log_reason = 'jsinterp does not support this subtest:\n%s' % test['code']
elif 'code' not in test:
log_reason = 'No code in subtest, skipping'
elif 'asserts' not in test:
log_reason = 'No assertion in subtest, skipping'
else:
log_reason = None

if log_reason is None:
variables = test.get('globals')
code = test['code']
call = None

if variables is not None:
code = 'function f(%s){%s}' % ((''.join(variables.keys())), code)
call = ('f',) + tuple(v for v in variables.values())
my_log.debug('globals: %s' % variables)
my_log.debug(code)

jsi = JSInterpreter(code, objects=variables)
for assertion in test['asserts']:
if 'value' in assertion:
if call is None:
call = assertion['call']

if call is not None:
my_log.debug('call: %s(%s)' % (call[0], ', '.join(str(arg) for arg in call[1:])))

self.assertEqual(jsi.call_function(*call), assertion['value'])
else:
my_log.info('No value in assertion, skipping')
else:
my_log.info(log_reason)

return test_template


# And add them to TestJSInterpreter
for testcase in defs:
reason = testcase['skip'].get('jsinterp', False)
tname = 'test_' + str(testcase['name'])
i = 1
while hasattr(TestJSInterpreter, tname):
tname = 'test_%s_%d' % (testcase['name'], i)
i += 1

if reason is True:
log_reason = 'Entirely'
elif not any('asserts' in test for test in testcase['subtests']):
log_reason = '''There isn't any assertion'''
else:
log_reason = None

if log_reason is None:
test_method = generator(testcase, logging.getLogger('.'.join((log.name, tname))))
test_method.__name__ = str(tname)
if reason is not False:
test_method.__unittest_skip__ = True
test_method.__unittest_skip_why__ = reason
setattr(TestJSInterpreter, test_method.__name__, test_method)
del test_method
else:
log.info('Skipping %s:%s' % (tname, log_reason))


class TestJSInterpreterOrig(unittest.TestCase):
def test_basic(self):
jsi = JSInterpreter('function x(){;}')
self.assertEqual(jsi.call_function('x'), None)

jsi = JSInterpreter('function x3(){return 42;}')
self.assertEqual(jsi.call_function('x3'), 42)

jsi = JSInterpreter('var x5 = function(){return 42;}')
self.assertEqual(jsi.call_function('x5'), 42)

def test_calc(self):
jsi = JSInterpreter('function x4(a){return 2*a+1;}')
self.assertEqual(jsi.call_function('x4', 3), 7)

def test_empty_return(self):
jsi = JSInterpreter('function f(){return; y()}')
self.assertEqual(jsi.call_function('f'), None)

def test_morespace(self):
jsi = JSInterpreter('function x (a) { return 2 * a + 1 ; }')
self.assertEqual(jsi.call_function('x', 3), 7)

jsi = JSInterpreter('function f () { x = 2 ; return x; }')
self.assertEqual(jsi.call_function('f'), 2)

def test_strange_chars(self):
jsi = JSInterpreter('function $_xY1 ($_axY1) { var $_axY2 = $_axY1 + 1; return $_axY2; }')
self.assertEqual(jsi.call_function('$_xY1', 20), 21)

def test_operators(self):
jsi = JSInterpreter('function f(){return 1 << 5;}')
self.assertEqual(jsi.call_function('f'), 32)

jsi = JSInterpreter('function f(){return 19 & 21;}')
self.assertEqual(jsi.call_function('f'), 17)

jsi = JSInterpreter('function f(){return 11 >> 2;}')
self.assertEqual(jsi.call_function('f'), 2)

def test_array_access(self):
jsi = JSInterpreter('function f(){var x = [1,2,3]; x[0] = 4; x[0] = 5; x[2] = 7; return x;}')
self.assertEqual(jsi.call_function('f'), [5, 2, 7])

def test_parens(self):
jsi = JSInterpreter('function f(){return (1) + (2) * ((( (( (((((3)))))) )) ));}')
self.assertEqual(jsi.call_function('f'), 7)

jsi = JSInterpreter('function f(){return (1 + 2) * 3;}')
self.assertEqual(jsi.call_function('f'), 9)

def test_assignments(self):
jsi = JSInterpreter('function f(){var x = 20; x = 30 + 1; return x;}')
self.assertEqual(jsi.call_function('f'), 31)

jsi = JSInterpreter('function f(){var x = 20; x += 30 + 1; return x;}')
self.assertEqual(jsi.call_function('f'), 51)

jsi = JSInterpreter('function f(){var x = 20; x -= 30 + 1; return x;}')
self.assertEqual(jsi.call_function('f'), -11)

def test_comments(self):
'Skipping: Not yet fully implemented'
return
jsi = JSInterpreter('''
function x() {
var x = /* 1 + */ 2;
var y = /* 30
* 40 */ 50;
return x + y;
}
''')
self.assertEqual(jsi.call_function('x'), 52)

jsi = JSInterpreter('''
function f() {
var x = "/*";
var y = 1 /* comment */ + 2;
return y;
}
''')
self.assertEqual(jsi.call_function('f'), 3)

def test_precedence(self):
jsi = JSInterpreter('''
function x() {
var a = [10, 20, 30, 40, 50];
var b = 6;
a[0]=a[b%a.length];
return a;
}''')
self.assertEqual(jsi.call_function('x'), [20, 20, 30, 40, 50])

def test_call(self):
jsi = JSInterpreter('''
function x() { return 2; }
function y(a) { return x() + a; }
function z() { return y(3); }
''')
self.assertEqual(jsi.call_function('z'), 5)


if __name__ == '__main__':
unittest.main()
Loading