Skip to content

Update dependencies to latest stable versions #4410

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Oct 30, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,8 @@ Changed

If user wants to retrieve and view all the attributes, they can use ``--attr all`` CLI command
argument (same as before). (improvement) #4396
* Update various internal dependencies to latest stable versions (greenlet, pymongo, pytz,
stevedore, tooz). #4410

Fixed
~~~~~
Expand Down
3 changes: 2 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -383,7 +383,8 @@ requirements: virtualenv .sdist-requirements install-runners
$(VIRTUALENV_DIR)/bin/pip install "six==1.11.0"

# Fix for Travis CI caching issue
$(VIRTUALENV_DIR)/bin/pip uninstall -y "pytz"
$(VIRTUALENV_DIR)/bin/pip uninstall -y "pytz" || echo "not installed"
$(VIRTUALENV_DIR)/bin/pip uninstall -y "python-dateutil" || echo "not installed"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need more workarounds for what looks like Travis caching and pip weirdness.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:(

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW related to that, I got this on local dev env unable to build from sources on a clean Ubuntu Xenial VM:

Generating requirements.txt for contrib/runners/action_chain_runner
===========================================================
Requirements written to: contrib/runners/action_chain_runner/requirements.txt
# Fix for Travis CI race
virtualenv/bin/pip install "six==1.11.0"
Requirement already satisfied: six==1.11.0 in ./virtualenv/lib/python2.7/site-packages
You are using pip version 9.0.3, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
# Fix for Travis CI caching issue
virtualenv/bin/pip uninstall -y "pytz"
Uninstalling pytz-2018.7:
  Successfully uninstalled pytz-2018.7
Traceback (most recent call last):
  File "virtualenv/bin/pip", line 11, in <module>
    sys.exit(main())
  File "/st2/virtualenv/local/lib/python2.7/site-packages/pip/__init__.py", line 248, in main
    return command.main(cmd_args)
  File "/st2/virtualenv/local/lib/python2.7/site-packages/pip/basecommand.py", line 252, in main
    pip_version_check(session)
  File "/st2/virtualenv/local/lib/python2.7/site-packages/pip/utils/outdated.py", line 102, in pip_version_check
    installed_version = get_installed_version("pip")
  File "/st2/virtualenv/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 838, in get_installed_version
    working_set = pkg_resources.WorkingSet()
  File "/st2/virtualenv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 644, in __init__
    self.add_entry(entry)
  File "/st2/virtualenv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 700, in add_entry
    for dist in find_distributions(entry, True):
  File "/st2/virtualenv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1949, in find_eggs_in_zip
    if metadata.has_metadata('PKG-INFO'):
  File "/st2/virtualenv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1463, in has_metadata
    return self.egg_info and self._has(self._fn(self.egg_info, name))
  File "/st2/virtualenv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1823, in _has
    return zip_path in self.zipinfo or zip_path in self._index()
  File "/st2/virtualenv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1703, in zipinfo
    return self._zip_manifests.load(self.loader.archive)
  File "/st2/virtualenv/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1643, in load
    mtime = os.stat(path).st_mtime
OSError: [Errno 2] No such file or directory: '/st2/virtualenv/local/lib/python2.7/site-packages/pytz-2018.7-py2.7.egg'
Makefile:363: recipe for target 'requirements' failed
make: *** [requirements] Error 1

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had similar issues here, even if I used old version of pytz. Looks like some pip related weirdness.

I got it to work by manually uninstalling python-dateutil and pytz once make requirements failed and then ran it again.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just tested it again with a fresh vm and a new virtualenv and it seems to work fine.


# Install requirements
#
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -702,8 +702,8 @@ def test_chain_task_passes_invalid_parameter_type_to_action(self, mock_request):
chain_runner.pre_run()

action_parameters = {}
expected_msg = ('Failed to cast value "stringnotanarray" \(type: str\) for parameter '
'"arrtype" of type "array"')
expected_msg = (r'Failed to cast value "stringnotanarray" \(type: str\) for parameter '
r'"arrtype" of type "array"')
self.assertRaisesRegexp(ValueError, expected_msg, chain_runner.run,
action_parameters=action_parameters)

Expand Down
4 changes: 2 additions & 2 deletions contrib/runners/local_runner/local_runner/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -182,8 +182,8 @@ def _run(self, action):

# Detect if user provided an invalid sudo password or sudo is not configured for that user
if self._sudo_password:
if re.search('sudo: \d+ incorrect password attempts', stderr):
match = re.search('\[sudo\] password for (.+?)\:', stderr)
if re.search(r'sudo: \d+ incorrect password attempts', stderr):
match = re.search(r'\[sudo\] password for (.+?)\:', stderr)

if match:
username = match.groups()[0]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -111,15 +111,15 @@ def test_fail_inspection(self):
{
'type': 'content',
'message': 'The action "std.noop" is not registered in the database.',
'schema_path': 'properties.tasks.patternProperties.^\w+$.properties.action',
'schema_path': r'properties.tasks.patternProperties.^\w+$.properties.action',
'spec_path': 'tasks.task3.action'
},
{
'type': 'context',
'language': 'yaql',
'expression': '<% ctx().foobar %>',
'message': 'Variable "foobar" is referenced before assignment.',
'schema_path': 'properties.tasks.patternProperties.^\w+$.properties.input',
'schema_path': r'properties.tasks.patternProperties.^\w+$.properties.input',
'spec_path': 'tasks.task1.input',
},
{
Expand All @@ -131,7 +131,7 @@ def test_fail_inspection(self):
'position 0 of expression \'<% succeeded()\''
),
'schema_path': (
'properties.tasks.patternProperties.^\w+$.'
r'properties.tasks.patternProperties.^\w+$.'
'properties.next.items.properties.when'
),
'spec_path': 'tasks.task2.next[0].when'
Expand All @@ -142,7 +142,7 @@ def test_fail_inspection(self):
'[{\'cmd\': \'echo <% ctx().macro %>\'}] is '
'not valid under any of the given schemas'
),
'schema_path': 'properties.tasks.patternProperties.^\w+$.properties.input.oneOf',
'schema_path': r'properties.tasks.patternProperties.^\w+$.properties.input.oneOf',
'spec_path': 'tasks.task2.input'
}
]
Expand Down
16 changes: 8 additions & 8 deletions contrib/runners/python_runner/tests/unit/test_pythonrunner.py
Original file line number Diff line number Diff line change
Expand Up @@ -104,10 +104,10 @@ def test_action_returns_non_serializable_result(self):

if six.PY2:
expected_result_re = (r"\[{'a': '1'}, {'h': 3, 'c': 2}, {'e': "
"<non_simple_type.Test object at .*?>}\]")
r"<non_simple_type.Test object at .*?>}\]")
else:
expected_result_re = (r"\[{'a': '1'}, {'c': 2, 'h': 3}, {'e': "
"<non_simple_type.Test object at .*?>}\]")
r"<non_simple_type.Test object at .*?>}\]")

match = re.match(expected_result_re, output['result'])
self.assertTrue(match)
Expand Down Expand Up @@ -624,17 +624,17 @@ def test_python_action_wrapper_action_script_file_doesnt_exist_friendly_error(se
wrapper = PythonActionWrapper(pack='dummy_pack_5', file_path=ACTION_1_PATH,
user='joe')

expected_msg = ('Failed to load action class from file ".*?list_repos_doesnt_exist.py" '
'\(action file most likely doesn\'t exist or contains invalid syntax\): '
'\[Errno 2\] No such file or directory')
expected_msg = (r'Failed to load action class from file ".*?list_repos_doesnt_exist.py" '
r'\(action file most likely doesn\'t exist or contains invalid syntax\): '
r'\[Errno 2\] No such file or directory')
self.assertRaisesRegexp(Exception, expected_msg, wrapper._get_action_instance)

def test_python_action_wrapper_action_script_file_contains_invalid_syntax_friendly_error(self):
wrapper = PythonActionWrapper(pack='dummy_pack_5', file_path=ACTION_2_PATH,
user='joe')
expected_msg = ('Failed to load action class from file ".*?invalid_syntax.py" '
'\(action file most likely doesn\'t exist or contains invalid syntax\): '
'No module named \'?invalid\'?')
expected_msg = (r'Failed to load action class from file ".*?invalid_syntax.py" '
r'\(action file most likely doesn\'t exist or contains invalid syntax\): '
r'No module named \'?invalid\'?')
self.assertRaisesRegexp(Exception, expected_msg, wrapper._get_action_instance)

def test_simple_action_log_messages_and_log_level_runner_param(self):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ def test_get_winexe_command_args(self):
[
'winexe',
'--interactive', '0',
'-U', 'MyDomain\Administrator3%bar3',
'-U', r'MyDomain\Administrator3%bar3',
'//localhost',
'dir'
]
Expand Down Expand Up @@ -123,7 +123,7 @@ def test_get_smbclient_command_args(self):
],
[
'smbclient',
'-U', 'MyDomain\Administrator3%bar3',
'-U', r'MyDomain\Administrator3%bar3',
'//localhost/E$',
'-c', 'dir'
],
Expand Down
4 changes: 2 additions & 2 deletions contrib/runners/windows_runner/windows_runner/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ def _get_winexe_command_args(self, host, username, password, command, domain=Non
args += ['--interactive', '0']

if domain:
args += ['-U', '%s\%s%%%s' % (domain, username, password)]
args += ['-U', r'%s\%s%%%s' % (domain, username, password)]
else:
args += ['-U', '%s%%%s' % (username, password)]

Expand All @@ -84,7 +84,7 @@ def _get_smbclient_command_args(self, host, username, password, command, share='

values = {'domain': domain, 'username': username, 'password': password}
if domain:
auth_string = '%(domain)s\%(username)s%%%(password)s' % values
auth_string = r'%(domain)s\%(username)s%%%(password)s' % values
else:
auth_string = '%(username)s%%%(password)s' % values

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -314,7 +314,7 @@ def _parse_share_information(self, stdout):

for line in lines:
line = line.strip()
split = re.split('\s{3,}', line)
split = re.split(r'\s{3,}', line)

if len(split) not in [1, 2]:
# Invalid line, skip it
Expand Down
11 changes: 6 additions & 5 deletions fixed-requirements.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Packages versions fixed for the whole st2 stack
# Note: greenlet is used by eventlet
greenlet==0.4.14
greenlet==0.4.15
# Note: 0.20.0 removed select.poll() on which some of our code and libraries we
# depend on rely
eventlet==0.24.1
Expand All @@ -19,15 +19,15 @@ requests[security]<2.15,>=2.14.1
apscheduler==3.5.3
gitpython==2.1.11
jsonschema==2.6.0
pymongo==3.7.1
pymongo==3.7.2
passlib==1.7.1
lockfile==0.12.2
python-gnupg==0.4.3
jsonpath-rw==1.4.0
pyinotify==0.9.6
semver==2.8.1
pytz==2018.5
stevedore==1.29.0
pytz==2018.6
stevedore==1.30.0
paramiko==2.4.2
networkx==1.11
python-keyczar==0.716
Expand All @@ -38,7 +38,7 @@ virtualenv==15.1.0
sseclient==0.0.19
python-editor==1.0.3
prompt-toolkit==1.0.15
tooz==1.62.0
tooz==1.63.1
zake==0.2.2
routes==2.4.1
flex==6.13.2
Expand All @@ -53,3 +53,4 @@ python-statsd==2.1.0
prometheus_client==0.1.1
mock==2.0.0
ujson==1.35
python-dateutil==2.7.4
2 changes: 1 addition & 1 deletion lint-configs/python/.flake8
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[flake8]
max-line-length = 100
ignore = E128,E402,E722
ignore = E128,E402,E722,W504
exclude=*.egg/*
12 changes: 6 additions & 6 deletions requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ git+https://github.com/StackStorm/orquesta.git@142a19ddabbeea6a8d9d10341ba705851
git+https://github.com/StackStorm/python-mistralclient.git#egg=python-mistralclient
git+https://github.com/StackStorm/st2-auth-backend-flat-file.git@master#egg=st2-auth-backend-flat-file
gitpython==2.1.11
greenlet==0.4.14
greenlet==0.4.15
gunicorn==19.9.0
ipaddr
jinja2
Expand All @@ -34,14 +34,14 @@ prometheus_client==0.1.1
prompt-toolkit==1.0.15
psutil==5.4.7
pyinotify==0.9.6
pymongo==3.7.1
pymongo==3.7.2
pyrabbit
python-dateutil
python-dateutil==2.7.4
python-editor==1.0.3
python-gnupg==0.4.3
python-json-logger
python-statsd==2.1.0
pytz==2018.5
pytz==2018.6
pywinrm==0.3.0
pyyaml==3.13
rednose
Expand All @@ -51,8 +51,8 @@ routes==2.4.1
semver==2.8.1
six==1.11.0
sseclient==0.0.19
stevedore==1.29.0
tooz==1.62.0
stevedore==1.30.0
tooz==1.63.1
ujson==1.35
unittest2
webob==1.8.2
Expand Down
2 changes: 1 addition & 1 deletion scripts/travis/install-requirements.sh
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/usr/bin/env bash

if [ "${TASK}" = 'compilepy3 ci-py3-unit' ] || [ "${TASK}" = 'ci-py3-integration' ]; then
pip install "tox==3.1.3"
pip install "tox==3.5.2"

# Install runners
. virtualenv/bin/activate
Expand Down
2 changes: 2 additions & 0 deletions st2actions/tests/unit/policies/test_concurrency.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,8 @@ def setUp(self):
MockLiveActionPublisherNonBlocking.wait_all()

def tearDown(self):
MockLiveActionPublisherNonBlocking.wait_all()

for liveaction in LiveAction.get_all():
action_service.update_status(
liveaction, action_constants.LIVEACTION_STATUS_CANCELED)
Expand Down
2 changes: 2 additions & 0 deletions st2actions/tests/unit/policies/test_concurrency_by_attr.py
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,8 @@ def setUp(self):
MockLiveActionPublisherNonBlocking.wait_all()

def tearDown(self):
MockLiveActionPublisherNonBlocking.wait_all()

for liveaction in LiveAction.get_all():
action_service.update_status(
liveaction, action_constants.LIVEACTION_STATUS_CANCELED)
Expand Down
8 changes: 4 additions & 4 deletions st2actions/tests/unit/test_actions_registrar.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,8 @@ def test_pack_name_missing(self):
content = yaml.safe_load(fd)
action_name = str(content['name'])
action_db = Action.get_by_name(action_name)
self.assertEqual(action_db.pack, 'dummy', 'Content pack must be ' +
'set to dummy')
expected_msg = 'Content pack must be set to dummy'
self.assertEqual(action_db.pack, 'dummy', expected_msg)
Action.delete(action_db)

@mock.patch.object(action_validator, '_is_valid_pack', mock.MagicMock(return_value=True))
Expand Down Expand Up @@ -138,6 +138,6 @@ def test_action_update(self):
content = yaml.safe_load(fd)
action_name = str(content['name'])
action_db = Action.get_by_name(action_name)
self.assertEqual(action_db.pack, 'wolfpack', 'Content pack must be ' +
'set to wolfpack')
expected_msg = 'Content pack must be set to wolfpack'
self.assertEqual(action_db.pack, 'wolfpack', expected_msg)
Action.delete(action_db)
8 changes: 4 additions & 4 deletions st2actions/tests/unit/test_notifier.py
Original file line number Diff line number Diff line change
Expand Up @@ -110,8 +110,8 @@ def test_notify_triggers(self):
liveaction_db.notify = NotificationSchema(on_success=on_success,
on_failure=on_failure)
liveaction_db.start_timestamp = date_utils.get_datetime_utc_now()
liveaction_db.end_timestamp = (liveaction_db.start_timestamp +
datetime.timedelta(seconds=50))
liveaction_db.end_timestamp = \
(liveaction_db.start_timestamp + datetime.timedelta(seconds=50))
LiveAction.add_or_update(liveaction_db)

execution = MOCK_EXECUTION
Expand Down Expand Up @@ -181,8 +181,8 @@ def test_notify_triggers_jinja_patterns(self, dispatch):
data={'stdout': '{{action_results.stdout}}'})
liveaction_db.notify = NotificationSchema(on_success=on_success)
liveaction_db.start_timestamp = date_utils.get_datetime_utc_now()
liveaction_db.end_timestamp = (liveaction_db.start_timestamp +
datetime.timedelta(seconds=50))
liveaction_db.end_timestamp = \
(liveaction_db.start_timestamp + datetime.timedelta(seconds=50))

LiveAction.add_or_update(liveaction_db)

Expand Down
2 changes: 1 addition & 1 deletion st2api/st2api/controllers/resource.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ def parameter_validation(validator, properties, instance, schema):
"description": "Input parameters for the action.",
"type": "object",
"patternProperties": {
"^\w+$": util_schema.get_action_parameters_schema()
r"^\w+$": util_schema.get_action_parameters_schema()
},
'additionalProperties': False,
"default": {}
Expand Down
8 changes: 4 additions & 4 deletions st2api/tests/unit/controllers/v1/test_workflow_inspection.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,15 +73,15 @@ def test_inspection_return_errors(self):
{
'type': 'content',
'message': 'The action "std.noop" is not registered in the database.',
'schema_path': 'properties.tasks.patternProperties.^\w+$.properties.action',
'schema_path': r'properties.tasks.patternProperties.^\w+$.properties.action',
'spec_path': 'tasks.task3.action'
},
{
'type': 'context',
'language': 'yaql',
'expression': '<% ctx().foobar %>',
'message': 'Variable "foobar" is referenced before assignment.',
'schema_path': 'properties.tasks.patternProperties.^\w+$.properties.input',
'schema_path': r'properties.tasks.patternProperties.^\w+$.properties.input',
'spec_path': 'tasks.task1.input',
},
{
Expand All @@ -93,7 +93,7 @@ def test_inspection_return_errors(self):
'position 0 of expression \'<% succeeded()\''
),
'schema_path': (
'properties.tasks.patternProperties.^\w+$.'
r'properties.tasks.patternProperties.^\w+$.'
'properties.next.items.properties.when'
),
'spec_path': 'tasks.task2.next[0].when'
Expand All @@ -104,7 +104,7 @@ def test_inspection_return_errors(self):
'[{\'cmd\': \'echo <% ctx().macro %>\'}] is '
'not valid under any of the given schemas'
),
'schema_path': 'properties.tasks.patternProperties.^\w+$.properties.input.oneOf',
'schema_path': r'properties.tasks.patternProperties.^\w+$.properties.input.oneOf',
'spec_path': 'tasks.task2.input'
}
]
Expand Down
4 changes: 2 additions & 2 deletions st2client/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ jsonpath-rw==1.4.0
jsonschema==2.6.0
prettytable
prompt-toolkit==1.0.15
python-dateutil
python-dateutil==2.7.4
python-editor==1.0.3
pytz==2018.5
pytz==2018.6
pyyaml==3.13
requests[security]<2.15,>=2.14.1
six==1.11.0
Expand Down
2 changes: 1 addition & 1 deletion st2client/st2client/utils/jsutil.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
# a..b
# abc.
# a(*
SIMPLE_EXPRESSION_REGEX = "^([a-zA-Z0-9\-_]+\.)*([a-zA-Z0-9\-_]+)$"
SIMPLE_EXPRESSION_REGEX = r"^([a-zA-Z0-9\-_]+\.)*([a-zA-Z0-9\-_]+)$"
SIMPLE_EXPRESSION_REGEX_CMPL = re.compile(SIMPLE_EXPRESSION_REGEX)


Expand Down
Loading