Skip to content

[AIRFLOW-116] Add a version view to display airflow version info #1504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
asfgit merged 1 commit into from
May 19, 2016
Merged

[AIRFLOW-116] Add a version view to display airflow version info #1504

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
.DS_Store
.ipynb*
.coverage
airflow/git_version
airflow/www/static/coverage/
airflow.db
airflow.cfg
Expand Down
2 changes: 2 additions & 0 deletions airflow/www/app.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,8 @@ def create_app(config=None):
base.MenuLink(category='Docs',
name='Github',url='https://github.com/airbnb/airflow'))

av(vs.VersionView(name='Version', category="About"))

av(vs.DagRunModelView(
models.DagRun, Session, name="DAG Runs", category="Browse"))
av(vs.DagModelView(models.DagModel, Session, name=None))
Expand Down
30 changes: 30 additions & 0 deletions airflow/www/templates/airflow/version.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
{#
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
#}
{% extends "airflow/master.html" %}

{% block body %}
{{ super() }}
<h2>{{ title }}</h2>
{% set version_label = 'Version' %}
{% if airflow_version %}
<h4>{{ version_label }} : <a href="https://pypi.python.org/pypi/airflow/{{ airflow_version }}">{{ airflow_version }}</a></h4>
{% else %}
<h4>{{ version_label }} : Not Available</h4>
{% endif %}
<h4>Git Version :{% if git_version %} {{ git_version }} {% else %} Not Available {% endif %}</h4>
<hr>

{% endblock %}
29 changes: 29 additions & 0 deletions airflow/www/views.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
import sys

import os
import pkg_resources
import socket
import importlib

Expand All @@ -28,6 +29,7 @@
from past.builtins import basestring

import inspect
import subprocess
import traceback

import sqlalchemy as sqla
Expand Down Expand Up @@ -61,6 +63,7 @@

from airflow.operators import BaseOperator, SubDagOperator

from airflow.utils.logging import LoggingMixin
from airflow.utils.json import json_ser
from airflow.utils.state import State
from airflow.utils.db import provide_session
Expand Down Expand Up @@ -2291,6 +2294,32 @@ class UserModelView(wwwutils.SuperUserMixin, AirflowModelView):
column_default_sort = 'username'


class VersionView(wwwutils.SuperUserMixin, LoggingMixin, BaseView):
@expose('/')
def version(self):
# Look at the version from setup.py
try:
airflow_version = pkg_resources.require("airflow")[0].version
except Exception as e:
airflow_version = None
self.logger.error(e)

# Get the Git repo and git hash
git_version = None
try:
with open("airflow/git_version") as f:
Copy link
Member

@mistercrunch mistercrunch May 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the current working directory may not be the one you think it is. I'd use something like os.path.join(settings.AIRFLOW_HOME, 'git_version')

Copy link
Contributor Author

@r39132 r39132 May 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, though I am using os.path.join(*[settings.AIRFLOW_HOME, 'airflow', 'git_version']) here and in setup.py

git_version = f.readline()
except Exception as e:
self.logger.error(e)

# Render information
title = "Version Info"
return self.render('airflow/version.html',
title=title,
airflow_version=airflow_version,
git_version=git_version)


class ConfigurationView(wwwutils.SuperUserMixin, BaseView):
@expose('/')
def conf(self):
Expand Down
224 changes: 136 additions & 88 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
from setuptools import setup, find_packages, Command
from setuptools.command.test import test as TestCommand

import logging
import os
import sys

logger = logging.getLogger(__name__)

# Kept manually in sync with airflow.__version__
version = '1.7.0'

Expand Down Expand Up @@ -35,6 +38,44 @@ def run(self):
os.system('rm -vrf ./build ./dist ./*.pyc ./*.tgz ./*.egg-info')


def git_version(version):
Copy link
Member

@mistercrunch mistercrunch May 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+doc string

Copy link
Contributor Author

@r39132 r39132 May 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

try:
import git
except ImportError:
logger.warn('gitpython not found: Cannot compute the git version.')
return ''
try:
repo = git.Repo('.git')
except ImportError:
Copy link
Member

@mistercrunch mistercrunch May 19, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't be an ImportError in that case, put more lines in the try, and catch many exceptions in that same try block with many except sections

Copy link
Contributor Author

@r39132 r39132 May 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. using Exception.

logger.warn('Git repo not found: Cannot compute the git version.')
return ''
sha = repo.head.commit.hexsha
Copy link
Member

@mistercrunch mistercrunch May 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

references to repo are potentially out of scope here. It should be in the same try block or within an else for that try block.

Copy link
Contributor Author

@r39132 r39132 May 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't else make sense if there is a reasonable default as mentioned on http://stackoverflow.com/a/14590478/1110993

Copy link
Contributor Author

@r39132 r39132 May 19, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

def git_version(version):
    """
    Return a version to identify the state of the underlying git repo. The version will
    indicate whether the head of the current git-backed working directory is tied to a
    release tag or not : it will indicate the former with a 'release:{version}' prefix
    and the latter with a 'dev0' prefix. Following the prefix will be a sha of the current
    branch head. Finally, a "dirty" suffix is appended to indicate that uncommitted changes
    are present.
    """
    repo = None
    try:
        import git
        repo = git.Repo('.git')
    except ImportError:
        logger.warn('gitpython not found: Cannot compute the git version.')
        return ''
    except Exception as e:
        logger.warn('Git repo not found: Cannot compute the git version.')
        return ''
    if repo:
        sha = repo.head.commit.hexsha
        if repo.is_dirty():
            return '.dev0+{sha}.dirty'.format(sha=sha)
        # commit is clean
        # is it release of `version` ?
        try:
            tag = repo.git.describe(
                match='[0-9]*', exact_match=True,
                tags=True, dirty=True)
            assert tag == version, (tag, version)
            return '.release:{version}+{sha}'.format(version=version,
                                                     sha=sha)
        except git.GitCommandError:
            return '.dev0+{sha}'.format(sha=sha)
    else:
        return 'no_git_version'

if repo.is_dirty():
return '.dev0+{sha}.dirty'.format(sha=sha)
# commit is clean
# is it release of `version` ?
try:
tag = repo.git.describe(
match='[0-9]*', exact_match=True,
tags=True, dirty=True)
assert tag == version, (tag, version)
return '.release:{version}+{sha}'.format(version=version,
sha=sha)
except git.GitCommandError:
return '.dev0+{sha}'.format(sha=sha)


def write_version(filename=os.path.join('airflow', 'git_version')):
cnt = """%(git_revision)s"""
text = cnt % {'git_revision':
Copy link
Member

@mistercrunch mistercrunch May 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the python community is moving away from the modulo format operator in favor of "{}".format() function.

Copy link
Contributor Author

@r39132 r39132 May 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done... replaced with
text = "{}".format(git_version(version))

git_version(version)}
try:
with open(filename, 'w') as a:
Copy link
Member

@mistercrunch mistercrunch May 19, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't silence this error unless we're expecting the write to fail, let it raise!

Copy link
Contributor Author

@r39132 r39132 May 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.. I've removed the try-except..

def write_version(filename=os.path.join(*[settings.AIRFLOW_HOME,
                                          'airflow',
                                          'git_version'])):
    text = "{}".format(git_version(version))
    with open(filename, 'w') as a:
        a.write(text)

a.write(text)
except Exception as e:
logger.error(e)
Copy link
Member

@mistercrunch mistercrunch May 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logger.exception(e) is usually better as it provides the stack trace.



async = [
'greenlet>=0.4.9',
'eventlet>= 0.9.7',
Expand Down Expand Up @@ -100,91 +141,98 @@ def run(self):
devel_hadoop = devel_minreq + hive + hdfs + webhdfs + kerberos
devel_all = devel + all_dbs + doc + samba + s3 + slack + crypto + oracle + docker

setup(
name='airflow',
description='Programmatically author, schedule and monitor data pipelines',
license='Apache License 2.0',
version=version,
packages=find_packages(),
package_data={'': ['airflow/alembic.ini']},
include_package_data=True,
zip_safe=False,
scripts=['airflow/bin/airflow'],
install_requires=[
'alembic>=0.8.3, <0.9',
'babel>=1.3, <2.0',
'chartkick>=0.4.2, < 0.5',
'croniter>=0.3.8, <0.4',
'dill>=0.2.2, <0.3',
'python-daemon>=2.1.1, <2.2',
'flask>=0.10.1, <0.11',
'flask-admin>=1.4.0, <2.0.0',
'flask-cache>=0.13.1, <0.14',
'flask-login==0.2.11',
'future>=0.15.0, <0.16',
'funcsigs>=0.4, <1',
'gunicorn>=19.3.0, <19.4.0', # 19.4.? seemed to have issues
'jinja2>=2.7.3, <3.0',
'markdown>=2.5.2, <3.0',
'pandas>=0.15.2, <1.0.0',
'pygments>=2.0.1, <3.0',
'python-dateutil>=2.3, <3',
'requests>=2.5.1, <3',
'setproctitle>=1.1.8, <2',
'sqlalchemy>=0.9.8',
'thrift>=0.9.2, <0.10',
'Flask-WTF==0.12'
],
extras_require={
'all': devel_all,
'all_dbs': all_dbs,
'async': async,
'celery': celery,
'crypto': crypto,
'devel': devel_minreq,
'devel_hadoop': devel_hadoop,
'doc': doc,
'docker': docker,
'druid': druid,
'gcp_api': gcp_api,
'hdfs': hdfs,
'hive': hive,
'jdbc': jdbc,
'mssql': mssql,
'mysql': mysql,
'oracle': oracle,
'postgres': postgres,
'rabbitmq': rabbitmq,
's3': s3,
'samba': samba,
'slack': slack,
'statsd': statsd,
'vertica': vertica,
'ldap': ldap,
'webhdfs': webhdfs,
'kerberos': kerberos,
'password': password,
'github_enterprise': github_enterprise,
'qds': qds,
'cloudant': cloudant
},
classifiers={
'Development Status :: 5 - Production/Stable',
'Environment :: Console',
'Environment :: Web Environment',
'Intended Audience :: Developers',
'Intended Audience :: System Administrators',
'License :: OSI Approved :: Apache Software License',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3.4',
'Topic :: System :: Monitoring',
},
author='Maxime Beauchemin',
author_email='maximebeauchemin@gmail.com',
url='https://github.com/airbnb/airflow',
download_url=(
'https://github.com/airbnb/airflow/tarball/' + version),
cmdclass={'test': Tox,
'extra_clean': CleanCommand,
},
)
def do_setup():
write_version()
setup(
name='airflow',
description='Programmatically author, schedule and monitor data pipelines',
license='Apache License 2.0',
version=version,
packages=find_packages(),
package_data={'': ['airflow/alembic.ini', "airflow/git_version"]},
include_package_data=True,
zip_safe=False,
scripts=['airflow/bin/airflow'],
install_requires=[
'alembic>=0.8.3, <0.9',
'babel>=1.3, <2.0',
'chartkick>=0.4.2, < 0.5',
'croniter>=0.3.8, <0.4',
'dill>=0.2.2, <0.3',
'python-daemon>=2.1.1, <2.2',
'flask>=0.10.1, <0.11',
'flask-admin>=1.4.0, <2.0.0',
'flask-cache>=0.13.1, <0.14',
'flask-login==0.2.11',
'future>=0.15.0, <0.16',
'funcsigs>=0.4, <1',
'gitpython>=2.0.2',
'gunicorn>=19.3.0, <19.4.0', # 19.4.? seemed to have issues
'jinja2>=2.7.3, <3.0',
'markdown>=2.5.2, <3.0',
'pandas>=0.15.2, <1.0.0',
'pygments>=2.0.1, <3.0',
'python-dateutil>=2.3, <3',
'requests>=2.5.1, <3',
'setproctitle>=1.1.8, <2',
'sqlalchemy>=0.9.8',
'thrift>=0.9.2, <0.10',
'Flask-WTF==0.12'
],
extras_require={
'all': devel_all,
'all_dbs': all_dbs,
'async': async,
'celery': celery,
'crypto': crypto,
'devel': devel_minreq,
'devel_hadoop': devel_hadoop,
'doc': doc,
'docker': docker,
'druid': druid,
'gcp_api': gcp_api,
'hdfs': hdfs,
'hive': hive,
'jdbc': jdbc,
'mssql': mssql,
'mysql': mysql,
'oracle': oracle,
'postgres': postgres,
'rabbitmq': rabbitmq,
's3': s3,
'samba': samba,
'slack': slack,
'statsd': statsd,
'vertica': vertica,
'ldap': ldap,
'webhdfs': webhdfs,
'kerberos': kerberos,
'password': password,
'github_enterprise': github_enterprise,
'qds': qds,
'cloudant': cloudant
},
classifiers={
'Development Status :: 5 - Production/Stable',
'Environment :: Console',
'Environment :: Web Environment',
'Intended Audience :: Developers',
'Intended Audience :: System Administrators',
'License :: OSI Approved :: Apache Software License',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3.4',
'Topic :: System :: Monitoring',
},
author='Maxime Beauchemin',
author_email='maximebeauchemin@gmail.com',
url='https://github.com/airbnb/airflow',
download_url=(
'https://github.com/airbnb/airflow/tarball/' + version),
cmdclass={'test': Tox,
'extra_clean': CleanCommand,
},
)


Copy link
Contributor Author

@r39132 r39132 May 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 seems standard. I need to install flake8 on my box now that we don't have landscape.io running. But, I'm pretty sure 2 is adequate. Your earlier comment was when there were 3.

Please remove comment.

Copy link
Member

@mistercrunch mistercrunch May 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PEP8 says 2 lines before classes and function definition but not before an if. It's really not shocking though when landscape.io starts policing again it will note this as a "code smell". You can get flake8 to install commit hooks so that you see those on commit. flake8 --install-hook

Copy link
Contributor Author

@r39132 r39132 May 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Contributor Author

@r39132 r39132 May 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are all addressed in a new PR : https://github.com/apache/incubator-airflow/pull/1523/files

if __name__ == "__main__":
do_setup()