This is tutorial on how to develop your own X-road services with Python and JSON using X-road REST interface. More basic REST example can be found at https://github.com/Roksnet/Python-json-adapter-hello .
For real life scenario, you need access to your information system’s database. Data handling is based on SQLAlchemy library which can be used with any popular RDBMS, including SQLite, Postgresql, MySQL, Oracle, MS-SQL etc. If you do not have access to the database, you can still follow this tutorial using local SQLite database.
Python 3 is required. As operation system you can use any modern Linux distribution, for example Ubuntu.
For development environment you can also use Microsoft Windows. For installing Python in Windows, we recommend to install Cygwin (http://cygwin.com/setup-x86_64.exe) and select Python 3 among installation options.
If you have Python ready, open terminal window and create sandbox for our project environment:
virtualenv-3 /srv/python-json-demoActivate the sandbox environment for your terminal:
. /srv/python-json-demo/bin/activateMost of files here are intended to develop the server side (to provide X-road services). If your objective is to use X-road services provided by other systems and you need to develop client only, then see client.py.
For providing services you can either clone this example package (and modify it’s contents if you like), or you can create a brand new project with cookiecutter and use this demo package as source for example files.
If you like to create a new project, you need to install cookiecutter:
pip install cookiecutter
Change directory to something that will become a parent of the project directory. Let cookiecutter generate a new Pyramid project:
cookiecutter gh:Pylons/pyramid-cookiecutter-starter
You will be asked for:
- project name and repo name - choose whatever you like
- template language – we prefer mako, but actually it is not important as we are not developing front-end here
- backend – choose sqlalchemy
Change directory into the package directory (json_populationdb).
In config file development.ini setup database connection. Default is local SQLite database:
sqlalchemy.url = sqlite:///%(here)s/json_populationdb.sqlite
See SQLAlchemy documentation for connection string format according to your DBMS. For example, PostgreSQL connection format is
sqlalchemy.url = postgresql://username:password@host/dbname
MySQL connection format is:
sqlalchemy.url = mysql://username:password@host/dbname
For database access we are using SQLAlchemy. SQLAlchemy comes with two options to query and modify data: either object-relational mapper (ORM) or plain SQL.
For complicated business and query logic it is recommended to use ORM. For each database table you have to describe corresponding Python class. ORM classes belong to the models subdirectory. The cookiecutter has generated a demo model in models/mymodel.py. In this demo package you will find data model description in models/populationdb.py. If you change data model then your model classes must be imported in models/__init.py:
from .populationdb import Person, Document
(When project was generated by cookiecutter then remove references to mymodel.)
If you are in hurry then you can omit creating ORM classes and use plain SQL instead. If you are not using an existing database then you must create the database manually. (If using ORM, database can be initialized automatically, generated from ORM classes.)
If you created a new project with cookiecutter and created your own data model then it would be nice if you would exclude original MyModel from files view/default.py and tests.py.
Create code for services your server will provide. This demo app contains example services as views/services/*py. If you have described ORM classes then you can compose queries using SQLAlchemy ORM. If not, you can use SQLAlchemy with plain SQL. There are examples for both options in the persons list service views/services/person.py.
Service functions have decorators which refer to the route name. Route names must be mapped to route URLs in routes.py.
Create service description file (OpenAPI Specification). Example service description file is static/persondata.yaml. The service description file is useful for programmers (and code generators) who will develop clients that are using your services and need to know the structure of service input and output messages. In terms of the security server, each service must have its own description file, but a service may include several endpoints. For each endpoint, description file contains a path description.
Write client code. The demo project contains demo client code in client.py. To develop client of a service of some other service provider, you need to know the input and output parameters of the service. For this purpose you need to investigate service description file or example messages provided by service provider.
Install:
python setup.py develop
In case your are not working with an existing database, the database needs to be initialized. We are using Alembic. Alembic is a tool that automates upgrades of your data model at the database. Omit this step if you are connecting to an existing database or if your data model is managed somewhere else.
To initialize and upgrade the database using Alembic, create first revision of data structure according to your data model (omit this command if you did not modify data model files):
alembic -c development.ini revision --autogenerate -m "init"Initialize the database:
alembic -c development.ini upgrade head
If you do not have an existing database then you need some example data to be added to the database. Initial demo data can be inserted in scripts/initialize_db.py (function setup_models). To load the demo data into the database, run:
initialize_json_populationdb_db development.ini
When you are ready to test the services, run the server locally:
pserve --reload development.ini
Service description file is served at the following URL:
http://SERVER:6543/static/persondata.yaml
For running client, open another terminal window, activate the sandbox environment:
. /srv/python-json-demo/bin/activateRun the client:
python -m json_populationdb.client
If you see that services are working fine, you can continue setting services up in your security server.
Open browser at user interface URL of your security server (port 4000). We assume that you have the information system already certified. Navigate to services page and press ADD REST. Modal window appears, insert following values:
- URL Type: OpenAPI 3 Description
- URL: http://SERVER:6543/static/persondata.yaml
- Service Code: persondata
Save and you see the URL. Click on the service code. You see Service URL field filled with root URI of your server. Append /services to the Service URL (http://SERVER:6543/services). Save.
Open tab ENDPOINTS. You should see here all endpoints you have described in the description file. Add access rights to your service consumer. Access rights may be granted either for service (tab SERVICE PARAMETERS) or individually for each endpoint (tab ENDPOINTS).
Close service form and enable the web service (click on the right of the URL).
To test client against X-road security server, modify client.py to use URL of the security server instead of local URL. You need to modify:
- Your security server’s IP address
- X-road service ID data (according to the service provider in the security server)
- X-road client data (according to the client data in the security server)
- Your user ID (2 letter country code + personal code)
Owner of the client information system is responsible for using correct user ID value according to the real authenticated user.
After those changes you can run client again to check how it works via X-road.
When your services are ready for production, it is recommended to serve them in Linux under Apache web server instead of pserve.
For installing Pyramid app under Apache, you need mod-wsgi for Apache. The following instructions have been tested in Ubuntu Linux. Note that security server sends data with chunked encoding which was not supported by older versions of mod-wsgi. If you have mod-wsgi received from the Ubuntu repository and it happens to be older than 4.7.0, remove it because it does not support chunked encoding:
sudo apt-get remove libapache2-mod-wsgi-py3
To install mod_wsgi:
pip install mod_wsgi mod_wsgi-express module-config > /etc/apache2/mods-available/wsgi.load a2enmod wsgiIn case you do not have the real database and you would like to use the local SQLite database, copy it’s file into the place which will be accessible for the server process:
sudo mkdir /srv/python-json-demo/var sudo cp json_populationdb.sqlite /srv/python-json-demo/var sudo chown -R www-data /srv/python-json-demo/var
Example production configuration file is production.ini. Modify database connect string in the configuration file according to your database. When using local SQLite database, it might be like:
sqlalchemy.url = sqlite:///srv/python-json-demo/json_populationdb.sqlite
Copy configuration file into the sandbox:
sudo mkdir /srv/python-json-demo/etc sudo cp production.ini /srv/python-json-demo/etc/populationdb.ini
Install the package contents into the sandbox:
sudo /srv/python-json-demo/bin/python setup.py install
Copy WSGI app file into sandbox:
sudo mkdir /srv/python-json-demo/apache sudo cp apache/populationdb.wsgi /srv/python-json-demo/apache/populationdb.wsgi
Copy example Apache configuration and enable:
sudo cp apache/adapter.conf /etc/apache2/conf-available sudo a2enconf adapter
File adapter.conf describes that URL /populationdb will be alias for our WSGI app. You may wish to modify configuration file to restrict access to the app etc.
You may need to edit /etc/apache2/envvars to set LANG to UTF-8 character set:
export LANG=en_US.utf8Restart Apache:
sudo service apache2 restart
Now you have a local SOAP server running at: http://SERVER/populationdb/services .
Copy service description file to be served under Apache (assume that web root is /var/www/html):
sudo mkdir /var/www/html/static sudo cp json_populationdb/static/persondata.yaml /var/www/html/static/persondata.yaml
Configure your security server like described before, but with service URL http://SERVER/populationdb/services.
Test services with client.