Evaluate and test the SDR deployment document

[llivermore]

Naturalis will set up their own instance of the SDR using current documentation. We will record (and respond) to any questions or feedback here.

[TomDijkema]

Hi Oliver/Laurence,

I am currently trying to run the SDR locally to test the implementation before setting it up on a remote server. This process until now gave me some issues that the documentation did not directly address but I could fix, I can list those if you like. However, I am now getting pretty stuck at the last step: the deployment of the Ansible playbook (I skipped the SSL portion for my local installment).

In an attempt to host the SDR locally I attempted several things but started with the suggested 'hosts' file were I defined the localhost ip as the host:

127.0.0.1 ansible_connection=local

This keeps giving me the following result:

ansible-playbook deploy-galaxy.yml
[DEPRECATION WARNING]: include is deprecated, use include_tasks/import_tasks instead. See 
https://docs.ansible.com/ansible-core/2.14/user_guide/playbooks_reuse_includes.html for details. This feature will be
 removed in version 2.16. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[WARNING]: Could not match supplied host pattern, ignoring: galaxyservers
[WARNING]: Could not match supplied host pattern, ignoring: remoteservers

PLAY [galaxyservers,remoteservers] ***********************************************************************************
skipping: no hosts matched

PLAY RECAP ***********************************************************************************************************

Which I interpret as the host is not valid or I am probably doing something wrong in the way you should deploy this on localhost in comparison to a remote server.
I also tried some other methods like the ones suggested in: https://gist.github.com/alces/caa3e7e5f46f9595f715f0f55eef65c1, and for example tried editing the hosts variable in the deploy-galaxy.yml file to localhost, but with the same result.

Could you give me a hint on what I am doing wrong? Ansible and Galaxy are completely new to me so I am probably overseeing something.

Best regards, Tom

[OliverWoolland]

Hello @TomDijkema! Thanks for trying out the process :) I have a few responses!

1. I wouldn't personally try to use the Ansible scripts to deploy the SDR locally, the playbooks were not written with that in mind and it would be hard to undo the changes if any problems are found. I'd suggest setting up a virtual machine and deploying to that
2. I suspect your specific issue here is missing the [remoteservers] tag in your hosts file? (see here)
3. I have recently updated the deployment documentation in response to some feedback! It might be worth giving it another skim

[TomDijkema]

Hi @OliverWoolland,

Thanks, we have set up a remote server and now this part indeed works as expected in the manual.
I am, however, encountering a new error which seems to inherit from the python code itself when executing the second playbook for the enhanced SDR features (the first playbook has finished successfully after some attempts :) ).
Could you please have a look at the error message for me and let me know if this is something wrong in the code or if I need to change the configuration files?
I'll leave you with the error message and configuration files.

Error:

TASK [Create bootstrap admin] ***************************************************************
fatal: [ip]: FAILED! => changed=true 
  cmd: |-
    python3 '/srv/SDR/deployment/galaxy-scripts/add-bootstrap-admin.py' --master-api-key 'xiqnaejull' --admin-email 'dissco@sdr.com' --admin-user 'ubuntu' --server 'localhost'
  delta: '0:00:00.481564'
  end: '2022-12-09 14:12:34.882872'
  msg: non-zero return code
  rc: 1
  start: '2022-12-09 14:12:34.401308'
  stderr: |-
    Traceback (most recent call last):
      File "/srv/SDR/deployment/galaxy-scripts/add-bootstrap-admin.py", line 20, in <module>
        gi = GalaxyInstance(url=args.server, key=args.master_api_key)
      File "/usr/local/lib/python3.10/dist-packages/bioblend/galaxy/__init__.py", line 83, in __init__
        super().__init__(url, key, email, password, verify=verify)
      File "/usr/local/lib/python3.10/dist-packages/bioblend/galaxyclient.py", line 67, in __init__
        raise ValueError(f"Missing scheme in url {url}")
    ValueError: Missing scheme in url localhost
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>

PLAY RECAP **********************************************************************************
ip                : ok=14   changed=11   unreachable=0    failed=1    skipped=0    rescued=0    ignored=0    

Values in the encrypted sdr-secret.yml file:

api key: xiqnaejull
admin email: dissco@sdr.com
admin user: <user>
teklia:
bardecode:
ansible_ssh_pass: <server user password>
ansible_become_pass: <become pass>
vault_id_secret: <the vault password present in .vault-password.txt>

[OliverWoolland]

Hi @TomDijkema - excellent news that it has gone as planned. Please be a little careful posting IP addresses and passwords here!

Could you please double check you are working with the latest version of this repository? I would have expected this commit to solve your issue.

If you do have the latest version and that problem has persisted I will have to have a look in to it!

[TomDijkema]

Apologies, forgot to mute them, edited previous comment.
I will check the version I pulled.

[TomDijkema]

Ok, I pulled the latest version and retried. It now encounters a 502 Bad Gateway, originating from the domain itself. If you navigate to https://sdr.dissco.tech (our domain for sdr) you can see the error code. This is the nginx result from running the first playbook, maybe something in there went wrong. Checked our server config but ports 80 and 443 should be accessible from anywhere, also included the SSL certificates in the server (called foo with the correct extension) and connection is verified.

I am not sure if this is a problem with our server config or SDR, it looks like ports 80 and 443 are up and running with nginx master just fine. If so I will check this with Sam next week.

Error in playbook:

TASK [Create bootstrap admin] ********************************************************************************************************************************
fatal: [ip]: FAILED! => changed=true 
  cmd: |-
    python3 '/srv/SDR/deployment/galaxy-scripts/add-bootstrap-admin.py' --master-api-key 'xiqnaejull' --admin-email 'dissco@sdr.com' --admin-user '<user>' --server 'http://localhost'
  delta: '<delta>'
  end: '2022-12-09 15:18:47.793975'
  msg: non-zero return code
  rc: 1
  start: '2022-12-09 15:18:47.477817'
  stderr: |-
    Traceback (most recent call last):
      File "/srv/SDR/deployment/galaxy-scripts/add-bootstrap-admin.py", line 27, in <module>
        for existing_user in gi.users.get_users():
      File "/usr/local/lib/python3.10/dist-packages/bioblend/galaxy/users/__init__.py", line 74, in get_users
        return self._get(deleted=deleted, params=params)
      File "/usr/local/lib/python3.10/dist-packages/bioblend/galaxy/client.py", line 134, in _get
        raise ConnectionError(
    bioblend.ConnectionError: GET: error 502: b'<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.18.0 (Ubuntu)</center>\r\n</body>\r\n</html>\r\n', 0 attempts left: <html>
    <head><title>502 Bad Gateway</title></head>
    <body>
    <center><h1>502 Bad Gateway</h1></center>
    <hr><center>nginx/1.18.0 (Ubuntu)</center>
    </body>
    </html>
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>

PLAY RECAP ***************************************************************************************************************************************************
ip                : ok=13   changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0 

[OliverWoolland]

Ok thanks for trying that, did you rerun only deploy-sdr.yml or did you run deploy-galaxy.yml as well?

If only deploy-sdr can I suggest rerunning deploy-galaxy first and seeing if that helps?

[TomDijkema]

I think i did rerun it, but let me try it again

[OliverWoolland]

Sadly the 502 is rather hard to debug. If needed, a procedure I have used is:

• ssh to remote machine
• stop the galaxy service sudo systemctl stop galaxy
• become the galaxy user sudo su galaxy
• move to galaxy server folder cd /srv/galaxy/
• activate the galaxy virtual env source venv/bin/activate
• launch server manually, watch for any error output /srv/galaxy/venv/bin/galaxyctl start --foreground

[TomDijkema]

Ok, when I manually start the Galaxy server as you described it gives the following logs:

supervisord is not running
supervisord is not running
/usr/bin/tail: cannot open '/srv/galaxy/var/gravity/log/gunicorn.log' for reading: No such file or directory
/usr/bin/tail: cannot open '/srv/galaxy/var/gravity/log/celery.log' for reading: No such file or directory
/usr/bin/tail: cannot open '/srv/galaxy/var/gravity/log/celery-beat.log' for reading: No such file or directory
/usr/bin/tail: cannot open '/srv/galaxy/var/gravity/log/handler_0.log' for reading: No such file or directory
/usr/bin/tail: cannot open '/srv/galaxy/var/gravity/log/handler_1.log' for reading: No such file or directory
/usr/bin/tail: cannot open '/srv/galaxy/var/gravity/log/handler_2.log' for reading: No such file or directory
/usr/bin/tail: no files remaining
2022-12-09 16:09:42,001 WARN No file matches via include "/srv/galaxy/var/gravity/supervisor/supervisord.conf.d/*.conf"
2022-12-09 16:09:42,001 INFO Included extra file "/srv/galaxy/var/gravity/supervisor/supervisord.conf.d/_default_.d/galaxy_celery-beat_celery-beat.conf" during parsing
2022-12-09 16:09:42,001 INFO Included extra file "/srv/galaxy/var/gravity/supervisor/supervisord.conf.d/_default_.d/galaxy_celery_celery.conf" during parsing
2022-12-09 16:09:42,001 INFO Included extra file "/srv/galaxy/var/gravity/supervisor/supervisord.conf.d/_default_.d/galaxy_gunicorn_gunicorn.conf" during parsing
2022-12-09 16:09:42,001 INFO Included extra file "/srv/galaxy/var/gravity/supervisor/supervisord.conf.d/_default_.d/galaxy_standalone_handler_0.conf" during parsing
2022-12-09 16:09:42,001 INFO Included extra file "/srv/galaxy/var/gravity/supervisor/supervisord.conf.d/_default_.d/galaxy_standalone_handler_1.conf" during parsing
2022-12-09 16:09:42,001 INFO Included extra file "/srv/galaxy/var/gravity/supervisor/supervisord.conf.d/_default_.d/galaxy_standalone_handler_2.conf" during parsing
Error: Another program is already listening on a port that one of our HTTP servers is configured to use.  Shut this program down first before starting supervisord.
For help, use /srv/galaxy/venv/bin/supervisord -h

I guess some other service is running on the port supervisord wants to run, on which port should it run according to your configuration?

[OliverWoolland]

Thanks for trying that :) I agree that looks like a problem! The set up uses standard ports so 80 or 443 I would expect

[TomDijkema]

Hmm, strange. Server states nginx running on the ports, but that does seem intentional no?

tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      23194/nginx: master 
tcp6       0      0 :::80                   :::*                    LISTEN      23194/nginx: master 
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      23194/nginx: master 

[OliverWoolland]

For that to be a problem seems very strange to me.

Maybe bring nginx down too? sudo systemctl stop nginx then restarting galaxyctl