Basic Workflow examples requires another service?

[vsoch]

Hi! I'm trying to follow the tutorial - I've installed it, done merlin config (not included in the docs there) and also init'd the workflow. When I run the command as shown here https://merlin.readthedocs.io/en/latest/merlin_workflows.html I get

I think I'm possibly missing something - was I supposed to configure a service? Is there a complete tutorial with step by step instructions I could do somewhere? Thanks!

[lucpeterson]

Have you configured merlin per the instructions in the tutorial? https://merlin.readthedocs.io/en/latest/modules/installation/installation.html#id3
In particular, you should be running the "merlin server" command to get a container running a server

[vsoch]

I didn't know there was a tutorial! 😆 I see redis now. Would it make sense to have a link to that on the workflows page? Like "before you do this, you should have followed the instructions here." I started at the workflows page, then went to the install page to pip install, and didn't realize there was an intermediate step.

[vsoch]

Is it possible to run redis just locally, or via an external service? I'm planning to run this on kubernetes, and the redis service will be another container in a pod, meaning the service will be available (and I don't want merlin to launch singularity or docker or podman). I'm reading here https://merlin.readthedocs.io/en/latest/merlin_server.html.

[lucpeterson]

Yes you can start your own redis server locally without a container or have one running with an external server and configure your configuration file to point to it. This can be complicated, so merlin server aims to make this easier. You can also run with the —local flag to bypass a server and run serially, which is sufficient for the basic example workflows

[lucpeterson]

Here are some instructions if you want to configure your container to run your own server: https://merlin.readthedocs.io/en/latest/modules/installation/installation.html#id5

[lucpeterson]

Advanced configuration docs are here: https://merlin.readthedocs.io/en/latest/merlin_config.html

[lucpeterson]

Running via Kubernetes would be pretty cool. If you figure out the configuration and/or docs, we should add it!

[vsoch]

The docker-compose example should work! For testing I'm just spinning up redis locally, but for the Flux Operator we have separate containers that provide services: https://flux-framework.org/flux-operator/tutorials/services.html. There are two modes (and likely I'll test and think about both). For one - the service is provided as a sidecar container. So an indexed job with N containers running merlin would have one sidecar per container. That wouldn't be ideal if all nodes running merlin (via flux) need access to the same database. The second design (which I haven't tested yet, but technically it's simpler than the above) is to bring up a single (shared) service for all the worker nodes to access. Glancing at these files, the hardest thing (for the Flux Operator) will be to generate the shared config files and volumes in advance. Likely I'll just do this manually for now and make a tutorial, but (if you like this idea generally) this could be an opportunity to make a simple Kubernetes operator just for running these merlin workflows. They are fairly fun to make! I'm also thinking about if it might be possible to have the concept of a Flux Operator plugin - e.g., adding an ability to use "Flux Operator + Merlin" and then having some of the complexity handled by the plugin. I'll do more research/reading about this today - editing audio first but should have some time later.

Happy Saturday!

[vsoch]

Heyo! I have my docker-compose setup and I think I'm fairly close - but a question. Is redis requested to use ssl? I haven't generated certificates for it, and I'm wondering if this is redis or rabbitmq (or possibly I have a configuration error). I also had some trouble matching the environment variables for the various cert files (they generate with very different names) so I might have messed that up. Here is the current debug output:

$ docker exec -it merlin bash
root@eda5eada61b7:/workflow# merlin -lvl DEBUG run feature_demo/feature_demo.yaml
  
                                                
       *      
   *~~~~~                                       
  *~~*~~~*      __  __           _ _       
 /   ~~~~~     |  \/  |         | (_)      
     ~~~~~     | \  / | ___ _ __| |_ _ __  
    ~~~~~*     | |\/| |/ _ \ '__| | | '_ \ 
   *~~~~~~~    | |  | |  __/ |  | | | | | |
  ~~~~~~~~~~   |_|  |_|\___|_|  |_|_|_| |_|
 *~~~~~~~~~~~                                    
   ~~~*~~~*    Machine Learning for HPC Workflows                                 
              


[2023-03-20 22:48:08: INFO] Loading specification from path: /workflow/feature_demo/feature_demo.yaml
[2023-03-20 22:48:08: DEBUG] Creating Merlin spec object...
[2023-03-20 22:48:08: DEBUG] Successfully loaded specification: 
{'name': '$(NAME)', 'description': 'Run 10 hello worlds.'}
[2023-03-20 22:48:08: DEBUG] Merlin spec object created.
[2023-03-20 22:48:08: DEBUG] Verifying Merlin spec...
[2023-03-20 22:48:08: DEBUG] Merlin spec verified.
[2023-03-20 22:48:08: DEBUG] Creating Merlin spec object...
[2023-03-20 22:48:08: DEBUG] Successfully loaded specification: 
{'name': 'feature_demo', 'description': 'Run 10 hello worlds.'}
[2023-03-20 22:48:08: DEBUG] Merlin spec object created.
[2023-03-20 22:48:08: DEBUG] Creating Merlin spec object...
[2023-03-20 22:48:08: DEBUG] Successfully loaded specification: 
{'name': 'feature_demo', 'description': 'Run 10 hello worlds.'}
[2023-03-20 22:48:08: DEBUG] Merlin spec object created.
[2023-03-20 22:48:08: DEBUG] Creating Merlin spec object...
[2023-03-20 22:48:08: DEBUG] Successfully loaded specification: 
{'name': 'feature_demo', 'description': 'Run 10 hello worlds.'}
[2023-03-20 22:48:08: DEBUG] Merlin spec object created.
[2023-03-20 22:48:08: INFO] Study workspace is '/workflow/studies/feature_demo_20230320-224808'.
[2023-03-20 22:48:08: INFO] Reading app config from file /root/.merlin/app.yaml
[2023-03-20 22:48:08: DEBUG] Broker: connection = amqps
[2023-03-20 22:48:08: DEBUG] Broker: vhost = root
[2023-03-20 22:48:08: DEBUG] Broker: username = root
[2023-03-20 22:48:08: DEBUG] Broker: server = rabbitmq
[2023-03-20 22:48:08: DEBUG] Broker: password filepath = /root/.merlin/rabbit.pass
[2023-03-20 22:48:08: DEBUG] Broker: RabbitMQ using default port = 5671
[2023-03-20 22:48:08: DEBUG] Broker: connection = amqps
[2023-03-20 22:48:08: DEBUG] Broker: vhost = root
[2023-03-20 22:48:08: DEBUG] Broker: username = root
[2023-03-20 22:48:08: DEBUG] Broker: server = rabbitmq
[2023-03-20 22:48:08: DEBUG] Broker: password filepath = /root/.merlin/rabbit.pass
[2023-03-20 22:48:08: DEBUG] Broker: RabbitMQ using default port = 5671
[2023-03-20 22:48:08: DEBUG] broker: amqps://root:******@rabbitmq:5671/root
[2023-03-20 22:48:08: DEBUG] Broker: keyfile not present
[2023-03-20 22:48:08: DEBUG] Broker: certfile not present
[2023-03-20 22:48:08: DEBUG] Broker: ca_certs not present
[2023-03-20 22:48:08: DEBUG] Broker: ssl cert_reqs not present
[2023-03-20 22:48:08: DEBUG] Broker: ssl ssl_protocol not present
[2023-03-20 22:48:08: DEBUG] broker_ssl = True
[2023-03-20 22:48:08: DEBUG] Results backend: redis using default password = 
[2023-03-20 22:48:08: DEBUG] Results backend: password_file = 
[2023-03-20 22:48:08: DEBUG] Results backend: server = redis
[2023-03-20 22:48:08: DEBUG] Results backend: certs_path = None
[2023-03-20 22:48:08: DEBUG] Results Backend: keyfile not present
[2023-03-20 22:48:08: DEBUG] Results Backend: certfile not present
[2023-03-20 22:48:08: DEBUG] Results Backend: ca_certs not present
[2023-03-20 22:48:08: DEBUG] Results Backend: ssl cert_reqs not present
[2023-03-20 22:48:08: DEBUG] Results Backend: ssl ssl_protocol not present
[2023-03-20 22:48:08: DEBUG] Results backend: redis using default password = 
[2023-03-20 22:48:08: DEBUG] Results backend: password_file = 
[2023-03-20 22:48:08: DEBUG] Results backend: server = redis
[2023-03-20 22:48:08: DEBUG] Results backend: certs_path = None
[2023-03-20 22:48:08: DEBUG] results: redis://redis:6379/0
[2023-03-20 22:48:08: DEBUG] results: redis_backed_use_ssl = False
[2023-03-20 22:48:08: INFO] Overriding default celery config with 'celery.override' in 'app.yaml':
	visibility_timeout:	86400
[2023-03-20 22:48:08: DEBUG] Adapter config = {'type': 'local', 'dry_run': False, 'shell': '/bin/bash', 'batch_type': 'local'}
[2023-03-20 22:48:10: DEBUG] Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/merlin/main.py", line 898, in main
    args.func(args)
  File "/opt/conda/lib/python3.10/site-packages/merlin/main.py", line 191, in process_run
    router.run_task_server(study, args.run_mode)
  File "/opt/conda/lib/python3.10/site-packages/merlin/router.py", line 72, in run_task_server
    run_celery(study, run_mode)
  File "/opt/conda/lib/python3.10/site-packages/merlin/study/celeryadapter.py", line 64, in run_celery
    app.connection().connect()
  File "/opt/conda/lib/python3.10/site-packages/kombu/connection.py", line 274, in connect
    return self._ensure_connection(
  File "/opt/conda/lib/python3.10/site-packages/kombu/connection.py", line 433, in _ensure_connection
    return retry_over_time(
  File "/opt/conda/lib/python3.10/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
    return fun(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/kombu/connection.py", line 877, in _connection_factory
    self._connection = self._establish_connection()
  File "/opt/conda/lib/python3.10/site-packages/kombu/connection.py", line 812, in _establish_connection
    conn = self.transport.establish_connection()
  File "/opt/conda/lib/python3.10/site-packages/kombu/transport/pyamqp.py", line 201, in establish_connection
    conn.connect()
  File "/opt/conda/lib/python3.10/site-packages/amqp/connection.py", line 323, in connect
    self.transport.connect()
  File "/opt/conda/lib/python3.10/site-packages/amqp/transport.py", line 130, in connect
    self._init_socket(
  File "/opt/conda/lib/python3.10/site-packages/amqp/transport.py", line 209, in _init_socket
    self._setup_transport()
  File "/opt/conda/lib/python3.10/site-packages/amqp/transport.py", line 404, in _setup_transport
    self.sock.do_handshake()
  File "/opt/conda/lib/python3.10/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

[2023-03-20 22:48:10: ERROR] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

Here is my app.yaml

broker:
  name: rabbitmq
  server: rabbitmq
  password: /root/.merlin/rabbit.pass
celery:
  override:
    visibility_timeout: 86400
process:
  kill: kill {pid}
  status: pgrep -P {pid}
results_backend:
  db_num: 0
  name: redis
  port: 6379
  server: redis

and my docker-compose:

version: '3.9'

services:

  # This can also be set up with TLS
  # https://merlin.readthedocs.io/en/latest/modules/installation/installation.html#id7
  # see 2.4.2 "Redis TLS Service"
  redis:
    restart: always
    hostname: redis
    container_name: redis
    image: 'redis:latest'
    ports:
      - "6379:6379"
    networks:
      - rabbitmq

  rabbitmq:
    restart: always

    # The hostname should not be necessary - didn't work either way
    hostname: rabbitmq
    container_name: rabbitmq
    image: rabbitmq:3.8-management
    ports:
      - "15672:15672"
      - "15671:15671"
      - "5672:5672"
      - "5671:5671"
    environment:
      - RABBITMQ_SSL_CACERTFILE=/cert_rabbitmq/client_rabbitmq_certificate.pem
      - RABBITMQ_SSL_KEYFILE=/cert_rabbitmq/server_rabbitmq_key.pem
      - RABBITMQ_SSL_CERTFILE=/cert_rabbitmq/server_rabbitmq_certificate.pem
      - RABBITMQ_SSL_VERIFY=verify_none
      - RABBITMQ_SSL_FAIL_IF_NO_PEER_CERT=false
      - RABBITMQ_DEFAULT_USER=merlinu
      - RABBITMQ_DEFAULT_VHOST=/merlinu
      - RABBITMQ_DEFAULT_PASS=guest
    volumes:
      - ./merlinu/cert_rabbitmq:/cert_rabbitmq
    networks:
      - rabbitmq

  # Yer a weezard Harry!
  merlin:
    build: .
    container_name: merlin
    networks:
      - rabbitmq

    # I only added these because they weren't showing up (didn't change anything)
    # You can try with them removed
    links:
      - rabbitmq 
      - redis

networks:
  rabbitmq:
    driver: bridge

Also note that we have to now pin the rabbitmq management container to that version -if you go higher you'll get an error because they changed it to not accept envars (and only accept a config file).

Thanks for your help - I think I'm close (and am excited to at least try this out in the flux operator, once it's working in compose!)

[lucpeterson]

@koning or @ryannova do you have any ideas? Is this related to the weird conda SSL issue that pops up every once in a while (and I’m not sure what the fix is)?
How is the “merlin server” container configured, compared to this setup?

[vsoch]

@lucpeterson I just tested removing conda/mamba from the container (and installing to system python alongside flux) and I reproduced the same ssl error, if that helps.

[lucpeterson]

I’ve seen a similar error with incompatibilities between versions of celery/kombu and certifi. Trying different versions of certifi could maybe work?

[koning]

The broker lists as amqps so it is expecting a ssl config, you can try setting the broker to amqp or setting cert_reqs: none . The rabbitmq server is also configured with certs so it may need to have ssl configured for the initial handshake. The TSL configuration is definitely the most complicated component of the server system and we have seen issues with specific versions of openssl.

[vsoch]

I haven't added the redis ssl yet - it was presented like it was optional in the docs. I can do that now.