Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CAMP fails on the CityGo case-study #81

Closed
fchauvel opened this issue Oct 2, 2019 · 22 comments
Closed

CAMP fails on the CityGo case-study #81

fchauvel opened this issue Oct 2, 2019 · 22 comments

Comments

@fchauvel
Copy link
Member

fchauvel commented Oct 2, 2019

Characteristics

  • Issue Type: Case-study
  • Reproducibility: always
  • Severity: block
  • Tool/Service/Component: CAMP v0.6.3
  • Execution Environment:
  • Reporter: Franck Chauvel on behalf of AtoS

Description

We open here a new issue to group and summarize Issue #24, #68, #77 and #80, which we shall close accordingly.

The template directory, the camp.yml, as well as the docker-compose file must be reworked so that one can run all the way.

$ camp generate -d . --all
$ camp realize -d .
$ camp execute -d .

Alternatively, we must also rework the case-study description and documentation available at: https://stamp-project.github.io/camp/pages/citygo.html

Steps to reproduce

Run CAMP on the CityGo case-study. See samples/stamp/atos/. There are two separate scenarios, namely performance and functional.

Other files and URLs

Relationships

Help on issue template

Preview to follow the link or open file .github/ISSUE_DOC.md

@fermenreq
Copy link

fermenreq commented Oct 2, 2019

Hi @fchauvel,

According this issue:

Please, consider details above

Project folder structure

root@osboxes:/home/osboxes/Desktop/STAMP/stamp_docker_citygoApp# tree -L 2

├── apache
│   ├── app
│   ├── demo_site_1.conf
│   ├── demo_site.conf
│   ├── django.conf
│   ├── **docker-compose-apache-as-proxy.yml**
│   ├── docker-compose-apache.yml
│   ├── docker-compose-mapped.yml
│   ├── docker-compose.yml
│   ├── Dockerfile
│   ├── httpd.conf
│   ├── mpm_event.conf
│   └── requirements.txt
├── dashboard
│   ├── access.log
│   ├── backend
│   ├── citygo_settings
│   ├── dashboard
│   ├── DEBUG.log
│   ├── Dockerfile
│   ├── entrypoint.sh
│   ├── init-db.sql
│   ├── manage.py
│   ├── requirements.txt
│   └── static
├── diagram.png
├── LICENSE
├── nginx
│   ├── app
│   ├── DEBUG.log
│   ├── docker-compose-server.yml
│   ├── docker-compose.yml
│   ├── nginx
│   ├── **nginx_docker-compose.yml**
│   ├── README.md
│   ├── restart.sh
│   ├── stop.sh
│   ├── test.sh
│   └── up.sh
├── nohup.out
├── Postgres
│   ├── Dockerfile
│   ├── init-db.sql
│   └── postgresql.conf
├── README.md
└── Tests
       ├── citygo_tests
       ├── executeCamp.sh
       ├── reportTestResults.sh
       ├── Result.jtl
       ├── Test.jmx
       └── version2

18 directories, 49 files

CAMP evaluation over ATOS use case (CityGo)

  • Scenario 1: To deploy CityGo and performance stress test - only in Apache configuration with different performance parameters

  • The docker compose descriptors used to deploy CityGo over apache service is docker-compose-apache-as-proxy.yml

  • Scenario 2: To deploy CityGo and run functional test before different functional functions of CityGo: nginx, python variants, django, etc.

  • The docker compose descriptor used to deploy CityGo over Nginx service is nginx_docker-compose.yml

Objective: CAMP will generate X configurations as a docker-compose-configuration_X.yml. So this docker-compose-configuration_X.yml have to deploy same services as docker-compose.yml ( base yml file after CAMP execution)

@fchauvel
Copy link
Member Author

fchauvel commented Oct 2, 2019

Hi there,

I updated the performance scenario as we discussed. I also updated the related documentation. Please take a look and let me know if there are still things that look mysterious and that you do not know how to do.

I will continue tomorrow with the functional scenario.

Hope this helps.

@fermenreq
Copy link

fermenreq commented Oct 2, 2019

Hi @fchauvel ,

Thanks for your updated.

I ran my jenkins job to test it and I get the following output.

[Pipeline] { (CAMP generate)
[Pipeline] sh
docker exec -d camp sh -c cp -R /stamp_docker_citygoApp/apache/app/* ./samples/stamp/atos/performance/template/citygo/

docker exec -i camp sh -c cd samples/stamp/atos/performance/ && camp generate -d . --all

/usr/local/lib/python3.5/dist-packages/camp/generate.py:216: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  metamodel = load_yaml(data)
CAMP v0.6.3-git.2460e36 (MIT)
Copyright (C) 2017 -- 2019 SINTEF Digital

Loaded './camp.yml'.

 - Config. 1 in './out/config_1/configuration.yml'.
   Includes citygo, hub, postgres, apache (104, 128, 1664), browser, ubuntu...

 - Config. 2 in './out/config_2/configuration.yml'.
   Includes citygo, apache (24, 64, 384), hub, postgres, browser, ubuntu, p...

 - Config. 3 in './out/config_3/configuration.yml'.
   Includes citygo, hub, postgres, browser, apache (56, 64, 896), ubuntu, p...

 - Config. 4 in './out/config_4/configuration.yml'.
   Includes citygo, hub, postgres, apache (120, 128, 1920), browser, ubuntu...

 - Config. 5 in './out/config_5/configuration.yml'.
   Includes citygo, hub, postgres, apache (112, 128, 1792), browser, ubuntu...

 - Config. 6 in './out/config_6/configuration.yml'.
   Includes citygo, hub, postgres, browser, apache (8, 64, 128), ubuntu, py...

 - Config. 7 in './out/config_7/configuration.yml'.
   Includes citygo, hub, postgres, apache (32, 64, 512), browser, ubuntu, p...

 - Config. 8 in './out/config_8/configuration.yml'.
   Includes citygo, hub, postgres, apache (40, 64, 640), browser, ubuntu, p...

 - Config. 9 in './out/config_9/configuration.yml'.
   Includes citygo, hub, apache (48, 64, 768), postgres, browser, ubuntu, p...

 - Config. 10 in './out/config_10/configuration.yml'.
   Includes citygo, hub, postgres, apache (16, 64, 256), browser, ubuntu, p...

That's all folks!
exit
[Pipeline] }
[Pipeline] // stage

[Pipeline] stage
[Pipeline] { (CAMP realize)
[Pipeline] sh
+ docker exec -i camp sh -c cd samples/stamp/atos/performance/ && camp realize -d . && cp -R out/ /stamp_docker_citygoApp/

CAMP v0.6.3-git.2460e36 (MIT)
Copyright (C) 2017 -- 2019 SINTEF Digital

Loaded './camp.yml'.
Loading configurations from './out' ...

Error:
Vain substitution: No match found!'
  - pattern: 'FROM\s+camp/[a-zA-Z0-9\._-]+(?:\:[a-zA-Z0-9\._-]+)?'
  - target file: './out/config_1/images/citygo_0/Dockerfile'
Is the pattern appropriate? What about the file content?

That's all folks!

exit

@fchauvel
Copy link
Member Author

fchauvel commented Oct 2, 2019

This problem occurs when CAMP tries to replace the pattern camp/runtime in the Dockerfile of components that are involved in a stack. In the Citygo case, those are the 'python' and the 'citygo' components. Both Dockerfiles should have a FROM statement like FROM camp/runtime, which CAMPwill try to replace so that components are stacked properly.

From the error, it is the citygo component, but check as well the python component. Should one of these dockerfile be a multistage build, only replace the final FROM statement by camp/runtime.

@fermenreq
Copy link

fermenreq commented Oct 2, 2019

Hi,

This is what CAMP generated (python and citygo) components:

root@supersede:/var/jenkins_home/workspace/citygo_camp/out/config_1/images# cat python_0/Dockerfile

   #Installation of Python 2.7
   #Replaced by CAMP during realization
   FROM ubuntu:latest
   RUN apt-get install -y --no-install-recommends python2.7 \
    && rm -rf  /usr/bin/python \
    && ln -s /usr/bin/python2.7 /usr/bin/python

Seems it a wrong subsitution for the FROM:ubuntu:latest?

root@supersede:/var/jenkins_home/workspace/citygo_camp/out/config_1/images# cat citygo_0/Dockerfile

   FROM python:2.7
   ENV PYTHONUNBUFFERED 1
   ENV C_FORCE_ROOT true
   Set work directory
   WORKDIR /usr/src/app
   RUN chmod -R +x /usr/src/app
   COPY ./requirements.txt /usr/src/app/requirements.txt
   RUN pip install -r requirements.txt
   #copy entrypoint.sh
   COPY ./entrypoint.sh /usr/src/app/entrypoint.sh
   #copy project
   COPY . /usr/src/app/
   EXPOSE 8000
   #run entrypoint.sh
   ENTRYPOINT ["/usr/src/app/entrypoint.sh"]

Thanks,
Fernando Mendez - fernando.mendez@atos.net

@fchauvel
Copy link
Member Author

fchauvel commented Oct 2, 2019

Those two Dockerfiles are directly generated from the ones in the template folders, respectively from template\citygo and template\python. So it is actually these two Dockerfiles (the ones in the template directory) that must contain the marker camp/runtime. CAMP will actually copy these files from the template to the generated configurations, and will "search and replace" the fake image tag camp/rutime by the correct image tag.

This looks complicated but it is this very indirection on image tag, that lets CAMP assemble components in various ways, although in the citygo case there is only one possibility.

@fermenreq
Copy link

fermenreq commented Oct 2, 2019

Hi @fchauvel , thanks for your quick reply.

As my understanding we have to replace from both Dockerfiles FROM ubuntu:latest and FROM python:2.7 statement to FROM camp/runtime ?

Currently this subsitution exists for Apache Dockerfile and for Python Dockerfile

Thanks,
Fernando

@fermenreq
Copy link

Hi @fchauvel ,

I pushed new changes PR 79ae93a

@fermenreq
Copy link

fermenreq commented Oct 3, 2019

Hi @fchauvel ,

Succesfully CAMP configurations generated but errors on executions. See above

Test execution aborted!
Error: A shell command failed (code: 100)
$ bash build_images.sh
Check out logs in 'camp_execute.log'.
Sending build context to Docker daemon 2.048kB

 Step 1/2 : FROM ubuntu:latest
 ---> ea4c82dcd15a
 Step 2/2 : RUN apt-get install -y --no-install-recommends python2.7     && rm -rf  /usr/bin/python        && ln -s /usr/bin/python2.7 /usr/bin/python
 ---> Running in 6fbae0463c27
 Reading package lists...
 Building dependency tree...
 Reading state information...
 [91mE: Unable to locate package python2.7
 E: Couldn't find any package by glob 'python2.7'
 E: Couldn't find any package by regex 'python2.7'
 [0m The command '/bin/sh -c apt-get install -y --no-install-recommends python2.7     && rm -rf   /usr/bin/python     && ln -s /usr/bin/python2.7 /usr/bin/python' returned a non-zero code: 100

Also if I entered into camp container and build the image I will get same error before comment

root@c5f5c1fd5360:/camp/samples/stamp/atos/performance/out/config_1/images/python_0# docker build .

Step 1/2 : FROM ubuntu:latest
 ---> ea4c82dcd15a
Step 2/2 : RUN apt-get install -y --no-install-recommends python2.7     && rm -rf  /usr/bin/python     && ln -s /usr/bin/python2.7 /usr/bin/python
 ---> Running in d4fd070ab011
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package python2.7
E: Couldn't find any package by glob 'python2.7'
E: Couldn't find any package by regex 'python2.7'
The command '/bin/sh -c apt-get install -y --no-install-recommends python2.7     && rm -rf  /usr/bin/python     && ln -s /usr/bin/python2.7 /usr/bin/python' returned a non-zero code: 100

I guess after install python packages, you need to update package lists from Ubuntu repository:

Step 2/2 : RUN apt-get update && apt-get install -y --no-install-recommends python2.7 && rm -rf /usr/bin/python && ln -s /usr/bin/python2.7 /usr/bin/python

@fchauvel
Copy link
Member Author

fchauvel commented Oct 3, 2019

Hi there,

Looking at the error message, I understand that the command: RUN apt-get install -y --no-install-recommends python2.7 fails because APT cannot find a package named python2.7. From the top of my head, the python2.7 package will be installed on top of the ubuntu component, which is based on the Docker ubuntu:latest (according to the camp.yml). I am not sure whether Python2.7 is still supported in the lastest Ubuntu version, so I suggest to replace this by ubuntu:xenial for instance and to modify the performance/camp.yml, at the line 89, accordingly.

Besides, in the file template/python/Dockerfile, I would call apt-get update before to install python to refresh the local repositories, as follows:

FROM camp/runtime

RUN apt-get update \ 
    && apt-get install -y --no-install-recommends python2.7 \
    && rm -rf  /usr/bin/python \
    && ln -s /usr/bin/python2.7 /usr/bin/python

Finally, I saw that the file performance/template/apache/Dockerfile contains the marker `camp/runtime', while it must not. This Dockerfile will not be assembled with others because Apache runs alone in its own container. As it is now, it will lead—I guess—to a failure during execution, because Docker will not find how to build the image 'camp/runtime'.

@fermenreq
Copy link

fermenreq commented Oct 4, 2019

Hi @fchauvel,

Found an error and solved it by this commit 3b89744. Let's go to test again

Successfully built 059a4e3353f6
Successfully tagged camp-python_0:latest
Sending build context to Docker daemon  57.62MB

Step 1/11 : FROM camp-python_0
 ---> 059a4e3353f6
Step 2/11 : ENV PYTHONUNBUFFERED 1
 ---> Running in ab729e3a68a8
Removing intermediate container ab729e3a68a8
 ---> a011813adbb6
Step 3/11 : ENV C_FORCE_ROOT true
 ---> Running in 5bd170a2f243
Removing intermediate container 5bd170a2f243
 ---> 87167b82cca8
Step 4/11 : WORKDIR /usr/src/app
 ---> Running in d60661a9a94f
Removing intermediate container d60661a9a94f
 ---> 6f5a097c7434
Step 5/11 : RUN chmod -R +x /usr/src/app
 ---> Running in 664088bc6448
Removing intermediate container 664088bc6448
 ---> ab9ab9e05123
Step 6/11 : COPY ./requirements.txt /usr/src/app/requirements.txt
 ---> e0f89bafe3d9
Step 7/11 : RUN pip install -r requirements.txt
 ---> Running in d674b7c06ef2
[91m/bin/sh: 1: pip: not found
[0m The command '/bin/sh -c pip install -r requirements.txt' returned a non-zero code: 127

Thanks,
Fernando

@fermenreq
Copy link

Included a fix: 0c6fc82

@fermenreq
Copy link

fermenreq commented Oct 4, 2019

Included a fix that does NOT deploy Citygo using CAMP cause some errors with dependencies within dockerfiles images: 5c97f34

@fermenreq
Copy link

Hi,

Camp configuration has been generated but only the first one has been deployed with problems on deployment due to ports and address are already in use by other services.

Successfully built 9355eaab9b50
Successfully tagged config_1_web:latest

Creating network "config_1_my-app-bridge" with driver "bridge"
Creating volume "config_1_postgres_data" with default driver
Creating volume "config_1_static_volume" with default driver
Building db
Image for service db was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Pulling hub (selenium/hub:)...
Pulling selenium_chrome (selenium/node-chrome-debug:)...
Pulling mongo (mongo:4.0.6-xenial)...
Building apache
Image for service apache was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Building web
Image for service web was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating selenium_hub ... 
Creating my_mongo     ... 
Creating my_postgresql ... error

Creating my_mongo      ... error
rting userland proxy: listen tcp 0.0.0.0:5432: bind: address already in use'
Creating selenium_hub  ... done
ERROR: for my_mongo  Cannot start service mongo: b'driver failed programming external connectivity on endpoint my_mongo (78f1054c5c746222f5881dc7e9d40f39b38d2f36ec29b40b79dba3ee59f6d104): Error starting userland proxy: listen tcp 0.0.0.0:27017: bind: address already in use'
Creating selenium_chrome ... done

ERROR: for db  Cannot start service db: b'driver failed programming external connectivity on endpoint my_postgresql (6d1ac474c98b3093fb1173b335abe12eb64b1895bc93d501430dbe37a7b9165b): Error starting userland proxy: listen tcp 0.0.0.0:5432: bind: address already in use'

ERROR: for mongo  Cannot start service mongo: b'driver failed programming external connectivity on endpoint my_mongo (78f1054c5c746222f5881dc7e9d40f39b38d2f36ec29b40b79dba3ee59f6d104): Error starting userland proxy: listen tcp 0.0.0.0:27017: bind: address already in use'
Encountered errors while bringing up the project.


That's all folks!

I will make a commit to fix the issue ASAP.

Best,
Fernando

@fermenreq
Copy link

fermenreq commented Oct 7, 2019

Hi @fchauvel

It's OK that CAMP build and generates 10 configurations for CityGo but It fails on deployment step.

root@6ec36b3c8054://camp/samples/stamp/atos/performance# camp execute -d .
CAMP v0.6.3-git.98aa936 (MIT)
Copyright (C) 2017 -- 2019 SINTEF Digital

Loaded './camp.yml'.
Loading configurations from './out' ...

 - Executing ./out/config_1
   1. Building images ...
      $ bash build_images.sh (from './out/config_1/images')
   2. Starting Services ...
      $ docker-compose up -d (from './out/config_1')
   3. Running tests ...
      $ docker-compose run browser bzt citygo.jmx --option=modules.console.disable=false -report (from './out/config_1')
   4. Collecting reports ...
      $ docker ps --all --quiet --filter name=config_1_browser_run_ (from './out/config_1')
      $ docker cp :/browser/./ ./test-reports (from './out/config_1')
   5. Stopping Services ...
      $ docker-compose down --volumes --rmi all (from './out/config_1')

Test execution aborted!
 - Error: A shell command failed (code: 1)
   $ docker cp :/browser/./ ./test-reports
   Check out logs in 'camp_execute.log'.

must specify at least one container source


That's all folks!

Camp execute Logs

Image for service web was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating selenium_hub  ... done
Creating my_postgresql   ... done
Creating my_mongo        ... done
Creating selenium_chrome ... done
Creating my_apache       ... done
Creating my_web          ... done

camp@bash:./out/config_1$ docker-compose run browser bzt citygo.jmx --option=modules.console.disable=false -report
No such service: browser

camp@bash:./out/config_1$ docker ps --all --quiet --filter name=config_1_browser_run_

camp@bash:./out/config_1$ docker cp :/browser/./ ./test-reports
must specify at least one container source

camp@bash:./out/config_1$ docker-compose down --volumes --rmi all
Stopping my_web          ... done
Stopping my_apache       ... done
Stopping selenium_chrome ... done
Stopping my_mongo        ... done
Stopping my_postgresql   ... done
Stopping selenium_hub    ... done
Removing my_web          ... done
Removing my_apache       ... done
Removing selenium_chrome ... done
Removing my_mongo        ... done
Removing my_postgresql   ... done
Removing selenium_hub    ... done
Removing network config_1_my-app-bridge
Removing volume config_1_postgres_data
Removing volume config_1_static_volume

Run funtional Test :

docker exec -i my_web sh -c 'pytest -v backend/tests.py --html=pytest_report.html --junitxml=pytest_report_xml.xml --self-contained-html && exit'

Run performance Test in a Jenkins pipeline job:

 stage('Performance Test') {
        parallel BlazeMeterTest:{
                dir('Tests/citygo_tests'){
                    sh 'bzt citygo.jmx --option=modules.console.disable=false -report'
                }
        }, Analysis:{
             sleep 60
        },
        failFast: true|false
    }

As I see in camp.yml in browser component section:

    ...
    tests:
      command: bzt citygo.jmx --option=modules.console.disable=false -report
      reports:
        location: "./"
        pattern: ".xml"
        format: junit
    ...

Does CAMP needs Tests/citygo_tests as a location in camp.yml descriptor?

Best,
Fernando Mendez - fernando.mendez@atos.net

@fchauvel
Copy link
Member Author

fchauvel commented Oct 7, 2019

Hi,

Yes, actually according to the camp.yml file, the tests are launched from the browser component /container. So the file template\browser\Dockerfile should yield an image with all the needed tools installed and files available. As it is now, I think we miss the files Tests/citygo_tests you mention.

@fchauvel
Copy link
Member Author

fchauvel commented Oct 7, 2019

Hi again,

Also, I see in the logs that CAMP is trying to run a container named "browser", which it expects to find according to the camp.yml file, so I would add a service in the docker-compose-file as follows:

services:
  browser:
    build: ./browser

This service will be run by CAMP when it lauched the tests, and then CAMP will fetch the test reports from there.

@fermenreq
Copy link

Hi,

I tested again and it gets the same output reported before. By other hand you have to know that test located in template/browser/Dockerfile are related for scenario 2 ( test: funtional test), but it's located in performance folder.

I just commit and I relocated test files according both scenearios. See here: 4109d70

@fermenreq
Copy link

fermenreq commented Oct 8, 2019

hi,

Fixed here: a81215b, 697d52a
1d15e57

@fermenreq
Copy link

fermenreq commented Oct 14, 2019

Hi @fchauvel ,

Some errors in tests deployment CI/CD:

JavaGreetingsShould

E   test_run_all_tests_three_times

Error in tests.acceptance.test_java.JavaGreetingsShould.test_run_all_tests_three_times
  File "/usr/lib/python3.5/unittest/case.py", line 59, in testPartExecutor yield
  File "/usr/lib/python3.5/unittest/case.py", line 601, in run
    testMethod()
  File "/camp/tests/acceptance/test_java.py", line 37, in test_run_all_tests_three_times
    report = self.scenario.fetch_test_report()
  File "/camp/tests/commons.py", line 85, in fetch_test_report
    return self._output.load_reports()
  File "/camp/camp/directories.py", line 175, in load_reports
    with open(self.test_report, "r") as yaml_file:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp952ku9gi/camp_k8wrozmi/out/test_report.yaml'

It seems test_report.yaml is not generated

See here

Thanks, Fernando

@fermenreq
Copy link

Hi,

According scenario 2 case study in functional test I get the following error when CAMP run realize configurations generated.

CAMP v0.6.4-git.2bbc8f5 (MIT)
Copyright (C) 2017 -- 2019 SINTEF Digital

Loaded './camp.yml'.

- Config. 1 in './out/config_1/configuration.yml'.
Includes ubuntu, python, nginx (True), mongo, citygo (v1.10.2), hub, pos...

- Config. 2 in './out/config_2/configuration.yml'.
Includes ubuntu, citygo (v2.2.6), python, nginx (False), mongo, hub, pos...

- Config. 3 in './out/config_3/configuration.yml'.
Includes ubuntu, python, nginx (False), mongo, citygo (v1.10.2), hub, po...

- Config. 4 in './out/config_4/configuration.yml'.
Includes ubuntu, citygo (v2.2.6), python, nginx (True), mongo, hub, post...

That's all folks!
+ exit
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (CAMP realize)
[Pipeline] sh
+ docker exec -i camp sh -c cd samples/stamp/atos/functional/ && camp realize -d . && cp -R out/ /stamp_docker_citygoApp/

CAMP v0.6.4-git.2bbc8f5 (MIT)
Copyright (C) 2017 -- 2019 SINTEF Digital

Loaded './camp.yml'.
Loading configurations from './out' ...

Error:
Vain substitution: No match found!'
- pattern: 'FROM\s+camp/[a-zA-Z0-9\._-]+(?:\:[a-zA-Z0-9\._-]+)?'
- target file: './out/config_2/images/citygo_0/Dockerfile'
Is the pattern appropriate? What about the file content?

Best,
Fernando

@fchauvel
Copy link
Member Author

Close as per v0.9.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants