Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wrong data in pm2 jlist property versioning #3276

Open
KaiSchwarz-cnic opened this issue Nov 13, 2017 · 17 comments
Open

wrong data in pm2 jlist property versioning #3276

KaiSchwarz-cnic opened this issue Nov 13, 2017 · 17 comments

Comments

@KaiSchwarz-cnic
Copy link

I called pm2 jlist. In data property pm2_env.versioning I get data to another repository. In our case: our webhook repository which is used to trigger the deployment process and thus restarts the pm2 apps.
For each of our pm2 apps we have this data covered in the above data property which doesn't make a lot sense. The purpose of that object data is imo to cover the versioning data of the app repository itself. I hope I am right here.

Our webhook creates always a new child process and executes the deployment script of the appropriate app. the child process always inherits the environment of the parent process.

===============================================================================
--- PM2 REPORT (Mon Nov 13 2017 12:40:25 GMT+0000 (UTC)) ----------------------
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 2.6.1
node version         : 6.10.2
node path            : /usr/lib/node_modules/pm2/bin/pm2
argv                 : /usr/bin/nodejs,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : node
uid                  : 1000
gid                  : 1000
uptime               : 36450min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 2.6.1
node version         : 6.11.5
node path            : /usr/bin/pm2
argv                 : /home/node/.nvm/versions/node/v6.11.5/bin/node,/usr/bin/pm2,report
argv0                : node
user                 : node
uid                  : 1000
gid                  : 1000
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Xeon(R) CPU E3-1271 v3 @ 3.60GHz
cpus nb              : 8
freemem              : 532135936
totalmem             : 33694265344
home                 : /home/node
===============================================================================
--- PM2 list -----------------------------------------------
┌───────────┬────┬─────────┬───────┬────────┬─────────┬────────┬─────┬────────────┬──────┬──────────┐
│ App name  │ id │ mode    │ pid   │ status │ restart │ uptime │ cpu │ mem        │ user │ watching │
├───────────┼────┼─────────┼───────┼────────┼─────────┼────────┼─────┼────────────┼──────┼──────────┤
│ CP3PRD    │ 6  │ cluster │ 26650 │ online │ 68      │ 2D     │ 0%  │ 421.4 MB   │ node │ disabled │
│ CP3PRD    │ 3  │ cluster │ 26161 │ online │ 68      │ 2D     │ 0%  │ 337.4 MB   │ node │ disabled │
│ CP3PRD    │ 4  │ cluster │ 26406 │ online │ 68      │ 2D     │ 0%  │ 407.8 MB   │ node │ disabled │
│ CP3PRD    │ 5  │ cluster │ 26615 │ online │ 68      │ 2D     │ 0%  │ 407.9 MB   │ node │ disabled │
│ CP3PRD    │ 2  │ cluster │ 26371 │ online │ 68      │ 2D     │ 0%  │ 326.8 MB   │ node │ disabled │
│ CP3PRD    │ 7  │ cluster │ 27008 │ online │ 67      │ 2D     │ 0%  │ 392.7 MB   │ node │ disabled │
│ CP3PRD    │ 8  │ cluster │ 27042 │ online │ 66      │ 2D     │ 0%  │ 441.3 MB   │ node │ disabled │
│ CP3PRD    │ 9  │ cluster │ 27076 │ online │ 66      │ 2D     │ 1%  │ 404.1 MB   │ node │ disabled │
│ PORT80APP │ 15 │ cluster │ 29843 │ online │ 0       │ 4h     │ 0%  │ 39.0 MB    │ node │ disabled │
│ TLDINFO   │ 0  │ cluster │ 27407 │ online │ 9       │ 3D     │ 0%  │ 101.4 MB   │ node │ disabled │
│ WEBHOOK   │ 16 │ cluster │ 560   │ online │ 0       │ 3h     │ 0%  │ 36.5 MB    │ node │ disabled │
└───────────┴────┴─────────┴───────┴────────┴─────────┴────────┴─────┴────────────┴──────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/home/node/.pm2/pm2.log last 20 lines:
PM2        | 2017-11-13 08:37:16: 14
PM2        | 2017-11-13 08:37:16: Stopping app:PORT80APP id:14
PM2        | 2017-11-13 08:37:16: App name:PORT80APP id:14 disconnected
PM2        | 2017-11-13 08:37:16: App [PORT80APP] with id [14] and pid [29094], exited with code [0] via signal [SIGINT]
PM2        | 2017-11-13 08:37:16: pid=29094 msg=process killed
PM2        | 2017-11-13 08:37:16: Starting execution sequence in -cluster mode- for app name:PORT80APP id:15
PM2        | 2017-11-13 08:37:26: App name:PORT80APP id:15 online
PM2        | 2017-11-13 08:39:07: Stopping app:WEBHOOK id:11
PM2        | 2017-11-13 08:39:07: App name:WEBHOOK id:11 disconnected
PM2        | 2017-11-13 08:39:07: App [WEBHOOK] with id [11] and pid [21068], exited with code [0] via signal [SIGINT]
PM2        | 2017-11-13 08:39:07: pid=21068 msg=process killed
PM2        | 2017-11-13 08:39:07: Starting execution sequence in -cluster mode- for app name:WEBHOOK id:11
PM2        | 2017-11-13 08:39:17: App name:WEBHOOK id:11 online
PM2        | 2017-11-13 08:39:49: 11
PM2        | 2017-11-13 08:39:49: Stopping app:WEBHOOK id:11
PM2        | 2017-11-13 08:39:49: App name:WEBHOOK id:11 disconnected
PM2        | 2017-11-13 08:39:49: App [WEBHOOK] with id [11] and pid [31075], exited with code [0] via signal [SIGINT]
PM2        | 2017-11-13 08:39:49: pid=31075 msg=process killed
PM2        | 2017-11-13 08:41:53: Starting execution sequence in -cluster mode- for app name:WEBHOOK id:16
PM2        | 2017-11-13 08:41:53: App name:WEBHOOK id:16 online

Have we to change anything in our process or is this indeed a bug?

@vmarchaud
Copy link
Contributor

You must set the cwd correctly when launching your application, could you share your ecosystem ?

@KaiSchwarz-cnic
Copy link
Author

So this is the configuration for one of our apps used in the webhook to redeploy:

CP3_PRD: {
    branches: ["master"],
    command: "./deploy.sh master CP3_PRD",
    events: ["push"],
    exit: false,
    group: "node",
    options: {
      cwd: "/opt/fnode",
      env: {
        NODE_ENV: "production",
      },
    },
    token: ".............................. ;o) ................................",
    user: "node",
  },

cwd points to the directory where the repository of that appropriate app got cloned to. That should be correct then for my understanding.

in the webhook logic, we set the process environment then as follows:

 tpl.options.env = Object.assign({}, process.env, tpl.options.env);//tpl corresponds to the config above

and finally execute the command through child_process.exec:

cp.exec(tpl.command, tpl.options, (err, stdout, stderr) => {
    //handling of the callback results        
});

Basically, I don't see there an issue? I mean, everything is restarting and working fine, just not these data fields.

@KaiSchwarz-cnic
Copy link
Author

KaiSchwarz-cnic commented Nov 16, 2017

Ehm.. can pm2 reload --update-env be the reason in the appropriate deployment script of the apps?

@KaiSchwarz-cnic
Copy link
Author

No, not the case. I've started our port 80 to 443 forwarding app manually: all data in these fields looked fine. triggered redeployment via webhook app -> data broken. --update-env is not part of the deployment script of that port forwarding app.

@KaiSchwarz-cnic
Copy link
Author

Do you need anything else @vmarchaud ?

@KaiSchwarz-cnic
Copy link
Author

Looks like tpl.options.env = Object.assign({}, process.env, tpl.options.env); is the reason for this.
Where the process environment of the existing process (parent) is getting set for the child process, but overwritten by the template env settings.
This got introduces as we had problems one day when restarting our apps, I'll keep this issue updated. Let me see if I can solve this. If you have any suggestion, let me know @vmarchaud.

@KaiSchwarz-cnic
Copy link
Author

KaiSchwarz-cnic commented Nov 20, 2017

introduced because of: #2756

If I change tpl.options.env = Object.assign({}, process.env, tpl.options.env); to tpl.options.env = Object.assign({}, tpl.options.env); necessary paths seem to be missing so that deployment process breaks. e.g. ./deploy.sh: line 5: npm: command not found

I am asking myself what you suggest to be the right way here. I really have no clue.

@vmarchaud
Copy link
Contributor

vmarchaud commented Nov 20, 2017

I'm pretty sure --update-env can't update the cwd of your application, did you set the cwd in the pm2 configuration of your application ?

@KaiSchwarz-cnic
Copy link
Author

KaiSchwarz-cnic commented Nov 20, 2017

here's the pm2.json we use for our main app:

{
  "apps": [{
    "name": "CP3PRD",
    "script": "fnode.js",
    "args": ["CP3_PRD"],
    "error_file": "/var/log/frontend/CP3/err.log",
    "out_file": "/var/log/frontend/CP3/out.log",
    "instances": 0,
    "cwd": "/opt/fnode",
    "env": {
      "NODE_ENV": "production"
    },
    "exec_mode": "cluster",
    "exec_interpreter": "node",
    "merge_logs": true,
    "log_date_format": "YYYY-MM-DD HH:mm:ss Z",
    "autorestart": true,
    "restart_delay": 10000,
    "watch": false,
    "wait_ready": true,
    "listen_timeout": 10000,
    "kill_timeout": 20000
  }]
}

That's the config we use to start the application.
The options part is reflected again accordingly in the webhook configuration which we use as mentioned:

options: {
      cwd: "/opt/fnode",
      env: {
        NODE_ENV: "production",
      },
    },

So I cannot imagine cwd is the reason behind.

@KaiSchwarz-cnic
Copy link
Author

KaiSchwarz-cnic commented Nov 20, 2017

I have the impression that pm2 itself reuses some options settings (which came from parent process and were generated for the webhook app) at least for the versioning part later on. Deployment itself seems to work correctly.

So as, settings from the parent process seem to be required to have paths to npm, et al set correctly, maybe further things I am not aware of currently - I am asking myself which object keys from the options object need to be cleaned up to get this working?

@KaiSchwarz-cnic
Copy link
Author

any further suggestions?

@vmarchaud
Copy link
Contributor

No suggestion in this case it will need to be inspected by someone (feel free to do it and open up a PR to fix any issue)

@KaiSchwarz-cnic
Copy link
Author

KaiSchwarz-cnic commented Nov 21, 2017

Thanks, as this goes very deep into pm2 architecture, I don't think I am able to cover this (and yes, I am very busy in addition). If I should get the time, yes I'll have an eye on it, but don't expect this to happen.

@KaiSchwarz-cnic
Copy link
Author

Are there news on this issue? I would also recommend to get the "S: Open for PR" flag removed, as I won't get that time in near future.

@wallet77
Copy link
Contributor

@Papakai this flag is not only for people who open issue, but for all people who want to contribute.

@KaiSchwarz-cnic
Copy link
Author

KaiSchwarz-cnic commented Feb 28, 2018

@wallet77 thanks, good to know. Still leaves me the question when / if this might get reviewed. Or at least when to get some informations which environment data to explicitely drop to avoid this data issue.

@wallet77
Copy link
Contributor

@Papakai we have a lot of work with the next release. I think we can check this one (and many other issues linked to versioning) after the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants