CP-2093: "Host internal management network" support #7

djs55 · 2011-02-03T21:57:13Z

This rebrands the badly-named "Guest installer network" to the more appropriate "Host internal management network", changes the IP address range being handled out from an arbitrary (possibly clashing) RFC1918 network to the IPv4 zeroconf range and ensures that xapi listens properly on port 80, even when the external management interface is disabled or xapi is restarted.

…4.* route being added automatically. Signed-off-by: David Scott <dave.scott@eu.citrix.com> Reviewed-by: Rob Hoes <rob.hoes@citrix.com>

…gement network" reflecting its new use. Transparently upgrade the old into the new. Signed-off-by: David Scott <dave.scott@eu.citrix.com> Reviewed-by: Rob Hoes <rob.hoes@citrix.com>

…management" network to 127.0.0.1 rather than the management IP so that it works when the management interface is disabled. Signed-off-by: David Scott <dave.scott@eu.citrix.com> Reviewed-by: Rob Hoes <rob.hoes@eu.citrix.com>

djs55 · 2011-02-09T13:59:08Z

I've taken Rob's advice and refactored the network creation logic such that the "upgrade" case is handled in the xapi upgrade code, rather than the create_networks code, to allow for easier GC later.

I've also tweaked it to update the network in-place rather than destroy/create, which should prevent problems should someone actually be using it during the upgrade.

…127-tampa-lcm CA-153127: Enable master connection watchdog

Update to 0.9.2

storage test: add a simple VDI.clone test

Move xml_http_rpc to Xcp_client

rrd2csv.opam: use xapi-client instead of xapi

CA-111642: Removed the .snk key from the repository. Created a new one a...

Add Api_messages module

Update where rrdd looks in sysfs for VBDs

Update following ocamlfind rename to 'xen-gnt'

Add travis integration

SCTX-1766: Configurable timeout for stunnel

Separate sparse dd

sparse_dd: Fix references to Vhd_lwt.File.use_unbuffered

Fix bug parsing `tap-ctl list` when filenames contain spaces

Report IPv4 addresses as well as IPV6 addresses.

Use upstream ocaml-base64, drop custom implementation

git: Add metadata to the result of `git archive`

CA-341597: Raise the open fd limit

CP-8359: Add check for subsystem device IDs

ezxenstore: update for safe-string

Using an off-cpu flamegraph I identified that concurrent PAM calls are slow due to a call to `sleep(1)`. `pam_authenticate` calls `crypt_r` which calls `NSSLOW_Init` which on first use will try to initialize the just `dlopen`-ed library. If it encounters a race condition it does a `sleep(1)`. This race condition can be quite reliably reproduced when performing a lot of PAM authentications from multiple threads in parallel. GDB can also be used to confirm this by putting a breakpoint on `sleep`: ``` #0 __sleep (seconds=seconds@entry=1) at ../sysdeps/unix/sysv/linux/sleep.c:42 #1 0x00007ffff1548e22 in freebl_RunLoaderOnce () at lowhash_vector.c:122 #2 0x00007ffff1548f31 in freebl_InitVector () at lowhash_vector.c:131 #3 NSSLOW_Init () at lowhash_vector.c:148 #4 0x00007ffff1b8f09a in __sha512_crypt_r (key=key@entry=0x7fffd8005a60 "pamtest-edvint", salt=0x7ffff31e17b8 "dIJbsXKc0", xapi-project#5 0x00007ffff1b8d070 in __crypt_r (key=key@entry=0x7fffd8005a60 "pamtest-edvint", salt=<optimized out>, xapi-project#6 0x00007ffff1dc9abc in verify_pwd_hash (p=p@entry=0x7fffd8005a60 "pamtest-edvint", hash=<optimized out>, nullok=nullok@entry=0) at passverify.c:111 xapi-project#7 0x00007ffff1dc9139 in _unix_verify_password (pamh=pamh@entry=0x7fffd8002910, name=0x7fffd8002ab0 "pamtest-edvint", p=0x7fffd8005a60 "pamtest-edvint", ctrl=ctrl@entry=8389156) at support.c:777 xapi-project#8 0x00007ffff1dc6556 in pam_sm_authenticate (pamh=0x7fffd8002910, flags=<optimized out>, argc=<optimized out>, argv=<optimized out>) at pam_unix_auth.c:178 xapi-project#9 0x00007ffff7bcef1a in _pam_dispatch_aux (use_cached_chain=<optimized out>, resumed=<optimized out>, h=<optimized out>, flags=1, pamh=0x7fffd8002910) at pam_dispatch.c:110 xapi-project#10 _pam_dispatch (pamh=pamh@entry=0x7fffd8002910, flags=1, choice=choice@entry=1) at pam_dispatch.c:426 xapi-project#11 0x00007ffff7bce7e0 in pam_authenticate (pamh=0x7fffd8002910, flags=flags@entry=1) at pam_auth.c:34 xapi-project#12 0x00000000005ae567 in XA_mh_authorize (username=username@entry=0x7fffd80028d0 "pamtest-edvint", password=password@entry=0x7fffd80028f0 "pamtest-edvint", error=error@entry=0x7ffff31e1be8) at xa_auth.c:83 xapi-project#13 0x00000000005adf20 in stub_XA_mh_authorize (username=<optimized out>, password=<optimized out>) at xa_auth_stubs.c:42 xapi-project#14 0x00000000004a0a6a in camlDune__exe__Bench_pam__pam_authenticate$27_320 () at ocaml/tests/bench/pam/bench_pam.ml:63 xapi-project#15 0x00000000004a1113 in camlEzbechamel_concurrent__worker_loop_359 () at ocaml/tests/bench/lib/concurrent/ezbechamel_concurrent.ml:36 xapi-project#16 0x00000000005935b9 in camlStdlib__Fun__protect_317 () xapi-project#17 0x00000000004a1955 in camlThread__fun_850 () xapi-project#18 0x00000000005d6401 in caml_start_program () xapi-project#19 0x00000000005cd0fd in caml_callback_exn () xapi-project#20 0x00000000005af810 in caml_thread_start () xapi-project#21 0x00007ffff79b7e25 in start_thread (arg=0x7ffff31e2700) at pthread_create.c:308 xapi-project#22 0x00007ffff71dbbad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 ``` `pam_start` and `pam_end` doesn't help here, because on `pam_end` the library is `dlclose`-ed, so on next `pam_authenticate` it will have to go through the initialization code again. (This initialization code would've belonged into `pam_start`, not `pam_authenticate`, but there are several layers here including a call to `crypt_r`). To avoid this link with `libcrypt` and call `crypt_r` once ourselves (and ensure it loads `libfreeblpriv3` by using the sha512 prefix). That way the library will stay loaded (we'll hold a reference count on it), and the `dlclose` done by PAM won't unload it. Confirmed that there are no `sleep` calls now, and the results are also visible when running the benchmark targeted to the with and without fix code: ``` ╭─────────────────────────────────────────────────┬───────────────────────────┬───────────────────────────┬───────────────────────────╮ │name │ major-allocated │ minor-allocated │ monotonic-clock │ ├─────────────────────────────────────────────────┼───────────────────────────┼───────────────────────────┼───────────────────────────┤ │ concurrent authenticate (sleep fix, actual):8 │ 0.0000 mjw/run│ 50.0000 mnw/run│ 27043467.0000 ns/run│ ╰─────────────────────────────────────────────────┴───────────────────────────┴───────────────────────────┴───────────────────────────╯ ╭────────────────────────────────────────┬───────────────────────────┬───────────────────────────┬───────────────────────────╮ │name │ major-allocated │ minor-allocated │ monotonic-clock │ ├────────────────────────────────────────┼───────────────────────────┼───────────────────────────┼───────────────────────────┤ │ concurrent authenticate (no reuse):8 │ 0.0000 mjw/run│ 50.0000 mnw/run│ 1029831372.0000 ns/run│ ╰────────────────────────────────────────┴───────────────────────────┴───────────────────────────┴───────────────────────────╯ ``` Without this fix using 2 threads to perform PAM authentication would result in a 38x slowdown compared to using no threads at all (which is what XAPI currently does). Signed-off-by: Edwin Török <edwin.torok@cloud.com>

This enables PAM to be used in multithreaded mode (currently XAPI has a global lock around auth). Using an off-cpu flamegraph I identified that concurrent PAM calls are slow due to a call to `sleep(1)`. `pam_authenticate` calls `crypt_r` which calls `NSSLOW_Init` which on first use will try to initialize the just `dlopen`-ed library. If it encounters a race condition it does a `sleep(1)`. This race condition can be quite reliably reproduced when performing a lot of PAM authentications from multiple threads in parallel. GDB can also be used to confirm this by putting a breakpoint on `sleep`: ``` #0 __sleep (seconds=seconds@entry=1) at ../sysdeps/unix/sysv/linux/sleep.c:42 #1 0x00007ffff1548e22 in freebl_RunLoaderOnce () at lowhash_vector.c:122 #2 0x00007ffff1548f31 in freebl_InitVector () at lowhash_vector.c:131 #3 NSSLOW_Init () at lowhash_vector.c:148 #4 0x00007ffff1b8f09a in __sha512_crypt_r (key=key@entry=0x7fffd8005a60 "pamtest-edvint", salt=0x7ffff31e17b8 "dIJbsXKc0", xapi-project#5 0x00007ffff1b8d070 in __crypt_r (key=key@entry=0x7fffd8005a60 "pamtest-edvint", salt=<optimized out>, xapi-project#6 0x00007ffff1dc9abc in verify_pwd_hash (p=p@entry=0x7fffd8005a60 "pamtest-edvint", hash=<optimized out>, nullok=nullok@entry=0) at passverify.c:111 xapi-project#7 0x00007ffff1dc9139 in _unix_verify_password (pamh=pamh@entry=0x7fffd8002910, name=0x7fffd8002ab0 "pamtest-edvint", p=0x7fffd8005a60 "pamtest-edvint", ctrl=ctrl@entry=8389156) at support.c:777 xapi-project#8 0x00007ffff1dc6556 in pam_sm_authenticate (pamh=0x7fffd8002910, flags=<optimized out>, argc=<optimized out>, argv=<optimized out>) at pam_unix_auth.c:178 xapi-project#9 0x00007ffff7bcef1a in _pam_dispatch_aux (use_cached_chain=<optimized out>, resumed=<optimized out>, h=<optimized out>, flags=1, pamh=0x7fffd8002910) at pam_dispatch.c:110 xapi-project#10 _pam_dispatch (pamh=pamh@entry=0x7fffd8002910, flags=1, choice=choice@entry=1) at pam_dispatch.c:426 xapi-project#11 0x00007ffff7bce7e0 in pam_authenticate (pamh=0x7fffd8002910, flags=flags@entry=1) at pam_auth.c:34 xapi-project#12 0x00000000005ae567 in XA_mh_authorize (username=username@entry=0x7fffd80028d0 "pamtest-edvint", password=password@entry=0x7fffd80028f0 "pamtest-edvint", error=error@entry=0x7ffff31e1be8) at xa_auth.c:83 xapi-project#13 0x00000000005adf20 in stub_XA_mh_authorize (username=<optimized out>, password=<optimized out>) at xa_auth_stubs.c:42 ``` `pam_start` and `pam_end` doesn't help here, because on `pam_end` the library is `dlclose`-ed, so on next `pam_authenticate` it will have to go through the initialization code again. (This initialization code would've belonged into `pam_start`, not `pam_authenticate`, but there are several layers here including a call to `crypt_r`). Upstream has fixed this problem >5 years ago by switching to libxcrypt instead. Signed-off-by: Edwin Török <edwin.torok@cloud.com>

Fix build on OS X

Use Stdext 2.0.0

Backport of 3b52b72 This enables PAM to be used in multithreaded mode (currently XAPI has a global lock around auth). Using an off-cpu flamegraph I identified that concurrent PAM calls are slow due to a call to `sleep(1)`. `pam_authenticate` calls `crypt_r` which calls `NSSLOW_Init` which on first use will try to initialize the just `dlopen`-ed library. If it encounters a race condition it does a `sleep(1)`. This race condition can be quite reliably reproduced when performing a lot of PAM authentications from multiple threads in parallel. GDB can also be used to confirm this by putting a breakpoint on `sleep`: ``` #0 __sleep (seconds=seconds@entry=1) at ../sysdeps/unix/sysv/linux/sleep.c:42 xapi-project#1 0x00007ffff1548e22 in freebl_RunLoaderOnce () at lowhash_vector.c:122 xapi-project#2 0x00007ffff1548f31 in freebl_InitVector () at lowhash_vector.c:131 xapi-project#3 NSSLOW_Init () at lowhash_vector.c:148 xapi-project#4 0x00007ffff1b8f09a in __sha512_crypt_r (key=key@entry=0x7fffd8005a60 "pamtest-edvint", salt=0x7ffff31e17b8 "dIJbsXKc0", xapi-project#5 0x00007ffff1b8d070 in __crypt_r (key=key@entry=0x7fffd8005a60 "pamtest-edvint", salt=<optimized out>, xapi-project#6 0x00007ffff1dc9abc in verify_pwd_hash (p=p@entry=0x7fffd8005a60 "pamtest-edvint", hash=<optimized out>, nullok=nullok@entry=0) at passverify.c:111 xapi-project#7 0x00007ffff1dc9139 in _unix_verify_password (pamh=pamh@entry=0x7fffd8002910, name=0x7fffd8002ab0 "pamtest-edvint", p=0x7fffd8005a60 "pamtest-edvint", ctrl=ctrl@entry=8389156) at support.c:777 xapi-project#8 0x00007ffff1dc6556 in pam_sm_authenticate (pamh=0x7fffd8002910, flags=<optimized out>, argc=<optimized out>, argv=<optimized out>) at pam_unix_auth.c:178 xapi-project#9 0x00007ffff7bcef1a in _pam_dispatch_aux (use_cached_chain=<optimized out>, resumed=<optimized out>, h=<optimized out>, flags=1, pamh=0x7fffd8002910) at pam_dispatch.c:110 xapi-project#10 _pam_dispatch (pamh=pamh@entry=0x7fffd8002910, flags=1, choice=choice@entry=1) at pam_dispatch.c:426 xapi-project#11 0x00007ffff7bce7e0 in pam_authenticate (pamh=0x7fffd8002910, flags=flags@entry=1) at pam_auth.c:34 xapi-project#12 0x00000000005ae567 in XA_mh_authorize (username=username@entry=0x7fffd80028d0 "pamtest-edvint", password=password@entry=0x7fffd80028f0 "pamtest-edvint", error=error@entry=0x7ffff31e1be8) at xa_auth.c:83 xapi-project#13 0x00000000005adf20 in stub_XA_mh_authorize (username=<optimized out>, password=<optimized out>) at xa_auth_stubs.c:42 ``` `pam_start` and `pam_end` doesn't help here, because on `pam_end` the library is `dlclose`-ed, so on next `pam_authenticate` it will have to go through the initialization code again. (This initialization code would've belonged into `pam_start`, not `pam_authenticate`, but there are several layers here including a call to `crypt_r`). Upstream has fixed this problem >5 years ago by switching to libxcrypt instead. Signed-off-by: Edwin Török <edwin.torok@cloud.com> Signed-off-by: Christian Lindig <christian.lindig@cloud.com>

Backport of 3b52b72 This enables PAM to be used in multithreaded mode (currently XAPI has a global lock around auth). Using an off-cpu flamegraph I identified that concurrent PAM calls are slow due to a call to `sleep(1)`. `pam_authenticate` calls `crypt_r` which calls `NSSLOW_Init` which on first use will try to initialize the just `dlopen`-ed library. If it encounters a race condition it does a `sleep(1)`. This race condition can be quite reliably reproduced when performing a lot of PAM authentications from multiple threads in parallel. GDB can also be used to confirm this by putting a breakpoint on `sleep`: ``` #0 __sleep (seconds=seconds@entry=1) at ../sysdeps/unix/sysv/linux/sleep.c:42 #1 0x00007ffff1548e22 in freebl_RunLoaderOnce () at lowhash_vector.c:122 #2 0x00007ffff1548f31 in freebl_InitVector () at lowhash_vector.c:131 #3 NSSLOW_Init () at lowhash_vector.c:148 #4 0x00007ffff1b8f09a in __sha512_crypt_r (key=key@entry=0x7fffd8005a60 "pamtest-edvint", salt=0x7ffff31e17b8 "dIJbsXKc0", #5 0x00007ffff1b8d070 in __crypt_r (key=key@entry=0x7fffd8005a60 "pamtest-edvint", salt=<optimized out>, #6 0x00007ffff1dc9abc in verify_pwd_hash (p=p@entry=0x7fffd8005a60 "pamtest-edvint", hash=<optimized out>, nullok=nullok@entry=0) at passverify.c:111 #7 0x00007ffff1dc9139 in _unix_verify_password (pamh=pamh@entry=0x7fffd8002910, name=0x7fffd8002ab0 "pamtest-edvint", p=0x7fffd8005a60 "pamtest-edvint", ctrl=ctrl@entry=8389156) at support.c:777 #8 0x00007ffff1dc6556 in pam_sm_authenticate (pamh=0x7fffd8002910, flags=<optimized out>, argc=<optimized out>, argv=<optimized out>) at pam_unix_auth.c:178 #9 0x00007ffff7bcef1a in _pam_dispatch_aux (use_cached_chain=<optimized out>, resumed=<optimized out>, h=<optimized out>, flags=1, pamh=0x7fffd8002910) at pam_dispatch.c:110 #10 _pam_dispatch (pamh=pamh@entry=0x7fffd8002910, flags=1, choice=choice@entry=1) at pam_dispatch.c:426 #11 0x00007ffff7bce7e0 in pam_authenticate (pamh=0x7fffd8002910, flags=flags@entry=1) at pam_auth.c:34 #12 0x00000000005ae567 in XA_mh_authorize (username=username@entry=0x7fffd80028d0 "pamtest-edvint", password=password@entry=0x7fffd80028f0 "pamtest-edvint", error=error@entry=0x7ffff31e1be8) at xa_auth.c:83 #13 0x00000000005adf20 in stub_XA_mh_authorize (username=<optimized out>, password=<optimized out>) at xa_auth_stubs.c:42 ``` `pam_start` and `pam_end` doesn't help here, because on `pam_end` the library is `dlclose`-ed, so on next `pam_authenticate` it will have to go through the initialization code again. (This initialization code would've belonged into `pam_start`, not `pam_authenticate`, but there are several layers here including a call to `crypt_r`). Upstream has fixed this problem >5 years ago by switching to libxcrypt instead. Signed-off-by: Edwin Török <edwin.torok@cloud.com> Signed-off-by: Christian Lindig <christian.lindig@cloud.com>

Release 0.3

David Scott added 3 commits February 9, 2011 13:44

CP-2093: write NOZEROCONF=yes to all ifcfg-* files, to avoid a 169.25…

2ad29d7

…4.* route being added automatically. Signed-off-by: David Scott <dave.scott@eu.citrix.com> Reviewed-by: Rob Hoes <rob.hoes@citrix.com>

CP-2093: Rename the "Guest installer network" the "Host internal mana…

d99e927

…gement network" reflecting its new use. Transparently upgrade the old into the new. Signed-off-by: David Scott <dave.scott@eu.citrix.com> Reviewed-by: Rob Hoes <rob.hoes@citrix.com>

johnelse pushed a commit to johnelse/xen-api that referenced this pull request Mar 20, 2015

Merge pull request xapi-project#7 from simonjbeaumont/kaifengz-CA-153…

99c1f65

…127-tampa-lcm CA-153127: Enable master connection watchdog

sharady pushed a commit to sharady/xen-api that referenced this pull request May 11, 2015

Merge pull request xapi-project#7 from djs55/new-version

d0c552d

Update to 0.9.2

sharady pushed a commit to sharady/xen-api that referenced this pull request May 11, 2015

Merge pull request xapi-project#7 from djs55/add-storage-test

a3a75db

storage test: add a simple VDI.clone test

koushikcgit pushed a commit to koushikcgit/xen-api that referenced this pull request Aug 7, 2015

Merge pull request xapi-project#7 from jonludlam/fix-build

42ff4ca

Move xml_http_rpc to Xcp_client

lippirk pushed a commit to lippirk/xen-api that referenced this pull request Jul 20, 2020

Merge pull request xapi-project#7 from mseri/patch-1

38225f3

rrd2csv.opam: use xapi-client instead of xapi

kc284 added a commit to kc284/xen-api that referenced this pull request Mar 11, 2021

Merge pull request xapi-project#7 from kc284/master

4793d68

CA-111642: Removed the .snk key from the repository. Created a new one a...

robhoes pushed a commit to robhoes/xen-api that referenced this pull request Jul 6, 2021

Merge pull request xapi-project#7 from johnelse/add-api-messages

0303e4d

Add Api_messages module

robhoes added a commit that referenced this pull request Sep 14, 2021

Merge pull request #7 from simonjbeaumont/cp-4056

c49deda

Update where rrdd looks in sysfs for VBDs

robhoes pushed a commit that referenced this pull request Sep 14, 2021

Merge pull request #7 from djs55/update

6611bd9

Update following ocamlfind rename to 'xen-gnt'

robhoes pushed a commit that referenced this pull request Sep 14, 2021

Merge pull request #7 from johnelse/travis

e8660b7

Add travis integration

psafont pushed a commit to psafont/xen-api that referenced this pull request Sep 20, 2021

Merge pull request xapi-project#7 from akshayramani/SCTX-1665

5632036

SCTX-1766: Configurable timeout for stunnel

robhoes pushed a commit to robhoes/xen-api that referenced this pull request Sep 21, 2021

Merge pull request xapi-project#7 from djs55/separate-sparse_dd

69c7d45

Separate sparse dd

robhoes pushed a commit to robhoes/xen-api that referenced this pull request Sep 21, 2021

Merge pull request xapi-project#7 from euanh/master

9652e5d

sparse_dd: Fix references to Vhd_lwt.File.use_unbuffered

robhoes pushed a commit to robhoes/xen-api that referenced this pull request Sep 21, 2021

Merge pull request xapi-project#7 from djs55/CP-13838

24e7b42

Fix bug parsing `tap-ctl list` when filenames contain spaces

psafont pushed a commit to psafont/xen-api that referenced this pull request Oct 19, 2021

Merge pull request xapi-project#7 from djs55/ipv4-monitor

747dc66

Report IPv4 addresses as well as IPV6 addresses.

robhoes pushed a commit to robhoes/xen-api that referenced this pull request Jan 10, 2022

Merge pull request xapi-project#7 from mseri/master

41cfb49

Use upstream ocaml-base64, drop custom implementation

psafont pushed a commit to psafont/xen-api that referenced this pull request Jan 10, 2022

Merge pull request xapi-project#7 from gaborigloi/add_gitarchiveinfo

e3453bc

git: Add metadata to the result of `git archive`

edwintorok added a commit to edwintorok/xen-api that referenced this pull request Jan 10, 2022

Merge pull request xapi-project#7 from rosslagerwall/CA-341597

f94a275

CA-341597: Raise the open fd limit

robhoes pushed a commit to robhoes/xen-api that referenced this pull request Apr 7, 2022

Merge pull request xapi-project#7 from johnelse/cp-8359

ed04edb

CP-8359: Add check for subsystem device IDs

psafont pushed a commit to psafont/xen-api that referenced this pull request Mar 17, 2023

Merge pull request xapi-project#7 from mseri/master

8b5f825

ezxenstore: update for safe-string

edwintorok pushed a commit to edwintorok/xen-api that referenced this pull request Jan 22, 2024

Merge pull request xapi-project#7 from djs55/portable-osx

f4affb0

Fix build on OS X

edwintorok pushed a commit to edwintorok/xen-api that referenced this pull request Jan 22, 2024

Merge pull request xapi-project#7 from jonludlam/smapiv3

82e61f9

Use Stdext 2.0.0

edwintorok pushed a commit to edwintorok/xen-api that referenced this pull request Sep 24, 2024

Merge pull request xapi-project#7 from djs55/release.0.3

06b9fa1

Release 0.3

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CP-2093: "Host internal management network" support #7

CP-2093: "Host internal management network" support #7

djs55 commented Feb 3, 2011

djs55 commented Feb 9, 2011

CP-2093: "Host internal management network" support #7

CP-2093: "Host internal management network" support #7

Conversation

djs55 commented Feb 3, 2011

djs55 commented Feb 9, 2011