Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Makefile.xcp that creates RPMs. #1

Closed
wants to merge 3 commits into from

Conversation

magthe
Copy link

@magthe magthe commented Dec 16, 2010

No description provided.

@matthiasgoergens
Copy link
Contributor

Someone should review the top-three commits (I am too tired). The lower ones (Adding the .gitginore and the css-file) seem harmless.

@xen-git
Copy link
Contributor

xen-git commented Apr 5, 2011

Failed to merge and build magthe/xen-api@dd19cc7 with ac556ef.

Failed when executing:
git merge dd19cc7
Error log:
requesting all changes
adding changesets
adding manifests
adding file changes
added 1677 changesets with 3604 changes to 636 files
updating to branch default
180 files updated, 0 files merged, 0 files removed, 0 files unresolved
ln -sf /usr/groups/xen/carbon/trunk/latest/manifest /local/builds/build-xen-git.hg/obj/manifest.input
======> 1302018223 api: cloning xen-api.git from git@github.com:xen-git
======> 1302018223 api: WARNING - no personal fork of xen-api. Cloning from git://hg.uk.xensource.com/git instead.
Initialized empty Git repository in /local/builds/build-xen-git.hg/myrepos/xen-api/.git/
======> 1302018239 api: xen-api placed into myrepos
Auto-merging ocaml/idl/OMakefile
CONFLICT (content): Merge conflict in ocaml/idl/OMakefile
Auto-merging ocaml/lvhdrt/OMakefile
CONFLICT (content): Merge conflict in ocaml/lvhdrt/OMakefile
Auto-merging xapi.spec
CONFLICT (content): Merge conflict in xapi.spec
Automatic merge failed; fix conflicts and then commit the result.

@xen-git
Copy link
Contributor

xen-git commented Apr 5, 2011

Build succeeded. Can merge magthe/xen-api@c264ab1 with ac556ef.

@xen-git
Copy link
Contributor

xen-git commented Apr 8, 2011

Build succeeded. Can merge magthe/xen-api@c264ab1 with 486794e.

if test -f $(JQUERY); then \
cp $(JQUERY) $(SDK)/docs/html; \
else; \
(cd $(SDK)/docs/html; wget http://jqueryjs.googlecode.com/files/jquery-1.1.3.1.pack.js); \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it good to download missing files directly from the build system? This might hide the fact if the files were really missed.

If it's for built outside our building system (e.g. XCP), I guess JonL must have already made some changes (e.g. on $CARBON_DISTFILES) to his Makefile variants in XCP branch.

The same for the next patch hunk.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you see the current build system assumes that the JQuery file is available from the filesystem (which AFAIK only holds true when building at Citrix).

Downloading the file during build is probably not so good, but it's arguably better than relying on the build happening at Citrix :-)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I said, it's probably already addressed/generlized in JonL's XCP building scripts which will eventually merge or substitute this one.

I would suggest to just leave this part as it is for now, or to issue an warning message to tell the user to define $CARBON_DISTFILES him/herself and place jquery (and possibly many other dependencies as well) in the right place.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then I'll just close the pull request completely and we'll wait for Jon's merge instead.

@jonludlam
Copy link
Contributor

We should remove jquery entirely. It's no longer needed.

@xen-git
Copy link
Contributor

xen-git commented Apr 8, 2011

Build succeeded. Can merge magthe/xen-api@c264ab1 with fb197f0.

@xen-git
Copy link
Contributor

xen-git commented Apr 8, 2011

Build succeeded. Can merge magthe/xen-api@c264ab1 with 2d3391e.

@magthe magthe closed this Apr 8, 2011
sharady pushed a commit to sharady/xen-api that referenced this pull request May 11, 2015
sharady pushed a commit to sharady/xen-api that referenced this pull request May 11, 2015
koushikcgit pushed a commit to koushikcgit/xen-api that referenced this pull request Aug 7, 2015
install missing qemu-vif-script
lippirk referenced this pull request in lippirk/xen-api Jul 20, 2020
kc284 added a commit to kc284/xen-api that referenced this pull request Mar 11, 2021
CP-5169: Updated repository name. Replaced hg_loc with git_loc so the component can be cuilt correctly
robhoes pushed a commit that referenced this pull request Jul 19, 2021
Functorize, add unit tests
robhoes pushed a commit that referenced this pull request Jul 19, 2021
Update to use rpclib rather than a bespoke xmlrpc lib
robhoes pushed a commit that referenced this pull request Sep 14, 2021
CA-111132: Make xcp-rrdd-iostat handle empty VBDs
robhoes pushed a commit that referenced this pull request Sep 14, 2021
Switch order of the data source/owner tuple
robhoes pushed a commit that referenced this pull request Sep 14, 2021
robhoes pushed a commit that referenced this pull request Sep 14, 2021
rrddump: Update for stdext 2.0.0
robhoes pushed a commit that referenced this pull request Sep 21, 2021
robhoes pushed a commit that referenced this pull request Sep 22, 2021
Add all the standard files.
robhoes pushed a commit that referenced this pull request Oct 21, 2021
Use Api_messages to get message priority
robhoes pushed a commit that referenced this pull request Jan 10, 2022
edwintorok referenced this pull request in edwintorok/xen-api Jan 10, 2022
CA-302981, CP-30032: Timeout on XAPI calls, and sandbox varstore-rm
robhoes pushed a commit that referenced this pull request Jan 10, 2022
Optionally use the message switch
robhoes pushed a commit that referenced this pull request Jan 10, 2022
psafont pushed a commit that referenced this pull request Apr 11, 2023
lib/xenstore: preserve logging from Xs_client_unix
edwintorok added a commit that referenced this pull request Oct 12, 2023
This enables PAM to be used in multithreaded mode (currently XAPI has a global lock around auth).

Using an off-cpu flamegraph I identified that concurrent PAM calls are slow due to a call to `sleep(1)`.
`pam_authenticate` calls `crypt_r` which calls `NSSLOW_Init` which on first use will try to initialize the just `dlopen`-ed library.
If it encounters a race condition it does a `sleep(1)`. This race condition can be quite reliably reproduced when performing a lot of PAM authentications from multiple threads in parallel.

GDB can also be used to confirm this by putting a breakpoint on `sleep`:
```
  #0  __sleep (seconds=seconds@entry=1) at ../sysdeps/unix/sysv/linux/sleep.c:42
  #1  0x00007ffff1548e22 in freebl_RunLoaderOnce () at lowhash_vector.c:122
  #2  0x00007ffff1548f31 in freebl_InitVector () at lowhash_vector.c:131
  #3  NSSLOW_Init () at lowhash_vector.c:148
  #4  0x00007ffff1b8f09a in __sha512_crypt_r (key=key@entry=0x7fffd8005a60 "pamtest-edvint", salt=0x7ffff31e17b8 "dIJbsXKc0",
  #5  0x00007ffff1b8d070 in __crypt_r (key=key@entry=0x7fffd8005a60 "pamtest-edvint", salt=<optimized out>,
  #6  0x00007ffff1dc9abc in verify_pwd_hash (p=p@entry=0x7fffd8005a60 "pamtest-edvint", hash=<optimized out>, nullok=nullok@entry=0) at passverify.c:111
  #7  0x00007ffff1dc9139 in _unix_verify_password (pamh=pamh@entry=0x7fffd8002910, name=0x7fffd8002ab0 "pamtest-edvint", p=0x7fffd8005a60 "pamtest-edvint", ctrl=ctrl@entry=8389156) at support.c:777
  #8  0x00007ffff1dc6556 in pam_sm_authenticate (pamh=0x7fffd8002910, flags=<optimized out>, argc=<optimized out>, argv=<optimized out>) at pam_unix_auth.c:178
  #9  0x00007ffff7bcef1a in _pam_dispatch_aux (use_cached_chain=<optimized out>, resumed=<optimized out>, h=<optimized out>, flags=1, pamh=0x7fffd8002910) at pam_dispatch.c:110
  #10 _pam_dispatch (pamh=pamh@entry=0x7fffd8002910, flags=1, choice=choice@entry=1) at pam_dispatch.c:426
  #11 0x00007ffff7bce7e0 in pam_authenticate (pamh=0x7fffd8002910, flags=flags@entry=1) at pam_auth.c:34
  #12 0x00000000005ae567 in XA_mh_authorize (username=username@entry=0x7fffd80028d0 "pamtest-edvint", password=password@entry=0x7fffd80028f0 "pamtest-edvint", error=error@entry=0x7ffff31e1be8) at xa_auth.c:83
  #13 0x00000000005adf20 in stub_XA_mh_authorize (username=<optimized out>, password=<optimized out>) at xa_auth_stubs.c:42
```

`pam_start` and `pam_end` doesn't help here, because on `pam_end` the library is `dlclose`-ed, so on next `pam_authenticate` it will have to go through the initialization code again.
(This initialization code would've belonged into `pam_start`, not `pam_authenticate`, but there are several layers here including a call to `crypt_r`).
Upstream has fixed this problem >5 years ago by switching to libxcrypt instead.

Signed-off-by: Edwin Török <edwin.torok@cloud.com>
psafont pushed a commit that referenced this pull request Jan 31, 2024
psafont pushed a commit that referenced this pull request Jan 31, 2024
psafont pushed a commit that referenced this pull request Jan 31, 2024
CA-107919: Reduce memory usage when writing large RRDs.
lindig pushed a commit to lindig/xen-api that referenced this pull request Feb 28, 2024
Backport of 3b52b72

This enables PAM to be used in multithreaded mode (currently XAPI has a global lock around auth).

Using an off-cpu flamegraph I identified that concurrent PAM calls are slow due to a call to `sleep(1)`.
`pam_authenticate` calls `crypt_r` which calls `NSSLOW_Init` which on first use will try to initialize the just `dlopen`-ed library.
If it encounters a race condition it does a `sleep(1)`. This race condition can be quite reliably reproduced when performing a lot of PAM authentications from multiple threads in parallel.

GDB can also be used to confirm this by putting a breakpoint on `sleep`:
```
  #0  __sleep (seconds=seconds@entry=1) at ../sysdeps/unix/sysv/linux/sleep.c:42
  xapi-project#1  0x00007ffff1548e22 in freebl_RunLoaderOnce () at lowhash_vector.c:122
  xapi-project#2  0x00007ffff1548f31 in freebl_InitVector () at lowhash_vector.c:131
  xapi-project#3  NSSLOW_Init () at lowhash_vector.c:148
  xapi-project#4  0x00007ffff1b8f09a in __sha512_crypt_r (key=key@entry=0x7fffd8005a60 "pamtest-edvint", salt=0x7ffff31e17b8 "dIJbsXKc0",
  xapi-project#5  0x00007ffff1b8d070 in __crypt_r (key=key@entry=0x7fffd8005a60 "pamtest-edvint", salt=<optimized out>,
  xapi-project#6  0x00007ffff1dc9abc in verify_pwd_hash (p=p@entry=0x7fffd8005a60 "pamtest-edvint", hash=<optimized out>, nullok=nullok@entry=0) at passverify.c:111
  xapi-project#7  0x00007ffff1dc9139 in _unix_verify_password (pamh=pamh@entry=0x7fffd8002910, name=0x7fffd8002ab0 "pamtest-edvint", p=0x7fffd8005a60 "pamtest-edvint", ctrl=ctrl@entry=8389156) at support.c:777
  xapi-project#8  0x00007ffff1dc6556 in pam_sm_authenticate (pamh=0x7fffd8002910, flags=<optimized out>, argc=<optimized out>, argv=<optimized out>) at pam_unix_auth.c:178
  xapi-project#9  0x00007ffff7bcef1a in _pam_dispatch_aux (use_cached_chain=<optimized out>, resumed=<optimized out>, h=<optimized out>, flags=1, pamh=0x7fffd8002910) at pam_dispatch.c:110
  xapi-project#10 _pam_dispatch (pamh=pamh@entry=0x7fffd8002910, flags=1, choice=choice@entry=1) at pam_dispatch.c:426
  xapi-project#11 0x00007ffff7bce7e0 in pam_authenticate (pamh=0x7fffd8002910, flags=flags@entry=1) at pam_auth.c:34
  xapi-project#12 0x00000000005ae567 in XA_mh_authorize (username=username@entry=0x7fffd80028d0 "pamtest-edvint", password=password@entry=0x7fffd80028f0 "pamtest-edvint", error=error@entry=0x7ffff31e1be8) at xa_auth.c:83
  xapi-project#13 0x00000000005adf20 in stub_XA_mh_authorize (username=<optimized out>, password=<optimized out>) at xa_auth_stubs.c:42
```

`pam_start` and `pam_end` doesn't help here, because on `pam_end` the library is `dlclose`-ed, so on next `pam_authenticate` it will have to go through the initialization code again.
(This initialization code would've belonged into `pam_start`, not `pam_authenticate`, but there are several layers here including a call to `crypt_r`).
Upstream has fixed this problem >5 years ago by switching to libxcrypt instead.

Signed-off-by: Edwin Török <edwin.torok@cloud.com>
Signed-off-by: Christian Lindig <christian.lindig@cloud.com>
lindig pushed a commit that referenced this pull request Feb 29, 2024
Backport of 3b52b72

This enables PAM to be used in multithreaded mode (currently XAPI has a global lock around auth).

Using an off-cpu flamegraph I identified that concurrent PAM calls are slow due to a call to `sleep(1)`.
`pam_authenticate` calls `crypt_r` which calls `NSSLOW_Init` which on first use will try to initialize the just `dlopen`-ed library.
If it encounters a race condition it does a `sleep(1)`. This race condition can be quite reliably reproduced when performing a lot of PAM authentications from multiple threads in parallel.

GDB can also be used to confirm this by putting a breakpoint on `sleep`:
```
  #0  __sleep (seconds=seconds@entry=1) at ../sysdeps/unix/sysv/linux/sleep.c:42
  #1  0x00007ffff1548e22 in freebl_RunLoaderOnce () at lowhash_vector.c:122
  #2  0x00007ffff1548f31 in freebl_InitVector () at lowhash_vector.c:131
  #3  NSSLOW_Init () at lowhash_vector.c:148
  #4  0x00007ffff1b8f09a in __sha512_crypt_r (key=key@entry=0x7fffd8005a60 "pamtest-edvint", salt=0x7ffff31e17b8 "dIJbsXKc0",
  #5  0x00007ffff1b8d070 in __crypt_r (key=key@entry=0x7fffd8005a60 "pamtest-edvint", salt=<optimized out>,
  #6  0x00007ffff1dc9abc in verify_pwd_hash (p=p@entry=0x7fffd8005a60 "pamtest-edvint", hash=<optimized out>, nullok=nullok@entry=0) at passverify.c:111
  #7  0x00007ffff1dc9139 in _unix_verify_password (pamh=pamh@entry=0x7fffd8002910, name=0x7fffd8002ab0 "pamtest-edvint", p=0x7fffd8005a60 "pamtest-edvint", ctrl=ctrl@entry=8389156) at support.c:777
  #8  0x00007ffff1dc6556 in pam_sm_authenticate (pamh=0x7fffd8002910, flags=<optimized out>, argc=<optimized out>, argv=<optimized out>) at pam_unix_auth.c:178
  #9  0x00007ffff7bcef1a in _pam_dispatch_aux (use_cached_chain=<optimized out>, resumed=<optimized out>, h=<optimized out>, flags=1, pamh=0x7fffd8002910) at pam_dispatch.c:110
  #10 _pam_dispatch (pamh=pamh@entry=0x7fffd8002910, flags=1, choice=choice@entry=1) at pam_dispatch.c:426
  #11 0x00007ffff7bce7e0 in pam_authenticate (pamh=0x7fffd8002910, flags=flags@entry=1) at pam_auth.c:34
  #12 0x00000000005ae567 in XA_mh_authorize (username=username@entry=0x7fffd80028d0 "pamtest-edvint", password=password@entry=0x7fffd80028f0 "pamtest-edvint", error=error@entry=0x7ffff31e1be8) at xa_auth.c:83
  #13 0x00000000005adf20 in stub_XA_mh_authorize (username=<optimized out>, password=<optimized out>) at xa_auth_stubs.c:42
```

`pam_start` and `pam_end` doesn't help here, because on `pam_end` the library is `dlclose`-ed, so on next `pam_authenticate` it will have to go through the initialization code again.
(This initialization code would've belonged into `pam_start`, not `pam_authenticate`, but there are several layers here including a call to `crypt_r`).
Upstream has fixed this problem >5 years ago by switching to libxcrypt instead.

Signed-off-by: Edwin Török <edwin.torok@cloud.com>
Signed-off-by: Christian Lindig <christian.lindig@cloud.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants