Multiple keys and delegation

[adfel70]

Our Apache server hosts wsgi application (mod_wsgi) and uses mod_gssapi for authentication. moreover, the application uses delegation to pass the authenticated user to a backend service as follow:

client request --> HTTP/apache_1@OUR_REALM --> HTTP/backend_server@OUR_REALM

This works perfect, but now we need to add load balancer in front of the Apache server:

                                            -->  HTTP/apache_1@OUR_REALM --> HTTP/backend_server@OUR_REALM
client request -->  HTTP/balancer@OUR_REALM |  
                                            -->  HTTP/apache_2@OUR_REALM --> HTTP/backend_server@OUR_REALM

blancer server just forwards requests - no security configuration whatsoever and works as expected.

To make this work, we did the following:

1. created new principal HTTP/balancer@OUR_REALM with constrained delegation to the backend service.
2. Added the new principal to each of the existing keytabs of apache_1 and apache_2 servers:
   On server apache_1:

		KVNO 	Principal
		---- 	-------------------
			1 	HTTP/apache_1@OUR_REALM
			1	HTTP/balancer@OUR_REALM

**On server apache_2:**

		KVNO 	Principal
		---- 	-------------------
			1 	HTTP/apache_2@OUR_REALM
			1	HTTP/balancer@OUR_REALM

While testing our configuration we issued http requests to our balancer and got the following error in Apache's error log:
GSSError:(('Unspecified GSS failure. Minor code may provide more information', 851968), ('Matching credential not found (filename: /var/run/httpd/krbcache/user_principal@OUR_REALM)', -1765328243))

Investigating the user's ccache using klist -c /var/run/httpd/krbcache/user_principal@OUR_REALM showed (start and expiration columns are omitted):

# example from server apache_1
Ticket cache: FILE:/var/run/httpd/krbcache/user_principal@OUR_REALM
Default principal: user_principal@OUR_REALM

Service principal
HTTP/balancer@OUR_REALM

krbtgt/OUR_REALM@OUR_REALM
for client HTTP/apache_1@OUR_REALM

--> Its seems that even though we executed requests to the balancer, the delegation ticket was still issued to HTTP/apache_1@OUR_REALM (tgt for client HTTP/apache_1@OUR_REALM). Same behavior was occurred on requests that were forwarded to apache_2 server.

We tried to change the order of the principals inside the keytabs so HTTP/balancer@OUR_REALM will be the first principal:

on apache_1:

KVNO 	Principal
---- 	-------------------
	1	HTTP/balancer@OUR_REALM
	1 	HTTP/apache_1@OUR_REALM

on apache_2:

KVNO 	Principal
---- 	-------------------
	1	HTTP/balancer@OUR_REALM
	1 	HTTP/apache_2@OUR_REALM

And now we got the opposite: requests issued to the balancer server were succeeded while requests to the concrete servers apache_1 or apache_2 were failed. Obviously, this is better than before since we got a working balancer, but still - its preferable to make the concrete servers answering as well (balancer may fail, monitoring and more).

Is there a way making both principals work for each of our balancer servers?

Thanks.

[simo5]

Can you please provide your configuration ?

[simo5]

For multiple keys in a server normally we'd recommend to set:
GssapiAcceptorName {HOSTNAME}

If you r balancer will not change the request from the client at all so that your apache server see that they actually contacted the balancer (this means your apache server is configured with 2 server names, their real one and the the balancer name), then this option should fix the problem for you.

[adfel70]

Thanks @simo5, here is the relevant configuration section:

AuthType GSSAPI
GssapiCredstore keytab:/etc/httpd/apache.keytab
GssapiCredstore client_keytab:/etc/httpd/apache.keytab
GssapiUseS4U2Proxy On
GssapiDelegCcacheDir /var/run/httpd/krbcache
GssapiImpersonate On

Had a filling I could use GssapiAcceptorName {HOSTNAME} but I am getting an error:

GSS ERROR gss_acquire_cred[_from]() failed to get server creds: [Unspecified GSS failure. Minor code may provide more information (No key table entry found matching {HOSTNAME}/@)]

We currently run with mod_gssapi 1.5.1 and I think {HOSTNAME} is only supported from 1.6.0. I'll try to upgrade and see how it works.
Any idea where can I find rpm package?
Thanks again for all you help.

[simo5]

Fedora has 1.6.1+, you can rebuild from Fedora source if you are on a EL platform, they should be compatible.

[adfel70]

Ok, installed version 1.6.3, added GssapiAcceptorName {HOSTNAME} and now I get something weird:

First request always works, but also sets the principal. So, if I executed the request to the balancer and it was forwarded to apache_1, it will only succeed with the balancer principal (requests to apache_1 by its hostname will failed). But if the first request was with apache_1 principal, requests to the balancer that are forwarded to apache_1 will fail. The error is
GSS ERROR gss_acquire_cred[_from]() failed to get server creds: [Unspecified GSS failure. Minor code may provide more information (Principal in credential cache does not match desired name)]
Any idea?

[simo5]

@adfel70 yes I think I know what is going on and it is potentially a bug, or will need some docs to explain how this needs to be set up.

Basically we are initializing a ccache with a TGT in one name, and later when the second attempt comes we try to reinitialize the same ccache but with a different TGT.

What is the default ccache type on your system (you should find it in krb5.conf) ?

I believe this might work with a ccache collection, but if you have a FILE ccache by default it would give this error.

To test you can create a directory like:
mkdir /var/run/httpd/srvcache

And add this to the configuration:
GssapiCredstore ccache:DIR:/var/run/httpd/srvcache

This will instruct mod_auth_gssapi to use a ccache collection of type DIR in /var/run/httpd/srvcache and should hopefully allow you to have multiple credentials valid at the same time.

[adfel70]

Thanks again @simo5 for all your help.
We tried your advice, our current configuration is as follow:

AuthType GSSAPI
GssapiCredstore keytab:/etc/httpd/apache.keytab
GssapiCredstore client_keytab:/etc/httpd/apache.keytab
GssapiCredstore ccache:DIR:/var/run/httpd/srvcache
GssapiUseS4U2Proxy On
GssapiDelegCcacheDir /var/run/httpd/krbcache
GssapiAcceptorName {HOSTNAME}

Problem remains... :(
By the way, if we manually destroy the cache using kdestroy -c /var/run/httpd/srvcache/tkt we can issue request to the other acceptor's name, but then again it becomes the only acceptor.
We could not find the required parameter's name for setting default ccache type in krb5.conf (we are using version 1.15.1 of krb5 library). Do you know how can we verify it is configured properly? (though the ccache dir is indeed being used...)

Do you think this is a bug?

[simo5]

Unfortunately I think this is a bug that will need code changes to be addressed

[simo5]

I think we never tested both GssapiAcceptorName {HOSTNAME} and GssapiUseS4U2Proxy On at the same time, and the interaction between the two is not working well.

However there may be a workaround for you.
Can you try to do the following before starting apache?

As the user apache:
$ KRB5CCNAME=DIR:/var/run/httpd/srvcache
$ kinit -k keytab:/etc/httpd/apache.keytab HTTP/apache_1@OUR_REALM
$ kinit -k keytab:/etc/httpd/apache.keytab HTTP/balancer@OUR_REALM

Then with kilist -A you should see two caches in the collection, one for each principal.
If that is the case and you then start apache are you able to make it work at all ?

[simo5]

@adfel70 if you have a way to try a custom build it would be really nice to know if PR #238 fully fixes your issue.

[adfel70]

I built your branch with the fix and it works perfect! :)

I removed GssapiCredstore ccache:DIR:/var/run/httpd/srvcache as the dir empty. No use of this ccache, is that makes sense?

Thanks a lot

[simo5]

Thanks for the feedback, this is great, yeah the explicit ccache is ignored in this specifica case, instead you will see two new cacches appear in /var/run/httpd/krbcache named accpetor_

These are not delegated ccaches, they are actually server ccaches, one for each hostname you have keys for.
The reason I put files there is that this directory is required for this case, which is the only one that need "support ccaches", and it will avoid adding new configuration options.