[BUG] Unstopped thread when using Kerberos authentication

[jansohn]

Driver version

7.0.0

SQL Server version

Microsoft SQL Server 2016 (SP2) (KB4052908) - 13.0.5026.0 (X64) 
	Mar 18 2018 09:11:49 
	Copyright (c) Microsoft Corporation
	Standard Edition (64-bit) on Windows Server 2016 Standard 10.0 <X64> (Build 14393: ) (Hypervisor)

Client Operating System

Windows 10

JAVA/JVM version

java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

Problem description

1. Expected behaviour: All threads should be stopped when shutting down the application in Tomcat

2. Actual behaviour: One thread is still running after shutdown and produces a possible memory leak

3. Error message/stack trace:

   WARNING: The web application [test] appears to have started a thread named [Thread-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
    sun.net.dns.ResolverConfigurationImpl.notifyAddrChange0(Native Method)
    sun.net.dns.ResolverConfigurationImpl$AddressChangeListener.run(ResolverConfigurationImpl.java:144)
   

4. I debugged the web application and found out that the thread is started in a static initialization block in sun.net.dns.ResolverConfigurationImpl (see stack trace below):

   static {
     AccessController.doPrivileged(new PrivilegedAction() {
        public Void run() {
           System.loadLibrary("net");
           return null;
        }
     });
     init0();
     ResolverConfigurationImpl.AddressChangeListener var0 = new ResolverConfigurationImpl.AddressChangeListener();
     var0.setDaemon(true);
     var0.start();
   }
   

   Daemon Thread [localhost-startStop-2] (Class load: ResolverConfigurationImpl)	
   	owns: Object  (id=91)	
   	owns: Object  (id=92)	
   	owns: EntityManagerFactoryDelegate  (id=93)	
   	owns: ApplicationContextListener  (id=94)	
   	owns: StandardContext  (id=95)	
   	ResolverConfiguration.open() line: 56 [local variables unavailable]	
   	DnsContextFactory.serversForUrls(DnsUrl[]) line: 149	
   	DnsContextFactory.getContext(String, DnsUrl[], Hashtable<?,?>) line: 81	
   	dnsURLContext.getRootURLContext(String, Hashtable<?,?>) line: 71	
   	dnsURLContext(GenericURLDirContext).getAttributes(String, String[]) line: 100	
   	KrbServiceLocator.lambda$getKerberosService$1(Context, String) line: 168	
   	1747531283.run() line: not available	
   	AccessController.doPrivileged(PrivilegedExceptionAction<T>, AccessControlContext) line: not available [native method]	
   	AccessController.doPrivileged(PrivilegedExceptionAction<T>, AccessControlContext, Permission...) line: 713	
   	KrbServiceLocator.getKerberosService(String, String) line: 166	
   	Config.getKDCFromDNS(String) line: 1171	
   	Config.getKDCList(String) line: 1057	
   	NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]	
   	NativeMethodAccessorImpl.invoke(Object, Object[]) line: 62	
   	DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
   	Method.invoke(Object, Object...) line: 498	
   	KerbAuthentication$2.isRealmValid(String) line: 301	
   	KerbAuthentication.getRealmValidator(String) line: 312	
   	KerbAuthentication.enrichSpnWithRealm(String, boolean) line: 254	
   	KerbAuthentication.<init>(SQLServerConnection, String, int) line: 231	
   	SQLServerConnection.logon(SQLServerConnection$LogonCommand) line: 3557	
   	SQLServerConnection.access$000(SQLServerConnection, SQLServerConnection$LogonCommand) line: 81	
   	SQLServerConnection$LogonCommand.doExecute() line: 3541	
   	SQLServerConnection$LogonCommand(TDSCommand).execute(TDSWriter, TDSReader) line: 7240	
   	SQLServerConnection.executeCommand(TDSCommand) line: 2869	
   	SQLServerConnection.connectHelper(ServerPortPlaceHolder, int, int, boolean, boolean, boolean, int) line: 2395	
   	SQLServerConnection.login(String, String, int, String, FailoverInfo, int, long) line: 2042	
   	SQLServerConnection.connectInternal(Properties, SQLServerPooledConnection) line: 1889	
   	SQLServerConnection.connect(Properties, SQLServerPooledConnection) line: 1120	
   	SQLServerDriver.connect(String, Properties) line: 700	
   	DriverManager.getConnection(String, Properties, Class<?>) line: 664	
   	DriverManager.getConnection(String, Properties) line: 208	
   	DefaultConnector.connect(Properties, Session) line: 98	
   	DatabaseLogin(DatasourceLogin).connectToDatasource(Accessor, Session) line: 162	
   	ServerSession(DatabaseSessionImpl).setOrDetectDatasource(boolean) line: 207	
   	ServerSession(DatabaseSessionImpl).loginAndDetectDatasource() line: 760	
   	EntityManagerFactoryProvider.login(DatabaseSessionImpl, Map, boolean) line: 265	
   	EntityManagerSetupImpl.deploy(ClassLoader, Map) line: 731	
   	EntityManagerFactoryDelegate.getAbstractSession() line: 205	
   	EntityManagerFactoryDelegate.createEntityManagerImpl(Map, SynchronizationType) line: 305	
   	EntityManagerFactoryImpl.createEntityManagerImpl(Map, SynchronizationType) line: 337	
   	EntityManagerFactoryImpl.createEntityManager() line: 303	
   	ApplicationContextListener.createEntityManager() line: 229	
   	[...]
   

The thread itself is started by a Java core class but my question is if we can prevent the thread from starting in the first place (stopping it correctly seems to be out of the question as there is no reference available).
I think others using Kerberos for authentication (with username/password) should experience this problem, too. It would be nice to know if this is a general problem or if I'm doing something wrong.
Currently the code is embedded in a bigger web application (with JPA and others) but I'll try to strip it down to a minimal example if necessary.

[cheenamalhotra]

Hi @jansohn

We have removed this piece of code in PR #839 and the fix was part of v7.1.2-preview. Would like you to test and confirm if the new driver version resolves your problem?

Thanks,
Cheena

[jansohn]

I just tested it with 7.1.2.jre8-preview and 7.1.3.jre8-preview. The problem persists, there are no notable differences.

[cheenamalhotra]

Hi @jansohn

Can you confirm if you see the exact same error as in above stacktrace or some other by using newer version of the driver? Please make sure your application doesn't have multiple driver versions referenced as according to the stacktrace, the code path in question (Reflective access to Config.getKDCList()) no longer exists as I mentioned in the PR link above.

Please verify in a standalone Java App and do share us repro it you see the issue.

[jansohn]

I'll check this next week!

[jansohn]

Updated stacktrace with 7.1.2.jre8-preview:

Daemon Thread [localhost-startStop-1] (Class load: ResolverConfigurationImpl)	
	owns: Object  (id=93)	
	owns: Object  (id=94)	
	owns: EntityManagerFactoryDelegate  (id=95)	
	owns: ApplicationContextListener  (id=96)	
	owns: StandardContext  (id=97)	
	ResolverConfiguration.open() line: 56 [local variables unavailable]	
	DnsContextFactory.serversForUrls(DnsUrl[]) line: 149	
	DnsContextFactory.getContext(String, DnsUrl[], Hashtable<?,?>) line: 81	
	DnsContextFactory.urlToContext(String, Hashtable<?,?>) line: 120	
	DnsContextFactory.getInitialContext(Hashtable<?,?>) line: 64	
	NamingManager.getInitialContext(Hashtable<?,?>) line: 684	
	InitialDirContext(InitialContext).getDefaultInitCtx() line: 313	
	InitialDirContext(InitialContext).init(Hashtable<?,?>) line: 244	
	InitialDirContext(InitialContext).<init>(Hashtable<?,?>) line: 216	
	InitialDirContext.<init>(Hashtable<?,?>) line: 101	
	DNSUtilities.findSrvRecords(String) line: 43	
	DNSKerberosLocator.isRealmValid(String) line: 38	
	KerbAuthentication$2.isRealmValid(String) line: 293	
	KerbAuthentication.findRealmFromHostname(KerbAuthentication$RealmValidator, String) line: 321	
	KerbAuthentication.enrichSpnWithRealm(String, boolean) line: 255	
	KerbAuthentication.<init>(SQLServerConnection, String, int) line: 231	
	SQLServerConnection.logon(SQLServerConnection$LogonCommand) line: 3557	
	SQLServerConnection.access$000(SQLServerConnection, SQLServerConnection$LogonCommand) line: 81	
	SQLServerConnection$LogonCommand.doExecute() line: 3541	
	SQLServerConnection$LogonCommand(TDSCommand).execute(TDSWriter, TDSReader) line: 7233	
	SQLServerConnection.executeCommand(TDSCommand) line: 2869	
	SQLServerConnection.connectHelper(ServerPortPlaceHolder, int, int, boolean, boolean, boolean, int) line: 2395	
	SQLServerConnection.login(String, String, int, String, FailoverInfo, int, long) line: 2042	
	SQLServerConnection.connectInternal(Properties, SQLServerPooledConnection) line: 1889	
	SQLServerConnection.connect(Properties, SQLServerPooledConnection) line: 1120	
	SQLServerDriver.connect(String, Properties) line: 700	
	DriverManager.getConnection(String, Properties, Class<?>) line: 664	
	DriverManager.getConnection(String, Properties) line: 208	
	DefaultConnector.connect(Properties, Session) line: 98	
	DatabaseLogin(DatasourceLogin).connectToDatasource(Accessor, Session) line: 162	
	ServerSession(DatabaseSessionImpl).setOrDetectDatasource(boolean) line: 207	
	ServerSession(DatabaseSessionImpl).loginAndDetectDatasource() line: 760	
	EntityManagerFactoryProvider.login(DatabaseSessionImpl, Map, boolean) line: 265	
	EntityManagerSetupImpl.deploy(ClassLoader, Map) line: 731	
	EntityManagerFactoryDelegate.getAbstractSession() line: 205	
	EntityManagerFactoryDelegate.createEntityManagerImpl(Map, SynchronizationType) line: 305	
	EntityManagerFactoryImpl.createEntityManagerImpl(Map, SynchronizationType) line: 337	
	EntityManagerFactoryImpl.createEntityManager() line: 303	
	ApplicationContextListener.createEntityManager() line: 229	
	[...]

[jansohn]

Reproducer: https://github.com/jansohn/kerberos-timer

[peterbae]

Hi @jansohn, thanks for your investigation. I can also see that a daemon thread has been left behind with your reproduction code. We'll start looking into this issue, and will let you know when we have an update.

[ulvii]

Hi @jansohn,

That particular piece of code was introduced in PR #40, to automatically append the realm name to the SPN for Kerberos cross-realm authentication (when server's realm is different than the default realm). It makes use of the JNDI DNS, which spins up a daemon thread in the background, but does not provide an API to release it. I agree that leaving a thread running is an issue and did some research to find alternatives to JNDI DNS, but looks like there is no other JDK library that would let us achieve the same functionality. There are however some third party libraries that provide API to retrieve DNS records, but it is highly unlikely that we will introduce a new dependency to the driver, mainly because there is already a connection property serverSpn that lets the users specify realm name for the scenario where the default realm in the Kerberos configuration is different than the realm of the server.

All of that being said, we are currently leaning toward reverting PR #40 and asking users to explicitly specify realm name in serverSpn property. We are also definitely open to suggestions that would let us keep the functionality introduced in PR #40 and also fix the issue with a thread being left behind without introducing a new dependency to the driver.

I am adding @pierresouchay - the author of the PR #40 to the thread to get his opinion on this issue too.

[pierresouchay]

Hello @ulvii and @jansohn,

I was unaware of such Thread being initialized when using native DNS resolutions.

I have a few remarks:

• are you sure the Thead (a daemon one) is leaked when classloader is released (I mean, if you start/stop twice the container, does 2 threads are kept)? Since this thread is spawn from JRE classes, I suspect it would be re-used by other class loader at next resolution, don't you think? In that case, it means that next DNS resolution would reuse it, so it might not be a "real" problem (not sure about that). I am also very surprised as I used it lots of time in various Application servers in the past without troubles (I mean in the 2000s... I am old :-) => I am very doubtful this is a real world issue (we are using this in dozens of projects in my company for 3 years). Many programs statically allocate stuff at startup and release at the end. As long a single resource is used, this is not a big issue IMHO, so unless start/stop app twice creates 2 thread, I am not sure it is a real life problem. Especially when the thread is for a so common thing as DNS resolution. (BTW, is does not leak classes since using only classes of JRE, so not gonna cause any class leak neither)
• I think this is a great feature to have within the Driver for several reasons: computing the REALM is not easy and took me months. Removing this feature will break existing users relying on this feature (such as my whole company where only IP is provided for Database and the driver is solving the whole REALM). Computing the SPN is hard, especially with multiple Realms, so that's a very precious part of the code that allow to do some Kerberos in real life (how many developers know what is a SPN BTW?)
• In order to not have any dependency over a third party library, I actually used what was in the JRE, but other libraries to exist to have similar result (as I did in my own first implementation, where I used org.xbill.DNS.* for things such as having access to TCP DNS when needed. Such library has very low dependencies (aka NONE) and would be able to do the job.

I really think removing this feature would be a shame: Kerberos is complicated (look at the workaround I had to do to make it work on multiple JVMs) and I spent lots of time on fixing it, and removing the feature would break some users code.

[ulvii]

Hi @pierresouchay,

Thank you for the detailed explanation. You are correct to say that the thread gets reused when reloading the driver, so this does not seem like a memory leak, but the ideal behavior would be to not have the thread left behind. Another reason why we were thinking of reverting the PR was because there are lots of workarounds in the code as you mentioned and it is not straightforward (sometimes nearly impossible) to maintain them, given the limitations and changes from different JDK vendors/versions.

@jansohn, do you have any particular issue in your application that's caused by the daemon thread or it is just a warning reported in Tomcat?

[pierresouchay]

@ulvii The code is complicated for some reasons: JVM support for Kerberos is not very consistent.

Some JVMs (SUN/Oracle) do parse the krb5.conf using the class sun.security.krb5.Config, but some are parsing it wrong, especially on Windows where some versions of JVM do think there is one realm to rule them all (aka Active Directory), thus those JVMs do not resolve realms properly (I had to look at JRE source code to see that), thus, I did use feature testing to ensure the JVM is working accordingly ( https://github.com/Microsoft/mssql-jdbc/pull/40/files#diff-579fbf11bb1ece322ce26ed03202e7d1R357 ), so it is unlikely that this code breaks (it will fallback on DNS in that case).

DNS is cool to solve REALMs, unfortunately, it is not very often properly configured, thus, the 2nd method (using DNS) is less likely work since it expects lots of configuration to be in place. Moreover, the file krb5.conf can contain some overrides regarding realms that might be very important in some cases. I did put it as a fallback for:

• Windows JRE with bad Kerberos REALM resolution (most of them), see Added automatic detection of REALM in SPN needed for Cross Domain authentication. #40 (comment)
• Other JVMs such as IBM or other that do not have sun.security.krb5.Config

While you might choose to remove reflection to simplify code (in that case, users will have to configure all of their DNS records properly), but removing the feature is IMHO a huge mistake, because nobody understand Kerberos completely and having the logic in the driver is what make it work.

In my company, we are doing cross REALMs calls using it across 9 different REALMs with different levels of trust (mixing Hadoop and AD domains) for years, it had been impossible before that.

So, yes, this is not fun, but I think the driver HAS to hide the gory details, otherwise, what is the point?