HTTP Basic Authentication error (was: Repeating issues with the extension)

[navennn]

Hi,
Up until recently everything worked pretty much fine, but then came some thunderbird and lighting update (thunderbird wouldn't load correctly). I was waiting for a new version with a hope that it'll fix my issues, but unfortunately it did not.

It seems that the extension is sending bad password and/or username. When I go to the EWS webpage I can authenticate just fine, it only asks me for the password once, but in thunderbird it asks me for password three times. Before the latest release it used to show my password sometimes and it was missing half the characters. My password is over 25 characters long now, and it was always showing the first 11 characters.
Perhaps it is asking for a username first, then for the password while the box is saying 'password' all the time?

In ews settings of calendar I tried
username
domain\username
domain/username
username@domain

OS is Linux Mint 18.2 64
Version 52.6.0
User Agent Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0
Exchange Calendar 4.0.0-beta5 true exchangecalendar@extensions.1st-setup.nl
Lightning 5.4.6 true {e2fda1a4-762b-4020-b5ad-a41df1933103}
Manually sort folders 1.1.1 true tbsortfolders@xulforum.org
Provider for Google Calendar 3.3 true {a62ef8ec-5fdc-40c2-873c-223b8a6925cc}
ThunderKeepPlus 0.10.0 true thunderkeepplus@gmail.com

Also right now the folder structure when adding a new calendar keeps loading forever and it never works in the end (Check server and mailbox). This was also an issue in previous version.

In logs I tried to obscure sensitive information, so it says like domain.com, or mycompanyemailaddress@domain.com.
Starting from line 387 is right after I input my password to the box. In case of this log it was done 3 times.

thunderbird.log

[Trim]

Hello,

Thanks for the report. I'll try to explain what I've seen while trying to debug the "password shown in dialog box". To get a password, ExchangeCalendar does this step (I'm writting what I remember, that's the overall idea):

• Check the extension cache to see if the password has already be given by the user
• If not found, check if it is saved in the Thunderbird Password manager
• Finally, ask the password to the user:
  • The extension count the number of fail to authenticate and stop after 3 times (we should add a dialog to explain we reached this count and we stop to ask the password with two options: reset this counter or really stop it now)
  • It creates a dialog asking the password. This dialog box needs to not be the standard one from Thunderbird, because the user need to see: for which domain and for which user the password is required.
  • either the user cancel the password request and the application stop to ask it to the user, or the user provide a password and a new attempt to authenicate is done.

So, yes, you need to provide just a password, maybe there's a bug in these parts:

• the dialog box
• the way the password is given to the Thunderbird HTTP request handler (HTTP channel)

The workaround provided by 4.0.0-beta5 for the issue "password shown in dialog box" fixes the way the password is given to the HTTP channel: Thunderbird chanel API give us a way to update the password simply with channel.password = urlencode(password). That was he way used up to now, but it seems bugus as the password were written inisde the user part of the HTTP request instead of the password part.

So, now we use another way from the channel API: channel.userPass = urlencode(username) + : + urlencode(password). That seemed to work to set correctly the channel username and password as the password isn't shown any more in the dialog.

To summary, I think we could have bugs in these parts (starting with the most probable):

• the dialog box trunk the password
• the channel API is still not working as expected even with the userPass workaround
• the urlencode method used trunk the values

For thoes who are interested to view this codes: HTTP requests are handled in common/exchangeapi/ecExchangeRequest.js and passwords are managed by common/interface/mivExchangeAuthPrompt2/.

There's also two ways to get the password:

• either before creating any HTTP request (so it's directly asked by ecExchnageRequest)
• or by finding an HTTP authenitcation error and trying to fix it (here's the mivExchangeAuthProvider implements call back to get these errors and try to fix them).

That's just an overall summary of the overall process. I need to read your logs (thank you!).

I don't know exactly what are the options provided by the ExchangeCalendar extension, but can you tell us if the extension password cache is enabled (not sure if there's an option) and if your are using the Thunderbird password manager ?

[navennn]

Hi,
Thank you for your reply.

I am not too sure what does the first question mean. Is it a separate cache from the application one? I did a bit of searching in the options and could only find Thunderbird's own cache.
However:
ul0XXXXX username # ls -lh /home/username/.thunderbird/cw4dpk9x.my.name/calendar-data/
total 3,6M
-rw-r--r-- 1 username username 2,0M Jan 30 07:26 cache.sqlite
-rw-r--r-- 1 username username 32K Jan 30 07:26 cache.sqlite-shm
-rw-r--r-- 1 username username 0 Jan 30 07:26 cache.sqlite-wal
-rw-r--r-- 1 username username 224K Jan 26 16:35 deleted.sqlite
-rw-r--r-- 1 username username 1,4M Jan 30 07:10 local.sqlite

ul0XXXXX username # ls -lh /home/username/.thunderbird/cw4dpk9x.my.name/exchange-data/
total 42M
-rw-r--r-- 1 username username 9,3M Jan 29 03:36 254915e8-049b-4cd5-870d-3dc23a0ed4e6.offlineCache.sqlite
-rw-r--r-- 1 username username 1,1M Jan 29 03:48 8d6a67a5-6bf8-47a2-becd-263762c85528.offlineCache.sqlite
-rw-r--r-- 1 username username 31M Jan 26 16:35 f89b711f-c965-4164-9a8f-6a7becc0c9b5.offlineCache.sqlite
drwxr-xr-x 2 username username 4,0K Jan 18 13:27 tmp

Yes, I am using thunderbird password manager and the password does appear there.

[Trim]

For the username + domain format, I know that user@domain is working (that's the way I'm using the extension). To use this way, set this text in the user name exchange settings dialog.

You could use the old (before windows 2003 IIRC) way to format this with domain\user. To use this way, set the user in user field and domain in the domain field of the exchange settings dialog.

The last one caused issues and we so recommand to use the first one. IIRC, the way compatible with pre-2003 servers has some limitations like the username and domain lengths.

I've just read the logs and unfortunately, it don't contains the interesting part (authentication log). Could you try again with a more verbose way for the authentication process (you can find it in the exchange settings in Thunderbird properties dialog) ?

[navennn]

So, I set all the logs to max possible verbosity now

I can't change the username since after clicking 'check server and mailbox' it just hangs as if there was no network connection.

It appears that there is the very same error (same cause?) when authenticating and also when trying to change the username in EWS settings.

1st-setup:2018-0-2 7:54:33.153:mivExchangeAuthPrompt2: asyncPromptAuthNotifyCallback: Already showing a prompt or trying to get the password for URL 'https://cas.mycompany.com/EWS/Exchange.asmx'. Not going to try again until the active one has finished. (_asyncPromptAuthNotifyCallback in mivExchangeAuthPrompt2.js:265)
1st-setup:2018-0-2 7:54:46.360:[RES PN GNM] this._weAreSyncing changed from 'true' to 'false' (set weAreSyncing in mivExchangeCalendar.js:8203)
1st-setup:2018-0-2 7:54:46.364:[PN SCO] this._weAreSyncing changed from 'true' to 'false' (set weAreSyncing in mivExchangeCalendar.js:8203)

(removed log)

[navennn]

Hi, any ideas @Trim ?

[Trim]

Hello,

I've just found these lines in your logs:

1st-setup:2018-0-2 7:52:55.443:mivExchangeAuthPrompt2: asyncPromptAuthNotifyCallback: username=domain\username (_asyncPromptAuthNotifyCallback in mivExchangeAuthPrompt2.js:324)
1st-setup:2018-0-2 7:52:55.443:mivExchangeAuthPrompt2: asyncPromptAuthNotifyCallback: WWW-Authenticate:Negotiate (_asyncPromptAuthNotifyCallback in mivExchangeAuthPrompt2.js:335)
1st-setup:2018-0-2 7:52:55.443:mivExchangeAuthPrompt2: asyncPromptAuthNotifyCallback: WWW-Authenticate:NTLM (_asyncPromptAuthNotifyCallback in mivExchangeAuthPrompt2.js:335)
1st-setup:2018-0-2 7:52:55.443:mivExchangeAuthPrompt2: asyncPromptAuthNotifyCallback: WWW-Authenticate:Basic realm="cas.mycompany.com" (_asyncPromptAuthNotifyCallback in mivExchangeAuthPrompt2.js:335)
1st-setup:2018-0-2 7:52:55.444:mivExchangeAuthPrompt2: asyncPromptAuthNotifyCallback: WWW-Authenticate HTTP response header not found ! (_asyncPromptAuthNotifyCallback in mivExchangeAuthPrompt2.js:345)
1st-setup:2018-0-2 7:52:55.444:mivExchangeAuthPrompt2: asyncPromptAuthNotifyCallback: asking password to user. (_asyncPromptAuthNotifyCallback in mivExchangeAuthPrompt2.js:350)

I see some bugs with this log:

• the Exchange server gives us a realm and the log don't see it (normally, it should write what it found)
• the last line saying there's no WWW-Auhenticate header shows me that there was an exception while reading HTTP headers from the response.

As your Exchange server seems to support NTLM, could you try to follow steps from issue #112 ?

Maybe you'll be able to authenticate with NTLM while we are looking for a fix.

[Trim]

After taking some time, I've found an issue in the way the extension tries to authentication with Basic HTTP authentication while reading realm from HTTP respons headers.

I believed that my Exchange provider were using it, but I only have the NTLM option.
NTLM can work without Kerberos / Active Directory, I was confused due these are all Microsoft technologies, sorry :/

So, I can't test my proposed fixes. Could you try them ?

Either by building code from my Trim/exchangecalendar/fix-reading-realm-from-www-authenticate branch either by downloading my build from my cloud.

[navennn]

Hi,
So, the NTLM workaround worked. When thunderbird actually worked.

I would like to try it, but thunderbird simply will not turn on. It was similar for the older version as well. beta-4, beta-5. I don't know what causes it, it seems to start working after some time, but now I restarted the machine and TB several times and still can't get it to work.
The only method I found to access TB is to fallback to v45 where the extensions aren't compatible ( so EWS is disabled).
For sure it is caused by EWS. If I disable it (falling back to 45 and going back to 52.6.0) thunderbird works.

At the moment:
Thunderbird 52.6
Lightning 5.4
EWS exchange provider exchangecalendar-v4.0.0-beta5_fix-basic-auth
From your download link.

test-thunderbird.log

There's not much log I can attach, I hope this is enough.

#141 seems like the same issue

[Trim]

For your last issue (Thunderbird hangs up), we'll continue discussion in #141

I'm keeping this bug open to ask community to check if the proposed patch works for a user using HTTP Basic authentication.

If someone can confirm this works, it will be easier to take the decision to merge it.

[skojten]

Hi,
I have had problems with logging in to our server, similar to what others have described. Something I have noticed is that having a '%' in e.g. password makes it in principle impossible to connect to the server, while an identical password where the '%' is changed to something else works perfectly. My own conclusion is that this could have to do with MIME encoding in e.g. basic authentication, but I am not the person to investigate that.
Nevertheless, after (and only after) skipping the '%' in the password the 'beta5_fix-basic-auth' works for me.

[zuckel]

I just tried v4.0.0-beta5 with 261d3d2 cherry-picked on top and it did not help. I'm not 100% sure if I have the same problem though. Password prompts started popping up 2 or 3 weeks ago and will sometimes accept and other times not accept my password. I've also seen parts of my password being scrambled together with the username in the description of the dialog. Should I try another version?

[Trim]

@skojten

Thanks, I'll do a PR with the fix.

For the issue with the % char in password, I don't know how to fix it. The current code is adding manually the password in the current channel. To do it, it uses the URIEncodeComponent util from Mozilla. That seems the right way, but it doesn't work.

Personnaly, I think we should find a way to stop customizing the channels by:

1. Removing password cache and requires users to use the Thunderbird password manager to use this extension
2. Finding a way to announce to user for which calendar the user/password prompt is asked

@zuckel
I hopped to not see again the password in dialog with the 4.0.0-beta5 workaround. The issue is due to the fact the extension manually updates the communication channel and it seems Firefox has modified some code in release 52. Unfortunately, I didn't find where the issues was exactly, but I found this workaround implemented in beta5: setting user and password in one command worked better than just setting the password.

[frevib]

https://github.com/Trim/exchangecalendar/tree/fix-reading-realm-from-www-authenticate works, thanks!