Likely Dumb Newbie Questions

[whitedavidp]

So I use an older version of TBird as my email client on the desktop. Don't want to update it as that will break a bunch of plugins I wrote and use daily... Until now, I have been getting by using "app passwords" for gmail and hotmail/outlook accounts. But this morning I get an email from MS saying that in the near future password sign ins are being discontinued (they why did I have to go through all that rigamarole a short while ago?) so I went looking and found this project which seems so very promising to someone in my condition.

Right now I am focused only on my hotmail/outlook account. I have done my best to configure things and am basically following the info the the example config file. But:

1. I am still not quite sure what I need for client_id and client_secret. I just do not think I have another source to take these from. I do use BlueMail on my Android. And I am not even sure it that app will work when MS forces these changes (right now I am using app password same as in TBird). So I have to get these values somehow. But I a not very sure how.
2. I have tried going to the Entra/Azure link provided but it will not let me have access. I am so new to this I have no idea why or what to do.

So I am stuck at this point. I am sure this is all newbie stuff. But to me it is way over my pay grade. Can anyone point me in the right direction?

Thanks

[whitedavidp]

I just realized that I might be able to install a modern TBird on a spare computer, let it self-configure and they try to get/use info from that.

I have it connecting and when I go into the Security->Passwords and show myself what is there, I am wondering if the URL (1st column) and Password (3rd column) are what I need for oauth2_scope and client_secret? I assume that the client_id is just my full login email user id (xxx@outlook.com.

Am I on the right track?

So when I put those values into place and try to download (via pop) a message I have just sent to myself, I no longer get errors or prompts. But the connection login takes a really long time and nothing is actually downloaded.

So I am still not in heaven. Any tips are appreciated. Thanks

[whitedavidp]

Ok a little more progress...

I found some client ids for TBird here. And could confirm the url's in the sample file as well.

Now I am still prompted to authorize account. So when I do this (external or not) I get an error to the extent I cannot use my personal account and must use a school or work account (neither of which do I have).

Any tips? Thanks

[simonrob]

Thanks for following up – glad you managed to make progress with setting things up.

Re: your latest message –Microsoft have recently for some unknown reason decided to change the scope values for OAuth 2.0 access. Could you try the following value for oauth2_scope (adding the equivalent for IMAP if you need that functionality):

https://outlook.office.com/POP.AccessAsUser.All https://outlook.office.com/SMTP.SendAsApp offline_access

[whitedavidp]

Hi and thanks so much for your help. I tried the suggested change and, sadly, no joy still. The proxy prompts me to authorize the account as expected. But when I do so, I get the exact, same result as shown above. This time, I also clicked the sign-in options link and was shown the following:

Not at all sure what this means. But between this and the message given on the previous dialog, it seems that there is something to do with MS knowing my account is personal but I am taken to an authorize page that is for organizations.

I am feeling quite close now and learning enough to be dangerous. But this is still mostly Greek to me :-)

Thanks

[whitedavidp]

I did some more looking at the TBird code referenced above. I noticed a couple things:

"outlook.office365.com",
     [
       "login.microsoftonline.com",
-      "https://outlook.office365.com/IMAP.AccessAsUser.All https://outlook.office365.com/POP.AccessAsUser.All https://outlook.office365.com/SMTP.Send offline_access",
+      "https://outlook.office.com/IMAP.AccessAsUser.All https://outlook.office.com/POP.AccessAsUser.All https://outlook.office.com/SMTP.Send offline_access",
     ],

and

[
     "login.microsoftonline.com",
-    [
-      "08162f7c-0fd2-4200-a84a-f25a4db0b584", // Application (client) ID
-      "TxRBilcHdC6WGBee]fs?QR:SJ8nI[g82", // @see App registrations | Certificates & secrets
+    {
+      clientId: "9e5f94bc-e8a4-4e73-b8be-63364c29d753", // Application (client) ID
       // https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-v2-protocols#endpoints
-      "https://login.microsoftonline.com/common/oauth2/v2.0/authorize",
-      "https://login.microsoftonline.com/common/oauth2/v2.0/token",
-    ],
+      authorizationEndpoint:
+        "https://login.microsoftonline.com/common/oauth2/v2.0/authorize",
+      tokenEndpoint:
+        "https://login.microsoftonline.com/common/oauth2/v2.0/token",
+      redirectionEndpoint: "https://localhost",
+    },
   ],

And so I started trying stuff out. One thing I noted was that the redirection endpoint in the code is https: while that in the example config file (and thereby also in my own) was http: (no s). That got me a little further along. But still no joy.

Then I tried the client_id = 9e5f94bc-e8a4-4e73-b8be-63364c29d753 and the following for the scope:
oauth2_scope = https://outlook.office.com/IMAP.AccessAsUser.All https://outlook.office.com/POP.AccessAsUser.All https://outlook.office.com/SMTP.Send offline_access
And that actually ended up getting me to the point where I could finally authorize via the proxy via the normal-looking, MS routine.

So now on to the next challenge: actually getting inbound POP email. TBird periodically polls for new email on this account. Regardless of what follows, I seem to get prompted by the proxy to re-authorize every single time. And that seems strange and rather painful. Perhaps I am missing something?

And I cannot help but notice that sometimes, when I do try to authorize when prompted by the proxy, a window pops up middle of the screen and then disappears fairly quickly. Amazingly, I sometimes then see the following notification from the proxy:

And that is quite gratifying. However, sometimes I also get a failed notification and that seems odd. I have also seen TBird put up its failure dialog:

Even with all of this, I still have not received a test email I sent to myself at this account - although I just had a thought that perhaps it is trapped in the outlook spam filter and need to check that to be sure.

Thanks for any info/suggestions. Your time and help are much appreciated. I am not sure if you will want to update the project with my findings above. Best

[whitedavidp]

I just checked and the email I was expecting WAS trapped in the outlook spam filter (not sure why). So I reported it as not-spam and it got moved to the inbox. Despite this, the pattern described above remains the same and that email was never downloaded to my TBird client.

It appears that the pattern is:

1. Every time TBird tries to connect to outlook via the proxy, I am prompted to authorize by the proxy.
2. I have to use the proxy's menu twice each time. The first time always seems to fail and I am prompted again. The second time seems to succeed (in that I am not prompted again).
3. I see the above login failure dialog in TBird right after (or possibly congruent with) seeing the success notification from the proxy.
4. No email is download.

Thanks again!

[whitedavidp]

I decided to try and see if I can get any further using my gmail account on TBird. Getting setup went much faster. Once again, I used the TBird source code I found referenced above. And so my server entry is:

permission_url = https://accounts.google.com/o/oauth2/auth
token_url = https://oauth2.googleapis.com/token
oauth2_scope = https://mail.google.com/
redirect_uri = http://localhost
client_id = 406964657835-aq8lmia8j95dhl1a2bvharmfk3t1hgqj.apps.googleusercontent.com
client_secret = kSmqreRr0qwBWJgbf5Y-PjSU

Authorization started smoothly enough. But after the first authorization page, I got taken to this error:

I looked again at my server config and noticed again that the redirect was not https. So I changed that and tried it all again. Sadly, I came to the same end.

This makes me wonder which browser is being invoked to show/process these windows? I presumed it was the Windows default (which in my case is Firefox). So since this is google, I decided to try again after changing the default browser to Chrome. Sadly, still no joy.

Tips/suggestions are much appreciated. Thanks!

[whitedavidp]

Silly/stoopid me! I enabled external auth mode and tried again. I was able to succeed. As warned, the URL returned to the browser created a connection error. But when I copied the URL and pasted it, I was in business! Now to see how often I get asked to authorize... And I guess I will try with Outlook again this way. Best!

Well, I tried external authorization on Outlook as well. It appears to produce the same, exact results as described above using the internal method. Rats!

[whitedavidp]

Perhaps a little more progress... I found here another scope value to try with Outlook:

oauth2_scope = https://outlook.office.com/IMAP.AccessAsUser.All https://outlook.office.com/POP.AccessAsUser.All https://outlook.office.com/EWS.AccessAsUser.All https://outlook.office.com/SMTP.Send offline_access

This ended up with a different result in the authorization process and I actually got an email from MS saying "New app(s) have access to your data". Have not seen that before! Seems like progress. However, I still am seeing the same pattern of intermittent auth success/failure.

So I turned on debugging (had not thought of that before - what a dope I am) and checked the log. I am seeing:

2024-07-04 12:50:06,837: Caught exception while requesting OAuth 2.0 credentials for account xxx@outlook.com: URLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1000)'))

Is there a certificate somewhere in the proxy? Or is it using one on my system somewhere that may be the problem?

Thanks!

[simonrob]

I'm afraid I can't replicate this. With a Hotmail account (which I believe is treated the same as an Outlook account), and using the new scope values and the more recent Thunderbird client ID that you have used above (with no client secret), I can successfully authenticate and access the account.

You should definitely not be prompted to authenticate by the proxy every time you use the account. I was just about to point to debug mode, but I see you've done this already as I wrote this, which is good. That error is encountered when retrieving the token details, and may well be linked to the other issues you've encountered. You could try installing certifi?

Re: the issues you mentioned above where Thunderbird (which supports OAuth 2.0 natively, by the way) seems to just hang and not retrieve messages: it's important to note that this client is not a good choice for testing because it tries hard to simplify the connection process to the point where it modifies what you enter (such as encrypted vs. unencrypted connection details), often giving the impression it is using the values you've given when in fact it is not. As a result, when investigating an issue like this, it is always best to use one of the methods outlined in the troubleshooting section of the readme so that you can be totally sure of what is actually happening.

[whitedavidp]

Thanks so much. I just noticed that at the top of the log file I see the following:

2024-07-03 07:39:29,353: Running in a packaged/frozen environment - imported SSL certificates from certifi``

And I am running the pre-made executable from the .zip file on Windows 10. And that maps to your comment as well. It is interesting that Outlook should complain, however, and Gmail does not. Either way, I have no idea how I might update certifi (or anything else python-wise on Windows. I am presuming that Python and its supporting packages are somehow wrapped-up inside the .exe file. But trying to open it with 7-zip shows only one entry (endpoint-rule-set-1.json) and I cannot view that but I can extract it and then look at its contents. Not sure I see anything promising in there.

Regarding my TBird. I actually still use TBird v45 as anything newer will kill all the many plugins I have written and use daily. That version does NOT seem to support oauth and hence my searching for and finding (to my great joy) your excellent project.

[whitedavidp]

Logically, it seems that if there is a problem with a cert, that cert must be inside the Windows executable. TBird is not using crypto to the proxy. Does that need to be rebuilt with later certifi? I find it hard to believe the faulty cert is on the Outlook side. Thanks

[whitedavidp]

Since I have another, cleaner/simpler Windows machine on my LAN, I decided to copy over my setup there and run it. When I use that proxy, it works just great! So this convinces me that the issue has to do with a certificate on my original machine, NOT within python or anywhere else.

I would rather run the proxy locally than over my LAN like this. Doing so simplifies things for my simple mind. So now I have to try and see how/if I can normalize the computer's certificates from the other machine to my local one. It is making my brain hurt :-(

Could the problem comes from encrypting the data that is stored in the config file as a result of the authorization process? Is there some way to test this by turning off that encryption?

Thanks

[whitedavidp]

Wow, I am beginning to think I can cure the common cold (if I work hard enough)...

Realizing that there was indeed a problem on my desktop when I discovered that my other Windows machine on the LAN worked fine, I decided to explore the differences in the stored certificates between the 2 machines - which are otherwise quite similar.

I used the following powershell script on each machine and then diff'ed the two results:

Get-ChildItem cert:\ -Recurse | Where-Object {$_ -is [System.Security.Cryptography.X509Certificates.
X509Certificate2] -and $_.NotAfter -lt (Get-Date)} | Add-Member ScriptProperty -Name IssuedTo -Value { $this.GetNameInfo
( 'SimpleName', $false) } -PassThru |  Add-Member ScriptProperty -Name IssuedBy -Value { $this.GetNameInfo( 'SimpleName'
, $true ) } -PassThru | Select-Object -Property IssuedTo,IssuedBy,FriendlyName,NotAfter | Format-Table -AutoSize > C:\tm
p\expired_certs.txt

This demonstrated that the machine having troubles had a bunch of certificates installed that were not found on the working machine. I have no idea how the certificate being used is selected. So, after fully backing up the troubled machine, I went through the diff's one-by-by and searched for each cert that was not matched on the working machine on the basis of the IssuedTo and NotAfter columns. I had to look in both user and machine certs to find them all but I managed to remove the ones that were not present on the working machine and proved with a final diff showing no changes that I had them in sync.

As soon as I ran the proxy and authorized the Outlook account, all is well - working perfectly as expected and seen on the other machine..

Thanks so much for all of your time, patience, and help with this.