SmtpClient.ConnectAsync wrong CancellationToken behaviour

[SirRufo]

Describe the bug
CancellationToken did not behave as expected in SmtpClient.ConnectAsync.

To Reproduce

using MailKit.Net.Smtp;

using System;
using System.Diagnostics;
using System.Threading;
using System.Threading.Tasks;

namespace SmtpConnectTimeout.ConsoleApp
{
    static class Program
    {
        private const string Host = "example.com"; // an ip address of a shutdown/non-existing system
        private const int Port = 25;
        private const bool UseSsl = false;

        static async Task Main(string[] args)
        {
            var sw = new Stopwatch();

            var cts = new CancellationTokenSource();
            var token = cts.Token;
            cts.CancelAfter(4000);

            using (var smtp = new SmtpClient())
            {
                Console.Write("connecting ");
                sw.Restart();

                var connectTask = smtp.ConnectAsync(Host, Port, UseSsl, token);
                do
                {
                    Console.Write(".");
                    var task = await Task.WhenAny(connectTask, Task.Delay(1000));

                } while (!connectTask.IsCompleted);
                Console.Write(" ");

                try
                {
                    await connectTask;
                    Console.WriteLine("OK {0:#,##0}ms", sw.ElapsedMilliseconds);
                }
                catch (Exception ex)
                {
                    Console.WriteLine("FAIL {0:#,##0}ms", sw.ElapsedMilliseconds);
                    Console.WriteLine("{0}: {1}", ex.GetType().Name, ex.Message);
                }
            }
        }
    }
}

Expected behavior
I would expect that after around 4,000ms the operation is canceled.

Screenshots

connecting .......................................... FAIL 42,068ms
OperationCanceledException

This has not stopped because of the CancellationToken but because of a timeout exception which is now (wrong) an OperationCanceledException.

Without passing the token I get this

connecting .......................................... FAIL 42,048ms
SocketException: Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat [2606:2800:220:1:248:1893:25c8:1946]:25

Additional context
This was tested on .net FX 4.7.2 and .net Core 2.2

[jstedfast]

Thanks.