Unable to contact AWS Memcached instances from Windows

[thoffmann-fms]

I have three AWS Memcached instances and only one is working. The traces I've run point to an SSL Certificate issue, but I have the 5 recommended AWS Root Certs installed. Has anyone else had this issue? The only error Enyim is logging is a "socket reset".

09:00:08.395 INFORMATION - DEBUG: Memcached:Log.314
  Debug Use NodeLocator: Enyim.Caching.Memcached.SingleNodeLocator. Current server count: 1
09:00:08.401 INFORMATION - DEBUG: Memcached:Log.314
  Information Memcached server address - XXXX.serverless.use1.cache.amazonaws.com:11211
09:00:11.038 INFORMATION - DEBUG: Memcached:Log.314
  Debug ExecuteOperationAsync(Enyim.Caching.Memcached.Protocol.Binary.StoreOperation)
09:00:11.554 INFORMATION - DEBUG: Memcached:Log.314
  Debug Pool has been inited for Unspecified/XXXX.serverless.use1.cache.amazonaws.com:11211 with 5 sockets
09:00:11.565 INFORMATION - DEBUG: Memcached:Log.314
  Information MemcachedInitPool-cost: 518.5943ms
09:00:11.573 INFORMATION - DEBUG: Memcached:Log.314
  Debug Acquiring stream from pool. Unspecified/XXXX.serverless.use1.cache.amazonaws.com:11211
09:00:11.578 INFORMATION - DEBUG: Memcached:Log.314
  Debug Socket 2fafbca6-8639-41b5-9534-6a982648c5e0 was reset
09:00:11.582 INFORMATION - DEBUG: Memcached:Log.314
  Debug Socket was reset. InstanceId 2fafbca6-8639-41b5-9534-6a982648c5e0
09:00:11.591 INFORMATION - DEBUG: Memcached:Log.314
  Debug pooledSocket.WriteAsync...
09:00:11.601 INFORMATION - DEBUG: Memcached:Log.314
  Debug Enyim.Caching.Memcached.Protocol.Binary.StoreOperation.ReadResponseAsync...
09:00:21.626 INFORMATION - DEBUG: Memcached:Log.314
  Debug Releasing socket 2fafbca6-8639-41b5-9534-6a982648c5e0
09:00:21.633 INFORMATION - DEBUG: Memcached:Log.314
  Debug Are we alive? True

OpenSSL logs an SSL issue, "unable to get local issuer certificate"

C:\Program Files\OpenSSL-Win64\bin>openssl s_client -showcerts -connect XXXXX.serverless.use1.cache.amazonaws.com:11211
Connecting to 10.102.11.0
CONNECTED(000001F0)
depth=3 C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=2 C=US, O=Amazon, CN=Amazon Root CA 1
verify return:1
depth=1 C=US, O=Amazon, CN=Amazon RSA 2048 M02
verify return:1
depth=0 CN=*.serverless.use1.cache.amazonaws.com
verify return:1
---
Certificate chain
 0 s:CN=*.serverless.use1.cache.amazonaws.com
   i:C=US, O=Amazon, CN=Amazon RSA 2048 M02
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 27 00:00:00 2024 GMT; NotAfter: Sep 25 23:59:59 2025 GMT
-----BEGIN CERTIFICATE-----
MIIGGjCCBQKgAwIBAgIQBl49qDc3bH8VtelS2cY+XjANBgkqhkiG9w0BAQsFADA8
...
sM8Xuyp7NnzFDF+gE8kWpGmKGbMVJ7nBVgJyZ0tXU2aDIWpkE9PTNf4dIYWLR9iQ
3qhXYa9OCy5MJ3COgIk7I71ER+W4Bov9LDNGrNoi
-----END CERTIFICATE-----
 1 s:C=US, O=Amazon, CN=Amazon RSA 2048 M02
   i:C=US, O=Amazon, CN=Amazon Root CA 1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 23 22:25:30 2022 GMT; NotAfter: Aug 23 22:25:30 2030 GMT
-----BEGIN CERTIFICATE-----
MIIEXjCCA0agAwIBAgITB3MSSkvL1E7HtTvq8ZSELToPoTANBgkqhkiG9w0BAQsF
...
slI2yayq0n2TXoHyNCLEH8rpsJRVILFsg0jc7BaFrMnF462+ajSehgj12IidNeRN
4zl+EoNaWdpnWndvSpAEkq2P
-----END CERTIFICATE-----
 2 s:C=US, O=Amazon, CN=Amazon Root CA 1
   i:C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 25 12:00:00 2015 GMT; NotAfter: Dec 31 01:00:00 2037 GMT
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgITBn+USionzfP6wq4rAfkI7rnExjANBgkqhkiG9w0BAQsF
...
0FE6/V1dN2RMfjCyVSRCnTawXZwXgWHxyvkQAiSr6w10kY17RSlQOYiypok1JR4U
akcjMS9cmvqtmg5iUaQqqcT5NJ0hGA==
-----END CERTIFICATE-----
 3 s:C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2
   i:C=US, O=Starfield Technologies, Inc., OU=Starfield Class 2 Certification Authority
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  2 00:00:00 2009 GMT; NotAfter: Jun 28 17:39:16 2034 GMT
-----BEGIN CERTIFICATE-----
MIIEdTCCA12gAwIBAgIJAKcOSkw0grd/MA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV
...
VsyuLAOQ1xk4meTKCRlb/weWsKh/NEnfVqn3sF/tM+2MR7cwA130A4w=
-----END CERTIFICATE-----
---
Server certificate
subject=CN=*.serverless.use1.cache.amazonaws.com
issuer=C=US, O=Amazon, CN=Amazon RSA 2048 M02
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5566 bytes and written 437 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Protocol: TLSv1.3
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)
---

[thoffmann-fms]

I have one update. AWS Memcached instances can require useSslStream to be true. That setting is "Encryption in transit" set to Enabled. My first cache had the value set to Disabled and UseSslStream was defaulting to False.

        MemcachedClientOptions options = new MemcachedClientOptions();
        options.AddServer("endpoint.cache.amazonaws.com", 11211);
        options.UseSslStream = true;

Using SSL fixes the normal clustered caches to work with the client, but the Serverless version does not respond back after the write occurs in PooledSocket.cs.

This works as expected.

        public async Task WriteAsync(IList<ArraySegment<byte>> buffers)
        {
            CheckDisposed();

            try
            {
                if (_useSslStream)
                {
                    foreach (var buf in buffers)
                    {
                        await _sslStream.WriteAsync(buf.Array, 0, buf.Count).ConfigureAwait(false);
                    }

                    await _sslStream.FlushAsync().ConfigureAwait(false);
                }

This section times out (regardless of the timeout setting).

        public async Task ReadAsync(byte[] buffer, int offset, int count)
        {
            try
            {
                CheckDisposed();

                int read = 0;
                int shouldRead = count;

                while (read < count)
                {
                    try
                    {
                        int currentRead = (_useSslStream
                            ? await _sslStream.ReadAsync(buffer, offset, shouldRead).ConfigureAwait(false)
                            : await _inputStream.ReadAsync(buffer, offset, shouldRead).ConfigureAwait(false));

[cnblogs-dudu]

Please try EnyimMemcachedCore 3.3.3-pre2.

dotnet add package EnyimMemcachedCore --version 3.3.3-pre2

[thoffmann-fms]

I tried 3.3.3-Pre2, no joy.

09:38:35.24 Debug Use NodeLocator: Enyim.Caching.Memcached.SingleNodeLocator. Current server count: 1
09:38:44.29 Debug ExecuteOperationAsync(Enyim.Caching.Memcached.Protocol.Binary.StoreOperation)
09:38:45.17 Debug Pool has been inited for Unspecified/XXX-ui4pdp.serverless.use1.cache.amazonaws.com:11211 with 5 sockets
09:38:45.17 Information MemcachedInitPool-cost: 879.041ms
09:38:45.18 Debug Acquiring stream from pool. Unspecified/XXX-ui4pdp.serverless.use1.cache.amazonaws.com:11211
09:38:45.18 Debug Socket a916c6c9-afd2-4d39-bf81-9ca2c65d1d79 was reset
09:38:45.19 Debug Socket was reset. InstanceId a916c6c9-afd2-4d39-bf81-9ca2c65d1d79
09:38:45.19 Debug pooledSocket.WriteAsync...
09:38:45.20 Debug Enyim.Caching.Memcached.Protocol.Binary.StoreOperation.ReadResponseAsync...
09:38:55.23 Debug Releasing socket a916c6c9-afd2-4d39-bf81-9ca2c65d1d79
09:38:55.23 Debug Are we alive? True

---

Test code:

        _LoggerFactory = new CacheLoggerFactory();

        MemcachedClientOptions options = new MemcachedClientOptions();
        options.AddServer(lessCache, 11211);
        options.UseSslStream = true;
        //options.SocketPool = new SocketPoolOptions();
        //options.SocketPool.ConnectionTimeout = new TimeSpan(0, 10, 0);

        MemcachedClientConfiguration config = new MemcachedClientConfiguration(_LoggerFactory, options);
        _Cache = new MemcachedClient(_LoggerFactory, config);

        bool saved = await _Cache.StoreAsync(StoreMode.Set, "tjh1", "testvalue", new TimeSpan(1, 2, 0));

[marceloboeira]

@thoffmann-fms did you ever manage to make this work?

[thoffmann-fms]

Sorry, no. I didn't really need the SSL feature, so I just turned if off in AWS (Encryption in transit-Disabled) and UseSslStream=false and moved on.

[marceloboeira]

@thoffmann-fms thanks for the reply. I'm using Elasticache Serverless, which doesn't allow disabling in-transit encryption... gonna have to figure out a workaround.

[marceloboeira]

@thoffmann-fms this worked for me:

...
            services.AddEnyimMemcached(options =>
            {
                    options.UseSslStream = true;
                    options.Protocol = Enyim.Caching.Memcached.MemcachedProtocol.Text;
                    options.AddServer(...);
                    options.SslClientAuth = new System.Net.Security.SslClientAuthenticationOptions
                        {
                            TargetHost = "$host",
                            RemoteCertificateValidationCallback = (sender, certificate, chain, sslPolicyErrors) => true
                        };
                    }
                }
            });
...