Warning with redis storage: You're running Souin with the default storage that is not optimized and for development purpose

[bgeneto]

I'm getting this warning message after having compiled the latest cache handler version, even though I'm using Redis storage (see Dockerfile below):

You're running Souin with the default storage that is not optimized and for development purpose.  
We recommend to use at least one of the storages from https://github.com/darkweak/storages

Here is the Dockerfile used to build the image...

FROM caddy:builder AS builder

RUN xcaddy build --with github.com/caddy-dns/duckdns \
                 --with github.com/caddy-dns/cloudflare \
                 --with github.com/darkweak/storages/redis/caddy \
                 --with github.com/caddyserver/cache-handler

FROM caddy

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

And below is the relevant global cache config in my Caddyfile:

{
        auto_https disable_redirects
        order cache before file_server
        order rewrite before file_server
        log {
                level warn
        }
        cache {
                default_cache_control public
                log_level error
                ttl 7200s
                stale 600s
                max_cacheable_body_bytes 2097152
                allowed_http_verbs GET HEAD POST
                redis {
                        url keydb:6379
                }
        }
}

[...]

        handle @somehost {
                cache
                root * /srv/app/somehost
                file_server
        }

It used to work previously, are there any breaking changes? TIA.

[darkweak]

Hello @bgeneto do you have some error logs above? If Souin/cache-handler fallbacks on the default storage, that's because it couldn't connect to the redis instance. Is your keydb reachable?

[bgeneto]

Hello @bgeneto do you have some error logs above?

Unfortunately there is no fallback log message. Only messages below. I'd tried hard to make it work...

{"level":"warn","ts":1726568026.1681597,"logger":"http.handlers.cache","msg":"You're running Souin
 with the default storage that is not optimized and for development purpose. We recommend to use at least one of the storages from https://github.com/darkweak/storages"}
{"level":"debug","ts":1726568026.1681979,"logger":"http.handlers.cache","msg":"Storer initialized: []types.Storer{(*storage.Default)(0xc0000e62e0)}."}
{"level":"debug","ts":1726568026.1682246,"logger":"http.handlers.cache","msg":"Surrogate storage initialized."}
{"level":"debug","ts":1726568026.1682472,"logger":"http.handlers.cache","msg":"Set Souin
 as Cache-Status name"}
{"level":"debug","ts":1726568026.16826,"logger":"http.handlers.cache","msg":"Enable GraphQL logic due to your custom HTTP verbs setup."}
{"level":"debug","ts":1726568026.168281,"logger":"http.handlers.cache","msg":"Allow 3 method(s). [GET HEAD POST]."}
{"level":"debug","ts":1726568026.1683073,"logger":"http.handlers.cache","msg":"The cache logic will run as : &{Strict:true Bypass_request:false Bypass_response:false}"}
{"level":"info","ts":1726568026.168325,"logger":"http.handlers.cache","msg":"Set backend timeout to 10s"}
{"level":"info","ts":1726568026.1683397,"logger":"http.handlers.cache","msg":"Set cache timeout to 10s"}
{"level":"info","ts":1726568026.1683526,"logger":"http.handlers.cache","msg":"Souin
 configuration is now loaded."}
{"level":"warn","ts":1726568026.175706,"logger":"http.handlers.cache","msg":"You're running Souin
 with the default storage that is not optimized and for development purpose. We recommend to use at least one of the storages from https://github.com/darkweak/storages"}
{"level":"debug","ts":1726568026.1757329,"logger":"http.handlers.cache","msg":"Storer initialized: []types.Storer{(*storage.Default)(0xc0000e62e0)}."}
{"level":"debug","ts":1726568026.1757607,"logger":"http.handlers.cache","msg":"Surrogate storage initialized."}
{"level":"debug","ts":1726568026.175776,"logger":"http.handlers.cache","msg":"Set Souin
 as Cache-Status name"}
{"level":"debug","ts":1726568026.1757874,"logger":"http.handlers.cache","msg":"Enable GraphQL logic due to your custom HTTP verbs setup."}
{"level":"debug","ts":1726568026.1758065,"logger":"http.handlers.cache","msg":"Allow 6 method(s). [GET HEAD PROPFIND GET HEAD POST]."}
{"level":"debug","ts":1726568026.175828,"logger":"http.handlers.cache","msg":"The cache logic will run as : &{Strict:true Bypass_request:false Bypass_response:false}"}
{"level":"info","ts":1726568026.1758401,"logger":"http.handlers.cache","msg":"Set backend timeout to 10s"}
{"level":"info","ts":1726568026.1758528,"logger":"http.handlers.cache","msg":"Set cache timeout to 10s"}
{"level":"info","ts":1726568026.1758654,"logger":"http.handlers.cache","msg":"Souin
 configuration is now loaded."}

If Souin/cache-handler fallbacks on the default storage, that's because it couldn't connect to the redis instance.

I don't know... I think that it is not falling back but not detecting the storage plug-in...

Is your keydb reachable?

Yes. It is...

❯ docker exec -it caddy ping keydb
PING keydb (172.24.0.2): 56 data bytes
64 bytes from 172.24.0.2: seq=0 ttl=64 time=0.139 ms
64 bytes from 172.24.0.2: seq=1 ttl=64 time=0.137 ms
64 bytes from 172.24.0.2: seq=2 ttl=64 time=0.136 ms

❯ docker exec -it caddy telnet keydb 6379
Connected to keydb

[bgeneto]

It follows a minimal Dockerfile, compose.yaml and Caddyfile to test...
just run something like docker compose up -d and then docker logs <container_id> to check the issue.

https://drive.google.com/file/d/1tFvsq_rxU5aav6ViR-38-693kWAOl3yw/view?usp=sharing

Thanks for your help!

[darkweak]

@bgeneto I found the bug

The check about the storages to use and the generated Uuid to identify redis weren't good.
This enables souin-client as default client name according to the go-redis expectations https://github.com/darkweak/souin/pull/551/files#diff-b6c42537d079f776a1500f1dcc8333e78f34e9bfec8d9e6817994dcb9a480228
This function https://github.com/darkweak/souin/pull/551/files#diff-e1704a111d38befa7770754690d6c92b01e974385878b10fc285f9dd6d279868R119-R122 centralize the check for each available storage.
This line https://github.com/darkweak/souin/pull/551/files#diff-e1704a111d38befa7770754690d6c92b01e974385878b10fc285f9dd6d279868R206 check all storages with the previous function.

That should work, you can already use it

RUN xcaddy build --with github.com/caddy-dns/cloudflare \
                 --with github.com/darkweak/storages/go-redis/caddy \
                 --with github.com/darkweak/souin@v1.7.1=github.com/darkweak/souin@5a79ab7fe694a33db6dc6014d79834cbb363bc5f \
                 --with github.com/darkweak/souin/plugins/caddy@5a79ab7fe694a33db6dc6014d79834cbb363bc5f

[popcorn]

@darkweak, is this fix shipped to caddyserver/cache-handler or do I have to use darkweak/souin?

I'm having the identical issue as @bgeneto

FROM caddy:2.8.4-builder AS builder

RUN xcaddy build \
    --with github.com/caddyserver/cache-handler \
    --with github.com/darkweak/storages/go-redis/caddy

with this caddy.json

{
  "apps": {
    "cache": {
      "log_level": "DEBUG",
      "ttl": "5m",
      "headers": [
        "Content-Type",
        "Authorization"
      ],
      "cache_name": "saas_custom_domains_cache",
      "redis": {
        "url": "localhost:6379"
      }
    },
    "http": {
      "servers": {
        "tls_terminator": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "localhost"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "cache"
                },
                {
                  "handler": "static_response",
                  "body": "Hello, world!",
                  "status_code": 200
                }
              ]
            }
          ],
          "logs": {}
        }
      }
    },
    "tls": {
      "automation": {
        "on_demand": {
          "ask": "http://localhost:3000/caddy/ask"
        }
      },
      "cache": {
        "capacity": 100000
      }
    }
  },
  "admin": {
    "identity": {
      "issuers": [
        {
          "module": "acme",
          "email": "drago+dev@saascustomdomains.com"
        }
      ]
    }
  },
  "logging": {
    "logs": {
      "default": {
        "exclude": [
          "http.log.access"
        ],
        "writer": {
          "output": "file",
          "filename": "/var/log/caddy-dev/caddy.log",
          "roll": true,
          "roll_size_mb": 64,
          "roll_keep": 20
        },
        "encoder": {
          "format": "json",
          "time_format": "iso8601"
        }
      },
      "log0": {
        "writer": {
          "output": "file",
          "filename": "/var/log/caddy-dev/access.log",
          "roll": true,
          "roll_size_mb": 64,
          "roll_keep": 20
        },
        "encoder": {
          "format": "json",
          "time_format": "iso8601"
        },
        "include": [
          "http.log.access"
        ]
      }
    }
  },
  "storage": {
    "module": "file_system",
    "root": "~/caddy-dev/data"
  }
}

And I get these logs:

...
{"level":"warn","ts":"2025-03-20T16:56:10.318Z","logger":"http.handlers.cache","msg":"You're running Souin with the default storage that is not optimized and for development purpose. We recommend to use at least one of the storages from https://github.com/darkweak/storages"}
{"level":"info","ts":"2025-03-20T16:56:10.319Z","logger":"http.handlers.cache","msg":"Set backend timeout to 10s"}
{"level":"info","ts":"2025-03-20T16:56:10.320Z","logger":"http.handlers.cache","msg":"Set cache timeout to 10ms"}
{"level":"info","ts":"2025-03-20T16:56:10.321Z","logger":"http.handlers.cache","msg":"Souin configuration is now loaded."}
...

My Redis instance is reachable from Caddy container. I confirmed this by installing redis-cli in the container and running it:

~ # redis-cli -h localhost
localhost:6379> ping
PONG

In Safari Networking tool I see we got cache hit (which I assume comes from default storage):

Is there anything I'm missing here?

[darkweak]

@popcorn you have to use the latest version of Souin instead of the cache-handler because I didn't tag the new version yet.

[popcorn]

Ok thank you!

Btw, are the caddyserver/cache-handler and darkweak/souin configurations in my Caddy JSON completely interchangeable?

Will I easily be able to build my Caddy later with cache-handler instead of souin without changing the Caddy JSON config? Or do you not guarantee this?

[darkweak]

It's the same configuration using either Souin or cache-handler.

[popcorn]

Hmm, it seems building with commit specified didn't help me. I still see the same issue.

FROM caddy:2.8.4-builder AS builder

RUN xcaddy build \
  --with github.com/darkweak/storages/go-redis/caddy \
  --with github.com/darkweak/souin@v1.7.1=github.com/darkweak/souin@5a79ab7fe694a33db6dc6014d79834cbb363bc5f \
  --with github.com/darkweak/souin/plugins/caddy@5a79ab7fe694a33db6dc6014d79834cbb363bc5f

I am now wondering if there's another issue and Souin is failing to connect to my Redis but I can't figure it out.

I can see the Redis is running locally:

/srv # ps
PID   USER     TIME  COMMAND
    1 root      0:00 {start.sh} /bin/bash /usr/local/bin/start.sh
    9 root      0:00 redis-server *:6379
   10 root      0:00 caddy run --config /etc/caddy/caddy-dev.json
   26 root      0:00 sh
   35 root      0:00 ps

And I can connect to it manually:

/srv # redis-cli
127.0.0.1:6379> ping
PONG

I cannot figure this out for the life of me. Is there a way to get more logs out of Souin to see what's going on under the hood?

[darkweak]

FROM caddy:2.8.4-builder AS builder

RUN xcaddy build \
  --with github.com/darkweak/storages/go-redis/caddy \
  --with github.com/darkweak/souin/plugins/caddy@v1.7.5

[popcorn]

Ahh ok, missed that. I rebuilt, but happens still.

Built with:

FROM caddy:2.8.4-builder AS builder

RUN xcaddy build \
      --with github.com/darkweak/storages/go-redis/caddy \
      --with github.com/darkweak/souin/plugins/caddy@v1.7.5

Logs:

{"level":"warn","ts":"2025-03-21T11:46:07.009Z","logger":"http.handlers.cache","msg":"You're running Souin with the default storage that is not optimized and for development purpose. We recommend to use at least one of the storages from https://github.com/darkweak/storages"}
{"level":"info","ts":"2025-03-21T11:46:07.011Z","logger":"http.handlers.cache","msg":"Set backend timeout to 10s"}
{"level":"info","ts":"2025-03-21T11:46:07.013Z","logger":"http.handlers.cache","msg":"Set cache timeout to 100ms"}
{"level":"info","ts":"2025-03-21T11:46:07.015Z","logger":"http.handlers.cache","msg":"Souin configuration is now loaded."}
{"level":"info","ts":"2025-03-21T11:46:07.102Z","logger":"admin.identity.cache.maintenance","msg":"started background certificate maintenance","cache":"0x40003c7e00"}

Here's my full caddy.json:

{
  "apps": {
    "cache": {
      "log_level": "debug",
      "ttl": "5m",
      "headers": [
        "Content-Type",
        "Authorization"
      ],
      "cache_name": "saas_custom_domains_cache",
      "distributed": true,
      "timeout": {
        "backend": "10s",
        "cache": "100ms"
      },
      "redis": {
        "url": "127.0.0.1:6379"
      }
    },
    "http": {
      "servers": {
        "tls_terminator": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "localhost"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "cache"
                },
                {
                  "handler": "static_response",
                  "status_code": 200,
                  "body": "Hello, cache!"
                }
              ]
            }
          ],
          "logs": {}
        }
      }
    },
    "tls": {
      "automation": {
        "on_demand": {
          "ask": "http://localhost:3000/caddy/ask"
        }
      },
      "cache": {
        "capacity": 100000
      }
    }
  },
  "admin": {
    "identity": {
      "issuers": [
        {
          "module": "acme",
          "email": "drago+dev@saascustomdomains.com"
        }
      ]
    }
  },
  "logging": {
    "logs": {
      "default": {
        "exclude": [
          "http.log.access"
        ],
        "writer": {
          "output": "file",
          "filename": "/var/log/caddy-dev/caddy.log",
          "roll": true,
          "roll_size_mb": 64,
          "roll_keep": 20
        },
        "encoder": {
          "format": "json",
          "time_format": "iso8601"
        }
      },
      "log0": {
        "writer": {
          "output": "file",
          "filename": "/var/log/caddy-dev/access.log",
          "roll": true,
          "roll_size_mb": 64,
          "roll_keep": 20
        },
        "encoder": {
          "format": "json",
          "time_format": "iso8601"
        },
        "include": [
          "http.log.access"
        ]
      }
    }
  },
  "storage": {
    "module": "file_system",
    "root": "~/caddy-dev/data"
  }
}

Do you see anything obviously wrong with my configuration?

Is there any way to add more logging around the cache connection code so it's easier to debug this?

[popcorn]

I'll build with patched Souin and report back.

[popcorn]

Found the solution for my issue!

I patched darkweak/souin with extra logging in darkweak/souin@df12b2d and built Caddy.

I found that I had to explicitly set "redis": { "found": true, "url": "..." } in Caddy JSON config for this to work.

This was confusing because configuration file https://github.com/caddyserver/cache-handler/blob/master/configuration.json that is mentioned in the docs doesn't have found anywhere.

Anyway, with found explicitly defined in redis configuration, both caddyserver/cache-handler and darkweak/souin work for me and find the Redis storage.