Initial setup with syslog ingestion and Unraid

[Zxurian]

Saw the 0.40 post on reddit and decided to try centralizing my homelab logs. I'm not an expert by any means, but dangerous enough to lookup and try solving things myself. Got stuck at a few points though. I followed the install docs at https://logtide.dev/docs/getting-started/, then https://logtide.dev/docs/syslog/ and approached it from the "follow the docs to get things running" approach and got 99% of the way there. My environment is structed that all compose stacks are handled by Komodo.

Setup Note 1

Using the basic installation, everything came up, but when trying to access the front end for the first time, I was getting a Failed to Fetch error. After some research and poking, saw the following in the logtide/docker.env.example that is referenced on the installation page

# Frontend API URL - AUTO-DETECTED in most cases, no configuration needed!
#
# How auto-detection works:
# - Access via http://192.168.1.100:3000 → API auto-detected at http://192.168.1.100:8080
# - Access via https://logtide.example.com (port 80/443) → Uses relative URLs (assumes reverse proxy)
#
# Only set this if you have a non-standard setup:
# PUBLIC_API_URL=http://custom-backend-host:8080

Only by uncommenting the PUBLIC_API_URL and setting it to the docker hostname reference was the dashboard signin/signup page able to work (in my case, http://docker-02.mydomain.network:8080). For a "Quick start" impression that the getting-started page gives, might be a good idea to at least list this one setup change if initial deployment fails auto-detection. (Can provide details of my enviornment if you want to diagnose why auto-detection failed)

Setup Note 2

While I run docker, I wouldn't say I'm "comfortable" with docker yet. Within the docker compose file, you've got fluent-bit listed with a profile declaration. I wasn't aware of what the profile did and originally couldn't figure out why the fluent-bit image wasn't coming up with the full stack. May I suggest adding a tip on the syslog setup page to note that you have to add COMPOSE_PROFILES=logging to make sure the fluent image starts?

Onto syslog ingestion setup. Per the syslog page, I setup the 4 files with the default data provided.

Setup Note 3

Under the Docker Compose Setup section, it differs from the default service described in however https://github.com/logtide-dev/logtide/blob/main/docker/docker-compose.yml . The differences being that in the docker-compose.yml file, there is an extra lua file wrap_logs.lua which isn't defined on the syslog setup page, and it also has the profiles declaration, whereas in the syslog setup page, there is no profiles delcaration. Should the syslog setup page compose match what's being distributed in the default stack compose?

Once fluent-bit was running, I attempted to setup my Unraid server to push syslog to it. First pitfall was nothing was coming through. Looked at fluent-bit logs (via Dozzle) and saw it was repeatedly getting an "authorization failed" when trying to talk to logtide. Looking at compose for fluent-bit, there's a reference to an environment variable LOGTIDE_API_KEY: ${FLUENT_BIT_API_KEY:-} that isn't mentioned anywhere in the docs. Taking a guess, I created a new Project in Logtide, generated a new API key, then set that env variable with the new API key and redeployed. Now fluent-bit was forwarding logs to logtide. However within the logs, the service was coming across as just syslog. Given this, it stands to reason that any other server I send syslog from is all going to show up in the same project. Is there a way to have a separate project per server that is sending syslog logs over? or did I miss how to do that and am using the wrong API key.

Looking at the fluent-bit.conf it looks like Unraid's syslog messages don't follow the standard RFC3164 format (Example Unraid syslog)

Jan 16 10:00:58 unraid-01 move: After script finished with result: OK
Jan 16 10:00:58 unraid-01 move: Cleaning lock and stop files
Jan 16 10:00:58 unraid-01 move: Action ended at: 10:00:58
Jan 16 10:00:58 unraid-01 move: Elapsed time: 37 sec

Is there a way to adjust the input/filter/parser so that any logs coming from a specific server are sent through a specific regex for parsing?

[Polliog]

Hey, thanks a ton for the detailed feedback! This is exactly the kind of real-world testing that helps improve the docs. Let me address the points you raised:

Re: PUBLIC_API_URL Auto-Detection You're correct the auto-detection logic assumes a standard setup (access via port 3000, API at 8080) and tends to break on orchestrators like Komodo. I'll add a troubleshooting section for this. For now, hardcoding it in your .env is the correct fix:
Bash

PUBLIC_API_URL=http://your-docker-host:8080

(I'd love to see your Komodo config later if you're up for sharing, it would help me make the detection smarter).

Docker Compose Profiles Good catch. Fluent Bit is hidden behind a logging profile so it doesn't start by default for users who don't need syslog. I definitely missed making that clear in the docs. To start it, you can run:

# To start with Fluent Bit / syslog support:
docker compose --profile logging up -d

Docs vs docker-compose.yml Mismatch Yeah, the syslog docs page is lagging behind the actual compose file. The missing wrap_logs.lua and profiles declaration need to be synced. I'll get that fixed ASAP.

FLUENT_BIT_API_KEY That is a big miss on my part, sorry about that! Here is the actual workflow:

1. Create a project in the LogTide dashboard.
2. Go to Project Settings → API Keys and generate a new key.
3. Add it to your .env: FLUENT_BIT_API_KEY=lp_your_key_here
4. Restart: docker compose --profile logging restart fluent-bit

Separating Logs by Server Currently, syslog uses the ident field (program name) as the service, falling back to hostname. To separate them cleanly (e.g., Unraid vs Proxmox), you have two main options:

Option A: Different ports per server (Cleanest / Recommended) This gives you full isolation. You'd set up two input blocks:

# Unraid on port 5514
[INPUT]
    Name        syslog
    Parser      syslog-unraid
    Listen      0.0.0.0
    Port        5514
    Mode        udp
    Tag         syslog.unraid

# Proxmox on port 5515
[INPUT]
    Name        syslog
    Parser      syslog-rfc3164
    Listen      0.0.0.0
    Port        5515
    Mode        udp
    Tag         syslog.proxmox
    ```
    
Option B: Rewrite Tags (If you must use port 514) If you need everything on the standard port, you can use a rewrite_tag filter to route logs based on the hostname:
  ```ini
  [FILTER]
    Name          rewrite_tag
    Match         syslog.*
    Rule          $host ^unraid.* syslog.unraid false
    Rule          $host ^proxmox.* syslog.proxmox false

[FILTER]
    Name          lua
    Match         syslog.unraid
    script        /fluent-bit/etc/parse_unraid.lua
    call          parse_unraid

If you share a sample of the Unraid syslog format, I can help you write that custom parser!

Unraid Syslog Format If you're seeing parsing issues or wrong timestamps, check the logs (docker compose logs fluent-bit) and try setting SYSLOG_TZ_OFFSET in your .env (e.g., SYSLOG_TZ_OFFSET=1 for CET).

Thanks again for taking the time to write this up! I'll be updating the docs based on your feedback this week. Let me know if you hit any other errors in the docs. I'm going to fix the reported one.

[Zxurian]

Happy to share my komodo config. The toml for the logtide stack is listed below. As far as implementation, I'm not using any reverse proxies (yet), so initial access was just through http://<docker hostname>:3000.

As far as the unraid logging, I attached a sample log. Looks like the date format isn't standard rfc, so regex would have to be updated for that. I'm already setting SYSLOG_TZ_OFFSET in .env, however because I hate DST, Perhaps a feature request specifying a timezone (America/New_York) instead of an offset would work better? Otherwise I'll have to remember to update it between -5 & -4 twice a year.

And as far as the splitting of syslogs, While I would like to use option A for splitting unraid/proxmox, thinking about other hardware in my homelab, there are some that don't let you specify the rsyslog port, so I'd probably go with option B. Looking at the [OUTPUT], blocks, I'm guessing that I can just add another block to match against syslog.unraid, then use a different X-API-Key to put it into it's own Project?

[[stack]]
name = "logtide"
[stack.config]
server = "docker-02"
poll_for_updates = true
linked_repo = "repo-name-redacted"
clone_path = "/etc/komodo"
run_directory = "stacks/logtide"
config_files = [
  { path = "fluent-bit.conf" },
  { path = "parsers.conf" },
  { path = "map_syslog_level.lua" },
  { path = "extract_container_id.lua" }
]
environment = """
# Volumes
DEFAULT_DOCKER_DATA=[[DEFAULT_DOCKER_DATA]]

# LogTide Docker Configuration
# =============================
# Copy this file to .env and set secure passwords

COMPOSE_PROFILES=logging

# =============================================================================
# REQUIRED - Set these before starting
# =============================================================================

# Database password (use a strong random password)
DB_PASSWORD=<redacted>

# Redis password (use a strong random password)
REDIS_PASSWORD=<redacted>

# API key encryption secret (minimum 32 characters)
API_KEY_SECRET=<redacted>

# =============================================================================
# OPTIONAL - Customize if needed
# =============================================================================

# Database settings (defaults work for most setups)
DB_NAME=logtide
DB_USER=logtide

# Frontend API URL - AUTO-DETECTED in most cases, no configuration needed!
#
# How auto-detection works:
# - Access via http://192.168.1.100:3000 → API auto-detected at http://192.168.1.100:8080
# - Access via https://logtide.example.com (port 80/443) → Uses relative URLs (assumes reverse proxy)
#
# Only set this if you have a non-standard setup:
PUBLIC_API_URL=http://docker-02.<redacted>.<redacted>:8080

# Docker images (pin versions for production stability)
# LOGTIDE_BACKEND_IMAGE=logtide/backend:0.4.1
# LOGTIDE_FRONTEND_IMAGE=logtide/frontend:0.4.1

# Fluent Bit image (for log collection)
# Default: fluent/fluent-bit:4.2.2
# For Raspberry Pi / ARM64, you may need to build your own image or use:
#   - cr.fluentbit.io/fluent/fluent-bit:4.2.2 (official multi-arch)
#   - Custom ARM64 build
# FLUENT_BIT_IMAGE=fluent/fluent-bit:4.2.2
SYSLOG_TZ_OFFSET=-5
FLUENT_BIT_API_KEY=<redacted>

# =============================================================================
# EMAIL NOTIFICATIONS (optional)
# =============================================================================
# SMTP_HOST=smtp.example.com
# SMTP_PORT=587
# SMTP_USER=your_email@example.com
# SMTP_PASS=your_smtp_password
# SMTP_FROM=alerts@yourdomain.com

# =============================================================================
# HORIZONTAL SCALING (advanced)
# =============================================================================
# For scaling, use docker-compose.traefik.yml:
#   docker compose -f docker-compose.yml -f docker-compose.traefik.yml up -d --scale backend=3
#
# See docs: https://logtide.dev/docs/deployment#horizontal-scaling
"""

unraid-01-log-snippet.log

[Polliog]

@Zxurian Thanks for sharing the Komodo config! That helps understand the setup. The auto-detection assumes direct browser-to-backend connectivity on predictable ports, but with orchestrators like Komodo managing the stack, the hostname
resolution can differ. So yes the PUBLIC_API_URL. in this setup is required
Great idea about the timezone feature I'll add support for IANA timezone names (America/New_York, Europe/Rome, etc.) instead of just numeric offsets. That way DST is handled automatically. I've noted this for a future release.

about the splitting syslog, yes, exactly you can add multiple [OUTPUT] blocks with different API keys to route to different projects:

# Route Unraid logs to "unraid-01" project                                                                                                                                                                                        
[OUTPUT]                                                                                                                                                                                                                          
    Name                http                                                                                                                                                                                                      
    Match               syslog.unraid                                                                                                                                                                                             
    Host                ${LOGTIDE_API_HOST}                                                                                                                                                                                       
    Port                8080                                                                                                                                                                                                      
    URI                 /api/v1/ingest/single                                                                                                                                                                                     
    Format              json_lines                                                                                                                                                                                                
    Header              X-API-Key ${UNRAID_API_KEY}                                                                                                                                                                               
    Header              Content-Type application/json                                                                                                                                                                             
    Json_date_key       time                                                                                                                                                                                                      
    Json_date_format    iso8601                                                                                                                                                                                                   
    Retry_Limit         3                                                                                                                                                                                                         
    tls                 Off                                                                                                                                                                                                       
                                                                                                                                                                                                                                  
# Route Proxmox logs to "proxmox" project                                                                                                                                                                                         
[OUTPUT]                                                                                                                                                                                                                          
    Name                http                                                                                                                                                                                                      
    Match               syslog.proxmox                                                                                                                                                                                            
    Host                ${LOGTIDE_API_HOST}                                                                                                                                                                                       
    Port                8080                                                                                                                                                                                                      
    URI                 /api/v1/ingest/single                                                                                                                                                                                     
    Format              json_lines                                                                                                                                                                                                
    Header              X-API-Key ${PROXMOX_API_KEY}                                                                                                                                                                              
    Header              Content-Type application/json                                                                                                                                                                             
    Json_date_key       time                                                                                                                                                                                                      
    Json_date_format    iso8601                                                                                                                                                                                                   
    Retry_Limit         3                                                                                                                                                                                                         
    tls                 Off                                                                                                                                                                                                       
                                                                                                                                                                                                                                  
# Catch-all for other syslog sources                                                                                                                                                                                              
[OUTPUT]                                                                                                                                                                                                                          
    Name                http                                                                                                                                                                                                      
    Match               syslog.*                                                                                                                                                                                                  
    Host                ${LOGTIDE_API_HOST}                                                                                                                                                                                       
    Port                8080                                                                                                                                                                                                      
    URI                 /api/v1/ingest/single                                                                                                                                                                                     
    Format              json_lines                                                                                                                                                                                                
    Header              X-API-Key ${FLUENT_BIT_API_KEY}                                                                                                                                                                           
    Header              Content-Type application/json                                                                                                                                                                             
    Json_date_key       time                                                                                                                                                                                                      
    Json_date_format    iso8601                                                                                                                                                                                                   
    Retry_Limit         3                                                                                                                                                                                                         
    tls                 Off                                                                                                                                                                                                       

Then in your .env:

UNRAID_API_KEY=lp_your_unraid_project_key                                                                                                                                                                                         
PROXMOX_API_KEY=lp_your_proxmox_project_key                                                                                                                                                                                       
FLUENT_BIT_API_KEY=lp_your_default_project_key

in theory the order should matter, more specific matches (syslog.unraid) should come before the catch-all (syslog.*).

looking at your log snippet, it's actually valid RFC3164 just without the priority field which some syslog implementations omit. The standard parser should work:
if you're seeing parsing issues, it might be that Unraid is sending without the prefix. Try this parser in parsers.conf:

  # RFC3164 without priority field (some systems omit it)                                                                                                                                                                           
  [PARSER]                                                                                                                                                                                                                          
      Name        syslog-rfc3164-nopri                                                                                                                                                                                              
      Format      regex                                                                                                                                                                                                             
      Regex       ^(?<time>[A-Z][a-z]{2} [ ]?[0-9]{1,2} [0-9:]+) (?<host>[^ ]+) (?<ident>[a-zA-Z0-9_\/\.\-]+)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$                                                                  
      Time_Key    time                                                                                                                                                                                                              
      Time_Format %b %d %H:%M:%S                                                                                                                                                                                                    
      Time_Offset -0500                                                                                                                                                                                                             
      Time_Keep   On    

Let me know if you run into any other issues

[Zxurian]

just using regex101, looks like the new regex detects the unraid log format. https://regex101.com/r/M4jeYB/1 I will try the new implementations this weekend.

[Polliog]

Cool, tell me if you have any problems implementing it