Integrate SHA256 checksums into build and release workflow with GitHub Actions enhancement

.github/copilot-instructions.md

@@ -2,12 +2,12 @@
 
 ## Project Overview
 
-**go-pugleaf** is a modern NNTP server and web gateway for Usenet/NetNews built in Go. It provides a complete newsgroup platform with full RFC 3977 compliant NNTP server implementation, modern web interface for browsing, efficient article fetching and threading, SQLite-based storage with per-group databases, and spam flagging/moderation tools.
+**go-pugleaf** is a modern NNTP server and web gateway for Usenet/NetNews built in Go. It provides a complete newsgroup platform with full RFC 3977 compliant NNTP server implementation, modern web interface for browsing, efficient article fetching and threading, SQLite-based storage with per-group databases, spam flagging/moderation tools, and bridge integrations for Fediverse (ActivityPub) and Matrix protocols.
 
 ### High-Level Repository Information
 - **Type**: Production newsgroup server system (medium-to-large codebase)
-- **Size**: 150 Go files across 13 internal packages, 21 command applications
-- **Languages**: Go 1.24.3+ (primary), HTML/CSS/JavaScript (web frontend), Shell scripts (build system)
+- **Size**: 149 Go files across 13 internal packages, 21 command applications
+- **Languages**: Go 1.25.0+ (primary), HTML/CSS/JavaScript (web frontend), Shell scripts (build system)
 - **Frameworks**: Gin web framework, custom NNTP protocol implementation, SQLite3 database
 - **Target Runtime**: Linux/Unix systems (Windows support not tested)
 - **Dependencies**: Minimal external dependencies - SQLite3, Gin, Go crypto libraries
@@ -16,23 +16,38 @@
 
 ### Essential Prerequisites
 ```bash
-# Always ensure Go 1.24.3+ is available
-go version  # Must be 1.24.3 or higher
+# Always ensure Go 1.25.0+ is available
+go version  # Must be 1.25.0 or higher
 
 # Always run module commands before building
 go mod tidy && go mod verify
 ```
 
 ### Build Commands (Always run in repository root)
 ```bash
-# Build all applications (14+ binaries) - takes ~15-30 seconds
+# Build all applications (13 binaries) - takes ~15-30 seconds
 ./build_ALL.sh
 
 # Build individual applications - takes ~1-2 seconds each
+# 15 individual build scripts available:
 ./build_webserver.sh      # Main web interface
 ./build_fetcher.sh        # Article fetcher
 ./build_nntp-server.sh    # NNTP server
 ./build_expire-news.sh    # Article expiration tool
+./build_merge-active.sh   # Merge active files utility
+./build_merge-descriptions.sh  # Merge newsgroup descriptions
+./build_nntpmgr.sh        # NNTP management tool
+./build_analyze.sh        # NNTP analysis tool
+./build_history-rebuild.sh     # History rebuild utility
+./build_fix-references.sh      # Fix article references
+./build_fix-thread-activity.sh # Fix thread activity
+./build_import_flat-files.sh   # Import flat files
+./build_recover-db.sh          # Database recovery
+./build_rslight_importer.sh    # RSLight importer
+./build_TestMsgIdItemCache.sh  # Message ID cache tester
+
+# Note: Some cmd applications don't have build scripts:
+# benchmark_hash, extract_hierarchies, history-demo, parsedates, test-nntp, usermgr
 
 # Build output goes to build/ directory
 ls -la build/  # List all built binaries
@@ -96,15 +111,34 @@ golangci-lint run
   ├── nntp-fetcher/     # Article fetching from NNTP providers  
   ├── nntp-server/      # NNTP server implementation
   ├── expire-news/      # Article expiration and cleanup
-  └── [17 other tools]  # Import, analysis, migration utilities
+  ├── merge-active/     # Merge active files utility
+  ├── merge-descriptions/ # Merge newsgroup descriptions utility
+  ├── nntpmgr/          # NNTP management tool
+  ├── nntp-analyze/     # NNTP analysis tool
+  ├── history-rebuild/  # History rebuild utility
+  ├── fix-references/   # Fix article references
+  ├── fix-thread-activity/ # Fix thread activity
+  ├── import-flat-files/ # Import flat files
+  ├── recover-db/       # Database recovery
+  ├── rslight-importer/ # RSLight importer
+  ├── test-MsgIdItemCache/ # Message ID cache tester
+  └── [6 other tools]   # benchmark_hash, extract_hierarchies, history-demo, 
+                        # parsedates, test-nntp, usermgr (no build scripts)
 
 /internal/              # 13 internal packages (core business logic)
   ├── nntp/            # NNTP protocol implementation (60+ files)
   ├── web/             # Web server handlers and templates  
   ├── database/        # SQLite abstraction and per-group databases
   ├── config/          # Configuration management
   ├── models/          # Data models and structures
-  └── [8 other packages] # cache, embedded, history, processor, etc.
+  ├── cache/           # Caching layer
+  ├── processor/       # Article processing
+  ├── history/         # History tracking
+  ├── utils/           # Utility functions
+  ├── preloader/       # Data preloading
+  ├── spam/            # Spam detection and filtering
+  ├── fediverse/       # Fediverse integration
+  └── matrix/          # Matrix protocol support
 
 /web/                   # Frontend assets
   ├── templates/       # HTML templates (Gin templating)
@@ -141,6 +175,11 @@ golangci-lint run
 - Overview data caching for performance
 - Migration system for schema updates
 
+**Bridge Integrations** (New in v0.4.8.6+):
+- **Fediverse Bridge** (`internal/fediverse/`): ActivityPub protocol support for bridging NNTP to Mastodon/Pleroma
+- **Matrix Bridge** (`internal/matrix/`): Matrix protocol support for bridging newsgroups to Matrix rooms
+- **Spam Filtering** (`internal/spam/`): SpamAssassin integration with fast rule-based filtering
+
 ### Dependencies Not Obvious from Layout
 - **Race Detection**: All builds use `-race` flag for concurrency safety
 - **Version Injection**: Build scripts inject version from `appVersion.txt`
@@ -154,7 +193,9 @@ golangci-lint run
 **Database Schema**: Add migrations to `internal/database/migrations/`
 **Configuration**: Update `internal/config/` and `configs/sample.yaml`
 **Build Process**: Modify individual `build_*.sh` scripts
-**New Applications**: Add to `cmd/` directory with corresponding build script
+**New Applications**: Add to `cmd/` directory with corresponding build script (build_<name>.sh)
+**Bridge Configuration**: Update `internal/fediverse/`, `internal/matrix/`, or `internal/spam/` for integration features
+**Spam Rules**: Update SpamAssassin configuration or quick rules in `internal/spam/`
 
 ### Key Files by Importance
 1. `cmd/web/main.go` - Main web application entry point
@@ -163,6 +204,9 @@ golangci-lint run
 4. `internal/database/database.go` - Database abstraction layer
 5. `build_ALL.sh` - Master build script for all applications
 6. `go.mod` - Dependencies and Go version requirements
+7. `internal/fediverse/bridge.go` - Fediverse integration (ActivityPub)
+8. `internal/matrix/bridge.go` - Matrix room bridging
+9. `internal/spam/filter.go` - Spam filtering system
 
 ### Validation and CI/CD
 Currently **no GitHub Actions or automated CI/CD**. Validation is manual:
@@ -179,7 +223,7 @@ Currently **no GitHub Actions or automated CI/CD**. Validation is manual:
 - **Per-Group Databases**: Articles stored in separate SQLite files per newsgroup
 
 ### Troubleshooting Common Issues
-- **Build Failures**: Check Go version (must be 1.24.3+), run `go mod tidy`
+- **Build Failures**: Check Go version (must be 1.25.0+), run `go mod tidy`
 - **Missing Templates**: Ensure `web/` directory present for web server
 - **Memory Issues**: Known issue with long-running fetcher and webserver processes
 - **NNTP Testing**: Use telnet to localhost:1119 for manual NNTP server testing

.github/workflows/release.yml

@@ -34,14 +34,6 @@ jobs:
         with:
           go-version: '1.25.x'
 
-      - name: Cache Go modules
-        uses: actions/cache@v4
-        with:
-          path: ~/go/pkg/mod
-          key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
-          restore-keys: |
-            ${{ runner.os }}-go-
-
       - name: Install dependencies
         run: go mod download
 
@@ -117,12 +109,33 @@ jobs:
             fi
           done
 
+      - name: Generate SHA256 checksums
+        run: |
+          OS="${{ matrix.os }}"
+          ARCH="${{ matrix.arch }}"
+          CHECKSUMS_FILE="checksums-${OS}-${ARCH}.sha256"
+
+          echo "Generating SHA256 checksums for ${OS}-${ARCH} binaries..."
+          cd build
+          sha256sum * > "../${CHECKSUMS_FILE}"
+          cd ..
+
+          echo "CHECKSUMS_FILE=${CHECKSUMS_FILE}" >> $GITHUB_ENV
+          echo ""
+          echo "Generated checksums for ${OS}-${ARCH}:"
+          echo "================================="
+          cat "${CHECKSUMS_FILE}"
+
       - name: Create release archive
         run: |
           VERSION="${{ steps.version.outputs.version }}"
           OS="${{ matrix.os }}"
           ARCH="${{ matrix.arch }}"
           ARCHIVE_NAME="go-pugleaf-v${VERSION}-${OS}-${ARCH}"
+
+          # Include checksums in the archive
+          cp "${{ env.CHECKSUMS_FILE }}" build/checksums.sha256
+
           if [ "${{ matrix.os }}" = "windows" ]; then
             zip -r "${ARCHIVE_NAME}.zip" build/ README.md LICENSE
             echo "ARCHIVE_FILE=${ARCHIVE_NAME}.zip" >> $GITHUB_ENV
@@ -135,7 +148,9 @@ jobs:
         uses: actions/upload-artifact@v4
         with:
           name: go-pugleaf-${{ matrix.os }}-${{ matrix.arch }}
-          path: ${{ env.ARCHIVE_FILE }}
+          path: |
+            ${{ env.ARCHIVE_FILE }}
+            ${{ env.CHECKSUMS_FILE }}
 
   release:
     name: Create release
@@ -163,6 +178,38 @@ jobs:
         with:
           path: artifacts
 
+      - name: Create comprehensive checksums file
+        run: |
+          echo "Creating comprehensive SHA256 checksums file..."
+          echo "# Go-Pugleaf v${{ steps.version.outputs.version }} - SHA256 Checksums" > SHA256SUMS.txt
+          echo "# Generated on $(date -u '+%Y-%m-%d %H:%M:%S UTC')" >> SHA256SUMS.txt
+          echo "" >> SHA256SUMS.txt
+
+          # Process each platform's checksums
+          for artifact_dir in artifacts/go-pugleaf-*; do
+            if [ -d "$artifact_dir" ]; then
+              platform=$(basename "$artifact_dir" | sed 's/go-pugleaf-//')
+              echo "## Platform: $platform" >> SHA256SUMS.txt
+
+              # Find and process checksums file for this platform
+              checksums_file=$(find "$artifact_dir" -name "checksums-*.sha256" | head -1)
+              if [ -f "$checksums_file" ]; then
+                echo "Processing checksums for $platform..."
+                while IFS= read -r line; do
+                  # Extract hash and filename
+                  hash=$(echo "$line" | cut -d' ' -f1)
+                  filename=$(echo "$line" | cut -d' ' -f2-)
+                  echo "$hash  $filename ($platform)" >> SHA256SUMS.txt
+                done < "$checksums_file"
+              fi
+              echo "" >> SHA256SUMS.txt
+            fi
+          done
+
+          echo "Comprehensive checksums file created:"
+          echo "====================================="
+          cat SHA256SUMS.txt
+
       - name: Create release
         uses: softprops/action-gh-release@v2
         with:
@@ -193,8 +240,27 @@ jobs:
             - macOS (amd64, arm64)
             - Windows (amd64)
 
+            ### Binary Verification:
+            All binaries include SHA256 checksums for integrity verification:
+            - **SHA256SUMS.txt** - Comprehensive checksums for all platforms
+            - **checksums-{os}-{arch}.sha256** - Individual platform checksums
+            - Each archive also contains a `checksums.sha256` file for verification after extraction
+
+            **Verification Examples:**
+            ```bash
+            # Verify individual platform binaries
+            sha256sum -c checksums-linux-amd64.sha256
+
+            # Verify after archive extraction
+            tar -xzf go-pugleaf-v${{ steps.version.outputs.version }}-linux-amd64.tar.gz
+            cd build/
+            sha256sum -c checksums.sha256
+            ```
+
             Extract the archive for your platform and run the binaries.
             See README.md for usage instructions.
-          files: artifacts/*/*
+          files: |
+            artifacts/*/*
+            SHA256SUMS.txt
           draft: false
           prerelease: false

.gitignore

@@ -29,6 +29,8 @@ go.work
 /demo_history
 internal/nntp/analysis_*
 update.tar.gz
+checksums.sha256
+checksums.sha256.archive
 active_files/local-mode.active.filtered
 active_files/local-mode.active.filtered.sorted
 active_files/local-mode.active.new

.update

@@ -1 +1 @@
-2cc6dbd00dc59836970e79b6ed101020c952164e222a52c6a0694294d8da5bfb  update.tar.gz
+3e20a56cf38c0b7be78fb3791dacccf4a9b59f6127c55bc6fb1ca9999d266958  update.tar.gz

README.md

@@ -180,6 +180,43 @@ Or build a single tool manually, e.g.:
 go build -o build/usermgr ./cmd/usermgr
 ```
 
+### Building and Release
+
+**Build all binaries:**
+```bash
+# Build all binaries (automatically generates checksums)
+./build_ALL.sh
+```
+
+**Generate checksums manually:**
+```bash
+# Generate SHA256 checksums for all executables in build/
+./createChecksums.sh
+```
+
+**Build and create release package:**
+```bash
+# Build all binaries and create release package with checksums
+./build_ALL.sh update
+```
+
+This creates:
+- `checksums.sha256` - SHA256 hashes for all individual executables (with build/ paths)
+- `checksums.sha256.archive` - SHA256 hashes with relative paths for archive inclusion
+- `update.tar.gz` - Compressed archive of all binaries including checksums.sha256
+- `.update` - SHA256 hash of the tar.gz file
+
+**Verify checksums:**
+```bash
+# Verify all executable checksums (from repository root)
+sha256sum -c checksums.sha256
+
+# Verify checksums after extracting release archive
+tar -xzf update.tar.gz
+cd extracted-directory/
+sha256sum -c checksums.sha256  # Verify all executables in release
+```
+
 ## 📚 Binary Documentation
 
 go-pugleaf includes command-line applications for various newsgroup management tasks.

build_ALL.sh

@@ -17,4 +17,8 @@ rm -v build/*
 ./build_recover-db.sh
 ./build_expire-news.sh
 
+# Always generate checksums after building
+echo "Generating checksums for built executables..."
+./createChecksums.sh
+
 test "$1" = "update" && ./createUpdate.sh

createChecksums.sh

@@ -0,0 +1,45 @@
+#!/bin/sh -e
+
+# Generate SHA256 checksums for all executables in build directory
+
+BUILD_DIR="build"
+CHECKSUMS_FILE="checksums.sha256"
+
+echo "Generating SHA256 checksums for executables in $BUILD_DIR/"
+
+# Check if build directory exists
+if [ ! -d "$BUILD_DIR" ]; then
+    echo "Error: Build directory '$BUILD_DIR' not found"
+    echo "Please run ./build_ALL.sh first to build all executables"
+    exit 1
+fi
+
+# Check if there are any files in build directory
+if [ ! "$(ls -A $BUILD_DIR)" ]; then
+    echo "Error: Build directory '$BUILD_DIR' is empty"
+    echo "Please run ./build_ALL.sh first to build all executables"
+    exit 1
+fi
+
+# Remove existing checksums file
+rm -f "$CHECKSUMS_FILE"
+
+# Generate checksums for all files in build directory
+echo "Creating $CHECKSUMS_FILE with SHA256 hashes..."
+sha256sum "$BUILD_DIR"/* > "$CHECKSUMS_FILE"
+
+# Also create a version with relative paths for inclusion in the release archive
+cd "$BUILD_DIR"
+sha256sum * > "../${CHECKSUMS_FILE}.archive"
+cd ..
+
+# Display the checksums file
+echo ""
+echo "Checksums generated successfully:"
+echo "================================="
+cat "$CHECKSUMS_FILE"
+
+echo ""
+echo "Checksums file created: $CHECKSUMS_FILE"
+echo "Archive checksums file created: ${CHECKSUMS_FILE}.archive (for inclusion in release)"
+echo "Number of executables: $(wc -l < $CHECKSUMS_FILE)"

createUpdate.sh

@@ -1,8 +1,20 @@
 #!/bin/sh -e
+# Generate checksums for individual executables
+echo "Generating checksums for individual executables..."
+./createChecksums.sh
+
+# Copy archive checksums file into build directory for inclusion in release archive
+echo "Including checksums in release archive..."
+cp checksums.sha256.archive build/checksums.sha256
+
 rm -f update.tar.gz
 tar -f update.tar.gz -C build -c -v -z .
 sha256sum update.tar.gz > .update
 HASH=$(cat .update|cut -d" " -f1)
 du -b update.tar.gz
 cat .update
+echo ""
+echo "Release archive created with individual executable checksums included"
+echo "Individual executable checksums available in: checksums.sha256"
+echo "Checksums are also included in the release archive as checksums.sha256"
 exit 0