Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg

# Virtual environments
.venv/
venv/
ENV/
env/

# IDE
.vscode/
.idea/
*.swp
*.swo
*~

# Testing
.pytest_cache/
.coverage
htmlcov/
.coverage.*
coverage.xml
*.cover
.hypothesis/

# Documentation
docs/_build/

# Git
.git/
.gitignore

# OS
.DS_Store
Thumbs.db

# Project specific
test_dest/
*.log

20 changes: 17 additions & 3 deletions .github/workflows/python-app.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,29 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python }}
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python }}
cache: 'pip'
- name: Cache pip dependencies
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt', '**/pyproject.toml') }}
restore-keys: |
${{ runner.os }}-pip-
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install flake8 pytest pytest-cov pytest-integration
pip install flake8 pytest pytest-cov pytest-integration mypy ruff
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
pip install -e .
pip install -e ".[test]"
- name: Lint with ruff
run: |
ruff check . --output-format=github
- name: Type check with mypy
run: |
mypy gitlabber/ --ignore-missing-imports || true
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
Expand Down
55 changes: 53 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,46 @@
<!--next-version-placeholder-->
## [Unreleased]

## [2.0.0] - 2025-01-XX
## [2.0.1] - 2025-11-19

### Added
- **Secure Token Storage**: Store GitLab tokens securely using OS-native keyring
- macOS: Keychain
- Linux: Secret Service API (GNOME Keyring, KWallet)
- Windows: Windows Credential Manager
- **Automatic Token Retrieval**: Tokens are automatically retrieved from secure storage if no CLI token is provided
- **Token Resolution Priority**: CLI β†’ Stored β†’ Environment Variable
- **`--store-token` CLI Flag**: Store tokens securely in OS keyring (requires `keyring` package)
- **Docker Testing Infrastructure**: Test on Ubuntu environment matching CI
- `Dockerfile.test` for Ubuntu/Python 3.11 testing environment
- `docker-compose.test.yml` for easy test execution
- `scripts/test-docker.sh` helper script
- **Comprehensive Token Storage Documentation**: Added usage examples and best practices to README files
- **Docker Testing Documentation**: Added guide to `DEVELOPMENT.md` for running tests in Docker

### Changed
- Token resolution now includes secure storage as a source (automatic fallback)
- Updated GitHub Actions workflow:
- Use `actions/setup-python@v5` (from v2)
- Use `actions/checkout@v4` (from v3)
- Added pip caching with `actions/cache@v4` for faster builds
- Added `ruff` linting step
- Added `mypy` type checking step
- Install test dependencies with `[test]` extra

### Fixed
- Fixed all previously skipped CLI tests (`test_version_option`, `test_missing_token_error`, `test_missing_url_error`, `test_missing_dest_error`, `test_print_tree`, `test_sync_tree`)
- Fixed all previously skipped integration tests (`test_help`, `test_version`)
- Fixed environment isolation issues that caused CI test failures
- Fixed token resolution to properly check secure storage before falling back to environment variables
- Fixed all ruff linting errors (unused imports, f-strings without placeholders, equality comparisons)

### Security
- Tokens stored encrypted at rest by OS keyring
- Graceful fallback to environment variables if keyring unavailable
- No breaking changes to existing security practices

## [2.0.0] - 2025-11-18

### Added
- **Major Performance Feature**: Add `--api-concurrency` option for parallel API calls during tree building. This dramatically speeds up tree discovery for large GitLab instances with many groups and subgroups. Real-world performance improvements: **4-6x speedup** (e.g., 96s β†’ 16-21s for instances with 21+ subgroups). The feature includes:
Expand Down Expand Up @@ -265,7 +304,19 @@
### Fixed
### Security

[unreleased]: https://github.com/ezbz/gitlabber/compare/v1.1.8...HEAD
[unreleased]: https://github.com/ezbz/gitlabber/compare/v2.0.1...HEAD
[2.0.1]: https://github.com/ezbz/gitlabber/compare/v2.0.0...v2.0.1
[2.0.0]: https://github.com/ezbz/gitlabber/compare/v1.2.8...v2.0.0
[1.2.8]: https://github.com/ezbz/gitlabber/compare/v1.2.7...v1.2.8
[1.2.7]: https://github.com/ezbz/gitlabber/compare/v1.2.6...v1.2.7
[1.2.6]: https://github.com/ezbz/gitlabber/compare/v1.2.5...v1.2.6
[1.2.5]: https://github.com/ezbz/gitlabber/compare/v1.2.4...v1.2.5
[1.2.4]: https://github.com/ezbz/gitlabber/compare/v1.2.3...v1.2.4
[1.2.3]: https://github.com/ezbz/gitlabber/compare/v1.2.2...v1.2.3
[1.2.2]: https://github.com/ezbz/gitlabber/compare/v1.2.1...v1.2.2
[1.2.1]: https://github.com/ezbz/gitlabber/compare/v1.2.0...v1.2.1
[1.2.0]: https://github.com/ezbz/gitlabber/compare/v1.1.9...v1.2.0
[1.1.9]: https://github.com/ezbz/gitlabber/compare/v1.1.8...v1.1.9
[1.1.8]: https://github.com/ezbz/gitlabber/compare/v1.1.7...v1.1.8
[1.1.7]: https://github.com/ezbz/gitlabber/compare/v1.1.6...v1.1.7
[1.1.6]: https://github.com/ezbz/gitlabber/compare/v1.1.4...v1.1.6
Expand Down
23 changes: 23 additions & 0 deletions Dockerfile.test
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Dockerfile for testing on Ubuntu (matching CI environment)
FROM python:3.11-slim

# Install system dependencies
RUN apt-get update && apt-get install -y \
git \
build-essential \
&& rm -rf /var/lib/apt/lists/*

# Set working directory
WORKDIR /app

# Copy requirements first for better caching
COPY pyproject.toml ./
RUN pip install --no-cache-dir --upgrade pip setuptools wheel

# Install project in editable mode with test dependencies
COPY . .
RUN pip install --no-cache-dir -e ".[test]"

# Default command: run tests
CMD ["python", "-m", "pytest", "tests/", "-v"]

47 changes: 45 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Gitlabber

[![Python App](https://github.com/ezbz/gitlabber/actions/workflows/python-app.yml/badge.svg?branch=master)](https://github.com/ezbz/gitlabber/actions/workflows/python-app.yml)
[![Python App](https://github.com/ezbz/gitlabber/actions/workflows/python-app.yml/badge.svg?branch=main)](https://github.com/ezbz/gitlabber/actions/workflows/python-app.yml)
[![codecov](https://codecov.io/gh/ezbz/gitlabber/branch/main/graph/badge.svg)](https://codecov.io/gh/ezbz/gitlabber)
[![PyPI version](https://badge.fury.io/py/gitlabber.svg)](https://badge.fury.io/py/gitlabber)
[![PyPI downloads](https://img.shields.io/pypi/dm/gitlabber)](https://pypi.org/project/gitlabber/)
[![License](https://img.shields.io/pypi/l/gitlabber.svg)](https://pypi.python.org/pypi/gitlabber/)
[![Python versions](https://img.shields.io/pypi/pyversions/gitlabber)](https://pypi.python.org/pypi/gitlabber/)
[![Documentation Status](https://readthedocs.org/projects/gitlabber/badge/?version=latest&style=plastic)](https://app.readthedocs.org/projects/gitlabber/)
Expand Down Expand Up @@ -33,6 +34,11 @@ Gitlabber clones or pulls all projects under a subset of groups / subgroups by b
pip install -e .
```

* Optional: Install with secure token storage support:
```bash
pip install gitlabber[keyring]
```

* You'll need to create an [access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) from GitLab with API scopes `read_repository` and `read_api` (or `api`, for GitLab versions <12.0)

## Quick Start
Expand All @@ -57,6 +63,34 @@ Arguments can be provided via the CLI arguments directly or via environment vari
| include | -i | `GITLABBER_INCLUDE` |
| exclude | -x | `GITLABBER_EXCLUDE` |

### Secure Token Storage

Gitlabber supports secure token storage using OS-native keyring (Keychain on macOS, Secret Service on Linux, Windows Credential Manager). This allows you to store your GitLab token securely and avoid passing it via CLI or environment variables.

**Token Resolution Priority:**
1. CLI argument (`-t/--token`) - highest priority
2. Stored token (from secure storage)
3. Environment variable (`GITLAB_TOKEN`)

**Usage:**
```bash
# Install keyring (optional, for secure storage)
pip install gitlabber[keyring]

# Store token securely (one-time setup)
gitlabber --store-token -u https://gitlab.com
Enter token: [hidden input]
Token stored securely in keyring for https://gitlab.com βœ“

# Use stored token automatically (no -t flag needed)
gitlabber -u https://gitlab.com .

# Override with CLI token if needed
gitlabber -t <token> -u https://gitlab.com .
```

**Note:** If keyring is not installed, gitlabber falls back to environment variables or CLI arguments (current behavior).

To view the tree run the command with your includes/excludes and the `-p` flag. It will print your tree like so:

```bash
Expand Down Expand Up @@ -85,7 +119,7 @@ root [http://gitlab.my.com]

```bash
usage: gitlabber [-h] [-t token] [-T] [-u url] [--verbose] [-p] [--print-format {json,yaml,tree}] [-n {name,path}] [-m {ssh,http}]
[-a {include,exclude,only}] [-i csv] [-x csv] [-c N] [--api-concurrency N] [-r] [-F] [-d] [-s] [-g term] [-U] [-o options] [--version]
[-a {include,exclude,only}] [-i csv] [-x csv] [-c N] [--api-concurrency N] [-r] [-F] [-d] [-s] [-g term] [-U] [-o options] [--version] [--store-token]
[dest]

Gitlabber - clones or pulls entire groups/projects tree from gitlab
Expand Down Expand Up @@ -126,6 +160,7 @@ options:
-o options, --git-options options
provide additional options as csv for the git command
--version print the version
--store-token store token securely in OS keyring (requires keyring package)
```

### Examples
Expand Down Expand Up @@ -160,7 +195,15 @@ gitlabber --api-concurrency 10 -t <token> -u <url> .

# Use both API and git concurrency for maximum performance
gitlabber --api-concurrency 5 -c 10 -t <token> -u <url> .

# Store token securely for future use (one-time setup)
gitlabber --store-token -u https://gitlab.com

# Use stored token (no -t flag needed)
gitlabber -u https://gitlab.com .
```
<|tool▁call▁begin|>
run_terminal_cmd

## Common Use Cases

Expand Down
Loading