Skip to content

Release pixelator 2.0 #138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 73 commits into from
Jun 23, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
9a2b63d
Template update for nf-core/tools version 3.1.2
nf-core-bot Jan 20, 2025
d6c04f4
Bump version to `1.5.0dev`
fbdtemme Jan 22, 2025
903bef3
Merge pull request #127 from nf-core/sync-master
fbdtemme Jan 23, 2025
cbc915a
Template update for nf-core/tools version 3.2.0
nf-core-bot Jan 27, 2025
419381b
Template update for nf-core/tools version 3.2.1
nf-core-bot Apr 30, 2025
63e6ed4
Merge branch 'dev' into nf-core-template-merge-3.2.1
johandahlberg May 6, 2025
bae2d5d
Use v4 of github cache action
johandahlberg May 6, 2025
1b63118
Bring files in line with template
johandahlberg May 6, 2025
b6dc70d
Update lint config
johandahlberg May 6, 2025
e8d2792
Fix up README
johandahlberg May 6, 2025
cc20d47
Merge pull request #132 from nf-core/nf-core-template-merge-3.2.1
johandahlberg May 6, 2025
1f40be9
Move single-cell to single-cell-mpx
johandahlberg Apr 10, 2025
c17a027
Adding pna modules
johandahlberg Apr 10, 2025
fcd6ed7
WIP trying to get mpx to run
johandahlberg Apr 10, 2025
7584084
harshilAlignment on in vscode nextflow plugin
johandahlberg Apr 10, 2025
99e7a06
MPX branch of pipeline executes
johandahlberg Apr 10, 2025
355436c
Update container used
johandahlberg Apr 14, 2025
f307357
Update pixelator containers and test data paths
johandahlberg Apr 15, 2025
0dc0f71
wip updating test data
johandahlberg Apr 16, 2025
e963f64
Update config files for new testdata
johandahlberg Apr 17, 2025
644b2d5
Update underlying pixelator container
johandahlberg Apr 17, 2025
f008912
Update testdata paths
johandahlberg Apr 17, 2025
df589cc
Remove channel dump
johandahlberg Apr 17, 2025
c95c61e
Update schema.json
johandahlberg Apr 17, 2025
a1c0f3a
Update to pixelator 0.20.1 containers
johandahlberg May 6, 2025
0398281
Fix up paths to test data
johandahlberg May 6, 2025
dc26752
Update test snapshots
johandahlberg May 6, 2025
ac9a29c
Syncing nextflow.config and schema file
johandahlberg May 6, 2025
ab31a96
Add missing pna modules conf
johandahlberg May 8, 2025
1cf0097
Fix up default tests
johandahlberg May 15, 2025
822fa56
Fix up save all tests
johandahlberg May 15, 2025
1ddd381
WIP Setting up workflow test for PNA
johandahlberg May 16, 2025
2c8fe8e
Fix pipeline tests for PNA flow
johandahlberg May 16, 2025
790ec74
Update usage instructions
johandahlberg May 19, 2025
814b1ad
Split output into sections for PNA and MPX
johandahlberg May 19, 2025
a104877
Add back base paths in tests config
johandahlberg May 19, 2025
9dc8031
Fix formatting
johandahlberg May 19, 2025
e0c6f96
Update usage docs
johandahlberg May 19, 2025
5d276ea
Merge pull request #131 from PixelgenTechnologies/update-to-pixelator…
johandahlberg May 19, 2025
e22d71c
Update README to include information about PNA
johandahlberg May 19, 2025
1e3c403
Check for conda prof. and add warning to README
johandahlberg May 19, 2025
09e0269
Merge pull request #133 from PixelgenTechnologies/update-readme
johandahlberg May 20, 2025
d6479e2
Update the version targets
johandahlberg May 22, 2025
9e4142d
Prepare the changelog
johandahlberg May 22, 2025
b234ddf
Load missing pna modules
johandahlberg May 22, 2025
1a790b8
Fix up the tests
johandahlberg May 23, 2025
16f1d82
Merge pull request #135 from PixelgenTechnologies/bug-pna-modules-not…
johandahlberg May 23, 2025
dcc720d
Merge pull request #134 from PixelgenTechnologies/prepare-change-log
johandahlberg May 23, 2025
12ab86d
Disable running conda tests in CI tests
fbdtemme May 27, 2025
63525f0
Fix typo
fbdtemme May 27, 2025
0761336
Merge pull request #136 from PixelgenTechnologies/disable-conda-in-ci
fbdtemme May 27, 2025
0160485
Bump version to 2.0.0
fbdtemme May 27, 2025
78d7210
Merge pull request #137 from nf-core/bump-version-2.0
fbdtemme May 27, 2025
b194a6d
Fix typo in changelog
fbdtemme May 27, 2025
a7c6679
fix: add leading slash to `input` and `input_basedir` to handle base_…
fbdtemme Jun 18, 2025
75565a0
Apply suggestions from code review
fbdtemme Jun 18, 2025
16a4071
fix: add missing `save_json` default in nextflow.config
fbdtemme Jun 18, 2025
28643a6
Tweak changelog 2.0.0 release summary
fbdtemme Jun 18, 2025
e754896
Remove redundant null check in modules.pna.config
fbdtemme Jun 18, 2025
52693c0
Reword introduction
fbdtemme Jun 18, 2025
00292e0
Strip excessive whitespace from output.md
fbdtemme Jun 18, 2025
27f11cd
Fix missing validation of pnx and mpx layout_algorithm parameter
fbdtemme Jun 18, 2025
284e265
Remove redundant samplesheet_sha injection
fbdtemme Jun 18, 2025
7eda6ba
Remove redundant input transforms for AMPLICON
fbdtemme Jun 18, 2025
e2ebd24
Replace camelcase local variables with snake_case
fbdtemme Jun 18, 2025
379107c
Replace camelcase local variables with snake_case and strip some whit…
fbdtemme Jun 18, 2025
009014f
Fix typo and strip commented out code
fbdtemme Jun 18, 2025
619e13c
Comment out conda directives in all pixelator processes
fbdtemme Jun 18, 2025
f62cf75
Unify `--save_all` and `save_json` flags across pna and mpx workflows
fbdtemme Jun 18, 2025
1267634
Set save_json to true by default and disable `save_all`
fbdtemme Jun 18, 2025
48241b0
Update pipeline test snapshots
fbdtemme Jun 18, 2025
56f96fb
Remove edits to utils_nfcore_pipeline
fbdtemme Jun 18, 2025
837787e
Update 2.0.0 release date in CHANGELOG
fbdtemme Jun 23, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 10 additions & 31 deletions .github/workflows/awsfulltest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,57 +4,36 @@ name: nf-core AWS full size tests
# It runs the -profile 'test_full' on AWS batch

on:
pull_request:
branches:
- main
- master
workflow_dispatch:
pull_request_review:
types: [submitted]
release:
types: [published]

jobs:
run-platform:
name: Run AWS full tests
# run only if the PR is approved by at least 2 reviewers and against the master branch or manually triggered
if: github.repository == 'nf-core/pixelator' && github.event.review.state == 'approved' && github.event.pull_request.base.ref == 'master' || github.event_name == 'workflow_dispatch'
# run only if the PR is approved by at least 2 reviewers and against the master/main branch or manually triggered
if: github.repository == 'nf-core/pixelator' && github.event.review.state == 'approved' && (github.event.pull_request.base.ref == 'master' || github.event.pull_request.base.ref == 'main') || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
steps:
- name: Get PR reviews
uses: octokit/request-action@v2.x
if: github.event_name != 'workflow_dispatch'
id: check_approvals
continue-on-error: true
with:
route: GET /repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/reviews?per_page=100
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: Check for approvals
if: ${{ failure() && github.event_name != 'workflow_dispatch' }}
run: |
echo "No review approvals found. At least 2 approvals are required to run this action automatically."
exit 1

- name: Check for enough approvals (>=2)
id: test_variables
if: github.event_name != 'workflow_dispatch'
- name: Set revision variable
id: revision
run: |
JSON_RESPONSE='${{ steps.check_approvals.outputs.data }}'
CURRENT_APPROVALS_COUNT=$(echo $JSON_RESPONSE | jq -c '[.[] | select(.state | contains("APPROVED")) ] | length')
test $CURRENT_APPROVALS_COUNT -ge 2 || exit 1 # At least 2 approvals are required
echo "revision=${{ (github.event_name == 'workflow_dispatch' || github.event_name == 'release') && github.sha || 'dev' }}" >> "$GITHUB_OUTPUT"

- name: Launch workflow via Seqera Platform
uses: seqeralabs/action-tower-launch@v2
with:
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
compute_env: ${{ secrets.TOWER_COMPUTE_ENV }}
revision: ${{ github.sha }}
workdir: s3://${{ secrets.AWS_S3_BUCKET }}/work/pixelator/work-${{ github.sha }}
revision: ${{ steps.revision.outputs.revision }}
workdir: s3://${{ secrets.AWS_S3_BUCKET }}/work/pixelator/work-${{ steps.revision.outputs.revision }}
parameters: |
{
"hook_url": "${{ secrets.MEGATESTS_ALERTS_SLACK_HOOK_URL }}",
"outdir": "s3://${{ secrets.AWS_S3_BUCKET }}/pixelator/results-${{ github.sha }}"
"outdir": "s3://${{ secrets.AWS_S3_BUCKET }}/pixelator/results-${{ steps.revision.outputs.revision }}"
}
profiles: test_full

Expand Down
25 changes: 8 additions & 17 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,32 +36,23 @@ jobs:
- "24.04.2"
- "latest-everything"
filter: ["workflow", "function", "pipeline"]
profile: ["conda", "docker", "singularity"]
# TODO: Add conda back when pixelator is available on bioconda again
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assuming still not?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sadly not, yes. We depend on an up-to-date version of faiss and that is packaged in the pytorch conda channel, but we cannot use that in our bioconda recipes.

profile: ["docker", "singularity"]
shard: [1, 2, 3, 4, 5]
isMaster:
- ${{ github.base_ref == 'master' }}
exclude:
- isMaster: false
profile: "conda"
# - isMaster: false
# profile: "conda"
- isMaster: false
profile: "singularity"

steps:
- name: Check out pipeline code
uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
with:
fetch-depth: 0

- uses: actions/setup-python@f677139bbe7f9c59b41e40162b753c062f5d49a3 # v5
with:
python-version: "3.11"
architecture: "x64"

- uses: actions/setup-java@8df1039502a15bceb9433410b1a100fbe190c53b # v4
with:
distribution: "temurin"
java-version: "17"

- name: Set up Nextflow
uses: nf-core/setup-nextflow@v2
with:
Expand All @@ -83,7 +74,7 @@ jobs:
mkdir -p $NXF_SINGULARITY_LIBRARYDIR

- name: Cache pdiff
uses: actions/cache@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # v4
uses: actions/cache@v4
id: cache-pip-pdiff
with:
path: ~/.cache/pip
Expand All @@ -110,8 +101,8 @@ jobs:
- name: Disk space cleanup
uses: jlumbroso/free-disk-space@v1.3.1

- name: Start summary
id: print-test
- name: "Run pipeline with test data ${{ matrix.NXF_VER }} | ${{ matrix.test_name }} | ${{ matrix.profile }}"
continue-on-error: ${{ matrix.NXF_VER == 'latest-everything' }}
run: |
echo "## nf-test tests summary :rocket:" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/linting_comment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Download lint results
uses: dawidd6/action-download-artifact@80620a5d27ce0ae443b965134db88467fc607b43 # v7
uses: dawidd6/action-download-artifact@20319c5641d495c8a52e688b7dc5fada6c3a9fbc # v8
with:
workflow: linting.yml
workflow_conclusion: completed
Expand Down
33 changes: 0 additions & 33 deletions .github/workflows/release-announcements.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,39 +27,6 @@ jobs:

${{ steps.get_topics.outputs.topics }} #nfcore #openscience #nextflow #bioinformatics

send-tweet:
runs-on: ubuntu-latest

steps:
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5
with:
python-version: "3.10"
- name: Install dependencies
run: pip install tweepy==4.14.0
- name: Send tweet
shell: python
run: |
import os
import tweepy

client = tweepy.Client(
access_token=os.getenv("TWITTER_ACCESS_TOKEN"),
access_token_secret=os.getenv("TWITTER_ACCESS_TOKEN_SECRET"),
consumer_key=os.getenv("TWITTER_CONSUMER_KEY"),
consumer_secret=os.getenv("TWITTER_CONSUMER_SECRET"),
)
tweet = os.getenv("TWEET")
client.create_tweet(text=tweet)
env:
TWEET: |
Pipeline release! ${{ github.repository }} v${{ github.event.release.tag_name }} - ${{ github.event.release.name }}!

Please see the changelog: ${{ github.event.release.html_url }}
TWITTER_CONSUMER_KEY: ${{ secrets.TWITTER_CONSUMER_KEY }}
TWITTER_CONSUMER_SECRET: ${{ secrets.TWITTER_CONSUMER_SECRET }}
TWITTER_ACCESS_TOKEN: ${{ secrets.TWITTER_ACCESS_TOKEN }}
TWITTER_ACCESS_TOKEN_SECRET: ${{ secrets.TWITTER_ACCESS_TOKEN_SECRET }}

bsky-post:
runs-on: ubuntu-latest
steps:
Expand Down
13 changes: 8 additions & 5 deletions .nf-core.yml
Original file line number Diff line number Diff line change
@@ -1,13 +1,16 @@
lint:
actions_ci: false
multiqc_config: false
files_unchanged:
- assets/nf-core-pixelator_logo_light.png
files_exist:
- assets/multiqc_config.yml
- conf/igenomes.config
- conf/igenomes_ignored.config
nf_core_version: 3.1.1
files_unchanged:
- assets/nf-core-pixelator_logo_light.png
# Ignore this for now since we haven't switched the name of the
# main branch to main from master yet
- .github/PULL_REQUEST_TEMPLATE.md
multiqc_config: false
nf_core_version: 3.2.1
repository_type: pipeline
template:
author: Pixelgen Technologies AB
Expand All @@ -21,4 +24,4 @@ template:
- fastqc
- multiqc
- igenomes
version: 1.4.0
version: 2.0.0
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ repos:
- prettier@3.2.5

- repo: https://github.com/editorconfig-checker/editorconfig-checker.python
rev: "3.0.3"
rev: "3.1.2"
hooks:
- id: editorconfig-checker
alias: ec
Expand Down
6 changes: 5 additions & 1 deletion .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
{
"markdown.styles": ["public/vscode_markdown.css"]
"markdown.styles": [
"public/vscode_markdown.css"
],
"nextflow.formatting.harshilAlignment": true
}

71 changes: 71 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,77 @@
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [[2.0.0](https://github.com/nf-core/pixelator/releases/tag/2.0.0)] - 2025-06-23

This release is a major update of the nf-core/pixelator pipeline. It brings in support for the
Proximity Network Analysis (PNA) workflow in addition to the Molecular Pixelation (MPX) workflow.
Any run of the pipeline will now run the appropriate workflow depending on which design is specified in the input samplesheet.

### Enhancements & fixes

- [[PR #131](https://github.com/nf-core/pixelator/pull/131)] - Update to pixelator 0.20.1, and add PNA workflows
- [[PR #132](https://github.com/nf-core/pixelator/pull/132)] - Template update for nf-core/tools v3.2.1
- [[PR #133](https://github.com/nf-core/pixelator/pull/133)] - Documentation updates

### Parameters

| Old parameter | New parameter |
| ------------- | ----------------------------------------------------------- |
| | `--save_pna_demux_parquet` |
| | `--save_pna_demux_passed_reads` |
| | `--save_pna_demux_failed_reads` |
| | `--save_pna_collapsed_reads` |
| | `--save_pna_graph_pixelfile` |
| | `--save_pna_analysis_pixelfile` |
| | `--save_json` |
| | `--pna_amplicon_mismatches` |
| | `--pna_amplicon_remove_polyg` |
| | `--pna_amplicon_quality_cutoff` |
| | `--pna_demux_mismatches` |
| | `--pna_demux_output_chunk_reads` |
| | `--pna_demux_output_max_chunks` |
| | `--pna_demux_strategy` |
| | `--pna_collapse_mismatches` |
| | `--pna_collapse_algorithm` |
| | `--pna_graph_multiplet_recovery` |
| | `--pna_graph_leiden_iterations` |
| | `--pna_graph_initial_stage_leiden_resolution` |
| | `--pna_graph_refinement_stage_leiden_resolution` |
| | `--pna_graph_min_count` |
| | `--pna_graph_min_component_size_in_refinement` |
| | `--pna_graph_max_refinement_recursion_depth` |
| | `--pna_graph_initial_stage_max_edges_to_remove` |
| | `--pna_graph_refinement_stage_max_edges_to_remove` |
| | `--pna_graph_initial_stage_max_edges_to_remove_relative` |
| | `--pna_graph_refinement_stage_max_edges_to_remove_relative` |
| | `--pna_graph_graph_min_component_size_to_prune` |
| | `--pna_graph_component_size_min_threshold` |
| | `--pna_analysis_compute_proximity` |
| | `--pna_analysis_proximity_nbr_of_permutations` |
| | `--pna_analysis_compute_k_cores` |
| | `--pna_analysis_compute_svd_var_explained` |
| | `--pna_analysis_svd_nbr_of_pivots` |
| | `--pna_layout_no_node_marker_counts` |
| | `--pna_layout_layout_algorithm` |
| | `--pna_layout_pmds_pivots` |
| | `--pna_layout_wpmds_k` |

> [!NOTE]
> Parameter has been **updated** if both old and new parameter information is present.
> Parameter has been **added** if just the new parameter information is present.
> Parameter has been **removed** if new parameter information isn't present.

### Software dependencies

| Dependency | Old version | New version |
| ----------- | ----------- | ----------- |
| `pixelator` | 0.19.0 | 0.20.1 |

> [!NOTE]
> Dependency has been **updated** if both old and new parameter information is present.
> Dependency has been **added** if just the new parameter information is present.
> Dependency has been **removed** if new parameter information isn't present.

## [[1.4.0](https://github.com/nf-core/pixelator/releases/tag/1.4.0)] - 2024-01-22

### Enhancements & fixes
Expand Down
Loading