Baysor Not Running on Windows

[marsdenl]

Description

Hey @quentinblampe, here's a recap on the Baysor running issue on Windows System. It seems baysor is not being run properly and hence, no polygons are being generated.

Steps leading to issue on API:

sdata = sopa.io.merscope("Z:/Queries/Data/Batch5_region3/region_3/")
sopa.make_transcript_patches(
sdata,
patch_width=1000,
prior_shapes_key="cell_id",
unassigned_value=-1
)
sopa.segmentation.baysor(sdata, scale=6.25)

AssertionError: Could not find the segmentation polygons file in C:\Users\marsdenl.sopa_cache\12dec02d-0281-4d2c-bd54-dfc887679fea\transcript_patches\0

The equivalent commands on the CLI lead to the same error.

FYI:
sopa.utils.get_cache_dir(sdata)
WindowsPath('C:/Users/marsdenl/.sopa_cache/12dec02d-0281-4d2c-bd54-dfc887679fea')

As you suggested, I tried running baysor individually on one of the generated patches in the cache directory and it did start running but I then got:

baysor run -c config.toml transcripts.csv

[11:18:35] Info: Run R2d01a4d04
[11:18:35] Info: (2025-01-28) Run Baysor v0.7.0
[11:18:35] Info: Loading data...
ERROR: gene column '["g", "e", "n", "e"]' not found in the data frame
Stacktrace:
[1] error(s::String)
@ Base .\error.jl:35
[2] read_spatial_df(data_path::String; x_col::Symbol, y_col::Symbol, z_col::Symbol, gene_col::Symbol, filter_cols::Bool, drop_z::Bool)
@ Baysor.DataLoading C:\Users\marsdenl.julia\packages\Baysor\gQh5m\src\data_loading\data.jl:62
[3] read_spatial_df
@ C:\Users\marsdenl.julia\packages\Baysor\gQh5m\src\data_loading\data.jl:42 [inlined]
[4] load_df(data_path::String; min_molecules_per_gene::Int64, exclude_genes::Vector{String}, kwargs::@kwargs{x_col::Symbol, y_col::Symbol, z_col::Symbol, gene_col::Symbol, drop_z::Bool, filter_cols::Bool})
@ Baysor.DataLoading C:\Users\marsdenl.julia\packages\Baysor\gQh5m\src\data_loading\data.jl:91
[5] load_df
@ C:\Users\marsdenl.julia\packages\Baysor\gQh5m\src\data_loading\data.jl:90 [inlined]
[6] load_df(coordinates::String, data_opts::Baysor.Utils.DataOptions; kwargs::@kwargs{filter_cols::Bool})
@ Baysor.DataLoading C:\Users\marsdenl.julia\packages\Baysor\gQh5m\src\data_loading\cli_wrappers.jl:204
[7] run(coordinates::String, prior_segmentation::String; config::Baysor.Utils.RunOptions, x_column::Symbol, y_column::Symbol, z_column::Symbol, gene_column::Symbol, min_molecules_per_cell::Int64, scale::Float64, scale_std::String, n_clusters::Int64, prior_segmentation_confidence::Float64, output::String, plot::Bool, polygon_format::String, count_matrix_format::String)
@ Baysor.CommandLine C:\Users\marsdenl.julia\packages\Baysor\gQh5m\src\cli\main.jl:100
[8] run
@ C:\Users\marsdenl.julia\packages\Baysor\gQh5m\src\cli\main.jl:51 [inlined]
[9] command_main(ARGS::Vector{String})
@ Baysor C:\Users\marsdenl.julia\packages\Comonicon\F3QqZ\src\codegen\julia.jl:343
[10] command_main()
@ Baysor C:\Users\marsdenl.julia\packages\Comonicon\F3QqZ\src\codegen\julia.jl:90
[11] top-level scope
@ none:1

The gene column doesn't seem too weird:
transcripts.csv

So I then tried on MacOS. I couldn't transfer a whole sample to MacsOS as I am out of memory so I generated a .zarr file and exported it on my windows system then I downloaded it to my Mac. Once on MasOS, I generated transcript patches and ran baysor on a single patch (all patches would take quite a while) and it seemed to have worked fine. I encountered no errors, although I did not inspect generated polygons (frankly, I did not know where to find them...). Steps on Mac:

sdata = spatialdata.read_zarr("b5_r0.zarr")
sopa.make_transcript_patches(
sdata,
patch_width=1000,
prior_shapes_key="cell_id",
unassigned_value=-1
)
sopa.segmentation.baysor(sdata, scale=6.25, patch_index=2)

Hope this helps a bit! Let me know if anything is unlcear /

System

• OS: Windows and MacOS

[quentinblampey]

Thanks @marsdenl for investigating! Two interesting things:

• The error log ERROR: gene column '["g", "e", "n", "e"]' not found in the data frame shows that the string "gene" is turned into a list at some point. Could you please send me the config.toml file inside one of the patch directory?
• So Baysor is indeed failing, but when running Baysor through Sopa, it doesn't detect that Baysor is failing. Normally, Sopa should detect that, and provide a detailed error log, which is not the case here.

I'll try to have access to a Windows computer on Friday to investigate on this!

[marsdenl]

Yep you're absolutely right, the config from the patched files has separated each letter, see attached:
config.txt
I switched it back to "gene" and baysor worked perfectly on that patch :). I repeated the steps to make sure this was reproducible and it seems that this switch into a "g","e","n","e" list format happened only if i don't specify a config (but do specificy the scale):

sopa.segmentation.baysor(sdata, scale=6.25)

If I specify the config file myself then "gene" is in the right format and baysor runs fine (if I execute it manually on that patch).

sopa.segmentation.baysor(sdata, config = "Z:/Queries/example_config.toml")

Hope this helps!

[quentinblampey]

Great, we make some progress!

I see two possible issues. Could you please try the commands below? It should help me understand the issue.

import sopa
from sopa.utils import get_feature_key
import toml

sdata = sopa.io.toy_dataset()

feature_key = get_feature_key(sdata["transcripts"])
print(feature_key) # should print "genes"

config = {
    "data": {
        "gene": feature_key,
    },
}

with open("test_config.toml", "w") as f:
    toml.dump(config, f)

Normally, it should print genes, and then it should create the"test_config.toml" file below. I believe one of these two should be incorrect in your case (i.e., either the get_feature_key function, or the TOML writing)

[data]
gene = "genes"

If feature_key is ["g", "e", "n", "e", "s"], can you also show me the output below:

print(sdata["transcripts"].attrs)

[marsdenl]

Hmmm both worked fine on the toy dataset... So:

import sopa
from sopa.utils import get_feature_key
import toml

sdata = sopa.io.toy_dataset()

feature_key = get_feature_key(sdata["transcripts"])
print(feature_key) # should print "genes"

config = {
    "data": {
        "gene": feature_key,
    },
}

with open("test_config.toml", "w") as f:
    toml.dump(config, f)

Returns: genes

And the config file contains:

[data]
gene = "genes"

Finally, print(sdata["transcripts"].attrs) also returned genes:

{'spatialdata_attrs': {'feature_key': 'genes'}, 'transform': {'global': Affine (x, y -> x, y)
[ 10. 0. -1000.]
[ 0. 10. -6000.]
[0. 0. 1.], 'microns': Identity }}

So I gave it a go in my own dataset:

import sopa
from sopa.utils import get_feature_key
import toml

sdata = sopa.io.merscope("Z:/Queries/Data/Batch5_region0/region_0/")

feature_key = get_feature_key(sdata["Batch5_region0_region_0_transcripts"])
print(feature_key) # should print "genes"

config = {
    "data": {
        "gene": feature_key,
    },
}

with open("test_config.toml", "w") as f:
    toml.dump(config, f)

Returns: gene

But the toml file writing is wrong:

[data]
gene = [ "g", "e", "n", "e",]

And print(sdata["Batch5_region0_region_0_transcripts"].attrs) returns:

{'spatialdata_attrs': {'feature_key': 'gene'}, 'transform': {'global': Affine (x, y -> x, y)
[ 9.25946999e+00 0.00000000e+00 -9.59847344e+04]
[ 0.00000000e+00 9.25953007e+00 -5.32247656e+04]
[0. 0. 1.]}}

So it seems dataset specific...can we force the toml file writing to avoid the listing format?

Cheers

Luc

[quentinblampey]

Thanks for trying, this is very surprising...
My naive guess is that perhaps the gene column has an unusual dtype that TOML doesn't know how to convert.

Could you try forcing it to become a string before writing the config file? I.e. the commands below:

import sopa
from sopa.utils import get_feature_key
import toml

sdata = sopa.io.merscope("Z:/Queries/Data/Batch5_region0/region_0/")

feature_key = get_feature_key(sdata["Batch5_region0_region_0_transcripts"])
print(feature_key, type(feature_key)) # also show the type of feature_key

config = {
    "data": {
        "gene": str(feature_key), # here, I make sure it is converted to a string
    },
}

with open("test_config.toml", "w") as f:
    toml.dump(config, f)

Now the config file should look as expected. Let me know!

[marsdenl]

That did it, got converted fine. Nice one man! So now, no more need to force a config file, the default "built in" works :)

[quentinblampey]

Very nice! I updated the code to use str(feature_key) instead of feature_key, it will be available in the next sopa release!

[marsdenl]

Fantastic! Thanks. Let me know how you get on with Baysor on Windows :)

[quentinblampey]

Now that the previous issue is fixed, could you let me know what's the new issue related to Windows (using the master branch)?
Either the Sopa error logs, or the Baysor error logs

I can't have access to a Windows machine yet. Probably in 2 weeks

[marsdenl]

No worries for the Windows access, no rush. I haven't pulled the master branch yet so I just forced the right config using a pre-defined config file.

Here's what I run:

import spatialdata
import sopa
import numpy as np

sdata = sopa.io.merscope("Z:/Queries/Data/Batch5_region0/region_0/")

sopa.make_transcript_patches(
    sdata,
    patch_width=1000,
    prior_shapes_key="cell_id", 
    unassigned_value=-1
)

sopa.segmentation.baysor(sdata, config = "Z:/Queries/example_config.toml")

And here's the error:

[WARNING] (sopa._settings) Running without parallelization backend can be slow. Consider using a backend, e.g. via sopa.settings.parallelization_backend = 'dask', or export SOPA_PARALLELIZATION_BACKEND=dask.
100%|██████████████████████████████████████████████| 23/23 [00:01<00:00, 18.83it/s]
Reading transcript-segmentation outputs: 0%| | 0/23 [00:00<?, ?it/s]

AssertionError Traceback (most recent call last)
Cell In[25], line 1
----> 1 sopa.segmentation.baysor(sdata, config = "Z:/Queries/example_config.toml")

File ~\sopa\sopa\segmentation\methods_baysor.py:79, in baysor(sdata, config, min_area, delete_cache, recover, force, scale, key_added, patch_index)
76 assert patches_dirs, "Baysor failed on all patches"
78 gene_column = _get_gene_column_argument(config)
---> 79 resolve(sdata, patches_dirs, gene_column, min_area=min_area, key_added=key_added)
81 sdata.attrs[SopaAttrs.BOUNDARIES] = key_added
83 if delete_cache:

File ~\sopa\sopa\segmentation_transcripts.py:43, in resolve(sdata, patches_dirs, gene_column, min_area, key_added)
40 if min_area > 0:
41 log.info(f"Cells whose area is less than {min_area} microns^2 will be removed")
---> 43 patches_cells, adatas = _read_all_segmented_patches(patches_dirs, min_area)
44 geo_df, cells_indices, new_ids = _resolve_patches(patches_cells, adatas)
46 points_key = sdata[SopaKeys.TRANSCRIPTS_PATCHES][SopaKeys.POINTS_KEY].iloc[0]

File ~\sopa\sopa\segmentation_transcripts.py:142, in _read_all_segmented_patches(patches_dirs, min_area)
138 def _read_all_segmented_patches(
139 patches_dirs: list[str],
140 min_area: float = 0,
141 ) -> tuple[list[list[Polygon]], list[AnnData]]:
--> 142 outs = [
143 _read_one_segmented_patch(path, min_area)
144 for path in tqdm(patches_dirs, desc="Reading transcript-segmentation outputs")
145 ]
147 patches_cells, adatas = zip(*outs)
149 return patches_cells, adatas

File ~\sopa\sopa\segmentation_transcripts.py:143, in (.0)
138 def _read_all_segmented_patches(
139 patches_dirs: list[str],
140 min_area: float = 0,
141 ) -> tuple[list[list[Polygon]], list[AnnData]]:
142 outs = [
--> 143 _read_one_segmented_patch(path, min_area)
144 for path in tqdm(patches_dirs, desc="Reading transcript-segmentation outputs")
145 ]
147 patches_cells, adatas = zip(*outs)
149 return patches_cells, adatas

File ~\sopa\sopa\segmentation_transcripts.py:93, in _read_one_segmented_patch(directory, min_area, min_vertices)
89 def _read_one_segmented_patch(
90 directory: str, min_area: float = 0, min_vertices: int = 4
91 ) -> tuple[list[Polygon], AnnData]:
92 directory: Path = Path(directory)
---> 93 id_as_string, polygon_file = _find_polygon_file(directory)
95 loom_file = directory / "segmentation_counts.loom"
96 if loom_file.exists():

File ~\sopa\sopa\segmentation_transcripts.py:134, in _find_polygon_file(directory)
132 return False, old_baysor_path
133 new_baysor_path = directory / "segmentation_polygons_2d.json"
--> 134 assert new_baysor_path.exists(), f"Could not find the segmentation polygons file in {directory}"
135 return True, new_baysor_path

AssertionError: Could not find the segmentation polygons file in C:\Users\marsdenl.sopa_cache\4edce370-2cf9-4d78-9c3e-18ad3a9fc02f\transcript_patches\0

So the patching seems to work fine, the config file looks okay (gene written "gene"), but baysor is not running. However, I can run baysor manually using the CLI in one of the patches in the cache directory using: baysor run -c config.toml transcripts.csv and it works fine.

The error did not occur when I ran the same code on my MacOS system (although I only ran baysor on a single patch to save time and wasn't sure where to find the resulting polygons to investigate). So the issue invoking baysor might be windows specific?

Hope this helps :)

Cheers!

[quentinblampey]

Thanks @marsdenl for the extra information. Last week I changed the way Baysor is called, maybe it fixed the issue. Could you please use the master branch to see if it's solved? Else, I'll try to reproduce it when I'll have access to a Windows

[marsdenl]

Sorry only got to do this now! Just pulled the master branch and gave it another go:

import spatialdata
import sopa
import numpy as np

sdata = sopa.io.merscope("Z:/Queries/Data/Batch5_region0/region_0/")

sopa.make_transcript_patches(
    sdata,
    patch_width=1000,
    prior_shapes_key="cell_id", 
    unassigned_value=-1
)

sopa.segmentation.baysor(sdata, config = "Z:/Queries/example_config.toml")

Error:

FileNotFoundError Traceback (most recent call last)
Cell In[5], line 1
----> 1 sopa.segmentation.baysor(sdata, config = "Z:/Queries/example_config.toml")

File ~\sopa\sopa\segmentation\methods_baysor.py:72, in baysor(sdata, config, min_area, delete_cache, recover, force, scale, key_added, patch_index)
69 return
71 patches_dirs = get_transcripts_patches_dirs(sdata)
---> 72 settings._run_with_backend([partial(baysor_patch, patch_dir) for patch_dir in patches_dirs])
74 if force:
75 patches_dirs = [patch_dir for patch_dir in patches_dirs if (patch_dir / "segmentation_counts.loom").exists()]

File ~\sopa\sopa_settings.py:49, in Settings._run_with_backend(self, functions)
43 if len(functions) > 1:
44 log.warning(
45 "Running without parallelization backend can be slow. "
46 "Consider using a backend, e.g. via sopa.settings.parallelization_backend = 'dask', "
47 "or export SOPA_PARALLELIZATION_BACKEND=dask."
48 )
---> 49 return [f() for f in tqdm(functions)]
51 log.info(f"Using {self.parallelization_backend} backend")
52 return getattr(self, f"run{self.parallelization_backend}_backend")(functions)

File ~\sopa\sopa_settings.py:49, in (.0)
43 if len(functions) > 1:
44 log.warning(
45 "Running without parallelization backend can be slow. "
46 "Consider using a backend, e.g. via sopa.settings.parallelization_backend = 'dask', "
47 "or export SOPA_PARALLELIZATION_BACKEND=dask."
48 )
---> 49 return [f() for f in tqdm(functions)]
51 log.info(f"Using {self.parallelization_backend} backend")
52 return getattr(self, f"run{self.parallelization_backend}_backend")(functions)

File ~\sopa\sopa\segmentation\methods_baysor.py:102, in BaysorPatch.call(self, patch_dir)
98 _copy_segmentation_config(patch_dir / SopaFiles.TOML_CONFIG_FILE, self.config)
100 import subprocess
--> 102 result = subprocess.run(
103 self.baysor_command.split(), cwd=patch_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE
104 )
106 if result.returncode != 0 or not (patch_dir / "segmentation_counts.loom").exists():
107 message = f"Baysor error on patch {patch_dir.resolve()} with command {self.baysor_command}"

File ~\AppData\Local\miniconda3\envs\sopa\lib\subprocess.py:503, in run(input, capture_output, timeout, check, *popenargs, **kwargs)
500 kwargs['stdout'] = PIPE
501 kwargs['stderr'] = PIPE
--> 503 with Popen(*popenargs, **kwargs) as process:
504 try:
505 stdout, stderr = process.communicate(input, timeout=timeout)

File ~\AppData\Local\miniconda3\envs\sopa\lib\subprocess.py:971, in Popen.init(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, user, group, extra_groups, encoding, errors, text, umask, pipesize)
967 if self.text_mode:
968 self.stderr = io.TextIOWrapper(self.stderr,
969 encoding=encoding, errors=errors)
--> 971 self._execute_child(args, executable, preexec_fn, close_fds,
972 pass_fds, cwd, env,
973 startupinfo, creationflags, shell,
974 p2cread, p2cwrite,
975 c2pread, c2pwrite,
976 errread, errwrite,
977 restore_signals,
978 gid, gids, uid, umask,
979 start_new_session)
980 except:
981 # Cleanup if the child failed starting.
982 for f in filter(None, (self.stdin, self.stdout, self.stderr)):

File ~\AppData\Local\miniconda3\envs\sopa\lib\subprocess.py:1456, in Popen._execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, unused_restore_signals, unused_gid, unused_gids, unused_uid, unused_umask, unused_start_new_session)
1454 # Start the process
1455 try:
-> 1456 hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
1457 # no special security
1458 None, None,
1459 int(not close_fds),
1460 creationflags,
1461 env,
1462 cwd,
1463 startupinfo)
1464 finally:
1465 # Child is launched. Close the parent's copy of those pipe
1466 # handles that only the child should have open. You need
(...)
1469 # pipe will not close when the child process exits and the
1470 # ReadFile will hang.
1471 self._close_pipe_fds(p2cread, p2cwrite,
1472 c2pread, c2pwrite,
1473 errread, errwrite)

FileNotFoundError: [WinError 2] The system cannot find the file specified

This time the config file was not distributed across the cache patch directories.

[quentinblampey]

Okay, so it looks like it cannot find the baysor executable.

Can send me what these commands are showing?

from sopa.segmentation.methods._baysor import _get_baysor_executable_path
_get_baysor_executable_path()

Also, can you send me an example of command you use when executing baysor manually? Normally, the function _get_baysor_executable_path() should return the right path to the executable or the baysor alias, which is used afterward.

Sorry for the slow / blind debugging, I still don't have access to a Windows...

[marsdenl]

No worries at all :) Appreciate the help.

from sopa.segmentation.methods._baysor import _get_baysor_executable_path
_get_baysor_executable_path()

Returns: 'baysor'

For the manual run, the following command works in a patch of the cache directory.

baysor run -c config.toml transcripts.csv

Hope this helps!