Test example error

[XingerTang]

While running the test example, it raises the following error

Traceback (most recent call last):

File "/Users/evie/mambaforge/bin/AlphaImpute2", line 8, in <module>

sys.exit(main())

File "/Users/evie/mambaforge/lib/python3.10/site-packages/alphaimpute2/tinyhouse/Utils.py", line 8, in timer

values = func(*args, **kwargs)

File "/Users/evie/mambaforge/lib/python3.10/site-packages/alphaimpute2/alphaimpute2.py", line 304, in main

write_out_data(pedigree, args)

File "/Users/evie/mambaforge/lib/python3.10/site-packages/alphaimpute2/tinyhouse/Utils.py", line 8, in timer

values = func(*args, **kwargs)

File "/Users/evie/mambaforge/lib/python3.10/site-packages/alphaimpute2/alphaimpute2.py", line 255, in write_out_data

pedigree.writePhase(args.out + ".haplotypes")

File "/Users/evie/mambaforge/lib/python3.10/site-packages/alphaimpute2/tinyhouse/Pedigree.py", line 943, in writePhase

if ind.haplotypes.ndim == 2:  # diploid

AttributeError: 'tuple' object has no attribute 'ndim'

[gregorgorjanc]

@XingerTang do you have any sense of what is driving this error? can you also share here what exact command you use to get this error?

[XingerTang]

@gregorgorjanc The command I used to get this error is just the command to run the example code

cd docs
bash run_script.sh

The error is probably caused by the datatype of haplotypes in Pedigree.py of the submodule tinyhouse.

In line 155 of Pedigree.py, the haplotypes attribute of an object of the Individual class is initialized as a tuple.

self.haplotypes = (np.full(nLoci, 9, dtype = np.int8), np.full(nLoci, 9, dtype = np.int8))

However, in the function writePhase(), it tries to take the ndim of the haplotypes, which is not possible as it is a tuple instead of a numpy array.

[gregorgorjanc]

@XingerTang this is odd - tinyhouse and its haplotype attribute of the Individual class must be used elsewhere too, say in AlphaPeel. How is it handled there? Or do we never call for ind.haplotypes.ndim elsewhere?

What do you propose we do to address this issue?

[XingerTang]

@gregorgorjanc We only call ind.haplotype.ndim in writePhase() and writePhasePed() which are not frequently used.

I think it can be addressed by replacing the ind.haplotype.ndim with some other code and I believe it won't be a big problem though I need a bit more time.

[XingerTang]

@gregorgorjanc The problem is a little bit more complicated than I thought. I checked all the places where ind.haplotypes has been used and in some of them it is used as an ndarray, and for others it is used as a tuple of ndarrays.

For the ndarray case, ind.haplotypes is initialized in Pedigree.decode_alleles() and being used in Pedigree.encode_alleles() apart from writePhase() and writePhasePed(). But for the Pedigree.decode_alleles(), I find that the function will never be called as the condition to call it will never be satisfied (the condition is an argument ped which is not in the argument dictionary to be not None). For the Pedigree.encode_alleles(), it is only used in writePhase() and writePhasePed().

For the a tuple of ndarrays case, it is the one that is actually used when the program runs, no matter whether the input pedigree file exists or not.

I guess what happened is that, during the development of the program, the developers originally want to accept multiploid input, which the ndarray allows. Yet, later they decide to limit the case to only take diploid input, which the tuple is enough. So some of the functions written previously left and causes the mismatch of the datatypes.

If what I guess is correct, what I need to do is remove some of the functions that are not being used and then update writePhase() and writePhasePed().

[gregorgorjanc]

@XingerTang thanks for looking into this!

Which packages have you looked at? I think we should make sure these tools work (these use tinyhouse) - will they work with your proposed change in tiny house?

https://github.com/AlphaGenes/AlphaPeel

https://github.com/AlphaGenes/AlphaImpute2

https://github.com/AlphaGenes/AlphaFamImpute

https://github.com/AlphaGenes/AlphaPlantImpute2

https://github.com/AlphaGenes/AlphaAssign

[XingerTang]

@gregorgorjanc The datatype of ind.haplotypes in AlphaPeel and AlphaImpute2 are consistent, so I think the problems are only in the tinyhouse, it is where all the above inconsistencies occur. (And I just found that it is exactly the issue AlphaGenes/tinyhouse#1.)

If the issue is fixed, then the error won't be raised when writePhase() and writePhasePed() are called by the above programmes.

[XingerTang]

@gregorgorjanc I checked all 5 packages and all of them except AlphaPlantImpute2 are using the tuple datatype for ind.haplotypes. However, AlphaPlantImpute2 is using the ndarray datatype.

Another difference between the AlphaPlantImpute2 and the other 4 that the AlphaPlantImpute2 allows the input options -ped [PED [PED ...]] and -library LIBRARY which can take in PLINK files. However for the rest 4, they can only take in plink files with the input option -bfile [BFILE [BFILE ...]].

As AlphaPlantImpute2 is using the ndarray datatype, a simple modification of the datatype of the ind.haplotypes in tinyhouse would lead to inconsistencies of AlphaPlantImpute2. Since AlphaPlantImpute2 is quite different from the others in many ways, it is difficult to modify it to make it consistent with the others. So, the simplest way to solve everything is writing two separate systems in parallel.

[gregorgorjanc]

@XingerTang I think the best way forward would indeed be to first work with tuple datatype for ind.haplotypes since that is used more commonly across all the packages we are working with. If we work with these packages in the devel versions and then bump the version, we will get a working solution for these packages, and then we will later work with the AlphaPlantImpute2. Ideally, we would have the same framework for all the packages, but let's cross that bridge once we fix the other packages.

Having said all this I now wonder why we are in this situation! The other packages are somewhat older that the AlphaPlantImpute2, sort of. So, I wonder if recent changes in tinyhouse have been towards the use in AlphaPlantImpute2 (that is, using ndarray and not a tuple of ndarrays), which is causing the issue in other packages? Can you please check this?

[XingerTang]

@gregorgorjanc Sure! I will do that later!

[XingerTang]

@gregorgorjanc Yes, you are correct, tinyhouse is shifting from using tuple to ndarrays. We can first publish a stable version with tuple and then figure out how to move toward the ndarray on a separate branch.

[gregorgorjanc]

@XingerTang does this mean we should:

1. Be upgrading everything to ndarrays since this is where tinyhouse is going to?

2. Reverting some changes in tinyhouse and tweak AlphaPlantImpute2?

[XingerTang]

@gregorgorjanc I think we will first do 2 then do 1... But let's just wait for the response.

[gregorgorjanc]

@XingerTang which version/commit of tinyhouse uses only tuple of ndararys? Maybe we just stick to that version for now and port recent changes (help) to that, make it the main branch and push the current state with ndarray to devel branch?

Waiting for Steve, but noticed he moved to Ox. Have contacted him on Twitter;)