Incorrect Helix Summary Statistics in Refine3D run.out

[awenborn]

We have noticed a potential bug in the calculation of the helical refinement summary statistics reported in the Refine3D run.out and hoping it doesn't propagate further into other output e.g. 3D density maps etc.

Description:
Throughout a Refine3D job the helix summary statistics reported in run.out at the end of each iteration show sensible values for each half-refinement e.g.:

 Maximization ...
1.43/1.43 min ............................................................~~(,_,">
 Local searches of helical twist from -0.9 to -0.8 degrees, rise from 4.76 to 4.84 Angstroms.
 (Half 1) Refined helical twist = -0.850742 degrees, rise = 4.78917 Angstroms.
 (Half 2) Refined helical twist = -0.84834 degrees, rise = 4.78882 Angstroms.
 Averaged helical twist = -0.849541 degrees, rise = 4.78899 Angstroms.

At the end of the final iteration the values reported for (Half 2) seem to be populated by the initial input values for the job, rather than the refined values for Half 2, and the subsequent Averaged helical twist and rise values are skewed appropriately. e.g.

 Maximization ...
3.20/3.20 min ............................................................~~(,_,">
 Local searches of helical twist from -0.9 to -0.8 degrees, rise from 4.76 to 4.84 Angstroms.
 (Half 1) Refined helical twist = -0.849097 degrees, rise = 4.78878 Angstroms.
 (Half 2) Refined helical twist = -0.87 degrees, rise = 4.8 Angstroms.
 Averaged helical twist = -0.859549 degrees, rise = 4.79439 Angstroms.
 Auto-refine: Refinement has converged, stopping now...

From a (somewhat naive!) end-user perspective, the obvious concern is that these seemingly incorrectly populated values are also being used in the generation of output .mrc maps and that these are thusly skewed relative to the accuracy of the initial values supplied to the job.

Environment:

• OS: Rocky Linux release 8.7
• MPI runtime: OpenMPI 4.0.5
• RELION version: RELION-4.0.0-commit-1569f0 / Relion-5.0-beta-0-commit-de8302
• Memory: 128 GB
• GPU: GeForce RTX 3090

Dataset:

• Box size: 480 px
• Pixel size: 0.828 Å/px
• Number of particles: ~50,000

Job options:

• Type of job: Refine3D
• Number of MPI processes: 5
• Number of threads: 3

[biochem-fan]

@scheres

These values are initialized to the initial value (not the value from the previous iteration) at the beginning of every M step,

relion/src/ml_optimiser_mpi.cpp

Lines 1969 to 1970 in 7809a73

	helical_twist_half1 = helical_twist_half2 = helical_twist_initial;
	helical_rise_half1 = helical_rise_half2 = helical_rise_initial;

and updated to the refined values.

relion/src/ml_optimiser_mpi.cpp

Lines 2132 to 2133 in 7809a73

	helical_rise_half1 = mymodel.helical_rise[ith_recons];
	helical_twist_half1 = mymodel.helical_twist[ith_recons];

relion/src/ml_optimiser_mpi.cpp

Lines 2264 to 2265 in 7809a73

	helical_rise_half2 = mymodel.helical_rise[ith_recons];
	helical_twist_half2 = mymodel.helical_twist[ith_recons];

But half2 performs this only when !do_join_random_halves.
Thus, helical_rise_half2 and helical_twist_half2 remain the initial values.

The question is whether this affects the output. It looks like imposeHelicalSymmetryInRealSpace is not called in the final iteration, so this does not degrade the map? Am I right?

In any case, the output message should be repaired, because users might report these numbers in the paper and/or use them in future refinement jobs.

[awenborn]

Many thanks for looking into this and apologies for the double-posting here and on CCPEM; wasn't sure if it would get lost in the quantity of messages on the mailing list and thought raising an Issue here would be more appropriate.

[scheres]

I have now committed a fix for this. Thanks for the detailed investigation! It is part of relion ver5.0 for now. Perhaps @awenborn could test it too?

On 11/10/23 08:11, biochem_fan wrote: CAUTION: This email originated from outside of the LMB. Do not click links or open attachments unless you recognize the sender and know the content is safe. ***@***.**** @scheres <https://github.com/scheres> These values are initialized to the initial value (not the value from the previous iteration) at the beginning of every M step, https://github.com/3dem/relion/blob/7809a73daec9f9dbd5e9d359d4de46be834fcc63/src/ml_optimiser_mpi.cpp#L1969-L1970 and updated to the refined values. https://github.com/3dem/relion/blob/7809a73daec9f9dbd5e9d359d4de46be834fcc63/src/ml_optimiser_mpi.cpp#L2132-L2133 https://github.com/3dem/relion/blob/7809a73daec9f9dbd5e9d359d4de46be834fcc63/src/ml_optimiser_mpi.cpp#L2264-L2265 But half2 performs this only when |!do_join_random_halves|. Thus, |helical_rise_half2| and |helical_twist_half2| remain the initial values. The question is whether this affects the output. It looks like |imposeHelicalSymmetryInRealSpace| is not called in the final iteration, so this does not degrade the map? Am I right? In any case, the output message should be repaired, because users might report these numbers in the paper and/or use them in future refinement jobs. — Reply to this email directly, view it on GitHub <#1026 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFOHJCMCFS7SSYCWUGLHFYTYDXOTJAVCNFSM6AAAAAA7DUUHF2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBVGI3TIOBTHA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Sjors Scheres MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge Biomedical Campus Cambridge CB2 0QH, U.K. tel: +44 (0)1223 267061 http://www2.mrc-lmb.cam.ac.uk/groups/scheres

[biochem-fan]

@scheres Thanks!

@awenborn in case you don't want to upgrade to RELION 5.0 beta yet, I applied the same patch to the ver4.0 branch.

[awenborn]

We've tested this on the Relion 5.0 beta and can confirm that it has fixed the issue.

Many thanks both for the rapid response.