1 body reduced density matrix estimator produces incorrect results

[jptowns]

Collecting the "dm1b" estimator on a VMC calculation of FeO with a single determinant PBE orbital basis with no Jastrow factor should produce a diagonal density matrix. This system contains 22 electrons per spin channel, so a visualization of the 1rdm for, say, the "up" channel should show a strongly diagonal structure for the first 22 states. Yet, for QMCPACK built on 04 Sept. (#1046) it does not:

The salient input block is in the specification of the hamiltonian:

<?xml version="1.0"?>                                                                                                  
<qmcsystem>                                                                                                            
  <hamiltonian name="h0" type="generic" target="e">                                                                    
    <pairpot name="IonIon"   type="coulomb" source="i" target="i"/>                                                    
    <pairpot name="ElecElec" type="coulomb" source="e" target="e"/>                                                    
    <pairpot type="pseudo" name="PseudoPot" source="i" wavefunction="psi0" format="xml">                               
      <pseudo elementType="Fe" href="../wfns-and-pseudos/Fe.opt.xml"/>                                                 
      <pseudo elementType="O"  href="../wfns-and-pseudos/O.opt.xml"/>                                                  
    </pairpot>                                                                                                         
    <estimator type="dm1b" name="1rdms">                                                                               
      <parameter name="basis"            >  dm_basis     </parameter>                                                  
      <parameter name="points"           >  10           </parameter>                                                  
      <parameter name="samples"          >  10           </parameter>                                                  
      <parameter name="warmup"           >  1            </parameter>                                                  
      <parameter name="timestep"         >  0.25         </parameter>                                                  
    </estimator>                                                                                                       
  </hamiltonian>                                                                                                       
</qmcsystem>                                                                                                           

And the VMC calculation was specified by:

  <qmc method="vmc" move="pbyp" checkpoint="0">                                                                        
    <parameter name="walkers"       >       8  </parameter>                                                            
    <parameter name="timestep"      >  0.2000  </parameter>                                                            
    <parameter name="useDrift"      >      no  </parameter>                                                            
    <parameter name="steps"         >      64  </parameter>                                                            
    <parameter name="blocks"        >      64  </parameter>                                                            
    <parameter name="nonlocalmoves" >     yes  </parameter>                                                            
  </qmc>                                                                                                               

What is strange is that for commit #1021 the same input blocks produce something more expected:

In fact, these effects persist regardless of the presence of a Jastrow, and also persist at the DMC level.
Minimal working example included.

feo-mwe-1rdm.gz

[prckent]

Is the trace correct?

[jptowns]

Whoa! The trace is way off!
The trace of the rdms for the first figure (As of commit #1046) are about 11 and 0, for up and down spin channels, respectively. Should be ~22 and ~22.

The trace for the second figure (as of commit #1021) are 21.9/21.9 respectively. This is what is expected.

[jptowns]

Oh, also I know that there are currently some issues regarding OMP support so all calculations were performed on a desktop workstation with multiple MPI ranks each with 1 thread.

[jtkrogel]

Interesting. There aren't many PR's between #1021 and #1046 that could affect this. Have you narrowed down to the culprit PR (via bisection search, etc)?

[jptowns]

Have not done that. Will see what I can do.

[prckent]

Any updates on this?

[prckent]

Is this fixed by #1509 or are there still problems to be resolved?

[jtkrogel]

I will look into this soon. I suspect it may be a real vs. complex issue. The real orbitals are not orthogonal in general and might lead to the pattern in the 1rdm seen above.

[prckent]

Could we measure using a distinct complex-valued orbital set or an orthogonal real-valued set even with the real build?

[jtkrogel]

If the set of orbitals is orthogonal within QMCPACK, then everything should work fine. I would prefer to require orthogonal orbitals within QMCPACK overall.

[Paul-St-Young]

Is there any update on this issue?

I still get 11 and 0 as the traces for the 1RDMs of up and down electrons using Josh's example above.

                    QMCPACK 3.9.9

       (c) Copyright 2003-  QMCPACK developers

                    Please cite:
 J. Kim et al. J. Phys. Cond. Mat. 30 195901 (2018)
      https://doi.org/10.1088/1361-648X/aab9c3

  Git branch: develop
  Last git commit: 02efd65977f97fc8ca54340640701553741445de
  Last git commit date: Tue Sep 22 08:04:13 2020 -0500