Should compare and harmonise with previous Toby et. al. paper

[jamesrhester]

The only previous attempt to tame large powder data sets appears to be Toby et. al. (2003). The document should recognise this. Ideally our recommendations should not contradict the recommendations of that paper without good reason, as it is likely that existing pdCIF input code has been written to that spec. I've added a second appendix to provide a comparison.

[jamesrhester]

An initial review of the paper and an implied outcome of recent powder CheckCIF discussions with the IUCr office suggests that all information about a chemical species (a phase) should be included in the same data block as the structure. This means that the powder dictionary should add child data names of _pd_phase.id to the CHEMICAL_* categories, and then inclusion of these categories be the necessary outcome of the rules as stated.

[jamesrhester]

I've created a powder dictionary pull request to make chemical information per-phase.

[jamesrhester]

Below is the text I've put in a pull request for the powder guidelines:

Summary of TvDL approach

For brevity,
Toby, von Dreele, and Larson (2003) is abbreviated as TvDL below. The TvDL prescription is given in terms
of D diffraction measurements and P phases.

Four kinds of data blocks are described:

1. A separate publication block

2. A separate overall information block

3. P Chemical species blocks

4. D Diffraction data blocks

Key points:

1. When either D > 1 or P > 1, all D and P are placed in separate blocks. When
   D = P = 1, all the above blocks are merged into a single block. This is for
   simplicity in implementation.

2. The chemical species blocks include structural and chemical information, as
   would be seen in a single crystal data block.

3. The diffraction data blocks include loops over the constituent phases,
   loops for raw and calculated data, and potentially loops where the incident
   beam contains more than one wavelength.

4. Relationships between blocks (e.g. which chemical species are present in
   which diffraction data set) are indicated using block pointers and block
   identifiers.

Block identifiers/pointers

In TvDL, blocks are connected using block pointers. Nothing in the present
standard prevents the use of these pointers or identifiers in addition to
_audit_dataset.id.

Separate blocks

The publication and overall information blocks would be combined into a
single block in the present guide. This is not strictly necessary.

Chemical species blocks

Rule 4 combines categories that have child key data names of
_structure.id and _pd_phase.id. This results in the same contents.

Diffraction data blocks

Diffraction data blocks are the most complex of those described in TvDL.
In terms of child key data names of Set categories, TvDL diffraction data
blocks contain:

1. Diffractograms together with the measurement conditions and instruments -
   children of _diffrn.id, _pd_diffractogram.id, and _pd_instr.id.

2. Phase table (TvDL 3.4.1). Combinations of children of _pd_phase.id
   and _pd_diffractogram.id giving all categories in the second row of Table 2.

3. Wavelength table - _diffrn_radiation_wavelength.id. Our recommendations
   would put this in a separate block.

4. Refln loop. TvDL includes _pd_refln.phase_id as a column in the loop,
   so that hkl from all phases are contained in a single loop for a single
   diffractogram.

The present guidelines reproduce only part 1 of the TvDL recommendations.

[jamesrhester]

The key difference with TvDL is in the construction of the "diffraction data" blocks, where we are recommending a subset of the contents of a TvDL diffraction data block, and additionally creating P X D blocks for the reflection and phase tables. I don't see this as a big problem, as I don't expect a lot of software has been written to ingest the TvDL diffraction data blocks (unlike the phase blocks, which are just standard core CIF structural data readable by all and sundry). Even software that has been written will still find the diffraction data, and will probably not be fatally bothered if it finds neither wavelength table, reflection table, or the phase table in the diffraction data block.

We do have the option of setting _audit.schema to Custom in certain data blocks if we think it more worthwhile to loop phases for each diffractogram.

Thoughts?

[rowlesmr]

Was reading through the paper today. Many of the loops seemed to be present to allow all of the block pointers to properly talk between all of the phase and diffractogram blocks.

The chemical species blocks in TvDL seem to be vanilla, except that they should contain block pointers to the diffractograms from which their information was calculated. Is there a way now to capture that Structure 1 was refined from Diffractograms A and B?

I think we can follow their guidelines pretty closely. TBH, I like the idea of the phase table(s) giving quant and/or PO values in with the diffractogram. Same with the reflection table. (It's how pdCIFplotter accepts data, and my TOPAS CIF macros output it. - see SI in https://journals.iucr.org/j/issues/2022/03/00/yr5087/)

I'm also trying to think on how to get outputs from programs. In TOPAS, I can quite easily loop over diffractograms, and then the phases used to model that diffractogram. This makes it easy to output wavelengths, diffraction conditions, instrument settings, phase tables on a per diffractogram basis while writing a CIF file sequentially. You can then write all the information about each phase, before going on to the next diffractogram. Looping over phases first, finding each diffractogram in which they appear is pretty much impossible.

[rowlesmr]

One way around the program-specific issues is to have a small program (for ease of getting it off the ground only has to work with modern, up-to-date CIF2.0) that can read in the dictionaries, read in the CIF, make up all the database tables, and then have a series of knobs you can set to decide how it is re-written as a "compliant" CIF (whatever that means).

[jamesrhester]

Is there a way now to capture that Structure 1 was refined from Diffractograms A and B?

No, because we don't currently have the concept of "a refinement". To do that we'd have to create a category and identifier, e.g. _refine.id, which could then be associated with structures and diffractograms. Essentially it would allow you to label all of the parameters, constraints, constant values, and data for a refinement. As a given refinement could involve many diffractograms, and a diffractogram could participate in more than one refinement, there's a many-to-many relationship, so you'd have to define a separate table that would look as follows:

_refine.id    _refine.diffractogram_id 
1                   a                      
1                   b                 
1                   c                    
2                   b                 
2                   c 

[jamesrhester]

One way around the program-specific issues is to have a small program (for ease of getting it off the ground only has to work with modern, up-to-date CIF2.0) that can read in the dictionaries, read in the CIF, make up all the database tables, and then have a series of knobs you can set to decide how it is re-written as a "compliant" CIF (whatever that means).

That is true - it could even be put behind a web service at the IUCr (although one of us would have to write it). Such software would completely ignore the concept of "Set" and "Loop" categories when reading in, assume all categories with key data names are looped, and then partition up the result when outputting.

I'm also trying to think on how to get outputs from programs. In TOPAS, I can quite easily loop over diffractograms, and then the phases used to model that diffractogram. This makes it easy to output wavelengths, diffraction conditions, instrument settings, phase tables on a per diffractogram basis while writing a CIF file sequentially. You can then write all the information about each phase, before going on to the next diffractogram. Looping over phases first, finding each diffractogram in which they appear is pretty much impossible.

In Python and presumably other languages you can use StringIO in the place of file output to accumulate the contents of a each data block in parallel, so that the loop over phases could distribute the information into the relevant data block before output. And if you use a library like PyCIFRW you would just have CifBlock objects to accumulate information into. But I don't know if Topas macros provide that sort of ability.

I do like the idea of us providing a transformer service, especially if it means the proposal as currently written is acceptable.

[rowlesmr]

Yes. It would be very permissive on at it would read in. Any duplicate information can be discarded, and then you can choose a "standard" to write out your data.

I have a feeling that it's either impossible, or would be a bodge in Topas to do such a thing. You're pretty much limited to doing the thing you're doing right now, at least in user-space.