Implement the classes Imo and ImoFlatFile

[ziotom78]

This PR adds two classes to the framework:

1. Imo is an abstract class
2. ImoFlatFile is a concrete class that provides methods to read the IMO from a YAML file, much like the one created by instrumentdb (https://github.com/ziotom78/instrumentdb)

Examples of use are in file tests/test_imo.py. They are like the following:

import litebird_sim as lbs
from uuid import UUID

# Specify the folder that contains file "schema.yaml"
imo = lbs.Imo(flatfile_location="/storage/litebird/mock_imo")

# Access data using the release number for a data file (in this case, 1.0)
data_file = imo.query("releases/1.0/focal_plane/beams/horn01/horn01_grasp")

assert data_file.uuid == UUID("bd8e16eb-2e9d-46dd-a971-f446e953b9dc")
assert data_file.metadata["ellipticity"] == 0.0
assert data_file.metadata["fwhm_deg"] == 1.0

# If you know the UUID, access the data file directly
data_file = imo.query(UUID("bd8e16eb-2e9d-46dd-a971-f446e953b9dc"))

Things to do:

• Implement ImoFlatFile
• Implement tests
• Include information about the IMO in the report produced by Simulation objects
• Write the documentation

I would like to wait for comments before writing the documentation.

[dpole]

Thanks Maurizio for this. I had a look at the imofile.py but I can not provide comments on the overall design. If you want a feedback on that, I'd need some general description of the logic (either here or in an offline chat). If there's something specific you'd like people to look at, please let me know, I can try to focus on it. In the meanwhile I propose a couple of minor changes.

[dpole]

As we discussed previously, I’m willing to help with the integration of the temporary imo into the litebrid_dim. Is it still relevant? Where can I find the codes and/or data we have to interface?

[ziotom78]

Hi @dpole , the repository is here but it's private: https://github.com/litebird/litebird_imo. You should request access to @paganol (BTW, can you please add me too, Luca?)

Something we should do ASAP is to fix the key names used in the metadata, so that measurement units are always indicated. For instance, replace fknee to fknee_hz, bandwidth to bandwitdh_ghz, etc. Once people in the ST start using it, it will become increasingly difficult to change this kind of stuff.

Another thing to check is the presence of multiple "data files" for each quantity. I made a quick check, and it seems that the current database is large also because there are many older versions of data files saved. Our first internal release of litebird_imo should have "no history", i.e., every quantity should be associated to one and only one data file.

[dpole]

About access, it should be private only for the outside world and not for people in the litebird organization. I just tested that I'm able to checkout and push. Let me know if you have limitations instead.

[ziotom78]

About access, it should be private only for the outside world and not for people in the litebird organization. I just tested that I'm able to checkout and push. Let me know if you have limitations instead.

Oh, that's great, I didn't have time to create a PR for it, so I did not check…

[NicolettaK]

Hi @ziotom78, having more examples on how to retrieve information from the IMO flat file would be great! for example, how can I get the specifications (like sensitivity and resolution) for all the LB channels?

[paganol]

Hi @ziotom78, related to @NicolettaK's question, do we need to assign a release version to every data_file, in order to be able to read them using the full path? it seems so. The mock_imo works for /releases/1.0/focal_plane/beams/horn01/horn01_grasp, but then it does not for the others.

[ziotom78]

Hi @ziotom78, related to @NicolettaK's question, do we need to assign a release version to every data_file, in order to be able to read them using the full path? it seems so. The mock_imo works for /releases/1.0/focal_plane/beams/horn01/horn01_grasp, but then it does not for the others.

As it is now, you can access a data file that is not part of a data release only through its UUID. I believe this is fair, as we want people to be really sure of what they're doing when they are not using official IMO parameters.

[ziotom78]

Ok, I believe the PR is complete now.

I have made modifications to the Detector class, because this class is a good way to test how to interface the IMO. In this PR, the class Detector can be constructed using either read_detector_from_imo or read_detector_from_dict. The first function constructs a Detector object from an IMO reference (either a UUID or a path like /release/v0.10/Satellite/.../det01), and the second one reads the detector from a dict object. The latter permits to build Dictionary objects out of parameter files, as shown below.

Reading detectors from the IMO is easy:

imo = lbs.Imo(...)
uuid = UUID("78fe75f1-a011-44b6-86dd-445dc9634416")
det = lbs.read_detector_from_imo(imo, uuid)
print(det)

If one wants to simulate a custom detector, its definition can be read from a TOML file (e.g., a parameter file specified by the user running the script):

import litebird_sim as lbs, tomlkit

doc = tomlkit.parse("""
[my_detector]
name = "foo"
wafer = "bar"
pixel = "abc"
pixtype = "def"
channel = "ghi"
sampling_rate_hz = 12.0
fwhm_arcmin = 34.0
ellipticity = 56.0
net_ukrts = 78.0
fknee_mhz = 90.0
fmin_hz = 98.0
alpha = 76.0
pol = "jkl"
orient = "mno"
quat = [0.0, 1.0, 2.0, 3.0]
""")

det = lbs.read_detector_from_dict(doc["my_detector"])
print(det)

(Of course, it's trivial to adapt this latter example to read the definition from a JSON/YAML/... file.)

[dpole]

I just want to point out a couple of thing in #42 that we may want to coordinate with this PR.

In #42 I removed the Detector class because it becomes unnecessary once all the information about the detectors is stored in Observation.

I was not sure about how to choose the detector fields to be loaded from the IMO. For time being I'm using a mock code that receives the fields list as argument and load each item for each detector with some_method (which I suppose should be replaced with something from this PR)

litebird_sim/litebird_sim/observations.py

Line 67 in e7f764c

field_arr = np.array([some_method(field, d) for d in detectors])#XXX

[ziotom78]

In #42 I removed the Detector class because it becomes unnecessary once all the information about the detectors is stored in Observation.

I believe it's important to have a Detector class containing all the fields in the IMO, because these can be useful for other purposes than in the Observation class. (E.g., @giuspugl uses this information to generate nice plots of the wafers using Matplotlib, in the toast-litebird repository.) We could initialize an Observation object by passing a list of Detector objects, so that Observation just grabs the kind of information it needs.

I'm not sure how your line of code above using np.array works when you're considering non-numeric fields (e.g., strings) returned by some_method.