Test -1.1.0 #58
Test -1.1.0 #58
Conversation
|
Check out this pull request on Review Jupyter notebook visual diffs & provide feedback on notebooks. Powered by ReviewNB |
|
View / edit / reply to this conversation on ReviewNB steven-murray commented on 2020-06-29T14:38:23Z Obviously need to fill out the CRITERIA.
I'm thinking it could be something like:
|
|
View / edit / reply to this conversation on ReviewNB steven-murray commented on 2020-06-29T14:38:24Z I think we can say it has passed, if you go back to the positions before converting to CIRS.
I think you can also suggest a follow-up action that it would be useful for pyuvsim (and simulators in general) to be able to output source alt/az for each of the simulation times, to make for simpler comparisons.
Also mention that the source position errors found here are fine for power spectra in the validation effort, but will potentially lead to calibration errors if used for that purpose in future IDRs.
Also mention that too-low lmax primarily affects uniform beam (because of the inherent step function in time). Setting it higher resolves problems, but also increases compute time significantly. lwhitler commented on 2020-06-29T18:43:41Z Is it correct to go back to the positions before converting to CIRS, though? I'm not sure what was done for the RIMEz run that we're using for validation, but whatever it was, should we be validating that (it might be CIRS or it might not be) or the method that apparently gives the more correct results (not using CIRS coordinates)? steven-murray commented on 2020-06-30T14:07:21Z I think I'm operating under the assumption that we've screwed something up in the CIRS transform, so that it's less correct than the original (and less correct than whatever @zmartinot did for Validation sims). Ideally, we'd hear from him and we can sort the whole thing out. If not, I think we can get this one through (and we can perhaps do a follow-up test -1.1.1 with updated positions later). steven-murray commented on 2020-07-06T16:54:40Z OK, so this has been resolved -- apparently CIRS doesn't work for this version of |
|
View / edit / reply to this conversation on ReviewNB steven-murray commented on 2020-06-29T14:38:24Z You are implicitly using pyuvdata and h5py, so I'd keep them in the versions. |
|
View / edit / reply to this conversation on ReviewNB steven-murray commented on 2020-06-29T14:38:25Z I think the |
|
View / edit / reply to this conversation on ReviewNB steven-murray commented on 2020-06-29T14:38:26Z I'd put all steven-murray commented on 2020-07-06T16:58:03Z Looks like this has been done. |
|
View / edit / reply to this conversation on ReviewNB steven-murray commented on 2020-06-29T14:38:26Z Again, should put this script in the repo... also a few words about how they were created (i.e. with CASA). steven-murray commented on 2020-07-06T16:58:33Z Great! |
|
View / edit / reply to this conversation on ReviewNB steven-murray commented on 2020-06-29T14:38:27Z Can probably suggest why: |
|
View / edit / reply to this conversation on ReviewNB steven-murray commented on 2020-06-29T14:38:28Z Can we explain the oscillations in the auto difference? Is it due to differences in beam interpolation? lwhitler commented on 2020-07-03T16:59:31Z Not actually sure. Possibly? |
|
View / edit / reply to this conversation on ReviewNB steven-murray commented on 2020-06-29T14:38:29Z I think that unless we figure out what's going on here, we should revert to the original non-adjusted positions (keep this section, just use the original positions, which had smaller errors). |
|
View / edit / reply to this conversation on ReviewNB steven-murray commented on 2020-06-29T14:38:30Z Probably should say what the sim is here (uniform beam) |
|
View / edit / reply to this conversation on ReviewNB steven-murray commented on 2020-06-29T14:38:31Z Can add to this that the reason is that a sharp step like this requires infinite fourier modes to capture it, and that using more fourier modes extends the fidelity out to larger fringe rates. The effect of this in frequency/time is to induce oscillations.
|
|
View / edit / reply to this conversation on ReviewNB steven-murray commented on 2020-06-29T14:38:31Z ... resulting in a cutoff at the frequency at which the sources are not adequately resolved. |
|
Is it correct to go back to the positions before converting to CIRS, though? I'm not sure what was done for the RIMEz run that we're using for validation, but whatever it was, should we be validating that (it might be CIRS or it might not be) or the method that apparently gives the more correct results (not using CIRS coordinates)? View entire conversation on ReviewNB |
|
I think I'm operating under the assumption that we've screwed something up in the CIRS transform, so that it's less correct than the original (and less correct than whatever @zmartinot did for Validation sims). Ideally, we'd hear from him and we can sort the whole thing out. If not, I think we can get this one through (and we can perhaps do a follow-up test -1.1.1 with updated positions later). View entire conversation on ReviewNB |
|
Not actually sure. Possibly? View entire conversation on ReviewNB |
|
OK, so this has been resolved -- apparently CIRS doesn't work for this version of View entire conversation on ReviewNB |
|
Looks like this has been done. View entire conversation on ReviewNB |
|
Great! View entire conversation on ReviewNB |
|
View / edit / reply to this conversation on ReviewNB steven-murray commented on 2020-07-06T17:08:22Z I think we want to include a short statement here to the effect that position errors are likely mostly due to not converting to CIRS -- functionality that was not available in RIMEz at the time the validation was performed (either here or for validation simulations). A newer version of RIMEz is expected to rectify this, and a follow-up validation notebook will be produced at that time. |
|
Briefly, the differences in this notebook reflect that effectively different source positions, and explicit definition of beam functions, were input to the pyuvsim calulation than to the RIMEz calculation. Obviously this is simply user error, not some fundamental problem with pyuvsim. But further, the method used in the RIMEz invocation actually computes time integration based on the specified integration, a capability that pyuvsim does not have. A more appropriate comparison would be to use the point-source method in RIMEz, with the right beams and source positions of course. Obviously there are differences between the harmonic decomposition algorithm and the point-source method in RIMEz, as shown in the RIMEz demonstration_1 notebook, but for something like the HERA beam model, the relative difference is 1e-4 or less depending on details of the beam/sky/time-sampling. As far as relevance to the HERA Phase 1 validation simulations, the visibilities for a handful of point sources is really not a good metric anyway, as "real" sky models have diffuse structure, and in particular the main object of our attention is the cosmological signal which is fundamentally diffuse and in no way well approximated by a dirac delta distribution. A better test of integrating arbitrary diffuse functions is also shown in the RIMEz demonstration_1 notebook. Fringe rate transforms are worth considering in general, but I don't think idea that simply FFTing the output of a pyuvsim point source calculation is a appropriate reference. The bit about the angular band limit is again getting toward something useful but is properly addressed by simply understanding how the calculation is done. |
|
Better/additional notes:
to be continued... |
This notebook validates
RIMEzagainst the first generation ofpyuvsimreference simulations. Test criteria are still TBD and it's still being updated somewhat. Specifically, there is evidence for source position errors fromRIMEz, but there may be some assumptions we missed so we are following that up. All paths are also on the ASU cluster (Enterprise) and I am in the process of moving everything to Lustre. However, the general structure is there.