Streaming inference (#930)

* Introducting plan for streaming inference

* WIP - investigating streaming inference

* Small changes, adding comments for future Drew.

* End of day checkpoint

* Implementing kafka_stream_dataset and iterabel `StreamingDataProvider`.

* Adding a couple of comments.

* Incorporating Max's work into kafka_stream_dataset.

* Cleaning up the kafka_stream_dataset and renaming self.data_loader to be a public variable in `infer_stream.py`.

* Allowing a list of kafka topics to be defined for the consumer. Default is empty list. Single string will be converted to a list. The `data_location` URL parsing still works for a single topic.

* Fixing and adding tests.

* Fixing another test.

* Addressing several PR comments from copilot.

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Fixing incorrectly used function.

* Adding docstrings to producer.py for linting.

* Executing one cell in infer_stream_example.ipynb and fixing dataset_config in there too.

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Updating docs and notebooks with `reduce_dimensions` (#962)

Update documentations and notebooks to replace `umap` with `reduce_dimensions`.

Need to create the torch Generator on the correct device. (#924)

* Need to create the torch Generator on the correct device.

* test: cover dist_data_loader generator device selection

Agent-Logs-Url: https://github.com/lincc-frameworks/hyrax/sessions/4f76dfac-175d-46f2-85e5-a3501f55afef

Co-authored-by: drewoldag <47493171+drewoldag@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

Make training shuffle a train-only option; migrate config and use sam…

…plers for ordering (#923)

Speed up AI container startup. (#882)

Switch AI web-container setup scripts to use a pre-built artifact on main.
Artifact uses conda-pack to package an environment.
Adds a CI job to create the artifact. (currently 3.6GB)
Only one copy of artifact is kept on github, except in case of error creating the artifact.
Setup scripts download the artifact and unpack it before doing hyrax editable install.

Merge pull request #852 from lincc-frameworks/issue-820-listing-avail…

…able-verbs

Add `list_verbs()` function to Hyrax. Returns an alphabetically sorted list of existing Verbs along with required and optional groups and a short description of the Verb.

Update pytorch-ignite requirement from <=0.5.3 to <=0.5.4 (#857)

Updates the requirements on [pytorch-ignite](https://github.com/pytorch/ignite) to permit the latest version.
- [Release notes](https://github.com/pytorch/ignite/releases)
- [Commits](pytorch/ignite@v0.1.0...v0.5.4)

---
updated-dependencies:
- dependency-name: pytorch-ignite
  dependency-version: 0.5.4
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Drew Oldag <47493171+drewoldag@users.noreply.github.com>

Getting started documentation page + CIFAR/CNN updates (#586)

* WIP commit for getting started page.

* Making some progress on the getting started page. Updates to Cifar dataset (and rename to "dataset") and HyraxCNN.

* Fixing sphinx build, tweaking wording on getting started page.

* Added accuracy and confusion matrix to getting started notebook. But HyraxCNN is not performing as well as expected.

* Made the getting_started notebook pretty messy, but will clean it up tomorrow.

* Cleaned up getting started notebook, and made quick-fix adjustments to the cifar dataset class.

* Updating text and adding evaluation section to getting_started.rst. A little more polish on the notebook.

* Correct spelling errors, clean up imports, fix ruff linting error.

* Addressing PR comments.

* Calculate the min required width for the CIFAR dataset ids to reduce output file size.

* Fixing typo from PR comment.

Use simple model.onnx file name and save a copy of config (#574)

* Save and read ONNX model as model.onnx. Save a copy of the runtime config when running to_onnx.

* Fix a broken unit test.

Fixing issues encountered while trying to to-onnx applecider models. (#…

…565)

* Fixing issues encountered while trying to to-onnx applecider models.

* Move set_default_device to a function that is called once.

* Bug fixes in `engine`. Correctly import InferenceDataSetWriter. Fix way we write onnx_results.

* Copy the to_tensor.py file to the onnx output directory.

* Adding initial version of dynamo-based torch onnx export.