Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tensor slice sampler #329

Merged
merged 14 commits into from
May 5, 2023
Merged

Tensor slice sampler #329

merged 14 commits into from
May 5, 2023

Commits on Apr 23, 2023

  1. Create tfdata_sampler.py

    Initial version of new tf.data.Dataset sampler.
    owenvallis authored Apr 23, 2023
    Configuration menu
    Copy the full SHA
    0ef7de9 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2023

  1. Configuration menu
    Copy the full SHA
    71c65a9 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2d82e81 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    767e527 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    c3c15f4 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2023

  1. Configuration menu
    Copy the full SHA
    2208f1c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    9cbfa7a View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2023

  1. Configuration menu
    Copy the full SHA
    a4c172a View commit details
    Browse the repository at this point in the history
  2. Format file

    owenvallis committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    4dbf73e View commit details
    Browse the repository at this point in the history

Commits on May 1, 2023

  1. Remove as_numpy_iterator when creating the list of grouped datasets.

    * Also move class_list filter to before the group_by function
    * Apply the total_examples_per_class as a take() function on each
      grouped dataset
    * Remove as much casting as possible from the dataset. Certain functions
      expect an int64 though and require casting.
    owenvallis committed May 1, 2023
    Configuration menu
    Copy the full SHA
    473b2b2 View commit details
    Browse the repository at this point in the history

Commits on May 2, 2023

  1. Configuration menu
    Copy the full SHA
    6511b14 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    06ad948 View commit details
    Browse the repository at this point in the history

Commits on May 5, 2023

  1. Move augment_fn and load_fn to before the repeat and batch functions.

    This change means the aug and load functions apply per example now. This
    will make it easier to apply random augmentations per example and is
    more consistent with how we implemented it in the existing memory
    sampler.
    
    This change also improves the tests for all parts of the module.
    owenvallis committed May 5, 2023
    Configuration menu
    Copy the full SHA
    eba02d3 View commit details
    Browse the repository at this point in the history
  2. Add support for handling tuple and dict values for y.

    This change adds support for passing a callable to parse the correct
    class id element for batch sampling. By default y is assumed to be a 1D
    tensor with the class ids and the function is lambda y:y. Otherwise we
    accept an int or str and construct a parser to get the class id tensor.
    owenvallis committed May 5, 2023
    Configuration menu
    Copy the full SHA
    c6541c0 View commit details
    Browse the repository at this point in the history