Change to parallel reader #635
Conversation
| """ struct for one block : | ||
| contain label, label desc, feature, feature_desc | ||
| class SampleInfo(object): | ||
| """SampleInfo holds the necessary information to load an example from disk. |
|
|
||
| class DataReader(object): | ||
| """DataReader provides basic audio sample preprocessing pipeline including | ||
| I/O and augmentation transforming. |
There was a problem hiding this comment.
I/O and augmentation transforming -> data loading and data augmentation
|
|
||
| Args: | ||
| feature_file_list (str): File containing feature data related files. | ||
| label_file_list (str): File containing label data related files. |
There was a problem hiding this comment.
-> feature_file_list (str): File that lists the paths of all the feature data and their descriptions.
-> label_file_list (str): File that lists the paths of all the label data and their descriptions.
| label_file_list (str): File containing label data related files. | ||
| frame_dim (int): The final feature dimension of one frame after all | ||
| augmentation applied. | ||
| drop_frame_len (int): Lower threshold bound to filter samples having |
There was a problem hiding this comment.
-> drop_frame_len (int): The sequence length threshold above which the samples will be dropped
| batch_samples = [] | ||
| lod = [0] | ||
|
|
||
| if len(batch_samples) >= minimum_batch_size: |
There was a problem hiding this comment.
Can this if statement be merged with the one in above for loop? The resemble each other very much.
| parser.add_argument( | ||
| '--minimum_batch_size', | ||
| type=int, | ||
| default=32, |
There was a problem hiding this comment.
What is the proper default value for this argument?
|
|
||
| class SampleInfoBucket(object): | ||
| """SampleInfoBucket contains paths of several description files. Feature | ||
| description file contains necessary information to access samples' feature |
There was a problem hiding this comment.
Add a simple explanation of necessary information, you can use ( ) or such as, etc.
| """SampleInfoBucket contains paths of several description files. Feature | ||
| description file contains necessary information to access samples' feature | ||
| data and label description file contains necessary information to | ||
| access samples' label data. SampleInfoBucket is the minimum unit to do |
There was a problem hiding this comment.
and label description file contains necessary information to access samples' label data-->, the same with the label description file
| self._batch_buffer_size = batch_buffer_size | ||
| self._process_num = process_num | ||
|
|
||
| def generate_bucket_list(self, is_shuffle): |
There was a problem hiding this comment.
private function use function name "_generate_bucket_list" to discriminate
There was a problem hiding this comment.
If generate_bucket_list is exposed, we can shuffle the block for each epoch. Otherwise, the shuffling can only be done once.
|
We can merge this pr first. |
Resolves #630