DataBatch and NDArrayIter doc modified #6091

Roshrini · 2017-05-03T20:55:09Z

@mli @zackchase @madjam @nswamy @indhub @jiajiechen Please take a look.

madjam · 2017-05-03T21:22:12Z

python/mxnet/io.py

@@ -80,7 +80,22 @@ def get_list(shapes, types):
            return [DataDesc(x[0], x[1]) for x in shapes]

 class DataBatch(object):
-    """A data batch.
+    """Returns a batch of data.


Since this is a class, you are not describing what is does but what it encapsulates.

madjam · 2017-05-03T21:22:45Z

python/mxnet/io.py

-    """A data batch.
+    """Returns a batch of data.
+
+    MXNet's data iterator returns a batch of data in each `next` call.


madjam · 2017-05-03T21:27:29Z

python/mxnet/io.py

-          If not provided, the order of arg_names of the executor is assumed.
-          When working with Module this is the order of the data_names argument.
+          The *i*-th element describes the name and shape of ``data[i]``.
+          If not provided, by default the order of `arg_names` of the executor is assumed.


comma after default.

madjam · 2017-05-03T21:29:18Z

python/mxnet/io.py

+    """Returns a batch of data.
+
+    MXNet's data iterator returns a batch of data in each `next` call.
+    This data often contains `batch_size` number of examples.


Can this ever be different from batch_size?

madjam · 2017-05-03T21:29:53Z

python/mxnet/io.py

-          examples read is less than the batch size.
+          The number of examples padded at the end of a batch. It is used when the
+          total number of examples read is not divisible by the `batch_size`.
+          These are ignored in the result.


"These extra padded examples are ignored during processing."

Is the above a correct statement to make?

zackchase · 2017-05-04T07:35:26Z

python/mxnet/io.py

+    MXNet's data iterator returns a batch of data for each `next` call.
+    This data contains `batch_size` number of examples.
+
+    If the input data consists of images then, these images should be stored in a


comma issue -> " If the input data consists of images, then these images..."

zackchase · 2017-05-04T07:38:45Z

python/mxnet/io.py

+    >>> labels = np.ones([10, 1])
+    >>> dataiter = mx.io.NDArrayIter(datas, labels, 3, True, last_batch_handle='discard')
+    >>> dataiter
+    <mxnet.io.NDArrayIter object at 0x10bb2fd90>


Is it descriptive here to have an example where every component of every example has value 1?

zackchase · 2017-05-04T07:39:18Z

python/mxnet/io.py

+    >>> for batch in dataiter:
+    ...    batchidx += 1
+    ...
+    >>> batchidx  # Padding added after the examples read are over


can we make this more clear? "Padding added after the examples read are over"

piiswrong · 2017-05-04T09:49:37Z

python/mxnet/io.py

+    This data contains `batch_size` number of examples.
+
+    If the input data consists of images then, these images should be stored in a
+    4-D matrix of shape ``(batch_size, num_channel, height, width)``.


This depends on the layout. if provide_data gives DataDesc(layout='NHWC') then its (batch_size,, height, width, num_channel)

piiswrong · 2017-05-04T09:53:58Z

python/mxnet/io.py

+
+    Example usage:
+    ----------
+    >>> class CustomBatch(object):


No we don't want users to do this. Use mx.io.DataBatch

piiswrong · 2017-05-04T09:55:27Z

python/mxnet/io.py

    index : numpy.array, optional
          The example indices in this batch.
    bucket_key : int, optional
-          The key of the bucket, used for bucket IO.
+          The bucket key, used for bucketing module.
    provide_data : list of (name, shape), optional


this is deprecated.
It should be a list of DataDesc now

Roshrini · 2017-05-04T18:39:30Z

Addressed all comments.

piiswrong · 2017-05-06T11:52:25Z

python/mxnet/io.py

-          When working with Module this is the order of the label_names argument.
+          The bucket key, used for bucketing module.
+    provide_data : list of `DataDesc`, optional
+          A list of `DataDesc` objects having attributes


explain what the DataDescs are for

piiswrong · 2017-05-06T11:53:16Z

python/mxnet/io.py

+    >>> labels = np.ones([10, 1])
+    >>> dataiter = mx.io.NDArrayIter(data, labels, 3, True, last_batch_handle='discard')
+    >>> dataiter
+    <mxnet.io.NDArrayIter object at 0x10bb2fd90>


we generally don't need to show object pointers like this. Users know what they are because they are just constructed in previous line.

piiswrong · 2017-05-08T23:51:38Z

python/mxnet/io.py

+          name, shape, type and layout information of the data.
+    provide_label : list of `DataDesc`, optional
+          A list of `DataDesc` objects. `DataDesc` is used to store
+          name, shape, type and layout information of the data.


data -> label
The i-th elements describes the ... of data[i].
keep this sentence

nswamy · 2017-05-09T18:19:48Z

python/mxnet/io.py

+    If `layout` is set to 'NHWC' then, images should be stored in a 4-D matrix
+    of shape ``(batch_size, height, width, num_channel)``.
+    The channels are often in RGB order.
+
    Parameters
    ----------
    data : list of NDArray


list of NDArray, each array containing batch_size examples.

nswamy · 2017-05-09T18:20:32Z

python/mxnet/io.py

+    If `layout` is set to 'NHWC' then, images should be stored in a 4-D matrix
+    of shape ``(batch_size, height, width, num_channel)``.
+    The channels are often in RGB order.
+
    Parameters
    ----------
    data : list of NDArray
          A list of input data.
    label : list of NDArray, optional


list of NDArray, each array often containing a 1-dimensional array.

* DataBatch and NDArrayIter doc modified * fixes after review * fixes after review * wording changed * some more fixes * improvement * desc fix * Datadesc info added * minor addition * fix * fix * fix after review

* updated docstring for set_lr_mult and set_wd_mult * updated docstring per review * Fixed imdecode crash bug when flag=0 (#6134) * Fix (#6131) * Docs for MXRecordIO, MXIndexedRecordIO modified (#6013) * docs for MXIndexedRecordIO modified * changes after review * recordIO doc modified * changes after review * lint error * minor change * minor change after review * empty commit to retrigger build * changes after review * Update documentation for mx.callback.Speedometer. (#6058) * Update documentation for mx.callback.Speedometer. * Minor doc changes. * Use module instead of model in example code. * update doc for Load (#6092) * Installation instructions for MacOS and Cloud (#6012) * Fix NDArray bool checking (#6130) * fix shape order bug (#6136) * TOC click unfold (#6133) * [doc] new sphnix plugin (#6105) * update doc * rm * update * update ndarray * update mds * update * update * update * update * update * update * update image.md and others * update * [doc] use debug mode to build (#6151) * move ctc loss to contrib (#6154) * Fix for invalid numpy float indexing (#6144) * Fix python3 compatibilities (#6143) * [doc] small changes to tutorials (#6164) * [doc] Fix left toc link (#6162) * [example]ADD practical functions and options for speech_recognition example (#6141) * ADD practical functions and options for speech_recognition example * add missing stt_bi_graphemes_util.py and deepspeech.cfg template * Added reflection padding (#6123) * Added reflection padding * Lint fix * Added 5d reflection padding * Added failure in forward/backward for input dimensions other than 4 of 5 * Improved sanity check readability * Fixing LICENSE file and adding NOTICE (#6172) * Creating NOTICE. When code moves to Apache, it will need adjusting to the Apache format. * Replacing source header with full license text * doc improvement - softmax, metrics, and initializer (#5945) * doc improvement, softmaxoutput, initializer-constant, minor fixes * doc improvement, metrics * fix softmax doc, fix metric lint * softmax more fixes * add doc change in initializer.py. some minor fix in softmax_cross_entropy * doc change in initializer.py * fix grammer * fix * fix * fix * minor fix * fix * minor fix * DataBatch and NDArrayIter doc modified (#6091) * DataBatch and NDArrayIter doc modified * fixes after review * fixes after review * wording changed * some more fixes * improvement * desc fix * Datadesc info added * minor addition * fix * fix * fix after review * [Scala] Change version to 0.9.5-SNAPSHOT (#6173) * [scala] change version to 0.9.5-SNAPSHOT * API doc improvement Dropout and SoftmaxActivation (#6088) * doc improve for dropout oper * doc improve for SoftmaxActivation oper * fix * fix * Update documentation for mx.callback.do_checkpoint (#6059) * Update documentation for mx.callback.do_checkpoint * Use module instead of model for example code. * Update documentation for plot_graph. (#6098) * Update documentation for plot_graph. * Minor doc fix. * Restruct get started (#6167) * Change get started page * Small fix * Improve * Update documentation of Initializer.dumps() (#6128) * Doc Improvement - RMSProp and RMSPropAlex (#6107) * rmsprop * rmsprop alex * add link in optimizer.py * fix * fix * missed fix.. * Docforcs,fft,ifft (#6145) * fft.cc * add all * changed the description of set_lr_mult and set_wd_mult * Explicitly specify quiet in R install_version (#6171)

* DataBatch and NDArrayIter doc modified * fixes after review * fixes after review * wording changed * some more fixes * improvement * desc fix * Datadesc info added * minor addition * fix * fix * fix after review

* updated docstring for set_lr_mult and set_wd_mult * updated docstring per review * Fixed imdecode crash bug when flag=0 (apache#6134) * Fix (apache#6131) * Docs for MXRecordIO, MXIndexedRecordIO modified (apache#6013) * docs for MXIndexedRecordIO modified * changes after review * recordIO doc modified * changes after review * lint error * minor change * minor change after review * empty commit to retrigger build * changes after review * Update documentation for mx.callback.Speedometer. (apache#6058) * Update documentation for mx.callback.Speedometer. * Minor doc changes. * Use module instead of model in example code. * update doc for Load (apache#6092) * Installation instructions for MacOS and Cloud (apache#6012) * Fix NDArray bool checking (apache#6130) * fix shape order bug (apache#6136) * TOC click unfold (apache#6133) * [doc] new sphnix plugin (apache#6105) * update doc * rm * update * update ndarray * update mds * update * update * update * update * update * update * update image.md and others * update * [doc] use debug mode to build (apache#6151) * move ctc loss to contrib (apache#6154) * Fix for invalid numpy float indexing (apache#6144) * Fix python3 compatibilities (apache#6143) * [doc] small changes to tutorials (apache#6164) * [doc] Fix left toc link (apache#6162) * [example]ADD practical functions and options for speech_recognition example (apache#6141) * ADD practical functions and options for speech_recognition example * add missing stt_bi_graphemes_util.py and deepspeech.cfg template * Added reflection padding (apache#6123) * Added reflection padding * Lint fix * Added 5d reflection padding * Added failure in forward/backward for input dimensions other than 4 of 5 * Improved sanity check readability * Fixing LICENSE file and adding NOTICE (apache#6172) * Creating NOTICE. When code moves to Apache, it will need adjusting to the Apache format. * Replacing source header with full license text * doc improvement - softmax, metrics, and initializer (apache#5945) * doc improvement, softmaxoutput, initializer-constant, minor fixes * doc improvement, metrics * fix softmax doc, fix metric lint * softmax more fixes * add doc change in initializer.py. some minor fix in softmax_cross_entropy * doc change in initializer.py * fix grammer * fix * fix * fix * minor fix * fix * minor fix * DataBatch and NDArrayIter doc modified (apache#6091) * DataBatch and NDArrayIter doc modified * fixes after review * fixes after review * wording changed * some more fixes * improvement * desc fix * Datadesc info added * minor addition * fix * fix * fix after review * [Scala] Change version to 0.9.5-SNAPSHOT (apache#6173) * [scala] change version to 0.9.5-SNAPSHOT * API doc improvement Dropout and SoftmaxActivation (apache#6088) * doc improve for dropout oper * doc improve for SoftmaxActivation oper * fix * fix * Update documentation for mx.callback.do_checkpoint (apache#6059) * Update documentation for mx.callback.do_checkpoint * Use module instead of model for example code. * Update documentation for plot_graph. (apache#6098) * Update documentation for plot_graph. * Minor doc fix. * Restruct get started (apache#6167) * Change get started page * Small fix * Improve * Update documentation of Initializer.dumps() (apache#6128) * Doc Improvement - RMSProp and RMSPropAlex (apache#6107) * rmsprop * rmsprop alex * add link in optimizer.py * fix * fix * missed fix.. * Docforcs,fft,ifft (apache#6145) * fft.cc * add all * changed the description of set_lr_mult and set_wd_mult * Explicitly specify quiet in R install_version (apache#6171)

* DataBatch and NDArrayIter doc modified * fixes after review * fixes after review * wording changed * some more fixes * improvement * desc fix * Datadesc info added * minor addition * fix * fix * fix after review

* updated docstring for set_lr_mult and set_wd_mult * updated docstring per review * Fixed imdecode crash bug when flag=0 (apache#6134) * Fix (apache#6131) * Docs for MXRecordIO, MXIndexedRecordIO modified (apache#6013) * docs for MXIndexedRecordIO modified * changes after review * recordIO doc modified * changes after review * lint error * minor change * minor change after review * empty commit to retrigger build * changes after review * Update documentation for mx.callback.Speedometer. (apache#6058) * Update documentation for mx.callback.Speedometer. * Minor doc changes. * Use module instead of model in example code. * update doc for Load (apache#6092) * Installation instructions for MacOS and Cloud (apache#6012) * Fix NDArray bool checking (apache#6130) * fix shape order bug (apache#6136) * TOC click unfold (apache#6133) * [doc] new sphnix plugin (apache#6105) * update doc * rm * update * update ndarray * update mds * update * update * update * update * update * update * update image.md and others * update * [doc] use debug mode to build (apache#6151) * move ctc loss to contrib (apache#6154) * Fix for invalid numpy float indexing (apache#6144) * Fix python3 compatibilities (apache#6143) * [doc] small changes to tutorials (apache#6164) * [doc] Fix left toc link (apache#6162) * [example]ADD practical functions and options for speech_recognition example (apache#6141) * ADD practical functions and options for speech_recognition example * add missing stt_bi_graphemes_util.py and deepspeech.cfg template * Added reflection padding (apache#6123) * Added reflection padding * Lint fix * Added 5d reflection padding * Added failure in forward/backward for input dimensions other than 4 of 5 * Improved sanity check readability * Fixing LICENSE file and adding NOTICE (apache#6172) * Creating NOTICE. When code moves to Apache, it will need adjusting to the Apache format. * Replacing source header with full license text * doc improvement - softmax, metrics, and initializer (apache#5945) * doc improvement, softmaxoutput, initializer-constant, minor fixes * doc improvement, metrics * fix softmax doc, fix metric lint * softmax more fixes * add doc change in initializer.py. some minor fix in softmax_cross_entropy * doc change in initializer.py * fix grammer * fix * fix * fix * minor fix * fix * minor fix * DataBatch and NDArrayIter doc modified (apache#6091) * DataBatch and NDArrayIter doc modified * fixes after review * fixes after review * wording changed * some more fixes * improvement * desc fix * Datadesc info added * minor addition * fix * fix * fix after review * [Scala] Change version to 0.9.5-SNAPSHOT (apache#6173) * [scala] change version to 0.9.5-SNAPSHOT * API doc improvement Dropout and SoftmaxActivation (apache#6088) * doc improve for dropout oper * doc improve for SoftmaxActivation oper * fix * fix * Update documentation for mx.callback.do_checkpoint (apache#6059) * Update documentation for mx.callback.do_checkpoint * Use module instead of model for example code. * Update documentation for plot_graph. (apache#6098) * Update documentation for plot_graph. * Minor doc fix. * Restruct get started (apache#6167) * Change get started page * Small fix * Improve * Update documentation of Initializer.dumps() (apache#6128) * Doc Improvement - RMSProp and RMSPropAlex (apache#6107) * rmsprop * rmsprop alex * add link in optimizer.py * fix * fix * missed fix.. * Docforcs,fft,ifft (apache#6145) * fft.cc * add all * changed the description of set_lr_mult and set_wd_mult * Explicitly specify quiet in R install_version (apache#6171)

madjam reviewed May 3, 2017

View reviewed changes

madjam added the Doc label May 3, 2017

Roshrini force-pushed the io-docs branch from cd8744b to cd9e605 Compare May 4, 2017 02:00

zackchase suggested changes May 4, 2017

View reviewed changes

piiswrong reviewed May 4, 2017

View reviewed changes

madjam approved these changes May 4, 2017

View reviewed changes

piiswrong reviewed May 6, 2017

View reviewed changes

Roshrini force-pushed the io-docs branch from 0f03f6f to 6d78e70 Compare May 8, 2017 17:48

piiswrong reviewed May 8, 2017

View reviewed changes

Roshrini force-pushed the io-docs branch from 6d78e70 to b75e82b Compare May 8, 2017 23:55

Roshrini added 11 commits May 9, 2017 09:04

DataBatch and NDArrayIter doc modified

7e670bb

fixes after review

7919486

fixes after review

1af634d

wording changed

5df9cb5

some more fixes

9678080

improvement

a0f93e6

desc fix

dba48f6

Datadesc info added

990fddb

minor addition

39f7053

fix

67d836c

fix

bea611a

Roshrini force-pushed the io-docs branch from 71267af to bea611a Compare May 9, 2017 16:04

nswamy reviewed May 9, 2017

View reviewed changes

Roshrini and others added 2 commits May 9, 2017 11:44

fix after review

811ae62

Merge branch 'master' into io-docs

9dae36a

piiswrong merged commit b798eca into apache:master May 10, 2017

Roshrini deleted the io-docs branch July 14, 2017 05:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DataBatch and NDArrayIter doc modified #6091

DataBatch and NDArrayIter doc modified #6091

Roshrini commented May 3, 2017

madjam May 3, 2017

madjam May 3, 2017

madjam May 3, 2017

madjam May 3, 2017

madjam May 3, 2017

zackchase May 4, 2017

zackchase May 4, 2017

zackchase May 4, 2017

piiswrong May 4, 2017

piiswrong May 4, 2017

piiswrong May 4, 2017

Roshrini commented May 4, 2017

piiswrong May 6, 2017

piiswrong May 6, 2017

piiswrong May 8, 2017 •

edited

Loading

nswamy May 9, 2017 •

edited

Loading

nswamy May 9, 2017 •

edited

Loading

DataBatch and NDArrayIter doc modified #6091

DataBatch and NDArrayIter doc modified #6091

Conversation

Roshrini commented May 3, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roshrini commented May 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

piiswrong May 8, 2017 • edited Loading

Choose a reason for hiding this comment

nswamy May 9, 2017 • edited Loading

Choose a reason for hiding this comment

nswamy May 9, 2017 • edited Loading

Choose a reason for hiding this comment

piiswrong May 8, 2017 •

edited

Loading

nswamy May 9, 2017 •

edited

Loading

nswamy May 9, 2017 •

edited

Loading