[Tracking Issue] TFLite operator support

[leandron]

In #9187 we implemented quantised version of operators in TFLite frontend. Recently, I just noticed a few more operators (with varying priorities) that can be taken as beginner friendly tasks, for someone who's starting in TVM.

I'm creating this as a tracking issue to mark those operators that need to have quantization support implemented and tested:

• FLOOR_DIV ([TFLite][Frontend] Support quantized floor_div #15724)
• FLOOR_MOD ([TFLite][Frontend] Support quantized floor_mod #15733)
• DIV ([TFLite][Frontend] Support quantized div #15768)
• POW ([TFLite][Frontend] Support quantized Pow #15798)
• REVERSE_SEQUENCE ([TFLite][Frontend] Support quantized Reverse sequence #15915)
• SQUARE ([TFLite][Frontend] Support quantized SQUARE #15914)
• ELU ([TFLite][Frontend] Support quantized ELU #15821)
• LOCAL_RESPONSE_NORMALIZATION (LRN)
• L2_NORMALIZATION
• MIRROR_PAD ([TFLite] Add support for quantized mirror pad #16243)
• ARG_MIN ([TFLite][Frontend] Support quantized ARG_MIN #15922)
• NOT_EQUAL ([TFLite][Frontend] Support quantized NOT_EQUAL #15769)
• LESS ([TFLite][Frontend] Support quantized LESS #15746)
• LESS_EQUAL ([TFLite][Frontend] Support quantized LESS_EQUAL #15790)
• GREATER_EQUAL ([TFLite][Frontend] Support quantized GREATER_EQUAL #15775)

For each operator above, there are actions to be taken:

• Remove the conditional block if self.is_quantized(op): ...
• Implement unit tests
• ask any committer in the project to update this issue, reflecting the PR to support the operators on the list above

#9165 can be used as an example.

In case you're interested, a good starting point is the TFLite frontend:

tvm/python/tvm/relay/frontend/tflite.py

Lines 78 to 185 in bee073b

	self.convert_map = {
	"ABS": self.convert_abs,
	"ADD": self.convert_add,
	"ADD_N": self.convert_add_n,
	"ARG_MAX": self.convert_arg_max,
	"ARG_MIN": self.convert_arg_min,
	"AVERAGE_POOL_2D": self.convert_average_pool2d,
	"BATCH_TO_SPACE_ND": self.convert_batch_to_space_nd,
	"BATCH_MATMUL": self.convert_batch_matmul,
	"CAST": self.convert_cast,
	"CEIL": self.convert_ceil,
	"CONCATENATION": self.convert_concatenation,
	"CONV_2D": self.convert_conv2d,
	"COS": self.convert_cos,
	"DENSIFY": self.convert_densify,
	"DEPTH_TO_SPACE": self.convert_depth_to_space,
	"DEPTHWISE_CONV_2D": self.convert_depthwise_conv2d,
	"DEQUANTIZE": self.convert_dequantize,
	"DETECTION_POSTPROCESS": self.convert_detection_postprocess,
	"DIV": self.convert_div,
	"ELU": self.convert_elu,
	"EQUAL": self.convert_equal,
	"EXP": self.convert_exp,
	"EXPAND_DIMS": self.convert_expand_dims,
	"FAKE_QUANT": self.convert_fake_quant,
	"FILL": self.convert_fill,
	"FLOOR_DIV": self.convert_floor_div,
	"FLOOR_MOD": self.convert_floor_mod,
	"FLOOR": self.convert_floor,
	"FULLY_CONNECTED": self.convert_fully_connected,
	"GATHER": self.convert_gather,
	"GATHER_ND": self.convert_gather_nd,
	"GREATER_EQUAL": self.convert_greater_equal,
	"GREATER": self.convert_greater,
	"HARD_SWISH": self.convert_hard_swish,
	"L2_NORMALIZATION": self.convert_l2_normalization,
	"L2_POOL_2D": self.convert_l2_pool2d,
	"LEAKY_RELU": self.convert_leaky_relu,
	"LESS_EQUAL": self.convert_less_equal,
	"LESS": self.convert_less,
	"LOCAL_RESPONSE_NORMALIZATION": self.convert_lrn,
	"LOG": self.convert_log,
	"LOG_SOFTMAX": self.convert_log_softmax,
	"LOGICAL_AND": self.convert_logical_and,
	"LOGICAL_NOT": self.convert_logical_not,
	"LOGICAL_OR": self.convert_logical_or,
	"LOGISTIC": self.convert_logistic,
	"MATRIX_DIAG": self.convert_matrix_diag,
	"MATRIX_SET_DIAG": self.convert_matrix_set_diag,
	"MAX_POOL_2D": self.convert_max_pool2d,
	"MAXIMUM": self.convert_maximum,
	"MEAN": self.convert_reduce_mean,
	"MINIMUM": self.convert_minimum,
	"MIRROR_PAD": self.convert_mirror_pad,
	"MUL": self.convert_mul,
	"NEG": self.convert_neg,
	"NOT_EQUAL": self.convert_not_equal,
	"ONE_HOT": self.convert_one_hot,
	"PACK": self.convert_pack,
	"PAD": self.convert_pad,
	"PADV2": self.convert_pad,
	"POW": self.convert_pow,
	"PRELU": self.convert_prelu,
	"RANGE": self.convert_range,
	"QUANTIZE": self.convert_quantize,
	"REDUCE_ANY": self.convert_reduce_any,
	"REDUCE_MAX": self.convert_reduce_max,
	"REDUCE_MIN": self.convert_reduce_min,
	"REDUCE_PROD": self.convert_reduce_prod,
	"RELU": self.convert_relu,
	"RELU6": self.convert_relu6,
	"RELU_N1_TO_1": self.convert_relu_n1_to_1,
	"RESHAPE": self.convert_reshape,
	"RESIZE_BILINEAR": self.convert_resize_bilinear,
	"RESIZE_NEAREST_NEIGHBOR": self.convert_resize_nearest_neighbor,
	"ROUND": self.convert_round,
	"RSQRT": self.convert_rsqrt,
	"REVERSE_SEQUENCE": self.convert_reverse_sequence,
	"REVERSE_V2": self.convert_reverse_v2,
	"SELECT": self.convert_select,
	"SHAPE": self.convert_shape,
	"SIN": self.convert_sin,
	"SLICE": self.convert_slice,
	"SOFTMAX": self.convert_softmax,
	"SPACE_TO_BATCH_ND": self.convert_space_to_batch_nd,
	"SPACE_TO_DEPTH": self.convert_space_to_depth,
	"SPARSE_TO_DENSE": self.convert_sparse_to_dense,
	"SPLIT": self.convert_split,
	"SPLIT_V": self.convert_split_v,
	"SQRT": self.convert_sqrt,
	"SQUARE": self.convert_square,
	"SQUARED_DIFFERENCE": self.convert_squared_difference,
	"SQUEEZE": self.convert_squeeze,
	"STRIDED_SLICE": self.convert_strided_slice,
	"SUB": self.convert_sub,
	"SUM": self.convert_reduce_sum,
	"TAN": self.convert_tan,
	"TANH": self.convert_tanh,
	"TILE": self.convert_tile,
	"TOPK_V2": self.convert_topk_v2,
	"TRANSPOSE_CONV": self.convert_transpose_conv,
	"TRANSPOSE": self.convert_transpose,
	"UNPACK": self.convert_unpack,
	"UNIDIRECTIONAL_SEQUENCE_LSTM": self.convert_unidirectional_sequence_lstm,
	"WHERE": self.convert_select,
	"ZEROS_LIKE": self.convert_zeros_like,
	"NON_MAX_SUPPRESSION_V5": self.convert_nms_v5,
	}

[p3achyjr]

hi--if no one has taken this yet, can I give this a shot?

[leandron]

hi--if no one has taken this yet, can I give this a shot?

Absolutely @p3achyjr. Each operator can be submitted individually, so we have self contained, easier to review PRs.

Let us know if you need any help. Thanks!

[tlopex]

@leandron I see that the code has changed a lot from the example#9165. All tests are all combined to one function, so if I just need to remove the conditional block and test it? Besides, because I am totally new to it, I find it a little bit tough to have a test, can you offer me some simple ways?

[p3achyjr]

@tlopex if you're working on this too, would you like to divide up the operators so we don't do redundant work?

[tlopex]

@p3achyjr I'm glad to do that. What operators tasks have you doing now?

[p3achyjr]

I won't be able to work on this for the next couple weeks, so feel free to take anything :)

[leandron]

@leandron I see that the code has changed a lot from the example#9165. All tests are all combined to one function, so if I just need to remove the conditional block and test it? Besides, because I am totally new to it, I find it a little bit tough to have a test, can you offer me some simple ways?

Maybe #14667 can be an updated version of the way to support new operators? Have a look on this one and let me know.

[tlopex]

@p3achyjr So am I. I think I won't have time to do this work until the first few days of September. If you start to work on it, just inform me. Thanks.

[tlopex]

@leandron I have seen #14667 , which is a good example for us to mimic. However, it is not the latest version of code.

[p3achyjr]

I'm back now. I can take a look at FLOOR_DIV.

[tlopex]

@p3achyjr Okay, got it. I am now trying to work on SQUARE.

[p3achyjr]

Finally getting around to this--a draft PR is at #15724

[tlopex]

Got it. Actually I don't know clearly how to write a test for unary element operation... And I will try again in following days.

[p3achyjr]

@tlopex maybe you can try something like this first?

tvm/tests/python/frontend/tflite/test_forward.py

Lines 2249 to 2251 in e3055c1

	def _test_ceil(data, quantized, int_quant_dtype=tf.int8):
	"""One iteration of ceil"""
	return _test_unary_elemwise(math_ops.ceil, data, quantized, int_quant_dtype=int_quant_dtype)

I'm happy to keep working on the elemwise ones :)

[tlopex]

@p3achyjr Thanks. You set me a really great example. I'll try it again.

[Lunderberg]

It looks like this may be causing some flaky CI failures. I'm noticing some failures in tests/python/frontend/tflite/test_forward.py (link) on unrelated PRs. These are occurring in the _test_floor_mod function, added as part of #15733,

[Lunderberg]

Confirmed, when running test_all_elem with only _test_forward_elemwise_quantized(_test_floor_mod) enabled, the test fails about 50% of the time. (44 failures out of 100 iterations).

[p3achyjr]

I'm not able to create a revert--has one already been created?

[p3achyjr]

@Lunderberg what error messages do you see when the tests fail? I wonder if the range should be (0,150) since mod can't be negative, and whether that's causing the failures.

[Lunderberg]

The error message is below (full test output here). Unfortunately, this doesn't indicate a clear location where it occurs, only that the final result was mismatched.

AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1
Mismatched elements: 1 / 18 (5.56%)
Max absolute difference: 129
Max relative difference: 0.50588235
 x: array([[181, 175, 227, 191, 159, 182],
       [135, 188, 128, 206, 131, 148],
       [234, 173, 165, 200, 165, 233]], dtype=uint8)
 y: array([[181, 175, 227, 191, 159, 182],
       [135, 188, 255, 206, 131, 148],
       [234, 173, 165, 200, 165, 233]], dtype=uint8)

[Lunderberg]

Between the mismatches having suspicious 255 expected values, and this comment, I agree that something is incorrect with the range of inputs/outputs. Testing different permutations (e.g. excluding negatives, excluding zero) in _test_elemwise_qnn_out_range didn't result in any combinations that became a valid test, so I suspect it won't be a simple numerical change.

My current guess is that there's something going on with the data generation. Currently, _test_forward_elemwise_quantized generates random uint8 data for the test case, but that could produce zero for the RHS of floormod. That would produce undefined behavior, with different results based on different optimizations.

[p3achyjr]

I see what you're saying--maybe we can add min/max overrides for _test_forward_elemwise_quantized. I'm surprised that div and floor_div aren't failing in this case though, since the rhs can generate 0s :/.

May I ask how you're running these tests multiple times? This seems to just tell you how to run it one time.

[Lunderberg]

I see what you're saying--maybe we can add min/max overrides for _test_forward_elemwise_quantized.

That's what I'm thinking as well. It looks like it currently uses the same range for both quantization and for data generation. I think it will need to override the data generation range to exclude zero from the denominator, but to keep zero in the quantization range as zero may occur in the output.

I'm surprised that div and floor_div aren't failing in this case though, since the rhs can generate 0s :/.

Agreed, as I would expect the same problem to effect any operator with a restricted domain. My guess is that there's some optimization that assumes the inputs to be valid (a legal assumption, as the output is typically undefined when the denominator is zero), and that that optimization is affecting floormod differently from floordiv. It probably would be good to track that optimization down at some point, if it occurs at the TVM level, but I don't think that should delay the re-enabling of the unit test.

May I ask how you're running these tests multiple times?

It's a bit of a hacky way to do so. I commented out everything in test_all_elemwise except for the _test_forward_elemwise_quantized(_test_floor_mod) line, then added a parametrized pytest fixture to the file. When running pytest as usual (python3 -mpytest -sv tests/python/frontend/tflite/test_forward.py::test_all_elemwise), it then repeats every test the number of times specified.

import pytest

@pytest.fixture(params=list(range(100)), autouse=True)
def repeat_all_tests(request):
    return request.param

I suppose I could have just made a for loop, but I was lazy and this let me use pytest's pass/fail counter instead of making my own :P.

[p3achyjr]

I ran some tests--it looks like div, floor_div, and floor_mod all fail. Here is an example of an offending input:

Fail.
Input Node: ['inq_1']
Input Data: [array([[ 12, 215, 181, 106,  92,  19],
       [ 58, 160, 194,  62, 231, 143],
       [ 24, 128,  91, 219,  69,  68]], dtype=uint8)]
TFLite: [[127 129 130 128 125 128]
 [127 131 129 127 129 135]
 [127   0 125 128 127 126]]
TVM: [[127 129 130 128 125 128]
 [127 131 129 127 129 135]
 [127 255 125 128 127 126]]
Raw Data:  [array([[152,  88, 238,   4, 202,  53],
       [ 98, 228,  85,  95, 173, 253],
       [247,  65, 180,  60,  78, 172]], dtype=uint8), array([[ 12, 215, 181, 106,  92,  19],
       [ 58, 160, 194,  62, 231, 143],
       [ 24, 128,  91, 219,  69,  68]], dtype=uint8)]

so at the index where the elements diverge (-5), the input data is 128. Presumably, this is the zero-point for quantization, so min/max overrides won't work.

My question is whether 128 is always the zero point, or if we determine it some other way. If it is, we can just prevent 128 in the RHS. @Lunderberg @leandron do either of you know?

[tlopex]

@p3achyjr

            # with respect to its fp32 input range, defined in fake_quant.
            # s = 255/(fmax-fmin);  m = -fmin*s (the zero point)
            for i in input_arrays:
                try:
                    quant_scale = 255 / (input_range[i][1] - input_range[i][0])
                except ZeroDivisionError:
                    print("Min and max of the input range for tensor " + i + " can't be equal")
                mean = -input_range[i][0] * quant_scale
                input_stats[i] = (mean, quant_scale)

Here, if the range is symmetrical, the zero point will always be 128.

[p3achyjr]

Thanks @tlopex!

I'm working on a draft PR to regenerate the input values if any of them are at the zero point. Hopefully I can have it up soon.

[Lunderberg]

@p3achyjr Rather than regenerating, would it be easier to alter the generated values? That would avoid potentially regenerating data over and over.

# Assuming we know that the operator is mod/div/floormod/floordiv,
# and the generated denominator is in a np.ndarray named "denominator".
denominator[denominator==0] += 1

[p3achyjr]

Thanks for that! sorry not a numpy guy.

[tlopex]

@p3achyjr May I ask you how did you run the tests. Why I use some commands like python3 -mpytest -sv tests/python/frontend/tflite/test_forward.py::test_all_elemwise always get passed even I used offending input you gave. It bothers me for a while.

[p3achyjr]

@tlopex @Lunderberg had a comment above^^

import pytest

@pytest.fixture(params=list(range(100)), autouse=True)
def repeat_all_tests(request):
    return request.param

put that at the bottom of the test file and then run the docker script as usual.

[tlopex]

@p3achyjr Thanks.

[leandron]

I found a couple PRs by @tlopex #15992 and #15915 that were suffering from CI flakiness in past. So I'm retying them again.