[Autoscheduler][Sparse] Add sparse dense end to end model tuning support for x86/arm cpu & Some bug fix #7635

jcf94 · 2021-03-11T10:26:54Z

#7313 has introduced the tuning support for Sparse, now this PR brings the end to end model tuning support.

cc @merrymercy @comaniac @FrozenGene @yuchaoli

This PR also contains some bug fix:

Add a separate op strange for conv2d_nhwc, conv2d_nhwc.winograd and depthwise_conv2d_nhwc in ARM CPU that AutoScheduler can perform better
Fix a dense bug in TFLite frontend, which will broken the LayoutRewrite process of AutoScheduler

comaniac

LGTM

merrymercy · 2021-03-12T00:58:10Z

cc @antinucleon

merrymercy · 2021-03-12T00:58:47Z

Do you have any performance numbers or comparisons against existing manual schedules?

jcf94 · 2021-03-12T01:43:15Z

Do you have any performance numbers or comparisons against existing manual schedules?

This PR currently just enables the ability of tuning E2E Sparse network, makes it from 0 to 1.

My colleague @yuchaoli has some results on ARM mobile phone, but currently there seems to have some problem with the TVM main branch that ARM cannot get best performance as we expected. I'll spend some time to fix them.
Later we can add ARM results to TLCBench or write a blog for this.

python/tvm/relay/op/strategy/arm_cpu.py

ANSHUMAN87 · 2021-03-12T08:00:54Z

python/tvm/topi/nn/sparse.py

@@ -470,6 +470,38 @@ def _traverse(t):
    return sparse_input_map


+def random_bsr_matrix(m, n, bs_r, bs_c, density, dtype):


IMO this should not be part of Topi. Either you can put where it is used or I testing.

IMO this should not be part of Topi. Either you can put where it is used or I testing.

Fine, just I'm finding that this has been used in many different places. I'll try to find a better postion.

Moved to topi/sparse/utils.

I am sorry for such delayed response, i missed your reply somehow. What my suggestion is, random_bsr_matrix() does not qualify to be in Topi unless it is required by some Ops. What i could see it is just utility for Tutorial, so lets keep this utility func in Tutorial file itself. Otherwise we have one more option, we can put it as part of tvm.testing which can help other tutorials and testcases as well.

python/tvm/relay/op/strategy/arm_cpu.py

python/tvm/relay/op/strategy/x86.py

ANSHUMAN87 · 2021-03-13T06:07:26Z

python/tvm/relay/analysis/sparse_dense.py

+                1 - sparsity,
+            )
+            register_task_input_buffer(
+                "default", prefix + "W_data", tvm.runtime.ndarray.array(sparse_weight.data)


Lets not hard-code it, we can use the {name + ".data", name + ".indices", name + ".indptr"}

The problem is that we cannot get the "name" during measuring.

Okay, thanks for clarification. But i just wonder if name is not available than, how the logic above prefix is working (i mean the line number 98). Its in the same flow right ? Please let me know in case i am mistaken.

tutorials/auto_scheduler/tune_network_arm.py

ANSHUMAN87 · 2021-03-13T07:15:33Z

#7313 has introduced the tuning support for Sparse, now this PR brings the end to end model tuning support.

cc @merrymercy @comaniac @FrozenGene @yuchaoli

This PR also contains some bug fix:
* Add a separate op strange for conv2d_nhwc, conv2d_nhwc.winograd and depthwise_conv2d_nhwc in ARM CPU that AutoScheduler can perform better

* Fix a dense bug in TFLite frontend, which will broken the LayoutRewrite process of AutoScheduler

I was just wondering whether we can divide this PR into 2, one with bug fix for conv2d and other one with sparse_dense auto scheduler & TFLite bug fix. These 2 are quite unrelated, may not be good to go in 1 PR.

jcf94 · 2021-03-15T02:01:27Z

#7313 has introduced the tuning support for Sparse, now this PR brings the end to end model tuning support.
cc @merrymercy @comaniac @FrozenGene @yuchaoli
This PR also contains some bug fix:
* Add a separate op strange for conv2d_nhwc, conv2d_nhwc.winograd and depthwise_conv2d_nhwc in ARM CPU that AutoScheduler can perform better

* Fix a dense bug in TFLite frontend, which will broken the LayoutRewrite process of AutoScheduler
I was just wondering whether we can divide this PR into 2, one with bug fix for conv2d and other one with sparse_dense auto scheduler & TFLite bug fix. These 2 are quite unrelated, may not be good to go in 1 PR.

Yeah, I agree.

ANSHUMAN87 · 2021-03-15T06:51:21Z

python/tvm/relay/frontend/tflite.py

@@ -1872,7 +1872,7 @@ def convert_fully_connected(self, op):
                out_dtype="int32",
            )
        else:
-            out = _op.nn.dense(in_expr, weight_expr)
+            out = _op.nn.dense(in_expr, weight_expr, units=weight_shape[0])


One small suggestion, if possible, can we add a test case which will indicate the issue fixed here. So that it will help in future breaks.

merrymercy · 2021-03-17T03:02:38Z

tutorials/auto_scheduler/tune_network_x86.py

+    if use_sparse:
+        # This is a test workload that manually transforms a dense model to sparse
+        # Check `tutorials/frontend/deploy_sparse.py` for more examples on how to import a
+        # pretrained model.
+
+        def random_sparse_dense_params(func, params, density, BS_R, BS_C):
+            def deepcopy(param_dic):
+                ret = {}
+                for k, v in param_dic.items():
+                    ret[k] = tvm.nd.array(v.asnumpy())
+                return ret
+
+            new_params = deepcopy(params)
+            dense_weight_names = relay.analysis.sparse_dense._search_dense_op_weight(func)
+            for item in dense_weight_names:
+                name = str(item)
+                shape = new_params[name].shape
+                if shape[0] % BS_R == 0 and shape[1] % BS_C == 0:
+                    new_w = random_bsr_matrix(
+                        shape[0], shape[1], BS_R, BS_C, density, "float32"
+                    ).todense()
+                    new_params[name] = tvm.nd.array(new_w)
+            return new_params
+
+        bs_r = 1
+        sparsity = 0.85
+
+        # Currently we only support to conver dense matmul to sparse dense matmul
+        mod, params = ddo.simplify_fc_transpose.convert(mod["main"], params)
+        params = random_sparse_dense_params(mod, params, BS_R=bs_r, BS_C=1, density=1 - sparsity)
+        mod, params = ddo.bsr_dense.convert(mod, params, (bs_r, 1), sparsity_threshold=0.8)
+
+        mod = tvm.IRModule.from_expr(mod)


Wrap this as a function. Do not let this huge block of code confuse readers who only want to know how to use auto-scheduler for regular networks.

Wrap this as a function. Do not let this huge block of code confuse readers who only want to know how to use auto-scheduler for regular networks.

Simplified this part, and moved the big block to sparse.utils.

jcf94 · 2021-03-26T02:33:05Z

@merrymercy @FrozenGene Currently removed the modifications about ARM CPU from this PR.
Will add them in a seperate PR.

tutorials/auto_scheduler/tune_network_arm.py

merrymercy · 2021-03-26T03:33:36Z

I only have one comment on the tutorial. We should make the tutorials more readable and modifiable for new users. Other parts look good to me.

Dismiss for the fix about ARM will be in another PR.

…ort for x86/arm cpu & Some bug fix (apache#7635) * Add sparse dense end to end model tuning support * Add sparse tuning for arm network * Bug fix for tflite frontend dense with layout rewrite * Move the random_bsr_matrix to sparse.utils

jcf94 added 5 commits March 11, 2021 18:25

Add sparse dense end to end model tuning support

9e47c62

Update

e477927

Update

73b0346

Merge branch 'main' of https://github.com/apache/incubator-tvm into main

9d86fe4

Lint fix

da3fb50

jcf94 changed the title ~~[Autoscheduler] Add sparse dense end to end model tuning support~~ [Autoscheduler][Sparse] Add sparse dense end to end model tuning support Mar 11, 2021

jcf94 added 2 commits March 11, 2021 21:08

Lint fix

b319eeb

Lint fix

c62079c

comaniac approved these changes Mar 11, 2021

View reviewed changes

jcf94 and others added 5 commits March 12, 2021 09:52

Merge branch 'main' into sparse_model

ca43bd3

Add sparse tuning for arm network

0206afe

Add sparse support for arm network

7a708dc

Update

5d0cc86

Update

77941d1

jcf94 changed the title ~~[Autoscheduler][Sparse] Add sparse dense end to end model tuning support~~ [Autoscheduler][Sparse] Add sparse dense end to end model tuning support for x86 & arm cpu Mar 12, 2021

jcf94 mentioned this pull request Mar 12, 2021

[arm cpu][auto scheduler] Auto scheduler @ arm cpu can not reproduce paper's performance #7625

Closed

jcf94 linked an issue Mar 12, 2021 that may be closed by this pull request

[arm cpu][auto scheduler] Auto scheduler @ arm cpu can not reproduce paper's performance #7625

Closed

Update

cb47cda

FrozenGene requested changes Mar 12, 2021

View reviewed changes

python/tvm/relay/op/strategy/arm_cpu.py Outdated Show resolved Hide resolved

ANSHUMAN87 reviewed Mar 12, 2021

View reviewed changes

jcf94 added 2 commits March 12, 2021 17:03

Bug fix for tflite frontend dense with layout rewrite

5379674

Move the random_bsr_matrix to sparse.utils

8f4fc1d

jcf94 changed the title ~~[Autoscheduler][Sparse] Add sparse dense end to end model tuning support for x86 & arm cpu~~ [Autoscheduler][Sparse] Add sparse dense end to end model tuning support for x86/arm cpu & Some bug fix Mar 12, 2021

jcf94 added 2 commits March 12, 2021 18:28

Update

46498fb

Update

03c455c

FrozenGene previously requested changes Mar 12, 2021

View reviewed changes

python/tvm/relay/op/strategy/arm_cpu.py Outdated Show resolved Hide resolved

python/tvm/relay/op/strategy/x86.py Outdated Show resolved Hide resolved

Bug fix

373020b

ANSHUMAN87 reviewed Mar 13, 2021

View reviewed changes

tutorials/auto_scheduler/tune_network_arm.py Show resolved Hide resolved

ANSHUMAN87 reviewed Mar 15, 2021

View reviewed changes

merrymercy requested changes Mar 17, 2021

View reviewed changes

jcf94 added 5 commits March 24, 2021 10:12

Remove the modification of ARM CPU

5325b52

Update

c032319

Update

98de8d8

Update

2f030eb

Update

a02e98d

merrymercy reviewed Mar 26, 2021

View reviewed changes

tutorials/auto_scheduler/tune_network_arm.py Outdated Show resolved Hide resolved

jcf94 added 3 commits March 26, 2021 16:56

Update

6c09f8d

Update

2b404c8

Update

52a1855

merrymercy approved these changes Mar 26, 2021

View reviewed changes

Lintfix

2a92eeb

jcf94 merged commit 612f6ce into apache:main Mar 30, 2021

jcf94 deleted the sparse_model branch March 30, 2021 09:54

junrushao mentioned this pull request Nov 1, 2021

Apache TVM v0.8 Release Note Candidate #9416

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Autoscheduler][Sparse] Add sparse dense end to end model tuning support for x86/arm cpu & Some bug fix #7635

[Autoscheduler][Sparse] Add sparse dense end to end model tuning support for x86/arm cpu & Some bug fix #7635

jcf94 commented Mar 11, 2021 •

edited

Loading

comaniac left a comment

merrymercy commented Mar 12, 2021

merrymercy commented Mar 12, 2021

jcf94 commented Mar 12, 2021 •

edited

Loading

ANSHUMAN87 Mar 12, 2021 •

edited

Loading

jcf94 Mar 12, 2021 •

edited

Loading

jcf94 Mar 12, 2021

ANSHUMAN87 Mar 13, 2021

ANSHUMAN87 Mar 13, 2021

jcf94 Mar 15, 2021

ANSHUMAN87 Mar 15, 2021

ANSHUMAN87 commented Mar 13, 2021

jcf94 commented Mar 15, 2021

ANSHUMAN87 Mar 15, 2021

merrymercy Mar 17, 2021

jcf94 Mar 26, 2021

jcf94 commented Mar 26, 2021

merrymercy commented Mar 26, 2021 •

edited

Loading

		@@ -470,6 +470,38 @@ def _traverse(t):
		return sparse_input_map


		def random_bsr_matrix(m, n, bs_r, bs_c, density, dtype):

[Autoscheduler][Sparse] Add sparse dense end to end model tuning support for x86/arm cpu & Some bug fix #7635

[Autoscheduler][Sparse] Add sparse dense end to end model tuning support for x86/arm cpu & Some bug fix #7635

Conversation

jcf94 commented Mar 11, 2021 • edited Loading

comaniac left a comment

Choose a reason for hiding this comment

merrymercy commented Mar 12, 2021

merrymercy commented Mar 12, 2021

jcf94 commented Mar 12, 2021 • edited Loading

ANSHUMAN87 Mar 12, 2021 • edited Loading

Choose a reason for hiding this comment

jcf94 Mar 12, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ANSHUMAN87 commented Mar 13, 2021

jcf94 commented Mar 15, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jcf94 commented Mar 26, 2021

merrymercy commented Mar 26, 2021 • edited Loading

jcf94 commented Mar 11, 2021 •

edited

Loading

jcf94 commented Mar 12, 2021 •

edited

Loading

ANSHUMAN87 Mar 12, 2021 •

edited

Loading

jcf94 Mar 12, 2021 •

edited

Loading

merrymercy commented Mar 26, 2021 •

edited

Loading