[WIP][tool] Add scripts to generate test data for WPT WebNN operations tests #22

BruceDai · 2022-11-02T16:13:02Z

@huningxin @fdwr I added these scripts to generate float64 inputs test data and float32 expected data for WebNN concat tests of WPT, PTAL, thanks.

BruceDai · 2022-11-03T02:50:06Z

Link web-platform-tests/wpt#36782 .

BruceDai · 2022-11-03T13:02:17Z

test/tool/gen-concat.js

+      let pos = 0;
+      for (const shape of inputShapes) {
+        const size = sizeOfShape(shape);
+        const precisionData =


This Line65 is to convert origin float64 data to float32 before invoking the implementation of concat operation, and below Line71 is to convert computed result to be float32 data based on our last synced solution. @huningxin PTAL, thank.

BruceDai · 2022-11-03T13:07:14Z

test/tool/resources/concat.json

@@ -0,0 +1,303 @@
+{
+  "tests": [


This tests part is reused the tests part of test_data file for WPT WebNN concat tests.

BruceDai · 2022-11-03T13:10:03Z

test/tool/test_data/concat_data.json

@@ -0,0 +1,370 @@
+{
+  "inputsData": {


These generated test data of inputsData and expectedData are used as the data of test_data file for WPT WebNN concat tests.

fdwr · 2022-11-08T06:46:20Z

test/tool/test_data/clamp_data.json

@@ -0,0 +1,344 @@
+{
+  "inputsData": {
+    "float64Data": [


So this float64 data was cast to float32 first (and then back to float64) before passing to the baseline reference engine? Likewise, the data is also cast to float32 in the same way when passed to the web API implementation?

We just want to be sure that both inputs see the same initial tensor. If the tests are passing, then I guess they must be cast consistently.

So this float64 data was cast to float32 first (and then back to float64) before passing to the baseline reference engine? Likewise, the data is also cast to float32 in the same way when passed to the web API implementation?

Yes. The float64 data was cast to float32 by same invoking new Float32Array(float64Data) before passing to baseline reference engine and web API implementation.

fdwr · 2022-11-10T05:27:28Z

test/tool/README.md

+node gen-concat.js resources\concat.json
+```
+
+, then you could see similar below output on the console, and find new / updated test data file under `test_data` folder. 


Then typo

Updated, thanks.

fdwr · 2022-11-10T05:47:33Z

test/tool/gen-clamp.js

+  const min = -10.0;
+  const max = 10.0;
+  const savedDataFile = path.join(path.dirname(process.argv[1]), 'test_data', 'clamp_data.json');
+  const jsonDict = utils.readJsonFile(process.argv[2]);


(ignorable, just something I learned today) Huh, so I see that argv[2] in Node.js is the equivalent of argv[1] in C and Python, because argv[0] is actually node.exe itself rather than the main program/script file. Weird.

fdwr · 2022-11-10T05:55:08Z

test/tool/test_data/clamp_data.json

@@ -0,0 +1,344 @@
+{


[minor] I see other files follow hyphen case (gen-concat.js). Should these too? (clamp-data.json)

fdwr · 2022-11-10T06:02:13Z

test/tool/gen-gemm.js

+      toSaveDataDict['expectedData'][expectedDataCategory] =
+          utils.getPrecisionData(result, precisionType);
+    } else {
+      continue;


Is the continue not redundant given the next line will reach the bottom of the for loop? If you want to keep it, consider inverting the if test, moving the continue to the top and unindenting the normal code flow (fewer branch points make it a little easier to follow):

if (toSaveDataDict['expectedData'][expectedDataCategory] !== undefined) { // The expected data is already defined. continue; } const precisionType = test.type; const precisionDataA = utils.getPrecisionDataFromDataDict(toSaveDataDict['inputsData'], test.inputs.a.data, precisionType); ...

fdwr · 2022-11-10T06:07:42Z

test/tool/gen-concat.js

+  const tests = jsonDict.tests;
+
+  for (const test of tests) {
+    const expectedDataCategory = test.expected.data;


Selecting which named entry is the source of your data feels more like a source than a category. How about expectedDataSource?

Updated by expectedDataSource, thanks.

fdwr · 2022-11-10T06:28:16Z

test/tool/utils.js

+    // If found, it replaces it by a trio
+    if ( value instanceof Int8Array ||
+         value instanceof Uint8Array ||
+         value instanceof Uint16Array ||


Int16Array?

I referred to WebNN MLOperandType where didn't define "int16" type, so here I didn't add Int16Array condition, and the condition of Uint16Array is for float16 data handler.

BruceDai

@fdwr Thanks for your reviewing, I've updated PR to address your comments, please take another look, thanks.

BruceDai · 2022-11-10T08:45:59Z

test/tool/utils.js

+    // If found, it replaces it by a trio
+    if ( value instanceof Int8Array ||
+         value instanceof Uint8Array ||
+         value instanceof Uint16Array ||


I referred to WebNN MLOperandType where didn't define "int16" type, so here I didn't add Int16Array condition, and the condition of Uint16Array is for float16 data handler.

BruceDai · 2022-11-10T08:47:07Z

test/tool/gen-concat.js

+  const tests = jsonDict.tests;
+
+  for (const test of tests) {
+    const expectedDataCategory = test.expected.data;


Updated by expectedDataSource, thanks.

BruceDai · 2022-11-10T08:47:24Z

test/tool/README.md

+node gen-concat.js resources\concat.json
+```
+
+, then you could see similar below output on the console, and find new / updated test data file under `test_data` folder. 


Updated, thanks.

BruceDai · 2024-04-16T03:47:45Z

Thanks all for reviewing and sharing guidelines. 👍 👍
Let me close this earlier several PRs, and new PR of generating baseline scripts including fp16 baseline data will submit later.

BruceDai mentioned this pull request Nov 2, 2022

[webnn] Add tests for concat operation. web-platform-tests/wpt#36782

Merged

BruceDai force-pushed the gen_concat_test_data branch from 7ede4c3 to e91e690 Compare November 2, 2022 16:38

[tool] Add scripts to generate test data for WebNN concat tests of WPT.

1230193

BruceDai force-pushed the gen_concat_test_data branch from e91e690 to 1230193 Compare November 3, 2022 12:59

BruceDai commented Nov 3, 2022

View reviewed changes

BruceDai changed the title ~~[tool] Add scripts to generate test data for WebNN concat tests of WPT.~~ [tool] Add scripts to generate test data for WPT WebNN operations tests Nov 4, 2022

[tool] Add scripts to generate test data for WebNN clamp tests of WPT.

d1c1c62

BruceDai force-pushed the gen_concat_test_data branch from a8141f1 to d1c1c62 Compare November 8, 2022 02:07

fdwr reviewed Nov 8, 2022

View reviewed changes

[tool] Add scripts to generate test data for WebNN gemm tests of WPT.

3c1fb4c

fdwr reviewed Nov 10, 2022

View reviewed changes

[tool] Addressed comments.

a69f625

BruceDai commented Nov 10, 2022

View reviewed changes

fdwr approved these changes Nov 10, 2022

View reviewed changes

[tool] Renaming inputDataCategory to inputDataSource.

43f68b6

fdwr approved these changes Nov 11, 2022

View reviewed changes

fdwr mentioned this pull request Dec 6, 2022

[WIP] generate test data for WebNN matmul tests of WPT #23

Closed

BruceDai changed the title ~~[tool] Add scripts to generate test data for WPT WebNN operations tests~~ [WIP][tool] Add scripts to generate test data for WPT WebNN operations tests Mar 13, 2023

BruceDai closed this Apr 16, 2024

[WIP][tool] Add scripts to generate test data for WPT WebNN operations tests #22

[WIP][tool] Add scripts to generate test data for WPT WebNN operations tests #22

Uh oh!

Conversation

BruceDai commented Nov 2, 2022

Uh oh!

BruceDai commented Nov 3, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fdwr Nov 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BruceDai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BruceDai commented Apr 16, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fdwr Nov 10, 2022 •

edited

Loading