Open
Description
When I run the tests locally, most of them fail. Also a dialogue pops up to ask if I want to save one of the notebooks
Most of them fail I believe because the images dont match. I looked at the files, and all of the look the same, but maybe it thinks they look different. Maybe comparing PNGs would be better?
yarn run test
yarn run v1.19.2
$ env JEST_PUPPETEER_CONFIG=./config/jest/jest-puppeteer.config.js jest --runInBand --verbose=true --detectOpenHandles --config ./config/jest/jest.config.js
Determining test suites to run...[W 20:39:54.567 LabApp] All authentication is disabled. Anyone who can connect to this server will be able to run code.
[I 20:39:55.902 LabApp] JupyterLab extension loaded from /usr/local/Caskroom/miniconda/base/envs/ibis-vega-transform/lib/python3.7/site-packages/jupyterlab
[I 20:39:55.903 LabApp] JupyterLab application directory is /usr/local/Caskroom/miniconda/base/envs/ibis-vega-transform/share/jupyter/lab
[I 20:39:55.906 LabApp] Serving notebooks from local directory: /Users/saul/p/ibis-vega-transform
[I 20:39:55.906 LabApp] The Jupyter Notebook is running at:
[I 20:39:55.906 LabApp] http://localhost:8080/
[I 20:39:55.906 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
console.log test/ui/test.ts:229
## Running "charting-example.ipynb"
[W 20:40:00.358 LabApp] Notebook examples/performance-charts.ipynb is not trusted
[W 20:40:00.365 LabApp] Notebook examples/charting-example.ipynb is not trusted
INFO: Started server process [92232]
INFO: Uvicorn running on http://0.0.0.0:59491 (Press CTRL+C to quit)
INFO: Waiting for application startup.
INFO: Application startup complete.
[I 20:40:00.720 LabApp] Kernel started: e3a651da-b07e-40ad-8492-d657a14f8a2e
[I 20:40:00.734 LabApp] Kernel started: b21450b8-bfe1-495f-91f7-5919fd79a706
[IPKernelApp] ERROR | No such comm target registered: ibis-vega-transform:compiler
[IPKernelApp] ERROR | No such comm target registered: ibis-vega-transform:compiler
[I 20:40:08.438 LabApp] Build is up to date
[I 20:40:08.854 LabApp] Kernel restarted: b21450b8-bfe1-495f-91f7-5919fd79a706
INFO: Reporting span 30c04757b7533a6e:41ceea03b33579ca:ffded251c796b7a8:1 browser.compileSpec
INFO: Reporting span 30c04757b7533a6e:3fe2d87804cac78:ffded251c796b7a8:1 browser.vegaEmbed
INFO: Reporting span 30c04757b7533a6e:ffded251c796b7a8:a2995eeccea36028:1 browser.renderModel
console.log test/ui/test.ts:253
Found charts: 1
INFO: Reporting span 30c04757b7533a6e:ac95546e57be8303:a2995eeccea36028:1 browser.transform
INFO: Reporting span 30c04757b7533a6e:3d6ef33d2f01d6e2:a2995eeccea36028:1 browser.transform
INFO: Reporting span 30c04757b7533a6e:475f0e403b0e79be:a2995eeccea36028:1 browser.transform
INFO: Reporting span 30c04757b7533a6e:429e0ee91115ca54:a2995eeccea36028:1 browser.transform
INFO: Reporting span 30c04757b7533a6e:5d2c5875c3121846:a2995eeccea36028:1 browser.transform
INFO: Reporting span 30c04757b7533a6e:56c7a28706fef1a4:a2995eeccea36028:1 browser.transform
console.log test/ui/test.ts:261
Setting download folder: /Users/saul/p/images/charting-example
console.log test/ui/test.ts:164
File saved at: /Users/saul/p/images/charting-example/visualization.svg
console.log test/ui/test.ts:165
File renamed to: /Users/saul/p/images/charting-example/0.svg
console.log test/ui/test.ts:280
Original images folder: /Users/saul/p/ibis-vega-transform/images/charting-example
console.log test/ui/test.ts:199
Comparing images: "0.svg"
console.log test/ui/test.ts:200
/Users/saul/p/ibis-vega-transform/images/charting-example/0.svg
console.log test/ui/test.ts:201
/Users/saul/p/images/charting-example/0.svg
console.log test/ui/test.ts:209
{
isSameDimensions: true,
dimensionDifference: { width: 0, height: 0 },
rawMisMatchPercentage: 1.1513410108568976,
misMatchPercentage: '1.15',
diffBounds: { top: 666, left: 237, bottom: 949, right: 922 },
analysisTime: 1261,
getImageDataUrl: [Function]
}
console.log test/ui/test.ts:229
## Running "ibis-altair-extraction.ipynb"
console.log test/ui/test.ts:229
## Running "interactive-slider.ipynb"
console.log test/ui/test.ts:229
## Running "omnisci-vega-example.ipynb"
[I 20:42:07.896 LabApp] Starting buffering for e3a651da-b07e-40ad-8492-d657a14f8a2e:1ffa6aa6-870c-4ad6-91a1-bd4b63ca6790
[I 20:42:07.897 LabApp] Starting buffering for b21450b8-bfe1-495f-91f7-5919fd79a706:ab568c93-8ab2-4161-a4e7-2ee96a01a950
[W 20:42:09.553 LabApp] Notebook examples/charting-example.ipynb is not trusted
[W 20:42:09.565 LabApp] Notebook examples/performance-charts.ipynb is not trusted
[W 20:42:10.148 LabApp] Notebook examples/omnisci-vega-example.ipynb is not trusted
[I 20:42:10.359 LabApp] Kernel started: 82351599-f707-4a25-bb01-3791c110967c
[I 20:42:17.893 LabApp] Kernel restarted: 82351599-f707-4a25-bb01-3791c110967c
[I 20:42:19.110 LabApp] Build is up to date
INFO: Reporting span 76a0faac88c3c1d1:28af9d419b23d430:e993a7e02ef673c5:1 browser.compileSpec
INFO: Reporting span 76a0faac88c3c1d1:b202b2383344cceb:e993a7e02ef673c5:1 browser.vegaEmbed
INFO: Reporting span 76a0faac88c3c1d1:e993a7e02ef673c5:1a644b09f479e532:1 browser.renderModel
console.log test/ui/test.ts:253
Found charts: 1
console.log test/ui/test.ts:261
Setting download folder: /Users/saul/p/images/omnisci-vega-example
INFO: Reporting span 76a0faac88c3c1d1:26cc93b71d8ab551:1a644b09f479e532:1 browser.transform
console.log test/ui/test.ts:164
File saved at: /Users/saul/p/images/omnisci-vega-example/visualization.svg
console.log test/ui/test.ts:165
File renamed to: /Users/saul/p/images/omnisci-vega-example/0.svg
console.log test/ui/test.ts:280
Original images folder: /Users/saul/p/ibis-vega-transform/images/omnisci-vega-example
console.log test/ui/test.ts:199
Comparing images: "0.svg"
console.log test/ui/test.ts:200
/Users/saul/p/ibis-vega-transform/images/omnisci-vega-example/0.svg
console.log test/ui/test.ts:201
/Users/saul/p/images/omnisci-vega-example/0.svg
console.log test/ui/test.ts:209
{
isSameDimensions: true,
dimensionDifference: { width: 0, height: 0 },
rawMisMatchPercentage: 0,
misMatchPercentage: '0.00',
diffBounds: { top: 606, left: 476, bottom: 0, right: 0 },
analysisTime: 348,
getImageDataUrl: [Function]
}
[I 20:42:50.553 LabApp] Kernel shutdown: 82351599-f707-4a25-bb01-3791c110967c
[I 20:42:54.587 LabApp] Starting buffering for b21450b8-bfe1-495f-91f7-5919fd79a706:3373411d-5447-4852-a2be-571135a38c78
[I 20:42:54.587 LabApp] Starting buffering for e3a651da-b07e-40ad-8492-d657a14f8a2e:e5ddb438-84f3-41f4-a245-fac2e68a953f
console.log test/ui/test.ts:229
## Running "performance-charts.ipynb"
[W 20:42:55.700 LabApp] Notebook examples/performance-charts.ipynb is not trusted
[W 20:42:55.709 LabApp] Notebook examples/charting-example.ipynb is not trusted
[I 20:43:04.085 LabApp] Kernel restarted: e3a651da-b07e-40ad-8492-d657a14f8a2e
[I 20:43:05.152 LabApp] Build is up to date
INFO: Reporting span 91f23b4b96ed2827:46a15660395bfac:68dbe4c7d2c69d27:1 browser.compileSpec
INFO: Reporting span cc63502bd76ae0f0:f3f14f0c462254fe:aa4291b947ecaf5e:1 browser.compileSpec
INFO: Reporting span 91f23b4b96ed2827:7cb585f01f74437e:68dbe4c7d2c69d27:1 browser.vegaEmbed
INFO: Reporting span cc63502bd76ae0f0:c211aa6adc411f36:aa4291b947ecaf5e:1 browser.vegaEmbed
INFO: Reporting span 91f23b4b96ed2827:68dbe4c7d2c69d27:68219de9377e5bfb:1 browser.renderModel
INFO: Reporting span cc63502bd76ae0f0:aa4291b947ecaf5e:c827d875909afb22:1 browser.renderModel
console.log test/ui/test.ts:253
Found charts: 2
INFO: Reporting span 91f23b4b96ed2827:b747a24778ab9aa4:68219de9377e5bfb:1 browser.transform
INFO: Reporting span 91f23b4b96ed2827:206f6c66071ee26:68219de9377e5bfb:1 browser.transform
INFO: Reporting span 91f23b4b96ed2827:dbf10a1d59799b6f:68219de9377e5bfb:1 browser.transform
INFO: Reporting span 91f23b4b96ed2827:454f1fa0c38198d0:68219de9377e5bfb:1 browser.transform
INFO: Reporting span 91f23b4b96ed2827:da2006d55cc4c869:68219de9377e5bfb:1 browser.transform
INFO: Reporting span 91f23b4b96ed2827:59ae9dcd17bf109f:68219de9377e5bfb:1 browser.transform
console.log test/ui/test.ts:261
Setting download folder: /Users/saul/p/images/performance-charts
INFO: Reporting span cc63502bd76ae0f0:9845549a4195e069:c827d875909afb22:1 browser.transform
console.log test/ui/test.ts:164
File saved at: /Users/saul/p/images/performance-charts/visualization.svg
console.log test/ui/test.ts:165
File renamed to: /Users/saul/p/images/performance-charts/0.svg
console.log test/ui/test.ts:164
File saved at: /Users/saul/p/images/performance-charts/visualization.svg
console.log test/ui/test.ts:165
File renamed to: /Users/saul/p/images/performance-charts/1.svg
console.log test/ui/test.ts:280
Original images folder: /Users/saul/p/ibis-vega-transform/images/performance-charts
console.log test/ui/test.ts:199
Comparing images: "0.svg"
console.log test/ui/test.ts:200
/Users/saul/p/ibis-vega-transform/images/performance-charts/0.svg
console.log test/ui/test.ts:201
/Users/saul/p/images/performance-charts/0.svg
console.log test/ui/test.ts:209
{
isSameDimensions: true,
dimensionDifference: { width: 0, height: 0 },
rawMisMatchPercentage: 1.1513410108568976,
misMatchPercentage: '1.15',
diffBounds: { top: 666, left: 237, bottom: 949, right: 922 },
analysisTime: 1301,
getImageDataUrl: [Function]
}
console.log test/ui/test.ts:199
Comparing images: "1.svg"
console.log test/ui/test.ts:200
/Users/saul/p/ibis-vega-transform/images/performance-charts/1.svg
console.log test/ui/test.ts:201
/Users/saul/p/images/performance-charts/1.svg
console.log test/ui/test.ts:209
{
isSameDimensions: true,
dimensionDifference: { width: 0, height: 0 },
rawMisMatchPercentage: 0.22698427504953797,
misMatchPercentage: '0.23',
diffBounds: { top: 48, left: 127, bottom: 509, right: 462 },
analysisTime: 426,
getImageDataUrl: [Function]
}
console.log test/ui/test.ts:229
## Running "vega-compiler.ipynb"
console.log test/ui/test.ts:292
[I 20:44:10.322 LabApp] Starting buffering for b21450b8-bfe1-495f-91f7-5919fd79a706:fd9d58ed-50e6-4f3d-805f-2d3282af3094
[I 20:44:10.323 LabApp] Starting buffering for e3a651da-b07e-40ad-8492-d657a14f8a2e:f131b4bc-83ae-4229-bb33-7fbf449c30b2
FAIL test/ui/test.ts (251.952s)
Test Ibis-Vega-Transform
✕ should open and run notebook correctly (62416ms)
✕ should open and run notebook correctly (30005ms)
✕ should open and run notebook correctly (30008ms)
✓ should open and run notebook correctly (55116ms)
✕ should open and run notebook correctly (43723ms)
✕ should open and run notebook correctly (30003ms)
● Test Ibis-Vega-Transform › should open and run notebook correctly
expect(received).toBe(expected) // Object.is equality
Expected: true
Received: false
285 | );
286 | const isBelowThreshold = value => value < threshold;
> 287 | expect(results.every(isBelowThreshold)).toBe(true);
| ^
288 | await page.waitFor(2 * 1000);
289 |
290 | // Close dialog "Are you sure you want to leave..."
at test/ui/test.ts:287:47
● Test Ibis-Vega-Transform › should open and run notebook correctly
TimeoutError: Navigation timeout of 30000 ms exceeded
230 |
231 | // Go to specific notebook file
> 232 | await page.goto(`http://localhost:8080/lab/tree/examples/${file}.ipynb`);
| ^
233 | await page.waitForSelector(
234 | '.lm-DockPanel-tabBar li[data-type="document-title"]'
235 | );
at node_modules/puppeteer/lib/LifecycleWatcher.js:142:21
at FrameManager.navigateFrame (node_modules/puppeteer/lib/FrameManager.js:94:17)
at Frame.goto (node_modules/puppeteer/lib/FrameManager.js:406:12)
at Page.goto (node_modules/puppeteer/lib/Page.js:672:12)
at test/ui/test.ts:232:7
-- ASYNC --
at Frame.<anonymous> (node_modules/puppeteer/lib/helper.js:111:15)
at Page.goto (node_modules/puppeteer/lib/Page.js:672:49)
at Page.<anonymous> (node_modules/puppeteer/lib/helper.js:112:23)
at test/ui/test.ts:232:18
-- ASYNC --
at Page.<anonymous> (node_modules/puppeteer/lib/helper.js:111:15)
at test/ui/test.ts:232:18
● Test Ibis-Vega-Transform › should open and run notebook correctly
TimeoutError: Navigation timeout of 30000 ms exceeded
230 |
231 | // Go to specific notebook file
> 232 | await page.goto(`http://localhost:8080/lab/tree/examples/${file}.ipynb`);
| ^
233 | await page.waitForSelector(
234 | '.lm-DockPanel-tabBar li[data-type="document-title"]'
235 | );
at node_modules/puppeteer/lib/LifecycleWatcher.js:142:21
at FrameManager.navigateFrame (node_modules/puppeteer/lib/FrameManager.js:94:17)
at Frame.goto (node_modules/puppeteer/lib/FrameManager.js:406:12)
at Page.goto (node_modules/puppeteer/lib/Page.js:672:12)
at test/ui/test.ts:232:7
-- ASYNC --
at Frame.<anonymous> (node_modules/puppeteer/lib/helper.js:111:15)
at Page.goto (node_modules/puppeteer/lib/Page.js:672:49)
at Page.<anonymous> (node_modules/puppeteer/lib/helper.js:112:23)
at test/ui/test.ts:232:18
-- ASYNC --
at Page.<anonymous> (node_modules/puppeteer/lib/helper.js:111:15)
at test/ui/test.ts:232:18
● Test Ibis-Vega-Transform › should open and run notebook correctly
expect(received).toBe(expected) // Object.is equality
Expected: true
Received: false
285 | );
286 | const isBelowThreshold = value => value < threshold;
> 287 | expect(results.every(isBelowThreshold)).toBe(true);
| ^
288 | await page.waitFor(2 * 1000);
289 |
290 | // Close dialog "Are you sure you want to leave..."
at test/ui/test.ts:287:47
● Test Ibis-Vega-Transform › should open and run notebook correctly
TimeoutError: Navigation timeout of 30000 ms exceeded
230 |
231 | // Go to specific notebook file
> 232 | await page.goto(`http://localhost:8080/lab/tree/examples/${file}.ipynb`);
| ^
233 | await page.waitForSelector(
234 | '.lm-DockPanel-tabBar li[data-type="document-title"]'
235 | );
at node_modules/puppeteer/lib/LifecycleWatcher.js:142:21
-- ASYNC --
at Frame.<anonymous> (node_modules/puppeteer/lib/helper.js:111:15)
at Page.goto (node_modules/puppeteer/lib/Page.js:672:49)
at Page.<anonymous> (node_modules/puppeteer/lib/helper.js:112:23)
at test/ui/test.ts:232:18
-- ASYNC --
at Page.<anonymous> (node_modules/puppeteer/lib/helper.js:111:15)
at test/ui/test.ts:232:18
Test Suites: 1 failed, 1 total
Tests: 5 failed, 1 passed, 6 total
Snapshots: 0 total
Time: 254.296s, estimated 305s
Ran all test suites.
INFO: Shutting down
[I 20:44:10.519 LabApp] Shutting down 2 kernels
INFO: Finished server process [92232]
[I 20:44:10.624 LabApp] Kernel shutdown: e3a651da-b07e-40ad-8492-d657a14f8a2e
[I 20:44:10.625 LabApp] Kernel shutdown: b21450b8-bfe1-495f-91f7-5919fd79a706
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Metadata
Assignees
Labels
No labels