test(examples): e2e test for examples #80

akihikokuroda · 2024-10-10T15:00:31Z

Which issue(s) does this pull-request address?

Closes: #62

Description

This PR changes add making examples run during e2e testing.

Checklist

I have read the contributor guide
Linting passes: yarn lint or yarn lint:fix
Formatting is applied: yarn format or yarn format:fix
Unit tests pass: yarn test:unit
E2E tests pass: yarn test:e2e
Tests are included
Documentation is changed or added
Commit messages and PR title follow conventional commits

akihikokuroda · 2024-10-10T15:01:20Z

For now, there is only one test for "memory/agentMemory.ts" added. I want to know if this is in the right direction or not.
Thanks!

Tomas2D

We don't want to modify the existing examples directory.
Testing output is not important.

Please see the Python example that I mentioned in the related Github issue.

akihikokuroda · 2024-10-10T15:35:11Z

@Tomas2D There are not changes in the /examples directory. The tests of the examples are under /tests/examples directory that executes the examples as-is and check if the examples generate any output.

Tomas2D · 2024-10-10T18:04:56Z

Oh. I wanted to say that I don't want 1:1 mapping between the tests/examples and the examples directories.

I would like to have one example test file, test/e2e/examples.ts, which executes (recursively) all files under the examples directory. This file will contain filters (so you can ignore some concrete examples or run them conditionally).

.github/workflows/examples-tests.yml

akihikokuroda · 2024-10-10T18:08:59Z

@Tomas2D Thanks for review and direction. I'll work on it.

akihikokuroda · 2024-10-11T01:24:20Z

yarn test:examples runs the all examples
EXAMPLE_NAME=<example name pattern> yarn test:examples runs the tests that matches the pattern (I couldn't find the way to pass the pattern as an argument of the yarn command)

The interactive examples and some other are failing. I'll look into the failures.

tests/e2e/examples.test.ts

.github/workflows/examples-tests.yml

tests/setup.examples.ts

vitest.examples.config.ts

package.json

akihikokuroda · 2024-10-12T16:13:27Z

In the last commit:

move the examples.test.ts to /tests/examples
limit the number of concurrent executions to 5

akihikokuroda · 2024-10-14T00:36:03Z

The last commit adds the configuration that can exclude examples from the tests in case some examples may not be expected work successfully.

Tomas2D · 2024-10-14T07:12:42Z

tests/examples/examples.test.ts

+    dot: false,
+    realpath: true,
+  });
+  const config = JSON.parse(fs.readFileSync("./tests/examples/example_test.cfg", "utf8"));


I don't see the point of adding a next file. Declare ignored paths in this file directly.
Note: don't use a sync operation in an async environment

Tomas2D · 2024-10-14T07:14:53Z

tests/examples/examples.test.ts

+    it.concurrent(`Run ${example}`, async () => {
+      await semaphore.acquire();


Why can't you just use https://vitest.dev/config/#sequence-concurrent with https://vitest.dev/config/#maxconcurrency? It seems better.

Tomas2D · 2024-10-14T07:15:47Z

tests/examples/examples.test.ts

+    it.concurrent(`Run ${example}`, async () => {
+      await semaphore.acquire();
+      try {
+        execSync(`yarn start -- ${example}`).toString();


Why do you call toString() even though you don't print anything?

btw, I think you should print something.

Tomas2D · 2024-10-14T07:19:10Z

vitest.examples.config.ts

Move this file under examples.

Tomas2D · 2024-10-14T07:19:26Z

vitest.examples.config.ts

+    passWithNoTests: true,
+    testTimeout: 120 * 1000,
+    printConsoleTrace: true,
+    setupFiles: ["./tests/setup.examples.ts"],


Use the same setup as the E2E tests.

or ideally just import e2e test config and spread it here

export define defineConfig({ ...e2eConfig, // changes })

I'm not sure how to do it. What necessary is undefine afterEach function defined in the "setup.ts".

vitest.examples.config.ts

package.json

Tomas2D

So far so good!

tests/examples/examples.test.ts

akihikokuroda · 2024-10-15T20:54:02Z

There are still 3 example errors. I added them in the exclude list for now. They are:

Run examples/cache/fileCache.ts 964ms
Run examples/llms/providers/bam_verbose.ts 1718ms
Run examples/llms/providers/bam.ts 1919ms

Here is the errors.

 FAIL  tests/examples/examples.test.ts > E2E Examples > Run examples/cache/fileCache.ts
Error: Command failed: yarn start -- examples/cache/fileCache.ts <<< "Hello world"
node:internal/modules/run_main:129
    triggerUncaughtException(
    ^

[SerializerError: Class "UnconstrainedCache" was not found!] {
  isFatal: true,
  isRetryable: false,
  context: {},
  [errors]: []
}

Node.js v20.17.0

 FAIL  tests/examples/examples.test.ts > E2E Examples > Run examples/llms/providers/bam_verbose.ts
Error: Command failed: yarn start -- examples/llms/providers/bam_verbose.ts <<< "Hello world"
node:internal/modules/run_main:129
    triggerUncaughtException(
    ^

[LLMError: LLM has occurred an error!] {
  isFatal: false,
  isRetryable: false,
  context: {},
  [errors]: [
    HttpError: Model not found
        at clientErrorWrapper (file:///Users/akihikokuroda/development/repositories/agenticflow/bee-agent-framework/node_modules/@ibm-generative-ai/node-sdk/dist/chunk-D3OWGAYN.js:321:13)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async file:///Users/akihikokuroda/development/repositories/agenticflow/bee-agent-framework/node_modules/p-queue-compat/dist/esm/index.js:118:26 {
      error: 'Not Found',
      status_code: 404,
      extensions: {
        code: 'NOT_FOUND',
        state: { model_id: 'meta-llama/llama-3-1-70b-instruct' }
      },
      [cause]: {
        status_code: 404,
        error: 'Not Found',
        message: 'Model not found',
        extensions: {
          code: 'NOT_FOUND',
          state: { model_id: 'meta-llama/llama-3-1-70b-instruct' }
        }
      }
    }
  ]
}

Node.js v20.17.0

 ❯ clientErrorWrapper node_modules/@ibm-generative-ai/node-sdk/dist/chunk-D3OWGAYN.js:321:13

 FAIL  tests/examples/examples.test.ts > E2E Examples > Run examples/llms/providers/bam.ts
Error: Command failed: yarn start -- examples/llms/providers/bam.ts <<< "Hello world"
node:internal/modules/run_main:129
    triggerUncaughtException(
    ^

[LLMError: LLM has occurred an error!] {
  isFatal: false,
  isRetryable: false,
  context: {},
  [errors]: [
    HttpError: Model not found
        at clientErrorWrapper (file:///Users/akihikokuroda/development/repositories/agenticflow/bee-agent-framework/node_modules/@ibm-generative-ai/node-sdk/dist/chunk-D3OWGAYN.js:321:13)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async file:///Users/akihikokuroda/development/repositories/agenticflow/bee-agent-framework/node_modules/p-queue-compat/dist/esm/index.js:118:26 {
      error: 'Not Found',
      status_code: 404,
      extensions: {
        code: 'NOT_FOUND',
        state: { model_id: 'meta-llama/llama-3-1-70b-instruct' }
      },
      [cause]: {
        status_code: 404,
        error: 'Not Found',
        message: 'Model not found',
        extensions: {
          code: 'NOT_FOUND',
          state: { model_id: 'meta-llama/llama-3-1-70b-instruct' }
        }
      }
    }
  ]
}

Node.js v20.17.0

 ❯ clientErrorWrapper node_modules/@ibm-generative-ai/node-sdk/dist/chunk-D3OWGAYN.js:321:13

Tomas2D · 2024-10-15T21:14:05Z

Regarding BAM (llama 3.1 is accessible only to some users). Let's ignore it for now. I will resolve this internally.
Change EXAMPLE_NAME to INCLUDE or INCLUDE_PATTERN
Add EXCLUDE or EXCLUDE_PATTERN optional env and use ignore property in the glob function
What wraps my head around is this not working test examples/cache/fileCache.ts. It works on my machine.
- Try to 1. run the example without vitest (yarn start examples/cache/fileCache.ts or 2. replace os.tmpdir with some different static path.

akihikokuroda · 2024-10-16T14:56:18Z

The environment variable names are now "INCLUDE_PATTERN" and "EXCLUDE_PATTERN".
The fileCache.ts works by replacing the os.tmpdir for me. Thanks!

Tomas2D · 2024-10-17T14:11:05Z

tests/examples/examples.test.ts

+    if (exclude.includes(example.slice(prefixLength))) {
+      continue;
+    }


This can be simplified.

if (!exclude.startsWith('examples') { continue; }

The condition is to exclude such as when
examples is examples/templates/objects.ts
and exclude contains templates/objects.ts.

So the items in exclude list don't need to have examples/.

Tomas2D · 2024-10-17T14:12:01Z

tests/examples/examples.test.ts

+      continue;
+    }
+    it.concurrent(`Run ${example}`, async () => {
+      const { stdout } = await execAsync(`yarn start -- ${example} <<< "Hello world"`);


we should also check for stderr being empty and return code being equal to 0

Tomas2D · 2024-10-17T14:13:21Z

The os.tmpdir() should be cross-platform. Which operating system are you using? This should work

akihikokuroda · 2024-10-17T14:15:53Z

It's M1 Mac. Sonoma 14.5

Signed-off-by: Akihiko Kuroda <akihikokuroda2020@gmail.com>

Tomas2D self-requested a review October 10, 2024 15:27

Tomas2D requested changes Oct 10, 2024

View reviewed changes

Tomas2D reviewed Oct 10, 2024

View reviewed changes

.github/workflows/examples-tests.yml Outdated Show resolved Hide resolved

Tomas2D reviewed Oct 10, 2024

View reviewed changes

.github/workflows/examples-tests.yml Outdated Show resolved Hide resolved

akihikokuroda force-pushed the examples-e2e branch 2 times, most recently from 3c87777 to d32d11c Compare October 11, 2024 01:16

akihikokuroda requested a review from Tomas2D October 11, 2024 01:24

Tomas2D force-pushed the main branch from 717ed26 to 65b7da4 Compare October 11, 2024 11:46

Tomas2D requested changes Oct 11, 2024

View reviewed changes

tests/e2e/examples.test.ts Outdated Show resolved Hide resolved

akihikokuroda force-pushed the examples-e2e branch 2 times, most recently from 85726e9 to 043c459 Compare October 11, 2024 19:46

Tomas2D requested changes Oct 11, 2024

View reviewed changes

.github/workflows/examples-tests.yml Outdated Show resolved Hide resolved

.github/workflows/examples-tests.yml Show resolved Hide resolved

tests/setup.examples.ts Outdated Show resolved Hide resolved

vitest.examples.config.ts Outdated Show resolved Hide resolved

Tomas2D assigned akihikokuroda Oct 11, 2024

akihikokuroda force-pushed the examples-e2e branch from 043c459 to 493ef76 Compare October 11, 2024 20:15

Tomas2D reviewed Oct 11, 2024

View reviewed changes

package.json Outdated Show resolved Hide resolved

akihikokuroda force-pushed the examples-e2e branch from 493ef76 to 960e407 Compare October 12, 2024 16:10

akihikokuroda force-pushed the examples-e2e branch 2 times, most recently from ff6eed3 to 56ee5ee Compare October 14, 2024 00:33

akihikokuroda changed the title ~~[WIP] test(examples): e2e test for examples~~ test(examples): e2e test for examples Oct 14, 2024

Tomas2D requested changes Oct 14, 2024

View reviewed changes

akihikokuroda force-pushed the examples-e2e branch 3 times, most recently from d95f4e7 to 11b92b8 Compare October 14, 2024 19:55

Tomas2D reviewed Oct 14, 2024

View reviewed changes

package.json Outdated Show resolved Hide resolved

akihikokuroda force-pushed the examples-e2e branch from 11b92b8 to c12f26d Compare October 14, 2024 20:30

Tomas2D requested changes Oct 15, 2024

View reviewed changes

tests/examples/examples.test.ts Outdated Show resolved Hide resolved

akihikokuroda force-pushed the examples-e2e branch from c12f26d to 783bbe5 Compare October 15, 2024 20:47

akihikokuroda force-pushed the examples-e2e branch 2 times, most recently from 8b13f9b to d4b7b92 Compare October 16, 2024 14:55

akihikokuroda force-pushed the examples-e2e branch from d4b7b92 to c11d59a Compare October 16, 2024 14:59

Tomas2D reviewed Oct 17, 2024

View reviewed changes

test(examples): e2e test for examples

96a57d8

Signed-off-by: Akihiko Kuroda <akihikokuroda2020@gmail.com>

akihikokuroda force-pushed the examples-e2e branch from c11d59a to 96a57d8 Compare October 17, 2024 15:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(examples): e2e test for examples #80

test(examples): e2e test for examples #80

akihikokuroda commented Oct 10, 2024 •

edited

Loading

akihikokuroda commented Oct 10, 2024

Tomas2D left a comment

akihikokuroda commented Oct 10, 2024

Tomas2D commented Oct 10, 2024

akihikokuroda commented Oct 10, 2024

akihikokuroda commented Oct 11, 2024 •

edited

Loading

akihikokuroda commented Oct 12, 2024

akihikokuroda commented Oct 14, 2024

Tomas2D Oct 14, 2024

Tomas2D Oct 14, 2024

Tomas2D Oct 14, 2024

Tomas2D Oct 14, 2024

Tomas2D Oct 14, 2024

Tomas2D Oct 14, 2024

Tomas2D Oct 14, 2024

akihikokuroda Oct 14, 2024

Tomas2D left a comment

akihikokuroda commented Oct 15, 2024

Tomas2D commented Oct 15, 2024

akihikokuroda commented Oct 16, 2024

Tomas2D Oct 17, 2024

akihikokuroda Oct 17, 2024

Tomas2D Oct 17, 2024

Tomas2D commented Oct 17, 2024

akihikokuroda commented Oct 17, 2024

		it.concurrent(`Run ${example}`, async () => {
		await semaphore.acquire();

test(examples): e2e test for examples #80

Are you sure you want to change the base?

test(examples): e2e test for examples #80

Conversation

akihikokuroda commented Oct 10, 2024 • edited Loading

Which issue(s) does this pull-request address?

Description

Checklist

akihikokuroda commented Oct 10, 2024

Tomas2D left a comment

Choose a reason for hiding this comment

akihikokuroda commented Oct 10, 2024

Tomas2D commented Oct 10, 2024

akihikokuroda commented Oct 10, 2024

akihikokuroda commented Oct 11, 2024 • edited Loading

akihikokuroda commented Oct 12, 2024

akihikokuroda commented Oct 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Tomas2D left a comment

Choose a reason for hiding this comment

akihikokuroda commented Oct 15, 2024

Tomas2D commented Oct 15, 2024

akihikokuroda commented Oct 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Tomas2D commented Oct 17, 2024

akihikokuroda commented Oct 17, 2024

akihikokuroda commented Oct 10, 2024 •

edited

Loading

akihikokuroda commented Oct 11, 2024 •

edited

Loading