Feature: Tooling to run tests for Unity packages

[antja0]

Hi!

First of all, thanks to you and all contributors for this repository ❤️

Background

I'm currently open sourcing some Unity packages (tools etc.) from our company to be used with the new Unity Package Manager.
I'm using this repository's action to run tests for those packages.

• Package I'm currently developing: https://github.com/3d-group/unity-simple-notifications (open source)
• Unity docs on custom packages: https://docs.unity3d.com/Manual/CustomPackages.html

Problem

It seems like Unity requires an Unity project to run tests.

To circumvent this problem I created an empty Unity project at my repository and ignored it with .npmignore file and just ran tests there using the projectPath variable.

But I cannot help thinking that it is a bit silly to include empty Unity project just for running tests 😄

Suggested solution

I think this could be automated and included in this action like so:

1. Check if there is package.json in project root in order to detect if project is an upm (unity package manager) package.
   or perhaps this could be told via env variable?
2. Create new Unity project (either copy it from somewhere or download from Unity website?)
3. Edit manifest.json at UnityProject/Packages/

Add following lines to dependencies and testables sections:

{
  "dependencies": {
    "com.company.package": "https://github.com/company/repository.git"
  },
  "testables": [ "com.company.package" ]
}

Perhaps Unity project could also be cached for future action runs?

[webbertakken]

Hi @antja0 and thank you for your feature request.

We are open to improve test runner to account for cases like this.

I think your suggested solution might be a great first iteration. Unity itself is able to create a project, which might be the appropriate way to use here, because a new project may differ per unity version.

Another alternative would be to run the tests directly in C#. I'm not very experienced in Unity tests (looks like nobody is yet), so it would be helpful if you could determine whether you think this would work in general, and for your projects specifically.

Looking forward to hear your thoughts on this.

[BLaZeKiLL]

you could also look into how open upm recommends setting up upm respositories.

basically they use the default unity project as the repo and create release branches using git sub tree, example this way I can have multiple upm packages in 1 repo also

[trudeaua21]

Hey, it's been quite a while since this was active, but I'm developing an open source Unity Package as well and came across this issue.

I was able to get a flow similar to what @antja0 described working locally for my purposes. Next time I get around to working on it (which should be within the next week), I'll try seeing if the same method works on the remote GitHub workflow server (using this and unity-activate). If I end up getting it working, mind if I take a stab at contributing an implementation of this in the action?

Here's my preliminary thoughts on how we could go about implementing this:

• take an input named something like testPackage to determine whether a Unity package is being tested, and maybe an input of the name/absolute file location of the package
• like @webbertakken suggested, use the Unity command Unity.exe -batchMode -createProject ./DummyProject -quit to create a Unity project (maybe one that could be optionally cached?)
• like @antja0 suggested, use a bash/batch script to modify the manifest.json file of the dummy project to add the dependency to the package being tested
• run tests

This process worked for me (sans the bash script to modify the manifest file - I did that in VSCode manually 😅 ), so I feel confident it would work well here!

Edit: I also agree with @BLaZeKiLL - it would be good to document the option of maintaining the package within an otherwise empty Unity project. Since that prevents the package from being installed with git through the Package Manager, though, I still think it would be good to implement creation of an empty project through this action.

[GabLeRoux]

like @antja0 suggested, use a bash/batch script to modify the manifest.json file of the dummy project to add the dependency to the package being tested

Sounds like a good use case for jq cli. Here's how we can insert additional key value:

echo '{"hello": "world"}' | jq --arg foo bar '. + {foo: $foo}'
# {
#   "hello": "world",
#   "foo": "bar"
# }

And some other examples (in the comments too)

[trudeaua21]

I may have figured it out - skip to the final edit

Hey, I'm working on getting this implemented, but I'm new to Docker and running into an issue where the Docker.build test seems to be failing (here's the failing check on my fork, and here's the diff). I would super appreciate it if someone could help me out with this!

The only things I've changed to make the tests fail are adding the jq cli apt-get installation call in the Dockerfile, and adding a basic log in the main workflow to test whether jq installed properly. Oddly enough, all the Unity test suites pass, and jq seems to have installed properly per the test logs, but the Docker.build test fails with the following error:

FAIL src/model/docker.test.js (9.997 s)
  ● Docker › builds

    The process '/usr/bin/docker' failed with exit code 127

      at ExecState._setResult (node_modules/@actions/exec/src/toolrunner.ts:653:17)
      at ExecState.CheckComplete (node_modules/@actions/exec/src/toolrunner.ts:634:12)
      at ChildProcess.<anonymous> (node_modules/@actions/exec/src/toolrunner.ts:507:15)

I've tried looking into this specific error, but due to my lack of Docker experience I'm having a bit of trouble interpreting what I've found.

EDIT - to potentially save someone a Google search - it seems that 127 signifies an illegal command

EDIT 2 - I believe I figured out the reason, being that the test Docker image is using Alpine, which can't use apt-get. This would make sense with the error codes I was encountering. Would it be alright if I changed this to an Ubuntu image, or some other image compatible with apt-get? I only ask because it doesn't look like jq cli is available for apk.

[trudeaua21]

Hey, sorry to ping anyone on the thread if they don't want it. Just wanted to say in case anyone Googles and finds this thread that I'm still in the process of working on this - I had intended to finish it back in October but had some problems getting older versions of Unity to run tests from a package in my local environment. I'm picking it back up again now and hopefully I'll resolve the issue while plugging away.

Edit - Really close now! I'm planning on submitting my PR for this soon - I have it functionally complete and tested through the main workflow. I'm going to make sure that it actually works on one of my own packages before submitting the PR, which may take me a couple days since the package I'm going to test this on doesn't have tests yet 😅

[trudeaua21]

Sorry again in case this pings anyone who doesn't want it - the feature is pretty much complete on my fork. I've added tests to the workflows so I'm pretty sure it works, but I'd like to test it myself on an actual package first before I open my PR.

That said, I'm dealing with an injury atm, so I don't really have the capacity to test it on a real package for at least a couple weeks. If anyone else reading this feels like it, you can give it a shot and test it on your own package before that, and if you can just let me know here whether it works and that will be enough for me to fill out the PR. If not, then I'll just get to it in a few weeks. Thanks everyone! 🎉

[Renardjojo]

Hi @trudeaua21 ! I tested your feature on my package template, but I ran issues each time. Can you help me ?
Here you can find my test project and here is the action that I use :

name: TestPackage2
on:
  workflow_dispatch:

jobs:
  testAllPackageModesLikeInTheReadme:
    name: Test package mode 📦 in ${{ matrix.testMode }} on version ${{ matrix.unityVersion }}
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        projectPath:
          - ./
        unityVersion:
          - 2019.2.11f1
          #- 2020.3.29f1
        testMode:
          - editmode
          #- playmode
    steps:
      ###########################
      #         Checkout        #
      ###########################
      - uses: actions/checkout@v2
         
      #######################
      #         Test        #
      #######################
      # Configure test runner
      - name: Run tests
        id: packageEditMode
        uses: trudeaua21/unity-test-runner@main
        env:
          # Personal license
          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
        with:
          projectPath: ${{ matrix.projectPath }}
          unityVersion: ${{ matrix.unityVersion }}
          testMode: ${{ matrix.testMode }}
          artifactsPath: ${{ matrix.testMode }}-packageArtifacts
          packageMode: true
         
      #############################
      #   Upload artifacts        #
      #############################
      - name: Upload test results
        uses: actions/upload-artifact@v2
        with:
          name: Package test results (edit mode)
          path: ${{ steps.packageEditMode.outputs.artifactsPath }}
          retention-days: 14

If I use 2019.2.11f1 licence with this code, I have this issue :

###########################
#   Testing in EditMode   #
###########################

Aborted (core dumped)
cat: write error: Resource temporarily unavailable
Initiating legacy licensing module

LICENSE SYSTEM [2022228 20:33:49] Next license update check is after 2022-03-01T20:30:13

And if I try with more recent licence (generate with my hub), I had this log :

Package name found: com.open-source-unity-package.test-action
Creating an empty Unity project to add the package to.
Adding package to the temporary project's dependencies and testables...

Packages/mainfest.json was not created properly. This indicates a problem with the Action, not with your package. Aborting...

Error: The process '/usr/bin/docker' failed with exit code 1

I hope this could help to understand providing of the issue

[trudeaua21]

@Renardjojo Thanks for the info! I'll be doing some refactoring for some suggested changes I received from one of the maintainers, so I'll make sure to keep these errors in mind while I'm modifying things.

Did you run the tests on an open-source package? If not, no worries! If so, I'll probably fork it so I can dig deeper into why the issue is occurring there and not on my own test project. My bad, I missed your link the first time I read your message - I'll try to make sure I can get the tests working on your project before I ask for re-review on my PR!

[trudeaua21]

Quick update @Renardjojo - I think at least one of your issues (the one when using the 2019.2.11f1 license was caused by not having a Tests folder (which will likely need to actually have tests in it) in the package. I'll add a condition to check for that case to make that error visible to the developer!

As far as the other issue goes (the one where you used a more recent license), I can't really figure out why it failed there. My first thought was that it was something to do with the license, but I messed around with the licensing on my own project for a bit and was unable to reproduce a failure on that specific step. I was even able to run my own test workflow with the same version as the one where the workflow failed, and the tests ended up running correctly.

My guess is that it's either a license issue that I just don't know enough about to debug, or that maybe Unity just sort of had a random exception. For now, I've just added a little more logging to the step where that second failure occurred.

Thank you so much for your help in testing this out! 😄 If you end up adding some tests to your example project and the issues persist, I'll be happy to take another look!

[Renardjojo]

I test again but it still doesn't work (same error with Tests folder). I have "GUID [...] conflicts with" error and "cat: /github/workspace/editmode-packageArtifacts/editmode-results.xml: No such file or directory" errors.
I have a question about this tool. Is it possible to just install a package in VM and just return if it was correctly installed in unity (see if error or warning happened during installation ? ) like user could do when they install a package thanks to git url ? I made a package template to speed up package creation process and I simply need to check if a user that clone this template creates a package that can be installed in unity. Is it possible with this tool ?

[trudeaua21]

@Renardjojo

About Errors with the Action

I'm sorry to hear that you're still having those errors - are you able to get any package tests running on your own machine? To do so, follow these steps:

• make sure you have your package located somewhere on your machine
• Follow the steps here to add tests to the package if they aren't present already: https://docs.unity3d.com/Manual/cus-tests.html (note that the last step, Enabling Tests for a Package, will have to be done after the next couple of steps)
• open an existing or new project in the Unity editor
• use the Unity Package Manager to add your package to the project you have open
• Follow the steps from the link above in the section titled Enabling Tests for a Package to make it so that your package's tests are picked up by the project that's open in Unity
• Use the steps here to run your tests: https://docs.unity3d.com/Packages/com.unity.test-framework@1.1/manual/workflow-run-test.html
• If you've done everything right, the tests from your package should be showing up in the Unity Test Runner window and passing

If you aren't able to successfully get the tests working this way, then I can't guarantee that this action will work as intended. That said, if you can successfully get tests running using the above steps, and the action still doesn't work after that, then that's definitely a problem.

I should probably document this somewhere, to be honest - I'll likely have to contribute to the game-ci documentation repo as part of this work anyhow, so I'll make sure to include a snippet about this requirement!

About your Other Question

About your question:

Is it possible to just install a package in VM and just return if it was correctly installed in unity (see if error or warning happened during installation ? ) like user could do when they install a package thanks to git url ?

The short answer is maybe? Honestly, I'm not sure what it would take to make something like that.

This is because Unity doesn't really have a way to install a package from the package manager through command line commands. Based on this blurb from UPM, a Unity Package Manager/registry, it would seem like just manually editing a project's manifest file is enough to "install" a package to the project.

The problem is that it's pretty hard to define what "correctly installed" means in your scenario. I don't really have experience using Unity from the command line past my bit of work on this action, so I don't really know what I would look for in Unity's logging to see if it has any problems processing a package's contents.

I will say that if you use this action to run tests on a package, and everything works correctly, then it's likely that the package would work properly in an actual Unity project. However, this isn't really an intended or supported use-case of this action, I'd say - the only real point of the functionality I've built in this action is to run tests from a package.

In my opinion, there's a couple ways to achieve your goal:

1. On your own machine, you could try to add a package to a Unity project's project manifest incorrectly (you could do this by having a copy of the package on your machine, adding the package to the project using the Package Manager, then going into the project manifest file and changing the path to the package to something incorrect). Then, from the command line, you could run the project using Unity Editor commands and see if it throws any errors related to the package's path being wrong. If that happens, I think the tool you're looking for could be built by doing this same process on a GitHub Action.
2. You could simply recommend that before users try adding their package to a project using a git URL, they clone the package to their own machine and add it to a project on their own machine using the Unity Package Manger window. If that works, then it's safe to say that the package would work. This sort of pushes the problem off to the user, but I think it would be a faster process than trying to do what you're looking for in a CI/CD environment.

Apologies for the lengthy message! Please let me know if you have any more questions/clarifications on this topic.