Contributing Guidelines

Contributing to Surface

• Setup Surface Locally
  • VSCode configuration
  • Running tests
• Contributing to Surface (after you setup it locally)
• Dependencies
• Troubleshooting
• Specific Surface modules
  • Scanners

The following sections covers a fully detailed list of steps on how to set up and start Surface locally. The first sections are both useful for development but also for users, if they desire to not use Docker.

Setup Surface Locally

1. Make sure you have Python 3.9 or higher
   • https://www.python.org/downloads/
   • brew install python3.9 (restart the Terminal/iTerm after the installation is completed)
   • apt install python3.9
   • ...
2. git clone git@github.com:surface-security/surface.git (ssh config is required - https://docs.github.com/en/authentication/connecting-to-github-with-ssh)
3. cd ./surface
4. Install OS dependencies
   • macOS - brew install openssl mysql freetds
   • macOS - xcode-select --install
   • if macOS Catalina (10.15): read this - https://stackoverflow.com/questions/58272830/python-crashing-on-macos-10-15-beta-19a582a-with-usr-lib-libcrypto-dylib
   • Ubuntu/Debian (Linux) - sudo apt-get install openssl libssl-dev libldap2-dev libsasl2-dev libmysqlclient-dev (for Kali, replace libmysqlclient-dev with default-libmysqlclient-dev)
5. Upgrade your pip - python3 -m pip install --upgrade pip
6. Create and activate virtual environment: using default venv, for instance:

   # MacOS / Linux:
   python3 -m venv venv3
   source venv3/bin/activate
   
   # Windows:
   python3 -m venv venv3
   venv3\Scripts\activate.bat
   

7. Install requirements:

   python3 -m pip install -r requirements_dev.txt
   

   If macOS use export LDFLAGS=-L/usr/local/opt/openssl/lib before the previous the commands above
   Also in macOS, read https://dev.to/ruivieira/fixing-libcrypto-ansible-crashes-on-macos-mm2

8. Create local env file for variables, usernames, passwords: touch local.env
9. (Optional) Default Surface configuration will use a SQLite file which is enough for most development purposes. If you'd rather use MySQL in Docker:
   • Create MySQL docker container bash ../dev/1-create_mysql_container.sh
   • Add the mysql config in local.env

     SURF_DATABASE_URL=mysql://root:@127.0.0.1:3306/surface
     SURF_DATABASE_PASSWORD=""
     

10. Apply all migrations to the database python3 manage.py migrate
11. You can create a super user with python3 manage.py createsuperuser
12. Run the application: python3 manage.py runserver
13. Log in with your created super user account.

Note: You can exit the virtual environment with deactivate.

Note: there are many virtualenv options out there if you don't like venv, it's just the default one. virtualenvwrapper-win, hatch, pyenv, or any other will do just fine.

VSCode Configuration

Sample .vscode/settings.json that includes:

• autocomplete, pytest, sonarqube link, autoformatting (black) and some exclusions from tree browser

{
    "python.autoComplete.extraPaths": [
        "surface",
    ],
    "python.analysis.extraPaths": [
        "surface",
    ],
    "python.testing.pytestArgs": [
        "surface",
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.formatting.provider": "black",
    "files.exclude": {
        "**/__pycache__": true,
        "**/.pytest_cache": true,
        "**/mysql.sql": true
    }
}

You can also use this launch.json example that includes Python debug config, Django's runserver and launchers for commands with and without arguments:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Run Command",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/surface/manage.py",
            "args": [
                "${fileBasenameNoExtension}",
            ],
            "django": true,
            "justMyCode": false
        },
        {
            "name": "Python: Run Command With 1 Arg",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/surface/manage.py",
            "args": [
                "${fileBasenameNoExtension}", "${input:someArgs}"
            ],
            "django": true,
            "justMyCode": false
        },
        {
            "name": "Python: Run Command With 2 Args",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/surface/manage.py",
            "args": [
                "${fileBasenameNoExtension}", "${input:someArgs}", "${input:someArgs2}"
            ],
            "django": true,
            "justMyCode": false
        },
        {
            "name": "Python: Django",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/surface/manage.py",
            "args": [
                "runserver"
            ],
            "django": true,
            "justMyCode": false
        },
        {
            "name": "Debug Unit Test",
            "type": "python",
            "request": "test",
            "justMyCode": false,
        }
    ],
    "inputs": [
        {
          "id": "someArgs",
          "type": "promptString",
          "description": "enter argument"
        },
        {
            "id": "someArgs2",
            "type": "promptString",
            "description": "enter argument 2"
          }
    ]
}

Running tests

Surface and their apps have a battery of unit tests what should be running green at all times. Before changing anything, make sure you have an healthy test environment by running the existing tests.

If you're using sqlite in Surface, no need to perform the below steps. For MySQL:

1. Start MySQL (if you haven't already). With docker, that's as easy as: docker-compose -f dev/docker-compose-dev.yml up mysql.
2. Make sure you have at least SURF_DATABASE_URL set up in surface/local.env, which will tell Surface to use that database.

• if using MySQL from Docker: SURF_DATABASE_URL=mysql://root:@127.0.0.1:23306/surface

1. Run pytest inside the root of the project:

• Using -n4 (multiprocessing for tests) option with pytest is recommended for a speedier run.

Contributing to Surface (after you setup it locally)

Please be a good citizen and write descriptive, short and concise commit messages. Run pylint locally, otherwise the style checks will fail and that could be avoided effortlessly.

1. Create a new branch for your work: git checkout -b my-awesome-branch
2. Write new code or make changes.
   • If you're adding new functionality, use python3 manage.py startapp to automatically generate the correct file tree for you.
   • Add its code dependencies to YOURAPP/requirements.txt even if they're already included by other apps (but make sure to mention same versions otherwise pip will go crazy). These allow us to keep track of each app's dependencies and uses.
   • Update base requirements surface/requirements.txt with -r YOURAPP/requirements.txt.
   • If you do add an app-specific requirements.txt, add that ADD line to docker/Dockerfile in the highlighted section.
3. Whenever you are editing anything in the models, make sure you run - python3 manage.py makemigrations to update the database tables.
4. After you have new migrations or you get new changes from Git, make sure you run again python3 manage.py migrate just to update your table and make sure there are no incompatible changes being pushed.
5. Use a linter to check the quality of the code - we recommend Visual Studio code. Please try to fix the reported issues/warnings.
6. Follow Code Style Guidelines section.
7. Add all files you want to add to the repository - git add <filename(s)>
   • Ideally, commits are brief messages explaining what is added to the project and should be easy to read as well. Avoid committing too many files simultaneously and avoid cryptic commit messages. Both make it harder to review.
   • Do not push your local.env or any other sensitive file. In addition, even though mentioned in the Code Style Guidelines, do not push code with secrets in it.
8. Commit your changes to your branch - git commit -m "MESSAGE"
9. Push the new created branch to GitHub - git push --set-upstream origin my-awesome-branch (you need to defined the upstream only at the first push)
10. Create a new Pull Request (PR) in GitHub from source as being your new branch
11. After the PR is created and the pipeline is passing, assign the core team as reviewer because it is required to have at least one one approval to merge.

Dependencies

Surface can use mulitple stacks of services, like Elasticsearch, dkron, MySQL and the list continues to grow. Plus, you probably have your own. To create a working docker-compose for you, ensure that:

1. Take a look at dev/docker-compose-in-a-box.yml
2. Copy it to dev/docker-compose.yml and make your own adjustments: Read the comments, comment or uncomment the services you don't want
3. Pay attention to the mentioned/required changes in your local.env
4. Launch it with docker-compose -f dev/docker-compose.yml up

Specific Surface modules

Scanners

Check scanners/CONTRIBUTING.md