Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(docs): update README to reflect manager move #1519

Merged
merged 7 commits into from
Mar 12, 2024
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 30 additions & 101 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,126 +2,55 @@

![Build and Test](https://github.com/Jigsaw-Code/outline-server/actions/workflows/build_and_test_debug.yml/badge.svg?branch=master) [![Mattermost](https://badgen.net/badge/Mattermost/Outline%20Community/blue)](https://community.internetfreedomfestival.org/community/channels/outline-community) [![Reddit](https://badgen.net/badge/Reddit/r%2Foutlinevpn/orange)](https://www.reddit.com/r/outlinevpn/)

This repository has all the code needed to create and manage Outline servers on
DigitalOcean. An Outline server runs instances of Shadowsocks proxies and
provides an API used by the Outline Manager application.
Outline Server is a tool to create and manage your own VPN servers using Shadowsocks proxies. If you prefer a graphical interface, you can use the [Outline Manager](https://github.com/Jigsaw-Code/outline-apps/).
sbruens marked this conversation as resolved.
Show resolved Hide resolved

Go to https://getoutline.org for ready-to-use versions of the software. **To join our Outline Community, [sign up for the IFF Mattermost](https://internetfreedomfestival.org/wiki/index.php/IFF_Mattermost).**
**Components:**

## Components
- **Outline Server** ([`src/shadowbox`](src/shadowbox)): The core proxy server that runs and manages [outline-ss-server](https://github.com/Jigsaw-Code/outline-ss-server/), a Shadowsocks backend. It provides a REST API for access key management.

The system comprises the following components:
- **Metrics Server** ([`src/metrics_server`](src/metrics_server)): A REST service for optional, anonymous metrics sharing.

- **Outline Server**: a proxy server that runs a Shadowsocks instance for each
access key and a REST API to manage the access keys. The Outline Server runs
in a Docker container in the host machine.
**Join the Outline Community** by signing up for the [IFF Mattermost](https://wiki.digitalrights.community/index.php?title=IFF_Mattermost)!

See [`src/shadowbox`](src/shadowbox)
## Shadowsocks and Anti-Censorship

- **Outline Manager:** an [Electron](https://electronjs.org/) application that
can create Outline Servers on the cloud and talks to their access key
management API to manage who has access to the server.
Outline's use of Shadowsocks means it benefits from ongoing improvements that strengthen its resistance against detection and blocking.

See [`src/server_manager`](src/server_manager)
**Key Protections:**

- **Metrics Server:** a REST service that the Outline Server talks to
if the user opts-in to anonymous metrics sharing.
- **AEAD ciphers** are mandatory.
- **Probing resistance** mitigates detection techniques.
- **Protection against replayed data.**
- **Variable packet sizes** to hinder identification.

See [`src/metrics_server`](src/metrics_server)
[Read more](docs/shadowsocks.md) about Shadowsocks resistance against detection and blocking.
sbruens marked this conversation as resolved.
Show resolved Hide resolved

## Code Prerequisites
## Installation

In order to build and run the code, you need the following installed:
**Prerequisites**

- [Node](https://nodejs.org/en/download/) LTS (`lts/hydrogen`, version `18.16.0`)
- [NPM](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) (version `9.5.1`)
- Manager-specific
- [Wine](https://www.winehq.org/download), if you would like to generate binaries for Windows.
- Server-specific
- [Docker](https://docs.docker.com/engine/install/), to build the Docker image and to run the integration test.
- [docker-compose](https://docs.docker.com/compose/install/), to run the integration test.

> 💡 NOTE: if you have `nvm` installed, run `nvm use` to switch to the correct node version!
1. **Install dependencies**

Install dependencies with:
```sh
npm install
```

```sh
npm install
```
1. **Start the server**

This project uses [NPM workspaces](https://docs.npmjs.com/cli/v7/using-npm/workspaces/).
```sh
npm run action shadowbox/server/start
```

## Build System
Exploring further options:

We have a very simple build system based on package.json scripts that are called using `npm run`
and a thin wrapper for what we call build "actions".
- **Refer to the README:** Find additional configuration and usage options in the core server's [`README`](src/shadowbox/README.md).
- **Learn about the build system:** For in-depth build system information, consult the [build documentation](docs/build.md).

We've defined a package.json script called `action` whose parameter is a relative path:
1. **To clean up**

```shell
npm run action $ACTION
```

This command will define a `run_action()` function and call `${ACTION}.action.sh`, which must exist.
The called action script can use `run_action` to call its dependencies. The $ACTION parameter is
always resolved from the project root, regardless of the caller location.

The idea of `run_action` is to keep the build logic next to where the relevant code is.
It also defines two environmental variables:

- ROOT_DIR: the root directory of the project, as an absolute path.
- BUILD_DIR: where the build output should go, as an absolute path.

> ⚠️ To find all the actions in this project, run `npm run action:list`

### Build output

Building creates the following directories under `build/`:

- `web_app/`: The Manager web app.
- `static/`: The standalone web app static files. This is what one deploys to a web server or runs with Electron.
- `electron_app/`: The launcher desktop Electron app
- `static/`: The Manager Electron app to run with the electron command-line
- `bundled/`: The Electron app bundled to run standalone on each platform
- `packaged/`: The Electron app bundles packaged as single files for distribution
- `invite_page`: the Invite Page
- `static`: The standalone static files to be deployed
- `shadowbox`: The Proxy Server

The directories have subdirectories for intermediate output:

- `ts/`: Autogenerated Typescript files
- `js/`: The output from compiling Typescript code
- `browserified/`: The output of browserifying the JavaScript code

To clean up:

```
npm run clean
```

## Shadowsocks Resistance Against Detection and Blocking

Shadowsocks used to be blocked in some countries, and because Outline uses Shadowsocks, there has been skepticism about Outline working in those countries. In fact, people have tried Outline in the past and had their servers blocked.

However, since the second half of 2020 things have changed. The Outline team and Shadowsocks community made a number of improvements that strengthened Shadowsocks beyond the censor's current capabilities.

As shown in the research [How China Detects and Blocks Shadowsocks](https://gfw.report/talks/imc20/en/), the censor uses active probing to detect Shadowsocks servers. The probing may be triggered by packet sniffing, but that's not how the servers are detected.

Even though Shadowsocks is a standard, it leaves a lot of room for choices on how it's implemented and deployed.

First of all, you **must use AEAD ciphers**. The old stream ciphers are easy to break and manipulate, exposing you to simple detection and decryption attacks. Outline has banned all stream ciphers, since people copy old examples to set up their servers. The Outline Manager goes further and picks the cipher for you, since users don't usually know how to choose a cipher, and it generates a long random secret, so you are not vulnerable to dictionary-based attacks.

Second, you need **probing resistance**. Both shadowsocks-libev and Outline have added that. The research [Detecting Probe-resistant Proxies](https://www.ndss-symposium.org/ndss-paper/detecting-probe-resistant-proxies/) showed that, in the past, an invalid byte would trigger different behaviors whether it was inserted in positions 49, 50 or 51 of the stream, which is very telling. That behavior is now gone, and the censor can no longer rely on that.

Third, you need **protection against replayed data**. Both shadowsocks-libev and Outline have added such protection, which you may need to enable explicitly on ss-libev, but it's the default on Outline.

Fourth, Outline and clients using shadowsocks-libev now **merge the SOCKS address and the initial data** in the same initial encrypted frame, making the size of the first packet variable. Before the first packet only had the SOCKS address, with a fixed size, and that was a giveaway.

The censors used to block Shadowsocks, but Shadowsocks has evolved, and in 2021, it was ahead again in the cat and mouse game.

In 2022 China started blocking seemingly random traffic ([report](https://www.opentech.fund/news/exposing-the-great-firewalls-dynamic-blocking-of-fully-encrypted-traffic/)). While there is no evidence they could detect Shadowsocks, the protocol ended up blocked.

As a reponse, we [added a feature to the Outline Client](https://github.com/Jigsaw-Code/outline-client/pull/1454) that allows service managers to specify in the access key a prefix to be used in the Shadowsocks initialization, which can be used to bypass the blocking in China.

Shadowsocks remains our protocol of choice because it's simple, well understood and very performant. Furthermore, it has an enthusiastic community of very smart people behind it.
```sh
npm run clean
```
23 changes: 23 additions & 0 deletions docs/build.md
sbruens marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Build Actions

We have a very simple build system based on package.json scripts that are called using `npm run`
and a thin wrapper for what we call build "actions".

We've defined a package.json script called `action` whose parameter is a relative path:

```sh
npm run action $ACTION
```

This command will define a `run_action()` function and call `${ACTION}.action.sh`, which must exist.
The called action script can use `run_action` to call its dependencies. The $ACTION parameter is
always resolved from the project root, regardless of the caller location.

The idea of `run_action` is to keep the build logic next to where the relevant code is.
It also defines two environmental variables:

- `ROOT_DIR`: the root directory of the project, as an absolute path.
- `BUILD_DIR`: where the build output should go, as an absolute path.

> [!TIP]
> To find all the actions in this project, run `npm run action:list`
25 changes: 25 additions & 0 deletions docs/shadowsocks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Shadowsocks Resistance Against Detection and Blocking

Shadowsocks used to be blocked in some countries, and because Outline uses Shadowsocks, there has been skepticism about Outline working in those countries. In fact, people have tried Outline in the past and had their servers blocked.

However, since the second half of 2020 things have changed. The Outline team and Shadowsocks community made a number of improvements that strengthened Shadowsocks beyond the censor's current capabilities.

As shown in the research [How China Detects and Blocks Shadowsocks](https://gfw.report/talks/imc20/en/), the censor uses active probing to detect Shadowsocks servers. The probing may be triggered by packet sniffing, but that's not how the servers are detected.

Even though Shadowsocks is a standard, it leaves a lot of room for choices on how it's implemented and deployed.

First of all, you **must use AEAD ciphers**. The old stream ciphers are easy to break and manipulate, exposing you to simple detection and decryption attacks. Outline has banned all stream ciphers, since people copy old examples to set up their servers. The Outline Manager goes further and picks the cipher for you, since users don't usually know how to choose a cipher, and it generates a long random secret, so you are not vulnerable to dictionary-based attacks.

Second, you need **probing resistance**. Both shadowsocks-libev and Outline have added that. The research [Detecting Probe-resistant Proxies](https://www.ndss-symposium.org/ndss-paper/detecting-probe-resistant-proxies/) showed that, in the past, an invalid byte would trigger different behaviors whether it was inserted in positions 49, 50 or 51 of the stream, which is very telling. That behavior is now gone, and the censor can no longer rely on that.

Third, you need **protection against replayed data**. Both shadowsocks-libev and Outline have added such protection, which you may need to enable explicitly on ss-libev, but it's the default on Outline.

Fourth, Outline and clients using shadowsocks-libev now **merge the SOCKS address and the initial data** in the same initial encrypted frame, making the size of the first packet variable. Before the first packet only had the SOCKS address, with a fixed size, and that was a giveaway.

The censors used to block Shadowsocks, but Shadowsocks has evolved, and in 2021, it was ahead again in the cat and mouse game.

In 2022 China started blocking seemingly random traffic ([report](https://www.opentech.fund/news/exposing-the-great-firewalls-dynamic-blocking-of-fully-encrypted-traffic/)). While there is no evidence they could detect Shadowsocks, the protocol ended up blocked.

As a reponse, we [added a feature to the Outline Client](https://github.com/Jigsaw-Code/outline-apps/pull/1454) that allows service managers to specify in the access key a prefix to be used in the Shadowsocks initialization, which can be used to bypass the blocking in China.

Shadowsocks remains our protocol of choice because it's simple, well understood and very performant. Furthermore, it has an enthusiastic community of very smart people behind it.
Loading
Loading