Archivum

Archivum is a full-stack web app focused on building a data warehouse for the site Twitch.tv. When configured it will track the chats of given channels in real time and store all messages, donations, livestreams, bans and more to a central database and provide an API and website for accessing this data.

Setup

First you need to start by generating Twitch access and client tokens with the right permissions. I suggest using swiftyspiffy's website for this.

As an individual the only permission you'll need is chat:read. If you're the owner of a channel, or have moderator permissions, bits:read and channel:read:subscriptions would be recommended for accuracy.

The Database

Currently the only supported database back-end is Mysql. So you'll need to ensure there's a running instance of MySql for the app(s) to access.

This can be running on the hardware, through Docker, in a Kubernetes cluster, or any other way you might configure the app to access it.

Creating the Config

Once you have your access and client token, you'll want to create the config file. The default path for config files is ./config_files/config.yml. This path can be configured with the CONFIG_PATH environment variable.

Inside this file you'll want to setup your values. It'll look something like this:

twitchNickname: YourLoginNameHere
accessToken: YourAccessTokenHere
clientId: YourClientIdHere
databaseUsername: YourMySqlDatabaseUsernameHere
sqlUserPassword: YourSqlUserPasswordHere
databaseHostAddress: localhost:3306 # This is the default value.
channels: ["ChannelOneNameHere", "ChannelTwoNameHere"] # Up to 100 are allowed.

# Optional
logLevel: Info # Set your actual desired level.
database: twitch_tracker_db # This is the default name.
pastebinApiKey: YourPastebinApiKeyHere # Required to use the report generator app.

Most values (including secrets) can use the environment to define them instead. The available list of environment variables is as such:

TWITCH_ACCESS_TOKEN, TWITCH_CLIENT_ID, DATABASE_USERNAME, DATABASE_HOST_ADDRESS, DATABASE_PASSWORD, and PASTEBIN_API_KEY

Running

Once you've setup the config and MySql, you can run the tracker in one of three ways.

• Binary
• Dockerfile
• Kubernetes

Binary

Running the binary is as simple as building the target like so:

cargo run --release -p twitch_chat_tracker

And grabbing/running the binary from ./target/release/twitch_chat_tracker If you setup your config properly it should connect to Twitch's servers and start writting any incoming data to the database.

Dockerfile

The docker image for the tracker can be built with the following command:

docker build --progress=plain -t twitch-chat-logger:latest -f twitch_chat_tracker/Dockerfile .;

With any optional environment variables passed in as -e VARIABLE="value". Then run:

docker run --rm twitch-chat-logging:latest

To actually run the image.

If you setup your config properly it should connect to Twitch's servers and start writting any incoming data to the database.

Kubernetes

There's already an available deployment file under twitch_chat_tracker/kube that has most things already preconfigured. You just have to put in your own desired values.

Changing:

• spec.template.spec.containers.image to pull your correct Docker image.
• CONFIG_PATH under spec.template.spec.env
• DATABASE_HOST_ADDRESS under spec.template.spec.env and;
• TRACKED_CHANNELS under spec.template.spec.env.

I suggest also creating a secret in your cluster for this. The file should look something like so:

apiVersion: v1
kind: Secret
metadata:
  name: twitch-chat-logger
type: Opaque
stringData:
  twitchAccessToken: "YourTwitchAccessTokenHere"
  twitchClientId: "YourTwitchClientTokenHere"
  pastebinApiKey: "YourPastebinApiKeyHere"

This will plug in to the expected values in the existing deployment file.

If you setup your config and cluster properly it should connect to Twitch's servers and start writting any incoming data to the database.