Package: Peer to peer data channels

[ShishKabab]

To make initial data sync between two devices possible using storex-sync, and to enable other real-time use cases between two devices in the future, we need some mechanism to pass messages between two devices. We'd like to use Firebase Realtime Database for this purpose, and enable an optional connection through WebRTC if possible for faster data transfer. For now, we'll only accomodate the single-user, multi-device scenario. At the highest level, we need the following interfaces:

type PeerID = string | number
interface PeerInfo {
    id : PeerID
    label : string
}
interface PeerConnectionManager {
    events : EventEmitter

    getOnlinePeers() : Promise<Array<PeerInfo>>
    connectToPeer(id : PeerID, options : { confirmDataReceival : boolean }) : Promise<PeerConnection>
}
interface PeerConnection {
    events : EventEmitter
    send(data : string) : Promise<void>
    disconnect() : Promise<void>
}

PeerConnectionManager.events can emit the following events:

events.emit('incomingConnection', event : { connection : PeerConnection })
events.emit('connectionClosed', event : { reason : 'peer-requested-closed' | 'connection-error' } )

PeerConnection.events can emit the following events:

event.emit('data', event : { data : string })

As a first step, we'll need to implement the FirebasePeerConnectionManager:

• Stores presence information for the current device adapted from this code: https://firebase.google.com/docs/firestore/solutions/presence#using_presence_in_realtime_database (don't go into connecting Firebase with Firestore)
• Listens at a special path relay/negotiation/{userId}/{peerId} for changes. Here only one value is stored at a time, in the format { initiatior : PeerID, sessionId : number | string, updatedWhen : Timestamp }.
• If a device wants to connect to another one, they write a message to relay/negotiation/{userId}/{peerId} where peerId is the ID of the target device. The target picks it up, and emits PeerConnectionManager.on('incomingConnection') with the PeerConnection
• A PeerConnection sends messages by writing to the relay/session/messages/{sessionId} object, one message at a time in the following format { data : string, updatedWhen : Timestamp }. If the connection was established with confirmDataReceival, it also listens to relay/session/acknowledgements/{sessionId}, which stores a single object as { peerId : PeerID, updatedWhen : Timestamp }

As a second step, we under the hood use simple-peer to establish a direct P2P connection through WebRTC. Basically when the initial negotion is done, but before emitting the incomingConnection event, you try to use the existing PeerConnection to exchange establish a connection as described in the simple-peer docs, and wrap that connection in a PeerConnection class. This way the consuming side does not know whether under the hood WebRTC or Firebase is used.

Very important:

• Writing Firebase security rules: make sure data types are valid, and messages are of reasonable size. And of course that you check for user IDs :)
• Error handling: be paranoid, think about what might go wrong and give enough information to the consumer of this class to provide a good UX.
• Time-outs: there could be time-outs in a lot of places, and we need to signal to the consumer something seems slow, and that the connection failed/dropped

The first step can be completely developed independently from the Memex codebase in a seperate Node.js package. You can copy the @worldbrain/storex package layout to get started. I cannot stress enough to develop this entirely using a TDD approach, meaning writing tests first, then the actual code, bit by bit, in the smallest units possible. You can do this using the Firebase emulator.

As for the second step, you can use create-react-app to create a test project, and yarn link the package you're creating, even though you'll need need React. But at least it provides a working Webpack setup quickly.

Whether the package should be part of the Storex ecosystem, or not, is still unclear to me.