Skip to content

Extract src/websockets from Polykey to js-ws #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 149 commits into from
Sep 25, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
149 commits
Select commit Hold shift + click to select a range
e525dbc
chore: copied over websocket domain from Polykey/websocket
amydevs Aug 17, 2023
8e16eb1
feat: generic WebSocketClient
amydevs Aug 17, 2023
21ae525
feat: WebSocketServer and WebSocketConnection
amydevs Aug 23, 2023
6d9a7b2
feat: tests now use generated certs for TLS
amydevs Aug 23, 2023
867ec6d
feat: streamIds are now bigints + added conversion utility functions …
amydevs Aug 23, 2023
6365700
fix: stream id to/from parser
amydevs Aug 24, 2023
0a76685
fix: toStreamId properly copies buffer
amydevs Aug 24, 2023
43a4498
feat: WebSocketStream WritableStream
amydevs Aug 24, 2023
dcd3c21
lintfix
amydevs Aug 25, 2023
90e1360
feat: WebSocketStream chunking
amydevs Aug 25, 2023
7fea3e6
feat: WebSocketClient connections are now managed by WebSocketConnect…
amydevs Aug 25, 2023
0404cda
fix: WebSocketStream error handling
amydevs Aug 25, 2023
7877c6a
feat: WebSocketConnection is now more generic
amydevs Aug 25, 2023
8a80720
feat: created WebSocketConnectionMap
amydevs Aug 25, 2023
76be5f1
feat: WebSocketClient connection configuration
amydevs Aug 28, 2023
ed4d444
feat: WebSocketStream buffering and tests
amydevs Aug 28, 2023
9108f2c
feat: WebSocketStream error handling
amydevs Aug 30, 2023
0b9bb93
feat: WebSocketConnection stopping with error codes
amydevs Aug 30, 2023
4bb95e3
feat: mocking for WebSocketStream tests
amydevs Aug 30, 2023
22dd260
feat: WebSocketStream bidi closing
amydevs Aug 30, 2023
a0b4f46
feat: added ability to specify codeToReason and reasonToCode from Web…
amydevs Aug 30, 2023
513856e
lintfix
amydevs Aug 30, 2023
24cdfa9
chore: deleted old WebSocketStream
amydevs Aug 30, 2023
a09fde1
fix: https/ws server errors are propagated onto WebSocketServer
amydevs Aug 30, 2023
52d2a06
feat: errors on WebSocket are now bubbled up
amydevs Aug 30, 2023
8a58602
fix: errors on parsing WebSocketStream messages are now bubbled up
amydevs Aug 30, 2023
633f6e5
fix: buffering writableDesiredSize was going into negative
amydevs Aug 30, 2023
9052a47
fix: WebSocketConnectionMap is now using correct `resource-counter` d…
amydevs Aug 31, 2023
d7610aa
feat: added specifying force stop/destroy to connections and streams
amydevs Aug 31, 2023
ac391db
feat: aligned WebSocketClient to use WebSocketConnection
amydevs Aug 31, 2023
c1eed24
fix: writableDesiredSize should be decremented before stream messages…
amydevs Aug 31, 2023
73f7ae6
fix: bubble up WebSocketStreamDestroyEvents on the server
amydevs Aug 31, 2023
a84fa4c
fix: close and error frames will be ignored on closed streams
amydevs Aug 31, 2023
bce0e7d
fix: readable controller properly closes on WebSocketStream
amydevs Sep 1, 2023
59d7e5c
feat: WebSocketStream tests for varying sized writes
amydevs Sep 1, 2023
051b2cd
lintfix
amydevs Sep 1, 2023
a7e41f2
chore: upgraded to node 20
amydevs Sep 1, 2023
e669551
fix: `single write within buffer size` now uses correct buffer size
amydevs Sep 1, 2023
29dbd2d
feat: migrated events to use js-events system
amydevs Sep 1, 2023
f4d399f
feat: simultaneous writing tests for WebSocketStream
amydevs Sep 1, 2023
8b46495
chore: renamed `StreamType` to `StreamMessageType`
amydevs Sep 1, 2023
baf00ca
fix: renamed ErrorWebSocketStream errors
amydevs Sep 1, 2023
44ddbd6
fix: made a VarInt type that SteamId is
amydevs Sep 1, 2023
3740a78
chore: upgraded js-async-init to 1.9.1 for implicit lifecycle events
amydevs Sep 4, 2023
9addf86
lintfix
amydevs Sep 4, 2023
e447669
fix: WebSocketStream lifecycle methods are now all awaited
amydevs Sep 4, 2023
3cc29fb
feat: WebSocketStream lifecycle tests
amydevs Sep 4, 2023
5a9a641
feat: moved parsing/generation of stream/connection messages to separ…
amydevs Sep 4, 2023
d65262a
lintfix
amydevs Sep 4, 2023
1c1eeaa
fix: exported every thing at root on index.ts
amydevs Sep 4, 2023
b6c4b7f
feat: WebSocketStreamQueue
amydevs Sep 6, 2023
8c1ddfc
feat: changed backpressure to work additively rather than by setting …
amydevs Sep 6, 2023
34fbee3
fix: WebSocketStream shutdowns WebSocketStreamQueue
amydevs Sep 6, 2023
188f4ab
fix: changed 'delete's in WebSocketStreamQueue to set to undefined fo…
amydevs Sep 6, 2023
9476888
fix: WebSocketStream WritableStream write is now looped rather than r…
amydevs Sep 6, 2023
8e66969
fix: readable queue is now cleared earlier on closing ReadableStream
amydevs Sep 6, 2023
ba2c3af
chore: WebSocketStreamQueue documentation
amydevs Sep 6, 2023
0b28258
fix: WebSocketStreamQueue dequeue tests to include byteLengths
amydevs Sep 6, 2023
425a58a
chore: execute WebSocketStream test write/read functions more closely
amydevs Sep 6, 2023
c34755f
fix: readableController is automatically closed off of error in signa…
amydevs Sep 6, 2023
e2b4646
feat: error propagation test for WebSocketStream
amydevs Sep 7, 2023
7367eeb
fix: closing ReadableStream should correctly close the opposing Writa…
amydevs Sep 7, 2023
ec2218f
feat: stream cancelling tests
amydevs Sep 7, 2023
8592b0c
lintfix
amydevs Sep 7, 2023
2e7716f
fix: propagate logger on WebSocketClient
amydevs Sep 7, 2023
dfd3784
feat: benchmarks for baseline + streams
amydevs Sep 7, 2023
e63478d
fix: changed to KiB naming for benches
amydevs Sep 7, 2023
6f07141
feat: benchmark for baseline tcp socket
amydevs Sep 7, 2023
f09e7d0
fix: stream benchmarks now no longer create a new stream for each ite…
amydevs Sep 7, 2023
44bb2f2
lintfix
amydevs Sep 7, 2023
830ddb9
feat: connection benchmarks
amydevs Sep 7, 2023
58d8348
feat: streamIds are encoded on stream creation rather than stream send
amydevs Sep 7, 2023
29cbe3f
fix: updated connection benchmark to follow changes to streamIds
amydevs Sep 7, 2023
7100d95
feat: minimized array concats
amydevs Sep 7, 2023
ab66880
chore: benchmarks for parser/generator improvements
amydevs Sep 7, 2023
58a50cc
lintfix
amydevs Sep 8, 2023
45bbbbf
fix: connection benchmarks were silently failing
amydevs Sep 8, 2023
aa1a0ed
feat: unknown errors for WebSocketStream
amydevs Sep 8, 2023
c142667
fix: connection benchmark removed type wrangling
amydevs Sep 8, 2023
3e9efea
fix: fixed recursive stopping on streams when a connection is stopped
amydevs Sep 8, 2023
c5df0b3
feat: WebSocket tests
amydevs Sep 8, 2023
5a9a4c9
fix: removed utils.test.ts
amydevs Sep 8, 2023
ca680a6
fix: properly stop server and connection if they aren't already stopping
amydevs Sep 8, 2023
38f8e9d
fix: WebSocketConnectionMap counter is protected
amydevs Sep 8, 2023
37ddaae
feat: WebSocketStream metadata
amydevs Sep 8, 2023
06b702b
lintfix
amydevs Sep 8, 2023
73a1972
fix: correct metadata for WebSocketClient
amydevs Sep 8, 2023
f59a714
chore: added README.md
amydevs Sep 8, 2023
5958805
chore: added .gitlab-ci.yml
amydevs Sep 8, 2023
6f0285c
fix: streams created by peers must initiate with an Ack message
amydevs Sep 8, 2023
dda1e9e
fix: made StreamErrorCode an object constant rather than an enum
amydevs Sep 11, 2023
05ce328
fix: config changes to make keepalive and connection start timeouts i…
amydevs Sep 11, 2023
98f4f88
fix: made class comment on WebStreamQueue a docblock
amydevs Sep 11, 2023
4b06dd1
fix: more strict constraints on stopping server from https server errors
amydevs Sep 11, 2023
4a161c2
feat: WebSocketConnection errors with stricter typing
amydevs Sep 11, 2023
5a0b58e
feat: reworked WebSocketConnection events
amydevs Sep 13, 2023
7b1d477
fix: WebSocketConnection.stop now correctly emits close event
amydevs Sep 13, 2023
e2d9308
feat: WebSocketConnection close event test
amydevs Sep 13, 2023
9c2e3fc
lintfix
amydevs Sep 13, 2023
709de3f
feat: StreamMap is now managed by WebSocketConnection
amydevs Sep 14, 2023
ce0b921
feat: WebSocketServer and WebSocketClient lifecycle is now managed by…
amydevs Sep 15, 2023
4f5c4a0
lintfix
amydevs Sep 15, 2023
c378621
feat: WebSocketStream lifecycle is now managed by events
amydevs Sep 15, 2023
1cb4a71
feat: WebSocketStream is now StartStop
amydevs Sep 15, 2023
63e404d
fix: added back WebSocketStream WritablePull
amydevs Sep 15, 2023
53a24dc
chore: benchmarks after js-events integration
amydevs Sep 15, 2023
5ffcf4d
fix: WebSocketStream single write within buffer test
amydevs Sep 15, 2023
37083bb
fix: WebSocketStream error tests
amydevs Sep 15, 2023
3f114c2
feat: use of Buffer.allocUnsafe wherever possible
amydevs Sep 18, 2023
da460da
chore: update benchmarks for buffer.allocUnsafe changes
amydevs Sep 18, 2023
f88efae
fix: WebSocketStream is now using .push for sends instead of .unshift
amydevs Sep 18, 2023
6c5f4b4
fix: promise refactoring for WebSocketStream so that pull/write promi…
amydevs Sep 18, 2023
9e16ca5
fix: naming of tcp baseline benchmarks
amydevs Sep 18, 2023
c36d7e1
feat: verifyPeer on both server and client
amydevs Sep 18, 2023
784ffb1
feat: WebSocketStream StreamMessageType is now represented by 16 bits
amydevs Sep 18, 2023
ef12a13
fix: tests with unneeded writablestreams closed
amydevs Sep 18, 2023
f1a8a6c
fix: updated bench tls config
amydevs Sep 18, 2023
7e6e066
feat: server tests
amydevs Sep 18, 2023
102c8fa
feat: TLS client/server verification tests
amydevs Sep 18, 2023
ce78131
feat: stream closing tests
amydevs Sep 18, 2023
a4762bb
lintfix
amydevs Sep 18, 2023
831e04a
chore: bump js-errors and js-events
amydevs Sep 20, 2023
6f2c47e
fix: removed refernces to unidirectional streams in WebSocketConnection
amydevs Sep 20, 2023
0433875
fix: streamIds now increases as a lesser rate due to bidi only
amydevs Sep 20, 2023
64ce6d5
fix: WebSocketConnection error handling
amydevs Sep 20, 2023
9543af1
lintfix
amydevs Sep 20, 2023
27626e8
feature: added ability to inject https server
amydevs Sep 20, 2023
3ec1632
fix: WebSocketConnection start error handling
amydevs Sep 20, 2023
24d445e
feat: WebSocketConnection Error -> Close Event Flow
amydevs Sep 21, 2023
a4481e6
feat: WebSocketClient now redispatches Connection errors as Client er…
amydevs Sep 21, 2023
ca53ae4
feat: WebSocketStream Error -> Close Event Flow
amydevs Sep 21, 2023
410304a
lintfix
amydevs Sep 21, 2023
60af7de
chore: update inline documentation
amydevs Sep 21, 2023
ea664d9
fix: connectTimeoutTime is now injected through ctx of client
amydevs Sep 21, 2023
7b343d4
feat: WebSocketClient tests
amydevs Sep 21, 2023
acdc8a6
chore: bump js-async-init to 1.10.0
amydevs Sep 22, 2023
f918999
fix: custom verifyCallback is now supplied arrays of DERs instead
amydevs Sep 22, 2023
3d71693
feat: custom verifyCallback TLS Verification tests
amydevs Sep 22, 2023
8c2992b
feat: added path to start options
amydevs Sep 22, 2023
9a6bb58
lintfix
amydevs Sep 22, 2023
d7c834e
feat: ErrorWebSocketConnectionTLS
amydevs Sep 23, 2023
7920417
lintfix
amydevs Sep 23, 2023
2e4bca3
chore: added better inline documentation
amydevs Sep 23, 2023
9489160
feat: hostnames are now resolved for server and client
amydevs Sep 23, 2023
34d3ed8
fix: moved related tests from WebSocket.test.ts to WebSocketServer.te…
amydevs Sep 25, 2023
18e3844
lintfix
amydevs Sep 25, 2023
94bfef8
fix: WebSocketConnection timeout executes properly now
amydevs Sep 25, 2023
d7006d9
chore: WebSocketConnection keepalive tests
amydevs Sep 25, 2023
518e744
fix: typing for WebSocketServer so that a provided `opts.server` will…
amydevs Sep 25, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
chore: renamed StreamType to StreamMessageType
  • Loading branch information
amydevs committed Sep 21, 2023
commit 8b4649586c4171599a23bd12f3ac235d373e262c
6 changes: 3 additions & 3 deletions src/WebSocketConnection.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import { ready } from '@matrixai/async-init/dist/CreateDestroyStartStop';
import { Evented } from '@matrixai/events';
import WebSocketStream from './WebSocketStream';
import * as errors from './errors';
import { fromStreamId, promise, StreamType, toStreamId } from './utils';
import { fromStreamId, promise, StreamMessageType, toStreamId } from './utils';
import * as events from './events';

const timerCleanupReasonSymbol = Symbol('timerCleanupReasonSymbol');
Expand Down Expand Up @@ -163,8 +163,8 @@ class WebSocketConnection extends EventTarget {
if (stream == null) {
const messageType = message.at(0);
if (
messageType === StreamType.CLOSE ||
messageType === StreamType.ERROR
messageType === StreamMessageType.CLOSE ||
messageType === StreamMessageType.ERROR
) {
return;
}
Expand Down
3 changes: 0 additions & 3 deletions src/WebSocketServer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,6 @@ import WebSocketConnectionMap from './WebSocketConnectionMap';
* Otherwise, errors will just be ignored.
*
* Events:
* - serverStop
* - serverError
* - serverConnection
* - connectionStream - when new stream is created from a connection
* - connectionError - connection error event
* - connectionDestroy - when connection is destroyed
Expand Down
42 changes: 21 additions & 21 deletions src/WebSocketStream.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import {
fromVarInt,
never,
promise,
StreamType,
StreamMessageType,
StreamShutdown,
toVarInt,
} from './utils';
Expand Down Expand Up @@ -104,7 +104,7 @@ class WebSocketStream implements ReadableWritablePair<Uint8Array, Uint8Array> {
return;
}
// Send ACK on every read as there will be more usable space on the buffer.
await this.streamSend(StreamType.ACK, controller.desiredSize!);
await this.streamSend(StreamMessageType.ACK, controller.desiredSize!);
},
cancel: async (reason) => {
this.logger.debug(`readable aborted with [${reason.message}]`);
Expand Down Expand Up @@ -146,7 +146,7 @@ class WebSocketStream implements ReadableWritablePair<Uint8Array, Uint8Array> {
}
// Decrement the desired size by the amount of bytes written
this.writableDesiredSize -= bytesWritten;
await this.streamSend(StreamType.DATA, data);
await this.streamSend(StreamMessageType.DATA, data);

if (isChunkable) {
await writeHandler(chunk.subarray(bytesWritten), controller);
Expand Down Expand Up @@ -207,7 +207,7 @@ class WebSocketStream implements ReadableWritablePair<Uint8Array, Uint8Array> {
* @param payloadSize - The number of bytes that the receiver can accept.
*/
protected async streamSend(
type: StreamType.ACK,
type: StreamMessageType.ACK,
payloadSize: number,
): Promise<void>;
/**
Expand All @@ -216,7 +216,7 @@ class WebSocketStream implements ReadableWritablePair<Uint8Array, Uint8Array> {
* @param data - The payload to send.
*/
protected async streamSend(
type: StreamType.DATA,
type: StreamMessageType.DATA,
data: Uint8Array,
): Promise<void>;
/**
Expand All @@ -225,7 +225,7 @@ class WebSocketStream implements ReadableWritablePair<Uint8Array, Uint8Array> {
* @param shutdown - Signifies whether the ReadableStream or the WritableStream has been shutdown.
*/
protected async streamSend(
type: StreamType.ERROR,
type: StreamMessageType.ERROR,
shutdown: StreamShutdown,
code: bigint,
): Promise<void>;
Expand All @@ -235,28 +235,28 @@ class WebSocketStream implements ReadableWritablePair<Uint8Array, Uint8Array> {
* @param shutdown - Signifies whether the ReadableStream or the WritableStream has been shutdown.
*/
protected async streamSend(
type: StreamType.CLOSE,
type: StreamMessageType.CLOSE,
shutdown: StreamShutdown,
): Promise<void>;
protected async streamSend(
type: StreamType,
type: StreamMessageType,
data_?: Uint8Array | number,
code?: bigint,
): Promise<void> {
let data: Uint8Array | undefined;
if (type === StreamType.ACK && typeof data_ === 'number') {
if (type === StreamMessageType.ACK && typeof data_ === 'number') {
data = new Uint8Array(4);
const dv = new DataView(data.buffer);
dv.setUint32(0, data_, false);
} else if (type === StreamType.DATA) {
} else if (type === StreamMessageType.DATA) {
data = data_ as Uint8Array;
} else if (type === StreamType.ERROR) {
} else if (type === StreamMessageType.ERROR) {
const errorCode = fromVarInt(code!);
data = new Uint8Array(1 + errorCode.length);
const dv = new DataView(data.buffer);
dv.setUint8(0, data_ as StreamShutdown);
data.set(errorCode, 1);
} else if (type === StreamType.CLOSE) {
} else if (type === StreamMessageType.CLOSE) {
data = new Uint8Array([data_ as StreamShutdown]);
} else {
never();
Expand Down Expand Up @@ -286,10 +286,10 @@ class WebSocketStream implements ReadableWritablePair<Uint8Array, Uint8Array> {
}),
);
}
const type = message[0] as StreamType;
const type = message[0] as StreamMessageType;
const data = message.subarray(1);
const dv = new DataView(data.buffer, data.byteOffset, data.byteLength);
if (type === StreamType.ACK) {
if (type === StreamMessageType.ACK) {
try {
const bufferSize = dv.getUint32(0, false);
this.writableDesiredSize = bufferSize;
Expand All @@ -309,7 +309,7 @@ class WebSocketStream implements ReadableWritablePair<Uint8Array, Uint8Array> {
),
);
}
} else if (type === StreamType.DATA) {
} else if (type === StreamMessageType.DATA) {
if (this._readableEnded) {
return;
}
Expand All @@ -324,12 +324,12 @@ class WebSocketStream implements ReadableWritablePair<Uint8Array, Uint8Array> {
return;
}
this.readableController.enqueue(data);
} else if (type === StreamType.ERROR || type === StreamType.CLOSE) {
} else if (type === StreamMessageType.ERROR || type === StreamMessageType.CLOSE) {
try {
const shutdown = dv.getUint8(0) as StreamShutdown;
let isError = false;
let reason: any;
if (type === StreamType.ERROR) {
if (type === StreamMessageType.ERROR) {
isError = true;
const errorCode = toVarInt(data.subarray(1)).data;
reason = await this.codeToReason('recv', errorCode);
Expand Down Expand Up @@ -398,10 +398,10 @@ class WebSocketStream implements ReadableWritablePair<Uint8Array, Uint8Array> {
// Shutdown the write side of the other stream
if (isError) {
const code = await this.reasonToCode('send', reason);
await this.streamSend(StreamType.ERROR, StreamShutdown.Write, code);
await this.streamSend(StreamMessageType.ERROR, StreamShutdown.Write, code);
this.readableController.error(reason);
} else {
await this.streamSend(StreamType.CLOSE, StreamShutdown.Write);
await this.streamSend(StreamMessageType.CLOSE, StreamShutdown.Write);
}
if (this._readableEnded && this._writableEnded) {
this.destroyProm.resolveP();
Expand All @@ -428,10 +428,10 @@ class WebSocketStream implements ReadableWritablePair<Uint8Array, Uint8Array> {
// Shutdown the read side of the other stream
if (isError) {
const code = await this.reasonToCode('send', reason);
await this.streamSend(StreamType.ERROR, StreamShutdown.Read, code);
await this.streamSend(StreamMessageType.ERROR, StreamShutdown.Read, code);
this.writableController.error(reason);
} else {
await this.streamSend(StreamType.CLOSE, StreamShutdown.Read);
await this.streamSend(StreamMessageType.CLOSE, StreamShutdown.Read);
}
if (this._readableEnded && this._writableEnded) {
this.destroyProm.resolveP();
Expand Down
4 changes: 2 additions & 2 deletions src/utils/utils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ function fromVarInt(varInt: bigint): Uint8Array {
const fromStreamId = fromVarInt as (streamId: StreamId) => Uint8Array;
const toStreamId = toVarInt as (array: Uint8Array) => Parsed<StreamId>;

enum StreamType {
enum StreamMessageType {
DATA = 0,
ACK = 1,
ERROR = 2,
Expand All @@ -117,6 +117,6 @@ export {
fromVarInt,
toStreamId,
fromStreamId,
StreamType,
StreamMessageType,
StreamShutdown,
};
6 changes: 3 additions & 3 deletions tests/WebSocketStream.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import { fc, testProp } from '@fast-check/jest';
import WebSocketStream from '@/WebSocketStream';
import WebSocketConnection from '@/WebSocketConnection';
import * as events from '@/events';
import { promise, StreamType } from '@/utils';
import { promise, StreamMessageType } from '@/utils';
import * as config from '@/config';
import * as testUtils from './utils';

Expand Down Expand Up @@ -41,8 +41,8 @@ jest.mock('@/WebSocketConnection', () => {
let stream = instance.connectedConnection!.streamMap.get(streamId);
if (stream == null) {
if (
data.at(0) === StreamType.CLOSE ||
data.at(0) === StreamType.ERROR
data.at(0) === StreamMessageType.CLOSE ||
data.at(0) === StreamMessageType.ERROR
) {
return;
}
Expand Down