• Protobuf generator and GRPC
  • Description
  • Comparison
  • Install
  • Generator
    • API
    • CLI
    • Parser
    • Issues with protoc
  • GRPC
    • Setup gRPC instance
    • Sending requests
      • Simple example
      • Cancel requests
      • Mask
      • All options
      • Web Tools
    • Extraction types
    • Deep readonly
    • Default values
    • Unwrap and mapping data
  • Messages and typings
    • Primitives
    • Wrappers
    • Messages
    • Enums
    • Services
    • Oneof
    • Required and Optional
    • Deprecated
    • Defaults

Protobuf generator and GRPC

Description

Protobuf version 3 is used to work with the backend: https://developers.google.com/protocol-buffers/docs/proto3

Messages are sent in binary form and do not contain message structures.

This library was created for simple convenient work with the protocol.

It consists of 2 parts:

• generator - converts proto files into TS code
• query library - uses the result of the generator to send and receive requests

Comparison

Features	Google	Protobufjs	pbf	Current lib
Official client	+	-	-	-
Simple install	- (binary)	+ (full js)	+ (full js)	+ (full js)
Data as simple object	-	+	+	+
Compile simples proto with imports	+	+	-	+
GRPC client	+	- (poor wrapper)	- (none)	+
Small client size (kb gzip)	- (50 binary + http)	+ (6.8 binary)	+ (3.3 binary )	+ (3.3 binary + 3.0 HTTP)
Tree Shaking (lib + generated code)	-	-	-	+
Dev tools (JSON)	+	-	-	+
Dev tools (binary)	-	-	-	+
Auto wrap StringValue	-	-	-	+
Auto wrap oneof	- (manual)	- (manual)	- (none)	+
Auto wrap masks	-	-	-	+

Install

npm i @whisklabs/grpc

Generator

Before using the library, you need to convert the proto files to the required format.

API

The conversion can be done programmatically.

import { generator } from '@whisklabs/grpc/generator';

const error = await generator({
  dir: 'source/folder', // path to root of proto folder
  out: 'result/folder', // output path for generated result
  version: '№123', // optional string for version
  exclude: /some|regexp/, // optional regexp for exclude files
  debug: true, // generate json debug files,
  messageRequired: false, // enable strict required mode for messages (default: false)
  // If we need package.json, set options packageName and packageVersion
  packageName: '@whisklabs/package-one', // generate package.json with name
  packageVersion: '0.1.10', // generate package.json with version
  packageUrl: 'git@github.com:whisklabs/npm.git', // set package url
  packageRegistry: 'https://npm.pkg.github.com/', // set package registry
});

console.log(error); // true | false

CLI

The conversion can be done via the command line.

PROTO_DIR=source/folder PROTO_OUT=result/folder grpc-generator

export PROTO_DIR=source/folder
export PROTO_OUT=result/folder

npx @whisklabs/grpc

Required ENV params:

• PROTO_DIR - path to root of proto folder
• PROTO_OUT - output path for generated result
• PROTO_VERSION - string of version (default is from npm package lib)
• PROTO_EXCLUDE - optional regexp for exclude files
• PROTO_DEBUG - true | false - generate json debug files
• PROTO_MESSAGE_REQUIRED - true | false - enable strict required mode for messages (default: false)

If we need package.json, set up both options:

• PROTO_PACKAGE_NAME - generate package.json with name
• PROTO_PACKAGE_VERSION - generate package.json with version
• PROTO_PACKAGE_URL - set package url
• PROTO_PACKAGE_REGISTRY - set package registry

Parser

Return JSON structure of .proto files.

import { readFileSync } from 'fs';
import { parser } from '@whisklabs/grpc';

const parsed = parser(readFileSync('some.proto', 'utf8'));

Issues with protoc

For custom options add in start of file or in 'google/protobuf/descriptor.proto'

import "google/protobuf/descriptor.proto";

extend google.protobuf.FieldOptions {
  bool required = 1001; // uniq number more 1000
}
extend google.protobuf.MessageOptions {
  bool message_required = 1001; // uniq number more 1000
}
extend google.protobuf.FileOptions {
  bool messages_required = 1001; // uniq number more 1000
}

For optional keyword use ptotoc 3.15+

GRPC

The query library is a factory of endpoints, for simultaneous work with different API servers.

Setup gRPC instance

// Importing library
import { grpcHTTP, StatusCode } from '@whisklabs/grpc';

// Server gRPC instance
export const grpc = grpcHTTP({
  // URL to API server [Required]
  server: 'https://example.com',
  // Use or not credentials [default = true]
  credentials: true,
  // Send binary data of request/response to grpc-web devtools [default = false]
  debug: false,
  // Send JSON data of request/response to grpc-web devtools [default = false]
  devtool: false,
  // Instance of any console. Work with combine debug or devtool [default = undefined]
  logger: console,
  // Setup global timeout in number milliseconds [default = undefined]
  timeout: undefined,

  // Proxy xhr before request
  transformRequest: ({ xhr, data, meta }) => {
    xhr.setRequestHeader('Authorization', 'ANY_TOKEN');
  },

  // Or example with modify data before send
  transformRequest: async ({ data }) => {
    if (isObject(data)) {
      (data as any).token = await SomeAction();
      return data;
    }
    return { ...data, x: 1 };
  },

  // Proxy result after request
  transformResponse: ({ xhr, data, meta }) => {
    if (!data.success) {
      console.log(data.error);
    }

    // Passthrough without changes
    return data;
  },

  // Proxy result after request (async example)
  transformResponse: async ({ data }) => {
    // We can globaly check errors
    if (!data.success) {
      // Notify
      alert(`v1.Grpc.Event.GRPCError ${data.error.message}`);

      // Prevent
      if (data.error.httpStatus === 403) {
        const reason = await forbidden();
        return { success: false, error: { message: reason } };
      }

      // Redirect
      if (data.error.grpcCode === StatusCode.UNAUTHENTICATED) {
        logout();
      }
    }

    // Passthrough without changes
    return data;
  },
});

Sending requests

After configuring gRPC instance and generating code from proto files, we can make requests to the server.

Simple example

// Importing generated code from proto files
import { whisk_api_user_v2_UserAPI_GetMe } from './proto';

// Importing gRPC instance
import { grpc } from './grpc';

// You not need use catch for more flatten code without try {}
const user = await grpc(whisk_api_user_v2_UserAPI_GetMe);

console.log(user.success && user.data.user?.email);
console.log(!user.success && user.error.message);

if (user.success) {
  // If success we have typed data
  // You don't need do any extra check
  console.log(user.data.user?.email);
} else {
  // If error we have same structure with optional fields
  console.log(
    user.error.message, // string?
    user.error.data, // unknown? | Object - lib tries do safe JSON.parse
    user.error.grpcCode, // number?
    user.error.httpStatus // number?
  );
}

// Destruction example for flatten style
const { data: me, error } = await grpc(whisk_api_user_v2_UserAPI_GetMe);

console.log(me?.user?.email);
console.log(error?.message);

if (me) {
  console.log(me.user?.email);
}

if (error) {
  console.log(error.message);
}

Cancel requests

It can be of two types:

• manual
• automatic

Example of manually canceling a request.

// Importing
import { grpcCancel } from '@whisklabs/grpc';

// Cancel controller
const cancel = grpcCancel();

document.body.addEventListener('click', () => {
  cancel();
  console.log('gRPC canceled');
});

// Method call
const result = await grpc(whisk_api_user_v2_UserAPI_GetMe, { cancel });

if (result.success) {
  console.log(result.data.user?.id);
}

Example of automatic request cancellation.

Tokens (strings) are used to identify requests. They operate within a single gRPC instance (independently of each other) and depend only on their name and nothing more.

This is more convenient than the manual version, because occurs in fully automatic mode.

// Start 1 request
grpc(whisk_api_user_v2_UserAPI_GetMe, { cancel: `some-token` });
// Stopped 1 request, because cancel ID already in progress,
// and after this start 2 request
grpc(whisk_api_user_v2_UserAPI_GetMe, { cancel: `some-token` });
// Start 3 request and NOT stopped 2 request,
// because it's another cancel ID
grpc(whisk_api_user_v2_UserAPI_GetMe, { cancel: `some-token-${123}` });

Mask

Because in grpc there are no null values for collection of fields it is necessary to list all reset variables in the form of an array of lines with ways for these fields.

import { whisk_api_user_v2_UserAPI_UpdateSettings } from './proto';
import { mask, maskWrap } from '@whisklabs/grpc';

// Manual example - the most reliable, because we do not have a single standard for masks
const settings = { personalDetails: { age: 1 } };
const mask = mask(settings);

await grpc(whisk_api_user_v2_UserAPI_UpdateSettings, { settings, mask });

// Wrapper (some projects)
await grpc(
  whisk_api_user_v2_UserAPI_UpdateSettings,
  maskWrap({ settings: { personalDetails: { age: 1 } } }, 'settings')
  // maskWrap({ settings: { personalDetails: { age: 1 } } }, 'settings', 'otherName')
);

// Built-in wrapper (some projects)
await grpc(
  whisk_api_user_v2_UserAPI_UpdateSettings,
  { settings: { personalDetails: { age: 1 } } },
  { mask: { field: 'settings'}
  // or with custom mask name
  // { mask: { field: 'settings', outField: 'otherName' }
);

// Semi-automatic wrapper for current object (some projects)
await grpc(
  whisk_api_user_v2_UserAPI_UpdateSettings,
  { settings: { personalDetails: { age: 1 } } },
  { mask: true }
);

// Automatic wrapper (waiting for the standard). Future - not implemented yet!
await grpc(
  whisk_api_user_v2_UserAPI_UpdateSettings,
  { settings: { personalDetails: { age: 1 } } }
);

All options

interface Meta {
  token: string;
}

const grpc = grpcHTTP<Meta>(...);

const result = await grpc(
  whisk_api_user_v2_UserAPI_UpdateSettings, // gRPC method
  { ... }, // gRPC params
  { // Query options
    mask: { field: 'settings'}, // { field: string, outField?: string } | boolean - mask
    cancel: `some-token-${123}`, // string | grpcCancel - cancel request
    onDownload: e => console.log(e.loaded / e.total), // download progress with ProgressEvent
    onUpload: e => console.log(e.loaded / e.total), // upload progress with ProgressEvent
    timeout: 2000, // number - timeout for this request with cancel request at the end
    meta: { // Meta - data for transformRequest and transformResponse methods
      token: 'CODE',
    },
  }
);

Web Tools

This library is compatible with gRPC-Web Developer Tools.

Chrome & Firefox Browser extension to aid gRPC-Web development

Chrome extension

Firefox extension

grpcHTTP({
  // Send binary data of request/response to grpc-web devtools [default = false]
  debug: true,
  // Send JSON data of request/response to grpc-web devtools [default = false]
  devtool: true,
  ...
});

Extraction types

import { ServiceRequest, ServiceResponse } from '@whisklabs/grpc';
import { whisk_api_user_v2_UserAPI_UpdateSettings } from './proto';

// Manual types
type request = ServiceRequest<whisk_api_user_v2_UserAPI_UpdateSettings>;
type response = ServiceResponse<whisk_api_user_v2_UserAPI_UpdateSettings>;

Deep readonly

You can switch request and response to DeepReadonly.

import { GRPCDeep, grpcHTTP } from '@whisklabs/grpc';

const grpc = grpcHTTP({ ... }) as GRPCDeep;

Also there are helper types same as base types.

import { FieldGetDeep, ServiceRequestDeep, ServiceResponseDeep } from '@whisklabs/grpc';

Default values

import { Default } from '@whisklabs/grpc';
import { whisk_api_user_v2_TestItem } from './proto';

const res = Default(whisk_api_user_v2_TestItem);

// res return new default object with required fields
// {
//   id: '',
//   array: [],
//   date: { day: 0, month: 0, year: 0 },
//   mapSearch: {},
//   name: '',
//   searches: [],
// };

Unwrap and mapping data

import { unwrap } from '@whisklabs/grpc';

const userEmail = await unwrap(
  grpc(whisk_api_user_v2_UserAPI_UpdateSettings, { id: 'abc' }),
  data => data.user?.email,
  e => {
    console.log(e);
    return 1000;
  }
);

const user = await grpc(whisk_api_user_v2_UserAPI_GetMe);
const item = await unwrap(
  user,
  async success => {
    /* ... */
  },
  async error => {
    /* ... */
  }
);

// throw as usual promise
const err = e => {
  throw e;
};

try {
  const request = grpc(whisk_api_user_v2_UserAPI_GetMe);
  const email = await unwrap(request, data => data.user?.email, err);
} catch (e) {
  console.error(e);
}

Messages and typings

This lib convert proto to TS, turning a dependency tree into a flat list for tree shaking.

To ensure non-intersection, absolute paths become part of the interface name separated by "_".

Primitives

All primitives are required fields

Message	TypeScript	Convert fn
double	number	parseFloat
float	number	parseFloat
int32	number	parseInt
int64	number	parseInt
uint32	number	parseInt
uint64	number	parseInt
sint32	number	parseInt
sint64	number	parseInt
fixed32	number	parseInt
fixed64	number	parseInt
sfixed32	number	parseInt
sfixed64	number	parseInt
bool	boolean	=== true
string	string	String
bytes	Uint8Array	-

Wrappers

Wrappers are special messages for indicate optional primitive.

This messages auto wrap/unwrap by this lib.

Message	Alias	Real code	TypeScript
google.protobuf.DoubleValue	double	{ value: number }	number?
google.protobuf.FloatValue	float	{ value: number }	number?
google.protobuf.Int64Value	int64	{ value: number }	number?
google.protobuf.UInt64Value	uint64	{ value: number }	number?
google.protobuf.Int32Value	int32	{ value: number }	number?
google.protobuf.UInt32Value	uint32	{ value: number }	number?
google.protobuf.BoolValue	bool	{ value: boolean }	boolean?
google.protobuf.StringValue	string	{ value: string }	string?
google.protobuf.BytesValue	bytes	{ value: Uint8Array }	Uint8Array?

Messages

Messages are same as interface in TS

message TestItem {
  string id = 1;
  google.protobuf.StringValue description = 2;
  InnerItem item = 10
  repeated bool array = 11;
  map<string, OtherItem> map_search = 12;

  message InnerItem {
    int32 id = 1;
  }
}

message OtherItem {
  float count = 1;
}

type TestItem = {
  id: string;
  description?: string;
  item?: TestItem_InnerItem;
  array: boolean[];
  mapSearch: Record<string, OtherItem>;
};

type TestItem_InnerItem = {
  id: number;
};

type OtherItem = {
  count: number;
};

Enums

As TS enum. 0 fields name with ending _INVALID or _UNSPECIFIED removed from typings.

message TestItem {
  Type id = 1;
  Type force = 2 [ required = true ];
  Direction dir = 3;
}

enum Type {
  TYPE_UNSPECIFIED = 0; // removed
  HEIGHT = 1;
  WIDTH = 2;
}

enum Direction {
  UP = 0; // Not used in our proto, but can be in third party proto
  DOWN = 1;
}

type TestItem = {
  id?: Type;
  forced: Type;
  dir?: Direction;
};

const enum Type {
  HEIGHT = 1,
  WIDTH = 2,
}

const enum Direction {
  UP = 0,
  DOWN = 1,
}

Services

This is information about server methods, what types of messages are used for request and response.

message Request {
  int32 id = 1;
}

message Response {
  string name = 1;
}

service UserAPI {
  rpc GetMe(Request) returns (Response) {
    // can be some options
  }
}

type Request = {
  id: number;
};

type Response = {
  name: string;
};

export type UserAPI_GetMe = Service<Field<Request>, Field<Response>>;

// get back

type request = ServiceRequest<UserAPI_GetMe>;
type response = ServiceResponse<UserAPI_GetMe>;

Oneof

This is a grouping of fields where only one of them can be set or read.

// file path: whisk/api/user/v2/user.proto
message TestOneof {
  string id = 1;

  oneof device {
    EthicalPreference device_type = 11;
    string custom_device = 12;
  }
}

export type whisk_api_user_v2_TestOneof = {
  id: string;
  device?:
    | { oneof: 'deviceType'; value: whisk_api_user_v2_EthicalPreference }
    | { oneof: 'customDevice'; value: string };
};

There are two helpers for work with oneof: oneof and oneis.

const data: whisk_api_user_v2_TestOneof = {
  id: '123',
  device: {
    oneof: 'customDevice',
    value: 'abc',
  },
};

// any value
const any = oneof(data.device);
// or
const val = data.device?.value;

// oneis example
if (oneis(data.device, 'customDevice')) {
  console.log(data.device?.value);
}

// If example
if (data.device?.oneof === 'customDevice') {
  console.log(data.device?.value);
}

// Limit value variants
const a = oneof(data.device, 'customDevice') ?? oneof(data.device, 'deviceType');
console.log(a);

// Function transform
const out = oneof(data.device, 'customDevice', v => `${v}a`) ?? oneof(data.device, 'deviceType', v => v * 3);
console.log(out);

// Switch example
let res: string | number | undefined;
switch (data.device?.oneof) {
  case 'customDevice':
    res = `${data.device.value}a`;
    break;
  case 'deviceType':
    res = data.device.value * 3;
    break;
}
console.log(res);

Required and Optional

By default all primitives, repeated and map are required. Other types are optional.

You can change this behaveour using option required or keyword optional.

message TestOneof {
  string id = 1;
  string name = 2 [ required = true ];
  string item = 3 [ required = false ];

  google.protobuf.StringValue description = 10;
  google.protobuf.StringValue test = 11 [ required = true ];
  google.protobuf.StringValue result = 12 [ required = false ];
  optional string story = 13;

  whisk.api.shared.v1.Time time = 30;
  whisk.api.shared.v1.Date date = 31 [ required = true ];
  whisk.api.shared.v1.Date date_new = 32 [ (required) = true ]; // ptotoc compatable
}

export type whisk_api_user_v2_TestOneof = {
  id: string;
  name: string;
  item?: string;

  description?: string;
  test: string;
  result?: string;
  story?: string;

  time?: whisk_api_shared_v1_Time;
  date: whisk_api_shared_v1_Date;
  dateNew: whisk_api_shared_v1_Date;
};

You can switch on strict required mode for messages with optional keyword:

• All fields are required by default, expect oneof.
• To mark a field as optional, you can add the keyword optional.
• Although this keyword has no effect on binary compatibility and can be added or removed, clients need to be updated so that they understand how to handle values correctly.
• For backward compatibility, wrappers can be used because they are not binary compatible with optional.
• If you cannot change the structure in any way, but you need to force the override, then in such a case it is permissible to use [(required) = true] or [(required) = false].

1. Global

   Work on all files in final

   PROTO_MESSAGE_REQUIRED=true npx @whisklabs/grpc@1

   import { generator } from '@whisklabs/grpc/generator';
   
   const error = await generator({
     messageRequired: true, // enable strict required mode for messages (default: false)
   });

2. Local per file

   Add option in start of proto file

   syntax = "proto3";
   
   package whisk.api.user.v2;
   
   // Force required mode for messages in file
   option (messages_required) = true; // false for disable
   
   message Test {}

3. Local per message

   message Day {
     int32 num = 1;
   }
   
   message Week {
     // Force required mode in message
     option (message_required) = true; // false for disable
   
     int32 num = 1;
     Day day = 2;
   }

Example:

message Test {
  // Primitive
  string id = 1; // required
  optional string text = 2; // optional

  // Messages
  Week current_week = 11; // required
  optional Week next_week = 12; // optional
  whisk.api.shared.v1.Time time = 13; // required
  optional whisk.api.shared.v1.Time time_after = 14; // optional

  // Wrappers (legacy)
  google.protobuf.StringValue description = 21; // optional

  // Force override (backward binary compatibility only)
  string item = 31 [ (required) = false ]; // optional
  google.protobuf.StringValue test = 32 [ (required) = true ]; // required

  // Repeated - can't work with optional!
  repeated bool array = 41; // required
  repeated bool array_2 = 42 [ (required) = false ]; // optional

  // Map - can't work with optional!
  map<string, bool> map_search = 51; // required
  map<string, bool> map_search_2 = 52 [ (required) = false ]; // optional

  // Oneof - can't work with optional!
  oneof device_description {
    DeviceType device_type = 61; // required
    DeviceType custom_device = 62 [ (required) = false ]; // optional
  }
}

message Week {
  int32 num = 1;
}

Result:

type Test = {
  id: string;
  text?: string;
  currentWeek: Week;
  nextWeek?: Week;
  time: whisk_api_shared_v1_Time;
  timeAfter?: whisk_api_shared_v1_Time;
  description: string;
  item?: string;
  test: string;
  array: boolean[];
  array_2?: boolean[];
  mapSearch: Record<string, boolean>;
  mapSearch_2?: Record<string, boolean>;
  deviceDescription?:
    | { oneof: 'deviceType'; value: Device_DeviceType }
    | { oneof: 'customDevice'; value?: Device_DeviceType };
};

Deprecated

For obsolete fields, you can mark them as deprecated.

message TestOneof {
  string id = 1 [ deprecated = true ];
  google.protobuf.StringValue description = 10 [ required = true, deprecated = true ];
}

export type whisk_api_user_v2_TestOneof = {
  /**  @deprecated true */
  id: string;
  /**  @deprecated true */
  description: string;
};

Defaults

If field is required, bit not present in message, it is installed by table:

Message	Default
double	0
float	0
int32	0
int64	0
uint32	0
uint64	0
sint32	0
sint64	0
fixed32	0
fixed64	0
sfixed32	0
sfixed64	0
enum	0
bool	false
string	''
bytes	Uint8Array