Logging: How to change BundleOptions

[ofrobots]

From @FalconerTC on November 9, 2017 22:28

I hit a quota limit for Stackdriver ingestion requests recently, which is 1000 / second. I found this happened because I have a distributed service that was not bundling well because the GoogleCloud logging config sends logs every 50ms, per source here. This seems like an odd value to me, considering it only allows a maximum of 50 servers to be logging per project. I see this is defined as the GAX setting BundleOptions but the only GAX options that can be configured are CallOptions, per the @google-cloud/logging documentation.

Is there a way to change any of these options for the Bunyan logger? If not, do you have other recommendations to better batch logging requests in highly-distributed setups? Thanks!

Copied from original issue: googleapis/nodejs-logging-bunyan#13

[ofrobots]

/cc @lukesneeringer @stephenplusplus

[ofrobots]

From @stephenplusplus on November 10, 2017 20:56

@lukesneeringer is there anyone we can ask from the google-gax repo? @landrito?

It appears the only option you can pass through on the request is: isBundling, which can only disable the previously hard-coded setting. I'm not sure that there's a way to override it at runtime.

Additionally/alternatively, maybe we should be using a better default value in the hardcoded file?

[ofrobots]

From @FalconerTC on November 29, 2017 17:50

Any updates or ideas on this?

[ofrobots]

From @JustinBeckwith on November 29, 2017 19:25

Calling @alexander-fenster :)

[ofrobots]

From @alexander-fenster on November 29, 2017 20:37

@landrito to comment :)

[ofrobots]

From @landrito on November 29, 2017 22:8

Hm. I filed an issue in gax-nodejs to see if dynamically setting the bundling at call time makes sense. I have a hunch that introducing the ability to override it at call time is going to introduce a lot of complexity (what do we do for calls made prior to dynamically changing the bundling, what to do for calls with different

@stephenplusplus do you think dynamically setting this at client instantiation would work to resolve this?

[ofrobots]

From @stephenplusplus on December 11, 2017 18:39

Those concerns make sense to me. I guess we wouldn't want to open up the setting at the method level, but during instantiation of the client. If there's anything else gax uses similar to BundlingConfig, we would want to make sure those could be passed through as well.

[ofrobots]

From @landrito on December 11, 2017 19:29

So overriding the bundle options should work at client initialization like so:

const Logging = require('@google-cloud/logging');

const clientConfigOverride = {
  "interfaces": {
    "google.logging.v2.LoggingServiceV2": {
      "methods": {
        "WriteLogEntries": {
          "bundling": {
            "element_count_threshold": {Your},
            "request_byte_threshold": {Overrides},
            "delay_threshold_millis": {Here}
          }
        }
      }
    }
  }
};

const logging = new Logging({
  clientConfig: clientConfigOverrde
});

Looking at how the logging-bunyan client is implemented, it passes the constructor opts to the logging client directly. So passing the clientConfig to the constructor of the logging-bunyan client should yield overridden bundling options.

[ofrobots]

From @stephenplusplus on December 11, 2017 19:39

Oh, that looks like a good enough solution to me. @FalconerTC what do you think?

[ofrobots]

From @FalconerTC on December 12, 2017 16:14

This is perfect. Exactly what I am looking for. Thanks for the help.

[ofrobots]

From @stephenplusplus on December 12, 2017 17:54

Great. I think what we need to do now is add a mention to the docs that this type of configuration is possible. @landrito is there something on the gax docs we can link to?

[ofrobots]

From @DominicKramer on June 15, 2018 17:46

@landrito Friendly ping.

[ofrobots]

@DominicKramer I would suggest moving the issue (using the issue mover) to the gax-repo as the outstanding task is to update the docs.