Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

upgrade package metrics #758

Merged
merged 74 commits into from
Dec 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
23bc924
internal/debug: start pprof server by new function StartPProf (#16532)
gzliudan Dec 13, 2024
92a7b64
metrics: export to InfluxDB (#16979)
gzliudan Dec 13, 2024
61f5a88
metrics: add force option for metric system (#17667)
gzliudan Dec 13, 2024
01a5298
metrics: added NewCounterForced (#17919)
gzliudan Dec 13, 2024
4202f23
core: more detailed metrics for block processing (#18119)
gzliudan Dec 13, 2024
3e4583e
cmd/utils: allow for multiple influxdb flags (#18520)
gzliudan Dec 13, 2024
7608787
metrics: remove redundant type specifiers (#19090)
gzliudan Dec 13, 2024
1557746
metrics/influxdb: add a timeout to the InfluxDB HTTP client (#19250)
gzliudan Dec 13, 2024
db9487f
core: split out detailed trie access metrics from insertion time (#19…
gzliudan Dec 13, 2024
ed427a9
metrics: fix expensive metrics flag processing (#19327)
gzliudan Dec 13, 2024
a577c71
metrics/prometheus: added prometheus metrics (#17077)
gzliudan Dec 13, 2024
1eb2ed8
core, metrics, p2p: expose various counter metrics for grafana (#19692)
gzliudan Dec 13, 2024
a4e113c
metrics: gather and export threads and goroutines (#19725)
gzliudan Dec 13, 2024
7456407
metrics: change links in README.md to https (#20182)
gzliudan Dec 13, 2024
332ac32
metrics: not compare float numbers directly (#20219)
gzliudan Dec 13, 2024
462999b
metrics: fix issues reported by staticcheck (#20365)
gzliudan Dec 13, 2024
9fee8a7
metrics: disable CPU stats (gosigar) on iOS (#20816)
gzliudan Dec 13, 2024
47ce406
metrics: make flawed test less flawed (#20818)
gzliudan Dec 13, 2024
1415bb6
metrics: add missing calls to Ticker.Stop in tests (#20866)
gzliudan Dec 13, 2024
c65d0cd
cmd/XDC: enable metrics for geth import command (#20738)
gzliudan Dec 13, 2024
32f974c
metrics/prometheus: define TYPE once, add tests (#21068)
gzliudan Dec 13, 2024
d7d54b0
metrics: replace gosigar with gopsutil (#21041)
gzliudan Dec 13, 2024
3dee667
metrics/exp: allow configuring metrics HTTP server on separate endpoi…
gzliudan Dec 13, 2024
730960f
metrics: make meter updates lock-free (#21446)
gzliudan Dec 13, 2024
bf4b42a
metrics: zero temp variable in updateMeter (#21470)
gzliudan Dec 13, 2024
d4f1b8a
metrics: fix the panic for reading empty cpu stats (#21864)
gzliudan Dec 13, 2024
9d082aa
cmd/XDC: dump config for metrics (#22083)
gzliudan Dec 13, 2024
1a844e4
metrics: remove uneeded syntax (#21921)
gzliudan Dec 13, 2024
ec0ae49
metrics: fix cast omission in cpu_syscall.go (#22262)
gzliudan Dec 13, 2024
5e9cb5d
metrics: add handler performance metrics (#22581)
gzliudan Dec 13, 2024
ebbcd60
metrics: use resetting histograms for rare packets (#22586)
gzliudan Dec 13, 2024
d16c72e
metrics/influxdb: don't push empty histograms, no measurement != 0 (#…
gzliudan Dec 13, 2024
73b81dd
metrics: use golang.org/x/sys/unix to support Solaris (#22584)
gzliudan Dec 13, 2024
29b72db
metrics/influxdb: support V2 (#23194)
gzliudan Dec 13, 2024
9807910
metrics: fix compilation for GOOS=js (#23449)
gzliudan Dec 13, 2024
2e5b342
metrics: add go:build lines (#23468)
gzliudan Dec 13, 2024
d77c1e5
metrics: replace strings.Replace with string.ReplaceAll (#24835)
gzliudan Dec 13, 2024
7b0a7e4
metrics/influxdb: temp solution to present counter meaningfully (#24811)
gzliudan Dec 13, 2024
2e34afe
rpc: swap out timer metrics to histograms (#25044)
gzliudan Dec 13, 2024
392b44c
metrics/influxdb: replace time.Tick with time.NewTicker (#24783)
gzliudan Dec 13, 2024
e8452c9
metrics: fix some typos (#25551)
gzliudan Dec 13, 2024
dddd6c5
metrics: improve reading Go runtime metrics (#25886)
gzliudan Dec 13, 2024
40f47a6
metrics/influxdb: fix time ticker leaks (#26507)
gzliudan Dec 13, 2024
76320b4
metrics/librato: use http package to replace http method names (#26535)
gzliudan Dec 13, 2024
6199c84
metrics: remove deprecated uses of math.rand (#26710)
gzliudan Dec 13, 2024
c616077
metrics: improve accuracy of CPU gauges (#26793)
gzliudan Dec 13, 2024
9eae124
metrics: add cpu counters (#26796)
gzliudan Dec 13, 2024
6e055a6
metrics/influxdb: reuse code between v1 and v2 reporters (#26963)
gzliudan Dec 13, 2024
5457030
all: ensure resp.body closed (#26969)
gzliudan Dec 13, 2024
aa5e3e2
metrics: make gauge_float64 and counter_float64 lock free (#27025)
gzliudan Dec 13, 2024
7015a9e
metrics: use atomic type (#27121)
gzliudan Dec 13, 2024
7d40ca6
metrics: use sync.map in registry (#27159)
gzliudan Dec 13, 2024
e524043
metrics: use slices package for sorting (#27493 #27909)
gzliudan Dec 13, 2024
51e49a8
metrics: NilTimer should still run the function to be timed (#27723)
gzliudan Dec 13, 2024
d43f6c0
metrics: NilResettingTimer.Time should execute the timed function (#2…
gzliudan Dec 13, 2024
3538050
metrics, cmd/geth: informational metrics (prometheus, influxdb, opent…
gzliudan Dec 13, 2024
1fe46b8
metrics/librato: rename receiver re to rep
gzliudan Dec 13, 2024
cc4ea7a
metrics: refactor metrics (#28035)
gzliudan Dec 13, 2024
2f36a79
metrics: add support for enabling metrics from env vars (#28118)
gzliudan Dec 13, 2024
49b5886
rpc: fix ns/µs mismatch in metrics (#28649)
gzliudan Dec 13, 2024
6f67b35
metrics: fix docstring names (#28923)
gzliudan Dec 13, 2024
2220156
cmd, core, metrics: always report expensive metrics (#29191)
gzliudan Dec 13, 2024
7f37ee2
metrics: fix docstrings (#29279)
gzliudan Dec 13, 2024
fd8782b
metrics: use min from go1.21 (#29307)
gzliudan Dec 13, 2024
99f605f
metrics/influxdb: skip float64-precision-dependent tests on arm64 (#2…
gzliudan Dec 13, 2024
c7565af
metrics: remove dependency on golang.org/exp (#29314)
gzliudan Dec 13, 2024
988465c
metrics: fix mismatched names in comments (#29348)
gzliudan Dec 13, 2024
cf90a7a
metrics: remove librato (#29624)
gzliudan Dec 13, 2024
a9cbcde
metrics: fix out of range error message (#29821)
gzliudan Dec 13, 2024
fb6a268
metrics: fix flaky test `TestExpDecaySampleNanosecondRegression` (#29…
gzliudan Dec 13, 2024
7e784f1
metrics: add test for `SampleSnapshot.Sum` (#29831)
gzliudan Dec 13, 2024
ea355e8
metrics: fix function comment (#29843)
gzliudan Dec 13, 2024
7d82f27
metrics: fix issues with benchmarks (#30667)
gzliudan Dec 13, 2024
6beee27
metrics, cmd/XDC: change init-process of metrics (#30814)
gzliudan Dec 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 16 additions & 7 deletions cmd/XDC/chaincmd.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@ import (
"github.com/XinFinOrg/XDPoSChain/eth/downloader"
"github.com/XinFinOrg/XDPoSChain/event"
"github.com/XinFinOrg/XDPoSChain/log"
"github.com/XinFinOrg/XDPoSChain/metrics"
"github.com/urfave/cli/v2"
)

Expand Down Expand Up @@ -71,6 +70,18 @@ It expects the genesis file as argument.`,
utils.GCModeFlag,
utils.CacheDatabaseFlag,
utils.CacheGCFlag,
utils.MetricsEnabledFlag,
utils.MetricsEnabledExpensiveFlag,
utils.MetricsEnableInfluxDBFlag,
utils.MetricsEnableInfluxDBV2Flag,
utils.MetricsInfluxDBEndpointFlag,
utils.MetricsInfluxDBDatabaseFlag,
utils.MetricsInfluxDBUsernameFlag,
utils.MetricsInfluxDBPasswordFlag,
utils.MetricsInfluxDBTagsFlag,
utils.MetricsInfluxDBTokenFlag,
utils.MetricsInfluxDBBucketFlag,
utils.MetricsInfluxDBOrganizationFlag,
},
Description: `
The import command imports blocks from an RLP-encoded form. The form can be one file
Expand Down Expand Up @@ -227,14 +238,12 @@ func importChain(ctx *cli.Context) error {
if ctx.Args().Len() < 1 {
utils.Fatalf("This command requires an argument.")
}
// Start metrics export if enabled
utils.SetupMetrics(ctx)
// Start system runtime metrics collection
go metrics.CollectProcessMetrics(3 * time.Second)

stack, _ := makeFullNode(ctx)
stack, cfg := makeFullNode(ctx)
defer stack.Close()

// Start metrics export if enabled
utils.SetupMetrics(&cfg.Metrics)

chain, chainDb := utils.MakeChain(ctx, stack)
defer chainDb.Close()

Expand Down
79 changes: 78 additions & 1 deletion cmd/XDC/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ import (
"github.com/XinFinOrg/XDPoSChain/common"
"github.com/XinFinOrg/XDPoSChain/eth/ethconfig"
"github.com/XinFinOrg/XDPoSChain/internal/flags"
"github.com/XinFinOrg/XDPoSChain/log"
"github.com/XinFinOrg/XDPoSChain/metrics"
"github.com/XinFinOrg/XDPoSChain/node"
"github.com/XinFinOrg/XDPoSChain/params"
"github.com/naoina/toml"
Expand Down Expand Up @@ -90,6 +92,7 @@ type XDCConfig struct {
Eth ethconfig.Config
Node node.Config
Ethstats ethstatsConfig
Metrics metrics.Config
XDCX XDCx.Config
Account account
StakeEnable bool
Expand Down Expand Up @@ -122,12 +125,14 @@ func defaultNodeConfig() node.Config {
return cfg
}

// makeConfigNode loads geth configuration and creates a blank node instance.
func makeConfigNode(ctx *cli.Context) (*node.Node, XDCConfig) {
// Load defaults.
cfg := XDCConfig{
Eth: ethconfig.Defaults,
XDCX: XDCx.DefaultConfig,
Node: defaultNodeConfig(),
Metrics: metrics.DefaultConfig,
StakeEnable: true,
Verbosity: 3,
NAT: "",
Expand Down Expand Up @@ -208,6 +213,9 @@ func makeConfigNode(ctx *cli.Context) (*node.Node, XDCConfig) {
}

utils.SetXDCXConfig(ctx, &cfg.XDCX, cfg.Node.DataDir)

applyMetricConfig(ctx, &cfg)

return stack, cfg
}

Expand All @@ -227,10 +235,13 @@ func applyValues(values []string, params *[]string) {
func makeFullNode(ctx *cli.Context) (*node.Node, XDCConfig) {
stack, cfg := makeConfigNode(ctx)

// Start metrics export if enabled
utils.SetupMetrics(&cfg.Metrics)

// Register XDCX's OrderBook service if requested.
// enable in default
utils.RegisterXDCXService(stack, &cfg.XDCX)
utils.RegisterEthService(stack, &cfg.Eth)
utils.RegisterEthService(stack, &cfg.Eth, cfg.Node.Version)

// Add the Ethereum Stats daemon if requested.
if cfg.Ethstats.URL != "" {
Expand Down Expand Up @@ -258,3 +269,69 @@ func dumpConfig(ctx *cli.Context) error {
os.Stdout.Write(out)
return nil
}

func applyMetricConfig(ctx *cli.Context, cfg *XDCConfig) {
if ctx.IsSet(utils.MetricsEnabledFlag.Name) {
cfg.Metrics.Enabled = ctx.Bool(utils.MetricsEnabledFlag.Name)
}
if ctx.IsSet(utils.MetricsEnabledExpensiveFlag.Name) {
log.Warn("Expensive metrics are collected by default, please remove this flag", "flag", utils.MetricsEnabledExpensiveFlag.Name)
}
if ctx.IsSet(utils.MetricsHTTPFlag.Name) {
cfg.Metrics.HTTP = ctx.String(utils.MetricsHTTPFlag.Name)
}
if ctx.IsSet(utils.MetricsPortFlag.Name) {
cfg.Metrics.Port = ctx.Int(utils.MetricsPortFlag.Name)
}
if ctx.IsSet(utils.MetricsEnableInfluxDBFlag.Name) {
cfg.Metrics.EnableInfluxDB = ctx.Bool(utils.MetricsEnableInfluxDBFlag.Name)
}
if ctx.IsSet(utils.MetricsInfluxDBEndpointFlag.Name) {
cfg.Metrics.InfluxDBEndpoint = ctx.String(utils.MetricsInfluxDBEndpointFlag.Name)
}
if ctx.IsSet(utils.MetricsInfluxDBDatabaseFlag.Name) {
cfg.Metrics.InfluxDBDatabase = ctx.String(utils.MetricsInfluxDBDatabaseFlag.Name)
}
if ctx.IsSet(utils.MetricsInfluxDBUsernameFlag.Name) {
cfg.Metrics.InfluxDBUsername = ctx.String(utils.MetricsInfluxDBUsernameFlag.Name)
}
if ctx.IsSet(utils.MetricsInfluxDBPasswordFlag.Name) {
cfg.Metrics.InfluxDBPassword = ctx.String(utils.MetricsInfluxDBPasswordFlag.Name)
}
if ctx.IsSet(utils.MetricsInfluxDBTagsFlag.Name) {
cfg.Metrics.InfluxDBTags = ctx.String(utils.MetricsInfluxDBTagsFlag.Name)
}
if ctx.IsSet(utils.MetricsEnableInfluxDBV2Flag.Name) {
cfg.Metrics.EnableInfluxDBV2 = ctx.Bool(utils.MetricsEnableInfluxDBV2Flag.Name)
}
if ctx.IsSet(utils.MetricsInfluxDBTokenFlag.Name) {
cfg.Metrics.InfluxDBToken = ctx.String(utils.MetricsInfluxDBTokenFlag.Name)
}
if ctx.IsSet(utils.MetricsInfluxDBBucketFlag.Name) {
cfg.Metrics.InfluxDBBucket = ctx.String(utils.MetricsInfluxDBBucketFlag.Name)
}
if ctx.IsSet(utils.MetricsInfluxDBOrganizationFlag.Name) {
cfg.Metrics.InfluxDBOrganization = ctx.String(utils.MetricsInfluxDBOrganizationFlag.Name)
}
// Sanity-check the commandline flags. It is fine if some unused fields is part
// of the toml-config, but we expect the commandline to only contain relevant
// arguments, otherwise it indicates an error.
var (
enableExport = ctx.Bool(utils.MetricsEnableInfluxDBFlag.Name)
enableExportV2 = ctx.Bool(utils.MetricsEnableInfluxDBV2Flag.Name)
)
if enableExport || enableExportV2 {
v1FlagIsSet := ctx.IsSet(utils.MetricsInfluxDBUsernameFlag.Name) ||
ctx.IsSet(utils.MetricsInfluxDBPasswordFlag.Name)

v2FlagIsSet := ctx.IsSet(utils.MetricsInfluxDBTokenFlag.Name) ||
ctx.IsSet(utils.MetricsInfluxDBOrganizationFlag.Name) ||
ctx.IsSet(utils.MetricsInfluxDBBucketFlag.Name)

if enableExport && v2FlagIsSet {
utils.Fatalf("Flags --influxdb-metrics.organization, --influxdb-metrics.token, --influxdb-metrics.bucket are only available for influxdb-v2")
} else if enableExportV2 && v1FlagIsSet {
utils.Fatalf("Flags --influxdb-metrics.username, --influxdb-metrics.password are only available for influxdb-v1")
}
}
}
29 changes: 21 additions & 8 deletions cmd/XDC/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -126,9 +126,6 @@ var (
utils.HTTPCORSDomainFlag,
utils.HTTPVirtualHostsFlag,
utils.EthStatsURLFlag,
utils.MetricsEnabledFlag,
utils.MetricsHTTPFlag,
utils.MetricsPortFlag,
//utils.FakePoWFlag,
//utils.NoCompactionFlag,
//utils.GpoBlocksFlag,
Expand Down Expand Up @@ -163,6 +160,23 @@ var (
utils.IPCPathFlag,
utils.RPCGlobalTxFeeCap,
}

metricsFlags = []cli.Flag{
utils.MetricsEnabledFlag,
utils.MetricsEnabledExpensiveFlag,
utils.MetricsHTTPFlag,
utils.MetricsPortFlag,
utils.MetricsEnableInfluxDBFlag,
utils.MetricsInfluxDBEndpointFlag,
utils.MetricsInfluxDBDatabaseFlag,
utils.MetricsInfluxDBUsernameFlag,
utils.MetricsInfluxDBPasswordFlag,
utils.MetricsInfluxDBTagsFlag,
utils.MetricsEnableInfluxDBV2Flag,
utils.MetricsInfluxDBTokenFlag,
utils.MetricsInfluxDBBucketFlag,
utils.MetricsInfluxDBOrganizationFlag,
}
)

func init() {
Expand Down Expand Up @@ -194,6 +208,7 @@ func init() {
app.Flags = append(app.Flags, rpcFlags...)
app.Flags = append(app.Flags, consoleFlags...)
app.Flags = append(app.Flags, debug.Flags...)
app.Flags = append(app.Flags, metricsFlags...)
flags.AutoEnvVars(app.Flags, "XDC")

app.Before = func(ctx *cli.Context) error {
Expand All @@ -204,6 +219,9 @@ func init() {
}
flags.CheckEnvVars(ctx, app.Flags, "XDC")

// Start system runtime metrics collection
go metrics.CollectProcessMetrics(3 * time.Second)

utils.SetupNetwork(ctx)
return nil
}
Expand Down Expand Up @@ -303,11 +321,6 @@ func startNode(ctx *cli.Context, stack *node.Node, cfg XDCConfig) {
if ctx.Bool(utils.LightModeFlag.Name) || ctx.String(utils.SyncModeFlag.Name) == "light" {
utils.Fatalf("Light clients do not support staking")
}
// Start metrics export if enabled
utils.SetupMetrics(ctx)

// Start system runtime metrics collection
go metrics.CollectProcessMetrics(3 * time.Second)

var ethereum *eth.Ethereum
if err := stack.Service(&ethereum); err != nil {
Expand Down
Loading
Loading