Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Heartbeat] Refactor TCP Monitor #17549

Merged
merged 37 commits into from
Apr 20, 2020
Merged
Show file tree
Hide file tree
Changes from 35 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
3e6a95a
Move TCP dialchain builder into TCP package
andrewvc Apr 6, 2020
579e946
Move some TCP setup code into tcpMonitor struct
andrewvc Apr 6, 2020
1821b51
Move more setup code into object
andrewvc Apr 6, 2020
8fbe821
Rename ConnCheck to less confusing DataCheck
andrewvc Apr 6, 2020
751d388
Factor out loadConfig in tcpMonitor
andrewvc Apr 6, 2020
1f39e60
Initialize dialer late, use tcpMonitor for dialer construction
andrewvc Apr 6, 2020
6fd2c13
Late no more builder for plain TCP dials w/o SOCKS5
andrewvc Apr 6, 2020
760ebb9
TCP Dialer now layers host correctly
andrewvc Apr 6, 2020
5da7f86
Cleanup weird code around scheme/endpoints
andrewvc Apr 6, 2020
5b49fb8
More cleanup + socks5 support is back
andrewvc Apr 6, 2020
f292bb0
Code cleanup
andrewvc Apr 6, 2020
d7a2cf8
FMT
andrewvc Apr 6, 2020
5dc587e
Cleanup byHostname jobs
andrewvc Apr 6, 2020
f14e864
Passing socks5 test
andrewvc Apr 6, 2020
8c44e6a
More comments
andrewvc Apr 7, 2020
0d2bb51
Add TLS cert mismatch test + custom resolver
andrewvc Apr 7, 2020
1252f47
Split up test files
andrewvc Apr 7, 2020
613b417
Add changelog
andrewvc Apr 7, 2020
dbff2cf
Remove unnecessary line change
andrewvc Apr 7, 2020
93acae1
Add resolver test
andrewvc Apr 8, 2020
84735f8
Improve parsing of TCP 'hostnames' and add tests
andrewvc Apr 8, 2020
7ecf956
mage fmt
andrewvc Apr 9, 2020
101811c
Incorporate PR feedback, don't test ipv6 on boxes that don't support it
andrewvc Apr 9, 2020
bef1e8c
Incorporate PR feedback for resolvers
andrewvc Apr 15, 2020
7ee6e3c
Check server teardown errors and use defer
andrewvc Apr 15, 2020
520618a
Incorporate PR feedback
andrewvc Apr 15, 2020
9c2dac7
Incorporate PR feedback
andrewvc Apr 15, 2020
0d3d86a
go fmt
andrewvc Apr 15, 2020
c016696
Merge remote-tracking branch 'origin/master' into refactor-tcp
andrewvc Apr 15, 2020
754100d
Incorporate PR feedback
andrewvc Apr 15, 2020
1765973
Fix FMT
andrewvc Apr 16, 2020
376770d
Fix resolvers
andrewvc Apr 16, 2020
cd4ed05
Fix resolver ref
andrewvc Apr 16, 2020
5eb9374
Merge remote-tracking branch 'origin/master' into refactor-tcp
andrewvc Apr 16, 2020
07d091d
Merge remote-tracking branch 'origin/master' into refactor-tcp
andrewvc Apr 19, 2020
7543cba
Rename net.go for clarity, fix imports
andrewvc Apr 20, 2020
eb7f476
Merge remote-tracking branch 'origin/master' into refactor-tcp
andrewvc Apr 20, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
*Heartbeat*

- Fixed excessive memory usage introduced in 7.5 due to over-allocating memory for HTTP checks. {pull}15639[15639]
- Fixed TCP TLS checks to properly validate hostnames, this broke in 7.x and only worked for IP SANs. {pull}17549[17549]

*Journalbeat*

Expand Down
16 changes: 15 additions & 1 deletion heartbeat/hbtest/hbtestutil.go
Original file line number Diff line number Diff line change
Expand Up @@ -153,10 +153,19 @@ func SummaryChecks(up int, down int) validator.Validator {
})
}

// ResolveChecks returns a lookslike matcher for the 'resolve' fields.
func ResolveChecks(ip string) validator.Validator {
andrewvc marked this conversation as resolved.
Show resolved Hide resolved
return lookslike.MustCompile(map[string]interface{}{
"resolve": map[string]interface{}{
"ip": ip,
"rtt.us": isdef.IsDuration,
},
})
}

// SimpleURLChecks returns a check for a simple URL
// with only a scheme, host, and port
func SimpleURLChecks(t *testing.T, scheme string, host string, port uint16) validator.Validator {

hostPort := host
if port != 0 {
hostPort = fmt.Sprintf("%s:%d", host, port)
Expand All @@ -165,6 +174,11 @@ func SimpleURLChecks(t *testing.T, scheme string, host string, port uint16) vali
u, err := url.Parse(fmt.Sprintf("%s://%s", scheme, hostPort))
require.NoError(t, err)

return URLChecks(t, u)
}

// URLChecks returns a validator for the given URL's fields
func URLChecks(t *testing.T, u *url.URL) validator.Validator {
return lookslike.MustCompile(map[string]interface{}{
"url": wrappers.URLFields(u),
})
Expand Down
198 changes: 0 additions & 198 deletions heartbeat/monitors/active/dialchain/builder.go

This file was deleted.

7 changes: 4 additions & 3 deletions heartbeat/monitors/active/dialchain/net.go
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ import (
// }
// }
func TCPDialer(to time.Duration) NetDialer {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I see that the file net.go contains only dialers. Do you think we can rename it to dialers.go?
Also, there are some functions in util.go that refer to transport.Dialer, I wonder if we can divide the stuff in these two files into dialers.go and layers.go. We could get rid of utils ;)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, good call.

return netDialer(to)
return CreateNetDialer(to)
}

// UDPDialer creates a new NetDialer with constant event fields and default
Expand All @@ -62,10 +62,11 @@ func TCPDialer(to time.Duration) NetDialer {
// }
// }
func UDPDialer(to time.Duration) NetDialer {
return netDialer(to)
return CreateNetDialer(to)
}

func netDialer(timeout time.Duration) NetDialer {
// CreateNetDialer returns a NetDialer with the given timeout.
func CreateNetDialer(timeout time.Duration) NetDialer {
return func(event *beat.Event) (transport.Dialer, error) {
return makeDialer(func(network, address string) (net.Conn, error) {
namespace := ""
Expand Down
9 changes: 0 additions & 9 deletions heartbeat/monitors/active/dialchain/util.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,6 @@ type timer struct {
s, e time.Time
}

// IDLayer creates an empty placeholder layer.
func IDLayer() Layer {
return _idLayer
}

var _idLayer = Layer(func(event *beat.Event, next transport.Dialer) (transport.Dialer, error) {
return next, nil
})

// ConstAddrLayer introduces a network layer always passing a constant address
// to the underlying layer.
func ConstAddrLayer(address string) Layer {
Expand Down
4 changes: 1 addition & 3 deletions heartbeat/monitors/active/http/task.go
Original file line number Diff line number Diff line change
Expand Up @@ -94,10 +94,8 @@ func newHTTPMonitorIPsJob(
return nil, err
}

settings := monitors.MakeHostJobSettings(hostname, config.Mode)

pingFactory := createPingFactory(config, port, tls, req, body, validator)
job, err := monitors.MakeByHostJob(settings, pingFactory)
job, err := monitors.MakeByHostJob(hostname, config.Mode, monitors.NewStdResolver(), pingFactory)

return job, err
}
Expand Down
3 changes: 1 addition & 2 deletions heartbeat/monitors/active/icmp/icmp.go
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,7 @@ func create(
pingFactory := monitors.MakePingIPFactory(createPingIPFactory(&config))

for _, host := range config.Hosts {
settings := monitors.MakeHostJobSettings(host, config.Mode)
job, err := monitors.MakeByHostJob(settings, pingFactory)
job, err := monitors.MakeByHostJob(host, config.Mode, monitors.NewStdResolver(), pingFactory)

if err != nil {
return nil, 0, err
Expand Down
12 changes: 7 additions & 5 deletions heartbeat/monitors/active/tcp/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ import (
"github.com/elastic/beats/v7/libbeat/common/transport/tlscommon"
)

type Config struct {
type config struct {
// check all ports if host does not contain port
Hosts []string `config:"hosts" validate:"required"`
Ports []uint16 `config:"ports"`
Expand All @@ -45,12 +45,14 @@ type Config struct {
ReceiveString string `config:"check.receive"`
}

var DefaultConfig = Config{
Timeout: 16 * time.Second,
Mode: monitors.DefaultIPSettings,
func defaultConfig() config {
return config{
Timeout: 16 * time.Second,
Mode: monitors.DefaultIPSettings,
}
}

func (c *Config) Validate() error {
func (c *config) Validate() error {
if c.Socks5.URL != "" {
if c.Mode.Mode != monitors.PingAny && !c.Socks5.LocalResolve {
return errors.New("ping all ips only supported if proxy_use_local_resolver is enabled`")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,18 +23,20 @@ import (
"net"
)

type ConnCheck func(net.Conn) error
// dataCheck executes over an open TCP connection using the send / receive
// parameters the user has defined.
type dataCheck func(net.Conn) error

var (
errNoDataReceived = errors.New("no data")
errRecvMismatch = errors.New("received string mismatch")
)

func (c ConnCheck) Validate(conn net.Conn) error {
func (c dataCheck) Check(conn net.Conn) error {
return c(conn)
}

func makeValidateConn(config *Config) ConnCheck {
func makeDataCheck(config *config) dataCheck {
send := config.SendString
recv := config.ReceiveString

Expand All @@ -52,7 +54,7 @@ func makeValidateConn(config *Config) ConnCheck {

func checkOk(_ net.Conn) error { return nil }

func checkAll(checks ...ConnCheck) ConnCheck {
func checkAll(checks ...dataCheck) dataCheck {
return func(conn net.Conn) error {
for _, check := range checks {
if err := check(conn); err != nil {
Expand All @@ -63,13 +65,13 @@ func checkAll(checks ...ConnCheck) ConnCheck {
}
}

func checkSend(buf []byte) ConnCheck {
func checkSend(buf []byte) dataCheck {
return func(conn net.Conn) error {
return sendBuffer(conn, buf)
}
}

func checkRecv(expected []byte) ConnCheck {
func checkRecv(expected []byte) dataCheck {
return func(conn net.Conn) error {
buf := make([]byte, len(expected))
if err := recvBuffer(conn, buf); err != nil {
Expand Down
Loading