Address leftover TODOs for remote YARA rules feature

[lucasmrod]

Here are the remaining TODOs for the parent issue: #14899

• REST API changes: TODO: Specify changes in the the REST API doc page as a PR to reference docs release branch. Put "No changes" if there are no changes necessary. Move this item to the engineering list below if engineering will design the API changes.

We need to add yara_rules to the /api/latest/fleet/config endpoints.

• Changes to paid features or tiers: TODO: Specify changes in pricing-features-table.yml as a PR to reference docs release branch. Specify "Fleet Free" and/or "Fleet Premium" if there are no changes to the pricing page necessary.

Document feature is available for free instances.

• Load testing: TODO

We need to add support to osquery-perf to be able to load test the feature (and then perform a load test that uses the feature).

[sgress454]

We need to add yara_rules to the /api/latest/fleet/config endpoints.

Do you mean we need to actually build this endpoint? The bullet point made it seem like this is about documentation but I don't see any such endpoint currently existing. The only new API I see as having shipped with this feature is /api/osquery/yara/{name}. Happy to document this but I don't see where non-/api/*/fleet endpoints are documented (probably because they're internal?)

We need to add support to osquery-perf to be able to load test the feature (and then perform a load test that uses the feature).

What would the desired load test be? Run a YARA query on a bunch of hosts where the rule is located remotely, to test that the rule caching works as designed?

[lucasmrod]

Do you mean we need to actually build this endpoint? The bullet point made it seem like this is about documentation but I don't see any such endpoint currently existing. The only new API I see as having shipped with this feature is /api/osquery/yara/{name}. Happy to document this but I don't see where non-/api/*/fleet endpoints are documented (probably because they're internal?)

Sorry for not being clear.
We added yara_rules to the config endpoint /api/latest/fleet/config in v4.61.0:
See

fleet/server/fleet/app.go

Line 523 in e759582

YaraRules []YaraRule `json:"yara_rules,omitempty"`

This bullet is just missing documentation in /api/latest/fleet/config endpoints:

fleet/docs/REST API/rest-api.md

Line 767 in e759582

### Get configuration

Probably just add yara_rules to some of the /api/latest/fleet/config example responses in rest-api.md.

PS: We don't need to document the newly added /api/osquery/yara/* API because that's osquery facing, not part of the user API.

What would the desired load test be? Run a YARA query on a bunch of hosts where the rule is located remotely, to test that the rule caching works as designed?

We need to add support to osquery-perf to simulate osquery requesting YARA rules using the new authenticated endpoint (to load test thousands of agents using this new endpoint).
osquery-perf: https://github.com/fleetdm/fleet/blob/main/cmd/osquery-perf/agent.go

It doesn't need to be an exact simulation of osquery, just something that allows us to load test the new endpoint.
An idea after reading remote-yara-rules.md:
osquery-perf could detect (somewhere here) if a live query has FROM yara in it and extract sigurl and then do an authenticated request to the endpoint (which is Fleet) to get the rule and then discard it. Then during load testing we just run a live query on all enrolled hosts SELECT * FROM yara WHERE path="/bin/ls" AND sigurl='https://example.fleetdm.com/api/osquery/yara/rule1.yar' (after configuring the rule1.yar in Fleet) and then check the load on the infrastructure.

Let me know if I'm making sense.

[sgress454]

Thanks @lucasmrod! Both make sense. For the load test I'll probably add the logic inside the runLiveQuery implementation so it can take advantage of the same options around failure and "no results", but branch off to do the request and return a different response.

[sgress454]

@lucasmrod I just looked into the yara_rules config and while it was added to the AppConfig struct as you pointed out, it's not actually being returned by the config endpoint. It would need to be added to the response here, which is easy enough but I'm not sure it's what we want. By default it would output the rule names and their contents, and I doubt we want the contents returned via the API when all that work was done to allow them to be private. I could redact the contents, but I'm not whether a list of rule names is useful enough to be returned, which is perhaps why it wasn't added to the response in the first place (or it may have just been overlooked).