Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Display ingested software on host details page. #19576

Merged
merged 8 commits into from
Jun 12, 2024
Merged

Display ingested software on host details page. #19576

merged 8 commits into from
Jun 12, 2024

Conversation

getvictor
Copy link
Member

@getvictor getvictor commented Jun 6, 2024 •
edited
Loading

#19348

Fixed host details page and device details page not showing the latest software.

  • During software ingestion, software titles are now added if needed and software items have their title_id field populated.
  • In addition, after refreshing via UI, the software will be re-fetched if it has been modified.

Added exclude_software query parameter to the /api/latest/fleet/hosts/:id endpoint to exclude software from the response.

PR for API doc change: #19617

Related issue filed for the Device User Page: #19618

Checklist for submitter

If some of the following don't apply, delete the relevant line.

  • Changes file added for user-visible changes in changes/, orbit/changes/ or ee/fleetd-chrome/changes.
    See Changes files for more information.
  • Added/updated tests
  • Manual QA for all new/changed functionality

@codecov Codecov
Copy link

codecov bot commented Jun 7, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 80.24691% with 32 lines in your changes missing coverage. Please review.

Project coverage is 62.50%. Comparing base (df16d76) to head (1ebc197).
Report is 20 commits behind head on main.

❗ Current head 1ebc197 differs from pull request most recent head 15a8444

Please upload reports for the commit 15a8444 to get more accurate results.

Files Patch % Lines
server/datastore/mysql/software.go 80.48% 17 Missing and 7 partials ⚠️
...s/tables/20240607133721_ReconcileSoftwareTitles.go 79.48% 6 Missing and 2 partials ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main   #19576      +/-   ##
==========================================
+ Coverage   62.37%   62.50%   +0.12%     
==========================================
  Files        1397     1398       +1     
  Lines      130578   130342     -236     
  Branches     3166     3162       -4     
==========================================
+ Hits        81454    81475      +21     
+ Misses      42887    42631     -256     
+ Partials     6237     6236       -1
Flag Coverage Δ
backend 63.36% <80.24%> (+0.14%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@getvictor getvictor marked this pull request as ready for review June 7, 2024 22:33
@getvictor getvictor requested review from a team as code owners June 7, 2024 22:33
lucasmrod
lucasmrod previously approved these changes Jun 11, 2024
View reviewed changes
Copy link
Member

@lucasmrod lucasmrod left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Left some questions.

server/datastore/mysql/migrations/tables/20240607133721_ReconcileSoftwareTitles.go
software s
WHERE
NOT EXISTS (SELECT 1 FROM software_titles st WHERE (s.name, s.source, s.browser) = (st.name, st.source, st.browser))
ON DUPLICATE KEY UPDATE software_titles.id = software_titles.id`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why UPDATE software_titles.id = software_titles.id?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is how it was done in ReconcileSoftwareTitles method, so I kept it consistent. I'm guessing it was done so that the INSERT statement doesn't fail. It does seem like INSERT IGNORE would be less wordy/faster.

server/service/hosts.go Show resolved Hide resolved
server/datastore/mysql/software.go Outdated Show resolved Hide resolved
server/datastore/mysql/software.go Outdated Show resolved Hide resolved
server/datastore/mysql/software.go Outdated Show resolved Hide resolved
server/datastore/mysql/software.go

// update new title ids for new software table entries
updateSoftwareStmt := `
UPDATE
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am worried about this update locking the two tables. Maybe load test will tell us if this is expensive.

Another approach is to insert the titles first, then insert the software entries with the existing and new title_ids.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This flow should only lock the new software items that are being inserted and the new titles that are being inserted. Yes, load test should tell us if we're getting long lock times when many hosts try to enroll and insert their software.

@lucasmrod
Copy link
Member

lucasmrod commented Jun 11, 2024 •
edited
Loading

Something to think about:

  • The vulnerabilities cron runs the following jobs sequentially: cron_vulnerabilities+cron_sync_host_software+cron_reconcile_software_titles+
    cron_sync_hosts_software_titles (maybe we should decouple it in the future)
  • @xpkoala Do we usually test/QA Fleet with vulnerabilities disabled? Which brings to my next question.
  • @sharon-fdm How many customers are using Fleet with vulnerabilities disabled? (I believe our metrics do have that information)

@getvictor getvictor force-pushed the victor/19348-software-host-details-page branch from dc8801b to 15a8444 Compare June 11, 2024 19:07
@getvictor
Copy link
Member Author

Something to think about:

  • The vulnerabilities cron runs the following jobs sequentially: cron_vulnerabilities+cron_sync_host_software+cron_reconcile_software_titles+
    cron_sync_hosts_software_titles (maybe we should decouple it in the future)
  • @xpkoala Do we usually test/QA Fleet with vulnerabilities disabled? Which brings to my next question.
  • @sharon-fdm How many customers are using Fleet with vulnerabilities disabled? (I believe our metrics do have that information)

Yes, I filed a feature request for it: #19546
It could be considered a bug -- I don't think we ever intended vulnerability processing to prevent software aggregation.

ghernandez345
ghernandez345 approved these changes Jun 12, 2024
View reviewed changes
Copy link
Contributor

@ghernandez345 ghernandez345 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FE looks good. Thanks for making this improvement 👍🏽

@getvictor getvictor merged commit 8c4c739 into main Jun 12, 2024
18 checks passed
@getvictor getvictor deleted the victor/19348-software-host-details-page branch June 12, 2024 13:38
georgekarrv pushed a commit that referenced this pull request Jun 20, 2024
@getvictor @georgekarrv
Display ingested software on host details page. (#19576)
d0bd4fe 
#19348 

Fixed host details page and device details page not showing the latest
software.
- During software ingestion, software titles are now added if needed and
software items have their title_id field populated.
- In addition, after refreshing via UI, the software will be re-fetched
if it has been modified.

Added `exclude_software` query parameter to the
`/api/latest/fleet/hosts/:id` endpoint to exclude software from the
response.

PR for API doc change: #19617

Related issue filed for the Device User Page:
#19618

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lucasmrod lucasmrod lucasmrod approved these changes

@ghernandez345 ghernandez345 ghernandez345 approved these changes

Assignees

@lucasmrod lucasmrod

@jacobshandling jacobshandling

Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@getvictor @lucasmrod @ghernandez345 @jacobshandling @gillespi314