Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates to SQL Server Plugin, better Azure Managed Instance support + more #4642

Merged
merged 8 commits into from
Sep 12, 2018

Conversation

m82labs
Copy link
Contributor

@m82labs m82labs commented Sep 6, 2018

closes #4222
closes #4597

Required for all PRs:

  • [ x ] Signed CLA.
  • [ x ] Associated README.md updated.
  • [ x ] Has appropriate unit tests.

Changes

  • Altered database IO stats to include stats on a per-file basis
  • Added a few new perf counters including Background Writer pages/sec which tracks indirect checkpoints
  • Refactored some queries to be compatible with Azure Managed Instances
  • Changed to the denisenkom SQL package, prior to the change this plugin would not work properly with managed instances
  • Some perf counters now grab data per database instead of just for the "_Total" counter instance
  • Moved some newer wait types into a proper category
  • Added a few new wait types to the list of types to ignore
  • Now filtering for wait types that have accumulated some amount of waits (100 ms)

The majority of these changes came from Denzil Ribeiro from Microsoft, so big thanks to Denzil!

@m82labs
Copy link
Contributor Author

m82labs commented Sep 6, 2018

@danielnelson I changed a dependency, not sure how to get that added into the deps.

@danielnelson
Copy link
Contributor

I think we need to modify the Gopkg.toml by hand:

-  name = "github.com/zensqlmonitor/go-mssqldb"
+  name = "github.com/denisenkom/go-mssqldb"

After this run dep ensure, this should be all we need but also run dep check to make sure all is well.

@danielnelson danielnelson added feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin area/sqlserver labels Sep 6, 2018
@danielnelson
Copy link
Contributor

I think we need dep ensure still. Do you have a list of the issues this addresses?

@m82labs
Copy link
Contributor Author

m82labs commented Sep 6, 2018 via email

@danielnelson
Copy link
Contributor

Maybe it matches some feature requests. Can you take a look at, being careful not to link yet, these issues: 4569, 4597, 4523, 4398, 4382

@m82labs
Copy link
Contributor Author

m82labs commented Sep 7, 2018

@danielnelson just added a fix for 4597, 4382 was already closed out via #4571

@danielnelson
Copy link
Contributor

Looks like that pr was for 4398, closed now.

@danielnelson danielnelson added this to the 1.8.0 milestone Sep 8, 2018
@m82labs
Copy link
Contributor Author

m82labs commented Sep 10, 2018

@danielnelson I am assuming the build issue is not related to my changes?

)
) OR (
) OR (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

un-needed extra spacing.

uptime INT
)

IF OBJECT_ID('master.sys.dm_os_sys_info') IS NOT NULL
IF OBJECT_ID('master.sys.dm_os_sys_info') IS NOT NULL
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

un-needed extra spacing

AND waiting_tasks_count > 0
ORDER BY
waiting_tasks_count DESC
AND wait_time_ms > 100
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this intended?

@m82labs
Copy link
Contributor Author

m82labs commented Sep 10, 2018 via email

@danielnelson
Copy link
Contributor

@m82labs Are there any README changes we should make with this change?

@m82labs
Copy link
Contributor Author

m82labs commented Sep 10, 2018 via email

@danielnelson
Copy link
Contributor

Can you check if this resolves 4222?

@m82labs
Copy link
Contributor Author

m82labs commented Sep 11, 2018

@danielnelson I think it does. I don't have anything to test it against, but I know when this was tested against Azure Managed Instances it would not work until the SQL package was changed.

@danielnelson
Copy link
Contributor

@m82labs One last question, all the changes are backwards compatible? New fields and tags are okay but are there any renames or type changes?

@m82labs
Copy link
Contributor Author

m82labs commented Sep 12, 2018 via email

@danielnelson
Copy link
Contributor

danielnelson commented Sep 12, 2018

That should be okay since the new data would be tagged.

@denzilribeiro
Copy link
Contributor

denzilribeiro commented Sep 12, 2018 via email

@danielnelson danielnelson merged commit b43165f into influxdata:master Sep 12, 2018
@danielnelson
Copy link
Contributor

Users always have the option to trim out unwanted data with tagpass, tagdrop, fieldpass, etc so thats no problem.

Thanks again @m82labs and @denzilribeiro

@erik-wramner
Copy link

The readme for the plugin specifies grant VIEW SERVER STATE and VIEW ANY DEFINITION and that doesn't work on Azure. Can you please share the privileges you granted to the telegraf account? Or ideally update the readme for the plugin.

@m82labs
Copy link
Contributor Author

m82labs commented Sep 13, 2018

@erik-wramner can you try granting VIEW DATABASE STATE. I'll try to get something set up to test today, but that might be the permission needed.

@denzilribeiro
Copy link
Contributor

denzilribeiro commented Sep 13, 2018

The Azure SQL DB (not managed instance) plugin part has to be reworked some. The permissions do work on Azure SQL managed instance aka - https://docs.microsoft.com/en-us/azure/sql-database/sql-database-managed-instance
Some changes needed

  • the AzureDB flag is global means once you set it you enable it for all connection strings rather than for only those that are Azure DB
  • Connection string has to go to a Database not a server given metrics are at a DB level.

To be honest the Azure SQL DB part of plugin has to be rethought as a whole and queries be made conditional based on version type, different dashboards to visualize the "per database" metrics I can help Mark with that but anytime after next 2 weeks, address that perhaps in a different PR.

@erik-wramner
Copy link

I would like to reopen this issue, please. Perhaps it is only documentation, but the plugin is still not working. I tried connecting directly to a database with a user with VIEW DATABASE STATE and the plugin crashes. First it complains about permissions:

VIEW SERVER STATE permission was denied on object 'server', database 'master'.

Then it crashes:

`panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x1226554]

goroutine 24 [running]:
github.com/influxdata/telegraf/plugins/inputs/sqlserver.(*SQLServer).accRow(0xc04240c400, 0x1fd9b51, 0x265, 0x1, 0xc0437ca180, 0xc, 0xc, 0x21b0580, 0xc0421f8860, 0x218bf80, ...)
/go/src/github.com/influxdata/telegraf/plugins/inputs/sqlserver/sqlserver.go:222 +0x754
github.com/influxdata/telegraf/plugins/inputs/sqlserver.(*SQLServer).gatherServer(0xc04240c400, 0xc04228c161, 0x9e, 0x1fd9b51, 0x265, 0x1, 0xc0437ca180, 0xc, 0xc, 0x21b0580, ...)
/go/src/github.com/influxdata/telegraf/plugins/inputs/sqlserver/sqlserver.go:178 +0x220
github.com/influxdata/telegraf/plugins/inputs/sqlserver.(*SQLServer).Gather.func1(0xc042039490, 0x21b0580, 0xc0421f8860, 0xc04240c400, 0xc04228c161, 0x9e, 0x1fd9b51, 0x265, 0x1, 0x0, ...)
/go/src/github.com/influxdata/telegraf/plugins/inputs/sqlserver/sqlserver.go:141 +0xb7
created by github.com/influxdata/telegraf/plugins/inputs/sqlserver.(*SQLServer).Gather
/go/src/github.com/influxdata/telegraf/plugins/inputs/sqlserver/sqlserver.go:139 +0x1da`

This is with the build telegraf-1.8.0~rc2_windows_amd64 which should contain the fix?

Please update the documentation or at least this issue with a working connection string for a user with well-defined permissions and (if necessary) what edition of SQL Database that is supported. I'm not using managed instance, which is still in preview.

@danielnelson
Copy link
Contributor

danielnelson commented Sep 19, 2018

@erik-wramner Can you open the panic as a new issue if is different from the stack on #4222? I reopened 4222.

@erik-wramner
Copy link

The stacks are the same. Is it possible that telegraf-1.8.0~rc2_windows_amd64 contains the old plugin? It is a new build, only a few days old, but it is odd that the line numbers are the same?

@danielnelson
Copy link
Contributor

@erik-wramner I don't think its possible, the changes in 1.8 mostly just modify the queries later in the file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/sqlserver feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin
Projects
None yet
5 participants