You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
we have a quite large PRTG Installation and sometimes it takes a long time that the prtg api does it thing ;)
No i tried to set RetryCount and RetryDelay higher but it does not seem to work:
I set RetryCount and RetryDelay to 20: Connect-PrtgServer "https://$($PRTGServer):$($PRTGPort)" (New-Credential $PRTGUser $PRTGPassHash) -PassHash -RetryCount 20 -RetryDelay 20
And now i expected that i see something like Retries remaining 20 instead of 4 ?
Am i just doing it wrong?
The screenshot is not manipulated in any form. It was just a half output of the get-prtgclient.
Sure, here we go with some additional informations:
Get-PrtgClient-Diagnostic
PSVersion : 5.1.17763.2803
PSEdition : Desktop
OS : Microsoft Windows Server 2019 Standard
PrtgAPIVersion : 0.9.17
Culture : de-DE
CLRVersion : .NET Framework 4.7.2 (461814)
PrtgVersion : 22.2.76.1705
PrtgLanguage : german.lng
The "call" of the actual Add-Sensor/Add-Device Sensor is quite complex and i just can't copy paste it in three lines.
This is my Setup:
Script: Update-PRTGDevices.ps1
Import-Module prtghelper.psm1 -force # This is a custom helper module for us internal to share some methods between scriptsConnect-PrtgServer"https://$($PRTGServer):$($PRTGPort)" (New-Credential$PRTGUser$PRTGPassHash) -PassHash -RetryCount 50-RetryDelay 50# Gather some devices from the crmforeach($erpdevicein$erpdevices){
$myDevice=Set-PRTGDevice-Parent $someParentGroup-ERPDevice $erpdevice# now we add / replace some sensors with the "same" method as on the the device.
}
Custom Module: prtghelper.psm1
functionSet-PRTGDevice{
param($Parent,$ERPDevice)
$device=$Parent|Get-Device-Name $DeviceName# Sometimes this just return nothing even if there is a correct deviceif(($device|Measure-Object).Count -gt0){
if(($device|Measure-Object).Count -gt1){
throw"$DeviceName in $($Parent.Name) not unique"
}
$device|Set-ObjectProperty Host $Hostname
}else{
$device=$Parent|Add-Device$DeviceName$($Hostname)
# THIS will "fail" and take aged on adding multiple devices. and i return a wrong result
}
return$device
}
PSVersion : 5.1.19041.4648
PSEdition : Desktop
OS : Microsoft Windows 10 Pro
PrtgAPIVersion : 0.9.19
Culture : en-US
CLRVersion : 533325
PrtgVersion : 24.2.96.1375
PrtgLanguage : english.lng
Targets : PrtgAPI.Request.PrtgTargetHelper
Server : localhost
UserName : prtgadmin
PassHash : 2154168053
RetryCount : 10
RetryDelay : 30
LogLevel : Trace, Request
Version : 24.2.96.1375
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 4
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 3
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 2
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 1
Add-Device : Could not resolve object: PRTG is taking too long to create the object. Confirm the object has been
created in the Web UI and then attempt resolution again manually.
At C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML\SyncWithArqivis.ps1:63 char:17
+ ... $device = Add-Device -Name $name -Host $_.album.url -Destination $g ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Add-Device], ObjectResolutionException
+ FullyQualifiedErrorId : PrtgAPI.ObjectResolutionException,PrtgAPI.PowerShell.Cmdlets.AddDevice
I think I understand what's happening here. There are two levels of retries
The retries that occur in RequestEngine relate to when the PRTG server throws an exception saying that your request timed out. If that occurs, we allow retrying a set number of times with an increasing backoff delay
The retries that occur in the RequestObject method pertain to the resolution of the created object. The API request asking for the created object is succeeding...it's just that the object hasn't been created yet.
I have effectively moved onto other projects from PrtgAPI, and am mostly providing basic level support for the product as-is until Paessler break API V1 compatibility. I think exposing something to allow specifying an object-creation-resolution-retry-count should be easy enough however, so I will work on doing this
Can you please try the latest pre-release build and advise how you go? PrtgAPI now takes the RetryCount and RetryDelay into consideration when resolving objects. I have updated the wiki to make note of this fact. Note that PrtgAPI uses a backoff algorithm when consuming the RetryDelay, wherein it will wait longer and longer and longer with each failed attempt. As such, it's recommended to keep your retry delay small (i.e. 3 seconds) and add a few extra retry attempts
So, the countdown "works" but it never actually resolves...
Connect-PrtgServer-Server $PrtgServer-Credential $credentials-PassHash -IgnoreSSL -RetryCount 10-RetryDelay 5
...
$device=Add-Device-Name $name-Host $_.album.url-Destination $group
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 10
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 9
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 8
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 7
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 6
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 5
The time it takes between tries is not linear, at first it takes 5 seconds before retrying... then 10... then 15... then 20... it looks like you don't just sleep $RetryDelay, but something akin to $i * $RetryDelay.
Then the other problem is that it never detects the device was actually added.
I cannot show you the name because it's a client's name, but as you can see Get-Device works only if I use -Name with a wildcard or with Where-Object, but using the plain string does not work.
The name has a value similar to this (maybe pipes/quotes cause troubles):
Foo "BAR-BAZ | BAK | Lorem Ipsum KEWK"
Maybe that bug is related to the Add-Device bug that does not resolve objects, because it checks with exact name?
What type of sensor are you adding? If it's something to do with SNMP, the created type may differ from what you're adding (see the wiki for more info). You have to specify $params.DynamicTrue = $true in that scenario
If $name contains a character that is considered a wildcard by PowerShell, this can prevent Get-Device from returning the expected results, because it attempts to filter the results returned by PRTG for those that match the wildcard expression. Wildcard characters include *, ?[ and ]. So a [ in a wildcard pattern won't match a [ literal. If you experience issues with the -Name parameter, you can use the New-SearchFilter cmdlet (aliased as flt) to be explicit
$PrtgServer='localhost'$PrtgUserName='admin@bla.com'$PrtgUserHash='123456789'$credentials=New-Credential-Username $PrtgUsername-Password $PrtgPasshashConnect-PrtgServer-Server $PrtgServer-Credential $credentials-PassHash -IgnoreSSL -RetryCount 10-RetryDelay 5$group=Get-Group-Name 'Test group'if (!$group) {
throw'Could not find group'
}
$name='Foo "BAR-BAZ | BAK | Lorem Ipsum KEWK"'$device=Add-Device-Name $name-Host '1.2.3.4'-Destination $group# This outputs this:
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 10
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 9
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 8
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 7
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 6
WARNING: 'Add-Device' failed to resolve device: object is still being created. Retries remaining: 5
I don't mention the sensor because the error is when I create the device, so it's the Add-Device call like my screenshot is showing.
As you see the name has no special characters besides quotes & pipes.
The -Verbose output will contain sensitive information, but the key thing we want to ascertain is to what extent the name in the URL it's querying for matches what you're specifying to the cmdlet, and whether the object is being included in the response and PrtgAPI is just ignoring it. I have a strong suspicion that the issue is related to your use of | pipe characters; I suspect PRTG or PowerShell is interpreting these in a special way
Ok, I found out it's the quotes and not the pipes. It used to work. I guess latest updates from PRTG broke something, the good news is that I can use any string to trigger it, will post traces soon.
On the basis that I'm escaping the quotes correctly, I feel like PrtgAPI is doing the right thing.
If you paste this URL into your web browser it will also give the XML response. You can then potentially fiddle around specifying different values to the filter_name parameter, which may enable you to tell Paessler what's wrong.
Alternatively, if you find that names with single quotes work, potentially modifying your names to use single quotes instead of double quotes would work? This can easily be done with a one-liner with PrtgAPI
Still what's weird is that PRTG creates the node with quotes just fine. And my workaround where I query the name using Where-Object on Get-Device in a loop also work just fine.
So my guess is that the regression is subtle... like only filter_name in the PRTG api regressed.
The purpose of the request retry feature is that in large PRTG installations (10,000 sensors) there can be quite a delay in attempting to perform certain actions, in particular creating new objects. Thus, when PrtgAPI repeatedly queries for the given object, and find again and again it hasn't been created yet, the assumption is that your sever must be under intense load; we want to do our best to guarantee that your request will succeed, hence why PrtgAPI will wait longer and longer and longer to maximize its chance at success.
The implication of this however is that you should not go too crazy modifying the default retry parameters; I believe the defaults might result in a total wait of around 30 seconds or so. The scaling behavior ensures that PrtgAPI's default retry rules will generally function well on both small and large PRTG installations, scaling up the delay as required for your system.
Activity
lordmilko commentedon Jul 5, 2022
I'm unable to reproduce this issue. Are you able to confirm what version of PrtgAPI you're running?
Has the above screenshot been manipulated, or is it the output of a script? (as opposed to raw commands being typed in a PowerShell prompt)
Can you also please provide the output of
Get-PrtgClient -Diagnosticxasz commentedon Jul 5, 2022
The screenshot is not manipulated in any form. It was just a half output of the get-prtgclient.
Sure, here we go with some additional informations:
The "call" of the actual Add-Sensor/Add-Device Sensor is quite complex and i just can't copy paste it in three lines.
This is my Setup:
Script: Update-PRTGDevices.ps1
Custom Module: prtghelper.psm1
Silex commentedon Jul 23, 2024
I have the same problem 😞 With PrtgApi 0.9.19
RetryCount when using
Add-Deviceseems to be hardcoded at 5.Silex commentedon Jul 23, 2024
Variable is correctly set:
Output:
Silex commentedon Jul 23, 2024
Ok I think I found the cause:
PrtgAPI/src/PrtgAPI/Request/PrtgClient.Generated.cs
Line 1197 in e3fcb40
It looks hardcoded here.
And it it's not then probably that
PrtgAPI/src/PrtgAPI/Request/RequestEngine.cs
Line 244 in e3fcb40
Silex commentedon Jul 23, 2024
@lordmilko do you have a workaround, other than rolling our own sleep-then-check-in-a-loop?
lordmilko commentedon Jul 24, 2024
Thanks @Silex,
I think I understand what's happening here. There are two levels of retries
RequestEnginerelate to when the PRTG server throws an exception saying that your request timed out. If that occurs, we allow retrying a set number of times with an increasing backoff delayRequestObjectmethod pertain to the resolution of the created object. The API request asking for the created object is succeeding...it's just that the object hasn't been created yet.I have effectively moved onto other projects from PrtgAPI, and am mostly providing basic level support for the product as-is until Paessler break API V1 compatibility. I think exposing something to allow specifying an object-creation-resolution-retry-count should be easy enough however, so I will work on doing this
Respect PrtgClient.RetryCount and RetryDelay when resolving objects (…
lordmilko commentedon Jul 24, 2024
Hi all,
Can you please try the latest pre-release build and advise how you go? PrtgAPI now takes the
RetryCountandRetryDelayinto consideration when resolving objects. I have updated the wiki to make note of this fact. Note that PrtgAPI uses a backoff algorithm when consuming theRetryDelay, wherein it will wait longer and longer and longer with each failed attempt. As such, it's recommended to keep your retry delay small (i.e. 3 seconds) and add a few extra retry attemptsSilex commentedon Aug 5, 2024
Thanks!
So, the countdown "works" but it never actually resolves...
The time it takes between tries is not linear, at first it takes 5 seconds before retrying... then 10... then 15... then 20... it looks like you don't just sleep
$RetryDelay, but something akin to$i * $RetryDelay.Then the other problem is that it never detects the device was actually added.
Silex commentedon Aug 5, 2024
Ok, by replacing with the following workaround it works (notice the -Resolve:$false):
That said, I found a weird bug:
I cannot show you the name because it's a client's name, but as you can see
Get-Deviceworks only if I use-Namewith a wildcard or withWhere-Object, but using the plain string does not work.The name has a value similar to this (maybe pipes/quotes cause troubles):
Maybe that bug is related to the
Add-Devicebug that does not resolve objects, because it checks with exact name?lordmilko commentedon Aug 7, 2024
What type of sensor are you adding? If it's something to do with SNMP, the created type may differ from what you're adding (see the wiki for more info). You have to specify
$params.DynamicTrue = $truein that scenarioIf
$namecontains a character that is considered a wildcard by PowerShell, this can preventGet-Devicefrom returning the expected results, because it attempts to filter the results returned by PRTG for those that match the wildcard expression. Wildcard characters include*,?[and]. So a[in a wildcard pattern won't match a[literal. If you experience issues with the-Nameparameter, you can use theNew-SearchFiltercmdlet (aliased asflt) to be explicitSilex commentedon Aug 7, 2024
Here's a testcase:
I don't mention the sensor because the error is when I create the device, so it's the
Add-Devicecall like my screenshot is showing.As you see the name has no special characters besides quotes & pipes.
lordmilko commentedon Aug 7, 2024
If you do
Set-PrtgClient -LogLevel Alland then dowhat does it respond with? And while it's timing out creating the device, if you look in the PRTG UI has the device been created?
Silex commentedon Aug 7, 2024
While it's timeing out I see the device created in PRTG, but PrtgAPI is still waiting for it.
I'll report what the "-LogLevel All" says soonish.
lordmilko commentedon Aug 7, 2024
The
-Verboseoutput will contain sensitive information, but the key thing we want to ascertain is to what extent the name in the URL it's querying for matches what you're specifying to the cmdlet, and whether the object is being included in the response and PrtgAPI is just ignoring it. I have a strong suspicion that the issue is related to your use of|pipe characters; I suspect PRTG or PowerShell is interpreting these in a special waySilex commentedon Aug 7, 2024
Ok, I found out it's the quotes and not the pipes. It used to work. I guess latest updates from PRTG broke something, the good news is that I can use any string to trigger it, will post traces soon.
Silex commentedon Aug 7, 2024
So, here are the trace of a working request and a failing request.
In both case, the device is created just fine, it's just the resolving of the new device that works in the first case and fails in the second case.
Here's the trace of a request working (name
Test):And here's the failing one (name
Test "Foobar"):Silex commentedon Aug 8, 2024
For the record I'm completely reinstalling PRTG because it behaves weirdly, will check if the bug is still happening in 24.3.98.1210
lordmilko commentedon Aug 8, 2024
Thanks @Silex,
On the basis that I'm escaping the quotes correctly, I feel like PrtgAPI is doing the right thing.
If you paste this URL into your web browser it will also give the XML response. You can then potentially fiddle around specifying different values to the
filter_nameparameter, which may enable you to tell Paessler what's wrong.Alternatively, if you find that names with single quotes work, potentially modifying your names to use single quotes instead of double quotes would work? This can easily be done with a one-liner with PrtgAPI
Silex commentedon Aug 8, 2024
Yeah, it looks like a PRTG regression.
Still what's weird is that PRTG creates the node with quotes just fine. And my workaround where I query the name using
Where-ObjectonGet-Devicein a loop also work just fine.So my guess is that the regression is subtle... like only
filter_namein the PRTG api regressed.Silex commentedon Aug 8, 2024
For the record single quotes don't have this problem.
Silex commentedon Aug 8, 2024
Anyway, about the original issue I think your fix works, but I think you'd not make it so the delay increases with each tries.
Even with the default values it's like it takes 3 seconds for the first try, then 6, then 9...
About the issue with the quotes, I'll reopen another case if needed.
lordmilko commentedon Aug 8, 2024
The purpose of the request retry feature is that in large PRTG installations (10,000 sensors) there can be quite a delay in attempting to perform certain actions, in particular creating new objects. Thus, when PrtgAPI repeatedly queries for the given object, and find again and again it hasn't been created yet, the assumption is that your sever must be under intense load; we want to do our best to guarantee that your request will succeed, hence why PrtgAPI will wait longer and longer and longer to maximize its chance at success.
The implication of this however is that you should not go too crazy modifying the default retry parameters; I believe the defaults might result in a total wait of around 30 seconds or so. The scaling behavior ensures that PrtgAPI's default retry rules will generally function well on both small and large PRTG installations, scaling up the delay as required for your system.
Silex commentedon Aug 8, 2024
Understood. I'd maybe consider renaming
RetryDelaytoRetryFactoror something that implies it's exponential.