Cleaning up critical logs for more efficient log monitoring

[jang00777]

Impact of the bug
As some modules were updated or data sources were migrated, some logs don't need to be on the critical log anymore

Describe the bug
Frequent but not much useful logs

e.g )

doccache
Failed to get gwmsmon_prod_maxused
No JSON object could be decoded

GQ
The workflow pdmvserv_task_TAU-RunIISummer20UL17wmLHEGEN-00010__v1_T_210115_190746_406 has been assigned for 60.61 days

How to reproduce it
Check the critical log page

Expected behavior
Clean up a piece of code related to these logs

Additional context and error message
@haozturk FYI

[haozturk]

Thanks for creating the issue @jang00777 My suggestion would be first to understand those logs before attempting to clean them. For instance, the second one is useful. It shows that this workflow in assigned state for too long which is true https://cmsweb.cern.ch/reqmgr2/fetch?rid=pdmvserv_task_TAU-RunIISummer20UL17wmLHEGEN-00010__v1_T_210115_190746_406 But its label is incorrect, it should be addHoc instead of GQ [1]

First log is related to this line for instance [2]. My suggestion would be first to investigate the use cases of this information that we get from gwmsmon. If we do not need it anymore, then, there is no need to keep a log of it.

[1] https://github.com/CMSCompOps/WmAgentScripts/blob/master/Unified/addHoc.py#L90
[2] https://github.com/CMSCompOps/WmAgentScripts/blob/master/utils.py#L1818

[jang00777]

Thanks for your quick feedback!

Yes, I agree. I created this issue as just a notification about what I started and I'm still learning and trying to understand what's going on, especially for gwmsmon (I'm checking these lines[1] and related things. is there anything else that I should check? I'll appreciate your advice!)

For the second one, I thought It would be not much useful because now we check those which stuck in assigned status from this page [2], but if you think it's still useful I can just change the label from GQ to addHoc.

[1]

WmAgentScripts/utils.py

Line 2320 in 9767ac3

def fetch_glidein_info(self, talk=True):

[2] https://dmytro.web.cern.ch/dmytro/cmsprodmon/stuck_workflows.php

[haozturk]

Speaking for the gwmsmon fields, we really need to check all the use cases and see if they are currently being used or not. For instance:

1. gwmsmon_totals is used to set sites_memory field of siteInfo class object. [1]
2. sites_memory is used within sitesByArch [2] and sitesByMemory [3] functions
3. sitesByArch function is called within sitesByArchs [4] function
4. sitesByMemory is called within restrictByMemory [5]. restrictByMemory has no usages!
5. sitesByArchs has no usages!
6. Therefore gwmsmon_totals has no usages in our code space and thus its logs and all relevant redundant function calls should be removed.

It would be great, if you can do a similar investigation for the other fields. If you find any usage, please let me know and we judge its importance.

[1] https://github.com/CMSCompOps/WmAgentScripts/blob/master/utils.py#L2321
[2] https://github.com/CMSCompOps/WmAgentScripts/blob/master/utils.py#L2358
[3] https://github.com/CMSCompOps/WmAgentScripts/blob/master/utils.py#L2390
[4] https://github.com/CMSCompOps/WmAgentScripts/blob/master/utils.py#L2377
[5] https://github.com/CMSCompOps/WmAgentScripts/blob/master/utils.py#L2401

For the second one, that's a good point. https://dmytro.web.cern.ch/dmytro/cmsprodmon/stuck_workflows.php is a perfect replacement for that functionality and thus we can remove it. But, ideally, I want to be able to reach all debugging tools from a single place rather than scattered links all over the place. Critical logs is good in that sense. One central place to reach everything. Perhaps, we should put a link to this from somewhere in critical logs.

[jang00777]

I checked others

For gwmsmon_prod_site_summary

1. gwmsmon_prod_maxused is used for updating sites_pressure field in fetch_glidein_info function [1] in siteInfo class object
2. sites_pressure is used in GQ [2] module and site_in_depletion function in mappor[3] module
3. But site_in_depletion always returns True ---> So no usage in this function
4. In GQ module [2], codes using site_pressure fill agents_down [4] and it seems that it is not used now [5]
5. Above all things, now GQ is in archive thus I think it's ok to remove that part

For gwmsmon_prod_maxused

1. gwmsmon_prod_maxused is used for updating cpu_pledges field in fetch_glidein_info function [6] in siteInfo class object
2. Basically fetch_glidein_info is called in when siteInfo object is constructed [7]
3. cpu_pledges is used in several modules [8] but these modules except for htmlor are in archive
4. In htmlor module, it is used for updating available_ratios and upcoming_ratios [10] but I'm not sure how important they are

[1] https://github.com/CMSCompOps/WmAgentScripts/blob/master/utils.py#L1521-L1536
[2] https://github.com/CMSCompOps/WmAgentScripts/blob/master/archive/Unified/GQ.py#L114
[3] https://github.com/CMSCompOps/WmAgentScripts/blob/master/Unified/mappor.py#L61
[4] https://github.com/CMSCompOps/WmAgentScripts/blob/master/archive/Unified/GQ.py#L122
[5] https://github.com/CMSCompOps/WmAgentScripts/blob/master/archive/Unified/GQ.py#L300-L304

[6] https://github.com/CMSCompOps/WmAgentScripts/blob/master/utils.py#L1511-L1519
[7] https://github.com/CMSCompOps/WmAgentScripts/blob/master/utils.py#L1488
[8] https://github.com/CMSCompOps/WmAgentScripts/search?q=cpu_pledges
[9]

WmAgentScripts/Unified/htmlor.py

Line 986 in a9cce7c

cpu = SI.cpu_pledges[site] if site in SI.cpu_pledges else 'N/A'

[10]

WmAgentScripts/Unified/htmlor.py

Lines 984 to 1013 in a9cce7c

	for site in sorted(getattr(SI,t)):
	site_se = SI.CE_to_SE(site)
	cpu = SI.cpu_pledges[site] if site in SI.cpu_pledges else 'N/A'
	disk = SI.disk[site_se] if site_se in SI.disk else 'N/A'
	if c==0:
	text+="<tr>"
	if not disk:
	ht_disk = '<a href=remaining_%s.html><font color=red>Disk available: %s</font></a>'%(SI.CE_to_SE(site),disk)
	else:
	ht_disk = 'Disk available: %s'%disk
	up_com = ""
	usage = for_max_running[site]['CpusInUse'] if site in for_max_running else 0
	ht_cpu = 'CPU current/max: %s / %s'%(usage,cpu)
	if site_se in upcoming:
	u = sum(upcoming[site_se][camp] for camp in upcoming[site_se])
	up_com += "<br><a href=GQ.json>Jobs available %d</a> (av./pl.=%.2f)"%(u, (u/float(cpu) if usage else 0))
	if usage: available_ratios[site] = u / float(cpu)
	## should there be an alarm here if a site is underdoing
	if (usage)<(0.8*cpu) and (u > usage):
	ht_cpu = '<font color=red>%s</font>'%(ht_cpu)
	for team in ['production','relval']:
	if site in upcoming_by_site[team]:
	if upcoming_by_site[team][site]:
	den = max(float(cpu),float(usage))
	if usage and team=='production' and upcoming_by_site[team][site] > den:
	upcoming_ratios[site] = upcoming_by_site[team][site] / den
	up_com +='<br>%s jobs upcoming %d'%(team, upcoming_by_site[team][site])

[jang00777]

There are 2 suggestions

1. How about automatically forgetting WFs in rejected-archived or aborted-archived status? I'm manually checking them and switching their status into forget status for the case of a known issue or a solved issue. Are there any problems that you expect when we automate this part [1] ?

2. Changing contents of sendLog in these lines [2], that is, collecting all WFs stuck in assigned and printing them at once like

There are ## workflows have been assigned for [min_time, max_time] days
link : https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?stuck_state=assigned&stuck_min=min_time&stuck_max=max_time

[1] https://github.com/CMSCompOps/WmAgentScripts/blob/master/Unified/injector.py#L178-L180
[2]

WmAgentScripts/Unified/addHoc.py

Lines 43 to 50 in f32478e

	for wf in wfs:
	assigned_log = filter(lambda change : change["Status"] in ["assigned"],wf['RequestTransition'])
	if assigned_log:
	then = assigned_log[-1]['UpdateTime']
	since = (now-then)/float(1*24*60*60.)
	if since>1.:
	print "workflow",wf['RequestName'],"is assigned since",then," that is",since,"days"
	sendLog('GQ','The workflow %s has been assigned for %.2f days'%(wf['RequestName'], since), level='critical')

[jang00777]

Re-checking for clean-up docCaches not used anymore

gwmsmon_totals

1. Whenever we call siteInfo object, fetch_glidein_info [1] is called and then gwmsmon_totals is used for updating site_memory and 'siteByMemory' function uses 'site_memory' [2]
2. There are no necessary usage of this function, [3] [4] [5]
   • For [3], the usage was removed from getting the white site list
   • For [4], memory_allowed derived from the function has no usage anywhere in the module
   • For [5], same as the above

gwmsmon_prod_maxused

1. It is used for updating cpu_pledges in fetch_glidein_info [6]
2. but cpu_pledges is updated from fetching SSB [7] before fetch_glidein_info is called [8], so it has to be removed

gwmsmon_prod_site_summary

1. It is used for updating sites_pressure in fetch_glidein_info [9]
2. As the GQ is deprecated, actually the only usage is in the mappor module [10]
3. In the mappor, it is used in site_in_depletion but the function always returns True [11]
4. site_in_depletion is used on this line [12] and considering the function is always True, sites_pressure is not used in fact thus can be removed

[1] https://github.com/CMSCompOps/WmAgentScripts/blob/python3-migration-fast/utils.py#L1535-L1538
[2] https://github.com/CMSCompOps/WmAgentScripts/blob/python3-migration-fast/utils.py#L1568-L1577
[3] #712
[4] https://github.com/CMSCompOps/WmAgentScripts/blob/python3-migration-fast/Unified/equalizor.py#L413
[5] https://github.com/CMSCompOps/WmAgentScripts/blob/python3-migration-fast/assign.py#L360-L372

[6] https://github.com/CMSCompOps/WmAgentScripts/blob/python3-migration-fast/utils.py#L1541-L1549
[7] https://github.com/CMSCompOps/WmAgentScripts/blob/python3-migration-fast/utils.py#L1628-L1669
[8] https://github.com/CMSCompOps/WmAgentScripts/blob/python3-migration-fast/utils.py#L1503

[9] https://github.com/CMSCompOps/WmAgentScripts/blob/python3-migration-fast/utils.py#L1551-L1566
[10] https://github.com/CMSCompOps/WmAgentScripts/search?q=sites_pressure
[11] https://github.com/CMSCompOps/WmAgentScripts/blob/python3-migration-fast/Unified/mappor.py#L59-L70
[12] https://github.com/CMSCompOps/WmAgentScripts/blob/python3-migration-fast/Unified/mappor.py#L75