Add non legacy builders (<10.9) #119

arjunsalyan · 2019-09-25T14:41:37Z

I will soon be doing the table related changes, but till then it would be better to not miss on the build history for the new builders.

#117

arjunsalyan · 2019-09-25T14:43:02Z

I have applied this to the test site:

http://ec2-52-34-234-111.us-west-2.compute.amazonaws.com/ports/all_builds/

codecov · 2019-09-25T15:02:42Z

Codecov Report

Merging #119 into master will increase coverage by 0.42%.
The diff coverage is 93.1%.

@@            Coverage Diff             @@
##           master     #119      +/-   ##
==========================================
+ Coverage   73.91%   74.34%   +0.42%     
==========================================
  Files          46       48       +2     
  Lines        1342     1364      +22     
==========================================
+ Hits          992     1014      +22     
  Misses        350      350

Impacted Files	Coverage Δ
app/ports/migrations/0011_populate_display_name.py	`100% <100%> (ø)`
app/ports/filters.py	`100% <100%> (ø)`	⬆️
app/ports/models/buildhistory.py	`21.42% <100%> (+0.81%)`	⬆️
app/ports/migrations/0010_builder_display_name.py	`100% <100%> (ø)`
app/ports/views.py	`57.75% <66.66%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cbf3672...f0a2e47. Read the comment docs.

mojca · 2019-09-25T21:06:38Z

I would ideally like to see just one instance of 10.6 on port health status, and just one entry in the filter.

arjunsalyan · 2019-09-26T16:46:29Z

I would ideally like to see just one instance of 10.6 on port health status, and just one entry in the filter.

I am working on the new table design, shall I simply hardcode the os_arch_key and display_name columns?

arjunsalyan · 2019-09-27T22:03:51Z

This PR is not in the stage of merging right now, as it will cause migration conflicts with #126 .

arjunsalyan · 2019-10-10T13:55:17Z

I have deployed this on the test site. I am utilising only one additional field display_name.

Does it seem right now? @mojca
http://ec2-52-34-234-111.us-west-2.compute.amazonaws.com

mojca · 2019-10-10T14:19:29Z

Can you please try to import at least a few builds (even if just ten builds per worker)? I don't yet see any builds shown after the 22nd of August to be able to see how it works.

arjunsalyan · 2019-10-10T14:25:52Z

Can you please try to import at least a few builds (even if just ten builds per worker)? I don't yet see any builds shown after the 22nd of August to be able to see how it works.

I have started the process, the fetching should finish within next 10-15 minutes.

arjunsalyan · 2019-10-10T14:28:59Z

Also, please do a hard refresh in the browser so that the changes to the javascript get reflected.

mojca · 2019-10-10T18:48:12Z

Can you please point me to some actual builds?

arjunsalyan · 2019-10-10T19:56:18Z

Can you please point me to some actual builds?

On the page: http://ec2-52-34-234-111.us-west-2.compute.amazonaws.com/ports/all_builds/ select builder 10.8.
Now, the first 20 builds refer to the builder 10.8_x86_64 (2285 to 2304) while the builds below it refer to the builder 10.8_x86_64_legacy.

The app is initially set to fetch only 20 builds for those builders which do not have any builds in the database, that is why we only get the latest 20 builds for the newly added 10.8_x86_64. We will need to decide this number.

And I also missed adding non-legacy builder for 10.6_i386.

mojca · 2019-10-10T20:10:43Z

Perfect, thank you. They are there now, but I didn't see them earlier.

The all_builds works precisely as desired. The port health on the other hand is picking up the legacy workers, maybe because you sorted by build number (but I didn't check).

That said, the port health has other flaws, like: the user doesn't see if the worker is offline, or the latest build is still pending (10.5/ppc has been offline for ages and port health shows just whatever was built last); I would attempt to fix this with buildbot 2.

Fetching just the latest 20 builds sounds fine. But we would need to distinguish between two different scenarios:

users or temporary servers who just want to test a new feature (we don't want them to fetch all builds, at least not automatically)
production server where we really want to fetch all builds (and we need to try to do this before they go "out of focus" and get deleted)

I'm ok with merging this now and fixing the issue with port health immediately after that.

mojca

Two hopefully minor changes to fix, and then we can merge it.

And then separately think of the best way to fetch all builds just on production machine.

mojca · 2019-10-10T20:15:47Z

app/ports/views.py

+        all_latest_builds = BuildHistory.objects.all().order_by('port_name', 'builder_name__display_name', '-build_id').distinct('port_name', 'builder_name__display_name')
+        port_latest_builds = list(BuildHistory.objects.filter(id__in=Subquery(all_latest_builds.values('id')), port_name__iexact=name).values('builder_name__name', 'builder_name__display_name', 'build_id', 'status'))


We can no longer sort by build id. Maybe by time_start?

Yes, thank you.

app/ports/migrations/0011_populate_display_name.py

arjunsalyan · 2019-10-20T18:00:14Z

This is probably ready to be merged now? I am sorry for the delay.

mojca · 2019-10-20T18:16:32Z

Cool, perfect.

Now we just need a way to fetch the old builds (fetch them all on the main server without having to fetch them on some test instances).

mojca · 2019-10-20T18:20:30Z

Ok, we are still missing a few fixes. Looking at http://ec2-52-34-234-111.us-west-2.compute.amazonaws.com/port/qt59-qtserialport/, the builds are still picking the wrong builder. It works correctly if you switch to a different tab.

It might make sense to clean up the code a bit, but for now at least we should fix that bug.

arjunsalyan · 2019-10-20T18:24:12Z

Ok, we are still missing a few fixes. Looking at http://ec2-52-34-234-111.us-west-2.compute.amazonaws.com/port/qt59-qtserialport/, the builds are still picking the wrong builder. It works correctly if you switch to a different tab.

I am sorry, could you please specify it more? Also, the recent changes (made today) have not been deployed to the test site due to migration conflicts.

mojca · 2019-10-20T18:58:49Z

I am sorry, could you please specify it more?

Hmmm ... I can no longer reproduce what I saw. The most likely explanation is that I was accidentally looking/clicking at the 10.7 builder which still points to the legacy build. It might have to do with me travelling on a bumpy road while trying to concentrate on the screen :) :) :)

arjunsalyan requested a review from mojca September 25, 2019 14:41

Add display_name field to Builder Model

79f238d

arjunsalyan force-pushed the new-builders branch from 97edd6b to 5a1af86 Compare October 9, 2019 15:04

arjunsalyan mentioned this pull request Oct 10, 2019

Allow filtering out deleted ports and loop over all ports to identify deleted ports #122

Closed

mojca approved these changes Oct 10, 2019

View reviewed changes

arjunsalyan added 2 commits October 20, 2019 23:19

Add new builders and populate display_name field

a9ea00f

Use display_name globally in filters and port health

f0a2e47

arjunsalyan force-pushed the new-builders branch from 5a1af86 to f0a2e47 Compare October 20, 2019 17:55

arjunsalyan mentioned this pull request Oct 20, 2019

Add function to loop over all ports to identify deleted ports #128

Closed

mojca merged commit 95392c5 into macports:master Oct 20, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add non legacy builders (<10.9) #119

Add non legacy builders (<10.9) #119

arjunsalyan commented Sep 25, 2019 •

edited

Loading

arjunsalyan commented Sep 25, 2019

codecov bot commented Sep 25, 2019 •

edited

Loading

mojca commented Sep 25, 2019

arjunsalyan commented Sep 26, 2019

arjunsalyan commented Sep 27, 2019

arjunsalyan commented Oct 10, 2019

mojca commented Oct 10, 2019

arjunsalyan commented Oct 10, 2019

arjunsalyan commented Oct 10, 2019

mojca commented Oct 10, 2019

arjunsalyan commented Oct 10, 2019 •

edited

Loading

mojca commented Oct 10, 2019

mojca left a comment

mojca Oct 10, 2019

arjunsalyan Oct 20, 2019

arjunsalyan commented Oct 20, 2019

mojca commented Oct 20, 2019

mojca commented Oct 20, 2019

arjunsalyan commented Oct 20, 2019

mojca commented Oct 20, 2019

		all_latest_builds = BuildHistory.objects.all().order_by('port_name', 'builder_name__display_name', '-build_id').distinct('port_name', 'builder_name__display_name')
		port_latest_builds = list(BuildHistory.objects.filter(id__in=Subquery(all_latest_builds.values('id')), port_name__iexact=name).values('builder_name__name', 'builder_name__display_name', 'build_id', 'status'))

Add non legacy builders (<10.9) #119

Add non legacy builders (<10.9) #119

Conversation

arjunsalyan commented Sep 25, 2019 • edited Loading

arjunsalyan commented Sep 25, 2019

codecov bot commented Sep 25, 2019 • edited Loading

Codecov Report

mojca commented Sep 25, 2019

arjunsalyan commented Sep 26, 2019

arjunsalyan commented Sep 27, 2019

arjunsalyan commented Oct 10, 2019

mojca commented Oct 10, 2019

arjunsalyan commented Oct 10, 2019

arjunsalyan commented Oct 10, 2019

mojca commented Oct 10, 2019

arjunsalyan commented Oct 10, 2019 • edited Loading

mojca commented Oct 10, 2019

mojca left a comment

Choose a reason for hiding this comment

mojca Oct 10, 2019

Choose a reason for hiding this comment

arjunsalyan Oct 20, 2019

Choose a reason for hiding this comment

arjunsalyan commented Oct 20, 2019

mojca commented Oct 20, 2019

mojca commented Oct 20, 2019

arjunsalyan commented Oct 20, 2019

mojca commented Oct 20, 2019

arjunsalyan commented Sep 25, 2019 •

edited

Loading

codecov bot commented Sep 25, 2019 •

edited

Loading

arjunsalyan commented Oct 10, 2019 •

edited

Loading