Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add scraper for numfocus sponsored projects #3

Merged
merged 1 commit into from
May 23, 2023

Conversation

andrew
Copy link
Contributor

@andrew andrew commented May 22, 2023

I've added a scraper for NumFOCUS's sponsored projects.

I left the scraper code commented out as it looks like you had issues with nokogiri on GitHub pages, but I've added in a generated csv from the output of the code, feel free to uncomment and run locally to verify.

For projects with no website I've used the numfocus website page (https://numfocus.org/project/openfhe for example)

I wasn't sure on the wording on the numfocus subproject markdown page, feel free to change it.

Also didn't add https://numfocus.org/sponsored-projects/affiliated-projects, if you'd like that added as well let me know.

I also didn't commit the Gemfile.lock, although it'd be a good idea to add one (unless it's causing you issues with github pages)

@ShaneCurcuru
Copy link
Collaborator

Thanks, this is awesome! Yes, I haven't had time to figure out the cleanest way to get a zero maintenance environment - which I think will mean just sticking with whatever GH Pages currently supports. And ensuring that we can add enough Actions or the like to do linting on commits, have a periodic re-scraping that auto-generates a PR once a month, etc. Any tips or assistance in better setting up the environment appreciated (especially for someone as expert with this stuff as you!)

@andrew
Copy link
Contributor Author

andrew commented May 23, 2023

A GitHub action set up to run on a schedule (like cron) would be good for updating the csv files.

You could also use github actions to build the html site using this action: https://github.com/actions/jekyll-build-pages which would give a little more control over the environment that the site gets built in than regular github pages.

@ShaneCurcuru
Copy link
Collaborator

LOL, I keep thinking I can just make Actions work without actually understanding them, but no, you can't. Off to investigate, or to just hack a bad solution for the time being.
https://github.com/Punderthings/fossfoundation/actions/runs/5058854157/jobs/9079550741

@ShaneCurcuru ShaneCurcuru merged commit ce27745 into Punderthings:main May 23, 2023
@andrew
Copy link
Contributor Author

andrew commented May 24, 2023

@ShaneCurcuru I took a brief look at the error in that action and I think I can fix/rework it, will take a look later.

That jekyll action I suggested seems to use quite a lot of fixed versions of dependencies, I can see why it got you into dependency hell! An easier approach might just be to have an action that generates and commits the html onto a branch, which then github pages can deploy without needing to install any extra dependencies.

@andrew andrew deleted the numfocus branch February 15, 2024 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants