Quickly and easily store data about your open source projects on GitHub and various Package Managers.
All updates to this project are documented in our CHANGELOG.
First, get your free SendGrid account here.
Next, update your environment with your SENDGRID_API_KEY.
git clone https://github.com/sendgrid/open-source-library-data-collector.git
cd open-source-library-data-collector
virtualenv venv
cp .env_sample .envEnvironment Variables
echo "export SENDGRID_API_KEY='YOUR_API_KEY'" > sendgrid.env
echo "sendgrid.env" >> .gitignore
source ./sendgrid.envUpdate your settings in .env
mysql -u USERNAME -p -e "CREATE DATABASE IF NOT EXISTS open_source_external_library_data";
mysql -u USERNAME -p open_source_external_library_data < db/data_schema.sql
cp config_sample.yml config.ymlUpdate the settings in config.yml
source venv/bin/activate
pip install -r requirements.txtUpdate the code in package_managers.py. The functions update_package_manager_data and update_db were customized for our particular needs. You will want to either subclass those functions in your own application or modify it to suit your needs. We will remove these customizations in a future release. Here is the GitHub issue for reference.
To run:
source venv/bin/activate
python app.py
- The SendGrid Service, starting at the free level
- virtualenv
- mysql
heroku login
heroku create
heroku addons:create cleardb:ignite
Access the cleardb DB and create the tables in db/data_schema.sql
heroku config:add ENV=prod
heroku config:add GITHUB_TOKEN=<<your_github_token>>
heroku config:add SENDGRID_API_KEY=<<your_sendgrid_api_key>>
heroku addons:create scheduler:standard
Configure the schedular addon in your Heroku dashboard to run python app.py at your desired frequency.
Test by running heroku run worker
If you are interested in the future direction of this project, please take a look at our milestones. We would love to hear your feedback.
We encourage contribution to our projects, please see our CONTRIBUTING guide for details.
Quick links:
open-source-library-data-collector is guided and supported by the SendGrid Developer Experience Team.
open-source-library-data-collector is maintained and funded by SendGrid, Inc. The names and logos for open-source-library-data-collector are trademarks of SendGrid, Inc.
