Initial Release #1
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Initial Django project created by `django-admin startproject <project_name>`
I suppose this file is required by django's runtime and thus should not be added to SVC
Add very simple (failing) test file along with .gitlab-ci.yml configuration file which initially aims only to perform tests in CI env
First install Django before executing tests
Introduces django rest framework dependency. First implementation returns only mock data.
Author `url` field had been renamed to `profile_url` but "url" name was still being used in AuthorSerializer's `field`.
This would lead to (misleading) error bellow during tests:
AssertionError: `HyperlinkedIdentityField` requires the request in the serializer context. Add `context={'request': request}` when instantiating the serializer.
Add test cases and also removed no longer used `views.articles` and `HttpRequestTest`
Adjust tests to import data from fixture file `./fixtures/articles_and_authors.json` instead of using mock values defined at `models.authors` and `models.articles`. Adjust `.gitlab-ci.yml` file in order to require postgres service, configure its variables, and install `libpq-dev` system package which is required by `libpq-dev` python package. Note, at first I thought Django's `migration` were not meant to be added to VCS. However, according to [Django's documentation](https://docs.djangoproject.com/en/1.7/topics/migrations/) it seems it is: > The migration files for each app live in a “migrations” directory inside of that app, and are designed to be committed to, and distributed as part of, its codebase. You should be making them once on your development machine and then running the same migrations on your colleagues’ machines, your staging machines, and eventually your production machines.
This file aims to allow user to define local variable while keeping default values used by CI environment untouched. Based on http://stackoverflow.com/questions/1626326/how-to-manage-local-vs-production-settings-in-django/1629770#1629770.
Avoid error with message *'database "web_scraper_test" already exists'* in CI environment by setting `--noinput` argument to test command. Setting `--noinput` to a non-interactive environment seems a reasonable action anyway
Removed no longer used mock values defined at `models.authors` and `models.articles`. Those values are no longer used since 0fa0d2f.
Restruct project layout into a more "django-like" project as presented in [The Django Book](http://djangobook.com/), with an extra "app" for model classes, here called "articles" under `~/articles` folder.
First implementation retrieves information only about Author and Articles from TechCrunch site. Implemented Spider using [scapy](https://github.com/scrapy/scrapy) library. Intial implementation highly based on [scrapy's documentation page](https://doc.scrapy.org/en/latest/intro/tutorial.html) and integration scrapy and ["Django implemented based on "Scraping a website using Scrapy and Django" post](http://marcela-campo.blogspot.com.br/2015/03/scraping-website-using-scrapy-and-django.html). Spider maybe be triggered by executing `scrapy crawl tech_crunch` command under `~/scraper` folder.
Aims to avoid [build failure](https://gitlab.com/fcv/simple-web-scraper/builds/4919500) with message like: $ gem install dpl /bin/bash: line 53: gem: command not found ERROR: Build failed: exit code 1
Aims to fix deploy build failure with message "gem: command not found" by setting job's image to ruby. Remove command `apt-get install -y gem` introduced by last commit since it failed to fix the build. See failed build [#4919642](https://gitlab.com/fcv/simple-web-scraper/builds/4919642).
Add Heroku's Procfile, see more at https://devcenter.heroku.com/articles/procfile.
Output generated by it is quite misleading and may lead people to understand it as an error. Also set setting's DEBUG to True aiming to track what is causing Bad Request (400) in Heroku env.
…onment variable when available Loads Database connection configuration from `DATABASE_URL` environment variable when it is present. Approach used to easy configuration in Heroku environment. See https://devcenter.heroku.com/articles/django-app-configuration See https://pypi.org/project/dj-database-url/, https://github.com/kennethreitz/dj-database-url
Add information regarding virtual env, dependencies install
PostgreSQL is required only by test job and not for other jobs. Thus moving its service definition from global level to test job level.
…oku env. It seems that database migrate script is not automatically triggered at Heroku environment at deployment time. Add command to trigger it according this Stackoverflow answer: http://stackoverflow.com/questions/36410107/running-manage-py-migrate-during-heroku-deployment/39305091#39305091
Add handling of invalid value and type when parsing a datetime, fallback to None
Allow different configuration of static folder by setting env variable "EXTERNAL_STATIC_FOLDER" This change aims to allow use of `collectstatic` in Acc / Prod environment (currently using Heroku). Implementation based on https://devcenter.heroku.com/articles/django-assets
'/static' is just an empty folder where static files will be copied to upon `$ python manage.py collectstatic --noinput`. This command is executed at Heroku environment. See: https://devcenter.heroku.com/articles/django-assets
Introduce WhiteNoise, used to serve static file at Acc / Prod environment. see https://devcenter.heroku.com/articles/django-assets
…eroku env.
Avoid error like one bellow:
...
File "/app/.heroku/python/lib/python2.7/site-packages/whitenoise/django.py", line 12, in <module>
"'DJANGO_SETTINGS_MODULE' environment variable must be set "
Try to trigger scrapy's crawler by using `release` directive since `worker` seems to have no effect.
Add `authors` and `tags` properties to `Article` and `social_medias` to `Author`.
Note, its value is set to True at untracked `local_settings.py` file.
Fix links to repository's `settings.py` and `gitlab-ci.yml` files
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.