instructions for dev setup

[steveharoz]

EDIT: I've updated these commands based on the advice below. It works with a new Digital Ocean instance of Ubuntu.

Update and upgrade the packages:

apt-get update
apt-get -y upgrade

Install nginx:
apt-get install -y nginx

Install supervisor:
apt-get install -y supervisor

Install python virtualenv:
apt-get install -y python-virtualenv

Install postgresql:
apt-get install -y postgresql postgresql-contrib

get project:
git clone https://github.com/steveharoz/curate_science.git

make curate_science/.env file:

DB_USER=sharoz
DB_PASS=mypassword
SECRET_KEY=somerandomkey

setup virtual environment

virtualenv curate_science --python=$(which python3.5)
source curate_science/bin/activate
cd curate_science

Switch to the postgres user:
sudo su - postgres

Type this to go to the postgres interactive shell:
psql

Create a database and set up user:

CREATE DATABASE curate;
CREATE USER [DB_USER] WITH PASSWORD '[DB_PASS]';
GRANT ALL PRIVILEGES ON DATABASE curate TO [DB_USER];
ALTER USER [DB_USER] CREATEDB;

Quit from the shell and switch back to the root user:

postgres=# \q
postgres@djtrump:~$ exit

Set up server:

pip install -r requirements.txt
python manage.py migrate

tools to generate html/js

apt install npm
npm install -g yarn
npm install webpack

(reopen terminal)

source curate_science/bin/activate
cd curate_science
yarn start
[type ctrl + c after it finishes]

start the server

source curate_science/bin/activate
cd curate_science
python manage.py runserver [::]:8000

Go to [your machine's ip]:8000

[mickaobrien]

@steveharoz It looks like you're not running the yarn start command in the curate_science directory. Try reopening the terminal, cd curate_science and then run yarn start.

[steveharoz]

Thanks. I'm still missing something. I got to the point where I can run the server and run yarn start. But I don't see the app when I check the site:

python manage.py runserver_plus

yarn start

Here's how it looks when I go to the machine's IP address in a browser:

I also tried [IP]:8000 (connection refused) and [IP]/admin (404 error).

Any pointers?

[xgui3783]

Connection refused sounds to me like you might be trying to access the server Django server from a different machine (or a docker container).

If that's the case, you might want to have Django listen on all interfaces (0.0.0.0:8000) rather than (127.0.0.1:8000)

Or setup a proxy and forward the traffic.

[steveharoz]

Thank you! Got it working with python manage.py runserver [::]:8000

I've updated the above procedure in case anyone else who's unfamiliar with the django stack needs help getting set up.

One last question: my copy of the site only has 3 articles. Is there a simple way to import the data from the main site? I don't really understand what to do with the DB migration instructions.

[eplebel]

sorry i just realized you might still be waiting for help with this. @alexkyllo any chance you have a moment to provide guidance to @steveharoz on how to migrate production DB content into his local copy?

also, i've now linked your more detailed setup instructions within our README file in case it can help others, thanks so much!

[alexkyllo]

I think some of the migrations I wrote do seed the database with some initial data, but perhaps if the functionality of the site has changed much, it may no longer be useful. Try applying the django migrations and see if that's sufficient.

If not, the easiest way to copy all production data into your local database is probably using django-admin dumpdata/loaddata. Run dumpdata on production, pipe the output to a file, and run loaddata on your local, being very careful not to do the converse, which would overwrite the production db with your local data.

Another possible way to log into the production database server, take a backup using pg_dump, and then restore that backup into your local database server. This is similarly dangerous and requires care.

[eplebel]

thanks @alexkyllo for your suggestions!

and as side note, i believe the catastrophic actions you describe would not be possible given @steveharoz doesn't have write access to the resources -- only could initiate a PR, right?

[alexkyllo]

Well, if what he needs is a copy of all data in the production database, then he will need shell access to the production server to get it, in which case he could do anything. Or, he needs someone with that access to do it for him and send him the data in a zip file or something.

[eplebel]

oh i see, thanks for the follow-up! sounds like safest option is to just provide him with a production DB data dump and then he can import it into his local instance. @steveharoz just let us know if this works with you.

[steveharoz]

Thanks for following up. Yeah, as @alexkyllo said, there were a couple entries in the initial DB, but they were too few and too simple to decently sample the capability of the site.

A DB dump would work. FYI: I'm doing some end-of-year catching up for the next week or two, so take your time.

[alexkyllo]

@steveharoz Would you prefer a Django (Python application level) data dump, or a PostgreSQL (database level) data dump? It would be the same data, just a different format and method for loading it.

[steveharoz]

I suppose with PostgreSQL, I could import it instead of "Create a database and set up user:" (see the post in this thread). Then I'd just need to add my own user afterwords.

[eplebel]

just thought of an idea that would potentially solve your problem AND benefit our own testing purposes. that is, migrate the entire current production DB into the staging DB so that we have more realistic data to test new changes/features with on our staging website.

of course making sure you don't do the converse (as you mentioned), though i assume we do have a backup of the production DB or some kind of mechanism through GCP to revert back to a previous state?

thoughts @alexkyllo ?

[alexkyllo]

We actually don't have any automated backup process in production yet, but we probably should. It's a little more involved since we're self-hosting the database on a VM (vs. using GCP's managed PostgreSQL as a service, which was expensive), but if you'd like, I can look into setting up a script to take a daily backup of the entire production DB and drop it in a file storage bucket. Then Steve or any other devs could just retrieve the latest backup from that bucket when they want to set up their dev environments. I believe I could also set up a retention policy to auto-delete old backups (at say, 30 days).

[eplebel]

I can look into setting up a script to take a daily backup of the entire production DB and drop it in a file storage bucket. Then Steve or any other devs could just retrieve the latest backup from that bucket when they want to set up their dev environments. I believe I could also set up a retention policy to auto-delete old backups (at say, 30 days).

yes, this seems pretty important, and sounds like a great plan. thanks so much for offering to do this (again i can pay you if you keep track of your hours). thanks!