deployment fixes

Author: 0xfaust

README.md

@@ -23,8 +23,8 @@ Make sure you have installed all of the following prerequisites on your developm
 1. Clone this repository to get all of the files necessary for building the project  
 ```$ git clone https://github.com/0xfaust/0xfaust.io```  
 2. Make copies of the Environment Variable files  
-```$ cd 0xfaust.io && cp .env.template .env && cp config/postgres/.env.template config/postgres/.env```
-3. Configure the ```.env``` and ```config/postgres/.env``` files for your environment.
+```$ cd 0xfaust.io && cp .env.template .env && cp config/datadog/.env.template config/datadog/.env && cp config/postgres/.env.template config/postgres/.env```
+3. Configure the ```.env```, ```config/datadog.env``` and ```config/postgres/.env``` files for your environment.
 4. Build and run all of the Docker Containers in detached mode.  
 ```$ docker-compose up -d --build```   
 5. The site can then be accessed via [localhost:8000](http://localhost:8000) in your browser.
@@ -38,7 +38,7 @@ Make sure you have installed all of the following prerequisites on your developm
 
 ## Deployment
 There are a few additional considerations and modifications that need to be made in order to host the project on a live system. Some of the considerations are outlined in Django's [Deployment Checklist](https://docs.djangoproject.com/en/3.0/howto/deployment/checklist/) and most of them have already been addressed using Environment Variables. Here are some additional steps to go from ```localhost``` to your server's IP.
-1. In ```config/nginx/conf.d/local.conf```, change ```localhost``` to the IP of your server.
+1. In ```config/nginx/nginx.conf```, change ```localhost``` to the IP of your server.
 2. In ```src/faust/settings.py``` change ```localhost``` to the list of IP address's and domains, e.g.  
 ```ALLOWED_HOSTS = ['xxx.xxx.xxx.xxx','0xfaust.io','www.0xfaust.io']```
 

config/datadog/.env.template

@@ -1,3 +1,6 @@
 DD_API_KEY=<DATADOG API KEY>
 DD_HOSTNAME=<HOSTNAME>
 DD_SITE=<DATADOG SITE (US OR EU)>
+DD_LOGS_ENABLED=<TRUE OR FALSE>
+DATADOG_USER=<DB USERNAME>
+DATADOG_PASSWORD=<DB PASS>

docker-compose.yaml

@@ -84,6 +84,7 @@ services:
       - DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true
       - DD_DOGSTATSD_SOCKET=/var/run/datadog/dsd.socket
       - DD_DOGSTATSD_ORIGIN_DETECTION=true
+      - DD_CONTAINER_EXCLUDE_LOGS="name:0xfaustio_keras"
     networks:
       - datadog_network
     ports:
@@ -95,8 +96,6 @@ services:
       - /var/socket/run/datadog:/var/run/datadog
       - ./config/datadog/keras.d/conf.yaml:/etc/datadog-agent/conf.d/keras.d/conf.yaml
       - keras_volume:/var/lib/keras/
-    labels:
-      com.datadoghq.ad.logs: '[{"source": "agent", "service": "datadog agent"}]'
     pid: host
     depends_on:
       - django
@@ -119,6 +118,8 @@ services:
     volumes:
       - keras_volume:/opt/services/keras/models
       - /var/socket/run/datadog:/var/run/datadog:ro
+    labels:
+      com.datadoghq.ad.logs: '[{"source": "keras", "service": "machine learning"}]'
     networks:
       - datadog_network
 

nn/Dockerfile

@@ -10,10 +10,10 @@ COPY requirements.txt /opt/services/keras
 RUN apt-get update -y
 RUN apt-get install -y python3-dev python3-pip build-essential
 RUN pip3 install --upgrade pip
-RUN pip3 install -r requirements.txt
+RUN pip3 install -r requirements.txt --no-cache-dir
 
 # copy project code
 COPY . /opt/services/keras
 
 # run training loop
-CMD ["python3", "./src/covnet.py"]
+CMD ["python3", "./src/covnet.py", "> /dev/null"]

nn/src/covnet.py

@@ -66,14 +66,14 @@ def on_epoch_end(self, epoch, accuracy, logs=None):
     statsd.event('Convolutional Netual Network Model Training Starting', 'Training starting with Batch Size {} for {} Epochs.'.format(epochs_train, batch_size_train), alert_type='info')
 
     with tracer.trace('model.fit'):
-        model.fit(train_images, train_labels, epochs=epochs_train, batch_size=batch_size_train, callbacks=[LogsAndMetricsCallback()])
+        model.fit(train_images, train_labels, epochs=epochs_train, batch_size=batch_size_train, callbacks=[LogsAndMetricsCallback()], verbose=0)
 
     with tracer.trace('model.evaluate'):
-        test_loss, test_acc = model.evaluate(test_images, test_labels)
+        test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=0)
 
     statsd.event('Convolutional Netual Network Model Training Finished', 'Training finished with Accuracy {} and Loss {}.'.format(test_acc, test_loss), alert_type='success')
 
-    print(test_acc)
+    #print(test_acc)
 
     statsd.gauge('keras.epoch', 0)
     statsd.gauge('keras.accuracy', 0)

requirements.txt

@@ -2,9 +2,4 @@ Django==3.0.7
 gunicorn==20.0.4
 psycopg2==2.8.5
 ddtrace==0.38.0
-tensorflow==2.2.0
-numpy==1.18.4
-scipy==1.4.1
-keras==2.3.1
-h5py==2.10.0
 datadog==0.36.0