EVA Packaging Enabled (georgia-tech-db#222)

1. Create package for Eva 2. Config file will be automatically copied to the ~/.eva directory 3. Datasets will be stored also inside ~/.eva unless the user specifies a different location 4. FIle movements are done by calling the configuration manager. 5. CIrcleCI build uses package to perform testing. 6. Updated Readme with the new installation instructions and build from source instructions.
cs1234500000 · Jan 26, 2022 · 93433bc · 93433bc
1 parent b528883
commit 93433bc
Show file tree

Hide file tree

Showing 237 changed files with 1,304 additions and 930 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -2,37 +2,41 @@ version: 2.1
 
 jobs:
   build-and-test:
-    machine:
-      image: ubuntu-1604:202007-01
+    docker:
+      - image: cimg/python:3.7
+    resource_class: large
     steps:
-      - checkout          
+      - checkout
+      - restore_cache:
+          keys:
+            - v1-dependencies-{{ checksum "setup.cfg" }}
+            # fallback to using the latest cache if no exact match is found
+            - v1-dependencies-
+      - run:
+          name: Install Package
+          command: |
+            python -m venv env37
+            . env37/bin/activate
+            pip install --upgrade pip
+            python setup.py install 
 
       - run:
-          name: Before Install
+          name: Generate Parser Files
           command: |
-            sh script/install/before_install.sh
-          environment:
-            DEBIAN_FRONTEND: noninteractive
+            sudo add-apt-repository ppa:openjdk-r/ppa
+            sudo -E apt install -y openjdk-8-jdk openjdk-8-jre
+            sh script/antlr4/generate_parser.sh
 
       - run:
-          name: Install
-          command: | 
-            export PATH="$HOME/miniconda/bin:$PATH"
-            sh script/install/install.sh
-            # allow no-password connection for root
-            sudo mysql -u root -e "SELECT User,Host FROM mysql.user; 
-                                   DROP USER 'root'@'localhost'; 
-                                   CREATE USER 'root'@'%' IDENTIFIED BY ''; 
-                                   GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION; 
-                                   FLUSH PRIVILEGES;" 
+          name: Install Test Dependencies
+          command: |
+            . env37/bin/activate
+            pip install flake8==3.9.0 pytest==6.1.2 pytest-cov==2.11.1 mock==4.0.3 coveralls==3.0.1
 
       - run:
           name: Test
-          command: | 
-            export PATH="$HOME/miniconda/bin:$PATH"
-            conda init bash
-            source ~/.bashrc
-            conda activate eva
+          command: |
+            . env37/bin/activate
             sh script/test/test.sh
             coveralls
 

diff --git a/.coveragerc b/.coveragerc
@@ -1,2 +1,2 @@
 [run]
-omit = src/parser/evaql/*
+omit = eva/parser/evaql/*
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -9,10 +9,10 @@ repos:
     rev: v1.5.3
     hooks:
     -   id: autopep8
-        args: ['-i', '--select=E,F', '--exclude=src/filters, src/parser/evaql']
+        args: ['-i', '--select=E,F', '--exclude=eva/filters, eva/parser/evaql']
 -   repo: https://gitlab.com/pycqa/flake8
     rev: 3.8.3
     hooks:
     -   id: flake8
         additional_dependencies: [flake8-typing-imports==1.6.0]
-        args: ['--select=E,F', '--exclude=src/filters, src/parser/evaql']
+        args: ['--select=E,F', '--exclude=eva/filters, eva/parser/evaql']
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -0,0 +1,116 @@
+# Contributing to EVA
+
+## Setting up Development Environment
+
+### Installation
+
+Installation of EVA involves setting a virtual environment using [miniconda](https://conda.io/projects/conda/en/latest/user-guide/install/index.html) and configuring git hooks.
+
+1. Clone the repository
+```shell
+git clone https://github.com/georgia-tech-db/eva.git
+```
+
+2. Install the dependencies.
+```shell
+sh script/install/before_install.sh
+export PATH="$HOME/miniconda/bin:$PATH"
+sh script/install/install.sh
+```
+
+<!-- 4. Install `docker` and `docker-compose`.
+Please refer to [official doc](https://docs.docker.com/engine/install/). -->
+
+### Client Testing
+
+1. Set up the server and client
+
+- Activate the conda environment: `conda activate eva`
+
+- Launch EVA database Server: `python eva/eva_server.py`
+
+- Launch CLI: `python eva/eva_cmd_client.py`
+
+2. Run the `UPLOAD` command in the client terminal:
+```mysql
+UPLOAD INFILE 'data/ua_detrac/ua_detrac.mp4' PATH 'test_video.mp4';
+```
+
+3. Run the `LOAD` command in the client terminal: (may take a while)
+```mysql
+LOAD DATA INFILE 'test_video.mp4' INTO MyVideo;
+```
+
+4. Below is a basic query that should work on the client
+```mysql
+SELECT id, data FROM MyVideo WHERE id < 5;
+```
+
+### Configure GPU (Recommended)
+
+1. If your workstation has a GPU, you need to first set it up and configure it. You can run the following command first to check your hardware capabilities. 
+
+    ```
+    ubuntu-drivers devices
+    ```
+
+    If you do have an NVIDIA GPU, and its not been configured yet, follow all the steps in this link carefully. `https://towardsdatascience.com/deep-learning-gpu-installation-on-ubuntu-18-4-9b12230a1d31`. 
+
+    Some pointers:
+    - When installing NVIDIA drivers, check the correct driver version for your GPU to avoid compatibiility issues. 
+    - When installing cuDNN, you will have to create an account. Make sure you get the correct deb files for your OS and architecture. 
+
+2. You can run the following code in a jupyter instance to verify your GPU is working well along with PyTorch.
+
+    ```
+    import torch
+    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
+    print(device)
+    ```
+
+    Output of `cuda:0` indicates the presence of a GPU. (Note: 0 indicates the index of the GPU in system. Incase you have multiple GPUs, the index needs to be accordingly changed)
+
+2. Now configure the `executor` section in `eva/eva.yml` as follows:
+
+    ```
+    gpus: {'127.0.1.1': [0]}
+    ```
+    
+    `127.0.1.1` is the loopback address on which the eva server is started. 0 refers to the GPU index to be used. 
+
+## Commiting and Testing
+
+1. Install git hooks in your .git/ directory. [optional, but recommended]
+```shell
+conda activate eva
+pre-commit install
+```
+
+2. Ensure that all the unit test cases (including the ones you have added) run succesfully and the coding style conventions are followed.
+```shell
+bash script/test/test.sh
+```
+
+## Packaging New Version of EVA
+
+1. Generate EVA grammar files.
+```shell
+bash script/antlr4/generate_parser.sh
+```
+
+2. Bump up version number in `setup.cfg` along with any additional dependencies.
+
+3. Create a new build locally.
+```shell
+python -m build
+```
+
+4. Upload build to pypi using credentials.
+```shell
+python -m twine upload dist/*
+```
+
+
+## Issues and PR's
+
+To file a bug or request a feature, please file a GitHub issue. Pull requests are welcome.
diff --git a/README.md b/README.md
@@ -17,56 +17,40 @@ EVA is a visual data management system (think MySQL for videos). It supports a d
 
 * EVA **improves accuracy** by introducing state-of-the-art model specialization and selection algorithms.
 
-## Table of Contents
-* [Installation](#installation)
-* [Client Testing](#client-testing)
-* [Docker](#docker)
-* [Development](#development)
-* [Architecture](#architecture)
-
-
 ## Installation
 
-Installation of EVA involves setting a virtual environment using [miniconda](https://conda.io/projects/conda/en/latest/user-guide/install/index.html) and configuring git hooks.
+### Dependency
+EVA requires Python 3.7 or later and JAVA 8. On Ubuntu, you can install the JAVA by `sudo -E apt install -y openjdk-8-jdk openjdk-8-jre`.
 
-1. Clone the repository
+### Recommended
+To install EVA, we recommend using virtual environment and pip:
 ```shell
-git clone https://github.com/georgia-tech-db/eva.git
+python3 -m venv env37
+. env37/bin/activate
+pip install --upgrade pip
+pip install evatestdb
 ```
 
-2. Install the dependencies.
+### Install From Source
 ```shell
-sh script/install/before_install.sh
-export PATH="$HOME/miniconda/bin:$PATH"
-sh script/install/install.sh
-```
-
-3. Connect mysql user root with normal account and no password
-```mysql
-sudo mysql -u root
-> SELECT User,Host FROM mysql.user;
-> DROP USER 'root'@'localhost';
-> CREATE USER 'root'@'%' IDENTIFIED BY '';
-> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION;
-> FLUSH PRIVILEGES;
-mysql -u root
+git clone https://github.com/georgia-tech-db/eva.git && cd eva
+python3 -m venv env37
+. env37/bin/activate
+pip install --upgrade pip
+sh script/antlr4/generate_parser.sh
+pip install .
 ```
-refer to [askubuntu](https://askubuntu.com/questions/766334/cant-login-as-mysql-user-root-from-normal-user-account-in-ubuntu-16-04)
-
-<!-- 4. Install `docker` and `docker-compose`.
-Please refer to [official doc](https://docs.docker.com/engine/install/). -->
 
-## Client Testing
+## Verify Installation
 
 1. Set up the server and client
+- Activate the virtual environment: `. env37/bin/activate`
 
-- Activate the conda environment: `conda activate eva`
+- Launch EVA database Server: `eva_server`
 
-- Launch EVA database Server: `python eva.py`
+- Launch CLI: `eva_client`
 
-- Launch CLI: `python eva_cmd_client.py`
-
-2. Run the `UPLOAD` command in the client terminal:
+2. Run the `UPLOAD` command in the client terminal (use the [ua_detrac.mp4](data/ua_detrac/ua_detrac.mp4) as an example):
 ```mysql
 UPLOAD INFILE 'data/ua_detrac/ua_detrac.mp4' PATH 'test_video.mp4';
 ```
@@ -81,28 +65,6 @@ LOAD DATA INFILE 'test_video.mp4' INTO MyVideo;
 SELECT id, data FROM MyVideo WHERE id < 5;
 ```
 
-
-
-<!-- ## Docker
-
-1. Standup EVA testing for CPU/GPU hardware.
-```shell
-docker-compose -f docker-compose.yml eva-test-[cpu/gpu] up 
-``` -->
-
-## Development
-
-1. Install git hooks in your .git/ directory. [optional, but recommended]
-```shell
-conda activate eva
-pre-commit install
-```
-
-2. Ensure that all the unit test cases (including the ones you have added) run succesfully and the coding style conventions are followed.
-```shell
-bash script/test/test.sh
-```
-
 ## Quickstart Tutorial
 
 ### Configure GPU (Recommended)
@@ -129,7 +91,7 @@ bash script/test/test.sh
 
     Output of `cuda:0` indicates the presence of a GPU. (Note: 0 indicates the index of the GPU in system. Incase you have multiple GPUs, the index needs to be accordingly changed)
 
-2. Now configure the `executor` section in `eva.yml` as follows:
+2. Now configure the `executor` section in `~/.eva/eva.yml` as follows:
 
     ```
     gpus: {'127.0.1.1': [0]}
@@ -141,28 +103,26 @@ bash script/test/test.sh
 
 1. Open a terminal instance and start the server:
     ```
-    python eva.py
+    eva_server
     ```
 
-2. Open another terminal instance. Start a jupyter lab/notebook instance, and navigate to `tutorials/object_detection.ipynb`
+2. Open another terminal instance. Start a jupyter lab/notebook instance, and navigate to [tutorials/object_detection.ipynb](tutorials/object_detection.ipynb)
 
 3. You might have to install ipywidgets to visualize the input video and output. Follow steps in `https://ipywidgets.readthedocs.io/en/latest/user_install.html` as per your jupyter environment. 
 
-4. Run each cell one by one. Each cell is self-explanatory. If everything has been configured correctly you should be able to see a ipywidgets Video instance with the bounding boxes output of the executed query.  
+4. Run each cell one by one. Each cell is self-explanatory. If everything has been configured correctly you should be able to see a ipywidgets Video instance with the bounding boxes output of the executed query.
 
-## Architecture
+## Documentation
 
-EVA consists of four core components:
-
-* EVAQL Query Parser
-* Query Optimizer
-* Query Execution Engine (Filters + Deep Learning Models)
-* Distributed Storage Engine
+You can find documentation and code snippets for EVA [here](https://evagatech.readthedocs.io/).
 
 ## Contributing
 
 To file a bug or request a feature, please file a GitHub issue. Pull requests are welcome.
 
+For information on installing from source and contributing to EVA, see our
+[contributing guidelines](./CONTRIBUTING.md).
+
 ## Contributors
 
 See the [people page](https://github.com/georgia-tech-db/eva/graphs/contributors) for the full listing of contributors.

diff --git a/api-docs/guide/getstarted.rst b/api-docs/guide/getstarted.rst
@@ -5,29 +5,32 @@ Get Started
 
 Installation
 -------
-If you haven't installed EVA yet, you can do so by following instructions `given here <https://github.com/georgia-tech-db/eva#installation>`_
+EVA requires Python 3.7 or later. To install EVA, we recommend using pip::
+
+    pip install evadb
+
 
 Running EVA Server
 -------
-EVA runs as a client-server architecture. So you need to start the server before you can connect to it via various interfaces::
+EVA runs as a client server architecture. So you need to start the server before you can connect to it via various interfaces:::
 
-    python eva.py
+    eva_server
 
 Querying EVA
 -------
 
 - Querying Via Command Line::
 
-You can use the CLI of EVA to do some quick testing or verifying if the installation was succesful::
+EVA offers a CLI interface to query the server for quick testing and debugging::
 
-    python eva_cmd_client.py
+    eva_client
     >>> UPLOAD INFILE 'data/ua_detrac/ua_detrac.mp4' PATH 'test_video.mp4';
     >>> LOAD DATA INFILE 'test_video.mp4' INTO MyVideo;
     >>> SELECT id, data FROM MyVideo WHERE id < 5
 
-- Experimenting With Jupyter Notebook::
+- From Python/Jupyter Code::
 
-You can connect to the EVA server via a Jupyter Notebook as well, just copy paste the following snipped::
+EVA also provides an API to connect to the server using Python code::
 
     from src.server.db_api import connect
     import nest_asyncio
@@ -39,3 +42,5 @@ Once the connection is established, you can run queries using the cursor::
 
     cursor.execute("""SELECT id, Unnest(FastRCNNObjectDetector(data)) FROM MyVideo""")
     response = cursor.fetch_all()
+
+- A sample jupyter notebook the performs object detection using EVA can be found `here <https://github.com/georgia-tech-db/eva/blob/master/tutorials/object_detection.ipynb>`_.