First Commit

Author: xLaszlo

.gitignore

@@ -0,0 +1,5 @@
+.venv/
+.vscode/
+.idea/
+__pycache__
+__pycache__/*

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) <year> <real name>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,127 @@
+# Example
+
+This project was generated with the Hypergol framework
+
+Please see documentation for instructions: [https://hypergol.readthedocs.io/en/latest/](https://hypergol.readthedocs.io/en/latest/)
+
+### Initialise git
+
+Hypergol is heavily integrated with git, all projects must be in a git repository to ensure code and data lineage (to record which data was created by which version of the code).
+
+Initialise git with:
+
+```git init .```
+
+Create the first commit (datasets record the last commit when they are created and without this there is nothing to record):
+
+```git commit -m "First Commit!"```
+
+The project now (and any time a file is changed but the change is not committed to the repo) is in a "dirty" stage. If you run a pipeline or train a model, the last commit will be recorded but that commit will not represent the code that is running! Add changes and commit:
+
+```
+git add .
+git commit -m "All the files!"
+```
+
+If there are files that shouldn't be checked in ever to git they should be to the `.gitignore` file before `git add .`
+
+Alternatively individual files can be added to git with `git add <filename>`.
+
+### Make the virtual environment
+
+Having dedicated virtual environment fully described by the projects `requirements.txt` is the recommended practice. Don't forget to `deactivate` the current virtual environment! Files from the environment are included in the projects `.gitignore` file and will ignored by git.
+
+```
+deactivate
+./make_venv.sh
+source .venv/bin/activate
+```
+
+
+### How to list existing Datasets (in Jupyter)
+
+```
+sys.path.insert(0, '<project_directory>/example')
+from hypergol import HypergolProject
+from data_models.example_datamodel_class import ExampleDatamodelClass
+project = HypergolProject(
+    projectDirectory='<project_directory>/example',
+    dataDirectory='<data_directory>'
+)
+ds = project.datasetFactory.get(dataType=ExampleDatamodelClass, name='sentences')
+# project.list_datasets(pattern='.*', asCode=True);
+```
+
+This will list all existing datasets that matches `pattern` as self contained executable code.
+
+
+### How to start Tensorboard
+
+It is recommended to start it in a screen session (`screen -S tensorboard`) so you can close the terminal window or if you disconnect from a remote Linux machine (reconnect with `screen -x tensorboard`). In the project directory:
+
+```
+screen -S tensorboard
+source .venv/bin/activate
+tensorboard --logdir=<data_directory>/example/tensorboard/
+```
+
+
+### How to train your model
+
+After implementing all components and required functions:
+
+```
+./train_example.sh
+```
+
+This will execute the model manager's run() function with the prescribed schedule (training steps, evaluation steps, etc.). Training can be stopped with Ctrl-C, this will won't result in the corruption of the output dataset (datasets must be closed properly to generate their chk file after they are read only). This is possible because the entire training happen in a `try/finally` block.
+
+### How to serve your model
+
+In the generated `models/serve_example.py` function specify the directory of the model to be served at:
+
+```
+MODEL_DIRECTORY = '<data_directory>/example/<branch>/models/<ModelName>/<epoch_number>'
+```
+
+then start serving with (port and host can be set in the shell script):
+
+```
+./serve_example.sh
+```
+
+
+### How to call your model from python with requests
+
+```
+import requests
+response = json.loads(requests.get('http://0.0.0.0:8000', headers={'accept': 'application/json'}).text)
+modelLongName = response['model']
+```
+
+This allows to verify if indeed the intended model is served. The generated training script sets training day and the commit hash at that point to be part of the long name and to ensure that the exact conditions of training are available at serving. Long name should be used in logging to identify which model created an output. From v0.0.10 the long name is returned in the header of the response of `/output` endpoint as well in the `x-model-long-name` field.
+
+To get the response of the model to a list of objects, see example below. Replace `ExampleOutput` with the correct output type and load a dataset into `ds`, use `list_datasets` from above to do this.
+
+```
+sys.path.insert(0, '<project_directory>/example')
+import requests
+from itertools import islice
+from data_models.example_model_output import ExampleModelOutput
+
+with ds.open('r') as dsr:
+    values = [value.to_data() for value in islice(dsr, 10)]
+
+response = requests.post(
+    'http://0.0.0.0:8000/output',
+    headers={
+        'accept': 'application/json',
+        'Content-Type': 'application/json',
+    },
+    data=json.dumps(values)
+)
+outputs = [ExampleModelOutput.from_data(v) for v in json.loads(response.text)]
+modelLongName = response.headers['x-model-long-name']
+```
+
+It is not recommended to do large scale evaluation through the API as the overhead per object is too high and it is single threaded.

data_models/__init__.py

@@ -0,0 +1,32 @@
+from typing import List
+from datetime import datetime
+
+from hypergol import BaseData
+
+from data_models.sentence import Sentence
+
+
+class Article(BaseData):
+
+    def __init__(self, articleId: int, url: str, title: str, text: str, publishDate: datetime, sentences: List[Sentence]):
+        self.articleId = articleId
+        self.url = url
+        self.title = title
+        self.text = text
+        self.publishDate = publishDate
+        self.sentences = sentences
+
+    def get_id(self):
+        return (self.articleId, )
+
+    def to_data(self):
+        data = self.__dict__.copy()
+        data['publishDate'] = data['publishDate'].isoformat()
+        data['sentences'] = [v.to_data() for v in data['sentences']]
+        return data
+
+    @classmethod
+    def from_data(cls, data):
+        data['publishDate'] = datetime.fromisoformat(data['publishDate'])
+        data['sentences'] = [Sentence.from_data(v) for v in data['sentences']]
+        return cls(**data)

data_models/article.py

@@ -0,0 +1,12 @@
+from hypergol import BaseData
+
+
+class ArticlePage(BaseData):
+
+    def __init__(self, articlePageId: int, url: str, body: str):
+        self.articlePageId = articlePageId
+        self.url = url
+        self.body = body
+
+    def get_id(self):
+        return (self.articlePageId, )

data_models/article_page.py

@@ -0,0 +1,26 @@
+from datetime import datetime
+
+from hypergol import BaseData
+
+
+class ArticleText(BaseData):
+
+    def __init__(self, articleTextId: int, publishDate: datetime, title: str, text: str, url: str):
+        self.articleTextId = articleTextId
+        self.publishDate = publishDate
+        self.title = title
+        self.text = text
+        self.url = url
+
+    def get_id(self):
+        return (self.articleTextId, )
+
+    def to_data(self):
+        data = self.__dict__.copy()
+        data['publishDate'] = data['publishDate'].isoformat()
+        return data
+
+    @classmethod
+    def from_data(cls, data):
+        data['publishDate'] = datetime.fromisoformat(data['publishDate'])
+        return cls(**data)

data_models/article_text.py

@@ -0,0 +1,28 @@
+from hypergol import BaseData
+
+
+class EvaluationOutput(BaseData):
+
+    def __init__(self, articleId: int, sentenceId: int, inputs: object, outputs: object, targets: object):
+        self.articleId = articleId
+        self.sentenceId = sentenceId
+        self.inputs = inputs
+        self.outputs = outputs
+        self.targets = targets
+
+    def get_id(self):
+        return (self.articleId, self.sentenceId, )
+
+    def to_data(self):
+        data = self.__dict__.copy()
+        data['inputs'] = BaseData.to_string(data['inputs'])
+        data['outputs'] = BaseData.to_string(data['outputs'])
+        data['targets'] = BaseData.to_string(data['targets'])
+        return data
+
+    @classmethod
+    def from_data(cls, data):
+        data['inputs'] = BaseData.from_string(data['inputs'])
+        data['outputs'] = BaseData.from_string(data['outputs'])
+        data['targets'] = BaseData.from_string(data['targets'])
+        return cls(**data)

data_models/evaluation_output.py

@@ -0,0 +1,12 @@
+from hypergol import BaseData
+
+
+class LabelledArticle(BaseData):
+
+    def __init__(self, labelledArticleId: int, articleId: int, labelId: int):
+        self.labelledArticleId = labelledArticleId
+        self.articleId = articleId
+        self.labelId = labelId
+
+    def get_id(self):
+        return (self.labelledArticleId, )

data_models/labelled_article.py

@@ -0,0 +1,14 @@
+from typing import List
+
+from hypergol import BaseData
+
+
+class ModelOutput(BaseData):
+
+    def __init__(self, articleId: int, sentenceId: int, posTags: List[str]):
+        self.articleId = articleId
+        self.sentenceId = sentenceId
+        self.posTags = posTags
+
+    def get_id(self):
+        return (self.articleId, self.sentenceId, )