Release 1.0.1

README.md

@@ -18,7 +18,7 @@ DeepPavlov is designed for
 * Issues [*github/issues/*](https://github.com/deeppavlov/DeepPavlov/issues)
 * Forum [*forum.deeppavlov.ai*](https://forum.deeppavlov.ai/)
 * Blogs [*medium.com/deeppavlov*](https://medium.com/deeppavlov)
-* Tutorials [*examples/*](examples) and [extended colab tutorials](https://github.com/deeppavlov/dp_tutorials)
+* [Extended colab tutorials](https://github.com/deeppavlov/dp_tutorials)
 * Docker Hub [*hub.docker.com/u/deeppavlov/*](https://hub.docker.com/u/deeppavlov/) 
     * Docker Images Documentation [*docs:docker-images/*](http://docs.deeppavlov.ai/en/master/intro/installation.html#docker-images)
 
@@ -34,13 +34,9 @@ Please leave us [your feedback](https://forms.gle/i64fowQmiVhMMC7f9) on how we c
 
 [Automatic Spelling Correction](http://docs.deeppavlov.ai/en/master/features/models/spelling_correction.html) | [Entity Linking](http://docs.deeppavlov.ai/en/master/features/models/entity_linking.html)
 
-[Russian SuperGLUE](http://docs.deeppavlov.ai/en/master/features/models/superglue.html)
-
-**Skills**
-
-[Open Domain Questions Answering](http://docs.deeppavlov.ai/en/master/features/skills/odqa.html)
+[Open Domain Questions Answering](http://docs.deeppavlov.ai/en/master/features/models/odqa.html) | [Frequently Asked Questions Answering](http://docs.deeppavlov.ai/en/master/features/models/faq.html)
 
-[Frequently Asked Questions Answering](http://docs.deeppavlov.ai/en/master/features/skills/faq.html)
+[Russian SuperGLUE](http://docs.deeppavlov.ai/en/master/features/models/superglue.html)
 
 **Embeddings**
 
@@ -97,24 +93,26 @@ evaluate and infer it:
 
 #### GPU requirements
 
-To run supported DeepPavlov models on GPU you should have [CUDA](https://developer.nvidia.com/cuda-toolkit) compatible
-with used GPU and [library PyTorch version](deeppavlov/requirements/pytorch.txt).
+By default, DeepPavlov installs models requirements from PyPI. PyTorch from PyPI could not support your device CUDA
+capability. To run supported DeepPavlov models on GPU you should have [CUDA](https://developer.nvidia.com/cuda-toolkit)
+compatible with used GPU and [PyTorch version](deeppavlov/requirements/pytorch.txt) required by DeepPavlov models.
+See [docs](https://docs.deeppavlov.ai/en/master/intro/quick_start.html#using-gpu) for details.
 
 ### Command line interface (CLI)
 
 To get predictions from a model interactively through CLI, run
 
 ```bash
-python -m deeppavlov interact <config_path> [-d]
+python -m deeppavlov interact <config_path> [-d] [-i]
 ```
 
-* `-d` downloads required data -- pretrained model files and embeddings
-  (optional).
+* `-d` downloads required data - pretrained model files and embeddings (optional).
+* `-i` installs model requirements (optional).
 
 You can train it in the same simple way:
 
 ```bash
-python -m deeppavlov train <config_path> [-d]
+python -m deeppavlov train <config_path> [-d] [-i]
 ```
 
 Dataset will be downloaded regardless of whether there was `-d` flag or not.
@@ -126,10 +124,11 @@ The data format is specified in the corresponding model doc page.
 There are even more actions you can perform with configs:
 
 ```bash
-python -m deeppavlov <action> <config_path> [-d]
+python -m deeppavlov <action> <config_path> [-d] [-i]
 ```
 
 * `<action>` can be
+    * `install` to install model requirements (same as `-i`),
     * `download` to download model's data (same as `-d`),
     * `train` to train the model on the data specified in the config file,
     * `evaluate` to calculate metrics on the same dataset,
@@ -140,6 +139,7 @@ python -m deeppavlov <action> <config_path> [-d]
       *<file_path>* if `-f <file_path>` is specified.
 * `<config_path>` specifies path (or name) of model's config file
 * `-d` downloads required data
+* `-i` installs model requirements
 
 
 ### Python
@@ -149,33 +149,26 @@ To get predictions from a model interactively through Python, run
 ```python
 from deeppavlov import build_model
 
-model = build_model(<config_path>, download=True)
+model = build_model(<config_path>, install=True, download=True)
 
 # get predictions for 'input_text1', 'input_text2'
 model(['input_text1', 'input_text2'])
 ```
-
-* where `download=True` downloads required data from web -- pretrained model
-  files and embeddings (optional),
-* `<config_path>` is path to the chosen model's config file (e.g.
-  `"deeppavlov/configs/ner/ner_ontonotes_bert_mult.json"`) or
-  `deeppavlov.configs` attribute (e.g.
+where
+* `install=True` installs model requirements (optional),
+* `download=True` downloads required data from web - pretrained model files and embeddings (optional),
+* `<config_path>` is model name (e.g. `'ner_ontonotes_bert_mult'`), path to the chosen model's config file (e.g.
+  `"deeppavlov/configs/ner/ner_ontonotes_bert_mult.json"`),  or `deeppavlov.configs` attribute (e.g.
   `deeppavlov.configs.ner.ner_ontonotes_bert_mult` without quotation marks).
 
 You can train it in the same simple way:
 
 ```python
 from deeppavlov import train_model 
 
-model = train_model(<config_path>, download=True)
+model = train_model(<config_path>, install=True, download=True)
 ```
 
-* `download=True` downloads pretrained model, therefore the pretrained
-model will be, first, loaded and then train (optional).
-
-Dataset will be downloaded regardless of whether there was ``-d`` flag or
-not.
-
 To train on your own data you need to modify dataset reader path in the
 [train config doc](http://docs.deeppavlov.ai/en/master/intro/config_description.html#train-config).
 The data format is specified in the corresponding model doc page. 
@@ -185,12 +178,9 @@ You can also calculate metrics on the dataset specified in your config file:
 ```python
 from deeppavlov import evaluate_model 
 
-model = evaluate_model(<config_path>, download=True)
+model = evaluate_model(<config_path>, install=True, download=True)
 ```
 
-
 ## License
 
 DeepPavlov is Apache 2.0 - licensed.
-
-##

deeppavlov/__init__.py

@@ -26,13 +26,16 @@
 
 
 # TODO: make better
-def train_model(config: [str, Path, dict], download: bool = False, recursive: bool = False) -> Chainer:
-    train_evaluate_model_from_config(config, download=download, recursive=recursive)
+def train_model(config: [str, Path, dict], install: bool = False,
+                download: bool = False, recursive: bool = False) -> Chainer:
+    train_evaluate_model_from_config(config, install=install, download=download, recursive=recursive)
     return build_model(config, load_trained=True)
 
 
-def evaluate_model(config: [str, Path, dict], download: bool = False, recursive: bool = False) -> dict:
-    return train_evaluate_model_from_config(config, to_train=False, download=download, recursive=recursive)
+def evaluate_model(config: [str, Path, dict], install: bool = False,
+                   download: bool = False, recursive: bool = False) -> dict:
+    return train_evaluate_model_from_config(config, to_train=False, install=install,
+                                            download=download, recursive=recursive)
 
 
 # check version

deeppavlov/_meta.py

@@ -1,4 +1,4 @@
-__version__ = '1.0.0'
+__version__ = '1.0.1'
 __author__ = 'Neural Networks and Deep Learning lab, MIPT'
 __description__ = 'An open source library for building end-to-end dialog systems and training chatbots.'
 __keywords__ = ['NLP', 'NER', 'SQUAD', 'Intents', 'Chatbot']

deeppavlov/core/commands/infer.py

@@ -12,7 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import json
-import pickle
 import sys
 from itertools import islice
 from logging import getLogger
@@ -24,15 +23,18 @@
 from deeppavlov.core.common.params import from_params
 from deeppavlov.core.data.utils import jsonify_data
 from deeppavlov.download import deep_download
+from deeppavlov.utils.pip_wrapper import install_from_config
 
 log = getLogger(__name__)
 
 
 def build_model(config: Union[str, Path, dict], mode: str = 'infer',
-                load_trained: bool = False, download: bool = False) -> Chainer:
+                load_trained: bool = False, install: bool = False, download: bool = False) -> Chainer:
     """Build and return the model described in corresponding configuration file."""
     config = parse_config(config)
 
+    if install:
+        install_from_config(config)
     if download:
         deep_download(config)
 

deeppavlov/core/commands/train.py

@@ -24,6 +24,7 @@
 from deeppavlov.core.data.data_learning_iterator import DataLearningIterator
 from deeppavlov.core.data.utils import get_all_elems_from_json
 from deeppavlov.download import deep_download
+from deeppavlov.utils.pip_wrapper import install_from_config
 
 log = getLogger(__name__)
 
@@ -70,12 +71,15 @@ def train_evaluate_model_from_config(config: Union[str, Path, dict],
                                      iterator: Union[DataLearningIterator, DataFittingIterator] = None, *,
                                      to_train: bool = True,
                                      evaluation_targets: Optional[Iterable[str]] = None,
+                                     install: bool = False,
                                      download: bool = False,
                                      start_epoch_num: Optional[int] = None,
                                      recursive: bool = False) -> Dict[str, Dict[str, float]]:
     """Make training and evaluation of the model described in corresponding configuration file."""
     config = parse_config(config)
 
+    if install:
+        install_from_config(config)
     if download:
         deep_download(config)
 

deeppavlov/core/data/simple_vocab.py

@@ -109,7 +109,7 @@ def load(self):
         self.reset()
         if self.load_path:
             if self.load_path.is_file():
-                log.info("[loading vocabulary from {}]".format(self.load_path))
+                log.debug("[loading vocabulary from {}]".format(self.load_path))
                 tokens, counts = [], []
                 for ln in self.load_path.open('r', encoding='utf8'):
                     token, cnt = self.load_line(ln)

deeppavlov/core/models/torch_model.py

@@ -99,7 +99,7 @@ def __init__(self, device: str = "gpu",
         # we need to switch to eval mode here because by default it's in `train` mode.
         # But in case of `interact/build_model` usage, we need to have model in eval mode.
         self.model.eval()
-        log.info(f"Model was successfully initialized! Model summary:\n {self.model}")
+        log.debug(f"Model was successfully initialized! Model summary:\n {self.model}")
 
     def init_from_opt(self, model_func: str) -> None:
         """Initialize from scratch `self.model` with the architecture built in  `model_func` method of this class
@@ -150,22 +150,22 @@ def load(self, fname: Optional[str] = None, *args, **kwargs) -> None:
         model_func = getattr(self, self.opt.get("model_name", ""), None)
 
         if self.load_path:
-            log.info(f"Load path {self.load_path} is given.")
+            log.debug(f"Load path {self.load_path} is given.")
             if isinstance(self.load_path, Path) and not self.load_path.parent.is_dir():
                 raise ConfigError("Provided load path is incorrect!")
 
             weights_path = Path(self.load_path.resolve())
             weights_path = weights_path.with_suffix(f".pth.tar")
             if weights_path.exists():
-                log.info(f"Load path {weights_path} exists.")
-                log.info(f"Initializing `{self.__class__.__name__}` from saved.")
+                log.debug(f"Load path {weights_path} exists.")
+                log.debug(f"Initializing `{self.__class__.__name__}` from saved.")
 
                 # firstly, initialize with random weights and previously saved parameters
                 if model_func:
                     self.init_from_opt(model_func)
 
                 # now load the weights, optimizer from saved
-                log.info(f"Loading weights from {weights_path}.")
+                log.debug(f"Loading weights from {weights_path}.")
                 checkpoint = torch.load(weights_path, map_location=self.device)
                 model_state = checkpoint["model_state_dict"]
                 optimizer_state = checkpoint["optimizer_state_dict"]
@@ -181,10 +181,10 @@ def load(self, fname: Optional[str] = None, *args, **kwargs) -> None:
                 self.optimizer.load_state_dict(optimizer_state)
                 self.epochs_done = checkpoint.get("epochs_done", 0)
             elif model_func:
-                log.info(f"Init from scratch. Load path {weights_path} does not exist.")
+                log.debug(f"Init from scratch. Load path {weights_path} does not exist.")
                 self.init_from_opt(model_func)
         elif model_func:
-            log.info(f"Init from scratch. Load path {self.load_path} is not provided.")
+            log.debug(f"Init from scratch. Load path {self.load_path} is not provided.")
             self.init_from_opt(model_func)
 
     @overrides

deeppavlov/core/trainers/fit_trainer.py

@@ -63,7 +63,7 @@ def __init__(self, chainer_config: dict, *, batch_size: int = -1,
                  max_test_batches: int = -1,
                  **kwargs) -> None:
         if kwargs:
-            log.info(f'{self.__class__.__name__} got additional init parameters {list(kwargs)} that will be ignored:')
+            log.warning(f'{self.__class__.__name__} got additional init parameters {list(kwargs)} that will be ignored:')
         self.chainer_config = chainer_config
         self._chainer = Chainer(chainer_config['in'], chainer_config['out'], chainer_config.get('in_y'))
         self.batch_size = batch_size

deeppavlov/deep.py

@@ -41,6 +41,7 @@
 parser.add_argument("-b", "--batch-size", dest="batch_size", default=None, help="inference batch size", type=int)
 parser.add_argument("-f", "--input-file", dest="file_path", default=None, help="Path to the input file", type=str)
 parser.add_argument("-d", "--download", action="store_true", help="download model components")
+parser.add_argument("-i", "--install", action="store_true", help="install model requirements")
 
 parser.add_argument("--folds", help="number of folds", type=int, default=5)
 
@@ -67,6 +68,8 @@ def main():
     args = parser.parse_args()
     pipeline_config_path = find_config(args.config_path)
 
+    if args.install or args.mode == 'install':
+        install_from_config(pipeline_config_path)
     if args.download or args.mode == 'download':
         deep_download(pipeline_config_path)
 
@@ -95,8 +98,6 @@ def main():
                              rabbit_virtualhost=args.rabbit_virtualhost)
     elif args.mode == 'predict':
         predict_on_stream(pipeline_config_path, args.batch_size, args.file_path)
-    elif args.mode == 'install':
-        install_from_config(pipeline_config_path)
     elif args.mode == 'crossval':
         if args.folds < 2:
             log.error('Minimum number of Folds is 2')

deeppavlov/download.py

@@ -145,7 +145,7 @@ def download_resource(url: str, dest_paths: Iterable[Union[Path, str]], headers:
 
 def download_resources(args: Namespace) -> None:
     if not args.all and not args.config:
-        log.error('You should provide either skill config path or -all flag')
+        log.error('You should provide either model config path or -all flag')
         sys.exit(1)
     elif args.all:
         downloads = get_configs_downloads()

deeppavlov/models/__init__.py

@@ -20,7 +20,7 @@
 
 if not os.environ.get('DP_SKIP_NLTK_DOWNLOAD'):
     with RedirectedPrints():
-        nltk.download('punkt')
-        nltk.download('stopwords')
-        nltk.download('perluniprops')
-        nltk.download('nonbreaking_prefixes')
+        nltk.download('punkt', quiet=True)
+        nltk.download('stopwords', quiet=True)
+        nltk.download('perluniprops', quiet=True)
+        nltk.download('nonbreaking_prefixes', quiet=True)

deeppavlov/models/classifiers/cos_sim_classifier.py

@@ -130,5 +130,5 @@ def save(self) -> None:
 
     def load(self) -> None:
         """Load classifier parameters"""
-        logger.info("Loading faq_model from {}".format(self.load_path))
+        logger.debug("Loading faq_model from {}".format(self.load_path))
         self.x_train_features, self.y_train = load_pickle(self.load_path)

deeppavlov/models/classifiers/re_bert.py

@@ -156,7 +156,7 @@ def get_hrt(self, sequence_output: Tensor, attention: Tensor, entity_pos: List)
 
     def load(self) -> None:
         if self.pretrained_bert:
-            log.info(f"From pretrained {self.pretrained_bert}.")
+            log.debug(f"From pretrained {self.pretrained_bert}.")
             self.config = AutoConfig.from_pretrained(
                 self.pretrained_bert, num_labels=self.n_classes, output_attentions=True, output_hidden_states=True
             )

deeppavlov/models/doc_retrieval/pop_ranker.py

@@ -57,11 +57,11 @@ class PopRanker(Component):
     def __init__(self, pop_dict_path: str, load_path: str, top_n: int = 3, active: bool = True,
                  **kwargs) -> None:
         pop_dict_path = expand_path(pop_dict_path)
-        logger.info(f"Reading popularity dictionary from {pop_dict_path}")
+        logger.debug(f"Reading popularity dictionary from {pop_dict_path}")
         self.pop_dict = read_json(pop_dict_path)
         self.mean_pop = np.mean(list(self.pop_dict.values()))
         load_path = expand_path(load_path)
-        logger.info(f"Loading popularity ranker from {load_path}")
+        logger.debug(f"Loading popularity ranker from {load_path}")
         self.clf = joblib.load(load_path)
         self.top_n = top_n
         self.active = active

deeppavlov/models/embedders/fasttext_embedder.py

@@ -50,7 +50,7 @@ def load(self) -> None:
         """
         Load fastText binary model from self.load_path
         """
-        log.info(f"[loading fastText embeddings from `{self.load_path}`]")
+        log.debug(f"[loading fastText embeddings from `{self.load_path}`]")
         self.model = fasttext.load_model(str(self.load_path))
         self.dim = self.model.get_dimension()
 

deeppavlov/models/kbqa/query_generator_base.py

@@ -164,7 +164,7 @@ def get_entity_ids(self, entities: List[str], tags: List[str], question: str) ->
         try:
             el_output = self.entity_linker([entities], [tags], [[question]], [None], [None])
         except json.decoder.JSONDecodeError:
-            log.info("not received output from entity linking")
+            log.warning("not received output from entity linking")
         if el_output:
             if self.use_el_api_requester:
                 el_output = el_output[0]
@@ -262,7 +262,7 @@ def find_top_rels(self, question: str, entity_ids: List[List[str]], triplet_info
             try:
                 ex_rels = self.wiki_parser(parser_info_list, queries_list)
             except json.decoder.JSONDecodeError:
-                log.info("find_top_rels, not received output from wiki parser")
+                log.warning("find_top_rels, not received output from wiki parser")
             if self.use_wp_api_requester and ex_rels:
                 ex_rels = [rel[0] for rel in ex_rels]
             ex_rels = list(set(ex_rels))

deeppavlov/models/kbqa/rel_ranking_infer.py

@@ -170,7 +170,7 @@ def __call__(self, questions_list: List[str],
                     try:
                         answer = sentence_answer(question, answer, entities, template_answer)
                     except:
-                        log.info("Error in sentence answer")
+                        log.warning("Error in sentence answer")
                 confidence = answers_with_scores[0][2]
             if self.return_confidences:
                 answers.append((answer, confidence))