Merge branch 'main' into parallelization_scribe

Author: nfahlgren

.all-contributorsrc

@@ -649,6 +649,47 @@
         "bug",
         "doc"
       ]
+    },
+    {
+      "login": "Sunitalama2",
+      "name": "Sunitalama2",
+      "avatar_url": "https://avatars.githubusercontent.com/u/195969937?v=4",
+      "profile": "https://github.com/Sunitalama2",
+      "contributions": [
+        "tutorial",
+        "doc"
+      ]
+    },
+    {
+      "login": "trogers623",
+      "name": "trogers623",
+      "avatar_url": "https://avatars.githubusercontent.com/u/195968055?v=4",
+      "profile": "https://github.com/trogers623",
+      "contributions": [
+        "tutorial",
+        "doc"
+      ]
+    },
+    {
+      "login": "eseigeldanf",
+      "name": "eseigeldanf",
+      "avatar_url": "https://avatars.githubusercontent.com/u/195968072?v=4",
+      "profile": "https://github.com/eseigeldanf",
+      "contributions": [
+        "bug",
+        "tutorial",
+        "doc"
+      ]
+    },
+    {
+      "login": "clhendrikse",
+      "name": "Chloe Hendrikse",
+      "avatar_url": "https://avatars.githubusercontent.com/u/70405404?v=4",
+      "profile": "https://github.com/clhendrikse",
+      "contributions": [
+        "tutorial",
+        "doc"
+      ]
     }
   ],
   "contributorsPerLine": 7,

README.md

@@ -7,8 +7,8 @@
 
 ### Table of contents
 1. [Supported platforms and dependencies](#dependencies)
-2. [Desktop installation step-by-step guide](#desktop)
-3. [Server/command-line step-by-step guide](#cli)
+2. [Server/command-line step-by-step guide](#cli)
+3. [Desktop installation step-by-step guide](#desktop)
 4. [Detailed installation instructions](#detailed)
     1. [Conda](#conda)
     2. [PyPI](#pypi)
@@ -18,14 +18,32 @@
 - macOS x86 (Intel) and M (ARM) processors
 - Windows 64-bit, x86 processors
 
-### Desktop installation step-by-step guide <a name="desktop"></a>
 
-<iframe src="https://scribehow.com/embed/Install_PlantCV_via_Jupyter_Lab_Desktop__cS9d6VcxRcuDPGZxDfQycw" width="100%" height="640" allowfullscreen frameborder="0"></iframe>
+### Server/command line step-by-step guide <a name="cli"></a>
+
+Use the server/command line installation if you plan to create PlantCV workflows and run workflows in parallel. 
+Click through our step-by-step guide below to install PlantCV through conda.
 
-### Server/command-line step-by-step guide <a name="cli"></a>
 
 <iframe src="https://scribehow.com/embed/Installing_PlantCV__MacOSLinux__awAP9Xm2SgWV4SMZadm9CQ" width="640" height="640" allowfullscreen frameborder="0"></iframe>
 
+
+!!!note
+    Once you have installed PlantCV, to get started see our [guide to using PlantCV with Jupyter Notebooks](https://plantcv.readthedocs.io/en/stable/jupyter/) 
+    and our [guide to developing workflows in PlantCV](https://plantcv.readthedocs.io/en/stable/analysis_approach/#developing-image-processing-workflows-workflow-development).
+
+---
+
+### Desktop installation step-by-step guide <a name="desktop"></a>
+This is a simple install option if you would just like to test out PlantCV.
+If you plan to use PlantCV for your analyses and run your workflows in parallel, we recommend using the command line installation above.
+
+Click through our step-by-step guide below to install PlantCV through the JupyterLab Desktop app.
+
+<iframe src="https://scribehow.com/embed/Install_PlantCV_via_Jupyter_Lab_Desktop__cS9d6VcxRcuDPGZxDfQycw" width="100%" height="640" allowfullscreen frameborder="0"></iframe>
+
+---
+
 ### Detailed installation instructions <a name="detailed"></a>
 
 PlantCV requires Python (tested with versions 3.9, 3.10, and 3.11) and these [Python packages](https://github.com/danforthcenter/plantcv/blob/main/pyproject.toml).

docs/installation.md

@@ -2,7 +2,7 @@
 
 The first function (`pcv.predict_kmeans`) takes a target image and uses a trained kmeans model produced by [`pcv.learn.train_kmeans`](train_kmeans.md) to classify regions of the target image by the trained clusters. The second function (`pcv.mask_kmeans`) takes a list of clusters and produces the combined mask from clusters of interest. The target and training images may be in grayscale or RGB image format.
 
-**plantcv.kmeans_classifier.predict_kmeans**(img, model_path="./kmeansout.fit", patch_size=10)
+**plantcv.predict_kmeans**(img, model_path="./kmeansout.fit", patch_size=10)
 
 **outputs** An image with regions colored and labeled according to cluster assignment
 
@@ -18,7 +18,7 @@ The first function (`pcv.predict_kmeans`) takes a target image and uses a traine
 - **Example use below**
 
 
-**plantcv.kmeans_classifier.mask_kmeans**(labeled_img, k, cat_list=None)
+**plantcv.mask_kmeans**(labeled_img, k, cat_list=None)
 
 **outputs** Either a combined mask of the requestedlist of clusters or a dictionary of each cluster as a separate mask with keys corresponding to the cluster number 
 

docs/kmeans_classifier.md

@@ -8,7 +8,7 @@ to run workflows in parallel.
 Create a configuration file from a template:
 
 ```bash
-plantcv-run-workflow --template my_config.txt
+plantcv-run-workflow --template my_config.json
 ```
 
 *class* **plantcv.parallel.WorkflowConfig**
@@ -64,8 +64,8 @@ Validate parameters/structure of configuration data.
 * **img_outdir**: (str, default = "."): path/name of output directory where images will be saved.
 
 
-* **tmp_dir**: (str, default = `None`): path/name of parent folder for the temporary directory, uses system default
-  temporary directory when `None`.
+* **tmp_dir**: (str, default = `"."`): path/name of parent folder for the temporary directory, defaults to the
+current working directory.
 
 
 * **start_date**: (str, default = `None`): start date used to filter images. Images will be analyzed that are newer 
@@ -107,15 +107,15 @@ for downstream analysis. The default, `filepath` will create groups of single im
 example of a multi-image group could be to pair VIS and NIR images (e.g. `["timestamp", "camera", "rotation"]`). Supported
 metadata terms are listed [here](pipeline_parallel.md).
 
-* **group_name** (str, default = `"imgtype"`): either a metadata term used to create a unique name for each image in an
+* **group_name** (str, default = `"auto"`): either a metadata term used to create a unique name for each image in an
 image group (created by `groupby`), or `"auto"` to generate a numbered image sequence `image1, image2, ...`. The resulting
 names are used to access individual image filepaths in a workflow.
 
 * **cleanup**: (bool, default =`True`): remove temporary job directory if `True`.
 
 
-* **append**: (bool, default = `True`): if `True` will append results to an existing json file. If `False`, will delete
-  previous results stored in the specified JSON file.
+* **append**: (bool, default = `False`): if `False`, will delete previous results stored in the specified JSON file.
+  If `True` will append results to an existing json file.
 
 
 * **cluster** (str, default = "LocalCluster"): LocalCluster will run PlantCV workflows on a single machine. All valid
@@ -155,32 +155,32 @@ After defining the cluster, parameters are used to define the size of and reques
 environment. These settings are defined in the `cluster_config` parameter. We define by default the following 
 parameters:
 
-**n_workers**: (int, required, default = 1): the number of workers/slots to request from the cluster. Because we 
+* **n_workers**: (int, required, default = 1): the number of workers/slots to request from the cluster. Because we 
 generally use 1 CPU per image analysis workflow, this is effectively the maximum number of concurrently running 
 workflows.
 
-**cores**: (int, required, default = 1): the number of compute cores per workflow. This should be left as 1 unless a 
+* **cores**: (int, required, default = 1): the number of compute cores per workflow. This should be left as 1 unless a 
 workflow is designed to use multiple CPUs/cores/threads.
 
-**memory**: (str, required, default = "1GB"): the amount of memory/RAM used per workflow. Can be set as a number plus 
+* **memory**: (str, required, default = "1GB"): the amount of memory/RAM used per workflow. Can be set as a number plus 
 units (KB, MB, GB, etc.).
 
-**disk**: (str, required, default = "1GB"): the amount of disk space used per workflow. Can be set as a number plus 
+* **disk**: (str, required, default = "1GB"): the amount of disk space used per workflow. Can be set as a number plus 
 units (KB, MB, GB, etc.).
 
-**log_directory**: (str, optional, default = `None`): directory where worker logs are stored. Can be set to a path or 
+* **log_directory**: (str, optional, default = `None`): directory where worker logs are stored. Can be set to a path or 
 environmental variable.
 
-**local_directory**: (str, optional, default = `None`): dask working directory location. Can be set to a path or 
+* **local_directory**: (str, optional, default = `None`): dask working directory location. Can be set to a path or 
 environmental variable.
 
-**job_extra_directives**: (dict, optional, default = `None`): extra parameters sent to the scheduler. Specified as a dictionary 
+* **job_extra_directives**: (dict, optional, default = `None`): extra parameters sent to the scheduler. Specified as a dictionary 
 of key-value pairs (e.g. `{"getenv": "true"}`).
 
 !!! note
     `n_workers` is the only parameter used by `LocalCluster`, all others are currently ignored. `n_workers`, `cores`,
     `memory`, and `disk` are required by the other clusters. All other parameters are optional. Additional parameters
-    defined in the [dask-jobqueu API](https://jobqueue.dask.org/en/latest/api.html) can be supplied.
+    defined in the [dask-jobqueue API](https://jobqueue.dask.org/en/latest/api.html) can be supplied.
 
 ### Example
 

docs/parallel_config.md

@@ -22,14 +22,15 @@ a configuration file can be edited and input.
 To create a configuration file, run the following:
 
 ```bash
-plantcv-run-workflow --template my_config.txt
+plantcv-run-workflow --template my_config.json
 
 ```
 
 The code above saves a text configuration file in JSON format using the built-in defaults for parameters. The parameters can be modified
 directly in Python as demonstrated in the [WorkflowConfig documentation](parallel_config.md). A configuration can be
 saved at any time using the `save_config` method to save for later use. Alternatively, open the saved config
-file with your favorite text editor and adjust the parameters as needed.
+file with your favorite text editor and adjust the parameters as needed (refer to the attributes section of
+[WorkflowConfig documentation](parallel_config.md) for details about each parameter).
 
 **Some notes on JSON format:**
 
@@ -103,7 +104,7 @@ Sample image filename: `cam1_16-08-06-16:45_el1100s1_p19.jpg`
     "filename_metadata": ["camera", "timestamp", "id", "other"],
     "workflow": "/home/mgehan/pat-edger/round1-python-pipelines/2016-08_pat-edger_brassica-cam1-splitimg.py",
     "img_outdir": "/shares/mgehan_share/raw_data/raw_image/2016-08_pat-edger/data/split-round1/split-cam1/output",
-    "tmp_dir": null,
+    "tmp_dir": "."",
     "start_date": null,
     "end_date": null,
     "imgformat": "jpg",
@@ -115,7 +116,7 @@ Sample image filename: `cam1_16-08-06-16:45_el1100s1_p19.jpg`
     "groupby": ["filepath"],
     "group_name": "auto",
     "cleanup": true,
-    "append": true,
+    "append": false,
     "cluster": "HTCondorCluster",
     "cluster_config": {
         "n_workers": 16,
@@ -179,7 +180,7 @@ in a list to the `filename_metadata` parameter.
     "filename_metadata": ["camera", "plantbarcode", "timestamp"],
     "workflow": "user-workflow.py",
     "img_outdir": "output_directory",
-    "tmp_dir": null,
+    "tmp_dir": ".",
     "start_date": null,
     "end_date": null,
     "imgformat": "jpg",
@@ -191,7 +192,7 @@ in a list to the `filename_metadata` parameter.
     "groupby": ["filepath"],
     "group_name": "auto",
     "cleanup": true,
-    "append": true,
+    "append": false,
     "cluster": "HTCondorCluster",
     "cluster_config": {
         "n_workers": 16,
@@ -231,7 +232,7 @@ To identify each image within our workflow, we will name them based on the `imgt
     "filename_metadata": ["imgtype", "timestamp", "id", "other"],
     "workflow": "/home/mgehan/pat-edger/round1-python-pipelines/2016-08_pat-edger_brassica-cam1-splitimg.py",
     "img_outdir": "/shares/mgehan_share/raw_data/raw_image/2016-08_pat-edger/data/split-round1/split-cam1/output",
-    "tmp_dir": null,
+    "tmp_dir": ".",
     "start_date": null,
     "end_date": null,
     "imgformat": "jpg",
@@ -243,7 +244,7 @@ To identify each image within our workflow, we will name them based on the `imgt
     "groupby": ["timestamp"],
     "group_name": "imgtype",
     "cleanup": true,
-    "append": true,
+    "append": false,
     "cluster": "HTCondorCluster",
     "cluster_config": {
         "n_workers": 16,

docs/pipeline_parallel.md

@@ -21,7 +21,7 @@ def __init__(self):
         self.workflow = ""
         self.img_outdir = "./output_images"
         self.include_all_subdirs = True
-        self.tmp_dir = None
+        self.tmp_dir = "."
         self.start_date = None
         self.end_date = None
         self.imgformat = "png"
@@ -31,9 +31,9 @@ def __init__(self):
         self.writeimg = False
         self.other_args = {}
         self.groupby = ["filepath"]
-        self.group_name = "imgtype"
+        self.group_name = "auto"
         self.cleanup = True
-        self.append = True
+        self.append = False
         self.cluster = "LocalCluster"
         self.cluster_config = {
             "n_workers": 1,

plantcv/parallel/__init__.py

@@ -5,6 +5,7 @@
 import time
 import datetime
 import plantcv.parallel
+import plantcv.utils
 import tempfile
 import shutil
 
@@ -119,6 +120,16 @@ def main():
     print(f"Processing results took {process_results_clock_time} seconds.", file=sys.stderr)
     ###########################################
 
+    # Convert json results to csv files
+    ###########################################
+    # Convert results start time
+    convert_results_start_time = time.time()
+    print("Converting json to csv... ", file=sys.stderr)
+    plantcv.utils.json2csv(config.json, config.json)
+    convert_results_clock_time = time.time() - convert_results_start_time
+    print(f"Processing results took {convert_results_clock_time} seconds.", file=sys.stderr)
+    ###########################################
+
     # Cleanup
     if config.cleanup is True:
         shutil.rmtree(config.tmp_dir)

plantcv/parallel/cli.py

@@ -22,7 +22,7 @@ def closing(gray_img, kernel=None):
         fatal_error("Input image must be grayscale or binary")
 
     # If image is binary use the faster method
-    if len(np.unique(gray_img)) == 2:
+    if len(np.unique(gray_img)) <= 2:
         bool_img = morphology.binary_closing(gray_img, kernel)
         filtered_img = np.copy(bool_img.astype(np.uint8) * 255)
     # Otherwise use method appropriate for grayscale images

plantcv/plantcv/closing.py

@@ -22,7 +22,7 @@ def fill_holes(bin_img):
     :return filtered_img: numpy.ndarray
     """
     # Make sure the image is binary
-    if len(np.shape(bin_img)) != 2 or len(np.unique(bin_img)) != 2:
+    if len(np.shape(bin_img)) != 2 or len(np.unique(bin_img)) > 2:
         fatal_error("Image is not binary")
 
     # Cast binary image to boolean

plantcv/plantcv/fill_holes.py

@@ -48,6 +48,7 @@ def predict_kmeans(img, model_path="./kmeansout.fit", patch_size=10):
     reshape_params = [[h - 2*mg + 1, w - 2*mg + 1], [h - 2*mg, w - 2*mg]]
     # Takes care of even vs odd numbered patch size reshaping
     labeled = train_labels.reshape(reshape_params[patch_size % 2][0], reshape_params[patch_size % 2][1])
+    labeled = labeled.astype("uint8")
     _debug(visual=labeled, filename=os.path.join(params.debug_outdir, "_labeled_img.png"))
     return labeled
 
@@ -70,7 +71,7 @@ def mask_kmeans(labeled_img, k, cat_list=None):
         mask_dict = {}
         L = [*range(k)]
         for i in L:
-            mask_light = np.where(labeled_img == i, 255, 0)
+            mask_light = np.where(labeled_img == i, 255, 0).astype("uint8")
             _debug(visual=mask_light, filename=os.path.join(params.debug_outdir, "_kmeans_mask_"+str(i)+".png"))
             mask_dict[str(i)] = mask_light
         return mask_dict
@@ -80,9 +81,9 @@ def mask_kmeans(labeled_img, k, cat_list=None):
     params.debug = None
     for idx, i in enumerate(cat_list):
         if idx == 0:
-            mask_light = np.where(labeled_img == i, 255, 0)
+            mask_light = np.where(labeled_img == i, 255, 0).astype("uint8")
         else:
-            mask_light = pcv.logical_or(mask_light, np.where(labeled_img == i, 255, 0))
+            mask_light = pcv.logical_or(mask_light, np.where(labeled_img == i, 255, 0).astype("uint8"))
     params.debug = debug
     _debug(visual=mask_light, filename=os.path.join(params.debug_outdir, "_kmeans_combined_mask.png"))
     return mask_light

plantcv/plantcv/kmeans_classifier.py

@@ -1,6 +1,6 @@
 # Automatically detect a color card and color correct to standard chip values
 
-from plantcv.plantcv import params
+from plantcv.plantcv import params, deprecation_warning
 from plantcv.plantcv.transform.detect_color_card import detect_color_card
 from plantcv.plantcv.transform.color_correction import get_color_matrix, std_color_matrix, affine_color_correction
 
@@ -28,7 +28,10 @@ def auto_correct_color(rgb_img, label=None, **kwargs):
     # Set lable to params.sample_label if None
     if label is None:
         label = params.sample_label
-
+    deprecation_warning(
+        "The 'label' parameter is no longer utilized, since color chip size is now metadata. "
+        "It will be removed in PlantCV v5.0."
+        )
     # Get keyword arguments and set defaults if not set
     labeled_mask = detect_color_card(rgb_img=rgb_img, min_size=kwargs.get("min_size", 1000),
                                      radius=kwargs.get("radius", 20),

plantcv/plantcv/transform/auto_correct_color.py

@@ -6,7 +6,7 @@
 import cv2
 import math
 import numpy as np
-from plantcv.plantcv import params, outputs, fatal_error
+from plantcv.plantcv import params, outputs, fatal_error, deprecation_warning
 from plantcv.plantcv._debug import _debug
 
 
@@ -108,7 +108,10 @@ def detect_color_card(rgb_img, label=None, **kwargs):
     # Set lable to params.sample_label if None
     if label is None:
         label = params.sample_label
-
+    deprecation_warning(
+        "The 'label' parameter is no longer utilized, since color chip size is now metadata. "
+        "It will be removed in PlantCV v5.0."
+        )
     # Get keyword arguments and set defaults if not set
     min_size = kwargs.get("min_size", 1000)  # Minimum size for _is_square chip filtering
     radius = kwargs.get("radius", 20)  # Radius of circles to draw on the color chips