Improve management of documentation by helper type and keywords

src/engine/test/helper_tests/README.md

@@ -7,6 +7,7 @@
     1. [engine-helper-test-validator](#engine-helper-test-validator)
     1. [engine-helper-test-generator](#engine-helper-test-generator)
     1. [engine-helper-test-generate-runner](#engine-helper-test-generate-runner)
+    1. [engine-helper-test-documentation](#engine-helper-test-documentation)
 1. [How to run all the tests](#how-to-run-all-the-tests)
 1. [How to create an output in particular and in general](#how-to-create-an-output-in-particular-and-in-general)
     1. [Create an output for a given helper function](#create-an-output-for-a-given-helper-function)
@@ -24,6 +25,10 @@
     4. [Target field property](#target-field-property)
     5. [Skipped property](#skipped-property)
     6. [Test property](#test-property)
+1. [How generate helper functions documentation](#how-generate-helper-functions-documentation)
+    1. [Generate documentation for a particular helper function](#generate-documentation-for-a-particular-helper-function)
+    2. [Generate documentation for a particular type of helper function](#generate-documentation-for-a-particular-type-of-helper-function)
+    3. [Generate documentation for all helper functions](#generate-documentation-for-all-helper-functions)
 
 
 # End to end tests for helper functions
@@ -67,6 +72,15 @@ helper/
 |       └── template.py
 |       └── test_data.py
 |       └── validator.py
+|     └── documentation_generator
+|       └── __init__.py
+|       └── __main__.py
+|       └── documentation.py
+|       └── exporter.py
+|       └── html_generator.py
+|       └── mark_down_generator.py
+|       └── pdf_generator.py
+|       └── types.py
 |    └── .gitignore
 |    └── pyproject.toml
 |    └── setup.cfg
@@ -186,6 +200,23 @@ optional arguments:
                         Absolute or relative path where the test cases were generated
 ```
 
+## engine-helper-test-documentation
+```bash
+usage: engine-helper-test-documentation [-h] [--input_file_path INPUT_FILE_PATH] [--folder_path FOLDER_PATH] [--exporter EXPORTER] [-o OUTPUT_PATH]
+
+Generates files containing documentation for a given helper
+
+options:
+  -h, --help            show this help message and exit
+  --input_file_path INPUT_FILE_PATH
+                        Absolute or relative path where the description of the helper function is located
+  --folder_path FOLDER_PATH
+                        Absolute or relative path where the directory that contains the descriptions of the auxiliary functions is located
+  --exporter EXPORTER   Absolute or relative path of the directory where the generated test files will be located
+  -o OUTPUT_PATH, --output_path OUTPUT_PATH
+                        Absolute or relative path of the directory where the generated documentation files will be located
+```
+
 # How to run all the tests
 
 After completing the installation, you must run the setupEnvironment.py that builds the environment in a certain location.
@@ -430,3 +461,27 @@ test:
     expected: any_value
     description: any description
 ```
+
+# How generate helper functions documentation
+By default, if a directory is not specified to store the created documentation, it will be saved in `/tmp/documentation`.
+Likewise, the default exporter type if one is not specified will be `markdown`.
+The commands below use absolute paths to the engine directory.
+
+## Generate documentation for a particular helper function
+```bash
+engine-helper-test-documentation --input_file_path test/helper_tests/helpers_description/map/int_calculate.yml -o test/helper_tests/documentation
+```
+## Generate documentation for a particular type of helper function
+```bash
+engine-helper-test-documentation --folder_path test/helper_tests/helpers_description/map -o test/helper_tests/documentation
+```
+```bash
+engine-helper-test-documentation --folder_path test/helper_tests/helpers_description/filter -o test/helper_tests/documentation
+```
+```bash
+engine-helper-test-documentation --folder_path test/helper_tests/helpers_description/transformation -o test/helper_tests/documentation
+```
+## Generate documentation for all helper functions
+```bash
+engine-helper-test-documentation --folder_path test/helper_tests/helpers_description/ -o test/helper_tests/documentation
+```

src/engine/test/helper_tests/engine_helper_test/src/documentation_generator/__main__.py

@@ -7,12 +7,11 @@
 from .documentation import *
 from .exporter import IExporter
 from .types import *
-import tempfile
 import subprocess
 
 
 def parse_arguments() -> argparse.Namespace:
-    arg_parser = argparse.ArgumentParser(description="Generates files containing test cases for a given helper")
+    arg_parser = argparse.ArgumentParser(description="Generates files containing documentation for a given helper")
     arg_parser.add_argument(
         "--input_file_path",
         help="Absolute or relative path where the description of the helper function is located",
@@ -28,7 +27,7 @@ def parse_arguments() -> argparse.Namespace:
     arg_parser.add_argument(
         "-o",
         "--output_path",
-        help="Absolute or relative path of the directory where the generated test files will be located",
+        help="Absolute or relative path of the directory where the generated documentation files will be located",
     )
 
     args = arg_parser.parse_args()
@@ -39,42 +38,41 @@ def parse_arguments() -> argparse.Namespace:
     return args
 
 
-def is_temp_path(path: Path):
-    path = path.resolve()
-    temp_dir = Path(tempfile.gettempdir()).resolve()
-    return str(path).startswith(str(temp_dir))
-
-
-def generate_documentation(parser: Parser, exporter: IExporter, file: Path, output_dir: Path):
+def generate_documentation(parser: Parser, exporter: IExporter, file: Path, output_path: Path):
     parser.load_yaml_from_file(file)
     documentation = parse_yaml_to_documentation(parser)
     exporter.create_document(documentation)
-    exporter.save(output_dir)
+    exporter.save(output_path)
 
 
 def main():
     args = parse_arguments()
     parser = Parser()
     exporter_type = args.exporter if args.exporter else "mark_down"
-    output_dir = Path(args.output_path if args.output_path else "/tmp/documentation")
-    if not is_temp_path(output_dir):
-        sys.exit("the output directory must be a temporary one")
+    output_path = Path(args.output_path if args.output_path else "/tmp/documentation")
     exporter = ExporterFactory.get_exporter(exporter_type)
     if args.input_file_path:
         command = f'engine-helper-test-validator --input_file_path {args.input_file_path}'
         try:
             subprocess.run(command, check=True, shell=True, stdout=subprocess.PIPE)
         except subprocess.CalledProcessError as e:
             sys.exit(e.stderr)
-        generate_documentation(parser, exporter, Path(args.input_file_path), output_dir)
+        generate_documentation(parser, exporter, Path(args.input_file_path), output_path)
     elif args.folder_path:
-        for file in Path(args.folder_path).iterdir():
-            if file.is_file() and (file.suffix in ['.yml', '.yaml']):
-                command = f'engine-helper-test-validator --folder_path {args.folder_path}'
-                try:
-                    subprocess.run(command, check=True, shell=True, stdout=subprocess.PIPE)
-                except subprocess.CalledProcessError as e:
-                    sys.exit(e.stderr)
-                generate_documentation(parser, exporter, file, output_dir)
+        folder_path = Path(args.folder_path)
+        yaml_files = folder_path.rglob('*.yml')
+        yaml_files = list(yaml_files) + list(folder_path.rglob('*.yaml'))
+
+        if not yaml_files:
+            sys.exit("No YAML files found in the specified directory.")
+
+        for file in yaml_files:
+            command = f'engine-helper-test-validator --input_file_path {file}'
+            try:
+                subprocess.run(command, check=True, shell=True, stdout=subprocess.PIPE)
+            except subprocess.CalledProcessError as e:
+                sys.exit(e.stderr)
+            generate_documentation(parser, exporter, file, output_path)
+
     else:
         sys.exit("It is necessary to indicate a file or directory that contains a configuration yaml")

src/engine/test/helper_tests/engine_helper_test/src/documentation_generator/mark_down_generator.py

@@ -1,10 +1,12 @@
 from .types import Documentation
 from .exporter import *
+from collections import defaultdict
 
 
 class MarkdownGenerator(IExporter):
     def __init__(self):
         self.content = []
+        self.all_contents = defaultdict(lambda: defaultdict(list))  # Structure to organize by type and keyword
         self.helper_name = ""
 
     def create_signature(self, doc: Documentation):
@@ -159,7 +161,33 @@ def create_document(self, doc: Documentation):
         #         self.content.append(f"  - **Expected**: `{example.expected}`")
         #     self.content.append(f"  - **Description**: {example.description}\n")
 
+        # Organiza el contenido por tipo de helper y luego por keyword
+        for keyword in doc.keywords:
+            self.all_contents[doc.helper_type][keyword].append('\n'.join(self.content))
+
     def save(self, output_dir: Path):
         output_dir.mkdir(parents=True, exist_ok=True)
-        with open((output_dir / self.helper_name).as_posix() + ".md", 'w') as file:
-            file.write('\n'.join(self.content))
+
+        home_file = output_dir / "index.md"
+        with open(home_file, 'w') as file:
+            file.write("# Helper Types\n")
+            for helper_type in sorted(self.all_contents.keys()):
+                file.write(f"- [{helper_type}](./{helper_type}/index.md)\n")
+
+        for helper_type, keywords in self.all_contents.items():
+            type_dir = output_dir / helper_type
+            type_dir.mkdir(parents=True, exist_ok=True)
+
+            type_index_file = type_dir / "index.md"
+            with open(type_index_file, 'w') as file:
+                file.write(f"# {helper_type}\n")
+                file.write(f"## Keywords\n")
+                for keyword in sorted(keywords.keys()):
+                    file.write(f"- [{keyword}](./{keyword}.md)\n")
+
+            for keyword, docs in keywords.items():
+                keyword_file = type_dir / f"{keyword}.md"
+                with open(keyword_file, 'w') as file:
+                    for doc in docs:
+                        file.write(doc)
+                        file.write("\n---\n")

src/engine/test/helper_tests/helpers_description/filter/binary_and.yml

@@ -6,7 +6,7 @@ metadata:
     Transforms the hexadecimal value into an unsigned integer and performs the AND bit by bit with the field hexadecimal
     transformed to unsigned int. If it is other than 0x0 then the operation evaluates to true.
   keywords:
-    - to define
+    - undefined
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/contains.yml

@@ -7,7 +7,7 @@ metadata:
     If they're not, the function evaluates to false.
     In case of error, the function will evaluate to false.
   keywords:
-    - to define
+    - undefined
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/exists.yml

@@ -7,7 +7,7 @@ metadata:
     If it does, the function will evaluate to true, otherwise it will be false.
     This helper function is used in the check stage.
   keywords:
-    - to define
+    - undefined
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/exists_key_in.yml

@@ -8,7 +8,7 @@ metadata:
     If the object contains the key, then the function will evaluate to true, otherwise it will be false.
     This helper function is typically used in the check stage.
   keywords:
-    - to define
+    - undefined
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/int_equal.yml

@@ -7,7 +7,8 @@ metadata:
     If they're not, the function evaluates to false. In case of error, the function will evaluate to false.
     This helper function is typically used in the check stage
   keywords:
-    - to define
+    - integer
+    - comparison
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/int_greater.yml

@@ -7,7 +7,8 @@ metadata:
     If they're not, the function evaluates to false. In case of error, the function will evaluate to false.
     This helper function is typically used in the check stage
   keywords:
-    - to define
+    - integer
+    - comparison
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/int_less.yml

@@ -7,7 +7,8 @@ metadata:
     If they're not, the function evaluates to false. In case of error, the function will evaluate to false.
     This helper function is typically used in the check stage
   keywords:
-    - to define
+    - integer
+    - comparison
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/int_less_or_equal.yml

@@ -7,7 +7,8 @@ metadata:
     If they're not, the function evaluates to false. In case of error, the function will evaluate to false.
     This helper function is typically used in the check stage
   keywords:
-    - to define
+    - integer
+    - comparison
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/int_not_equal.yml

@@ -7,7 +7,8 @@ metadata:
     If they're not, the function evaluates to false. In case of error, the function will evaluate to false.
     This helper function is typically used in the check stage
   keywords:
-    - to define
+    - integer
+    - comparison
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/ip_cidr_match.yml

@@ -11,7 +11,7 @@ metadata:
     where ddd is a decimal number of up to three digits in the range  0  to  255.
     This helper function is typically used in the check stage
   keywords:
-    - to define
+    - undefined
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/is_array.yml

@@ -7,7 +7,7 @@ metadata:
     If they're not, the function evaluates to false. In case of error, the function will evaluate to false.
     This helper function is typically used in the check stage
   keywords:
-    - to define
+    - undefined
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/is_boolean.yml

@@ -7,7 +7,7 @@ metadata:
     If they're not, the function evaluates to false. In case of error, the function will evaluate to false.
     This helper function is typically used in the check stage
   keywords:
-    - to define
+    - undefined
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/is_not_array.yml

@@ -7,7 +7,7 @@ metadata:
     If they're not, the function evaluates to false. In case of error, the function will evaluate to false.
     This helper function is typically used in the check stage
   keywords:
-    - to define
+    - undefined
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/is_not_boolean.yml

@@ -7,7 +7,7 @@ metadata:
     If they're not, the function evaluates to false. In case of error, the function will evaluate to false.
     This helper function is typically used in the check stage
   keywords:
-    - to define
+    - undefined
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/is_not_object.yml

@@ -7,7 +7,7 @@ metadata:
     If they're not, the function evaluates to false. In case of error, the function will evaluate to false.
     This helper function is typically used in the check stage
   keywords:
-    - to define
+    - undefined
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/is_not_string.yml

@@ -7,7 +7,7 @@ metadata:
     If they're not, the function evaluates to false. In case of error, the function will evaluate to false.
     This helper function is typically used in the check stage
   keywords:
-    - to define
+    - undefined
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/is_number.yml

@@ -8,7 +8,7 @@ metadata:
     This helper function is typically used in the check stage
 
   keywords:
-    - to define
+    - undefined
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/is_object.yml

@@ -8,7 +8,7 @@ metadata:
     This helper function is typically used in the check stage
 
   keywords:
-    - to define
+    - undefined
 
 helper_type: filter
 

src/engine/test/helper_tests/helpers_description/filter/is_public_ip.yml

@@ -17,7 +17,7 @@ metadata:
     This helper function is typically used in the check stage
 
   keywords:
-    - to define
+    - undefined
 
 helper_type: filter