[libc][hdrgen] Allow to treat hdrgen Python code as a Python module. #128955
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
February 26, 2025 13:23
Move the hdrgen code under a subdirectory to treat it as a Python module. This mimics the structure used by llvm/utils/lit and llvm/utils/mlgo-utils and simplifies integration of hdrgen to the build system which rely on Python modules. In addition to that, it clarifies which imports are coming from the hdrgen-specific helpers (e.g. "from type import ..." becomes "from hdrgen.type import ...". Leave the entrypoints (top-level main.py and yaml_to_classes.py) as-is: they can keep being referred by the CMake build system w/o any changes.
|
@llvm/pr-subscribers-libc Author: Alexey Samsonov (vonosmas) ChangesMove the hdrgen code under a subdirectory to treat it as a Python module. This mimics the structure used by llvm/utils/lit and Leave the entrypoints (top-level main.py and yaml_to_classes.py) as-is: Patch is 32.82 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/128955.diff 14 Files Affected:
diff --git a/libc/docs/dev/header_generation.rst b/libc/docs/dev/header_generation.rst
index a946106fc7097..2ffb4fffe6e0e 100644
--- a/libc/docs/dev/header_generation.rst
+++ b/libc/docs/dev/header_generation.rst
@@ -63,7 +63,7 @@ To add through the command line:
examine.
If you want to sort the functions alphabetically you can check out
-libc/utils/hdrgen/yaml_functions_sorted.py.
+``libc/utils/hdrgen/hdrgen/yaml_functions_sorted.py``.
Testing
@@ -90,7 +90,7 @@ Common Errors
.. code-block:: none
- "/llvm-project/libc/utils/hdrgen/yaml_to_classes.py", line 67, in yaml_to_classes function_data["return_type"]
+ "/llvm-project/libc/utils/hdrgen/hdrgen/yaml_to_classes.py", line 67, in yaml_to_classes function_data["return_type"]
If you receive this error or any error pertaining to
``function_data[function_specific_component]`` while building the headers
@@ -107,9 +107,9 @@ Common Errors
CMake Error at:
/llvm-project/libc/cmake/modules/LLVMLibCHeaderRules.cmake:86 (message):
- 'add_gen_hdr2' rule requires GEN_HDR to be specified.
+ 'add_gen_hdr' rule requires GEN_HDR to be specified.
Call Stack (most recent call first):
- /llvm-project/libc/include/CMakeLists.txt:22 (add_gen_header2)
+ /llvm-project/libc/include/CMakeLists.txt:22 (add_gen_header)
/llvm-project/libc/include/CMakeLists.txt:62 (add_header_macro)
If you receive this error, there is a missing YAML file, h_def file, or
@@ -119,7 +119,6 @@ Common Errors
| ``[header_name]``
| ``[../libc/include/[yaml_file.yaml]``
- | ``[header_name.h.def]``
| ``[header_name.h]``
| ``DEPENDS``
| ``{Necessary Depend Files}``
@@ -148,13 +147,13 @@ Common Errors
.. code-block:: none
- File "/llvm-project/libc/utils/hdrgen/header.py", line 60, in __str__ for
+ File "/llvm-project/libc/utils/hdrgen/hdrgen/header.py", line 60, in __str__ for
function in self.functions: AttributeError: 'HeaderFile' object has no
attribute 'functions'
When running ``ninja libc`` in the build directory to generate headers you
may receive the error above. Essentially this means that in
- ``libc/utils/hdrgen/header.py`` there is a missing attribute named functions.
+ ``libc/utils/hdrgen/hgrgen/header.py`` there is a missing attribute named functions.
Make sure all function components are defined within this file and there are
no missing functions to add these components.
diff --git a/libc/utils/hdrgen/hdrgen/__init__.py b/libc/utils/hdrgen/hdrgen/__init__.py
new file mode 100644
index 0000000000000..44390e67aacf4
--- /dev/null
+++ b/libc/utils/hdrgen/hdrgen/__init__.py
@@ -0,0 +1,3 @@
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
diff --git a/libc/utils/hdrgen/enumeration.py b/libc/utils/hdrgen/hdrgen/enumeration.py
similarity index 100%
rename from libc/utils/hdrgen/enumeration.py
rename to libc/utils/hdrgen/hdrgen/enumeration.py
diff --git a/libc/utils/hdrgen/function.py b/libc/utils/hdrgen/hdrgen/function.py
similarity index 99%
rename from libc/utils/hdrgen/function.py
rename to libc/utils/hdrgen/hdrgen/function.py
index 28af05f78d897..f039996584e31 100644
--- a/libc/utils/hdrgen/function.py
+++ b/libc/utils/hdrgen/hdrgen/function.py
@@ -8,7 +8,7 @@
import re
from functools import total_ordering
-from type import Type
+from hdrgen.type import Type
# These are the keywords that appear in C type syntax but are not part of the
diff --git a/libc/utils/hdrgen/gpu_headers.py b/libc/utils/hdrgen/hdrgen/gpu_headers.py
similarity index 98%
rename from libc/utils/hdrgen/gpu_headers.py
rename to libc/utils/hdrgen/hdrgen/gpu_headers.py
index 505adfa8fee8a..290bfe0649135 100644
--- a/libc/utils/hdrgen/gpu_headers.py
+++ b/libc/utils/hdrgen/hdrgen/gpu_headers.py
@@ -6,7 +6,7 @@
#
# ==-------------------------------------------------------------------------==#
-from header import HeaderFile
+from hdrgen.header import HeaderFile
class GpuHeaderFile(HeaderFile):
diff --git a/libc/utils/hdrgen/header.py b/libc/utils/hdrgen/hdrgen/header.py
similarity index 100%
rename from libc/utils/hdrgen/header.py
rename to libc/utils/hdrgen/hdrgen/header.py
diff --git a/libc/utils/hdrgen/macro.py b/libc/utils/hdrgen/hdrgen/macro.py
similarity index 100%
rename from libc/utils/hdrgen/macro.py
rename to libc/utils/hdrgen/hdrgen/macro.py
diff --git a/libc/utils/hdrgen/hdrgen/main.py b/libc/utils/hdrgen/hdrgen/main.py
new file mode 100755
index 0000000000000..25df41e506a1f
--- /dev/null
+++ b/libc/utils/hdrgen/hdrgen/main.py
@@ -0,0 +1,134 @@
+#!/usr/bin/env python3
+#
+# ===- Generate headers for libc functions ------------------*- python -*--==#
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ==------------------------------------------------------------------------==#
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+from hdrgen.header import HeaderFile
+from hdrgen.yaml_to_classes import load_yaml_file, fill_public_api
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Generate header files from YAML")
+ parser.add_argument(
+ "yaml_file",
+ help="Path to the YAML file containing header specification",
+ metavar="FILE",
+ type=Path,
+ nargs="+",
+ )
+ parser.add_argument(
+ "-o",
+ "--output",
+ help="Path to write generated header file",
+ type=Path,
+ required=True,
+ )
+ parser.add_argument(
+ "--json",
+ help="Write JSON instead of a header, can use multiple YAML files",
+ action="store_true",
+ )
+ parser.add_argument(
+ "--depfile",
+ help="Path to write a depfile",
+ type=Path,
+ )
+ parser.add_argument(
+ "--write-if-changed",
+ help="Write the output file only if its contents have changed",
+ action="store_true",
+ default=False,
+ )
+ parser.add_argument(
+ "-e",
+ "--entry-point",
+ help="Entry point to include; may be given many times",
+ metavar="SYMBOL",
+ action="append",
+ )
+ args = parser.parse_args()
+
+ if not args.json and len(args.yaml_file) != 1:
+ print("Only one YAML file at a time without --json", file=sys.stderr)
+ parser.print_usage(sys.stderr)
+ return 2
+
+ files_read = set()
+
+ def write_depfile():
+ if not args.depfile:
+ return
+ deps = " ".join(str(f) for f in sorted(files_read))
+ args.depfile.parent.mkdir(parents=True, exist_ok=True)
+ with open(args.depfile, "w") as depfile:
+ depfile.write(f"{args.output}: {deps}\n")
+
+ def load_yaml(path):
+ files_read.add(path)
+ return load_yaml_file(path, HeaderFile, args.entry_point)
+
+ def load_header(yaml_file):
+ merge_from_files = dict()
+
+ def merge_from(paths):
+ for path in paths:
+ # Load each file exactly once, in case of redundant merges.
+ if path in merge_from_files:
+ continue
+ header = load_yaml(path)
+ merge_from_files[path] = header
+ merge_from(path.parent / f for f in header.merge_yaml_files)
+
+ # Load the main file first.
+ header = load_yaml(yaml_file)
+
+ # Now load all the merge_yaml_files, and transitive merge_yaml_files.
+ merge_from(yaml_file.parent / f for f in header.merge_yaml_files)
+
+ # Merge in all those files' contents.
+ for merge_from_path, merge_from_header in merge_from_files.items():
+ if merge_from_header.name is not None:
+ print(
+ f"{merge_from_path!s}: Merge file cannot have header field",
+ file=sys.stderr,
+ )
+ return 2
+ header.merge(merge_from_header)
+
+ return header
+
+ if args.json:
+ contents = json.dumps(
+ [load_header(file).json_data() for file in args.yaml_file],
+ indent=2,
+ )
+ else:
+ [yaml_file] = args.yaml_file
+ header = load_header(yaml_file)
+ # The header_template path is relative to the containing YAML file.
+ template = header.template(yaml_file.parent, files_read)
+ contents = fill_public_api(header.public_api(), template)
+
+ write_depfile()
+
+ if (
+ not args.write_if_changed
+ or not args.output.exists()
+ or args.output.read_text() != contents
+ ):
+ args.output.parent.mkdir(parents=True, exist_ok=True)
+ args.output.write_text(contents)
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/libc/utils/hdrgen/object.py b/libc/utils/hdrgen/hdrgen/object.py
similarity index 100%
rename from libc/utils/hdrgen/object.py
rename to libc/utils/hdrgen/hdrgen/object.py
diff --git a/libc/utils/hdrgen/type.py b/libc/utils/hdrgen/hdrgen/type.py
similarity index 100%
rename from libc/utils/hdrgen/type.py
rename to libc/utils/hdrgen/hdrgen/type.py
diff --git a/libc/utils/hdrgen/yaml_functions_sorted.py b/libc/utils/hdrgen/hdrgen/yaml_functions_sorted.py
similarity index 100%
rename from libc/utils/hdrgen/yaml_functions_sorted.py
rename to libc/utils/hdrgen/hdrgen/yaml_functions_sorted.py
diff --git a/libc/utils/hdrgen/hdrgen/yaml_to_classes.py b/libc/utils/hdrgen/hdrgen/yaml_to_classes.py
new file mode 100644
index 0000000000000..ebe7781d449f7
--- /dev/null
+++ b/libc/utils/hdrgen/hdrgen/yaml_to_classes.py
@@ -0,0 +1,285 @@
+#!/usr/bin/env python3
+#
+# ===- Generate headers for libc functions -------------------*- python -*--==#
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ==-------------------------------------------------------------------------==#
+
+import yaml
+import argparse
+from pathlib import Path
+
+from hdrgen.enumeration import Enumeration
+from hdrgen.function import Function
+from hdrgen.gpu_headers import GpuHeaderFile as GpuHeader
+from hdrgen.header import HeaderFile
+from hdrgen.macro import Macro
+from hdrgen.object import Object
+from hdrgen.type import Type
+
+
+def yaml_to_classes(yaml_data, header_class, entry_points=None):
+ """
+ Convert YAML data to header classes.
+
+ Args:
+ yaml_data: The YAML data containing header specifications.
+ header_class: The class to use for creating the header.
+ entry_points: A list of specific function names to include in the header.
+
+ Returns:
+ HeaderFile: An instance of HeaderFile populated with the data.
+ """
+ header_name = yaml_data.get("header")
+ header = header_class(header_name)
+ header.template_file = yaml_data.get("header_template")
+ header.standards = yaml_data.get("standards", [])
+ header.merge_yaml_files = yaml_data.get("merge_yaml_files", [])
+
+ for macro_data in yaml_data.get("macros", []):
+ header.add_macro(
+ Macro(
+ macro_data["macro_name"],
+ macro_data.get("macro_value"),
+ macro_data.get("macro_header"),
+ )
+ )
+
+ types = yaml_data.get("types", [])
+ sorted_types = sorted(types, key=lambda x: x["type_name"])
+ for type_data in sorted_types:
+ header.add_type(Type(type_data["type_name"]))
+
+ for enum_data in yaml_data.get("enums", []):
+ header.add_enumeration(
+ Enumeration(enum_data["name"], enum_data.get("value", None))
+ )
+
+ functions = yaml_data.get("functions", [])
+ if entry_points:
+ entry_points_set = set(entry_points)
+ functions = [f for f in functions if f["name"] in entry_points_set]
+ sorted_functions = sorted(functions, key=lambda x: x["name"])
+ guards = []
+ guarded_function_dict = {}
+ for function_data in sorted_functions:
+ guard = function_data.get("guard", None)
+ if guard is None:
+ arguments = [arg["type"] for arg in function_data["arguments"]]
+ attributes = function_data.get("attributes", None)
+ standards = function_data.get("standards", None)
+ header.add_function(
+ Function(
+ function_data["return_type"],
+ function_data["name"],
+ arguments,
+ standards,
+ guard,
+ attributes,
+ )
+ )
+ else:
+ if guard not in guards:
+ guards.append(guard)
+ guarded_function_dict[guard] = []
+ guarded_function_dict[guard].append(function_data)
+ else:
+ guarded_function_dict[guard].append(function_data)
+ sorted_guards = sorted(guards)
+ for guard in sorted_guards:
+ for function_data in guarded_function_dict[guard]:
+ arguments = [arg["type"] for arg in function_data["arguments"]]
+ attributes = function_data.get("attributes", None)
+ standards = function_data.get("standards", None)
+ header.add_function(
+ Function(
+ function_data["return_type"],
+ function_data["name"],
+ arguments,
+ standards,
+ guard,
+ attributes,
+ )
+ )
+
+ objects = yaml_data.get("objects", [])
+ sorted_objects = sorted(objects, key=lambda x: x["object_name"])
+ for object_data in sorted_objects:
+ header.add_object(
+ Object(object_data["object_name"], object_data["object_type"])
+ )
+
+ return header
+
+
+def load_yaml_file(yaml_file, header_class, entry_points):
+ """
+ Load YAML file and convert it to header classes.
+
+ Args:
+ yaml_file: Path to the YAML file.
+ header_class: The class to use for creating the header (HeaderFile or GpuHeader).
+ entry_points: A list of specific function names to include in the header.
+
+ Returns:
+ HeaderFile: An instance of HeaderFile populated with the data.
+ """
+ with yaml_file.open() as f:
+ yaml_data = yaml.safe_load(f)
+ return yaml_to_classes(yaml_data, header_class, entry_points)
+
+
+def fill_public_api(header_str, h_def_content):
+ """
+ Replace the %%public_api() placeholder in the .h.def content with the generated header content.
+
+ Args:
+ header_str: The generated header string.
+ h_def_content: The content of the .h.def file.
+
+ Returns:
+ The final header content with the public API filled in.
+ """
+ header_str = header_str.strip()
+ return h_def_content.replace("%%public_api()", header_str, 1)
+
+
+def parse_function_details(details):
+ """
+ Parse function details from a list of strings and return a Function object.
+
+ Args:
+ details: A list containing function details
+
+ Returns:
+ Function: An instance of Function initialized with the details.
+ """
+ return_type, name, arguments, standards, guard, attributes = details
+ standards = standards.split(",") if standards != "null" else []
+ arguments = [arg.strip() for arg in arguments.split(",")]
+ attributes = attributes.split(",") if attributes != "null" else []
+
+ return Function(
+ return_type=return_type,
+ name=name,
+ arguments=arguments,
+ standards=standards,
+ guard=guard if guard != "null" else None,
+ attributes=attributes if attributes else [],
+ )
+
+
+def add_function_to_yaml(yaml_file, function_details):
+ """
+ Add a function to the YAML file.
+
+ Args:
+ yaml_file: The path to the YAML file.
+ function_details: A list containing function details (return_type, name, arguments, standards, guard, attributes).
+ """
+ new_function = parse_function_details(function_details)
+
+ with open(yaml_file, "r") as f:
+ yaml_data = yaml.safe_load(f)
+ if "functions" not in yaml_data:
+ yaml_data["functions"] = []
+
+ function_dict = {
+ "name": new_function.name,
+ "standards": new_function.standards,
+ "return_type": new_function.return_type,
+ "arguments": [{"type": arg} for arg in new_function.arguments],
+ }
+
+ if new_function.guard:
+ function_dict["guard"] = new_function.guard
+
+ if new_function.attributes:
+ function_dict["attributes"] = new_function.attributes
+
+ insert_index = 0
+ for i, func in enumerate(yaml_data["functions"]):
+ if func["name"] > new_function.name:
+ insert_index = i
+ break
+ else:
+ insert_index = len(yaml_data["functions"])
+
+ yaml_data["functions"].insert(insert_index, function_dict)
+
+ class IndentYamlListDumper(yaml.Dumper):
+ def increase_indent(self, flow=False, indentless=False):
+ return super(IndentYamlListDumper, self).increase_indent(flow, False)
+
+ with open(yaml_file, "w") as f:
+ yaml.dump(
+ yaml_data,
+ f,
+ Dumper=IndentYamlListDumper,
+ default_flow_style=False,
+ sort_keys=False,
+ )
+
+ print(f"Added function {new_function.name} to {yaml_file}")
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Generate header files from YAML")
+ parser.add_argument(
+ "yaml_file", help="Path to the YAML file containing header specification"
+ )
+ parser.add_argument(
+ "--output_dir",
+ help="Directory to output the generated header file",
+ )
+ parser.add_argument(
+ "--add_function",
+ nargs=6,
+ metavar=(
+ "RETURN_TYPE",
+ "NAME",
+ "ARGUMENTS",
+ "STANDARDS",
+ "GUARD",
+ "ATTRIBUTES",
+ ),
+ help="Add a function to the YAML file",
+ )
+ parser.add_argument(
+ "--entry-point",
+ action="append",
+ help="Entry point to include",
+ dest="entry_points",
+ )
+ parser.add_argument(
+ "--export-decls",
+ action="store_true",
+ help="Flag to use GpuHeader for exporting declarations",
+ )
+ args = parser.parse_args()
+
+ if args.add_function:
+ add_function_to_yaml(args.yaml_file, args.add_function)
+
+ header_class = GpuHeader if args.export_decls else HeaderFile
+ header = load_yaml_file(Path(args.yaml_file), header_class, args.entry_points)
+
+ header_str = str(header)
+
+ if args.output_dir:
+ output_file_path = Path(args.output_dir)
+ if output_file_path.is_dir():
+ output_file_path /= f"{Path(args.yaml_file).stem}.h"
+ else:
+ output_file_path = Path(f"{Path(args.yaml_file).stem}.h")
+
+ if args.export_decls:
+ with open(output_file_path, "w") as f:
+ f.write(header_str)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/libc/utils/hdrgen/main.py b/libc/utils/hdrgen/main.py
old mode 100755
new mode 100644
index d5a1c25e7ce20..3e53661573392
--- a/libc/utils/hdrgen/main.py
+++ b/libc/utils/hdrgen/main.py
@@ -8,127 +8,7 @@
#
# ==------------------------------------------------------------------------==#
-import argparse
-import json
-import sys
-from pathlib import Path
-
-from header import HeaderFile
-from yaml_to_classes import load_yaml_file, fill_public_api
-
-
-def main():
- parser = argparse.ArgumentParser(description="Generate header files from YAML")
- parser.add_argument(
- "yaml_file",
- help="Path to the YAML file containing header specification",
- metavar="FILE",
- type=Path,
- nargs="+",
- )
- parser.add_argument(
- "-o",
- "--output",
- help="Path to write generated header file",
- type=Path,
- required=True,
- )
- parser.add_argument(
- "--json",
- help="Write JSON instead of a header, can use multiple YAML files"...
[truncated]
|
michaelrj-google
approved these changes
Feb 26, 2025
|
(proceeding per previous LGTM for direction from Tue on Discord and Roland confirming offline it won't interfere with their downstream build system too much) |
joaosaffran
pushed a commit
to joaosaffran/llvm-project
that referenced
this pull request
Mar 3, 2025
…lvm#128955) Move the hdrgen code under a subdirectory to treat it as a Python module. This mimics the structure used by llvm/utils/lit and llvm/utils/mlgo-utils and simplifies integration of hdrgen to the build system which rely on Python modules. In addition to that, it clarifies which imports are coming from the hdrgen-specific helpers (e.g. "from type import ..." becomes "from hdrgen.type import ...". Leave the entrypoints (top-level main.py and yaml_to_classes.py) as-is: they can keep being referred by the CMake build system w/o any changes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Move the hdrgen code under a subdirectory to treat it as a Python module.
This mimics the structure used by llvm/utils/lit and
llvm/utils/mlgo-utils and simplifies integration of hdrgen to the build
system which rely on Python modules. In addition to that, it clarifies
which imports are coming from the hdrgen-specific helpers (e.g. "from
type import ..." becomes "from hdrgen.type import ...".
Leave the entrypoints (top-level main.py and yaml_to_classes.py) as-is:
they can keep being referred by the CMake build system w/o any changes.