v0.16 (#335)

* Add Python extension lowering pass

* Add DocstringAttribute

* Add extension module codegen

* Handle different argument counts efficiently

* Add warnings to extension lowering

* Fix module name

* Fix extension codegen

* Fix argument check

* Auto-convert Codon exceptions to Python exceptions

* Fix #183

* Fix #162; Fix #135

* Fix #155

* Fix CPython interface in codegen

* Fix #191

* Fix #187

* Fix #189

* Generate object file in pyext mode

* Convert Codon exceptions to Python exceptions

* Fix vtable init; Fix failing tests on Linux

* Fix #190

* Fix #156

* Fix union routing

* Remove need for import python

* Automatic @export and wrapping for toplevel functions

* Reorganize API

* Add Python extension IR structs

* Add special calls for no-suspend yield-expr

* Add special calls for no-suspend yield-expr

* pyextension.h support [wip]

* pyextension.h support [wip]

* pyextension.h support

* pyextension.h support for toplevel functions

* clang-format

* Add PyFunction::nargs field

* Update pyextension codegen (WIP)

* SUpport nargs

* Add support for @pycapture

* PyType codegen (WIP)

* Py method codegen (WIP)

* Add type ptr hook

* Add getset codegen

* Add type alloc function

* Add type pointer hook codegen

* Re-organize codegen

* Add member codegen

* Update module init codegen

* Update module init codegen

* Add support for typePtrHook and new to/from_py hooks

* Fix extension codegen

* Fix init codegen

* Fix init codegen; add "tp_new" slot

* Fix type hook

* Add extra flags

* Specialized wrappers (PyType specs)

* Add static Python link option

* Fix C imports

* Add guards

* Remove unused field

* Python mode only when pyExt set

* Update python module

* Fix assert

* Update codegen/passes

* Fix tuple parsing in index expression

* Fix empty tuple unification

* Do not Cythonize underscore fns

* clang-format

* Fix switch

* Add Py support for cmp/setitem

* Add Py support for cmp/setitem

* Add type is support

* GetSet support

* clang-format

* GetSet support (fixes)

* Avoid useless vtable alloc

* Add iter support

* Fix size_t capture bug

* clang-format

* Fix POD type unification with tuples

* Add __try_from_py__ API

* Fix annotation

* Add static reflection methods (setattr; internal.static.*); refactor PyExt to python.codon; handle errors and kwargs in PyExt

* Python compat fixes

* Update Python object conversions

* Fix PyErrors

* clang-format; add copyright

* Add PyFunction::keywords field

* Fix JIT MRO handling; Refactor out Jupyter support

* Refactor out Jupyter support

* Add support for custom linking args (link=[]) to TOML plugins

* Fix tests

* Use g++ instead of gcc

* Fix Jupyter CMAKE

* Fix Jupyter CMAKE

* Add _PyArg_Parser definition

* Add complex64 type

* Add extra complex64 tests

* Fix Python calls; add staticenumerate

* Fix call

* Fix calls

* Update pyext wrappers

* Fix staticenumerate; Support static calls in tuple()

* Fix pyext routing

* Add add/mul for tuples

* clang-format

* Fix pyext codegen

* Fix wrap_multiple

* Add seq_alloc_atomic_uncollectable

* Fix default generics issue

* Add binary/ternary ops

* Fix missing generic issue

* Fix number slots

* Update pow

* Remove unnecessary pyobj

* Fix allocation

* Refactor errors

* Add test extension

* Fix formatting

* clang-format

* Fix getitem/setitem/delitem in pyext

* Fix pyext iterators

* Add builtin pow() (fix #294)

* Fix #244

* Fix #231

* Fix #229

* Fix #205

* Update docs

* Fix error message

* Add pyext tests

* Add pyext support for @Property

* Add pyext support for toplevel fns and @tuple classes

* More pyext tests

* More pyext tests

* Fix file error checking

* More pyext tests

* Update pyext tests

* Update docs

* Add pyext test to CI

* Add pyext support for @tuple.__new__

* Add pyext support for @tuple.__new__

* Fix hetero-tuple issue with fn_overloads

* More pyext tests

* Bump versions

* Fix del magic in pyext

* Fix init magic for tuples in pyext

* Have test-pypi only run on develop branch

* Make exception type indices unnamed-addr

* Fix #316; Fix #317 (slash issue)

* Use uncollectible-alloc for vtable

* Fix #249

* Add pyext docs

* Fix #249; Fix clashing vtables; Fix super() and class_copy

* Add content-atomic type property instruction

* __contents_atomic__ support

* Update internal functions

* Use PIC when generating Python extension

* Cleanup

* Add Dockerfile & fix -fPIC

* Cleanup

* Fix setup.py

* Fix pyext fn iteration

* Fix CI

* clang-format

* Update long conversions in Py bridge

* Support wide-int to str conversions

* Fix test

* Add pow for arbitrary-width ints

* Fix Linux backtraces

* Cleanup

* Add more tests

* Fix docs; Remove tuple.__add__ for scalars

* Update docs

---------

Co-authored-by: Ibrahim Numanagić <ibrahimpasa@gmail.com>

.github/workflows/ci.yml

@@ -172,6 +172,7 @@ jobs:
           ln -s build/libcodonrt.${LIBEXT} .
           build/codon_test
           test/app/test.sh build
+          (cd test/python && CODON_DIR=$(pwd)/../../codon-deploy python3 setup.py build_ext --inplace && python3 pyext.py)
         env:
           CODON_PATH: ./stdlib
           PYTHONPATH: .:./test/python
@@ -232,7 +233,7 @@ jobs:
           path: codon-linux-x86_64.tar.gz
 
       - name: Publish on TestPyPI
-        if: startsWith(matrix.os, 'ubuntu')
+        if: github.ref == 'refs/heads/develop' && startsWith(matrix.os, 'ubuntu')
         uses: pypa/gh-action-pypi-publish@release/v1
         with:
           user: __token__

.gitignore

@@ -17,6 +17,7 @@
 build/
 build_*/
 install/
+install_*/
 extra/python/src/jit.cpp
 extra/jupyter/build/
 

CMakeLists.txt

@@ -1,16 +1,15 @@
 cmake_minimum_required(VERSION 3.14)
 project(
   Codon
-  VERSION "0.15.5"
+  VERSION "0.16.0"
   HOMEPAGE_URL "https://github.com/exaloop/codon"
   DESCRIPTION "high-performance, extensible Python compiler")
-set(CODON_JIT_PYTHON_VERSION "0.1.3")
+set(CODON_JIT_PYTHON_VERSION "0.1.4")
 configure_file("${PROJECT_SOURCE_DIR}/cmake/config.h.in"
                "${PROJECT_SOURCE_DIR}/codon/config/config.h")
 configure_file("${PROJECT_SOURCE_DIR}/cmake/config.py.in"
                "${PROJECT_SOURCE_DIR}/extra/python/codon/version.py")
 
-option(CODON_JUPYTER "build Codon Jupyter server" OFF)
 option(CODON_GPU "build Codon GPU backend" OFF)
 
 set(CMAKE_CXX_STANDARD 17)
@@ -69,6 +68,10 @@ add_custom_command(
           omp
   DEPENDS peg2cpp codon/parser/peg/openmp.peg)
 
+# Codon Jupyter library
+set(CODON_JUPYTER_FILES codon/util/jupyter.h codon/util/jupyter.cpp)
+add_library(codon_jupyter SHARED ${CODON_JUPYTER_FILES})
+
 # Codon runtime library
 set(CODONRT_FILES codon/runtime/lib.h codon/runtime/lib.cpp
                   codon/runtime/re.cpp codon/runtime/exc.cpp
@@ -181,6 +184,7 @@ set(CODON_HPPFILES
     codon/cir/llvm/llvm.h
     codon/cir/llvm/optimize.h
     codon/cir/module.h
+    codon/cir/pyextension.h
     codon/cir/cir.h
     codon/cir/transform/cleanup/canonical.h
     codon/cir/transform/cleanup/dead_code.h
@@ -218,7 +222,6 @@ set(CODON_HPPFILES
     codon/cir/value.h
     codon/cir/var.h
     codon/util/common.h
-    extra/jupyter/jupyter.h
     codon/compiler/jit_extern.h)
 set(CODON_CPPFILES
     codon/compiler/compiler.cpp
@@ -320,16 +323,10 @@ set(CODON_CPPFILES
     codon/cir/util/visitor.cpp
     codon/cir/value.cpp
     codon/cir/var.cpp
-    codon/util/common.cpp
-    extra/jupyter/jupyter.cpp)
+    codon/util/common.cpp)
 add_library(codonc SHARED ${CODON_HPPFILES})
 target_include_directories(codonc PRIVATE ${peglib_SOURCE_DIR} ${toml_SOURCE_DIR}/include ${semver_SOURCE_DIR}/include)
 target_sources(codonc PRIVATE ${CODON_CPPFILES} codon_rules.cpp omp_rules.cpp)
-if(CODON_JUPYTER)
-  add_compile_definitions(CODON_JUPYTER)
-  add_dependencies(codonc xeus-static nlohmann_json)
-  target_link_libraries(codonc PRIVATE xeus-static)
-endif()
 if(ASAN)
   target_compile_options(
     codonc PRIVATE "-fno-omit-frame-pointer" "-fsanitize=address"
@@ -427,7 +424,7 @@ endif()
 
 # Codon command-line tool
 add_executable(codon codon/app/main.cpp)
-target_link_libraries(codon PUBLIC ${STATIC_LIBCPP} fmt codonc Threads::Threads)
+target_link_libraries(codon PUBLIC ${STATIC_LIBCPP} fmt codonc codon_jupyter Threads::Threads)
 
 # Codon test Download and unpack googletest at configure time
 include(FetchContent)
@@ -463,7 +460,7 @@ target_link_libraries(codon_test fmt codonc codonrt gtest_main)
 target_compile_definitions(codon_test
                            PRIVATE TEST_DIR="${CMAKE_CURRENT_SOURCE_DIR}/test")
 
-install(TARGETS codonrt codonc DESTINATION lib/codon)
+install(TARGETS codonrt codonc codon_jupyter DESTINATION lib/codon)
 install(FILES ${CMAKE_BINARY_DIR}/libomp${CMAKE_SHARED_LIBRARY_SUFFIX} DESTINATION lib/codon)
 install(TARGETS codon DESTINATION bin)
 install(DIRECTORY ${CMAKE_BINARY_DIR}/include/codon DESTINATION include)

cmake/deps.cmake

@@ -169,50 +169,3 @@ if(APPLE AND APPLE_ARM)
                 "LIBUNWIND_ENABLE_SHARED ON"
                 "LIBUNWIND_INCLUDE_DOCS OFF")
 endif()
-
-if(CODON_JUPYTER)
-    CPMAddPackage(
-        NAME libzmq
-        VERSION 4.3.4
-        URL https://github.com/zeromq/libzmq/releases/download/v4.3.4/zeromq-4.3.4.tar.gz
-        EXCLUDE_FROM_ALL YES
-        OPTIONS "WITH_PERF_TOOL OFF"
-                "ZMQ_BUILD_TESTS OFF"
-                "ENABLE_CPACK OFF"
-                "BUILD_SHARED ON"
-                "WITH_LIBSODIUM OFF"
-                "WITH_TLS OFF")
-    CPMAddPackage(
-        NAME cppzmq
-        URL https://github.com/zeromq/cppzmq/archive/refs/tags/v4.8.1.tar.gz
-        VERSION 4.8.1
-        EXCLUDE_FROM_ALL YES
-        OPTIONS "CPPZMQ_BUILD_TESTS OFF")
-    CPMAddPackage(
-        NAME xtl
-        GITHUB_REPOSITORY "xtensor-stack/xtl"
-        VERSION 0.7.3
-        GIT_TAG 0.7.3
-        EXCLUDE_FROM_ALL YES
-        OPTIONS "BUILD_TESTS OFF")
-    CPMAddPackage(
-        NAME json
-        GITHUB_REPOSITORY "nlohmann/json"
-        VERSION 3.10.1)
-    CPMAddPackage(
-        NAME xeus
-        GITHUB_REPOSITORY "jupyter-xeus/xeus"
-        VERSION 2.2.0
-        GIT_TAG 2.2.0
-        EXCLUDE_FROM_ALL YES
-        PATCH_COMMAND patch -N -u CMakeLists.txt -b ${CMAKE_SOURCE_DIR}/cmake/xeus.patch || true
-        OPTIONS "BUILD_EXAMPLES OFF"
-                "XEUS_BUILD_SHARED_LIBS OFF"
-                "XEUS_STATIC_DEPENDENCIES ON"
-                "CMAKE_POSITION_INDEPENDENT_CODE ON"
-                "XEUS_DISABLE_ARCH_NATIVE ON"
-                "XEUS_USE_DYNAMIC_UUID ${XEUS_USE_DYNAMIC_UUID}")
-    if (xeus_ADDED)
-        install(TARGETS nlohmann_json EXPORT xeus-targets)
-    endif()
-endif()

codon/app/main.cpp

@@ -4,6 +4,7 @@
 #include <cstdio>
 #include <cstdlib>
 #include <fstream>
+#include <functional>
 #include <iostream>
 #include <sstream>
 #include <string>
@@ -14,7 +15,9 @@
 #include "codon/compiler/error.h"
 #include "codon/compiler/jit.h"
 #include "codon/util/common.h"
+#include "codon/util/jupyter.h"
 #include "llvm/Support/CommandLine.h"
+#include "llvm/Support/FileSystem.h"
 
 namespace {
 void versMsg(llvm::raw_ostream &out) {
@@ -83,7 +86,7 @@ void initLogFlags(const llvm::cl::opt<std::string> &log) {
     codon::getLogger().parse(std::string(d));
 }
 
-enum BuildKind { LLVM, Bitcode, Object, Executable, Library, Detect };
+enum BuildKind { LLVM, Bitcode, Object, Executable, Library, PyExtension, Detect };
 enum OptMode { Debug, Release };
 enum Numerics { C, Python };
 } // namespace
@@ -109,8 +112,9 @@ int docMode(const std::vector<const char *> &args, const std::string &argv0) {
   return EXIT_SUCCESS;
 }
 
-std::unique_ptr<codon::Compiler> processSource(const std::vector<const char *> &args,
-                                               bool standalone) {
+std::unique_ptr<codon::Compiler> processSource(
+    const std::vector<const char *> &args, bool standalone,
+    std::function<bool()> pyExtension = [] { return false; }) {
   llvm::cl::opt<std::string> input(llvm::cl::Positional, llvm::cl::desc("<input file>"),
                                    llvm::cl::init("-"));
   auto regs = llvm::cl::getRegisteredOptions();
@@ -163,9 +167,9 @@ std::unique_ptr<codon::Compiler> processSource(const std::vector<const char *> &
 
   const bool isDebug = (optMode == OptMode::Debug);
   std::vector<std::string> disabledOptsVec(disabledOpts);
-  auto compiler = std::make_unique<codon::Compiler>(args[0], isDebug, disabledOptsVec,
-                                                    /*isTest=*/false,
-                                                    (numerics == Numerics::Python));
+  auto compiler = std::make_unique<codon::Compiler>(
+      args[0], isDebug, disabledOptsVec,
+      /*isTest=*/false, (numerics == Numerics::Python), pyExtension());
   compiler->getLLVMVisitor()->setStandalone(standalone);
 
   // load plugins
@@ -296,21 +300,27 @@ int buildMode(const std::vector<const char *> &args, const std::string &argv0) {
                                     llvm::cl::desc("Pass given flags to linker"));
   llvm::cl::opt<BuildKind> buildKind(
       llvm::cl::desc("output type"),
-      llvm::cl::values(clEnumValN(LLVM, "llvm", "Generate LLVM IR"),
-                       clEnumValN(Bitcode, "bc", "Generate LLVM bitcode"),
-                       clEnumValN(Object, "obj", "Generate native object file"),
-                       clEnumValN(Executable, "exe", "Generate executable"),
-                       clEnumValN(Library, "lib", "Generate shared library"),
-                       clEnumValN(Detect, "detect",
-                                  "Detect output type based on output file extension")),
+      llvm::cl::values(
+          clEnumValN(LLVM, "llvm", "Generate LLVM IR"),
+          clEnumValN(Bitcode, "bc", "Generate LLVM bitcode"),
+          clEnumValN(Object, "obj", "Generate native object file"),
+          clEnumValN(Executable, "exe", "Generate executable"),
+          clEnumValN(Library, "lib", "Generate shared library"),
+          clEnumValN(PyExtension, "pyext", "Generate Python extension module"),
+          clEnumValN(Detect, "detect",
+                     "Detect output type based on output file extension")),
       llvm::cl::init(Detect));
   llvm::cl::opt<std::string> output(
       "o",
       llvm::cl::desc(
           "Write compiled output to specified file. Supported extensions: "
           "none (executable), .o (object file), .ll (LLVM IR), .bc (LLVM bitcode)"));
+  llvm::cl::opt<std::string> pyModule(
+      "module", llvm::cl::desc("Python extension module name (only applicable when "
+                               "building Python extension module)"));
 
-  auto compiler = processSource(args, /*standalone=*/true);
+  auto compiler = processSource(args, /*standalone=*/true,
+                                [&] { return buildKind == BuildKind::PyExtension; });
   if (!compiler)
     return EXIT_FAILURE;
   std::vector<std::string> libsVec(libs);
@@ -326,6 +336,7 @@ int buildMode(const std::vector<const char *> &args, const std::string &argv0) {
     extension = ".bc";
     break;
   case BuildKind::Object:
+  case BuildKind::PyExtension:
     extension = ".o";
     break;
   case BuildKind::Library:
@@ -358,6 +369,12 @@ int buildMode(const std::vector<const char *> &args, const std::string &argv0) {
     compiler->getLLVMVisitor()->writeToExecutable(filename, argv0, true, libsVec,
                                                   lflags);
     break;
+  case BuildKind::PyExtension:
+    compiler->getCache()->pyModule->name =
+        pyModule.empty() ? llvm::sys::path::stem(compiler->getInput()).str() : pyModule;
+    compiler->getLLVMVisitor()->writeToPythonExtension(*compiler->getCache()->pyModule,
+                                                       filename);
+    break;
   case BuildKind::Detect:
     compiler->getLLVMVisitor()->compile(filename, argv0, libsVec, lflags);
     break;
@@ -368,15 +385,7 @@ int buildMode(const std::vector<const char *> &args, const std::string &argv0) {
   return EXIT_SUCCESS;
 }
 
-#ifdef CODON_JUPYTER
-namespace codon {
-int startJupyterKernel(const std::string &argv0,
-                       const std::vector<std::string> &plugins,
-                       const std::string &configPath);
-}
-#endif
 int jupyterMode(const std::vector<const char *> &args) {
-#ifdef CODON_JUPYTER
   llvm::cl::list<std::string> plugins("plugin",
                                       llvm::cl::desc("Load specified plugin"));
   llvm::cl::opt<std::string> input(llvm::cl::Positional,
@@ -385,11 +394,6 @@ int jupyterMode(const std::vector<const char *> &args) {
   llvm::cl::ParseCommandLineOptions(args.size(), args.data());
   int code = codon::startJupyterKernel(args[0], plugins, input);
   return code;
-#else
-  fmt::print("Jupyter support not included. Please recompile with "
-             "-DCODON_JUPYTER.");
-  return EXIT_FAILURE;
-#endif
 }
 
 void showCommandsAndExit() {

codon/cir/attribute.cpp

@@ -41,6 +41,8 @@ std::ostream &MemberAttribute::doFormat(std::ostream &os) const {
 
 const std::string SrcInfoAttribute::AttributeName = "srcInfoAttribute";
 
+const std::string DocstringAttribute::AttributeName = "docstringAttribute";
+
 const std::string TupleLiteralAttribute::AttributeName = "tupleLiteralAttribute";
 
 std::unique_ptr<Attribute> TupleLiteralAttribute::clone(util::CloneVisitor &cv) const {

codon/cir/attribute.h

@@ -64,6 +64,26 @@ struct SrcInfoAttribute : public Attribute {
   std::ostream &doFormat(std::ostream &os) const override { return os << info; }
 };
 
+/// Attribute containing docstring from source
+struct DocstringAttribute : public Attribute {
+  static const std::string AttributeName;
+
+  /// the docstring
+  std::string docstring;
+
+  DocstringAttribute() = default;
+  /// Constructs a DocstringAttribute.
+  /// @param docstring the docstring
+  explicit DocstringAttribute(const std::string &docstring) : docstring(docstring) {}
+
+  std::unique_ptr<Attribute> clone(util::CloneVisitor &cv) const override {
+    return std::make_unique<DocstringAttribute>(*this);
+  }
+
+private:
+  std::ostream &doFormat(std::ostream &os) const override { return os << docstring; }
+};
+
 /// Attribute containing function information
 struct KeyValueAttribute : public Attribute {
   static const std::string AttributeName;

codon/cir/instr.cpp

@@ -114,6 +114,8 @@ types::Type *TypePropertyInstr::doGetType() const {
   switch (property) {
   case Property::IS_ATOMIC:
     return getModule()->getBoolType();
+  case Property::IS_CONTENT_ATOMIC:
+    return getModule()->getBoolType();
   case Property::SIZEOF:
     return getModule()->getIntType();
   default:

codon/cir/instr.h

@@ -269,7 +269,7 @@ class StackAllocInstr : public AcceptorExtend<StackAllocInstr, Instr> {
 /// Instr representing getting information about a type.
 class TypePropertyInstr : public AcceptorExtend<TypePropertyInstr, Instr> {
 public:
-  enum Property { IS_ATOMIC, SIZEOF };
+  enum Property { IS_ATOMIC, IS_CONTENT_ATOMIC, SIZEOF };
 
 private:
   /// the type being inspected