[llvm] Add a tool to check mustache compliance against the public spec

[ilovepi]

This is a cli tool to that tests the conformance of LLVM's mustache
implementation against the public Mustache spec, hosted at
https://github.com/mustache/spec. This is a revised version of the
patches in #111487.

Co-authored-by: Peter Chou peter.chou@mail.utoronto.ca

[ilovepi]

• [llvm] Add a tool to check mustache compliance against the public spec #142813 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[llvmbot]

@llvm/pr-subscribers-llvm-binary-utilities

Author: Paul Kirth (ilovepi)

Changes

This is a cli tool to that tests the conformance of LLVM's mustache
implementation against the public Mustache spec, hosted at
https://github.com/mustache/spec. This is a revised version of the
patches in #111487.

Co-authored-by: Peter Chou <peter.chou@mail.utoronto.ca>

---

Full diff: https://github.com/llvm/llvm-project/pull/142813.diff

6 Files Affected:

• (modified) llvm/CMakeLists.txt (+1)
• (modified) llvm/docs/CommandGuide/index.rst (+1)
• (added) llvm/docs/CommandGuide/llvm-mustachespec.rst (+13)
• (added) llvm/utils/llvm-mustachespec/CMakeLists.txt (+5)
• (added) llvm/utils/llvm-mustachespec/\ (+108)
• (added) llvm/utils/llvm-mustachespec/llvm-mustachespec.cpp (+157)

diff --git a/llvm/CMakeLists.txt b/llvm/CMakeLists.txt
index ed44b16bf9aeb..cb5c9f10e419d 100644
--- a/llvm/CMakeLists.txt
+++ b/llvm/CMakeLists.txt
@@ -1302,6 +1302,7 @@ if( LLVM_INCLUDE_UTILS )
   add_subdirectory(utils/yaml-bench)
   add_subdirectory(utils/split-file)
   add_subdirectory(utils/mlgo-utils)
+  add_subdirectory(utils/llvm-mustachespec)
   if( LLVM_INCLUDE_TESTS )
     set(LLVM_SUBPROJECT_TITLE "Third-Party/Google Test")
     add_subdirectory(${LLVM_THIRD_PARTY_DIR}/unittest ${CMAKE_CURRENT_BINARY_DIR}/third-party/unittest)
diff --git a/llvm/docs/CommandGuide/index.rst b/llvm/docs/CommandGuide/index.rst
index 643951eca2a26..ae8fff1574ad0 100644
--- a/llvm/docs/CommandGuide/index.rst
+++ b/llvm/docs/CommandGuide/index.rst
@@ -87,6 +87,7 @@ Developer Tools
    llvm-exegesis
    llvm-ifs
    llvm-locstats
+   llvm-mustachespec
    llvm-pdbutil
    llvm-profgen
    llvm-tli-checker
diff --git a/llvm/docs/CommandGuide/llvm-mustachespec.rst b/llvm/docs/CommandGuide/llvm-mustachespec.rst
new file mode 100644
index 0000000000000..498928c12e4f2
--- /dev/null
+++ b/llvm/docs/CommandGuide/llvm-mustachespec.rst
@@ -0,0 +1,13 @@
+llvm-mustachespec - LLVM tool to test Mustache Compliance Library
+=================================================================
+
+llvm-mustachespec test the mustache spec conformance of the LLVM
+mustache library. The spec can be found here https://github.com/mustache/spec
+
+    $ llvm-mustachespec input-file
+
+.. program:: llvm-mustachespec
+
+Outputs the number of tests failures and success in the spec
+
+
diff --git a/llvm/utils/llvm-mustachespec/CMakeLists.txt b/llvm/utils/llvm-mustachespec/CMakeLists.txt
new file mode 100644
index 0000000000000..2a94eda32c9bb
--- /dev/null
+++ b/llvm/utils/llvm-mustachespec/CMakeLists.txt
@@ -0,0 +1,5 @@
+add_llvm_utility(llvm-mustachespec
+  llvm-mustachespec.cpp
+  )
+
+target_link_libraries(llvm-mustachespec PRIVATE LLVMSupport)
diff --git "a/llvm/utils/llvm-mustachespec/\\" "b/llvm/utils/llvm-mustachespec/\\"
new file mode 100644
index 0000000000000..2161410e06428
--- /dev/null
+++ "b/llvm/utils/llvm-mustachespec/\\"
@@ -0,0 +1,108 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Simple drivers to test the mustache spec found here
+// https://github.com/mustache/
+// It is used to verify that the current implementation conforms to the spec
+// Simply download the spec and pass the test files to the driver
+//
+// Currently Triple Mustache is not supported, so we expect the following spec
+// test to fail:
+//    Triple Mustache
+//    Triple Mustache Integer Interpolation
+//    Triple Mustache Decimal Interpolation
+//    Triple Mustache Null Interpolation
+//    Triple Mustache Context Miss Interpolation
+//    Dotted Names - Triple Mustache Interpolation
+//    Implicit Iterators - Triple Mustache
+//    Triple Mustache - Surrounding Whitespace
+//    Triple Mustache - Standalone
+//    Triple Mustache With Padding
+//    Standalone Indentation
+//    Implicit Iterator - Triple mustache
+//
+// Usage:
+//  mustache path/to/test/file/test.json path/to/test/file/test2.json ...
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Mustache.h"
+#include "llvm/Support/WithColor.h"
+#include <string>
+
+using namespace llvm;
+using namespace llvm::json;
+using namespace llvm::mustache;
+
+cl::list<std::string> InputFiles(cl::Positional, cl::desc("<input files>"),
+                                 cl::OneOrMore);
+
+static ExitOnError ExitOnErr;
+
+void runThroughTest(StringRef InputFile) {
+  outs() << "Running Tests: " << InputFile << "\n";
+  std::unique_ptr<MemoryBuffer> Buffer =
+      ExitOnErr(errorOrToExpected(MemoryBuffer::getFile(InputFile)));
+
+  StringRef FileContent = Buffer->getBuffer();
+  json::Value Json = ExitOnErr(parse(FileContent));
+
+  // Get test
+  Array *Obj = Json.getAsObject()->getArray("tests");
+  size_t Total = 0;
+  size_t Success = 0;
+  for (Value V : *Obj) {
+    Object *TestCase = V.getAsObject();
+    StringRef TemplateStr = TestCase->getString("template").value();
+    StringRef ExpectedStr = TestCase->getString("expected").value();
+    StringRef Name = TestCase->getString("name").value();
+    Value *Data = TestCase->get("data");
+    Value *Partials = TestCase->get("partials");
+
+    if (!Data)
+      continue;
+
+    Template T = Template(TemplateStr);
+    if (Partials) {
+      for (auto &PartialPairs : *Partials->getAsObject()) {
+        const auto &[Partial, Str] = PartialPairs;
+        T.registerPartial(Str.getAsString()->str(), Partial.str());
+      }
+    }
+
+    std::string ActualStr;
+    raw_string_ostream OS(ActualStr);
+    T.render(*Data, OS);
+    if (ExpectedStr == ActualStr) {
+      Success++;
+    } else {
+      llvm::errs() << "Template: " << TemplateStr <<"\n";
+      if (Partials)
+        Partials->print(llvm::errs());
+      llvm::errs() << "JSON Data: "; Data->print(errs()); errs() << "\n";
+      outs() << "Test Failed: " << Name << "\n"
+             << "  Expected: \'" << ExpectedStr << "\'\n"
+             << "  Actual: \'" << ActualStr << "\'\n";
+      llvm::errs() << "==========\n";
+    }
+    Total++;
+  }
+
+  outs() << "Result " << Success << "/" << Total << " succeeded\n";
+}
+
+int main(int argc, char **argv) {
+  ExitOnErr.setBanner(std::string(argv[0]) + " error:");
+  cl::ParseCommandLineOptions(argc, argv);
+  for (const auto &FileName : InputFiles) {
+    runThroughTest(FileName);
+  }
+  return 0;
+}
diff --git a/llvm/utils/llvm-mustachespec/llvm-mustachespec.cpp b/llvm/utils/llvm-mustachespec/llvm-mustachespec.cpp
new file mode 100644
index 0000000000000..acd090a6d3abe
--- /dev/null
+++ b/llvm/utils/llvm-mustachespec/llvm-mustachespec.cpp
@@ -0,0 +1,157 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Simple drivers to test the mustache spec found at
+// https://github.com/mustache/
+//
+// It is used to verify that the current implementation conforms to the spec.
+// Simply download the spec and pass the test files to the driver.
+//
+// The current implementation only supports non-optional parts of the spec, so
+// we do not expect any of the dynamic-names, inheritance, or lambda tests to
+// pass. Additionally, Triple Mustache is not supported, so we expect the
+// following tests to fail:
+//    Triple Mustache
+//    Triple Mustache Integer Interpolation
+//    Triple Mustache Decimal Interpolation
+//    Triple Mustache Null Interpolation
+//    Triple Mustache Context Miss Interpolation
+//    Dotted Names - Triple Mustache Interpolation
+//    Implicit Iterators - Triple Mustache
+//    Triple Mustache - Surrounding Whitespace
+//    Triple Mustache - Standalone
+//    Triple Mustache With Padding
+//    Standalone Indentation
+//    Implicit Iterator - Triple mustache
+//
+// Usage:
+//  llvm-mustachespec path/to/test/file.json path/to/test/file2.json ...
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Mustache.h"
+#include <string>
+
+using namespace llvm;
+using namespace llvm::json;
+using namespace llvm::mustache;
+
+#define DEBUG_TYPE "llvm-mustachespec"
+
+static cl::OptionCategory Cat("llvm-mustachespec Options");
+
+cl::list<std::string> InputFiles(cl::Positional, cl::desc("<input files>"),
+                                 cl::OneOrMore);
+
+cl::opt<bool> ReportErrors("report-errors",
+                           cl::desc("Report errors in spec tests"),
+                           cl::cat(Cat));
+
+static ExitOnError ExitOnErr;
+
+struct TestData {
+  static Expected<TestData> createTestData(json::Object *TestCase,
+                                           StringRef InputFile) {
+    // If any of the needed elements are missing, we cannot continue.
+    // NOTE: partials are optional in the test schema.
+    if (!TestCase || !TestCase->getString("template") ||
+        !TestCase->getString("expected") || !TestCase->getString("name") ||
+        !TestCase->get("data"))
+      return createStringError(
+          llvm::inconvertibleErrorCode(),
+          "invalid JSON schema in test file: " + InputFile + "\n");
+
+    return TestData{TestCase->getString("template").value(),
+                    TestCase->getString("expected").value(),
+                    TestCase->getString("name").value(), TestCase->get("data"),
+                    TestCase->get("partials")};
+  }
+
+  TestData() = default;
+
+  StringRef TemplateStr;
+  StringRef ExpectedStr;
+  StringRef Name;
+  Value *Data;
+  Value *Partials;
+};
+
+static void reportTestFailure(const TestData &TD, StringRef ActualStr) {
+  LLVM_DEBUG(dbgs() << "Template: " << TD.TemplateStr << "\n");
+  if (TD.Partials) {
+    LLVM_DEBUG(dbgs() << "Partial: ");
+    LLVM_DEBUG(TD.Partials->print(dbgs()));
+    LLVM_DEBUG(dbgs() << "\n");
+  }
+  LLVM_DEBUG(dbgs() << "JSON Data: ");
+  LLVM_DEBUG(TD.Data->print(dbgs()));
+  LLVM_DEBUG(dbgs() << "\n");
+  outs() << "Test Failed: " << TD.Name << "\n";
+  if (ReportErrors) {
+    outs() << "  Expected: \'" << TD.ExpectedStr << "\'\n"
+           << "  Actual: \'" << ActualStr << "\'\n"
+           << " ====================\n";
+  }
+}
+
+static void registerPartials(Value *Partials, Template &T) {
+  if (!Partials)
+    return;
+  for (const auto &[Partial, Str] : *Partials->getAsObject())
+    T.registerPartial(Partial.str(), Str.getAsString()->str());
+}
+
+static json::Value readJsonFromFile(StringRef &InputFile) {
+  std::unique_ptr<MemoryBuffer> Buffer =
+      ExitOnErr(errorOrToExpected(MemoryBuffer::getFile(InputFile)));
+  return ExitOnErr(parse(Buffer->getBuffer()));
+}
+
+static void runTest(StringRef InputFile) {
+  outs() << "Running Tests: " << InputFile << "\n";
+  json::Value Json = readJsonFromFile(InputFile);
+
+  json::Object *Obj = Json.getAsObject();
+  Array *TestArray = Obj->getArray("tests");
+  // Even though we parsed the JSON, it can have a bad format, so check it.
+  if (!TestArray)
+    ExitOnErr(createStringError(
+        llvm::inconvertibleErrorCode(),
+        "invalid JSON schema in test file: " + InputFile + "\n"));
+
+  const size_t Total = TestArray->size();
+  size_t Success = 0;
+
+  for (Value V : *TestArray) {
+    auto TestData =
+        ExitOnErr(TestData::createTestData(V.getAsObject(), InputFile));
+    Template T(TestData.TemplateStr);
+    registerPartials(TestData.Partials, T);
+
+    std::string ActualStr;
+    raw_string_ostream OS(ActualStr);
+    T.render(*TestData.Data, OS);
+    if (TestData.ExpectedStr == ActualStr)
+      ++Success;
+    else
+      reportTestFailure(TestData, ActualStr);
+  }
+
+  outs() << "Result [" << Success << "/" << Total << "] succeeded\n";
+}
+
+int main(int argc, char **argv) {
+  ExitOnErr.setBanner(std::string(argv[0]) + " error: ");
+  cl::ParseCommandLineOptions(argc, argv);
+  for (const auto &FileName : InputFiles)
+    runTest(FileName);
+  return 0;
+}

[nikic]

I'd consider llvm-check-mustachespec or llvm-test-mustachespec to be more explicit.

[nikic]

Stray \ file?

[nikic]

As previously mentioned, I think it would be great to include this list directly in the tool so the user doesn't have to manually check which failures are expected and which aren't.

[ilovepi]

Yeah, I agree that would be nice. The current implementation is more of a direct runner for the spec file tests. That's nice, since it keeps things simple, but that also means there isn't a good way to mark something as XFAIL, since we're loading the test files individually at runtime.

I guess we could keep a list of test names that are expected to fail and report them if they match ... I'll try that, since it seems to not require too many changes from the current design.

[nikic]

LGTM

[nikic]

Wondering if we should just skip the optional ~ tests entirely...