canonical-data.json standardisation discussion (was: Malformed data?)

[zenspider]

It appears that all-your-base.json is malformed. Where allergies.json has the structure of:

{
    "allergic_to": {
        "description": [ ... ],
        "cases": [ { "description": "...", ... } ... ]
            }, ...
}

all-your-base.json has:

{
  "#": [ ... ],
  "cases": [ ... ]

cases should be wrapped in a function name, yes?

It appears that bin/jsonlint only checks that the json parses, not that it has good structure.

At the very least, I think this should be patched up and the README expanded to actually show the desired structure. Happy to do a PR for that, assuming I understand it already. 😀

[zenspider]

Looks like there are a lot of different structures involved. Please provide hints as to the correct syntax so I can parse this stuff.

[kytrinyx]

Looks like there are a lot of different structures involved.

Yeah, this sort of happened a bit at a time, and we weren't sure what the various needs of this data were going to be.

We now have enough data to decide on a file format, but I don't think anyone has gone through and figured out what the syntax should be yet.

[Insti]

@zenspider sounds like you're writing a parser, perhaps you can look through the existing data and tell us what structure we should be using to make parsing convenient.
Then we can document it and create issues to update the old data.

[kytrinyx]

@devonestes This is the issue we were talking about on twitter.

[catb0t]

I'm just gonna collect my thoughts from #376 here, because I think this needs fleshing out.

I believe we can simultaneously make the JSON easier for humans and programs to read, but the way it is now makes it very hard to make a generalising program.

@petertseng linked to examples of code in various tracks using canonical-data.json to generate exercises, and I feel they all share a common problem: because each exercise has a different structure, each exercise needs its own separate, different test generator program.

My goal with exercism.autogen-exercises is to generate all the tests for all the exercises at once which should be trivially possible. I don't want a different ${exercisename}-testgen.factor for each different JSON structure.

As it is right now, I could theoretically write code to map x-common's JSON keys to my own internal structure, but this requires a duplication across programs that read this data. Also, it's not scalable, and as such it would be genuinely beneficial to everyone to standardise the keys and their meanings.

I am personally willing to manually rewrite all the JSON in this repository to fit a predictable format, but I won't until we have a consensus.

[NobbZ]

I'd fully support a more generic structure which would make it unnecessary to have a generator for each exercise.

But I have to admit, I have no idea how it could look like. Since you already said you would change them, do you have an idea about the structure already @catb0t?

Also since it seems to be the right time, I want to request a feature for this generic format:

I had a sleepness night, of how I should handle changes in the canonical data as I wanted to have some versioning test. First I thought I'd could just use the date of the last change, but this would mean, that because of whitespace changes all earlier submissions would get "invalidated". Therefore I think it would be a good idea to version the canonical data as well.

[catb0t]

@catb0t wrote:

I'm thinking something like:

• For exercises with one input translating to one output, description, input and output.
• For exercises with multiple inputs / multiple outputs, description, input_N, output_N.

Note that it would be disadvantageous to use an array for multiple inputs / outputs where an array is not part of the exercise because it would be hard or impossible to tell the difference between multiple inputs and an actual array. We could have keys like input_multi which is an array of inputs, I suppose?

@petertseng wrote

For exercises with multiple inputs / multiple outputs, description,input_N, output_N.

[ ... ] Can we simultaneously make it easy for a human to read as well? [ ... ] in e.g. all-your-base's JSON, [ ... ] many tracks will pass in three inputs: input_base, input_digits, output_base, and then check that the output digits are as specified in output_digits. If the data then simply looked like "input_1": 2, "input_2": [1], "input_3": 10, "output": [1] I think it might not be clear what is the difference between input_1 and input_3 to a human, and I consider this important for being able to understand PRs that propose to change the test cases.

@petertseng makes a good point that input_N, etc, might harm readability especially since there are no comments in JSON, and I'm not really sure what to do about that.

I don't have a firm idea of what keys would fix Peter's point, which is a reason I haven't started rewriting it all myself yet.

Using descriptive English names makes it hard to access them programmatically, but using numbered keys makes it hard for people (not me, but other maintainers) to read. What strikes a balance?

This might be a little bit wild, so bear with me: what if we add a top-level key metadata, and it has this structure:

"cases": { "cases data..." }
"metadata": {
    "input_keys": [ "input_key1", "input_key2", "input_key3" ],
    "output_keys": [ "output_keyN" ]
}

That moves the mapping of human-readable keys from each track's generation code to the JSON itself. Then autogeneration code can read metadata to get the list of keys that are used in this cases structure.

[ ... ]How should I handle changes in the canonical data as I wanted to have some versioning test. [ ... ] I could just use the date of the last change, but this would mean, that because of whitespace changes all earlier submissions would get "invalidated". Therefore I think it would be a good idea to version the canonical data as well.

We could perhaps end up with:

"#": "..."
"cases": { "cases data..." }
"metadata": { "..." }
"version": {
    "version_hash": "shasum of minified version of this file",
    "version_time": "seconds since 1 Jan 1970 here"
}

And you can read the version key. Or perhaps I'm misunderstanding your point.

[NobbZ]

I do not understand the input_N stuff, but there came something into my mind.

{
  "exercise": "repeat",
  "examples": [
    {
      "function": "repeat",
      "description": "tests valid stuff",
      "input_count": 5,
      "input_string": "foo",
      "expected": "foofoofoofoofoo"
    },
    {
      "function": "repeat",
      "description": "tests failure",
      "input_count": -5,
      "input_string": "foo",
      "expected": { "error": "no negatives allowed" }
    }
  ]
}

Perhaps we can use this as a base, or throw it away instantly?

[zenspider]

@NobbZ:

    {
      "function": "repeat",
      "description": "tests failure",
      "input_count": -5,
      "input_string": "foo",
      "expected": { "error": "no negatives allowed" }
    }

and what ensures the order of the args? There's no metadata in place to declare argument names.

[zenspider]

@catb0t:

My goal with exercism.autogen-exercises is to generate all the tests for all the exercises at once which should be trivially possible [emphasis mine]. I don't want a different ${exercisename}-testgen.factor for each different JSON structure.

I don't. I think you can get a good start on it for most languages, but that idea doesn't take into consideration language call semantic differences (factor/forth vs assembly vs algol-based languages vs keyword arguments (smalltalk, ruby) as an example). Nor is it realistic about the level of finality. I think you can easily generate a rough draft for every exercise for a language, but it still needs to be reviewed, finalized, and styled by a human to be a good example to learn from.

[NobbZ]

I do not see any sense in specifying order of arguments in the canonical
testdata. There are different idioms and necessities in the various tracks.

Let's assume we have some data type and we write functions around it. Let's
call it list. In object oriented languages it will be the object we call a
method in so it will be completely out of the order of arguments. In elixir
we like to have this object like argument at the first position to be able
to pipe it around, while in Haskell it is preferred to have it last to be
able to use point free style and partial application.

So as you can see order of arguments has to be specifies by the tracks
maintainer a anyway.

Ryan Davis notifications@github.com schrieb am Mi., 21. Sep. 2016 23:47:

{
  "function": "repeat",
  "description": "tests failure",
  "input_count": -5,
  "input_string": "foo",
  "expected": { "error": "no negatives allowed" }
}

and what ensures the order of the args? There's no metadata in place to
declare argument names.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#336 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADmR8LftAf2rePU2wUD_ZgXKqYCjrzbks5qsaYCgaJpZM4JjxYn
.

[rbasso]

Maybe I'm a little late and out of topic, but I'll try anyway...

About automatically generated test suites

I know that it makes sense in some languages to think about automatically generating tests, but I belive that this is not a goal shared between all tracks.

I think it is impossible, in the general case, to auto-magically generate the test suite, unless we collapse all the types into the ones representable in JSON. I know that, at least in Haskell, that would be bad and wrong! 😄

That said, it is certainly possible to have a generator to automatically update a specific exercise, if the JSON structure is not changed.

Is it worthy?

That depends on how frequently the data and the structure are updated, but mostly on how fun is the process of writing and maintaining it. So I think it is not unreasonable. 👍

Alternatively - if the desire is really to have auto-magic test suites - it would be more compatible if the exercises where specified as stdin-stdout mappings. That would be similar to how online judge systems work, but I don't think it is exercism's destiny to follow that path.

About readability for humans and software

Considering that it is generally impossible to automatically generate test suites, I think it doesn't make sense to sacrifice human-readability too much, forging a JSON that is convenient for software but inconvenient for humans.

That doesn't mean we shouldn't standardize the files. We should, but remembering that the files are meant to be read first by humans, and then by software.

About the format

Maybe I'm the only one that doesn't got what is going on here, but I think that, until it is clear what is our goal here, we should avoid getting into the details of the specification.

Edit: Ok. I think I got it!. 😄

[rbasso]

What about something like this:

{
  "exercise": "cipher",
  "version": "0.1.0 or an object with more detailed information",
  "comments": [
    "Anything you can think of",
    "as a list of strings"
  ],
  "tests": [
    {
      "name": "encode",
      "description": "Encodes plaintext",
      "cases": [
        {
          "description": "Encodes simple text",
          "plaintext": "Secret message",
          "key": "asdf1234",
          "expected": "qwertygh"
        },
        {
          "description": "Encodes empty string",
          "plaintext": "",
          "key": "test1234",
          "expected": ""
        }
      ]
    },
    {
      "name": "decode",
      "description": "Decodes plaintext",
      "cases": [
        {
          "description": "Decodes simple text",
          "ciphertext": "qwertygh",
          "key": "asdf1234",
          "expected": "Secret message"
        },
        {
          "description": "Decodes empty string",
          "ciphertext": "",
          "key": "test1234",
          "expected": ""
        }
      ]
    }

• 👍 Allows groups of tests.
• 👍 Uses standard names where it doesn't affect human-readability.
• 👍 Captures most of the structure of existing tests.
• 👍 Doesn't damage readability too much.
• 👎 Mixes description, inputs and output in the same object.
• 👎 Gives special meaning for two case's keys: description and expected.
• 👎 Has no explicit ordering of cases' input data.

The descriptions could be mandatory or optional.

I would be possible to use multilevel grouping of tests, but I don't think that is used frequently.

Keeping the description, inputs and the expected output together, we have a structure that is more human-friendly, but not so convenient for processing.

@zenspider and @catb0t, would it be too difficult to separate description and expected from the other keys? Would it be reasonable for you to use an implicit alphabetic ordering for the remaining keys, instead of adding metadata?

[devonestes]

I've been thinking about this a bit recently, and I think the most generalized version of this we can get might be the best for as many different needs as possible. What we're really doing in most of these exercises is basically testing functions. There's input, and there's output. By trying to use keys in our JSON objects that are things like "plaintext" and "key", that's creating a need for knowledge about the exercise to accurately understand how those parts interact.

I think if we can generalize on that concept of a function that we're testing, that might be helpful both for human readability, and also for machine readability so we can possibly use this data for automatic tests.

So, here's my example:

{
  "exercise": "cipher",
  "version": "0.1.0 or an object with more detailed information",
  "comments": [
    "Anything you can think of",
    "as a list of strings"
  ],
  "tests": [
    {
      "description": "encodes simple text",
      "function": "encode",
      "input": ["Secret message", "asdf1234"],
      "output": "qwertygh"
    },
    {
      "description": "encodes empty string",
      "function": "encode",
      "input": ["", "test1234"],
      "output": ""
    },
    {
      "description": "decodes simple string",
      "function": "decode",
      "input": ["qwertygh", "asdf1234"],
      "output": "Secret message"
    }
  ]
}

I don't think there are any exercises that require anything other than input and output, but I haven't done too deep of an analysis on that. I'd love any feedback if there are edge cases that would need to be taken care of here. I know that based on the structure above I can think of reasonable ways to parse that and automatically create some skeletons for tests in Ruby, Elixir, Go, JavaScript and Python, but that's really all I can reasonably speak to since those are the only languages I have a decent amount of experience with.

Also, I sort of like the stripped down way of looking at this - when I look at that data I don't need to know the context of the exercise to know what's going on. I just know there's a thing called encode, and that takes some input and returns some output, and there's a text description of what's going on.

I'm not really 100% sure that this would give us everything we want, but I wanted to at least throw this idea out there to get feedback and see if it might be a starting point for an actually good idea!