How to implement EDIFACT Syntax elements

[nerdoc]

I just want to gather a few ideas on how to implement interchange syntax elements like UNB, UNH, UNZ etc.: Messages, Control Header, and other things.
There are syntax implemetation guidelines of the UNECE, and other sources in the internet, like here

Best thing would be to write some sort of "syntax description language" which describes these syntax elements, and parse them. There are a few possibilities for the notation of this "syntax description":

• external files (YAML, TOML, CSV etc.)
• python classes

I had an idea writing these descriptions as classes, like Django does this in it's ORM, describing a syntax element by a class using attributes:

class CharType(Enum):
    ALPHA = 0
    ALPHANUMERIC = 1
    NUMERIC = 2

class SyntaxElement:
    """Common description of an EDIFACT syntax element"""
    def __init__(
        self,
        id: str,
        mandatory: bool = False,
        type: CharType = None,
        length: int = None,
        max_length: int = None,
    ):
        # checks...

        self.mandatory = mandatory
        self.type = type
        self.max = False
        self.length = 0
        if length:
            self.length = length
        elif max_length:
            self.length = max_length
            self.max = True

The first bytes of an Interchange Control Header would be then:

class InterchangeControlHeader:
    syntax_identifier = SyntaxElement(
        id="0001", mandatory=True, type=CharType.ALPHA, length=4
    )
    syntax_version_number = SyntaxElement(
        id="0002", mandatory=True, type=CharType.ALPHANUMERIC, length=1
    )
    service_code_list_directory_version_number = SyntaxElement(
        id="0080", type=CharType.ALPHANUMERIC, max_length=6
    )
    character_encoding = SyntaxElement(
        id="0133", type=CharType.ALPHANUMERIC, max_length=3
    )

@JocelynDelalande what do you think about that?

[JocelynDelalande]

LGTM ! (good idea to implement it as python classes IMHO, that will be the simplest way).

A Simple note on Wording : SyntaxElement must be called Component to respect EDIFACT wording.

Do you want to use those classes to build the API ? (replacing dict-list current approach) or merely to do data validation, leaving an API with same style for Component and Elements level ?

[nerdoc]

Yeah, I just created a buzz-word - could be named better. Component - hm dunno. I found a list of "service codes" here: https://www.stylusstudio.com/edifact/40102/codelist.htm
Maybe ServiceCode could be appropriate too?
In the first place I just thought about data validation, not API. OMG, EDIFACT is complicated, Why did I start this project again ;-)
I think the API should be done via other structures. #19 seems a better place to discuss that, thanks for that. So, let's just keep this here in mind for validation.

[nerdoc]

It would be good to create something like SegmentTables which define how messages are structured. Here we have to include segment groups (for loops) too, which can be nested. I am still not sure if doing that using Python classes is the best way. or if just lists of e.g. tuples would be better...

From your link, "example of segment groups for the Extended Payment Order (PAYEXT)" - first group.

[
    ("UNH", "Message header", "M", 1),
    ("BGM", "Beginning of Message", "M", 1),
    ("BUS", "Business function", "C", 1),
    # ...
]

This looks a bit cleaner for creating these descriptions. But we would have to write another parser which handles nested groups etc. This is easier done in Python classes, but it is more boilerplate code to write when defining structures.
I like Django very much, and came up with this idea from how Django creates database tables in it's ORM just from Python class declarations which describe the data object. This would be similar here.
I don't know, @JocelynDelalande what do you think?

[JocelynDelalande]

I like Django very much, and came up with this idea from how Django creates database tables in it's ORM just from Python class declarations which describe the data object. This would be similar here.
I don't know, @JocelynDelalande what do you think?

I do not really have any strong opinion on that, sorry.

[nerdoc]

Do you know if there is an official downloadable data format for the implementation of those fields available anywhere? Something like a xml or even CSV which could be parsed?

[maanas]

Use of django classes is a good idea. Another alternative could be to use yml file where we specify the options. The nesting could be well taken care of by the yml files.

[nerdoc]

YAML would be an option, but I definitely want to keep pydifact as clean (few dependencies) as possible. So "Django"-like classes will be the way to go.

[srinirokz]

Where can i ask a questions. ??

[nerdoc]

@srinirokz Here, if its related to this issue. Else just start a new issue.

[nerdoc]

I asked www.stylusstudio.com where they have a good overview of EDIFACT data if I can parse their site using beautifulsoup and extract Service Code data from it. This would make it much easier to get a bigger amount of data for Service codes.
If they don't agree (which I think that will be the case), I'll asked the official UN source too by mail. I nowhere can see if the EDIFACT data are protected by patents etc. or if one can use them freely.

[theangryangel]

Hi people 👋 , I’ve been poking at pydifact today and am I right in reading this issue is about adding support for definitions, which would describe things like segment groups, etc and smartly reading/writing? Am I correct that without this it’s up to users or the library to split groups?

[nerdoc]

Yes, that's right. I just don't have time ATM to implement this, so development has slowed down a bit. But nevertheless, PRs are more than welcome... I'd like to create a high level API to read and write documents, including groups.

[theangryangel]

Cool, I just wanted to check that I wasn't fundamentally misunderstanding something :)

I spent far longer than I would like to admit yesterday getting a handle on a project, dealing with malformed sample edifact files, reading specs, etc. and by the end of the day I was going a bit cross-eyed 😅

Practically I wasn't "budgeting" for time to add support, but I may be down the path far enough that a PR or two will come this way over the next few days.