Create a diagram of the workflow

[abelsiqueira]

Create a diagram to help describe all steps and inputs of the workflow.

Related Issues:

#414

---

Last update: 27/06/2024

I've split the diagram into two parts. One for the pipeline and one for the DBs. They are both still WIP, and the DBs are mostly just showing the keys.

High level overview of compact usage

graph
    classDef db fill:#aaa,color:black,stroke:black,stroke-width:1px;
    classDef cls_tem fill:#0af,color:black;
    classDef cls_tio fill:#fc0,color:black;
    classDef cls_tc fill:#0f0,color:black;

    source["Some source (e.g. ESDL + EDR, CSV folder)"]:::db

    %% DBs/DataFrames
    initial_data[(Initial data)]:::db
    clustering_data[(Clustering data)]:::db
  
    %% Connections
    source --> TIO.read_csv_folder:::cls_tio
        --> initial_data
        --> TC.split_into_periods:::cls_tc
        --> TC.find_representative_periods:::cls_tc
        --> TC.write_clustering_results_to_tables:::cls_tc
        --> clustering_data
        --> TEM.EnergyProblem:::cls_tem
        --> TEM.create_model!:::cls_tem
        --> TEM.solve_model!:::cls_tem

Loading

graph
    classDef db fill:#222,color:white,stroke:white,stroke-width:1px;
    classDef object fill:#09f,color:black;
    classDef cls_tem fill:#3d3,color:black;
    classDef cls_tio fill:#77f,color:black;
    classDef cls_tc fill:#77f,color:black;
    classDef undefined fill:#f33,color:black;

    source["Some source (e.g. ESDL + EDR, CSV folder)"]
    scenario["Scenario specification ?"]:::undefined

    %% Actions and packages
    TIO("TulipaIO"):::cls_tio
    TEM("TulipaEnergyModel"):::cls_tem
    TC("TulipaClustering"):::cls_tc
    TP("Partitioner"):::undefined
    create_input_dataframes("create_input_dataframes"):::cls_tem
    create_internal_structures("create_internal_structures"):::cls_tem
    compute_constraints_partitions("compute_constraints_partitions"):::cls_tem
    create_model("create_model"):::cls_tem
    solve_model(solve_model):::cls_tem
    construct_dataframes(construct_dataframes):::cls_tem
    
    %% Objects
    con{{DB connection}}:::object
    table_tree{{table_tree}}:::object
    _graph{{graph}}:::object
    rps{{representative_periods}}:::object
    timeframe{{timeframe}}:::object
    cps{{constraints_partitions}}:::object
    dfs{{dataframes}}:::object
    model{{model}}:::object
    solution{{solution}}:::object

    %% DBs/DataFrames
    static_data[(Graph Data)]:::db
    raw_profiles_data[(Raw Profiles Data)]:::db
    cluster_data[(RP & Timeframe Cluster Data)]:::db
    clustered_profiles_data[(Clustered Profiles Data)]:::db
    partition_data[(Partitioned Data)]:::db
  
    %% Connections
    source --> TIO
        --> static_data & raw_profiles_data
        --> TC
        --> cluster_data & clustered_profiles_data
    scenario --> TP
        --> partition_data
    TIO --> con
    static_data & raw_profiles_data & cluster_data & clustered_profiles_data & partition_data --> create_input_dataframes
        --> table_tree
        --> create_internal_structures
        --> _graph & rps & timeframe
    _graph & rps --> compute_constraints_partitions
        --> cps
    _graph & rps & cps & timeframe --> construct_dataframes
        --> dfs
    _graph & rps & dfs & timeframe --> create_model
        --> model
        --> solve_model
        --> solution

Loading

The DB diagram is not automatically generated, here is a snapshot, the code is below.

// Use DBML to define your database structure
// Docs: https://dbml.dbdiagram.io/docs

// Types are limited to what is provided by SQL
Table Asset {
  asset string [primary key]
  extra_stuff unknown
}

Table Flow {
  from_asset string [primary key, ref: > Asset.asset]
  to_asset string [primary key, ref: > Asset.asset]
  extra_stuff unknown
}

Table Profile {
  profile_name string [primary key]
  timestep int [primary key]
  value float
}

Table AssetProfileReference {
  asset string [primary key, ref: > Asset.asset]
  profile_type string [primary key]
  profile_name string [ref: > Profile.profile_name]
}

Table FlowProfileReference {
  from_asset string [primary key, ref: > Flow.from_asset]
  to_asset string [primary key, ref: > Flow.to_asset]
  profile_type string [primary key]
  profile_name string [ref: > Profile.profile_name]
}

// --------- 
// All DB above are used by Clustering. Everything below is Model specific
// --------- 

Table RepPeriod {
  rep_period int [primary key]
  num_timesteps int
  resolution float
}

Table TimeframePeriod {
  period int [primary key]
  rep_period int [primary key, ref: > RepPeriod.rep_period]
  weight float
}

Table RepPeriodProfile {
  profile_name string [primary key, ref: > Profile.profile_name]
  rep_period int [primary key, ref: > RepPeriod.rep_period]
  profile_type string
  value float
}

Table TimeframeProfile {
  profile_name string [primary key]
  timestep int [primary key, ref: > TimeframePeriod.period]
  value float
}

Table AssetTimeframeProfileReference {
  asset string [primary key, ref: > Asset.asset]
  profile_name string [primary key, ref: > TimeframeProfile.profile_name]
}

Table AssetRepPeriodPartition {
  asset string [primary key, ref: > Asset.asset]
  rep_period int [primary key, ref: > RepPeriod.rep_period]
  specification enum
  partition string
}

Table FlowRepPeriodPartition {
  from_asset string [primary key, ref: > Flow.from_asset]
  to_asset string [primary key, ref: > Flow.to_asset]
  rep_period int [primary key, ref: > RepPeriod.rep_period]
  specification enum
  partition string
}

Table AssetTimeframePartition {
  asset string [primary key, ref: > Asset.asset]
  specification enum
  partition string
}

[clizbe]

Is it normal that I have a "Unable to render rich display" error?

[abelsiqueira]

Unfortunately, it is not rendering - and it's incomplete - so I am thinking of using another tool.

…

On Thu, Jan 18, 2024, 10:47 Lauren Clisby ***@***.***> wrote: Is it normal that I have a "Unable to render rich display" error? — Reply to this email directly, view it on GitHub <#415 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIE5UCTWP56UK7XOAWAIQ3YPDVRFAVCNFSM6AAAAABB6R44POVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJYGEZTMOBYGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[abelsiqueira]

Updated with app.diagrams.net / draw.io version.

[clizbe]

Works now! 👍

[clizbe]

@abelsiqueira @suvayu Maybe this is a good candidate for a wiki? 🙃

[abelsiqueira]

Have we decided what use we'll be moving to the wiki? We have 6 places for prose right now:

• README.md
• README.dev.md
• CONTRIBUTING
• Docs = Website
• Issues
• Discussions

So it might be useful to redefine some things to give use to the wiki. Also, it is a good time to bring up https://diataxis.fr, since it relates to documentation and purpose.

[clizbe]

Diataxis was a great read!
I agree we're spreading out a bit, but this is something that doesn't seem to really belong in an issue... Maybe it can go into the docs? Right now it's pretty dev-related, but eventually might be useful to the user.

[clizbe]

@abelsiqueira Is there a way we can have this generate with code? That way we can put it in the docs without screenshotting it and keep it updated in the future (maybe add it to our PR checklist?)

[suvayu]

Is there a way we can have this generate with code?

Thinking like an engineer @clizbe ;⁠-⁠)

We could explore using Mermaid.js. It's supported by GitHub. You can experiment @ https://mermaid.live/

[clizbe]

Great! Let's use that. It'll probably take some work for the first version but updating after that shouldn't be bad.
Or maybe we can shop around and see if there's a similar package that's better...

[abelsiqueira]

I've updated the diagram with the mermaid version. It is still a mix of current and desired structure. In the following weeks I hope to converge on something more concrete.