Refactor the model

[clizbe]

Discussed in #688

^{Originally posted by datejada July 1, 2024}

Input data names

• Double-check the is_seasonal name, since it is used by more assets than the storage (use_timeframe?)
• Align terminology with INES-spec #622

• Split assets-data for new capacity and existing capacity #774

create_input_dataframes

• Remove all csv related reading functions and use TulipaIO instead. #702
• Use strings instead of Symbols in the create_input_dataframes function  #705

• Verify if we can use only one df (without using a dict) for assets_profiles and assets_timeframe_profiles (better after having a pipeline with timeframe); have a pipeline example before changing that.
• table_tree stores all the data in dfs

create_internal_structures

• Refactor the inputs to use DuckDB as much as possible #706
• Use DuckDB directly to calculate the weight as sum of the mapping in the RepresentativePeriod. See, PR Use DuckDB to create the RepresentativePeriod structure #708
• Clean the graph such that we only store the data that are not created before to avoid duplication, e.g., only leave names for flows/edges.
  • Some of the operations here are inefficient, but they will be removed (e.g., groupby the profiles to store in the graph)
  • Have an internal table instead of an internal dictionary for the rep_periods_partitions. This allows more DuckDB integration.
    Example:

    graph["Midgard"].rep_periods_partitions = Dict(
        1 => [1:3, 4:5, 6:12],
        2 => [1:1, 2:2, ... 6:6],
    )
    asset | rep_period | k | begin | end
    "Mid" | 1          | 1 |     1 |   3
    "Mid" | 1          | 2 |     4 |   5
    "Mid" | 1          | 3 |     6 |  12
    

  • Double-check that where we need rep_periods profiles, maybe just call DuckDB, instead of creating all of them apriori.
• Change representative_periods to use DuckDB #713
• Possibly rename rep_periods_mapping #643
• Should we use the ENUM type from DuckDB for the asset type?  #717
• Refactor the groupby inside create_internal_structure to use DuckDB/TulipaIO in the for loops of profiles #738

compute_assets_partitions

• Update compute_assets_partitions! to use DuckDB more efficiently
  • To compute the partitions of timeframe. It should be possible to use a join or similar

compute_constraints_partitions

• Rename the variables and constraints dataframes in the model #642

compute_rp_partitions

• This function needs a full refactor when changing to table (DuckDB)
  • Big change

solve_model and solve_model!

• Return tables (a.k.a. DuckDB) or dataframes as the output format #115
  • Store the solution into tables (a.k.a. DuckDB) instead of the structure solution and the graph.
  • Clean the solution structure to (maybe?) don't store any vector/array (only keep objective function and some extra info). Even delete it?
  • Create a new table for the investments (assets, and flows)

create_model

• Change construct_dataframes to use tables (a.k.a. DuckDB) or input data frames #611
  • comment -> if we do change all the partitioning to dataframe including the constraints partition computation, then construct_dataframes is not necessary.
• constraints_partitions from Dict to df

• When using tables the filters will be on the tables not in the graph

• Add argument descriptions to all "add constraints" functions #723
  • Change sets and names to be verbose (e.g., Ai -> assets_to_invest)
• Double check the constraint dataframes AffExpr are correctly created with the aggregation in the model. Is unique working as intended?

• add_expression_terms_intra_rp_contraints will need a refactor, and it will determine changes we need before and after
• Notes on code structure:
  • constraints partition: lowest/highest -> to get the correct partition;
  • sum of workspace expressions: sum/unique -> to get the correct expression
  • profile_aggregation functions are in the table in the documentation

• There is no need to use Iterators.flatten. Use for ... for ... instead.
• Double-check getting rid of the lookups (even if the codecov is not 100%) in pro of the readability of the code. See discussion in Update UC and ramping constraints with accumulated units #804
• Double-check the order of the indices in all the elements, constraints, variables, and expressions. They should follow the same order to be more efficient (at least in GAMS and AIMMS is like that)
• Revisit comment in Update storage binary charging constraints #814 (review)
• Revisti comment in Update energy storage constraints with multi-year investment #823 (review)
• Revisit comment in Add vintage version for multi-year  #741 (comment)

AND THEN:

• Rename input data to make more sense
  • How to include Multi-Year?
• Separate the model data from the scenario data. The scenario data for multi-year is currently hard-coded in create_model!, this needs to be generalized and relocated (see Create model discount parameters #803).

[clizbe]

solve_model and solve_model!

• Store the solution into tables (a.k.a. DuckDB) instead of the structure solution and the graph.

@abelsiqueira @datejada Can I try tackling this one? Or should I wait?

[abelsiqueira]

@clizbe you can do it, it's mostly independent from the rest.