Run and interact with MATPOWER from Python [discussion]

[yasirroni]

Hi @rdzman, thanks for your great repository. This is not an issue, I just want to inform you that there is oct2py library that make it possible to run .m file from python. Thus, I make a library called matpower-pip that pack MATPOWER github package release into pypi package. Thus, it make MATPOWER able to be installed (downloaded) via pypi (standard python package distribution) using:

pip install matpower

or if user want to run matpower from python:

pip install matpower[octave]

Thus, it support running MATPOWER using:

from matpower import matpower

matpower.runpf() 

Making preprocessing and postprocessing in python possible using:

from matpower import matpower

mpc = matpower.eval('case9', verbose=False)

mpc = matpower.runpf(mpc)

---

Don't worry, no code from original MATPOWER is modified, thus it's original MATPOWER that runs on octave via oct2py on python.

I hope you like it!!! Or maybe, you can link to my repository for others that want to run MATPOWER from python and post processing there?

[rdzman]

Thanks @yasirroni! This looks to be quite useful. I have pretty limited experience with Python, but wanted to give it a try before announcing it more widely and linking to it from the MATPOWER website.

I was able to install it just fine, but when I try to run a small script (mp.py) with the following content ...

from matpower import matpower

mpc = matpower.eval('case9', verbose=False)

mpc = matpower.runpf(mpc)

... I get this error:

Traceback (most recent call last):
  File "mp.py", line 3, in <module>
    from matpower import matpower
ImportError: cannot import name 'matpower' from 'matpower' (/Users/ray/Library/Python/3.7/lib/python/site-packages/matpower/__init__.py)

Any suggestions?

[yasirroni]

Sorry, I just changed the API yesterday to support multiple separate instance of octave/MATPOWER to support multi threading.

Latest API is:

from matpower import start_instance

matpower = start_instance()

mpc = matpower.eval('case9', verbose=False)

mpc = matpower.runpf(mpc)

Better:

from matpower import start_instance

m = start_instance()

mpc = m.eval('case9', verbose=False)

mpc = m.runpf(mpc)

Sorry for the trouble,

[yasirroni]

Which API you prefer? As far as I know, the latest one is the best from Python side to support user that want to call multiple yet separate octave/MATPOWER in multi threading environment.

[yasirroni]

Also, the latest one will support version checking such as:

import matpower

print(matpower.__version__)

[rdzman]

It's running for me now.

In terms of API, before I comment, I'd like to understand how it works. Am I to understand that you can call any MATPOWER (or Octave) function as method of the m object?

That would appear to be the case, except that calling runopf() results in an error while writing to a MAT file.

[yasirroni]

What I (start_instance) do is:

1. Starting an octave client (support multiple separate instance that make each instance access different client)
2. Add MATPOWER path to octave path.
3. Run any .m file and return (also display) the results.

So yeah, it's agnostic and only do what octave/MATPOWER do.

As far as I remember, I never use octave to write MAT file. Can you share with me the code (Python) that you use? I will check it.

[rdzman]

from matpower import start_instance

m = start_instance()

mpopt = m.mpoption('verbose', 2);
mpc = m.loadcase('case9');
r1 = m.runopf(mpc, mpopt)

And the error I get is ...

Traceback (most recent call last):
  File "mp.py", line 7, in <module>
    r1 = m.runopf(mpc, mpopt)
  File "/Users/ray/Library/Python/3.7/lib/python/site-packages/oct2py/dynamic.py", line 100, in __call__
    return self._ref().feval(self.name, *inputs, **kwargs)
  File "/Users/ray/Library/Python/3.7/lib/python/site-packages/oct2py/core.py", line 386, in feval
    store_as=store_as, plot_dir=plot_dir)
  File "/Users/ray/Library/Python/3.7/lib/python/site-packages/oct2py/core.py", line 586, in _feval
    raise Oct2PyError(msg)
oct2py.utils.Oct2PyError: Octave evaluation error:
error: save: error while writing '' to MAT file

[yasirroni]

Until now, my use case only calling MATPOWER case, modify in Python, run powerflow, calculate losses, further modify the case, and rerun. As simple as that.

Not all MATPOWER function have been used by me, and also no test I use.

If there is an error, my hunce is that it is either permission to write and read files or oct2py limitations.

_

I will try to investigate your use case. Thank you.

[yasirroni]

Dear @rdzman, after investigating, the problem arise from unknown object data type <object opf_model> that can't be converted to Python data format. The data can be found in r.om in runopf.m.

Possible solution is:

1. To remove this data format on MATPOWER site

2. Option for user to not get this data format on results

3. Reformat the data format to native octave data format such as struct

4. Patch oct2py to skip this data and give user warning

[yasirroni]

This solve the problem (workaround), by requeting 7 outputs from runopf to evade <object opf_model>. This is number 2 solution:

2. Option for user to not get this data format on results

from matpower import start_instance

m = start_instance()

mpc = m.loadcase('case9');
mpopt = m.mpoption('verbose', 2);
[MVAbase, bus, gen, gencost, branch, f, et] = m.runopf(mpc, mpopt, nout=7)

[yasirroni]

[UPDATE!!!]
This seems pretty comprehensive step by step how to communicate between octave and python if UNSUPPORTED DATA TYPE EXISTS

# import start_instance to start matpower instance
from matpower import start_instance

# start instance
m = start_instance()

# use octave native to run some commands
m.eval("mpopt = mpoption('verbose', 2);")
m.eval("mpc = loadcase('case9');")
m.eval("r1 = runopf(mpc, mpopt);") # we avoid parse `r1` that containts unsupported `<object opf_model>`

# fech data to python (.eval is used because .pull is not working in acessing field)
r1_mpc = {}
r1_mpc['baseMVA'] = m.eval('r1.baseMVA;')
r1_mpc['version'] = m.eval('r1.version;')
r1_mpc['bus'] = m.eval('r1.bus;')
r1_mpc['gen'] = m.eval('r1.gen;')
r1_mpc['branch'] = m.eval('r1.branch;')
r1_mpc['gencost'] = m.eval('r1.gencost;')

# modify variable if necessary
[GEN_BUS, PG, QG, QMAX, QMIN, VG, MBASE, GEN_STATUS, PMAX, PMIN, MU_PMAX, 
 MU_PMIN, MU_QMAX, MU_QMIN, PC1, PC2, QC1MIN, QC1MAX, QC2MIN, QC2MAX, 
 RAMP_AGC, RAMP_10, RAMP_30, RAMP_Q, APF] = m.idx_gen(nout='max_nout')
gen_index = 2 # index of generator to be changed
gen_index_ = int(gen_index - 1) # -1 due to python indexing start from 0
PMAX_ = int(PMAX -1) # -1 due to python indexing start from 0
r1_mpc['gen'][gen_index_,PMAX_] = 110 # in this example, we modify PMAX to be 110

[PQ, PV, REF, NONE, BUS_I, BUS_TYPE, PD, QD, GS, BS, 
 BUS_AREA, VM, VA, BASE_KV, ZONE, VMAX, VMIN, LAM_P, 
 LAM_Q, MU_VMAX, MU_VMIN] = m.idx_bus(nout='max_nout')
bus_index = 7 # index of bus to be changed
bus_index_ = int(bus_index - 1) # -1 due to python indexing start from 0
PD_ = int(PD-1) # -1 due to python indexing start from 0
r1_mpc['bus'][bus_index_,int(PD-1)] = 80 # in this example, we modify PD to be 150

# push back value to octave client
m.push('mpc', r1_mpc) # push r1_mpc in python to mpc in octave

# test if our pushed variable can be used
m.eval("r1 = runopf(mpc, mpopt);")

# test if we can retrive pushed value
mpc = m.pull('mpc')

[rdzman]

Very interesting. This is helpful.

So, if I understand correctly, oct2py attempts to convert the return values of Octave function calls from native Octave data types to native Python data types, by saving to a temporary .mat file and then reading that file into Python.

It does seem that the ideal solution would be to go upstream and provide a general solution in oct2py for handling these unknown data types ... preferably in a way that preserves the data and allows the object to be recreated when passed back to Octave.

But using eval is a workaround. Btw, you could also do ...

m.eval("r1 = runopf(mpc, mpopt);") # we avoid parse `r1` that containts unsupported `<object opf_model>`
m.eval("r1 = rmfield(r1, 'om');")
r1 = m.pull("r1")

[yasirroni]

Since the variable itself preserved inside octave session, I think let a custom variable stay there is the best solution since there will be infinitely many possibilities of custom data type.

Remember that the custom data is not removed from octave, it's just not parse-able to python to be modified in python.

---

Regarding the data type, it's not always python native, but numpy data type and python native data type. That is because python didn't support matrix data type, thus there is numpy.

[lfb66]

Hello @yasirroni !
I have been working with python, especially with pandapower, but I need to work with some networks that are only available in matpower
Is it possible to work with matpower from python without having octave?

[rdzman]

MATPOWER requires MATLAB or Octave to run, so you'll have to have one of the two. You can call MATLAB functions from Python, including MATPOWER functions, once you have MATLAB and MATPOWER installed.

[yasirroni]

Is it possible to work with matpower from python without having octave?

As stated above by @rdzman, you need either MATLAB engine or Octave engine. That's why I make it possible to use pip install matpower from pypi. You can use that package using both octave and matlab. But, since I only have octave, I only fully support and investigate octave.

For using matlab engine, it should be like (I can't test it, that way I will not really know it):

import matlab.engine
from matpower import path_matpower

eng = matlab.engine.start_matlab()
eng.addpath(eng.genpath(path_matpower))

eng.runpf()

[yasirroni]

That's why, I hope @rdzman someday could mention my work on his website. *insert wink emoji

[rdzman]

Thanks @yasirroni. I can confirm that it does also work with MATLAB, but I get errors unless I add nargout=0 to the function calls. For example, the following runs as expected ...

import matlab.engine
from matpower import path_matpower

m = matlab.engine.start_matlab()
m.addpath(m.genpath(path_matpower))

r = m.runpf('case5', nargout=0)

m.eval("mpopt = mpoption('verbose', 2);", nargout=0)
m.eval("mpc = loadcase('case9');", nargout=0)
m.eval("r1 = runopf(mpc, mpopt);", nargout=0)

And, I have not forgotten about mentioning it on the website / mailing list ... just haven't had the time yet.

[yasirroni]

Thanks @rdzman for testing on MATLAB. I will add that to the docs, including option to use start_instance(engine='matlab').

[rdzman]

matpower-pip is now listed on the Related Links page on the MATPOWER website.

Thanks again, @yasirroni, for your work on this and for sharing it!

I've also announced it on the MATPOWER Discussion List.

[yasirroni]

Thank you very much. I will close this issue than. If you want anything to ask or request, I will gladly help, since your work really help me go through my university life since undergraduate . Currently working my master degree!!!

[rdzman]

Thanks @yasirroni.

Just fyi and for reference, in case you are not on the MATPOWER mailing list, Richard Lincoln mentioned his earlier rwl/oct2pypower.

[yasirroni]

Thanks @yasirroni.

Just fyi and for reference, in case you are not on the MATPOWER mailing list, Richard Lincoln mentioned his earlier rwl/oct2pypower.

Yeah, I was not in mailing list.

To be honest, my work is partly inspired by his project. But, his project require docker and not easy to be used. Furthermore, my approach directly embed MATPOWER to pypi, while his approach more like oct2py installation + tutorial to add .m file to octave path.

[yasirroni]

That is why in the latest README, after I remember his package from mailing archive, I mention his project. But yeah, I never able to use his approach though. I'm not familiar with docker and confused how to integrate docker with usual case of python.

[yasirroni]

And one more, few days ago, I register to the mailing list, but seems not receiving any message.

[yasirroni]

And one more, few days ago, I register to the mailing list, but seems not receiving any message.

Ah, it goes to spam. Fixed, should receive new update from email.

[chiru7187]

Firstly let me appreciate your ..you did a great job..

Thanks, @yasirroni.

I have some problems dealing with Octave.

from matpower import start_instance

matpower = start_instance()

mpc = matpower.eval('case9', verbose=False)

mpc = matpower.runpf(mpc)

1. After running this code...
   getting OSQP Error!
   and giving me the results.

2. SECOND PROBLEM IS
   warning: error caught while executing handle class delete method:
   'osqp_mex' undefined near line 40, column 13
   warning: struct: converting a classdef object into a struct overrides the access restrictions defined for properties. All properties are returned, including private and protected ones.
   warning: called from
   _pyeval at line 97 column 3

---

Oct2PyError Traceback (most recent call last)

Oct2PyError: Octave evaluation error:
error: save: error while writing '' to MAT file

Kindly answer me ..Thank you

[yasirroni]

You can make issue and discuss about matpower in Python here.

---

After running this code...
getting OSQP Error!
and giving me the results

OSQP Error is printed by MATPOWER 7.1 (or MATPOWER 7.0, I'm forget about this). You can make Issue to @rdzman to remove that printout or workaround if we don't specify to use OSPQ. Actually I'm planning to make that issue, but haven't do it yet.

---

warning: error caught while executing handle class delete method:
'osqp_mex' undefined near line 40, column 13
warning: struct: converting a classdef object into a struct overrides the access restrictions defined for properties. All properties are returned, including private and protected ones.
warning: called from
_pyeval at line 97 column 3

Can you share me the piece of code? Please make an issue on matpower-pip