get_psm3() reorders the dataframe columns randomly

[Arjannikov]

Every time I call get_psm3(), the returned data frame column order is different for the same query parameters.

CODE EXAMPLE

for count in range(3):
df, metadata = pvlib.iotools.get_psm3(
latitude=latitude, longitude=longitude,
api_key=NREL_API_KEY,
email=MY_EMAIL_ADDRESS
names=2019,
interval=60,
map_variables=True,
leap_day=True)
print(df.columns)

OUTPUT

Index(['Year', 'Month', 'Day', 'Hour', 'Minute', 'pressure', 'Dew Point', 'dni', 'ghi', 'dhi', 'albedo', 'temp_air', 'wind_speed', 'wind_direction'], dtype='object')

Index(['Year', 'Month', 'Day', 'Hour', 'Minute', 'Dew Point', 'pressure', 'wind_speed', 'dni', 'dhi', 'albedo', 'wind_direction', 'ghi', 'temp_air'], dtype='object')

Index(['Year', 'Month', 'Day', 'Hour', 'Minute', 'dhi', 'Dew Point', 'pressure', 'ghi', 'temp_air', 'wind_direction', 'dni', 'albedo', 'wind_speed'], dtype='object')

Versions:

• pvlib.__version__: '0.9.4'
• pandas.__version__: 1.5.3
• python: sys.version_info(major=3, minor=9, micro=13, releaselevel='final', serial=0)

[cwhanse]

Strange. I don't reproduce this behavior with Window/python 3.9.7, pandas 1.5.3, pvlib 0.9.4. What OS are you on?

Edit: nor with python 3.9.13

[kandersolar]

I can't reproduce different column orderings in each loop iteration in a single python session, but I can get different orderings by killing and starting a fresh python session. I am not sure why that would have any effect.

I checked the raw data returned from the PSM3 API call and it seems that the random ordering is on the API's part, not ours -- we just pass through the ordering we get (at least with map_variables=False, I didn't check with True). I thought I understood that the API had a layer where repeated calls with identical parameters would get a cached copy of the corresponding data. I can't square that with these observations.

Worth pointing out that a big PSM3 data update just took place; seems plausible that this is connected.

[kandersolar]

Ah ha:

pvlib-python/pvlib/iotools/psm3.py

Lines 161 to 166 in b11c9e2

	# convert pvlib names in attributes to psm3 convention (reverse mapping)
	# unlike psm3 columns, attributes are lower case and with underscores
	amap = {value: key.lower().replace(' ', '_') for (key, value) in
	VARIABLE_MAP.items()}
	attributes = [amap.get(a, a) for a in attributes]
	attributes = list(set(attributes)) # remove duplicate values

I suspect the set operation is to blame here: our API request includes the columns in random order and the API returns data to match.