Invalid port

[yoannHertienne]

Hello,

i'm using release v0.0.10 with Home Assistant 2023.5.2 and I'm getting this error

`Logger: custom_components.enphase_envoy
Source: custom_components/enphase_envoy/envoy_reader.py:189
Integration: Enphase Envoy (DEV)
First occurred: 9 mai 2023 à 08:48:15 (1086 occurrences)
Last logged: 11:52:33

Unexpected error fetching envoy Envoy SN data: Invalid port
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/httpx/urlparse.py", line 339, in normalize_port
port_as_int = int(port)
ValueError: invalid literal for int() with base 10: '<>_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 258, in _async_refresh
self.data = await self._async_update_data()
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 217, in _async_update_data
return await self.update_method()
File "/config/custom_components/enphase_envoy/init.py", line 49, in async_update_data
await envoy_reader.getData()
File "/config/custom_components/enphase_envoy/envoy_reader.py", line 368, in getData
await self._getEnphaseToken()
File "/config/custom_components/enphase_envoy/envoy_reader.py", line 297, in _getEnphaseToken
await self._refresh_token_cookies()
File "/config/custom_components/enphase_envoy/envoy_reader.py", line 308, in _refresh_token_cookies
token_validation_html = await self._async_fetch_with_retry(
File "/config/custom_components/enphase_envoy/envoy_reader.py", line 189, in _async_fetch_with_retry
resp = await client.get(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1754, in get
return await self.request(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1517, in request
request = self.build_request(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 346, in build_request
url = self._merge_url(url)
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 376, in _merge_url
merge_url = URL(url)
File "/usr/local/lib/python3.10/site-packages/httpx/_urls.py", line 113, in init
self._uri_reference = urlparse(url, **kwargs)
File "/usr/local/lib/python3.10/site-packages/httpx/_urlparse.py", line 246, in urlparse
parsed_port: typing.Optional[int] = normalize_port(port, scheme)
File "/usr/local/lib/python3.10/site-packages/httpx/_urlparse.py", line 341, in normalize_port
raise InvalidURL("Invalid port")
httpx.InvalidURL: Invalid port
`

[catsmanac]

Is this new install or was it running before?

[yoannHertienne]

@catsmanac it's was working before. It's working when I uninstall and install the integration again, but after some time I'm getting "Check the configuration: Invalid port"

[catsmanac]

Does it run again (for a while) when using the option to reload the envoy integration (integration screens, the 3 dots menu on the envoy integration) instead of uninstall/install it.

(herlaad = reload)

[yoannHertienne]

he tried to reload, but after some seconds, I'm getting the same error.

[catsmanac]

Can you enable debug, in the same dot menu, en reload it again. That will add more details to the log file so you may find some more details.

[yoannHertienne]

@catsmanac After enabling debug, I'm still getting same log

`Logger: custom_components.enphase_envoy
Source: custom_components/enphase_envoy/envoy_reader.py:189
Integration: Enphase Envoy (DEV)
First occurred: 10 mai 2023 à 21:06:31 (421 occurrences)
Last logged: 08:54:57

Unexpected error fetching envoy Envoy S/N data: Invalid port
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/httpx/_urlparse.py", line 339, in normalize_port
port_as_int = int(port)
ValueError: invalid literal for int() with base 10: 'MY_IPV6'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 258, in _async_refresh
self.data = await self._async_update_data()
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 217, in _async_update_data
return await self.update_method()
File "/config/custom_components/enphase_envoy/init.py", line 49, in async_update_data
await envoy_reader.getData()
File "/config/custom_components/enphase_envoy/envoy_reader.py", line 368, in getData
await self._getEnphaseToken()
File "/config/custom_components/enphase_envoy/envoy_reader.py", line 297, in _getEnphaseToken
await self._refresh_token_cookies()
File "/config/custom_components/enphase_envoy/envoy_reader.py", line 308, in _refresh_token_cookies
token_validation_html = await self._async_fetch_with_retry(
File "/config/custom_components/enphase_envoy/envoy_reader.py", line 189, in _async_fetch_with_retry
resp = await client.get(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1754, in get
return await self.request(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1517, in request
request = self.build_request(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 346, in build_request
url = self._merge_url(url)
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 376, in _merge_url
merge_url = URL(url)
File "/usr/local/lib/python3.10/site-packages/httpx/_urls.py", line 113, in init
self._uri_reference = urlparse(url, **kwargs)
File "/usr/local/lib/python3.10/site-packages/httpx/_urlparse.py", line 246, in urlparse
parsed_port: typing.Optional[int] = normalize_port(port, scheme)
File "/usr/local/lib/python3.10/site-packages/httpx/_urlparse.py", line 341, in normalize_port
raise InvalidURL("Invalid port")
httpx.InvalidURL: Invalid port
`

[yoannHertienne]

When anonymising my IPV6, I realized that the function port_as_int = int(port) is waiting a literal value for int() with base 10, but is receiving partially the IPV6 address.

Full IPV6 address is 2a01:e0a:XXX:XXXX:XXX:XXXX:fe7b:5f08 and is receiving only e0a:XXX:XXXX:XXX:XXXX:fe7b:5f08.

Not sure the port should be the IPV6 address.

Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/httpx/_urlparse.py", line 339, in normalize_port
port_as_int = int(port)
ValueError: invalid literal for int() with base 10: 'e0a:XXX:XXXX:XXX:XXXX:fe7b:5f08'

[catsmanac]

Can you change the host to ipv4 address? There's more reports about issues with ipv6 addresses.

[pvancald]

I confirm the exact same issue with ipv6. Had to disable it to restore correct running even if I used the ipv4 of the envoy during the setup.

[yoannHertienne]

Can you change the host to ipv4 address? There's more reports about issues with ipv6 addresses.

I was configuring envoy integration using local IPV4 address (e.g. 192.168.X.X.). I don't know, why is using IPV6 later.

[sh4rks]

Can you change the host to ipv4 address? There's more reports about issues with ipv6 addresses.

I was configuring envoy integration using local IPV4 address (e.g. 192.168.X.X.). I don't know, why is using IPV6 later.

Same for me. I had to disable ipv6 on my router.
A fix which will check that the ip is an ipv4 address will be benefit.

[catsmanac]

So you all had it showing the ipv6 address in the configuration screen, replaced it by the ipv4 address and yet the ipv6 was used?

Guess to fix you can delete the current envoy integration, restart HA and add it again with ipv4 replacing the ipv6 address it shows?

@posixx has code in his version that will reject any non ipv4 address during the setup.

(A sneaky way to see what it is using currently is to look at the file core.config_entries in the folder config/.storage. Look for envoy and you see the line with host:. You can change it there too but stop HA first before changing it. )

[yoannHertienne]

So you all had it showing the ipv6 address in the configuration screen, replaced it by the ipv4 address and yet the ipv6 was used?

Guess to fix you can delete the current envoy integration, restart HA and add it again with ipv4 replacing the ipv6 address it shows?

@posixx has code in his version that will reject any non ipv4 address during the setup.

(A sneaky way to see what it is using currently is to look at the file core.config_entries in the folder config/.storage. Look for envoy and you see the line with host:. You can change it there too but stop HA first before changing it. )

@catsmanac when checking the core.config_entries file in the config/.storage, I realised that it's using the IPV6 address.
I also confirm, that I never used the IPV6 for the envoy integration, but always the IPV4. I don't know why the IPV4 is replaced by IPV6 and by which process.

[catsmanac]

@yoannHertienne in your case you had it working all fine, then reinstalled the integration and it started this behavior? Did you also update HA itself then, like did this start with update to most recent HA? Any other updates that may help finding out why this now occurs?

[yoannHertienne]

@catsmanac In my case everything was working fine, but last HA updates should be the root cause of this issue.

I did the following HA updates:

• Home Assistant 2023.4.6 on May 5th 18h44
• Home Assistant 2023.5.1 on May 6th 07h57
  when checking the history of the current production, the issue seems to be starting on May 5th 19h00

as you can see, sometimes it changed during the night, without any actions from my side.

My way to fix it is to uninstall and reinstall the integration and then is working for some time and then I'm getting this issue. When checking when it's working, host is my IPV4 (just checked yet) and when it's not working, host is my IPV6

[catsmanac]

@yoannHertienne that's weird indeed. What platform (docker, hassos, linux) are you using to host HA? Is your envoy assigned a fixed IP address in your network?

[yoannHertienne]

@yoannHertienne that's weird indeed. What platform (docker, hassos, linux) are you using to host HA? Is your envoy assigned a fixed IP address in your network?

I'm using a Raspberrry Pi to host HA using Alpine Linux.

My Envoy has a fixed IP.

[catsmanac]

I used a test install on docker on ubuntu running 2023.4.6. I stopped HA and changed in the core.config_entries the ip to the ipv6 of the envoy and then restarted HA. Right after the start the IP in the config file was changed back to IPV4 address that was there. As said it's the reverse case, but it shows something happens during the start.

[yoannHertienne]

I used a test install on docker on ubuntu running 2023.4.6. I stopped HA and changed in the core.config_entries the ip to the ipv6 of the envoy and then restarted HA. Right after the start the IP in the config file was changed back to IPV4 address that was there. As said it's the reverse case, but it shows something happens during the start.

seems we're not the only one: home-assistant/core#92603

[catsmanac]

Same error, but different cause. That one had the port number twice, in your case the issue is with ipv6 address having : in the address being interpreted as start of the port number. Still trying to find if something changed in core since 2023.4.5 that can explain this.

[catsmanac]

I was doing a direct hhtp request to my envoy and ipv6 works, provided it has [ ] around it:

GET https://[fe80::21d:c0ff:fe7f:b63b]/auth/check_jwt
Authorization: Bearer {{token}}
Accept: application/json

I guess that in your core.config_entries when it showed an ipv6 address there's no [ ] around it?

[yoannHertienne]

I was doing a direct hhtp request to my envoy and ipv6 works, provided it has [ ] around it:

GET https://[fe80::21d:c0ff:fe7f:b63b]/auth/check_jwt
Authorization: Bearer {{token}}
Accept: application/json

I guess that in your core.config_entries when it showed an ipv6 address there's no [ ] around it?

Yes, in the core.config_entries, it's only the IPV6 of the envoy, without the [] around it.

[catsmanac]

I did some further testing and there's 2 things that play a role here. First, the automatic detection returns the ipv6 address and will overwrite any address in the config file each time HA starts. Second, the ipv6 communication is not working in the current code. It should use the [] around the ipv6 address but even with these added it just fails when I test it in docker or ubuntu. On windows it works, so it's probably some library version issue.

So what are the options? Now looking into the automatic detection and if that can be disabled. That would end the replacement of the ipv4 address with the ipv6 address in the config file. Or see if it will favor the ipv4 address. Other option is to disable ipv6 on you alpine linux if that is an option for you. The real fix is obviously to get the ipv6 comm supported, but that needs more digging and help.

[catsmanac]

The automatic detection can not simply be disabled for this and its mechanism is intended to find changes in addresses and automatic update. So that leaves disabling ipv6 in Linux as option for now until there's better support for ipv6 in HA auto detect and get requests it seems.

[yoannHertienne]

@catsmanac Thank you for your feedback. However, I'm wondering if the issue is also not coming from the fact that IPV6 address is not complete. Because in the core.config_entries file I can see the full IPV6 address of my envoy, where in the logs, it's only a part of it.

Full IPV6 : 2a01:e0a:XXX:XXXX:XXX:XXXX:fe7b:5f08
the one I see in logs : e0a:XXX:XXXX:XXX:XXXX:fe7b:5f08

[catsmanac]

I wonder if that is because it thinks e0a:XXX:XXXX:XXX:XXXX:fe7b:5f08 is the port number right after the first : as it seems to process is as if it is a ipv4. If you enable debug for envoy then you can see the actual http request it is using in the log like example below. I inserted the ipv6 address with [] around it in the config file and it used it completely until it got changed back to ipv4 by zeroconf.

HTTP GET Attempt #1: https://[fe80::21d:c0ff:fe7f:b63b]/auth/check_jwt:

[yoannHertienne]

@catsmanac to be able to make it run, I finally decided to change row 102 in your envoy_reader.py from
self.host = host.lower() to self.host = "<<MY_IPV4 entered as host>>:443"

[catsmanac]

@yoannHertienne that's a simple straightforward patch for now. As for 'your envoy_reader.py', it's @briancmpbll's work that made this available, all kudo's to him.

From your info it seems this started with upgrade to HA 2023.4.6. One would think that zeroconf returns the ipv6 since then and replaces the ipv4 entry in the config with the ipv6 and then fails. The HA release notes don't mention much on this.

[curtmgray]

I had a similar problem with this, the detail in @yoannHertienne was a great work around for my issue! I'll go with that for now until a better fix can be implemented. Is there anything I can provide to help with this?

[catsmanac]

Each integration has an option menu in the lower right corner. That includes a menu item System Options. In the System options panel one can disable detection of new Enphase devices.

When disabling this option as shown in above example, the file core.config_entries shows it in the envoy entry as true where it is false by default.

        "pref_disable_new_entities": true,

I am now looking into using that flag in the async_step_zeroconf code section of config_flow.py. Logic would be to not apply the discovered entry if it exist with same serialnumber. Current code will change the ip address as we now know very well.

This will keep it future compatible whenever ipv6 is better supported. Changing the setting of this back again will let it work as before. It will not impact current function as the system option switch can only be changed after the initial configuration is done.

I also found that the original envoy code in HA Core actually has a test to only allow IPV4 in the zero configuration code async_step_zeroconf. That part is not in @briancmpbll code. It could be re-added but I feel that using the system settings option is the better path forward to avoid future code changes again.

So @yoannHertienne and @curtmgray if you are willing, I have something to test using above method.

• Open config_flow.py and find the line async def async_step_zeroconf(

• Update the first part of the code to look like

    async def async_step_zeroconf(
        self, discovery_info: zeroconf.ZeroconfServiceInfo
    ) -> FlowResult:
         """Handle a flow initialized by zeroconf discovery."""
        serial = discovery_info.properties["serialnum"]
        _LOGGER.debug("Envoy zeroconf %s %s, %s",serial,discovery_info.host,self.ip_address)
        await self.async_set_unique_id(serial)
        for entry in self._async_current_entries(include_ignore=False):
            if entry.pref_disable_new_entities:
                _LOGGER.debug("Envoy zeroconf update ended, disable new: %s for %s ",entry.pref_disable_new_entities,serial)
                return self.async_abort(reason="pref_disable_new_entities")
        self.ip_address = discovery_info.host

• Undo your previous change of hardcoded ip
• Set the System configuration option to disable automatic detection for the envoy as shown above
• Stop HA
• Update the ip in core.config_entries to your ipv4
• "pref_disable_new_entities" in core.config_entries for the envoy should now be set to true if you disabled the system option
• Restart HA

When debug is enabled you'll see something like this in the log:

2023-05-19 22:24:01.202 DEBUG (MainThread) [custom_components.enphase_envoy.config_flow] Envoy zeroconf 122302045041 192.168.3.112, None
2023-05-19 22:24:01.203 DEBUG (MainThread) [custom_components.enphase_envoy.config_flow] Envoy zeroconf update ended, disable new: True for 122302045041 

[catsmanac]

@yoannHertienne excellent, did it solve the ipv6 issue?

[yoannHertienne]

@yoannHertienne excellent, did it solve the ipv6 issue?

Yes, it was working like "hardcoded" patch ;), meaning using the ipv4.

[catsmanac]

Great, I'll prepare it as a Push request for @briancmpbll

[catsmanac]

@yoannHertienne you may want to try PR 87 by @spasma to solve the ipv6 issue.

[yoannHertienne]

@catsmanac I'm trying. I will keep you posted.

[Coder84619]

Any update on a long term IPv6 fix?

[catsmanac]

Looks like the fix by @spasma works and and his pull request 87 in pending commit by @briancmpbll or whomever is authorized to do so.

[posixx]

I see this is committed to have autodiscovery disabled in order to prevent ip updates. In my version of the plugin i use a routine to make sure ithe ip is a ipv4 address, otherwise abort the zeroconf discovery. I think is is much cleaner option. What do you guys think?

[catsmanac]

The ipv6 issue is solved by #87. It now handles ipv6 and ipv4. The auto-discovery disablement is a HA system option that was not implemented in this integration and it was used the help out those that switched to ipv6 each time by auto-discovery. They would disable auto-discovery for envoy and configure envoy manually with ipv4 address and stay on that. If you only allow ipv4 in auto-discovery it will recreate the issues for those where auto-discovery returns the ipv6. So I'd recommend to leave as is.unless you actual issues.

[posixx]

Ah ok i will restore the disablement. I prefer to have the option to disable ipv6 detection instead of disable auto-diiscovery in general. Perhaps make this a configurable option on the initial config screen?
My whole network has ipv6 disabled but as the envoy has it's own ipv6 address as well as HA it needs to be disabled in code,,

[spasma]

Ah ok i will restore the disablement. I prefer to have the option to disable ipv6 detection instead of disable auto-diiscovery in general. Perhaps make this a configurable option on the initial config screen?

My whole network has ipv6 disabled but as the envoy has it's own ipv6 address as well as HA it needs to be disabled in code,,

If ipv6 is disabled as a whole in your network, how can HA auto-discover ipv6 devices in your network? I think it wont, or am i wrong?

[posixx]

You are wrong. Even if your routers etc have IPV6 disabled, devices like the envoy can expose ipv6 addresses, and communicate with other devices which do expose a ipv6 address. Like your windows PC, it by default also exposes ipv6 address even if it's on a ipv4-only network.

[catsmanac]

Ah ok i will restore the disablement. I prefer to have the option to disable ipv6 detection instead of disable auto-diiscovery in general. Perhaps make this a configurable option on the initial config screen? My whole network has ipv6 disabled but as the envoy has it's own ipv6 address as well as HA it needs to be disabled in code,,

Disable auto detection is an overall HA system feature that was not implemented in this integration until we added it as practical stop-gap to the ipv6 issue then. So supporting a HA system wide function is a good thing I think.

As for the disable ipv6 configuration option that could be an option. But wouldn't you have to configure the envoy manually as auto-detected host ip can't be changed in config screen I think and that might be the ipv6 one?!

[posixx]

Well i think we can solve that by disabling ipv6 default, and make ipv6 an option. So when not configured the ipv6 address will abort, like this (async_step_zeroconf):

self.ip_address = discovery_info.host
if not is_ipv4_address(self.ip_address) and not enable_ipv6:
return self.async_abort(reason="ipv6_not_supported")

As i'm not familiar with accessing the configurable options; how do i get the value of enable_ipv6 (now just as sample parameter) which is configured on the initial setup screen (and by default will be False)? Do you know this?

[catsmanac]

Below an example of a call to init of envoy reader in init.py passing the setting for disable_installer_account_use providing the value from the configuration item DISABLE_INSTALLER_ACCOUNT_USE using config.get. If the item does not exist it will default to False.

    envoy_reader = EnvoyReader(
.
.
.
        disable_installer_account_use=config.get(DISABLE_INSTALLER_ACCOUNT_USE, False),
        ),
    )

Mind that config values can only be specified at configuration time. To change these you'll need to remove and re-add the envoy. You can also use options and option.get, these can be changed on the fly with only needing a reload, not a remove/add. For this you would use options.get in init.py. But it would require a class EnvoyOptionsFlowHandler in config_flow.py

I would vote for enabled ipv6 as default as that is how it currently works. If you default to not enabled it may become a breaking change for current users with ipv6.

[posixx]

I already have setup the setting in the EnvoyReader, my question is about getting that value within the async_step_zeroconf routine.

I'm sure 99% of the current users which use this integration have an ipv4 based configuration at the moment. So if we make ipv6 default it can change the current configuration. If we disable ipv6 at default everything will keep working when the integration is updated after merges.

[posixx]

I've deleted the pull request as i think it cannot be done correctly within zeroconf. Maybe the only option is to check if newly discovered ipaddress is different format of existing ipaddress. So that way only ipv4 changes and ipv6 changes are executed, a change from ipv4 to ipv6 (or the other way) will not. What do you think?

[catsmanac]

In zeroconf, to access a configuration item specified in the configuration screen use _async_current_entries data

        for entry in self._async_current_entries(include_ignore=False):
            _LOGGER.debug("entry: %s",entry.data)

That gave me:

 entry: {'host': '192.168.3.112', 'username': 'blahl', 'password': 'blah', 'serial': '<hugenumber>', 'use_enlighten': True, 'name': 'Envoy <serial>'}

If you specify new config entry 'don't switch between ipv4 and ipv6' setting item don_switch_ipversion it will show in there and can be used as

            _LOGGER.debug("entry: %s",entry.data["don_switch_ipversion"])

See file .storage/core.config_entries for the data section.

[catsmanac]

and the discovery_info contains:

	(host='192.168.3.112', 
	addresses=['192.168.3.112', 'fe80::21d:c0ff:fe7f:b63b'], 
	port=80, 
	hostname='envoy.local.', 
	type='_enphase-envoy._tcp.local.', 
	name='envoy._enphase-envoy._tcp.local.', 
	properties={'txtvers': '1', 
	'protovers': '7.6.175', 
	'serialnum': '122302045041'})

[posixx]

How do you make the discovery_info content available like that?

[catsmanac]

Added a _Logger.debug for discovery_info, in configuration.yaml force envoy logging to debug, Restart HA and wait a bit until discovery fires. I then tidied the line manually to what you see.

logger:
#  default: info
  default: warning
  logs:
    custom_components.enphase_envoy: debug

[posixx]

Yes tried that but i got an error the discovery_info cannot be iterated. But nevermind; i can access the options manually.

I have created a pull request #110 for correct ip discovery on ipv4-only networks. I think this approach is the safest possible, and will not break existing installations.