Better support for broken caldav servers: do client-side parsing of rrules on expanded date searches

[tobixen]

Ref #147, #86 and probably many others ...

This caldav client library is expecting the caldav server to handle recurring events.
Unfortunately quite many servers does not support it fully. The list goes like this:

• Xandikos - reportedly, there should be full support for recurring events on newer versions, but I haven't been able to get the tests to pass.
• Radikale - does not work, ref Expand Recurring Events Kozea/Radicale#662
• Zimbra - while quite a lot of things break in their caldav support, recurring evens and expansion of them actually works
• Bedework - does not work
• Baikal - it works (but not for tasks)
• iCloud - it works (but quite some other things break - caldav is not officially supported)
• DAViCal - it works
• Google - it does not work
• SOGo - it does not work
• Nextcloud - it works (but not for tasks)
• Fastmail - it works

This issue is for making workarounds on the client side if needed, allowing consistent return values to the library user regardless of what the server supports.

• Under the test directory, some framework has been made for marking compatibility issues. This should probably be moved from the test directory, so that it's possible to attach known compatibility issues to a davclient object.
• The code should behave differently dependent on the compatibility issue list. Specifically, quite some servers have problems with recurring events. For non-compliant servers, the date search algorithm may need to take out all events, analyze the rrule given and filter out the interesting events.
• Issue date_search(..., expand=True) should probably do a better job on expanding the icalendar data #86 has some example code that may be used

Other incompatibility workarounds should be made as separate issues and should refer to this one.

[tobixen]

There is some python code contributed in #86 that may be useful for client-side expansion.

[metronidazole]

Fastmail - it works

Are you sure about that?

I tried your example for this library with fastmail and had some odd results. I have a recurring event starting on the first of June, and I found the number of events returned were the same regardless of whether expand was enabled.

len(testcal.date_search(start=datetime(2022,6, 3), end=datetime(2022, 6,27), expand=False))
2

len(testcal.date_search(start=datetime(2022,6, 3), end=datetime(2022, 6,27), expand=True))
2

Is that supposed to happen, or does that mean FastMail is faking expansion of recurring events?

[tobixen]

Fastmail - it works

Are you sure about that?

I created a dedicated fastmail account for testing the caldav library and signed up for a "free trial period". As far as I can remember, the test code was passing, which means there should be support for recurring events. I sent a request to the Fastmail support team to keep the account open for free after the trial period had expired, but all they could offer was a 25% discount ... so I can't test more with Fastmail.

I tried your example for this library with fastmail and had some odd results. I have a recurring event starting on the first of June, and I found the number of events returned were the same regardless of whether expand was enabled.

len(testcal.date_search(start=datetime(2022,6, 3), end=datetime(2022, 6,27), expand=False))
2

len(testcal.date_search(start=datetime(2022,6, 3), end=datetime(2022, 6,27), expand=True))
2

Is that supposed to happen, or does that mean FastMail is faking expansion of recurring events?

If I remember correct, the actual data there is one vCalendar containing multiple vEvents.

[mendhak]

Please could you expand (haha...) on that answer a bit?

If I do

events_data = calendar.date_search(start=from_date, end=to_date, expand=True)

Which property should I be looking at to get the recurring tasks?

I tried events_data[0].data but it's a giant string.

I tried events_data[0].vobject_instance which does show a lot of items, but I don't know how to iterate them.

I tried looking at events_data[0].vobject_instance.vevent but that only shows the first occurrence of the event.

Not sure how to proceed after that.

[tobixen]

This will be easier in the near future, as soon as the pull request #222 is merged (or check out the branch feature_split_expanded). Then, if you use calendar.search(event=True, start=..., stop=...) rather than calendar.date_search(), you will get a list, making it easier to iterate through everything.

This got stalled a bit, but I'll try to get it merged and released as v.0.10.1 real soon now.

[tobixen]

By the way, events_data[0].icalendar_instance.subcomponents is the easiest way to iterate through the expanded recurrences.

[mendhak]

Ah that's it, thanks! I see, so icalendar_instance is a good way to do this. I'll use this for now and I'll switch over to the new way in the future, many thanks for your work with this very tricky 'protocol'

[poudenes]

Hi All,

Suddenly I get those errors. Never seen them now many in log file:

2023-01-24 07:48:49.442 ERROR (SyncWorker_9) [root] CalDAV server does not support recurring events properly.  See https://github.com/python-caldav/caldav/issues/157
2023-01-24 07:48:49.467 ERROR (SyncWorker_9) [root] CalDAV server does not support recurring events properly.  See https://github.com/python-caldav/caldav/issues/157

Home Assistant 2023.1.7
Supervisor 2022.12.1
Operating System 9.4
Frontend 20230110.0 - latest

[tobixen]

Perhaps you have some recurring events in the calendar now, that you didn't have earlier. There is a fix in the pipeline ... home-assistant/core#85536

[poudenes]

Perhaps you have some recurring events in the calendar now, that you didn't have earlier. There is a fix in the pipeline ... home-assistant/core#85536

For couple of weeks no new events added in calendar. Some just for 1 hour or so.... So I was surprised with this error

[tobixen]

Well, something must have changed. If I'm not mistaken, the caldav library version was last updated in HomeAssistant in 202203, so unless you recently did a major upgrade of home-assistant it shouldn't be the caldav library. Then it must either be the content from the caldav server or the caldav server itself. I'm not sure how the calendar integration in HomeAssistant works, but I know it's doing a date search - perhaps it's only fetching events in a window around the current date, perhaps there is a recurring event in the calendar that until now has been outside this window.

In any case, the error is mostly harmless, except that recurring events most likely won't come up in the calendar ... and a fix is on the way.

[poudenes]

Well, something must have changed. If I'm not mistaken, the caldav library version was last updated in HomeAssistant in 202203, so unless you recently did a major upgrade of home-assistant it shouldn't be the caldav library. Then it must either be the content from the caldav server or the caldav server itself. I'm not sure how the calendar integration in HomeAssistant works, but I know it's doing a date search - perhaps it's only fetching events in a window around the current date, perhaps there is a recurring event in the calendar that until now has been outside this window.

In any case, the error is mostly harmless, except that recurring events most likely won't come up in the calendar ... and a fix is on the way.

HA version is 2023.1.7 so latest one. That can be the issue. But I see all my appointments in the calendar. So it work.