[TypeError]: fromisoformat: argument must be str

[DEEJCUK]

What happened?

Hi, I'm trying to get my data in grafana, i found your importer which seems perfect but the import keeps crashing at Inserted 430000 records over and over, always the same place, any ideas what might be going on here ?

export.zip is 131MB in size

Environment

Docker

Relevant log output

health-ingester  | Inserted 420000 records
influxdb        | [httpd] 172.21.0.8 - root [23/Jun/2024:06:56:59 +0000] "POST /write?db=health&precision=s HTTP/1.1 " 204 0 "-" "python-requests/2.31.0" c995d268-312d-11ef-8ace-0242ac15001c 28687
health-ingester  | Inserted 430000 records
health-ingester  | Traceback (most recent call last):
health-ingester  |   File "//app.py", line 197, in <module>
health-ingester  |     process_health_data(client)
health-ingester  |   File "//app.py", line 146, in process_health_data
health-ingester  |     rec = format_record(elem)
health-ingester  |           ^^^^^^^^^^^^^^^^^^^
health-ingester  |   File "//app.py", line 64, in format_record
health-ingester  |     date = parse_date_as_timestamp(record.get("startDate", 0))
health-ingester  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
health-ingester  |   File "/formatters.py", line 19, in parse_date_as_timestamp
health-ingester  |     return int(dt.fromisoformat(v).timestamp())
health-ingester  |                ^^^^^^^^^^^^^^^^^^^
health-ingester  | TypeError: fromisoformat: argument must be str
health-ingester exited with code 0
health-ingester  | Unzipping the export file...
health-ingester  | Export file unzipped!

docker-compose.yml

influx:
    image: influxdb:1.8
    container_name: influxdb
    restart: unless-stopped
    environment:
      - INFLUXDB_DB=health
    volumes:
      - ${CONFIG_ROOT}/influx:/var/lib/influxdb
    ports:
      - 8086:8086
    logging:   
      driver: loki
      options:
        loki-url: ${LOKI_URL}
        mode: non-blocking
        max-buffer-size: 4m
        loki-retries: "3"
  ingester:
    image: k0rventen/apple-health-grafana-ingester:v0.5.0
    container_name: health-ingester
    restart: unless-stopped
    volumes:
      - ${CONFIG_ROOT}/influx/export.zip:/export.zip
    logging:   
      driver: loki
      options:
        loki-url: ${LOKI_URL}
        mode: non-blocking
        max-buffer-size: 4m
        loki-retries: "3"

[k0rventen]

Hi ! Seems like the timestamp of a specific record isn't formatted as expected. Could you provide the following informations to help debug ?

• IOS version
• Any additional sources you are using (Apple Watch, smart/connected thing that appear in the Health app) ?

As this is processsing quite sensitive datas, I do not log any records. That makes troubleshooting harder, but if necessary I'll create a more verbose version for you to run so we can fix this issue.

Also on a separate note, this makes me realize that I should handle more gracefully such unexpected errors, rather than 'exploding' :)

Regards,

[DEEJCUK]

IOS Version is latest 17.5.1
all together i have quite a few sources over the years, Garmin, apple watch, withings, forrest, map my run, komoot etc.. in total i see 28 sources in health 😄

i asked ChatGPT to help me with code to skip the malformed records etc and now it finishes but i noticed this

health-ingester  | Export file is /export/apple_health_export/export.xml
health-ingester  | Removing potentially malformed XML..

and the end shows

health-ingester  | Inserted 430000 records                                                                                                                                                10:34:33 [10/38219]
health-ingester  | Total number of records: 439438
health-ingester  | pushing 13 sources !
health-ingester  | All done! You can now check grafana.

but only workouts are imported, every other dashboard is blank

i also noticed that at the end of the ingest, it wipes the DB and restarts by unzipping the export and starting over, so i again asked chatGPT to give me the code to stop this and now it looks at the export.zip date and if it hasnt changed it doesnt start over so that maybe an idea.. unless this is specific to my case that it restarts over
thanks

[DEEJCUK]

actually, on further checks, i DO see some data, BMI, Body Mass, some heartrate, but no sleep at all, and nothing on the health specific metrics dash

[k0rventen]

430000 records for a 130Mb export file is quite low, should be expecting around 3-4millions records. That's why not all of your data is showing up.

I've pushed what I believe is a first fix on this issue. I think some of your records do not have a startDate attr, and this wasn't handled properly. This might not 'fix' it, but at least it shouldn't crash. If the new ingester goes further, that means we are on the right track.

Could you please change the ingester image to k0rventen/apple-health-grafana-ingester:rolling in your compose file and try again ?
thx 😄

[DEEJCUK]

it doesnt stop now, but it doesnt continue past the same amount of records as before, and it also restarts over and over, when it restarts it seems to wipe the DB of any records

health-ingester  | Inserted 430000 records
health-ingester  | Total number of records: 439438
health-ingester  | pushing 13 sources !
health-ingester  | All done! You can now check grafana.
health-ingester exited with code 0
# docker-compose logs -f ingester
health-ingester  | Unzipping the export file...
health-ingester  | Export file unzipped!
health-ingester  | Influx is ready.
health-ingester  | Loading workout routes ...
health-ingester  | Opening Route 2021-05-25 7:04pm

i also see this, im not sure if its relevant ?

health-ingester  | Export file is /export/apple_health_export/export.xml
health-ingester  | Removing potentially malformed XML..
health-ingester  | Inserted 10000 records

[k0rventen]

Regarding the wiping of previous data this is expected behavior.

I would suggest re-exporting your health data from your iPhone and starting again with this 'fresh' export, it's not uncommon for the export to be borked (see #4 and #14. This is also why you see a 'Removing potentially malformed XML..' line at the start, I try to fix it before loading it as a quick and dirty prevention step but this might not fix all possible cases.)

If the new export doesn't work, I would suggest using an XML linter to verify the validity of the file ?

[DEEJCUK]

Ah Ha!
so my export was borked in multiple ways like this (and control characters everywhere EG: ^B^D^B^D):

name: FITNESS3�[[�

and the CDA xml export have the file terminated in the wrong place, i i moved the below to the end of the file from another place where it was wrongly placed

</organizer>
</entry>
</ClinicalDocument>

and i ended up

health-ingester  | Inserted 7810376 records
health-ingester  | Total number of records: 7811968
health-ingester  | pushing 31 sources !
health-ingester  | All done! You can now check grafana.
health-ingester exited with code 0

but it still remains that the ingest process restarts after this and wipes the DB

[k0rventen]

Ok, I think we weren't understanding each other. The ingester is supposed to clear the db when starting up. But it's not supposed to loop indefinitely.

I believe you added restart: unless-stopped to the ingester service definition (it's not in the template here), and this is what's making it restarts. Removing this line should fix this behavior.

Regarding the export itself, I'm glad we found the issue. I'll improve the documentation to mention explicitely the fact that the export can be messed up.

[DEEJCUK]

I believe you added restart: unless-stopped to the ingester service definition (it's not in the template here), and this is what's making it restarts. Removing this line should fix this behavior.

ahh! that makes sense, i assumed the ingester should run indefinitely and all my containers are set to restart like this so its a habit, thanks for your help on this

[DEEJCUK]

if its helpful in the future, i am using this on the exported xml files

vim -E -s export*.xml << EOF
:%s/[[:cntrl:]]//g #removes the control chars
:%s/\%uFFFD\[\[\%uFFFD//g #removes the unicode symbol i had
:update
:quit
EOF

which fixes the files