ingester container fails Importing data info influxdb

[rashamalek]

Trying to ingest the data, the ingester container fails due to a ParseError

$ docker-compose up ingester
Creating applehealthgrafana_ingester_1 ... done
Attaching to applehealthgrafana_ingester_1
ingester_1  | unzipping the export file..
ingester_1  | export file unzipped
ingester_1  | influx is ready
ingester_1  | loading workout routes
ingester_1  | opening Route 2022-04-04 5:09pm
.....
ingester_1  | opening Route 2022-05-18 1:52pm
ingester_1  | Traceback (most recent call last):
ingester_1  |   File "//app.py", line 147, in <module>
ingester_1  |     process_health_data()
ingester_1  |   File "//app.py", line 107, in process_health_data
ingester_1  |     for _, elem in etree.iterparse(export_path):
ingester_1  |   File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1249, in iterator
ingester_1  |     yield from pullparser.read_events()
ingester_1  |   File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1320, in read_events
ingester_1  |     raise event
ingester_1  |   File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1292, in feed
ingester_1  |     self._parser.feed(data)
ingester_1  | xml.etree.ElementTree.ParseError: syntax error: line 156, column 0
applehealthgrafana_ingester_1 exited with code 1

The export.zip contains the following files

$ ls -ltra
total 904176
-rw-r--r-- 1 root root 664488691 Mar  2 17:30 export.xml
-rw-r--r-- 1 root root 261333433 Mar  2 17:30 export_cda.xml
drwxr-xr-x 7 root root      4096 Mar  3 11:31 ..
drwxr-xr-x 2 root root     28672 Mar  3 11:32 workout-routes
drwxr-xr-x 2 root root      4096 Mar  3 11:32 electrocardiograms
drwxr-xr-x 4 root root      4096 Mar  3 11:32 .

Here is the container status:

#$ docker ps -a | grep health
693fa4d42d6f        k0rventen/apple-health-grafana-ingester   "/bin/sh -c 'pytho..."   58 minutes ago      Exited (1) 55 minutes ago                                                                                applehealthgrafana_ingester_1
6767e7e85b48        grafana/grafana:9.3.8                     "/run.sh"                2 hours ago         Up 2 hours                  0.0.0.0:3000->3000/tcp                                                       applehealthgrafana_grafana_1
aae540aca1c6        influxdb:1.8                              "/entrypoint.sh in..."   2 hours ago         Up 2 hours                  8086/tcp                                                                     applehealthgrafana_influx_1

[k0rventen]

Hi. thx for giving it a try,
Would you mind sharing the version of iOS in use ?
This error is kinda weird, it's like the whole export.xml isn't parse-able by the lib..
I will make a more verbose version that you could try, maybe that will give us more info about the error.

[rashamalek]

Sure,
phone is running iOS version 16.1
The export.xml seems to be parsed partially after running the ingester container multiple times, (ofcourse each time failing), I can see some routes under workout-routes dashboard, but still data is incomplete, also other two dashboards are completely empty.

and Here is the ingester log:

ingester_1  | opening Route 2022-05-16 5:03pm
ingester_1  | opening Route 2022-11-02 12:49pm
ingester_1  | opening Route 2022-12-22 10:48pm
...
...
...
ingester_1  | opening Route 2022-08-29 9:37am
ingester_1  | opening Route 2022-03-05 6:39pm
ingester_1  | opening Route 2023-01-07 2:22pm
ingester_1  | opening Route 2022-05-18 1:52pm
ingester_1  | Traceback (most recent call last):
ingester_1  |   File "//app.py", line 147, in <module>
ingester_1  |     process_health_data()
ingester_1  |   File "//app.py", line 107, in process_health_data
ingester_1  |     for _, elem in etree.iterparse(export_path):
ingester_1  |   File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1249, in iterator
ingester_1  |     yield from pullparser.read_events()
ingester_1  |   File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1320, in read_events
ingester_1  |     raise event
ingester_1  |   File "/usr/local/lib/python3.11/xml/etree/ElementTree.py", line 1292, in feed
ingester_1  |     self._parser.feed(data)
ingester_1  | xml.etree.ElementTree.ParseError: syntax error: line 156, column 0
applehealthgrafana_ingester_1 exited with code 1

[k0rventen]

Ok, so loading the workout routes from their folder works fine, but the health data export (which is another XML file export.xml) fails..
I can create a version that will dump the faulty line, but due to the private nature of the data, I would rather communicate with you through a more private medium. If that's alright with you, let me now how you prefer things to continue.

[rashamalek]

Hey @k0rventen we can continue the discussion my GitHub username a t gmail. had to write it this way to try to slow down spammers :)

[dk98126]

I'm having the same problem with the same IOS version. @rashamalek were you able to solve the problem?

[dk98126]

I found out that my export file was damaged. I tried to open it with web browser and was able to spot damaged lines.

After fixing it everything worked out great!