You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/load.md
+23-10Lines changed: 23 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,11 +12,12 @@ It also allows loading from URLs and `urllib` requests.
12
12
If a file extension is not provided or is not recognized, the compression format will be automatically detected by reading the file's **"magic number."**
13
13
14
14
If the detection fails, the file is considered uncompressed.
15
-
```
16
15
17
-
#### Load an uncompressed file given a path.
16
+
#### Load an uncompressed file given a path
18
17
19
18
```python
19
+
# -*- coding: utf-8 -*-
20
+
20
21
import jsonl
21
22
22
23
path ="file.jsonl"
@@ -35,11 +36,13 @@ iterator = jsonl.load(path)
35
36
print(tuple(iterator))
36
37
```
37
38
38
-
#### Load a compressed file given a path.
39
+
#### Load a compressed file given a path
39
40
40
41
Check [note](#note-compression) for more details
41
42
42
43
```python
44
+
# -*- coding: utf-8 -*-
45
+
43
46
import jsonl
44
47
45
48
path ="file.jsonl.gz"# gzip compressed file, but it can be ".bz2" or ".xz"
@@ -58,12 +61,14 @@ iterator = jsonl.load(path)
58
61
print(tuple(iterator))
59
62
```
60
63
61
-
#### Load a file from an open file object.
64
+
#### Load a file from an open file object
62
65
63
66
!!! tip
64
67
This is useful when you need to load a file from a custom source.
65
68
66
69
```python
70
+
# -*- coding: utf-8 -*-
71
+
67
72
import jsonl
68
73
69
74
path ="file.jsonl"
@@ -83,12 +88,14 @@ with open(path) as fp:
83
88
print(tuple(iterator))
84
89
```
85
90
86
-
#### Load from a URL.
91
+
#### Load from a URL
87
92
88
93
You can load a JSON Lines directly from a URL incrementally, if needed you can also create custom
89
94
requests using `urllib.request.Request`.
90
95
91
96
```python
97
+
# -*- coding: utf-8 -*-
98
+
92
99
import urllib.request
93
100
import jsonl
94
101
@@ -102,13 +109,15 @@ iterator = jsonl.load(req)
102
109
print(tuple(iterator))
103
110
```
104
111
105
-
#### Load a file containing broken lines.
112
+
#### Load a file containing broken lines
106
113
107
114
!!! warning
108
115
If the **broken** parameter is set to `False`, the function will raise an `Exception` when it encounters a broken line.
109
116
If set to `True`, the function will skip the broken line, continue reading the file, and log a warning message.
110
117
111
118
```python
119
+
# -*- coding: utf-8 -*-
120
+
112
121
import jsonl
113
122
114
123
# Create a file with broken JSON lines
@@ -124,12 +133,12 @@ print(tuple(iterator))
124
133
125
134
*Output:*
126
135
127
-
```console
136
+
```text
128
137
WARNING:root:Broken line at 2: Expecting ',' delimiter: line 2 column 1 (char 28)
0 commit comments