Some Tests fails with <Python3.14

[dl1jbe]

PyShp Version

3.1.4

Python Version

tested with Python 3.12, Python 3.13 and Python 3.14

Your code

Running pytest fails if Python 3.12 or Python 3.13 is used as interpreter: The failing tests are:

  tests/hypothesis_tests.py::test_dbf_reader_writer_roundtrip
  tests/hypothesis_tests.py::test_shapefile_reader_writer_roundtrip
 
Using Python 3.14 all tests pass.

Full stacktrace

____________________________________________________ test_dbf_reader_writer_roundtrip _____________________________________________________
  + Exception Group Traceback (most recent call last):
  |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4/tests/hypothesis_tests.py", line 818, in test_dbf_reader_writer_roundtrip
  |     @pytest.mark.hypothesis_dbf
  |                    ^^^
  |   File "/usr/lib/python3.12/site-packages/hypothesis/core.py", line 2281, in wrapped_test
  |     raise the_error_hypothesis_found
  | ExceptionGroup: Hypothesis found 2 distinct failures. (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4/tests/hypothesis_tests.py", line 829, in test_dbf_reader_writer_roundtrip
    |     written_fields, written_records = _write_fields_and_records_to_strict(w, fields, records)
    |                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4/tests/hypothesis_tests.py", line 806, in _write_fields_and_records_to_strict
    |     w.record(*rec_list)
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4-python3_12/install/usr/lib/python3.12/site-packages/shapefile.py", line 4381, in record
    |     self._record(record)
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4-python3_12/install/usr/lib/python3.12/site-packages/shapefile.py", line 4441, in _record
    |     raise ShapefileException(
    | shapefile.ShapefileException: Could not read as date: 9990101. Date values must be either a datetime.date object, a list, a YYYYMMDD string, or a missing value.
    | Falsifying example: test_dbf_reader_writer_roundtrip(
    |     codec_fields_and_records=('shift_jis_2004',
    |      [{'name': '0', 'field_type': 'C', 'size': 1, 'decimal': 0},
    |       {'name': '0', 'field_type': 'D', 'size': 8, 'decimal': 0}],
    |      [('', '9990101')]),
    | )
    | Explanation:
    |     These lines were always and only run by failing examples:
    |         /var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4-python3_12/install/usr/lib/python3.12/site-packages/shapefile.py:4441
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4/tests/hypothesis_tests.py", line 829, in test_dbf_reader_writer_roundtrip
    |     written_fields, written_records = _write_fields_and_records_to_strict(w, fields, records)
    |                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4/tests/hypothesis_tests.py", line 806, in _write_fields_and_records_to_strict
    |     w.record(*rec_list)
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4-python3_12/install/usr/lib/python3.12/site-packages/shapefile.py", line 4381, in record
    |     self._record(record)
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4-python3_12/install/usr/lib/python3.12/site-packages/shapefile.py", line 4537, in _record
    |     raise ShapefileException(
    | shapefile.ShapefileException: Shapefile Writer unable to pack incorrect sized value=datetime.date(999, 1, 1) (encoded as 7B) into field '0' (8B).
    | Falsifying example: test_dbf_reader_writer_roundtrip(
    |     codec_fields_and_records=('shift_jis_2004',
    |      [{'name': '0', 'field_type': 'D', 'size': 8, 'decimal': 0}],
    |      [(datetime.date(999, 1, 1),)]),
    | )
    | Explanation:
    |     These lines were always and only run by failing examples:
    |         /var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4-python3_12/install/usr/lib/python3.12/site-packages/shapefile.py:4537
    +------------------------------------
_________________________________________________ test_shapefile_reader_writer_roundtrip __________________________________________________
  + Exception Group Traceback (most recent call last):
  |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4/tests/hypothesis_tests.py", line 853, in test_shapefile_reader_writer_roundtrip
  |     @pytest.mark.hypothesis_dbf
  |                    ^^^
  |   File "/usr/lib/python3.12/site-packages/hypothesis/core.py", line 2281, in wrapped_test
  |     raise the_error_hypothesis_found
  | ExceptionGroup: Hypothesis found 2 distinct failures. (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4/tests/hypothesis_tests.py", line 864, in test_shapefile_reader_writer_roundtrip
    |     written_fields, written_records = _write_fields_and_records_to_strict(w, fields, records)
    |                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4/tests/hypothesis_tests.py", line 806, in _write_fields_and_records_to_strict
    |     w.record(*rec_list)
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4-python3_12/install/usr/lib/python3.12/site-packages/shapefile.py", line 4987, in record
    |     self.dbf_writer.record(*recordList, **recordDict)
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4-python3_12/install/usr/lib/python3.12/site-packages/shapefile.py", line 4381, in record
    |     self._record(record)
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4-python3_12/install/usr/lib/python3.12/site-packages/shapefile.py", line 4441, in _record
    |     raise ShapefileException(
    | shapefile.ShapefileException: Could not read as date: 9990101. Date values must be either a datetime.date object, a list, a YYYYMMDD string, or a missing value.
    | Falsifying example: test_shapefile_reader_writer_roundtrip(
    |     codes_codecs_fields_shapes_and_records=(28,
    |      'shift_jis_2004',
    |      [{'name': '0', 'field_type': 'C', 'size': 1, 'decimal': 0},
    |       {'name': '0', 'field_type': 'C', 'size': 1, 'decimal': 0},
    |       {'name': '0', 'field_type': 'C', 'size': 1, 'decimal': 0},
    |       {'name': '0', 'field_type': 'C', 'size': 1, 'decimal': 0},
    |       {'name': '0', 'field_type': 'D', 'size': 8, 'decimal': 0}],
    |      [NullShape #-1],
    |      [('', '', '', '', '9990101')]),
    | )
    | Explanation:
    |     These lines were always and only run by failing examples:
    |         /var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4-python3_12/install/usr/lib/python3.12/site-packages/shapefile.py:4441
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4/tests/hypothesis_tests.py", line 864, in test_shapefile_reader_writer_roundtrip
    |     written_fields, written_records = _write_fields_and_records_to_strict(w, fields, records)
    |                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4/tests/hypothesis_tests.py", line 806, in _write_fields_and_records_to_strict
    |     w.record(*rec_list)
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4-python3_12/install/usr/lib/python3.12/site-packages/shapefile.py", line 4987, in record
    |     self.dbf_writer.record(*recordList, **recordDict)
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4-python3_12/install/usr/lib/python3.12/site-packages/shapefile.py", line 4381, in record
    |     self._record(record)
    |   File "/var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4-python3_12/install/usr/lib/python3.12/site-packages/shapefile.py", line 4537, in _record
    |     raise ShapefileException(
    | shapefile.ShapefileException: Shapefile Writer unable to pack incorrect sized value=datetime.date(999, 1, 1) (encoded as 7B) into field '0' (8B).
    | Falsifying example: test_shapefile_reader_writer_roundtrip(
    |     codes_codecs_fields_shapes_and_records=(28,
    |      'shift_jis_2004',
    |      [{'name': '0', 'field_type': 'C', 'size': 1, 'decimal': 0},
    |       {'name': '0', 'field_type': 'C', 'size': 1, 'decimal': 0},
    |       {'name': '0', 'field_type': 'C', 'size': 1, 'decimal': 0},
    |       {'name': '0', 'field_type': 'D', 'size': 8, 'decimal': 0}],
    |      [NullShape #-1],
    |      [('', '', '', datetime.date(999, 1, 1))]),
    | )
    | Explanation:
    |     These lines were always and only run by failing examples:
    |         /var/tmp/portage/sci-libs/pyshp-3.1.4/work/pyshp-3.1.4-python3_12/install/usr/lib/python3.12/site-packages/shapefile.py:4537
    +------------------------------------

Other notes

No response

[JamesParrott]

Thanks Thomas.

[Edit]. I've reproduced this, and figured this out. It should be a fast fix (fingers crossed).

I naively didn't realise datetime.strftime(%Y%m%d) doesn't always zero pad date strings to 8 chars, when its Python uses certain versions of glibc. That's not very good for an expressly Pure Python library like ours - I apologise. The bug is essentially because of:

Welcome to Ubuntu 24.04.4 LTS (GNU/Linux 6.8.0-101-generic x86_64)

...

root@ubuntu-2gb-hel1-1:~# python3
Python 3.12.3 (main, Jan 22 2026, 20:57:42) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import datetime
>>> datetime.date(999,1,1)
datetime.date(999, 1, 1)
>>> datetime.date(999,1,1).strftime("%Y%m%d")
'9990101'

I.e. not '09990101'. I should've run the hypothesis tests on older Pythons sooner.

[JamesParrott]

Fix (basically datetime.date(999,1,1).strftime("%Y%m%d").zfill(8)) included in 3.1.5, plus lots more tests are now run, including all round-trippable unicode code points for each codec.