bpo-33578: Add getstate/setstate for CJK codec #6984

libcthorne · 2018-05-19T07:06:56Z

This implements getstate and setstate for the cjkcodecs multibyte incremental encoders/decoders, primarily to fix issues with seek/tell.

The encoder getstate/setstate is slightly tricky as the "state" is pending bytes + MultibyteCodec_State but only an integer can be returned. The approach I've taken is to encode this data into a long, similar to how .tell() encodes a "cookie_type" as a long.

https://bugs.python.org/issue33578

libcthorne · 2018-05-19T07:09:11Z

(Added GitHub name to bugs.python.org. CLA is signed already)

pablogsal · 2018-05-30T23:03:50Z

Modules/cjkcodecs/multibytecodec.c

+        if (_PyLong_AsByteArray(statelong, pendingbuffer,
+                                sizeof(pendingbuffer),
+                                PY_LITTLE_ENDIAN, 0) < 0) {
+            Py_DECREF(statelong);


Is it possible to refactor all error paths under one "error" label section and substitute these with gotosas indicated here so there is only one exit point from the function?

Modules/cjkcodecs/multibytecodec.c

+    unsigned long long flag;
+
+    if (self->pendingsize > 0) {
+        buffer = PyBytes_FromString((const char *)self->pending);


Modules/cjkcodecs/multibytecodec.c

+        return NULL;
+    }
+
+    buffersize = PyBytes_Size(buffer);


libcthorne · 2018-06-03T16:17:07Z

Thanks for the review @pablogsal, I've made the changes. Refactoring to use an error label helped catch a missing decref 👍

Modules/cjkcodecs/multibytecodec.c

pablogsal · 2018-06-03T17:16:14Z

Lib/test/test_io.py

+        f.write("\u0300")
+        f.close()
+
+        f = self.open(support.TESTFN, "r", encoding="jisx0213")


This test is leaking a file descriptor:

ResourceWarning: unclosed file <_io.TextIOWrapper name='@test_22392_tmp' mode='r' encoding='jisx0213'>

Ah, missed the last .close, thanks.

Lib/test/test_io.py

+        self.assertEqual(f.readline(), "\u3046\u3048\n")
+        self.assertEqual(f.tell(), p1)
+        f.close()
+


methane · 2018-06-04T09:46:05Z

Lib/test/test_io.py

+        f.close()
+
+    def test_seek_with_encoder_state(self):
+        f = self.open(support.TESTFN, "w", encoding="jisx0213")


Please use "euc_jis_2004" instead of "jisx0213" alias.
"JIS X 0213:2004" defines charset. There are three encoding for it; "ISO-2022-JP-2004", "EUC-JIS-2004", and "Shift-JIS 2004".
Additionally, there is an old version of it; "JIS X 0213:2000".

I don't know why "jisx0213" alias for "euc_jis_2004" was added, but it seems not predictable alias.

Thanks, I've changed it to use euc_jis_2004. Also replaced euc-jp with euc_jp.

Modules/cjkcodecs/multibytecodec.c

libcthorne · 2018-06-08T10:24:38Z

Need to fix the following warnings:

2018-06-07T23:35:38.9073811Z /opt/vsts/work/1/s/Modules/cjkcodecs/multibytecodec.c: In function ‘_multibytecodec_MultibyteIncrementalEncoder_setstate’:
2018-06-07T23:35:38.9095425Z /opt/vsts/work/1/s/Modules/cjkcodecs/multibytecodec.c:947:40: warning: passing argument 2 of ‘_PyLong_AsByteArray’ from incompatible pointer type [-Wincompatible-pointer-types]
2018-06-07T23:35:38.9112219Z      if (_PyLong_AsByteArray(statelong, &state,
2018-06-07T23:35:38.9128553Z                                         ^
2018-06-07T23:35:38.9142991Z In file included from ./Include/Python.h:81:0,
2018-06-07T23:35:38.9157853Z                  from /opt/vsts/work/1/s/Modules/cjkcodecs/multibytecodec.c:8:
2018-06-07T23:35:38.9175164Z ./Include/longobject.h:163:17: note: expected ‘unsigned char *’ but argument is of type ‘MultibyteIncrementalEncoderState * {aka struct <anonymous> *}’
2018-06-07T23:35:38.9191247Z  PyAPI_FUNC(int) _PyLong_AsByteArray(PyLongObject* v,

and Travis has picked up a checksum mismatch:

Error in file "./Modules/cjkcodecs/multibytecodec.c" on line 936:
Checksum mismatch!
Expected: 2ef45090fc274f1e
Computed: ded04a702b17468c

Update: done

methane · 2018-06-11T08:23:55Z

Modules/cjkcodecs/multibytecodec.c

+        return NULL;
+    }
+    if (!PyArg_ParseTuple(state, "OK;setstate(): illegal state argument",
+                          &buffer, &flag))


You can use S for better error message..
https://docs.python.org/3/library/codecs.html#codecs.IncrementalDecoder.getstate

methane · 2018-06-11T08:24:30Z

Modules/cjkcodecs/multibytecodec.c

+    unsigned long long flag;
+
+    if (!PyTuple_Check(state)) {
+        PyErr_SetString(PyExc_TypeError, "state argument must be a tuple");


PyArg_ParseTuple will raise TypeError. So this check may be redundant.

PyArg_ParseTuple is enough but the error message is not very user friendly:

>>> import codecs >>> decoder = codecs.getincrementaldecoder('euc_jp')() >>> decoder.setstate(1) Traceback (most recent call last): File "<stdin>", line 1, in <module> SystemError: new style getargs format but argument is not a tuple

Then, you can use ArgumentClinic again. object(subclass_of='&PyTuple_Type').

Thanks, that works nicely :)

>>> decoder.setstate(0) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: setstate() argument must be tuple, not int

methane · 2018-06-11T08:25:55Z

Modules/cjkcodecs/multibytecodec.c

+            return NULL;
+        }
+        Py_DECREF(ret);
+    } else {


Split } and else { in two lines.

methane · 2018-06-11T08:33:58Z

Modules/cjkcodecs/multibytecodec.c

+            return NULL;
+        }
+        self->pendingsize = buffersize;
+        memcpy(self->pending, bufferstr, self->pendingsize);


You need to check buffersize <= MAXDECPENDING

And test cases for possible wrong inputs are needed too.

methane · 2018-06-11T08:35:21Z

Modules/cjkcodecs/multibytecodec.c

+    unsigned long long flag;
+
+    if (self->pendingsize > 0) {
+        buffer = PyBytes_FromString((const char *)self->pending);


Use PyBytes_FromStringAndSize.

bedevere-bot · 2018-06-11T08:36:48Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

methane · 2018-06-11T08:46:57Z

Modules/cjkcodecs/multibytecodec.c

+
+    flag = self->state.ull;
+
+    return Py_BuildValue("NK", buffer, flag);


There is make_tuple() in this file. It can be used instead.

methane · 2018-06-11T08:48:18Z

Modules/cjkcodecs/multibytecodec.h

+    unsigned char pending[MAXENCPENDING];
+    Py_ssize_t pendingsize;
+    MultibyteCodec_State codecstate;
+} MultibyteIncrementalEncoderState;


This struct is used from only two place in single source file.
Move this to the file.

libcthorne · 2018-07-07T18:45:54Z

Changes to address feedback from @doerwalter have been pushed.

MultibyteCodec_State has been changed to a struct with single field and usage updated appropriately
MultibyteIncrementalEncoderState.pendingsize has been changed to an unsigned char to make its representation independent of endianness and 32/64 bit size
PyLong_FromByteArray and PyLong_ToByteArray calls are passed constant of 1 for little_endian parameter to ensure same value is returned on big/little endian systems
test_multibytecodec has been updated to match endian related changes and also has some readability improvements

methane · 2018-07-09T11:03:47Z

Modules/cjkcodecs/multibytecodec.c

+{
+    const char *pendingbuffer = NULL;
+    Py_ssize_t pendingsize;
+    MultibyteIncrementalEncoderState state;


Struct layout is different by compiler.
To build portable integer, you need to use just bytearray:

unsigned char state[1 + MAXPENDING*4 + 8]; ... state[0] = peindingsize; memcpy(state+1, pendingbuffer, pendingsize); int pos = 1 + pendingsize; memcpy(state+pos, self->state, 8);

Thanks, @methane. state should now be independent of x32/64, endianness, and compiler.

Latest commit changes:

Replace MultibyteIncrementalEncoderState with byte array

Update test_multibytecodec to match new byte array format (difference: pendingsize moved to the first byte + only pendingsize bytes stored in buffer position)

Also added test_getstate_returns_expected_value to test_multibytecodec so that the consistency of the state byte array structure is validated by the test suite.

all resolved

vstinner · 2018-10-19T23:43:25Z

@methane: you approved the change, but you didn't merge it. Do you want another core dev to approve the change, or did you just forget to merge it?

methane · 2018-10-20T01:22:20Z

@doerwalter Would you review this again before merging this?

doerwalter · 2018-10-22T15:25:21Z

Modules/cjkcodecs/_codecs_cn.c

@@ -51,6 +51,8 @@
        ;                                                              \
    }

+#define STATE_OFFSET 0
+
 /*


What's the meaning of this #define?

I felt c[STATE_OFFSET] was more explicit than c[0], but don't mind inlining it if you prefer. Alternatively, STATE_POSITION may be a better name, as I wanted to get across that only byte 0 is relevant in state->c for the CN codec.

Perhaps even CN_STATE_POSITION?

OK, I get it: c[8] is the merged version of the previous union, so it serves two purposes: it's a character array and a (very) small integer. IMHO this (and how it's used) should be documented where MultibyteCodec_State is defined.

If I were to document how I see MultibyteCodec_State now, I would say something along the lines of:

A union that provides 8 bytes of state for multibyte codecs. Codecs are free to use this how they want. Note: if you need to add a new member to this union, ensure that its byte order is independent of CPU endianness so that the return value of `getstate` doesn't differ between little and big endian CPUs.

Should I add this as a comment next to the union?

Sounds good to me. However MultibyteCodec_State no longer is a union, "if you need to add a new member to this union" should be phrased differently.

And STATE_OFFSET would have a comment like: "Codecs that have to store a small integer in MultibyteCodec_State store it at the following offset".

And yes, multibytecodec.h is a little light on documentation, but that's probably another patch.

I've added comments next to STATE_OFFSET (now CN_STATE_OFFSET for clarity) and MultibyteCodec_State. Please let me know if the explanations make sense.

Looks good to me, but I'd like @methane to make the final decision about merging this PR.

bedevere-bot · 2018-11-01T11:03:41Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x SLES 3.x has failed when building commit ac22f6a.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/16/builds/1829) and take a look at the build logs.
Check if the failure is related to this commit (ac22f6a) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/16/builds/1829

Click to see traceback logs

From https://github.com/python/cpython
 * branch            master     -> FETCH_HEAD
Reset branch 'master'

/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Modules/_testcapimodule.c:5062:5: warning: initialization from incompatible pointer type [enabled by default]
     {"bad_get", bad_get, METH_FASTCALL},
     ^
/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Modules/_testcapimodule.c:5062:5: warning: (near initialization for ‘TestMethods[173].ml_meth’) [enabled by default]

test_devpoll skipped -- test works only on Solaris OS family
test_tix skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_sqlite skipped -- No module named '_sqlite3'
test_ttk_guionly skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_startfile skipped -- object <module 'os' from '/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/os.py'> has no attribute 'startfile'
test_kqueue skipped -- test works only on BSD
test_readline skipped -- No module named 'readline'
python: /home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Modules/cjkcodecs/_codecs_iso2022.c:515: iso2022_decode: Assertion `dsg->mark != '\0'' failed.
Fatal Python error: Aborted

Current thread 0x000003ffb85f9710 (most recent call first):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_codecs.py", line 72 in check_state_handling_decode
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_codecs.py", line 2135 in test_decoder_state
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/case.py", line 610 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/case.py", line 658 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/runner.py", line 176 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/support/__init__.py", line 1928 in _run_suite
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/support/__init__.py", line 2022 in run_unittest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 179 in runtest_inner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 124 in runtest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest_mp.py", line 68 in run_tests_worker
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 587 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 571 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 627 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/regrtest.py", line 46 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/regrtest.py", line 50 in <module>
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/runpy.py", line 85 in _run_code
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/runpy.py", line 192 in _run_module_as_main
stty: 'standard input': Inappropriate ioctl for device
test_winconsoleio skipped -- test only relevant on win32
test_winreg skipped -- No module named 'winreg'
test_msilib skipped -- No module named '_msi'
test_ioctl skipped -- Unable to open /dev/tty
test_zipfile64 skipped -- test requires loads of disk-space bytes and a long time to run
test_ossaudiodev skipped -- [Errno 2] No such file or directory: '/dev/dsp'
test_winsound skipped -- No module named 'winsound'
test_tk skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
python: /home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Modules/cjkcodecs/_codecs_iso2022.c:515: iso2022_decode: Assertion `dsg->mark != '\0'' failed.
Fatal Python error: Aborted

Current thread 0x000003ffb4f79710 (most recent call first):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_codecs.py", line 72 in check_state_handling_decode
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_codecs.py", line 2135 in test_decoder_state
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/case.py", line 610 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/case.py", line 658 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/runner.py", line 176 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/support/__init__.py", line 1928 in _run_suite
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/support/__init__.py", line 2022 in run_unittest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 179 in runtest_inner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 134 in runtest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 301 in rerun_failed_tests
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 610 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 571 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 627 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/__main__.py", line 2 in <module>
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/runpy.py", line 85 in _run_code
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/runpy.py", line 192 in _run_module_as_main
make: *** [buildbottest] Aborted (core dumped)

bedevere-bot · 2018-11-01T11:07:23Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL 3.x has failed when building commit ac22f6a.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/21/builds/1850) and take a look at the build logs.
Check if the failure is related to this commit (ac22f6a) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/21/builds/1850

Click to see traceback logs

From https://github.com/python/cpython
 * branch            master     -> FETCH_HEAD
Reset branch 'master'

/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Modules/_testcapimodule.c:5062:5: warning: initialization from incompatible pointer type [enabled by default]
     {"bad_get", bad_get, METH_FASTCALL},
     ^
/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Modules/_testcapimodule.c:5062:5: warning: (near initialization for ‘TestMethods[173].ml_meth’) [enabled by default]
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s

test_ioctl skipped -- Unable to open /dev/tty
python: /home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Modules/cjkcodecs/_codecs_iso2022.c:515: iso2022_decode: Assertion `dsg->mark != '\0'' failed.
Fatal Python error: Aborted

Current thread 0x000003fffd12f700 (most recent call first):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/test_codecs.py", line 72 in check_state_handling_decode
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/test_codecs.py", line 2135 in test_decoder_state
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/case.py", line 610 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/case.py", line 658 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/runner.py", line 176 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/support/__init__.py", line 1928 in _run_suite
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/support/__init__.py", line 2022 in run_unittest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/runtest.py", line 179 in runtest_inner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/runtest.py", line 124 in runtest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/runtest_mp.py", line 68 in run_tests_worker
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/main.py", line 587 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/main.py", line 571 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/main.py", line 627 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/regrtest.py", line 46 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/regrtest.py", line 50 in <module>
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/runpy.py", line 85 in _run_code
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/runpy.py", line 192 in _run_module_as_main
test_winreg skipped -- No module named 'winreg'
test_startfile skipped -- object <module 'os' from '/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/os.py'> has no attribute 'startfile'
test_winsound skipped -- No module named 'winsound'
test_zipfile64 skipped -- test requires loads of disk-space bytes and a long time to run
test_devpoll skipped -- test works only on Solaris OS family
test_winconsoleio skipped -- test only relevant on win32
test_kqueue skipped -- test works only on BSD
test_msilib skipped -- No module named '_msi'
test_ossaudiodev skipped -- [Errno 2] No such file or directory: '/dev/dsp'
test_ttk_guionly skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
stty: standard input: Inappropriate ioctl for device
test_tix skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_tk skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
python: /home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Modules/cjkcodecs/_codecs_iso2022.c:515: iso2022_decode: Assertion `dsg->mark != '\0'' failed.
Fatal Python error: Aborted

Current thread 0x000003fffd3f3700 (most recent call first):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/test_codecs.py", line 72 in check_state_handling_decode
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/test_codecs.py", line 2135 in test_decoder_state
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/case.py", line 610 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/case.py", line 658 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/runner.py", line 176 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/support/__init__.py", line 1928 in _run_suite
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/support/__init__.py", line 2022 in run_unittest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/runtest.py", line 179 in runtest_inner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/runtest.py", line 134 in runtest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/main.py", line 301 in rerun_failed_tests
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/main.py", line 610 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/main.py", line 571 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/main.py", line 627 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/__main__.py", line 2 in <module>
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/runpy.py", line 85 in _run_code
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/runpy.py", line 192 in _run_module_as_main
make: *** [buildbottest] Aborted

bedevere-bot · 2018-11-01T11:12:31Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Debian 3.x has failed when building commit ac22f6a.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/13/builds/1863) and take a look at the build logs.
Check if the failure is related to this commit (ac22f6a) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/13/builds/1863

Click to see traceback logs

From https://github.com/python/cpython
 * branch                  master     -> FETCH_HEAD
Reset branch 'master'

/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Modules/_testcapimodule.c:5062:17: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
     {"bad_get", bad_get, METH_FASTCALL},
                 ^~~~~~~
/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Modules/_testcapimodule.c:5062:17: note: (near initialization for ‘TestMethods[173].ml_meth’)

test_tix skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_winreg skipped -- No module named 'winreg'
test_startfile skipped -- object <module 'os' from '/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/os.py'> has no attribute 'startfile'
test_winsound skipped -- No module named 'winsound'
python: /home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Modules/cjkcodecs/_codecs_iso2022.c:515: iso2022_decode: Assertion `dsg->mark != '\0'' failed.
Fatal Python error: Aborted

Current thread 0x000003ffa11f2700 (most recent call first):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_codecs.py", line 72 in check_state_handling_decode
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_codecs.py", line 2135 in test_decoder_state
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 610 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 658 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/runner.py", line 176 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/support/__init__.py", line 1928 in _run_suite
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/support/__init__.py", line 2022 in run_unittest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 179 in runtest_inner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 124 in runtest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest_mp.py", line 68 in run_tests_worker
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 587 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 571 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 627 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/regrtest.py", line 46 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/regrtest.py", line 50 in <module>
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/runpy.py", line 85 in _run_code
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/runpy.py", line 192 in _run_module_as_main
stty: 'standard input': Inappropriate ioctl for device
test_ttk_guionly skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_zipfile64 skipped -- test requires loads of disk-space bytes and a long time to run
test_winconsoleio skipped -- test only relevant on win32
test_ossaudiodev skipped -- [Errno 2] No such file or directory: '/dev/dsp'
test_ioctl skipped -- Unable to open /dev/tty
test_kqueue skipped -- test works only on BSD
test_devpoll skipped -- test works only on Solaris OS family
test_tk skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_msilib skipped -- No module named '_msi'
python: /home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Modules/cjkcodecs/_codecs_iso2022.c:515: iso2022_decode: Assertion `dsg->mark != '\0'' failed.
Fatal Python error: Aborted

Current thread 0x000003ffb5472700 (most recent call first):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_codecs.py", line 72 in check_state_handling_decode
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_codecs.py", line 2135 in test_decoder_state
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 610 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 658 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/runner.py", line 176 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/support/__init__.py", line 1928 in _run_suite
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/support/__init__.py", line 2022 in run_unittest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 179 in runtest_inner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 134 in runtest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 301 in rerun_failed_tests
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 610 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 571 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 627 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/__main__.py", line 2 in <module>
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/runpy.py", line 85 in _run_code
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/runpy.py", line 192 in _run_module_as_main
make: *** [buildbottest] Aborted

libcthorne · 2018-11-01T12:27:19Z

I'm looking into the buildbot failures to see if they're related.

failing test is ...Lib/test/test_codecs.py", line 2135 in test_decoder_state
only s390x builds seem to be failing (e.g. this one passes https://buildbot.python.org/all/#/builders/141/builds/728)
test passes locally (Ubuntu 16.04 x64)
builders were stable before this commit was merged
failure caused by _codecs_iso2022.c, and this commit changed that file, so it is quite likely related

If I understand correctly, the s390x systems are big endian, so there is probably a problem in the handling of endianness. I will look into emulating a big endian machine locally in a few hours from now to debug this.

Update: I managed to recreate locally and the problem seems to be in _multibytecodec_MultibyteIncrementalDecoder_getstate_impl. The casting to Py_ssize_t isn't correct (on both little and big endian) and isn't consistent with setstate parsing. I'm going to use PyLongObject instead, in a similar way to how encoder getstate/setstate works.

libcthorne · 2018-11-02T00:50:40Z

I've opened a PR (#10290) to address the issues found by the big-endian build bots.

This comment has been minimized.

Sign in to view

the-knights-who-say-ni added the CLA not signed label May 19, 2018

bedevere-bot added the awaiting review label May 19, 2018

pablogsal previously requested changes May 30, 2018

View reviewed changes

bedevere-bot added awaiting core review and removed awaiting review labels May 30, 2018

libcthorne force-pushed the fix-multibyte-state branch from 8b5ee96 to db1a055 Compare June 3, 2018 16:12

the-knights-who-say-ni added CLA signed and removed CLA not signed labels Jun 3, 2018

pablogsal reviewed Jun 3, 2018

View reviewed changes

Modules/cjkcodecs/multibytecodec.c Show resolved Hide resolved

libcthorne force-pushed the fix-multibyte-state branch from db1a055 to 6f9869d Compare June 3, 2018 16:36

pablogsal reviewed Jun 3, 2018

View reviewed changes

libcthorne force-pushed the fix-multibyte-state branch from 6f9869d to 0af0cc7 Compare June 3, 2018 17:30

methane reviewed Jun 4, 2018

View reviewed changes

libcthorne force-pushed the fix-multibyte-state branch from 0af0cc7 to 44727e7 Compare June 4, 2018 20:27

methane reviewed Jun 5, 2018

View reviewed changes

Modules/cjkcodecs/multibytecodec.c Outdated Show resolved Hide resolved

libcthorne force-pushed the fix-multibyte-state branch 2 times, most recently from be4dc14 to 3c18d61 Compare June 7, 2018 23:30

libcthorne commented Jun 7, 2018

View reviewed changes

Modules/cjkcodecs/multibytecodec.c Show resolved Hide resolved

libcthorne force-pushed the fix-multibyte-state branch from 3c18d61 to f60f50c Compare June 8, 2018 22:58

methane requested changes Jun 11, 2018

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting core review labels Jun 11, 2018

methane reviewed Jun 11, 2018

View reviewed changes

libcthorne force-pushed the fix-multibyte-state branch from c97199f to edf0b4e Compare July 7, 2018 18:25

methane reviewed Jul 9, 2018

View reviewed changes

libcthorne force-pushed the fix-multibyte-state branch 5 times, most recently from e43fb6f to b36149f Compare July 9, 2018 23:06

methane approved these changes Jul 11, 2018

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting change review labels Jul 11, 2018

zhangyangyu self-requested a review July 12, 2018 04:11

doerwalter reviewed Oct 22, 2018

View reviewed changes

bpo-33578: Implement multibyte encoder/decoder state methods

4aea0e1

libcthorne force-pushed the fix-multibyte-state branch from b36149f to 4aea0e1 Compare October 23, 2018 12:54

methane changed the title ~~bpo-33578: Implement multibyte encoder/decoder state methods~~ bpo-33578: Add getstate/setstate for multibyte codec Nov 1, 2018

methane changed the title ~~bpo-33578: Add getstate/setstate for multibyte codec~~ bpo-33578: Add getstate/setstate for CJK codec Nov 1, 2018

methane added the 🤖 automerge label Nov 1, 2018

miss-islington merged commit ac22f6a into python:master Nov 1, 2018

bedevere-bot removed the awaiting merge label Nov 1, 2018

libcthorne mentioned this pull request Nov 2, 2018

bpo-33578: Fix getstate/setstate for CJK decoder #10290

Merged

mattip mentioned this pull request Aug 25, 2024

cjk codecs are not properly using setstate/clearstate pypy/pypy#5010

Closed


		flag = self->state.ull;

		return Py_BuildValue("NK", buffer, flag);

bpo-33578: Add getstate/setstate for CJK codec #6984

bpo-33578: Add getstate/setstate for CJK codec #6984

Conversation

libcthorne commented May 19, 2018 • edited by methane Loading

This comment has been minimized.

libcthorne commented May 19, 2018

Choose a reason for hiding this comment

This comment was marked as resolved.

This comment was marked as resolved.

libcthorne commented Jun 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

libcthorne commented Jun 8, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bedevere-bot commented Jun 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

libcthorne commented Jul 7, 2018

Choose a reason for hiding this comment

libcthorne Jul 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vstinner commented Oct 19, 2018

methane commented Oct 20, 2018

doerwalter Oct 22, 2018 • edited Loading

Choose a reason for hiding this comment

libcthorne Oct 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bedevere-bot commented Nov 1, 2018

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented Nov 1, 2018

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented Nov 1, 2018

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

libcthorne commented Nov 1, 2018 • edited Loading

libcthorne commented Nov 2, 2018

libcthorne commented May 19, 2018 •

edited by methane

Loading

libcthorne commented Jun 8, 2018 •

edited

Loading

libcthorne Jul 9, 2018 •

edited

Loading

doerwalter Oct 22, 2018 •

edited

Loading

libcthorne Oct 22, 2018 •

edited

Loading

libcthorne commented Nov 1, 2018 •

edited

Loading