bitfield layout wrong in ctypes

[thesamprice]

BPO	39858
Nosy	@thesamprice
Files	ctypeSizeTest.py: Example test code

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-03-05.05:22:35.966>
labels = ['3.8', 'ctypes', 'type-bug', '3.7', '3.9']
title = 'bitfield layout wrong in ctypes'
updated_at = <Date 2020-03-05.15:57:41.683>
user = 'https://github.com/thesamprice'

bugs.python.org fields:

activity = <Date 2020-03-05.15:57:41.683>
actor = 'thesamprice'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['ctypes']
creation = <Date 2020-03-05.05:22:35.966>
creator = 'thesamprice'
dependencies = []
files = ['48954']
hgrepos = []
issue_num = 39858
keywords = []
message_count = 2.0
messages = ['363417', '363438']
nosy_count = 1.0
nosy_names = ['thesamprice']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39858'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']

[thesamprice]

if 8 1 byte fields are included in a ctype field, it allows an extra byte to be included in the packing when there is no room left for the next field.

If I put the bitfields in a child structure then I get expected results.

In [35]: run ctypeSizeTest.py
Size is 4 Expected 3
0 0x10000 a0
0 0x10001 a1
0 0x10002 a2
0 0x10003 a3
0 0x10004 a4
0 0x10005 a5
0 0x10006 a6
0 0x10007 a7
0 0x40008 b0 <- Expected to be at offset 1, not 0.
2 0xc0000 b1 <- Expected to be at offset 1, not 2
Size is 3 Expected 3
0 0x1 a
1 0x40000 b0
1 0xc0004 b1

[thesamprice]

Does not happen on windows.

Error is in cfield.c

#ifndef MS_WIN32
    } else if (bitsize /* this is a bitfield request */
        && *pfield_size /* we have a bitfield open */
        && dict->size * 8 >= *pfield_size
        && (*pbitofs + bitsize) <= dict->size * 8) {
        /* expand bit field */
        fieldtype = EXPAND_BITFIELD;
#endif

Specifically dict->size * 8 >= *pfield_size
if *bitofs == *pfield_size then the current field is filled, and expanding the bitfield should not be done.

Consider adding this *pfield_size != *bitofs
#ifndef MS_WIN32
    } else if (bitsize /* this is a bitfield request */
        && *pfield_size /* we have a bitfield open */
        && *pfield_size != *pbitofs /* Current field has been filled, start new one */
        && dict->size * 8 >= *pfield_size
        && (*pbitofs + bitsize) <= dict->size * 8) {
        /* expand bit field */
        fieldtype = EXPAND_BITFIELD;
#endif