Tkinter: C API changes are needed for Tcl 8.7 and 9.0 value types

[chrstphrchvz]

Bug report

I recently made changes to Tcl.pm (a Tcl wrapper for Perl) to support Tcl 8.7/9.0 (gisle/tcl.pm#42). While I was doing so, I briefly looked at _tkinter.c for comparison (particularly AsObj() and FromObj()). Although I have not attempted to run Tkinter with Tcl 8.7/9.0, I do notice a few issues in the code which would need to be addressed first.

• Tcl 9.0 currently renames the Tcl 8.5 "booleanString" value type to "boolean", which will break the existing check in FromObj(). Tcl has suggested an idiom for retrieving the Tcl_ObjType * of unregistered types like "booleanString", regardless of Tcl version (see https://core.tcl-lang.org/tcl/info/3bb3bcf2da5b); although I would think that approach entails initializing v->BooleanType from Tkapp_New() rather than lazily initializing tkapp->BooleanType from FromObj().

• TIP 568: As of Tcl 9.0, "bytearray" is unregistered; the suggested idiom can be used instead.

• TIP 484: As of Tcl 8.7, on platforms with 32-bit long, there are no longer separate 32-bit "int" and 64-bit "wideInt" value types; there is only a single 64-bit type. And as of Tcl 9.0, the "int" type is unregistered. It is possible to reliably get the Tcl_ObjType * of "int" from a value created using Tcl_NewIntObj(0) and of "wideInt" (if present) using e.g. Tcl_NewWideIntObj((Tcl_WideInt)(1) << 32). However it would not be appropriate for FromObj() to continue using Tcl_GetLongFromObj() when long is 32-bit, as that would retrieve an overflowed result for values within (LONG_MAX,ULONG_MAX] rather than force FromObj() to try again with Tcl_GetWideIntFromObj() (via fromWideIntObj()); an alternative would be to always use Tcl_GetWideIntFromObj() for both "int" and "wideInt" values.

I am not familiar with contributing to Python, but I may be able to propose fixes for the above issues if no one else already has or intends to. I believe the fixes for the above issues are backward compatible with Tcl 8.5 and 8.6, and so can be implemented and tested without other changes that are likely also needed (such as for Tcl-syntax APIs) before Tkinter is usable with Tcl 8.7/9.0 and Tk 8.7 (which will be compatible with both Tcl 8.7 and 9.0) Tcl and Tk 8.7/9.0.

Linked PRs

• gh-103194: Fix Tkinter’s Tcl value type handling for Tcl 8.7/9.0 #103846
• [3.13] gh-103194: Fix Tkinter’s Tcl value type handling for Tcl 8.7/9.0 (GH-103846) #119830
• [3.12] gh-103194: Fix Tkinter’s Tcl value type handling for Tcl 8.7/9.0 (GH-103846) #119831

[terryjreedy]

Christopher, thank you for posting and offering your assistence. My understanding is that 8.7 and 9.0 have been in alpha since 2020. Do you have any updated information on when there might be a beta, let along a final release? IOW, how urgent are the needed changes?

Our _/tkinter maintainer is currently constrained in his python time. _tkinter patches should be welcome. I just hope he will be available to review any. Our devguide should give you any info you need.

I maintain IDLE. From what I have read, the main feature of interest for me in 8.7 is change to Text and in 9.0 full unicode support. Are these in the alphas? If 9.0 does unicode support that works with tkinter, that will be the version I would want out windows and mac installers to install. I wonder though if people would push for 8.7. Should that be an issue?

tkinter requires 8.6 now, so supporting 8.5 is not an issue, but 8.6 will be on Linux for possibly a decade or more. Tip 568 suggests replacing ...GetByteArray.... with ...GetBytes..., but only in 8.7+. So it would seem that we will need separate code for 8.6 and 8.7/9.0. Correct?

[chrstphrchvz]

My understanding is that 8.7 and 9.0 have been in alpha since 2020. Do you have any updated information on when there might be a beta, let along a final release? IOW, how urgent are the needed changes?

It sounded like there has been an increased push to get 8.7 and 9.0 beta releases out since last year, but I do not know how imminent these really are. Some areas of improvement do not seem completely settled, e.g. Unicode support/error handling, replacing int with size_t/ssize_t in APIs, and maybe other things that do not matter as much to Tkinter. But I believe the changes I brought up here will still apply even if 8.7/9.0 releases are years away, and are likely a prerequisite to any work needed for e.g. TIP 474.

Our _/tkinter maintainer is currently constrained in his python time. _tkinter patches should be welcome. I just hope he will be available to review any. Our devguide should give you any info you need.

Thanks, I will look into creating a pull request for these changes.

I maintain IDLE. From what I have read, the main feature of interest for me in 8.7 is change to Text and in 9.0 full unicode support. Are these in the alphas? If 9.0 does unicode support that works with tkinter, that will be the version I would want out windows and mac installers to install. I wonder though if people would push for 8.7. Should that be an issue?

The last alpha releases were a couple years ago, so even if they contain these features, I would think there are noticeable differences by now in the development branches. I am not familiar enough with how 8.7 and 9.0 differ (on things like Unicode support) in ways that matter to Tkinter to summarize here. My understanding is that 8.7 is meant as a choice and/or more gradual transition for 8.6 users that really do not want C API or other breaking changes from 9.0. But even some Tcl core team members raise the idea of abandoning 8.7 in favor of 9.0: https://sourceforge.net/p/tcl/mailman/tcl-core/thread/0fdf01d929c6$9c027350$d40759f0$@yahoo.com/

tkinter requires 8.6 now, so supporting 8.5 is not an issue, but 8.6 will be on Linux for possibly a decade or more.

Is the 8.6 requirement already documented somewhere? I find documentation/comments which still mention 8.5.12 as the minimum.

Tip 568 suggests replacing ...GetByteArray.... with ...GetBytes..., but only in 8.7+. So it would seem that we will need separate code for 8.6 and 8.7/9.0. Correct?

I recently got confirmation that for the use case of wrappers like Tkinter, which only want to use Tcl_GetByteArrayFromObj() for values that already have "bytearray" type, they can continue using Tcl_GetByteArrayFromObj() for 8.6 compatibility as it will behave no differently from Tcl_GetBytesFromObj(): https://core.tcl-lang.org/tcl/info/3bb3bcf2da5b. The guidance in TIP 568 seems meant for other use cases which want to convert arbitrary Tcl values to "bytearray".

[terryjreedy]

There was an idea to require 8.6 instead of 8.5, but yes, we stopped at 8.5.12 in issue 91152.

even some Tcl core team members raise the idea of abandoning 8.7 in favor of 9.0

I think that this would be a lot easier for Python.

[chrstphrchvz]

I have a draft of the changes: main...chrstphrchvz:cpython:patch-103194. I have verified that they work under Tcl 8.6.13. The boolean value case in FromObj() does not appear to exercised by Tkinter’s existing tests, but here is a script which I believe exercises all relevant codepaths:

import tkinter
i = tkinter.Tcl()

# retrieve bytearray
i.eval("""
set x [binary decode hex 40]
# manually verify value type:
puts [::tcl::unsupported::representation $x]
""")
assert i.getvar('x') == b'@'

# retrieve booleanString
i.eval("""
# start with pure string
set x yes
puts [::tcl::unsupported::representation $x]

# this command causes x to have booleanString type:
expr {!$x}
puts [::tcl::unsupported::representation $x]
""")
assert i.getvar('x') == True

# retrieve int
i.eval('set x [expr 1+1]')
assert i.getvar('x') == 2

# retrieve wideInt (or int if wideInt is not present)
i.eval("""
set x [expr 1<<32]
puts [::tcl::unsupported::representation $x]
""")
assert i.getvar('x') == (1 << 32)

I think the only thing left to settle for this set of changes is whether FromObj() should still handle obsolete registered types in Tcl 8.7--specifically "int" on 32-bit and Tcl 8.4 "boolean"--even though Tcl itself will never create values with obsolete types; see latest comment on https://core.tcl-lang.org/tcl/info/3bb3bcf2da5b.

Also, at least one other person appears interested in having Tkinter work with Tk 8.7: one of the Tk Aqua developers, who has been working on extensive improvements which will only appear in 8.7, attempted to use Tkinter with 8.7 but encountered other issues: https://core.tcl-lang.org/tk/info/6b49149b4e7c.

[chrstphrchvz]

one of the Tk Aqua developers … attempted to use Tkinter with 8.7 but encountered other issues: https://core.tcl-lang.org/tk/info/6b49149b4e7c.

That issue seems to indicate another necessary change:

• TIP 622: Under Tcl 8.7, commands can return values with the unregistered "utf32string" type. (The current "string" type still exists for compatibility, but in Tcl 9.0 it will be dropped, and "utf32string" will be renamed to "string".) FromObj() likely needs to recognize this type, or else users will find they have to pass the results of various commands through str() when there was no need to under Tcl 8.6.

[chrstphrchvz]

A separate report for an attempt to build Tkinter against Tcl 8.7 (https://core.tcl-lang.org/tcl/info/ff255adc4cb5) indicates yet another necessary change:

• TIP 538: Tcl 8.7 can be built to use separately-built libtommath, and no longer recommends building with the bundled libtommath. tclTomMath.h requires tommath.h, which Tcl will not install a copy of even when the bundled libtommath is used; either tommath.h must be obtained separately, or TCL_NO_TOMMATH_H must be defined (although a few changes are currently needed for it to work: https://core.tcl-lang.org/tcl/info/2a5cb49733ff).

Package managers presumably would want Tkinter to build against separately-built Tcl/Tk (and in turn separately-built libtommath once they update to 8.7). But given that nothing else in CPython uses libtommath, how much would the official Python binary distributions prefer that Tcl continue building its bundled libtommath?

Edit: I have created a separate issue for this: #103839. It may make more sense to address that issue before making the other changes on the issue here.

[chrstphrchvz]

• TIP 622: Under Tcl 8.7, commands can return values with the unregistered "utf32string" type.

Tcl has agreed to register "utf32string": https://core.tcl-lang.org/tcl/vdiff/?from=27d3b99343&to=9883f837fc

However it does seem like Tkinter code should still use str() where appropriate, rather than assume that functions relying on FromObj() won’t return a _tkinter.Tcl_Obj instead of a Python string.

[chrstphrchvz]

More C API changes are needed for Tcl_Size. I have opened a separate issue for that (gh-112672), so I am narrowing the subject of this issue slightly.

[serhiy-storchaka]

Thank you for your great investigations @chrstphrchvz. I agree with all of your changes.