Python-CLU – Common Lightweight Utilities
Install it right now! Download or type: $ pip install -U python-clu
The deal with CLU is, it is a bunch of stuff. Like, a junk drawer… but a junk drawer
full of building sets: Legos, Robotix, Capsela
– and also magic interfaces that let somehow you plug Legos into Capselas and all
that. It’s a la carte: you can use one part of CLU (say the inline testing stuff)
and ignore the rest. Or just the exporter
, or just the predicates
collection. Naturally, these things taste great and also taste great together, too.
CLU started out as a packaged-up and sanely written version of the Python tools I wrote for my REPLs, divided up into a bunch of subordinate packages. It’s grown magnificently since then (if I do say so myself). And if you are wondering if “CLU” is a Tron reference, you are correct. “CLU” stands for “Common Lightweight Utilities” – it started out as an acronym for “Command-Line Utilities” but it has grown useful beyond that scope, indeed.
The subordinate packages are pretty well-divided: clu.config
contains
things to help with configuration, like config-file loading and saving,
and data structures like our famous namespaced keymaps.
clu.exporting
has the @export
decorator, a utility to easily label module
members for export by adding them to a modules’ __all__
and __dir__
members,
plus other stuff like real-name determination. Which, speaking of which, clu.naming
has additional easy functions for dealing with the names of your things and other peoples’,
and clu.repr
has stuff for easily printing, getting, and specifying an objects’
string representation.
You’ll like clu.abstract
if you like building class towers – we have ABCs
for all types of things. If you like dict
s, check out the aforementioned clu.config.keymap
and clu.dicts
modules – you’ll find things to build dictionary types, represent
them, and deal with them later. clu.dispatch
will help you manage
subprocesses easily, clu.version
is better then semver
(I promise), clu.testing
is great for inlining some quick unit tests, and
clu.scripts
and clu.repl
are what you want for whatever your
REPL environment may be… bpython
, ipython
,
ptpython
, and many others I am sure!
So do have a look around. Here’s an abridged breakdown of some things within:
-
config
: Tons of useful things for dealing with your app’s configuration. These are centered around lexically namespaced dictionaries called “KeyMaps”, which basic examples of which can be found inclu.config.keymap
. If your nested data looks like:data = { 'yo' : { 'dogg' : { 'i-heard' : { 'you-like' : "nested data" } } } }
… and you assign it to a keymap, like e.g.
keyed = keymap.Nested(data).flatten()
, then you can usekeyed
like so:assert keyed["yo:dogg:i-heard:you-like"] == "nested data"
… The
Flat
andNested
classes are cheap to construct and lightweight, and generally act like mostdict
instances in most of the expected ways. There is also an environment-variablekeymap
adapter inclu.config.env
, which if your app uses environment variables that look like likeMYAPP_CONFIG_THINGY
, they can be accessed throughenv.FrozenEnviron(appname='myapp')
orenv.Environ(appname='myapp')
like so:appenv = Environ(appname='myapp') assert appenv['config:thingy'] == "some value" appenv['config:doohickey'] = "a different value" # sets MYAPP_CONFIG_DOOHICKEY # in the current env
… and there are plenty of tools for rolling your own
KeyMap
s inclu.config
: theproxy
subpackage has aKeyMap
proxy class,ns
has utility functions for manipulating namespace strings,keymapview
has a class tower forKeyMap
key/value/item views, andabc
has, as you’d expect, all the abstract base classes upon which ourKeyMap
s are built, for your perusal and eventual enjoyment. -
constants
: Definitions of useful constants, as well as polyfills to allow certain dependencies from the Python standard library to work even when some of their moving parts are missing (i.e. Ye Olde Pythone 2.7, various PyPy implementations, etc). -
all
: Herein you will find some verrrrry interesting “meta” functions for CLU. You have theimport_all_modules(…)
function, which when provided your projects’ base path and appname, will return all the package names therein, whether class-based modules defined throughclu.exporting
or plain ol’ modules defined by files ending in “.py”, basically. It’s awesome.The companion function
import_clu_modules(¬)
is a shortcut just for everything within CLU, andinline_tests(…)
yields an iterator over every CLU module that defines inline tests throughclu.testing
. These are very useful within CLU; for your own project, write your own functions modeled after these! You can’t go wrong. -
fs
: Filesystem-related things. Submodules include:-
fs.filesystem
: classes representing filesystem primitives likeDirectory
,TemporaryDirectory
,TemporaryName
; a version oftempfile.NamedTemporaryFile
that works with or without the leading dot in the provided suffix (a peeve of mine); functions for common filesystem operations likewhich(…)
,back_ticks(…)
,rm_rf(…)
(be careful with that one), and so forth.- The
Directory
class is particularly extensive, with methods for Zipfile archiving, searching for importable Python files and others, file-suffix histograms, context-managed working-directory management, walking and reverse-walking, and much more.
- The
-
fs.appdirectories
: Started out as a pretty much wholesale duplication of the popularappdirs
package† – but it’s been kitted out with many CLUish things, e.g.clu.fs.filesystem.Directory
instances. A solid upgrade in my opinion! -
fs.misc
: a bunch of useful miscellany – noteworthy standouts include regex utility functions for constructing filename regex match or exclusion tests. -
fs.pypath
: functions for safe manipulation ofsys.path
–enhance(…)
,remove_paths(…)
,remove_invalid_paths()
, &c.
-
-
importing
: a whole class-based-module system! Basically you can do this:class Derived(clu.importing.base.Module): """ I heard you like docstrings """ yo = 'dogg' @export def yodogg(self): return "I heard you like" @export def nodogg(self): return None export(yo, name='yo') from clu.app import Derived as derived # TAKE NOTE!!! assert isinstance(derived, Module) assert derived.yo == 'dogg' assert derived.yodogg() == "I heard you like" assert derived.nodogg() is None
… see that? just by subclassing
clu.importing.base.Module
you can create something instantly importable. Awesome, wouldn’t you say? I would say. We’re working out the best way to create the baseModule
for your own project (the machinery is there but the semantics are not yet perfect) but do feel free to play around with the stuff inclu.importing.base
, and have a look at the implementation inclu.importing.proxy
of the “parameterized”ProxyModule
(there’s aChainModuleMap
container available in theproxy
submodule, too). -
repl
andscripts
: Tools useful in Python REPL environments. Inclu.repl
, there’s a while module full of ANSI printing and formatting utilities, many of them as classes, acolumnize
function for, uh, printing things in columns†, and amodules
submodule focused on printing things about your project’s modules and submodules.clu.scripts
contains a “repl.py” file, which is meant to be loaded into an interactive interpreter like e.g.:$ bpython -i «PATH_TO_CLU»/clu/scripts/repl.py
… which will set all of CLU up for you to use interactively. N.B. if you have your own script you want to include but stil want to use CLU, you can set the
CLU_USER_SCRIPT
environment variable to the path to your own script, and our “repl.py” script will pick it up.clu.scripts
also containstreeline
anddictroast
, which are CLU’s nascent command-line parsing utilities. They make use of the CLUKeyMap
idiom (q.v. note supra) and are being worked on right this very µsecond! -
typespace
: Contains a submoduletypespace.namespace
definingSimpleNamespace
(á la Python 3’stypes.SimpleNamespace
) and a slightly more usefulNamespace
ancestor class; these are used to furnish atypespace.types
namespace that contains everything in the standard-librarytypes
module, but with all the typenames shortened so they no longer gratuitously end in “Type”. Like e.g.types.ModuleType
is to be found intypespace.types.Module
,types.FunctionType
istypespace.types.Function
, et cetera, ad nauseum, so you can just dofrom clu.typespace import types
to lose the redundant “Type” naming suffix – which I don’t know about you but that annoys me, I mean we all know that these things are types because they are in the fuckingtypes
module and don’t need those overly verbose extra four characters there at the end. ALSO, there are additional useful types, from Python’s standard library and common packages, in thetypes
namespace. YOU’RE WELCOME!! -
enums
: Furnishesalias(…)
– which, spoiler alert, let you create aliases within your enums! Oh yes. Like so:from clu.enums import alias @unique class Numbers(Enum): ONE = 1 TWO = 2 THREE = 3 UNO = alias(ONE) DOS = alias(TWO) TRÉS = alias(THREE) assert Numbers.ONE == Numbers.UNO assert Numbers.TWO == Numbers.DOS assert Numbers.THREE == Numbers.TRÉS
… aaaaand BOOM, there you go! The
alias
members of an enum don’t show up when iterating the enum’s members, so they’re for your convenience. For a fantastic CLUish real-life example, have a look at theclu.repl.ansi
module, and the enums it defines – these use a metaclass to line the enum’s standard members up with some predefined class attributes defined in a third-party module; we usealias(…)
to provide more obvious lexical aliases of these attributes. Yes! -
dicts
: Functions and classes for wranglingdict
s (and actuallyMapping
andMutableMapping
descendants in general). There are ABCs for key/value/item views for your own mappings, a much betterChainMap
, written from scratch and improving oncollections.ChainMap
, plus aChainRepr
based onreprlib.Repr
, and a whole bunch of fast dict-merging functions. -
exporting
: This is kind of the heart of CLU. At the beginning of all internal CLU modules, you’ll find something using theExporter
, like this:from clu.exporting import Exporter exporter = Exporter(path=__file__) export = exporter.decorator()
… which I know, I know, the use of
__file__
there is kind of irritating. But thatexporter
instance is now registered with CLU, and whenever there’s a class or function you want made available outside the module, you can specify it like:@export def public_function(): print("I’m so public") class PrivateAncestor: """ I’m more private, don’t play with me """ @export class PublicThing(PrivateAncestor): """ I’m very public, indeed, and available """
At the end of the module (but before your CLU tests, q.v. note sub.) you put:
__all__, __dir__ = exporter.all_and_dir()
… which makes things clear what is public and what is private to consumers of your code, without messing around with underscore prefixes, which I find coarse, and vulgar. You’re not just labeling things for export, though:
clu.exporting
has tons of machinery for finding out the “true name” of a thing, be it a class (easy), a function (trickier but still easy), an instance (trickier still), and a module variable (totally very tricky indeed).For your own project, at some part in your code, all you have to do is:
from clu.exporting import ExporterBase
, and then subclassExporterBase
with your app’s name and base path. Like so:class Exporter(ExporterBase, basepath="/your/project", appname="MyProject"): pass
… then import your own importer (or call it whatever you want) at the top of your own CLUish modules and instantiate per the example supra. See also
clu.scripts.dictroast
for other tricks yourExporter
instances can perform. -
testing
: a handy inline unit-test writing setup. It works fine withcoverage
andpytest
– all you have to do is, after your line assigning__all__
and__dir__
at the end of the module proper, something like:def test(): from clu.testing.utils import inline @inline.precheck def before_everything_else(): """ This function runs once before testing """ # ... @inline def first_test(): """ This test does one thing """ # ... @inline def second_test(): """ This test does another thing """ # ... @inline.runif(something == something_else) def test_conditionally_run(): """ This test only does something if `something` == `something_else` """ # ... @inline.diagnostic def apres_testing(): """ This function runs once after testing """ # ... return inline.test(100) if __name__ == '__main__': sys.exit(test())
… running the module (with either
python -m yourapp.yourmodule
or, say, Nox) will show you each tests’ output. But see thatreturn inline.test(100)
line? That integer argument tells the CLU testrunner how many times to run each test (we here like to go a hundred times). You’ll get a report – each test function only prints its output on the first run, the other 99 in this case are muted – that shows you the time totals, which will look like this:… which the astute will note is the output from a non-CLU module.
clu.testing.utils
is very easy to integrate, as it doesn’t require any other part of CLU. Look over our noxfile for examples of how to run your inline tests as part of a larger suite.… there are other testing support tools in
clu.testing.utils
, have a look. There’s also aclu.testing.pytest
submodule, with some currently-under-developmentpytest
plugins. -
mathematics
: the future home for math-related stuff. All there is right now is aclamp(…)
function that works withnumpy
dtypes. -
naming
: functions for determining the names of things (even module constants and other random things that generally lack things like__name__
attributes) and for the importing and exporting of things by “qualified names” – for instance, you can usenaming.qualified_import(…)
to import a classCurveSet
from its packageinstakit.processors.curves
by doing:CurveSet = qualified_import('instakit.processors.curves.CurveSet')
… which that may be handier for you than composing and hard-coding an
import
statement. See also thenameof(…)
andmoduleof(…)
functions, and the utility functions (e.g.qualified_name(…)
) and predicates (e.g.isnative(…)
). Most of these functions take literally anything for their argument. -
predicates
: Ifclu.exporting
is the heart,clu.predicates
is all of the eyes, ears, fingers and toes of CLU.- Do you want to know if a class is a metaclass? We have an
ismetaclass(…)
predicate for it, for example. - Do you want to get an attribute from something you have, but not sure which attributes this something
may have (due to versioning or being passed around, or whatever)?
attr(thing, 'one', 'two', default=None)
will give youthing.one
if that exists,thing.two
, ifthing.one
doesn’t, andNone
if neither of them are there. - But… would you maybe like to get a tuple with all of the
one
attributes across multiple things?attr_across('one', *things)
will do exactly that. - Similar to the above,
pyattr(…)
does likeattr(…)
exceptone
will be expanded to__one__
, anditem(…)
searches over mapping keys rather than attribute names.
Sweet, right? These functions, and others in the
predicates
module, are based onaccessor
,aquirer
,searcher
, andcollator
– abstract functions that can themselves be used. In fact most ofclu.predicates
is written in the functional style – that is to say they are built successively out of multiplelambda
statements. If this is not your taste, everything there has been documented thoroughly for your use and you shouldn’t have to program in this style. But… if you do enjoy a spot of FP, the primitives in the module perhaps will interest you more than the more utilitarian contents. - Do you want to know if a class is a metaclass? We have an
-
sanitizer
: functions for cleaning up unicode. Right now there is just a list-basedsanitize(…)
function that tones down anything with high-value code points to the point where it can be safelyascii
-ified. -
typology
: This is likeclu.predicates
but with more predicates, many of which are based on typelists. The module is full of typelists and it uses them extensively in its predicates, viaisinstance(…)
,issubclass(…)
and a custom version of same calledsubclasscheck(…)
which tries very hard to work with what you give it (instead of just crapping out with aFalse
return value). You will want to explore this one! -
version
: This is basically my take on thesemver
package, only with improvements. I originally developed it for internal use in some of my Python projects, but it stands on its own. You can constructclu.version.VersionInfo
instances usingsemver
primitives amongst other things. The (skeletal)git_version
subpackage has a couple of Git-related functions, but what you really want is inread_version
– therein you’ll find aread_version_file(…)
function that uses theast
module to read one of those__version__.py
files I know you have lying around. Yes -
The
pytest
testsuite: CLU has a full set ofpytest
-compatible unit tests. In addition to copious module-specific CLU inline tests, you can run this test suite usingpytest
– or, if you’d like to run all the tests, use the fantastic Nox testrunner (which we use) along with our included noxfile.
… AND MORE!!! There really are a ton of useful things in here and one day I will list them (but not today). Have fun with it!! Download CLU today and explore – if you want a full list of everything CLU exports, install it and list them all in your terminal:
$ pip install -U python-clu
$ python -m clu
… which will give you a giant list that looks like this:
† – the original code for some modules in CLU (e.g.
clu.fs.appdirectories
, clu.repl.columnize
,
clu.version.read_version
) was vendored in from third-party code.
This has all been appropriately licensed and credits are inlined where due. Thanks to all CLU
contributors, big and small!