TEA(ish)

Authoring & Building Tcl Extensions with Teaish
Login

Authoring & Building Tcl Extensions with Teaish

Intro

This page covers the build-related aspects of Tcl extensions, as they pertain to teaish. It does not cover the C-related details of creating Tcl extensions. For example purposes this document uses the SQLite Tcl extension because it's readily handy and was the initial motivation for this project.

Having at least a basic working knowledge of Autosetup will be helpful for using teaish.

Creating a teaish build requires some combination of the following files:

These are covered in more detail in their own sections below, but see the next section for how to quickly generate new ones.

Example builds:

Creating a New Extension

The easiest way to create a new set of teaish build files is:

$ /path/to/teaish/configure \
  --teaish-create-extension=/output/dir

Which will create the output dir (if needed) and write the following files to it:

If all goes well then it should build as-is with:

$ cd the/output/dir
$ /path/to/teaish/configure \
  [--with-tcl=/path/to/tcl/install/prefix] # if needed
$ make test

And (un)installed with:

$ make install   # will run an additional test
$ make uninstall # in case of later regret!

It can be packaged up for distribution with:

$ make dist

The dist files are not cleaned up by make clean, but make undist will remove them.

Including Teaish in the Extension

A full copy of teaish and Autosetup can be installed directly to an existing extension's directory with:

$ /path/to/configure --teaish-install=/path/to/extension

When using --teaish-create-extension, teaish can be included by adding the --teaish-install flag with no value or with the value auto.

With that in place, the extension can be built using the conventional:

$ ./configure && make

To install a newer version of teaish over an existing copy in an extension, use:

$ /path/to/new/teaish/configure \
    --teaish-install=/path/to/extension \
    --teaish-force

But ACHTUNG: that will overwrite all files in the target directory which teaish believes it owns, including: configure, auto.def, pkgIndex.tcl.in, _teaish.tester.tcl.in1, Makefile.in.

Invoking Extension Builds

There are three ways to plug an extension into this build system:

  1. Place a copy of the the extension's teaish files in the main teaish directory and run ./configure.

  2. From a directory containing the extension's teaish files run /path/to/teaish/configure.

  3. From a completely unrelated directory run
    /path/to/teaish/configure --teaish-extension-dir=/path/to/extension ...

(Pass --help to the configure to see the list of options.)

After that, just type make test and (if all goes well)...

(Un)Installing Extensions

Simply:

$ make install

That will build the extension, if needed, but it won't implicitly run make test.

Immediately after the installation, the makefile will run a "package require" test on the extension to ensure that the newly-installed extension can be loaded that way. It will complain loudly if that part fails.

Tip: the first-ever failure of such a test was caused by the Tcl_PkgProvides() call in the extension's C file using the wrong case for the teaish -name.pkg. If your test fails without much useful information, check that first.

To uninstall:

$ make uninstall

That does nothing more than delete the directory into which the extension is installed. (Presumably only that extension is installed in it.)

Extension Definition Files

This section covers the various files an extension may include to influence aspects of its build or execution.

teaish.tcl

This script conceptually needs to do several things, though all of them have usable (though perhaps not useful) defaults.

This script is loaded by teaish very early on in the bootstrapping process so that the extension may extend the set of --flags available to the configure script. That enables the extension to influence, for example, the --help text.

When the script is initially loaded, it must not invoke any teaish-defined APIs, as the configuration API is not yet fully bootstrapped when the extension script is initially read. Any exceptions to this rule in the teaish API are clearly noted in their API docs.

However, the script is expected to run tell teaish a little bit about the extension by calling teaish-pkinfo-set with a set of flags, like:

teaish-pkginfo-set -name foo -version 0.0.1

Or its equivalent:

teaish-pkginfo-set {-name foo -version 0.0.1}

(The latter does not require backslash-escaping for multi-line invocations.)

It has a number of flags, all of which have defaults.

teaish-pkginfo-set may be called multiple times to set some options in later calls, but it must be called (if at all) by the time teaish-options (see below) returns. Calling it from teaish-configure (see below) may confuse the configuration process.

Next, it may define the following functions to influence the downstream configure process...

To reiterate:

Aside from where specifically noted, most teaish Tcl APIs (named teaish-*) are not valid to call until teaish-configure is called, at which point teaish has completed bootstrapping. teaish-options implementations must avoid the temptation to call arbitrary teaish-* APIs, deferring most until teaish-configure.

The only teaish-* APIs which is legal to call from the global scope, when the script is initially imported, before teaish-options is called, are:

  • teaish-get with a subset of its flags
  • teaish-pkginfo-set (then...)
  • teaish-pkginfo-get
  • teaish-get with its full set of flags

Autosetup's facilities may be used, insofar as they don't interefere with teaish's bootstrapping, but all feature-checking "really should" be deferred until teaish-configure is called.

See the hello example extension for examples.

teaish.config

For very simple cases, in particular script-only extensions, teaish.tcl may be replaced by a file named teaish.config. Its only contents must be arguments for teaish-pkginfo-set. For example:

{
  -name whutil
  -version 0.1
  -tm.tcl whutil.tcl
}

The contents of this file are read verbatim and treated as a Tcl list which gets passed to teaish-pkginfo-set. The contents are not eval'd.

If both teaish.tcl and teaish.config exist, which one is chosen is not specified. Use one or the other, but not both.

teaish.make.in

This optional makefile snippet gets included by the main Makefile and is intended to communicate info about how to build the extension.

This file is optional because the same information can be provided programmatically via teaish-configure (from teaish.tcl), but a makefile may be more convenient for some cases. It's also possible to programmatically generate arbitrary makefile code and inject that into the generated makefile instead of, or in addition to, teaish.make.in using teash-make-add and friends.

See #filtering for info on how makefiles get filtered.

At its simplest, this file looks something like:

# List of C source and/or object files:
tx.src = $(tx.dir)/src/tclsqlite3.c

# List of files to include with "make dist". Must be relative to
# the extension's dir.
tx.dist.files += src/tclsqlite3.c

# List of LDFLAGS:
tx.LDFLAGS  = @LDFLAGS_MATH@ @LDFLAGS_ICU@ @LDFLAGS_RT@ @LDFLAGS_ZLIB@

# List of CFLAGS/CPPFLAGS:
tx.CFLAGS   = @CFLAGS_ICU@ @OPT_FEATURE_FLAGS@

The main vars are:

("tx" is short for "teaish extension.")

The complete list of available variables, and their documentation, are in the generated Makefile, which gets filtered from /file/teaish/Makefile.in.

Note that the top-level makefile does not honor CFLAGS, CPPFLAGS and LDFLAGS passed directly to make. Instead, they must be passed to configure, and they'll get baked into the resulting makefile under a different name. (This behavior was adopted after seeing it used to good effect in other trees. It wasn't my idea, but it's a useful one for configure-scripted projects.)

Hooking into Various Build Phases

teaish.make.in may hook into the clean, distclean, test, install, uninstall, and dist targets by using make dependencies, as demonstrated below. Each of those targets is made up of layers of targets following this pattern:

Put differently, the dependencies look like:

In the simplest case, they can be hooked into like this:

clean: clean-extension
clean-extension:
    rm -f my-stuff.foo

The pattern is the same for each, but there's a caveat when hooking into the test and install targets: one has to be careful to account for parallel builds.

If running targets in parallel (using make -j#) is safe for one's build then the pattern shown above is sufficient. If, however, their order needs to be tightly controlled, follow this pattern instead:

test-post: test-extension
test-extension: test-core
    ...do stuff here...

That will ensure that the main tests (test-core) run before test-extension, even when building in multi-process mode using make's -j# flag.

teaish.pkginit.tcl

This optional script is intended to hold code which should be run immediately after the extension library is loaded.

A different file can be used for this purpose by passing one of -pkgInit.tcl or -pkgInit.tcl.in to the pkginit process.

If teaish.pkginit.tcl.in is found in the extension directory, it is used to generate the teaish.pkginit.tcl, filtering it the same way makefiles are. If not, a static teaish.pkginit.tcl is used if it's available.

In the context of the source calls, a scope-local var named dir will be set which holds the name of the directory from which the file is loaded. This is derived from conventional pkgIndex.tcl usage.

pkgIndex.tcl.in

By default a pkgIndex.tcl will be generated for an extension based on teaish's primary pkgIndex template. A package may override that in one of two ways:

That combination is required to ensure that teaish will not overwrite your pkgIndex.tcl. The separate step is required because teaish cannot otherwise distinguish between a hand-written pkgIndex.tcl and one generated in a prior build.

teaish.test.tcl

The configure process, by default, generates a file called teaish.tester.tcl in the build directory, generated from teaish's _teaish.tester.tcl.in1 (if it exists). It acts as a wrapper for running arbitrary tests on the extension. To disable this, add -pragmas {no-tester} to the package init. When including a whole copy of teaish into a custom extension's tree, _teaish.tester.tcl.in can be deleted to disable this support.

The extension may optionally include either a teaish.test.tcl.in (subject to filtering) or a _teaish.test.tcl1 (not filtered). In the former case configure will generate the latter from the former.

The make test target invokes _teaish.tester.tcl with three arguments:

  1. The absolute path of the module's DLL file.

  2. The module's "load prefix", for use as the 2nd argument to Tcl's load function.

  3. The absolute path of teaish/tester.tcl, which provides a small library of routines which are helpful for running tests.

_teaish.tester.tcl will do the following before running the tests:

Before source'ing the pkginit and test files, it sets a call-local dir var to the name of the directory from which teaish.pkginit.tcl resp. teaish.test.tcl were loaded. (This convention is adopted from Tcl's package command.)

Test script may run arbitrary Tcl code for testing. If the script propagates an error, make test will fail. teaish/tester.tcl (see above) provides a set of helpful routines for running basic tests.

Here are some example tests from the SQLite module:

test-expect 1.0-open {
  sqlite3 db :memory:
} {}

test-expect 1.1-select {
  db eval {select 'hi, world',1,2,3}
} {{hi, world} 1 2 3}

test-expect 99.0-db-close {db close} {}

Tests which need to read or write files need to take care to do so from/to the proper directory so that they will work properly when invoked via out-of-tree builds:

Distributing Extensions

Teaish provides basic support for generating a "dist" zip file and tarball for extensions by simply running make dist. By default the distribution includes only the extension, but see the next section for how to ensure that teaish itself is bundled along with the extension.

These dist rules can be disabled by passing disable-dist in the teaish-pkginfo-set -pragmas list in which case the dist recipe is left undefined so that the client makefile may define it. When doing an out-of-tree build, the dist rules are unconditionally disabled because handling of the paths for archival purposes becomes troublesome to do from Makefile code.

By default, teaish will automatically include its "own" files in the distribution bundle if they are present.

In addition, it will include any files or directories which are either:

Caveat(s):

If both zip and tar binaries2 are found at configure-time, running make dist will create:

(Noting that it assumes a tar which supports czf, which some older tar implementations do not.)

Each archive has a top-level directory named $NAME-$VERSION.

The clean and distclean targets do not delete those, but the undist target will.

To build a freshly-unpackaged extension, do:

# If teaish is included in the bundle (see below):
$ ./configure
# Otherwise:
$ /path/to/teaish/configure ...
$ make

Alternately, make dist can be told to include a full copy of teaish along with it...

Including Teaish in the Dist

Whether or not teaish itself is included in the make dist result is determined automatically:

The pkginfo pragmas full-dist and no-full-dist can be used to tell teaish to include, or not, itself in the dist bundle.

When full-dist is on then make dist will include all of teaish in the distribution, such that running ./configure from that directory is all that's needed to prepare it for building.

Template File Filtering

Several of the files mentioned in this doc are filtered in a manner conventional to GNU Autotools. Specifically, the follow the rules for Autosetup's make-template function, but the conventions are similar. In this project, all such files have a suffix of .in (by convention, not hard-coded limitation).

Input files may have any number of @PLACEHOLDERS@ in them, which will be filtered out during the configure process, each one mapped to an Autosetup define, and the results will be written to some target file (typically the same as the input's name minus any ".in" extension). All such placeholders must be resolved, using values provided by Autosetup, teaish, or set in teaish.tcl with Autosetup's define function. If any unresolved placeholders remain after the filtering step, configure will fail3.

To generate the whole list of placeholders, pass --teaish-dump-defines to any configure invocation. That will generate a file named config.defines.txt containing all of the exported flags in some legible form (but don't rely on its format).

Filering of such files happens very late in the configure process, as it must happen after all defines are set up.

Script-only Extensions

Script-only extensions are particularly simple to set up, and there are two approaches. All that's needed is a teaish.tcl and the accompanying script.

First, using a pkgIndex.tcl-style:

teaish-pkginfo-set {
  -name tmplish
  -version 1.0
  -pkgInit.tcl tmplish.tcl
  -pragmas {no-dll}
}

Or a tcl::tm-style:

teaish-pkginfo-set {
  -name tmplish
  -version 1.0
  -tm.tcl tmplish.tcl
}

Alternatively, use a teaish.config file instead.

Optionally, add a teaish.test.tcl script to test it with.

Examples of script-only extensions can be found in teaish/example/whutil and teaish/example/tmplish.

API Docs

The Tcl-level API docs are scattered around the following places:

Running ./configure --reference will format the many docs for reading. Search that for teaish- to find the Teaish APIs. By Autosetup conventions, functions are documented adding comments with @the-function-name, so searching for @teaish- in the Teaish core code will reveal the public APIs.

  1. ^ a b c The underscore in the name _teaish.tester.tcl.in is so that that rarely-edited file will stop interfering with tab completion. The underscore prefixes in other generated _teaish.* files serve the same purpose.
  2. ^ The zip-based build actually uses tar to do some of the heavy lifting, so building the a zip bundle counterintuitively requires tar.
  3. ^ Exception: commented-out lines in files named *[Mm]ake* are ignored if the line starts with # (ignoring leading whitespace). Aside from that exception, the content is semantically opaque to the validation, so it has no notion of what a "comment" is.