(This page is far from complete. See the <tt>toekin</tt> subdir of the source tree for the sources.)
The "Toekin" (pronounced "token") API is a rewrite of an older C++ mini-project, reimplemented in C (mainly as a getting-back-to-C exercise). It is geared towards tokenizing script-like grammars. While it does not contain any actual parsing/interpretation code, the token model should be flexible enough to support a wide variety of parsers/interpreters. For example, experimentation with the [http://www.hwaci.com/sw/lemon/|Lemon parser generator] has shown it to be fairly straightforward to integrate Toekin with Lemon. (An earlier version of this model was the basis for an experimental scripting engine with a custom parser, which eventually was abandoned for lack of a decent VM.)
The basic model revolves around a chain of tokens (defined by a simple structure type), where a chain is a doubly-linked list of tokens. Once a chain is in place, it effectively as an infinite-lookahead/lookback buffer. The tokeninizer itself builds up these chains and provides an API for navigating, manipulating, and managing them.