c-snippets  Toekin

(This page is far from complete. See the toekin subdir of the source tree for the sources.)

The "Toekin" (pronounced "token") API is a rewrite of an older C++ mini-project, reimplemented in C (mainly as a getting-back-to-C exercise). It is geared towards tokenizing script-like grammars. While it does not contain any actual parsing/interpretation code, the token model should be flexible enough to support a wide variety of parsers/interpreters. For example, experimentation with the Lemon parser generator has shown it to be fairly straightforward to integrate Toekin with Lemon. (An earlier version of this model was the basis for an experimental scripting engine with a custom parser, which eventually was abandoned for lack of a decent VM.)

The basic model revolves around a chain of tokens (defined by a simple structure type), where a chain is a doubly-linked list of tokens. Once a chain is in place, it effectively as an infinite-lookahead/lookback buffer. The tokeninizer itself builds up these chains and provides an API for navigating, manipulating, and managing them.