c-snippets  Artifact Content

Artifact b71ea7749c9a7143212d15b6a5801d6b7e31cb91:

Wiki page [Toekin] by stephan 2008-02-25 09:14:26.
D 2008-02-25T09:14:26
L Toekin
P 80585da2b6bb4656aea3c0a38aa5276e85b1adc8
U stephan
W 1128
(This page is far from complete. See the <tt>toekin</tt> subdir of the source tree for the sources.)

The "Toekin" (pronounced "token") API is a rewrite of an older C++ mini-project, reimplemented in C (mainly as a getting-back-to-C exercise). It is geared towards tokenizing script-like grammars. While it does not contain any actual parsing/interpretation code, the token model should be flexible enough to support a wide variety of parsers/interpreters. For example, experimentation with the [http://www.hwaci.com/sw/lemon/|Lemon parser generator] has shown it to be fairly straightforward to integrate Toekin with Lemon. (An earlier version of this model was the basis for an experimental scripting engine with a custom parser, which eventually was abandoned for lack of a decent VM.)

The basic model revolves around a chain of tokens (defined by a simple structure type), where a chain is a doubly-linked list of tokens. Once a chain is in place, it effectively as an infinite-lookahead/lookback buffer. The tokeninizer itself builds up these chains and provides an API for navigating, manipulating, and managing them.
Z 3a541d359f49fdd5b9bdd19555decfe5