c-snippets  Update of "Toekin"

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview

Artifact ID: b71ea7749c9a7143212d15b6a5801d6b7e31cb91
Page Name:Toekin
Date: 2008-02-25 09:14:26
Original User: stephan
Parent: 80585da2b6bb4656aea3c0a38aa5276e85b1adc8
Content

(This page is far from complete. See the toekin subdir of the source tree for the sources.)

The "Toekin" (pronounced "token") API is a rewrite of an older C++ mini-project, reimplemented in C (mainly as a getting-back-to-C exercise). It is geared towards tokenizing script-like grammars. While it does not contain any actual parsing/interpretation code, the token model should be flexible enough to support a wide variety of parsers/interpreters. For example, experimentation with the Lemon parser generator has shown it to be fairly straightforward to integrate Toekin with Lemon. (An earlier version of this model was the basis for an experimental scripting engine with a custom parser, which eventually was abandoned for lack of a decent VM.)

The basic model revolves around a chain of tokens (defined by a simple structure type), where a chain is a doubly-linked list of tokens. Once a chain is in place, it effectively as an infinite-lookahead/lookback buffer. The tokeninizer itself builds up these chains and provides an API for navigating, manipulating, and managing them.