libwhio

Welcome to the Fossil source code repository for libwhio, the WanderingHorse.net I/O library for C. whio is a C library encapsulating an i/o device and stream API. It originally developed as parts of two other libraries, but was found to be generic enough to fork out into its own library.

Author: Stephan Beal (http://wanderinghorse.net/home/stephan/)

License: Public Domain

Code status: Works for me! whio is a core component of libwhefs, is developed in close conjunction with that library, and seems to work well. It is also the basis for a set of JavaScript I/O classes based on the Google v8 JavaScript engine, and (unsurprisingly) has caused no Grief there, either.

Download: downloads page

What is whio?

The API provides an object oriented interface, called whio_dev, for interacting with random-access data stores. On top of this interface, concrete implementations are provided for FILE and in-memory data stores (via dynamic memory or a client-supplied memory range). All implementations have the same, fairly small, public interface.

It can essentially be used to wrap any random-access data stream, and adding wrappers for custom stream types is easy to do. For example libwhefs, an embedded filesystem library, uses a custom whio_dev implementation to provide access to "virtual" files (inside an embedded filesystem) using the same API as one can use for FILE and memory buffer access.

The API also provides another object oriented interface, whio_stream, for interacting with sequential streams. Implementations exist for FILE handles and whio_dev objects, which allows streams to act on any i/o source/destination supported by whio_dev. This is a more lightweight API than whio_dev, but is suitable when one only needs sequential access. This was originally developed as part of libc11n, a data serialization framework, but was forked out for inclusion into the more generic whio library.

Features:

Complete API docs (they're in the header files).
A simple object oriented interface, modeled after the standard C i/o API, for interacting with sequential- or random-access data stores. Includes an ioctl()-like interface for implementing customizations beyond what the public API provides for.
Provides device implementations for FILE handles (using either FILE objects or file descriptors) and in-memory buffers. That is, read/write from/to memory or a file using the same interface.
The in-memory i/o devices can be configured to use a client-specified memory range or dynamic memory. Dynamic buffers can be configured to stay at a fixed size or expand as needed when write() takes them out of bounds.
"Subdevices" allow any i/o device to be partitioned into several logical i/o devices, each of which can only read/write from/to a specified range within the parent device. These can be embedded to further partition an address range.
So-called "block devices" simplify uses where a device is partitioned into a number of records of a fixed size.
Does no copying or transformation of data between the client and the data store, so the abstraction penalty is quite low.
Very frugal memory usage, needing only minimal dynamic memory. (Not counting in-memory devices, which of course need memory for their storage.)
Uniform cleanup of device objects, regardless of the underlying storage. This makes avoiding leaks very easy.
Allows tying client-side data to a device, along with an optional destructor function to clean up the data when the device is closed.
Devices can, with a little work, be "stacked". That is, one implementation can be used to add features to another (e.g. buffering, device range fencing, or multiplexing).
Compiles quickly - about 1.1 seconds on a 2.6GHz machine.
Can be compounded into two files to simplify redistribution and re-use (one .h and one .c file). Use "make amal" to create the "amalgamation" build.

Misfeatures

Does not attempt to be the end-all/be-all of i/o interfaces. It does what i need it to do.
Does not have specific features to support asynchronous i/o.
Does not directly support thread locking. Multi-threaded access to any given i/o device must be carefully serialized by the client.
Because it is device-generic, it does not directly support device-level locking. flock() can be used by the client in conjunction with FILE-based devices, however.
Um... there are probably more, but none come to mind at the moment.

Requirements

The code is ANSI C, using some C99 features. The file-based i/o handlers, as opposed to the memory-based handlers, require some functions defined in POSIX-1.2001. All (or most) Unix systems will have the few POSIX functions the API requires.

It has been shown to compile using gcc 4.2.x on Linux x86/32, gcc 4.1 on Linux x86/64, tcc on Linux/x86, and gcc 3.4 on Solaris/Sparc, but older gcc versions explicitly require using the -std=c99 flag to enable the C99 features which whio uses.

There are some optional features which use zlib to provide gzip compression to/from whio_stream objects, and these features require zlib (which is installed by default on nearly every system on the planet). See whio_zlib.h for the routines and the macro which needs to be set to enable these functions.

Potential uses

Memory-based devices act nicely as dynamic string buffers.
In-memory compression of arbitrary data (using the optional whio_stream/gzip support).
An embedded filesystem, e.g. whefs uses whio_dev as its back-end and provides a higher-level whio_dev specialization which allows its embedded files to be manipulated using the whio_dev interface.
Many places where abstract access to a random-access data store is needed can benefit from being able to swap out storage back-ends. For any given data store, if a whio_dev implementation exists (or can be written) then the whio_dev and whio_stream APIs allows one to easily swap out storage for any given application. e.g. libc11n uses this to provide object de/serialization over arbitrary i/o channels. (In fact, this code is a generalized form of the i/o API from libc11n.)

News

13 Dec 2009:

Work has started on refactoring libwhefs (which is based on whio), to move the most basic embedded filesystem features into libwhio. The core functionality is working, meaning clients can now embedded "pseudofilesystems" into their apps using an optional component of libwhio.

8 June 2009:

Now compiles with gcc's -pedantic and -fstrict-aliasing flags. Seems to work, too.

9 March 2009:

whio is now used in a "real" project to implement I/O classes for JavaScript bindings: http://code.google.com/p/v8-juice/wiki/PluginWhio.

30 Dec 2008:

Experimentation has shown that we can, with only a few minor workarounds, use SWIG to generate script bindings for whio_dev. Once that works it should be easy to make whefs (which uses whio extensively) scriptable.