Data Pipes (Iteratees)

Iteratees may be viewed as accumulating part of a data pipe or a fold operation. A minimal data pipe consists of an Enumerator associated with an Iteratee. They are Producers and Consumers respectively. They are also cooporating co-routines.

An Enumerator pushes a sequence of datums, one at a time, into an Iteratee. When either the Enumerator has no further values to send or the Iteratee determines it is done (say it "takes" or consumes exactly 5 values and no more) the Enumerator/Iteratee pipe returns some accumulated value from the Iteratee. Hence the perspective of viewing an Enumerator/Iteratee combination as a "fold".

So for example a Enumerator of a list of Integer could be associated with an Iteratee that sums Integer values resulting in the sum total of the Integer list.

Enumeratees are one or more pipe sections that sit between an Enumerator and an Iteratee. From the perspective of the Enumerator an Enumeratee acts as an Iteratee. From the perspective of the Iteratee the Enumeratee acts as an Enumerator.

Enumerator -> Enumeratee1 -> ... -> EnumerateeN -> Iteratee

As the traditional terminology of Enumerator, Enumeratee and Iteratee is not very user friendly, this implementation opts for less imposing Pump, Hose and Tank metaphor respectively.

Pump -> Hose1 -> ... -> HoseN -> Tank

So for example, a Pump of a list of string feeds string values into a Hose that converts strings to integers and in turn the feeds those integers into an integer summing Tank.

A Pump combinator function takes two Pumps and combines them into a single Pump. Say combining two Pumps of string lists to act as a single Pump of the combined lists. Or combining two or more Pumps of text files to act as if all the files had been cat’d together.

Similarly Hoses could be nested vertically (one within the other) or joined horizontally with one piping data from one to the other.

Tanks can also be nested.

The basic combining step however is to combine a Hose with a Tank to produce a new Tank. That Tank in turn may be repeatably combined with N number of Hoses producing an ever increasing Tank. Finally to complete the data pipe the built up Tank is associated with a Pump and then data pipe is run to completion at which point an accumulated value is drained from the Tank.