FLEXI-STREAMS - Flexible bivalent streams for Common Lisp

Abstract
FLEXI-STREAMS implements "virtual" bivalent streams that can be layered atop real binary or bivalent streams and that can be used to read and write character data in various single- or multi-octet encodings which can be changed on the fly. It also supplies in-memory binary streams which are similar to string streams.
The library needs a Common Lisp implementation that supports Gray streams and relies on David Lichteblau's trivial-gray-streams to offer portability between different Lisps.
The code comes with a BSD-style license so you can basically do with it whatever you want.
Download shortcut: https://github.com/edicl/flexi-streams/releases/latest

Example usage
Download and installation
Support
The FLEXI-STREAMS dictionary
File positions
Acknowledgements

Example usage

The examples were created with LispWorks 4.4.6 pro on Windows. The following two functions create the same file:

(defun foo (pathspec)
  "With standard LispWorks streams."
  (with-open-file (out pathspec
                       :direction :output
                       :if-exists :supersede
                       :external-format '(:utf-8 :eol-style :crlf))
    (write-line "ÄÖÜ1" out))
  (with-open-file (out pathspec
                       :direction :output
                       :if-exists :append
                       :external-format '(:latin-1 :eol-style :lf))
    (write-line "ÄÖÜ2" out))
  (with-open-file (out pathspec
                       :direction :output
                       :if-exists :append
                       :element-type 'octet)
    (write-byte #xeb out)
    (write-sequence #(#xa3 #xa4 #xa5) out))
  (with-open-file (out pathspec
                       :direction :output
                       :if-exists :append
                       :external-format '(:unicode :little-endian nil :eol-style :crlf))
    (write-line "ÄÖÜ3" out)))

(defun bar (pathspec)
  "With a flexi stream."
  (with-open-file (out pathspec
                       :direction :output
                       :if-exists :supersede
                       :external-format '(:latin-1 :eol-style :lf))
    (setq out (make-flexi-stream out :external-format :utf-8))
    (write-line "ÄÖÜ1" out)
    (setf (flexi-stream-external-format out) '(:latin-1 :eol-style :lf))
    (write-line "ÄÖÜ2" out) 
    (write-byte #xeb out)
    (write-sequence #(#xa3 #xa4 #xa5) out)
    (setf (flexi-stream-external-format out) :ucs-2be)
    (write-line "ÄÖÜ3" out)))

And applying this function

(defun baz (pathspec)
  (let (result)
    (with-open-file (in pathspec :element-type 'octet)
      (setq in (make-flexi-stream in :external-format :utf-8))
      (push (read-line in) result)
      (push (read-byte in) result)
      (setf (flexi-stream-external-format in) '(:latin-1 :eol-style :lf))
      (push (read-line in) result) 
      (setf (flexi-stream-external-format in) :greek)
      (push (read-char in) result)
      (setf (flexi-stream-external-format in) :latin0)
      (let ((string (make-string 3 :element-type 'character)))
        (read-sequence string in)
        (push string result))
      (let ((octets (make-array 2 :element-type 'octet)))
        (read-sequence octets in)
        (push octets result))
      (setf (flexi-stream-external-format in) :ucs-2be)
      (push (read-line in) result))
    (nreverse result)))

to the file created above will yield the list

("ÄÖÜ1" 196 "ÖÜ2" #\λ "£€¥" #(0 196) "ÖÜ3")

For more examples see the source code of CL-RFC2047, Drakma, Chunga, or CL-WBXML.

Before you try to install FLEXI-STREAMS, first check that in your Lisp each character's character code is equal to its Unicode code point and that (CHAR-CODE #\Newline) and (CHAR-CODE #\Linefeed) have the same value (10). (This is the case for all relevant CL implementations which were in use when this library was written. It is not mandated by the ANSI standard, though.)

FLEXI-STREAMS together with this documentation can be downloaded from https://github.com/edicl/flexi-streams/releases/latest.

Before you install FLEXI-STREAMS you first need to install the trivial-gray-streams library unless you already have it.

FLEXI-STREAMS comes with a system definition for ASDF so you can install the library with

(asdf:oos 'asdf:load-op :flexi-streams)

if you've unpacked it in a place where ASDF can find it. Installation via asdf-install should also be possible, and there's a port to Gentoo Lisp thanks to Matthew Kennedy.

You can run a test suite which tests some (but not all) aspects of the library with

(asdf:oos 'asdf:test-op :flexi-streams)

This might take a while...

The current development version of FLEXI-STREAMS can be found at https://github.com/edicl/flexi-streams.

Support

The development version of flexi-streams can be found on github. Please use the github issue tracking system to submit bug reports. Patches are welcome, please use GitHub pull requests.

The FLEXI-STREAMS dictionary

External formats

EXTERNAL-FORMAT objects are used to denote the external formats of flexi streams. These objects are created using the MAKE-EXTERNAL-FORMAT function, and there are various readers to query their attributes. Once such an object is created it can't be changed.

An external format consists of a basic encoding (like ISO 8859-1 or UTF-8), a definition how line endings are denoted - by a carriage return character (ASCII 13), by a line feed character (ASCII 10), or by both of these characters in a row -, and optionally (for encodings that use units larger than 8 bits) information about the endianess of the encoding.

The following encodings are currently supported by FLEXI-STREAMS:

UTF-8 (denoted by the keyword :UTF-8),
UTF-16 (denoted by the keyword :UTF-16),
UTF-32 (denoted by the keyword :UTF-32),
all ISO 8859 character sets (denoted by keywords like :ISO-8859-15),
KOI8-R (denoted by the keyword :KOI8-R),
MAC-ROMAN (denoted by the keyword :MAC-ROMAN),
a couple of Windows code pages (denoted by the keyword :CODE-PAGE and an obligatory :ID argument), and
US-ASCII.

A couple of alternative names are allowed that are listed below:

:UTF-8 :UTF8

:UTF-16 :UTF16

:UCS-2

:UCS2

:UNICODE

:UTF-32 :UTF32

:UCS-4

:UCS4

:ISO-8859-1 :LATIN-1

:LATIN1

:ISO-8859-2 :LATIN-2

:LATIN2

:ISO-8859-3 :LATIN-3

:LATIN3

:ISO-8859-4 :LATIN-4

:LATIN4

:ISO-8859-5 :CYRILLIC

:ISO-8859-6 :ARABIC

:ISO-8859-7 :GREEK

:ISO-8859-8 :HEBREW

:ISO-8859-9 :LATIN-5

:LATIN5

:ISO-8859-10 :LATIN-6

:LATIN6

:ISO-8859-11 :THAI

:ISO-8859-13 :LATIN-7

:LATIN7

:ISO-8859-14 :LATIN-8

:LATIN8

:ISO-8859-15 :LATIN-9

:LATIN9

:LATIN-0

:LATIN0

:ISO-8859-16 :LATIN-10

:LATIN10

:CODE-PAGE :CODEPAGE

WIN32:CODE-PAGE (only on LWW)

:KOI8-R :KOI8R

:MAC-ROMAN :MAC

:MACINTOSH

:MACOS-ROMAN

:US-ASCII :ASCII

(Note that we treat UCS-2 exactly like UTF-16 although there are subtle differences. Also note that even though we support encodings like UTF-32 some Lisps only supports characters contained within the Basic Multilingual Plane (like LispWorks) or even less (like CMUCL), so if other characters are read from a flexi stream, READ-CHAR will try to be helpful and return the corresponding Unicode code point - an integer - instead. This might lead to an error if you're using functions like READ-LINE, though.)

Whenever a FLEXI-STREAMS function accepts an external format as one of its arguments, you can provide either an EXTERNAL-FORMAT object or a shortcut which can be a list or a symbol. The list shortcuts have a syntax similar to the one used by LispWorks - the cars are the names of and encoding and the cdrs of these lists correspond to the keyword arguments to MAKE-EXTERNAL-FORMAT, so for example

(:latin-1 :eol-style :crlf)

is equivalent to

(make-external-format :latin-1 :eol-style :crlf)

The symbol shortcuts are equivalent to calling MAKE-EXTERNAL-FORMAT without keyword arguments, i.e.

:thai

behaves like

(make-external-format :thai)

Finally, the following expansions are available:

:UCS-2LE (:UCS-2 :LITTLE-ENDIAN T)

:UCS-2BE (:UCS-2 :LITTLE-ENDIAN NIL)

:UCS-4LE (:UCS-4 :LITTLE-ENDIAN T)

:UCS-4BE (:UCS-4 :LITTLE-ENDIAN NIL)

:UTF-16LE (:UTF-16 :LITTLE-ENDIAN T)

:UTF-16BE (:UTF-16 :LITTLE-ENDIAN NIL)

:UTF-32LE (:UTF-32 :LITTLE-ENDIAN T)

:UTF-32BE (:UTF-32 :LITTLE-ENDIAN NIL)

:IBM437 (:CODE-PAGE :ID 437)

:IBM850 (:CODE-PAGE :ID 850)

:IBM852 (:CODE-PAGE :ID 852)

:IBM855 (:CODE-PAGE :ID 855)

:IBM857 (:CODE-PAGE :ID 857)

:IBM860 (:CODE-PAGE :ID 860)

:IBM861 (:CODE-PAGE :ID 861)

:IBM862 (:CODE-PAGE :ID 862)

:IBM863 (:CODE-PAGE :ID 863)

:IBM864 (:CODE-PAGE :ID 864)

:IBM865 (:CODE-PAGE :ID 865)

:IBM866 (:CODE-PAGE :ID 866)

:IBM869 (:CODE-PAGE :ID 869)

:WINDOWS-1250 (:CODE-PAGE :ID 1250)

:WINDOWS-1251 (:CODE-PAGE :ID 1251)

:WINDOWS-1252 (:CODE-PAGE :ID 1252)

:WINDOWS-1253 (:CODE-PAGE :ID 1253)

:WINDOWS-1254 (:CODE-PAGE :ID 1254)

:WINDOWS-1255 (:CODE-PAGE :ID 1255)

:WINDOWS-1256 (:CODE-PAGE :ID 1256)

:WINDOWS-1257 (:CODE-PAGE :ID 1257)

:WINDOWS-1258 (:CODE-PAGE :ID 1258)

Note that if you provide a shortcut, it will be converted to an EXTERNAL-FORMAT object first. So, if you're concerned about efficiency, create these objects once and re-use them.

[Function]
make-external-format name &key eol-style little-endian id => external-format

Creates and returns an EXTERNAL-FORMAT object. name is a symbol, eol-style is one of the keywords :CR, :LF, or :CRLF, and little-endian is a generalized boolean.
The default value for eol-style is the value of *DEFAULT-EOL-STYLE*. For Windows code pages, the default style is :CRLF. For :MAC-ROMAN, the default is :CR.
The default value for little-endian is the value of *DEFAULT-LITTLE-ENDIAN* - this value is ignored unless name denotes one of UTF-16 or UTF-32.
id must be an integer denoting a Windows code page known by FLEXI-STREAMS if name is :CODE-PAGE or WIN32:CODE-PAGE, otherwise the value is ignored. See the section about external formats for more info.
Examples (run on Windows):
CL-USER 1 > (make-external-format :latin-1)
#<FLEXI-STREAMS::EXTERNAL-FORMAT (:ISO-8859-1 :EOL-STYLE :CRLF) 2067DA84>

CL-USER 2 > (make-external-format :latin-1 :eol-style :lf)
#<FLEXI-STREAMS::EXTERNAL-FORMAT (:ISO-8859-1 :EOL-STYLE :LF) 2068B4D4>

CL-USER 3 > (make-external-format :ibm437)
#<FLEXI-STREAMS::EXTERNAL-FORMAT (:CODE-PAGE :ID 437 :EOL-STYLE :CRLF) 2069B33C>

CL-USER 4 > (make-external-format :ucs-2)
#<FLEXI-STREAMS::EXTERNAL-FORMAT (:UTF-16 :EOL-STYLE :CRLF :LITTLE-ENDIAN T) 206B4F4C>

CL-USER 5 > (make-external-format :ucs-2be)
#<FLEXI-STREAMS::EXTERNAL-FORMAT (:UTF-16 :EOL-STYLE :CRLF :LITTLE-ENDIAN NIL) 2067DBE4>

CL-USER 6 > (make-external-format :ucs-2be :eol-style :cr)
#<FLEXI-STREAMS::EXTERNAL-FORMAT (:UTF-16 :EOL-STYLE :CR :LITTLE-ENDIAN NIL) 206B54AC>

[Readers]
external-format-name external-format => name
external-format-eol-style external-format => eol-style
external-format-little-endian external-format => little-endian
external-format-id external-format => id

These methods can be used to query an EXTERNAL-FORMAT object for its attributes.

[Functions]
external-format-equal external-format-1 external-format-2 => generalized-boolean

Checks whether the two external formats external-format-1 and external-format-2 are equivalent with respect to their effects on flexi streams.

Examples (run on Windows):

CL-USER 1 > (make-external-format :ucs-4le)
#<FLEXI-STREAMS::EXTERNAL-FORMAT (:UTF-32 :EOL-STYLE :CRLF :LITTLE-ENDIAN T) 2067FB74>

CL-USER 2 > (external-format-equal * (make-external-format :utf32 :little-endian t))
T

CL-USER 3 > (make-external-format :code-page :id 437)
#<FLEXI-STREAMS::EXTERNAL-FORMAT (:CODE-PAGE :ID 437 :EOL-STYLE :CRLF) 2069428C>

CL-USER 4 > (external-format-equal * (make-external-format :ibm437))
T

[Special variable]
*default-eol-style*

The default value for the eol-style keyword argument of MAKE-EXTERNAL-FORMAT. Its initial value is :CRLF on Windows and :LF on other operating systems.

[Special variable]
*default-little-endian*

The default value for the little-endian keyword argument of MAKE-EXTERNAL-FORMAT. Its initial value corresponds to the endianess of the platform FLEXI-STREAMS is used on as revealed by the :LITTLE-ENDIAN feature.

[Condition]
external-format-condition

All conditions related to external formats are of this type. There's a slot for the external format which can be accessed with EXTERNAL-FORMAT-CONDITION-EXTERNAL-FORMAT.

[Reader]
external-format-condition-external-format condition => external-format

If condition is of type EXTERNAL-FORMAT-CONDITION, this function will return the associated external format. Note that there are situation which happen during the creation of external formats where this method returns NIL.

[Condition]
external-format-error

All errors related to external formats are of this type. This is a subtype of EXTERNAL-FORMAT-CONDITION.

[Condition]
external-format-encoding-error

All errors related to encoding problems with external formats are of this type. (This includes situation where an end of file is encountered in the middle of a multi-octet character.) When this condition is signalled during reading, USE-VALUE restart is provided. See also *SUBSTITUTION-CHAR* and the example for it. EXTERNAL-FORMAT-ENCODING-ERROR is a subtype of EXTERNAL-FORMAT-ERROR.

[Special variable]
*substitution-char*

If this value is not NIL, it should be a character which is used (as if by a USE-VALUE restart) whenever during reading an error of type EXTERNAL-FORMAT-ENCODING-ERROR would have been signalled otherwise.

CL-USER 1 > (defun foo ()
              ;; not a valid UTF-8 sequence
              (with-input-from-sequence (in '(#xe4 #xf6 #xfc))
                (setq in (make-flexi-stream in :external-format :utf8))
                (read-line in)))
FOO

CL-USER 2 > (foo)

Error: Unexpected value #xF6 in UTF-8 sequence.
  1 (continue) Specify a character to be used instead.
  2 (abort) Return to level 0.
  3 Return to top loop level 0.

Type :b for backtrace, :c <option number> to proceed,  or :? for other options

CL-USER 3 : 1 > :c
Type a character: x

Error: End of file while in UTF-8 sequence.
  1 (continue) Specify a character to be used instead.
  2 (abort) Return to level 0.
  3 Return to top loop level 0.

Type :b for backtrace, :c <option number> to proceed,  or :? for other options

CL-USER 4 : 1 > :c
Type a character: y
"xy"
T

CL-USER 5 > (handler-bind ((external-format-encoding-error (lambda (condition)
                                                          (use-value #\-))))
              (foo))
"--"
T

CL-USER 6 > (let ((*substitution-char* #\?))
              (foo))
"??"
T

[Restart]
accept-overlong-sequence

This is a restart which is established whenever a UTF-8 "overlong" sequence is encountered. If you invoke this restart, the corresponding code point will be accepted although it was encoded in an illegal way.

Flexi streams

Flexi streams are the core of the FLEXI-STREAMS library. You create them using the function MAKE-FLEXI-STREAM which takes an open binary stream (called the underlying stream) as its only required argument. A binary stream in this context means that if it's an input stream, you can read from it with READ-BYTE (or, as a workaround for LispWorks, you can at least apply READ-SEQUENCE to it where the sequence is an array of element type OCTET), and similarly for WRITE-BYTE (WRITE-SEQUENCE for LispWorks) and output streams. (Note that this specifically holds for bivalent streams like socket streams.)

A flexi stream behaves like an ordinary Lisp stream. It is an input stream if the underlying binary stream is an input stream, and it is an output stream when the underlying binary stream is an output stream. You can write characters as well as octets to an output flexi stream and similarly you can read characters and octets from an input flexi stream.

A flexi stream always has an external format associated with it which is deployed whenever you read characters from the stream or write characters to it. You can change the external format while you use the stream.

Once you're using a flexi stream you should not read from or write to the underlying stream directly anymore.

If you close a flexi stream, the underlying stream will also be closed. However, it also suffices to close the underlying stream directly should you not want to use the flexi stream anymore. So, the following usage (where IN is implicitly closed at the end) is OK:

(with-open-file (in "/foo/bar/baz.txt")
  (let ((flexi (make-flexi-stream in :external-format :hebrew)))
    (read-line flexi)))

Output flexi streams will try to keep track of the column they're in but you can also set the column directly. This value will be incremented by one for each character written to the stream and it will be set to 0 if you send a #\Newline character. The column will be set to NIL if an OCTET is sent to the stream. Once the column is NIL it'll stay like that unless it is explicitly set to another value.

Input flexi streams keep track of their position within the stream. This value is incremented by one for each OCTET read from the stream, and it is incremented by the number of octets actually read for each character read from the stream. So, if the encoding is UTF-8, reading the character #\ä (a-umlaut) will advance the position by two. If the encoding is UTF-32 and the end-of-line style is :CRLF, reading a #\Newline will advance the position by eight.

You can also set the bound of an input flexi stream. Initially it is NIL, but when it's an integer and the stream's position has gone beyond this bound, the stream will behave as if no more input is available.

Caveat: You can only unread a character from a flexi stream if you haven't changed the external format after you read it.

Caveat: The underlying stream should either be a binary stream (i.e. have an element type that is a subtype of integer) or it should explicitly use an external format with :LF as its end-of-line style. Otherwise it might perform unwanted conversion of line endings on its own. (LispWorks does this even if you write binary data to the stream using WRITE-SEQUENCE.)

[Standard class]
flexi-stream

Every flexi stream returned by MAKE-FLEXI-STREAM is of this type which is a subtype of STREAM.

[Standard class]
flexi-input-stream

A flexi stream is of this type if its underlying stream is an input stream. This is a subtype of FLEXI-STREAM.

[Standard class]
flexi-output-stream

A flexi stream is of this type if its underlying stream is an output stream. This is a subtype of FLEXI-STREAM.

[Standard class]
flexi-io-stream

A flexi stream is of this type if it is both a FLEXI-INPUT-STREAM as well as a FLEXI-OUTPUT-STREAM.

[Function]
make-flexi-stream stream &key external-format element-type column position bound => flexi-stream

Creates and returns a flexi stream, i.e. an object of type FLEXI-STREAM. stream is the underlying Lisp stream. external-format is the initial external format to be used by the stream, the default is the value of evaluating (MAKE-EXTERNAL-FORMAT :LATIN1). element-type is the initial element type of the flexi stream the default of which is LW:SIMPLE-CHAR for LispWorks and CHARACTER otherwise. column is the initial column of the stream and should only be provided for output streams, the default is 0. position is the initial octet position of the stream and must only be provided for input streams, the default is 0. bound should be NIL (the default) or an integer and must only be provided for input streams. If the octet position of the stream has gone beyond this bound, the stream will behave as if no more input is available. See the section about flexi streams for more information.

[Accessors]
flexi-stream-external-format flexi-stream => external-format
(setf (flexi-stream-external-format flexi-stream) external-format)
flexi-stream-element-type flexi-stream => element-type
(setf (flexi-stream-element-type flexi-stream) element-type)
flexi-stream-column flexi-output-stream => column
(setf (flexi-stream-column flexi-output-stream) column)
flexi-stream-position flexi-input-stream => position
(setf (flexi-stream-position flexi-input-stream) position)
flexi-stream-bound flexi-input-stream => bound
(setf (flexi-stream-bound flexi-input-stream) bound)

These methods can be used to get and set the corresponding attributes of a flexi stream.
(SETF FLEXI-STREAM-EXTERNAL-FORMAT) accepts keyword symbols (names of external formats), lists (which should be valid lists of parameters to MAKE-EXTERNAL-FORMAT), or EXTERNAL-FORMAT objects:
CL-USER 1 > (setf (flexi-stream-external-format *my-stream*) :ucs-4le)
#<FLEXI-STREAMS::EXTERNAL-FORMAT (:UTF-32 :EOL-STYLE :CRLF :LITTLE-ENDIAN T) 206920DC>

CL-USER 2 > (setf (flexi-stream-external-format *my-stream*) '(:ucs-2be :eol-style :br))
#<FLEXI-STREAMS::EXTERNAL-FORMAT (:UTF-16 :EOL-STYLE :BR :LITTLE-ENDIAN NIL) 20696934>

CL-USER 3 > (setf (flexi-stream-external-format *my-stream*) (make-external-format :ibm437))
#<FLEXI-STREAMS::EXTERNAL-FORMAT (:CODE-PAGE :ID 437 :EOL-STYLE :CRLF) 2068716C>

[Reader]
flexi-stream-stream flexi-stream => stream

This method returns the underlying stream of a flexi stream.

[Generic function]
unread-byte byte stream => nil

Similar to UNREAD-CHAR in that it "unreads" the last octet from stream which must be a flexi stream. Note that you can only call UNREAD-BYTE after a corresponding READ-BYTE, not after READ-CHAR.

[Generic function]
peek-byte stream &optional peek-type eof-error-p eof-value => byte

PEEK-BYTE is like PEEK-CHAR, i.e. it returns an octet from stream (which must be a flexi stream) without actually removing it. If peek-type is NIL, the next octet is returned, if peek-type is T, the next octet which is not 0 is returned, if peek-type is an octet, the next octet which equals peek-type is returned. eof-error-p and eof-value are interpreted as usual.
Note that the parameters aren't in the same order as with PEEK-CHAR because it doesn't make much sense to make stream an optional argument.

[Type]
octet

Just a shortcut for (UNSIGNED-BYTE 8).

[Condition]
flexi-stream-error

All errors related to flexi streams are of this type. This is a subtype of STREAM-ERROR.

[Condition]
flexi-stream-out-of-sync-error

This can happen if you're trying to write to an IO stream which had prior to that "looked ahead" while reading and now can't "rewind" to the octet where you should be.

[Condition]
flexi-stream-element-type-error

All errors related to problems with the element type of flexi streams are of this type. This is a subtype of FLEXI-STREAM-ERROR and has an additional slot for the element type which can be accessed with FLEXI-STREAM-ELEMENT-TYPE-ERROR-ELEMENT-TYPE.

[Reader]
flexi-stream-element-type-error-element-type condition => element-type

If condition is of type FLEXI-STREAM-ELEMENT-TYPE-ERROR, this function will return the offending element type.

In-memory streams

The library also provides in-memory binary streams which are modeled after string streams and behave very similar only that they deal with octets instead of characters and the underlying data structure is not a string but either a list or a vector. These streams can obviously be used as the underlying streams for flexi streams.

[Standard class]
in-memory-stream

Every in-memory stream returned by MAKE-IN-MEMORY-INPUT-STREAM or MAKE-IN-MEMORY-OUTPUT-STREAM is of this type which is a subtype of STREAM.

[Standard class]
in-memory-input-stream

Every in-memory stream returned by MAKE-IN-MEMORY-INPUT-STREAM is of this type which is a subtype of IN-MEMORY-STREAM.

[Standard class]
in-memory-output-stream

Every in-memory stream returned by MAKE-IN-MEMORY-OUTPUT-STREAM is of this type which is a subtype of IN-MEMORY-STREAM.

[Standard class]
list-stream

Every in-memory input stream is of this type if it reads from a list.

[Standard class]
vector-stream

Every in-memory stream is of this type if it reads from or writes to a vector.

[Generic function]
make-in-memory-input-stream sequence &key start end transformer => in-memory-input-stream

Returns a binary input stream (of type IN-MEMORY-INPUT-STREAM) which will supply, in order, the octets in the subsequence of sequence bounded by start (the default is 0) and end (the default is the length of sequence). sequence must either be a list or a vector of octets. Each octet returned will be transformed in turn by the optional transformer function.

[Function]
make-in-memory-output-stream &key element-type transformer => in-memory-output-stream

Returns a binary output stream (of type IN-MEMORY-OUTPUT-STREAM) which accepts objects of type element-type (a subtype of OCTET) and makes available a sequence (see GET-OUTPUT-STREAM-SEQUENCE) that contains the octets that were actually output. The octets stored will each be transformed by the optional transformer function.

[Generic function]
get-output-stream-sequence stream &key as-list => sequence

Returns a vector containing, in order, all the octets that have been output to the in-memory output stream stream. This operation clears any octets on stream, so the vector contains only those octets which have been output since the last call to GET-OUTPUT-STREAM-SEQUENCE or since the creation of the stream, whichever occurred most recently. If as-list is true the return value is coerced to a list.

[Generic function]
output-stream-sequence-length stream => length

Returns the current length of the underlying vector of the in-memory output stream stream, i.e. this is the length of the sequence that GET-OUTPUT-STREAM-SEQUENCE would return if called at this very moment.

[Macro]
with-input-from-sequence (var sequence &key start end transformer) statement* => result*

Creates an in-memory input stream from the sequence sequence using the parameters start and end (see MAKE-IN-MEMORY-INPUT-STREAM), binds var to this stream and then executes the statement* forms. A function transformer may optionally be specified to transform the returned octets. The stream is automatically closed on exit from WITH-OUTPUT-TO-SEQUENCE, no matter whether the exit is normal or abnormal. The return value of this macro is the return value of the last statement of statement*.

[Macro]
with-output-to-sequence (var &key as-list element-type transformer) statement* => sequence

Creates an in-memory output stream, binds var to this stream and then executes the statement* forms. The stream stores data of type element-type (a subtype of OCTET) which is (optionally) transformed by the function transformer prior to storage. The stream is automatically closed on exit from WITH-OUTPUT-TO-SEQUENCE, no matter whether the exit is normal or abnormal. The return value of this macro is a vector (or a list if as-list is true) containing the octets that were sent to the stream within the body of the macro.

[Condition]
in-memory-stream-error

All errors related to in-memory streams are of this type. This is a subtype of STREAM-ERROR.

[Condition]
in-memory-stream-closed-error

An error of this type is signalled if one tries to read from or write to an in-memory stream which had already been closed. This is a subtype of IN-MEMORY-STREAM-ERROR.

[Condition]
in-memory-stream-position-spec-error

Errors of this type are signalled if an erroneous position spec is used in conjunction with FILE-POSITION. This is a subtype of IN-MEMORY-STREAM-ERROR and has an additional slot for the position spec which can be accessed with IN-MEMORY-STREAM-POSITION-SPEC-ERROR-POSITION-SPEC.

[Reader]
in-memory-stream-position-spec-error-position-spec condition => position-spec

If condition is of type IN-MEMORY-STREAM-POSITION-SPEC-ERROR, this function will return the offending position spec.

Strings

This section collects a few convenience functions for strings conversions.

[Function]
string-to-octets string &key external-format start end => vector

Converts the Lisp string string from start to end to an array of octets corresponding to the external format designated by external-format. The defaults for start and end are 0 and the length of the string. The default for external-format is :LATIN1.
In spite of the name, string can be any sequence of characters, but the function is optimized for strings.

[Function]
octets-to-string sequence &key external-format start end => string

Converts the Lisp sequence sequence of octets from start to end to a string using the external format designated by external-format. The defaults for start and end are 0 and the length of the sequence. The default for external-format is :LATIN1.
This function is optimized for the case of sequence being a vector. Don't use lists if you are in hurry.

[Function]
octet-length string &key external-format start end => length

Returns the length of the subsequence of string from start to end in octets if encoded using the external format designated by external-format. The defaults for start and end are 0 and the length of string. The default for external-format is :LATIN1.
In spite of the name, string can be any sequence of characters, but the function is optimized for strings.

[Function]
char-length sequence &key external-format start end => length

Kind of the inverse of OCTET-LENGTH. Returns the length of the subsequence (of octets) of sequence from start to end in characters if decoded using the external format designated by external-format. The defaults for start and end are 0 and the length of the sequence. The default for external-format is :LATIN1. Note that this function doesn't check for the validity of the data in sequence.
This function is optimized for the case of sequence being a vector. Don't use lists if you are in hurry.

File positions

For flexi streams as well as for in-memory streams, FILE-POSITION will usually return NIL and do nothing when a second argument is supplied. This is correct w.r.t. the ANSI standard, but not very helpful. However, even with Gray streams there is no portable way to implement a better behaviour.

For LispWorks and CLISP, FILE-POSITION for flexi streams will work as if the function had been applied to the underlying stream, and for in-memory streams it will try to do something sensible if the underlying data structure is a vector (i.e. not a list). Patches for other Common Lisp implementations should be sent to the trivial-gray-streams maintainers.

Acknowledgements

Thanks to David Lichteblau for numerous portability patches. Thanks to Igor Plekhov for the KOI8-R code. Thanks to Anton Vodonosov for numerous patches and additions. Thanks to Hans Hübner for his work on making FLEXI-STREAMS faster.

`:UTF-8`	`:UTF8`
`:UTF-16`	`:UTF16`
	`:UCS-2`
	`:UCS2`
	`:UNICODE`
`:UTF-32`	`:UTF32`
	`:UCS-4`
	`:UCS4`
`:ISO-8859-1`	`:LATIN-1`
`:ISO-8859-1`	`:LATIN1`
`:ISO-8859-2`	`:LATIN-2`
`:ISO-8859-2`	`:LATIN2`
`:ISO-8859-3`	`:LATIN-3`
`:ISO-8859-3`	`:LATIN3`
`:ISO-8859-4`	`:LATIN-4`
`:ISO-8859-4`	`:LATIN4`
`:ISO-8859-5`	`:CYRILLIC`
`:ISO-8859-6`	`:ARABIC`
`:ISO-8859-7`	`:GREEK`
`:ISO-8859-8`	`:HEBREW`
`:ISO-8859-9`	`:LATIN-5`
`:ISO-8859-9`	`:LATIN5`
`:ISO-8859-10`	`:LATIN-6`
`:ISO-8859-10`	`:LATIN6`
`:ISO-8859-11`	`:THAI`
`:ISO-8859-13`	`:LATIN-7`
`:ISO-8859-13`	`:LATIN7`
`:ISO-8859-14`	`:LATIN-8`
`:ISO-8859-14`	`:LATIN8`
`:ISO-8859-15`	`:LATIN-9`
	`:LATIN9`
	`:LATIN-0`
	`:LATIN0`
`:ISO-8859-16`	`:LATIN-10`
`:ISO-8859-16`	`:LATIN10`
`:CODE-PAGE`	`:CODEPAGE`
`:CODE-PAGE`	`WIN32:CODE-PAGE (only on LWW)`
`:KOI8-R`	`:KOI8R`
`:MAC-ROMAN`	`:MAC`
	`:MACINTOSH`
	`:MACOS-ROMAN`
`:US-ASCII`	`:ASCII`

`:UCS-2LE`	`(:UCS-2 :LITTLE-ENDIAN T)`
`:UCS-2BE`	`(:UCS-2 :LITTLE-ENDIAN NIL)`
`:UCS-4LE`	`(:UCS-4 :LITTLE-ENDIAN T)`
`:UCS-4BE`	`(:UCS-4 :LITTLE-ENDIAN NIL)`
`:UTF-16LE`	`(:UTF-16 :LITTLE-ENDIAN T)`
`:UTF-16BE`	`(:UTF-16 :LITTLE-ENDIAN NIL)`
`:UTF-32LE`	`(:UTF-32 :LITTLE-ENDIAN T)`
`:UTF-32BE`	`(:UTF-32 :LITTLE-ENDIAN NIL)`
`:IBM437`	`(:CODE-PAGE :ID 437)`
`:IBM850`	`(:CODE-PAGE :ID 850)`
`:IBM852`	`(:CODE-PAGE :ID 852)`
`:IBM855`	`(:CODE-PAGE :ID 855)`
`:IBM857`	`(:CODE-PAGE :ID 857)`
`:IBM860`	`(:CODE-PAGE :ID 860)`
`:IBM861`	`(:CODE-PAGE :ID 861)`
`:IBM862`	`(:CODE-PAGE :ID 862)`
`:IBM863`	`(:CODE-PAGE :ID 863)`
`:IBM864`	`(:CODE-PAGE :ID 864)`
`:IBM865`	`(:CODE-PAGE :ID 865)`
`:IBM866`	`(:CODE-PAGE :ID 866)`
`:IBM869`	`(:CODE-PAGE :ID 869)`
`:WINDOWS-1250`	`(:CODE-PAGE :ID 1250)`
`:WINDOWS-1251`	`(:CODE-PAGE :ID 1251)`
`:WINDOWS-1252`	`(:CODE-PAGE :ID 1252)`
`:WINDOWS-1253`	`(:CODE-PAGE :ID 1253)`
`:WINDOWS-1254`	`(:CODE-PAGE :ID 1254)`
`:WINDOWS-1255`	`(:CODE-PAGE :ID 1255)`
`:WINDOWS-1256`	`(:CODE-PAGE :ID 1256)`
`:WINDOWS-1257`	`(:CODE-PAGE :ID 1257)`
`:WINDOWS-1258`	`(:CODE-PAGE :ID 1258)`