KMP's CL References
This is the writeup of failed issue STREAM-DEFINITION-BY-USER. Because it did not pass, it has no official standing other than as a historical document.
Several vendors have implemented this proposal anyway, so if you'd like to use this facility, you might check to see if it's available in your implementation of choice in spite of not being part the "official" standard.
The facility described here is commonly referred to as "Gray Streams", after David Gray, who wrote the proposal—please do not write this as "Grey Streams"!
Another facility of note that came later and may be available in some implementations is Franz's "Simple Streams". It is newer and addresses a broader range of issues, but its availability in this or that implementation may be different.
Click here to see my personal notes
on this issue.
--Kent Pitman (10-Mar-2001)
Issue: STREAM-DEFINITION-BY-USER References: CLtL pages 329-332, 378-381, and 384-385. Related issues: STREAM-INFO, CLOSED-STREAM-FUNCTIONS, STREAM-ACCESS, STREAM-CAPABILITIES Category: ADDITION Edit history: Version 1, 22-Mar-89 by David N. Gray Status: For discussion and evaluation; not proposed for inclusion in the standard at this time. Problem description: Common Lisp does not provide a standard way for users to define their own streams for use by the standard I/O functions. This impedes the development of window systems for Common Lisp because, while there are standard Common Lisp I/O functions and there are beginning to be standard window systems, there is no portable way to connect them together to make a portable Common Lisp window system. There are also many applications where users might want to define their own filter streams for doing things like printer device control, report formatting, character code translation, or encryption/decryption. Proposal STREAM-DEFINITION-BY-USER:GENERIC-FUNCTIONS Overview: Define a set of generic functions for performing I/O. These functions will have methods that specialize on the stream argument; they would be used by the existing I/O functions. Users could write additional methods for them in order to support their own stream classes. Define a set of classes to be used as the superclass of a stream class in order to provide some default methods. Classes: The following classes are to be used as super classes of user-defined stream classes. They are not intended to be directly instantiated; they just provide places to hang default methods. FUNDAMENTAL-STREAM [Class] This class is a subclass of STREAM and of STANDARD-OBJECT. STREAMP will return true for an instance of any class that includes this. (It may return true for some other things also.) FUNDAMENTAL-INPUT-STREAM [Class] A subclass of FUNDAMENTAL-STREAM. Its inclusion causes INPUT-STREAM-P to return true. FUNDAMENTAL-OUTPUT-STREAM [Class] A subclass of FUNDAMENTAL-STREAM. Its inclusion causes OUTPUT-STREAM-P to return true. Bi-direction streams may be formed by including both FUNDAMENTAL-OUTPUT-STREAM and FUNDAMENTAL-INPUT-STREAM. FUNDAMENTAL-CHARACTER-STREAM [Class] A subclass of FUNDAMENTAL-STREAM. It provides a method for STREAM-ELEMENT-TYPE which returns CHARACTER. FUNDAMENTAL-BINARY-STREAM [Class] A subclass of FUNDAMENTAL-STREAM. Any instantiable class that includes this needs to define a method for STREAM-ELEMENT-TYPE. FUNDAMENTAL-CHARACTER-INPUT-STREAM [Class] Includes FUNDAMENTAL-INPUT-STREAM and FUNDAMENTAL-CHARACTER-STREAM. It provides default methods for several generic functions used for character input. FUNDAMENTAL-CHARACTER-OUTPUT-STREAM [Class] Includes FUNDAMENTAL-OUTPUT-STREAM and FUNDAMENTAL-CHARACTER-STREAM. It provides default methods for several generic functions used for character output. FUNDAMENTAL-BINARY-INPUT-STREAM [Class] Includes FUNDAMENTAL-INPUT-STREAM and FUNDAMENTAL-BINARY-STREAM. FUNDAMENTAL-BINARY-OUTPUT-STREAM [Class] Includes FUNDAMENTAL-OUTPUT-STREAM and FUNDAMENTAL-BINARY-STREAM. Character input: A character input stream can be created by defining a class that includes FUNDAMENTAL-CHARACTER-INPUT-STREAM and defining methods for the generic functions below. STREAM-READ-CHAR stream [Generic Function] This reads one character from the stream. It returns either a character object, or the symbol :EOF if the stream is at end-of-file. Every subclass of FUNDAMENTAL-CHARACTER-INPUT-STREAM must define a method for this function. Note that for all of these generic functions, the stream argument must be a stream object, not T or NIL. STREAM-UNREAD-CHAR stream character [Generic Function] Un-does the last call to STREAM-READ-CHAR, as in UNREAD-CHAR. Returns NIL. Every subclass of FUNDAMENTAL-CHARACTER-INPUT-STREAM must define a method for this function. STREAM-READ-CHAR-NO-HANG stream [Generic Function] This is used to implement READ-CHAR-NO-HANG. It returns either a character, or NIL if no input is currently available, or :EOF if end-of-file is reached. The default method provided by FUNDAMENTAL-CHARACTER-INPUT-STREAM simply calls STREAM-READ-CHAR; this is sufficient for file streams, but interactive streams should define their own method. STREAM-PEEK-CHAR stream [Generic Function] Used to implement PEEK-CHAR; this corresponds to peek-type of NIL. It returns either a character or :EOF. The default method calls STREAM-READ-CHAR and STREAM-UNREAD-CHAR. STREAM-LISTEN stream [Generic Function] Used by LISTEN. Returns true or false. The default method uses STREAM-READ-CHAR-NO-HANG and STREAM-UNREAD-CHAR. Most streams should define their own method since it will usually be trivial and will always be more efficient than the default method. STREAM-READ-LINE stream [Generic Function] Used by READ-LINE. A string is returned as the first value. The second value is true if the string was terminated by end-of-file instead of the end of a line. The default method uses repeated calls to STREAM-READ-CHAR. STREAM-CLEAR-INPUT stream [Generic Function] Implements CLEAR-INPUT for the stream, returning NIL. The default method does nothing. Character output: A character output stream can be created by defining a class that includes FUNDAMENTAL-CHARACTER-OUTPUT-STREAM and defining methods for the generic functions below. STREAM-WRITE-CHAR stream character [Generic Function] Writes character to the stream and returns the character. Every subclass of FUNDAMENTAL-CHARACTER-OUTPUT-STREAM must have a method defined for this function. STREAM-LINE-COLUMN stream [Generic Function] This function returns the column number where the next character will be written, or NIL if that is not meaningful for this stream. The first column on a line is numbered 0. This function is used in the implementation of PPRINT and the FORMAT ~T directive. For every character output stream class that is defined, a method must be defined for this function, although it is permissible for it to always return NIL. STREAM-START-LINE-P stream [Generic Function] This is a predicate which returns T if the stream is positioned at the beginning of a line, else NIL. It is permissible to always return NIL. This is used in the implementation of FRESH-LINE. Note that while a value of 0 from STREAM-LINE-COLUMN also indicates the beginning of a line, there are cases where STREAM-START-LINE-P can be meaningfully implemented although STREAM-LINE-COLUMN can't be. For example, for a window using variable-width characters, the column number isn't very meaningful, but the beginning of the line does have a clear meaning. The default method for STREAM-START-LINE-P on class FUNDAMENTAL-CHARACTER-OUTPUT-STREAM uses STREAM-LINE-COLUMN, so if that is defined to return NIL, then a method should be provided for either STREAM-START-LINE-P or STREAM-FRESH-LINE. STREAM-WRITE-STRING stream string &optional start end [Generic Function] This is used by WRITE-STRING. It writes the string to the stream, optionally delimited by start and end, which default to 0 and NIL. The string argument is returned. The default method provided by FUNDAMENTAL-CHARACTER-OUTPUT-STREAM uses repeated calls to STREAM-WRITE-CHAR. STREAM-TERPRI stream [Generic Function] Writes an end of line, as for TERPRI. Returns NIL. The default method does (STREAM-WRITE-CHAR stream #\NEWLINE). STREAM-FRESH-LINE stream [Generic Function] Used by FRESH-LINE. The default method uses STREAM-START-LINE-P and STREAM-TERPRI. STREAM-FINISH-OUTPUT stream [Generic Function] Implements FINISH-OUTPUT. The default method does nothing. STREAM-FORCE-OUTPUT stream [Generic Function] Implements FORCE-OUTPUT. The default method does nothing. STREAM-CLEAR-OUTPUT stream [Generic Function] Implements CLEAR-OUTPUT. The default method does nothing. STREAM-ADVANCE-TO-COLUMN stream column [Generic Function] Writes enough blank space so that the next character will be written at the specified column. Returns true if the operation is successful, or NIL if it is not supported for this stream. This is intended for use by by PPRINT and FORMAT ~T. The default method uses STREAM-LINE-COLUMN and repeated calls to STREAM-WRITE-CHAR with a #\SPACE character; it returns NIL if STREAM-LINE-COLUMN returns NIL. Other functions: CLOSE stream &key abort [Generic Function] The existing function CLOSE is redefined to be a generic function, but otherwise behaves the same. The default method provided by class FUNDAMENTAL-STREAM sets a flag for OPEN-STREAM-P. The value returned by CLOSE will be as specified by the issue CLOSED-STREAM-OPERATIONS. OPEN-STREAM-P stream [Generic Function] This function [from proposal STREAM-ACCESS] is made generic. A default method is provided by class FUNDAMENTAL-STREAM which returns true if CLOSE has not been called on the stream. STREAMP object [Generic Function] INPUT-STREAM-P stream [Generic Function] OUTPUT-STREAM-P stream [Generic Function] These three existing predicates may optionally be implemented as generic functions for implementations that want to permit users to define streams that are not STANDARD-OBJECTs. Normally, the default methods provided by classes FUNDAMENTAL-INPUT-STREAM and FUNDAMENTAL-OUTPUT-STREAM are sufficient. Note that, for example, (INPUT-STREAM-P x) is not equivalent to (TYPEP x 'FUNDAMENTAL-INPUT-STREAM) because implementations may have additional ways of defining their own streams even if they don't make that visible by making these predicates generic. STREAM-ELEMENT-TYPE stream [Generic Function] This existing function is made generic, but otherwise behaves the same. Class FUNDAMENTAL-CHARACTER-STREAM provides a default method which returns CHARACTER. PATHNAME and TRUENAME are also permitted to be implemented as generic functions. There is no default method since these are not valid for all streams. Binary streams: Binary streams can be created by defining a class that includes either FUNDAMENTAL-BINARY-INPUT-STREAM or FUNDAMENTAL-BINARY-OUTPUT-STREAM (or both) and defining a method for STREAM-ELEMENT-TYPE and for one or both of the following generic functions. STREAM-READ-BYTE stream [Generic Function] Used by READ-BYTE; returns either an integer, or the symbol :EOF if the stream is at end-of-file. STREAM-WRITE-BYTE stream integer [Generic Function] Implements WRITE-BYTE; writes the integer to the stream and returns the integer as the result. Rationale: The existing I/O functions cannot be made generic because, in nearly every case, the stream argument is optional, and therefore cannot be specialized. Therefore, it is necessary to define a lower-level generic function to be used by the existing function. It also isn't appropriate to specialize on the second argument of PRINT-OBJECT because it is a higher-level function -- even when the first argument is a character or a string, it needs to format it in accordance with *PRINT-ESCAPE*. In order to make the meaning as obvious as possible, the names of the generic functions have been formed by prefixing "STREAM-" to the corresponding non-generic function. Having the generic input functions just return :EOF at end-of-file, with the higher-level functions handling the eof-error-p and eof-value arguments, simplifies the generic function interface and makes it more efficient by not needing to pass through those arguments. Note that the functions that use this convention can only return a character or integer as a stream element, so there is no possibility of ambiguity. Functions STREAM-LINE-COLUMN, STREAM-START-LINE-P, and STREAM-ADVANCE-TO-COLUMN may appear to be a reincarnation of the defeated proposal STREAM-INFO, but the motivation here is different. This interface needs to be defined if user-defined streams are to be able to be used by PPRINT and FORMAT ~T, which could be viewed as a separate question from whether the user can call then on system-defined streams. Current practice: No one currently supports exactly this proposal, but this is very similar to the stream interface used in CLUE. On descendants of the MIT Lisp Machine, streams can be implemented by users as either flavors, with methods to accept the various messages corresponding to the I/O operations, or as functions, which take a message keyword as their first argument. Examples: ;;;; Here is an example of how the default methods could be ;;;; implemented (omitting the most trivial ones): (defmethod STREAM-PEEK-CHAR ((stream fundamental-character-input-stream)) (let ((character (stream-read-char stream))) (unless (eq character :eof) (stream-unread-char stream character)) character)) (defmethod STREAM-LISTEN ((stream fundamental-character-input-stream)) (let ((char (stream-read-char-no-hang stream))) (and (not (null char)) (not (eq char :eof)) (progn (stream-unread-char stream char) t)))) (defmethod STREAM-READ-LINE ((stream fundamental-character-input-stream)) (let ((line (make-array 64 :element-type 'string-char :fill-pointer 0 :adjustable t))) (loop (let ((character (stream-read-char stream))) (if (eq character :eof) (return (values line t)) (if (eql character #\newline) (return (values line nil)) (vector-push-extend character line))))))) (defmethod STREAM-START-LINE-P ((stream fundamental-character-output-stream)) (equal (stream-line-column stream) 0)) (defmethod STREAM-WRITE-STRING ((stream fundamental-character-output-stream) string &optional (start 0) (end (length string))) (do ((i start (1+ i))) ((>= i end) string) (stream-write-char stream (char string i)))) (defmethod STREAM-TERPRI ((stream fundamental-character-output-stream)) (stream-write-char stream #\newline) nil) (defmethod STREAM-FRESH-LINE ((stream fundamental-character-output-stream)) (if (stream-start-line-p stream) nil (progn (stream-terpri stream) t))) (defmethod STREAM-ADVANCE-TO-COLUMN ((stream fundamental-character-output-stream) column) (let ((current (stream-line-column stream))) (unless (null current) (dotimes (i (- current column) t) (stream-write-char stream #\space))))) (defmethod INPUT-STREAM-P ((stream fundamental-input-stream)) t) (defmethod INPUT-STREAM-P ((stream fundamental-output-stream)) ;; allow the two classes to be mixed in either order (typep stream 'fundamental-input-stream)) (defmethod OUTPUT-STREAM-P ((stream fundamental-output-stream)) t) (defmethod OUTPUT-STREAM-P ((stream fundamental-input-stream)) (typep stream 'fundamental-output-stream)) ;;;; Following is an example of how the existing I/O functions could ;;;; be implemented using standard Common Lisp and the generic ;;;; functions specified above. The standard functions being defined ;;;; are in upper case. ;; Internal helper functions (proclaim '(inline decode-read-arg decode-print-arg check-for-eof)) (defun decode-read-arg (arg) (cond ((null arg) *standard-input*) ((eq arg t) *terminal-io*) (t arg))) (defun decode-print-arg (arg) (cond ((null arg) *standard-output*) ((eq arg t) *terminal-io*) (t arg))) (defun check-for-eof (value stream eof-errorp eof-value) (if (eq value :eof) (report-eof stream eof-errorp eof-value) value)) (defun report-eof (stream eof-errorp eof-value) (if eof-errorp (error 'end-of-file :stream stream) eof-value)) ;;; Common Lisp input functions (defun READ-CHAR (&optional input-stream (eof-errorp t) eof-value recursive-p) (declare (ignore recursive-p)) ; a mistake in CLtL? (let ((stream (decode-read-arg input-stream))) (check-for-eof (stream-read-char stream) stream eof-errorp eof-value))) (defun PEEK-CHAR (&optional peek-type input-stream (eof-errorp t) eof-value recursive-p) (declare (ignore recursive-p)) (let ((stream (decode-read-arg input-stream))) (if (null peek-type) (check-for-eof (stream-peek-char stream) stream eof-errorp eof-value) (loop (let ((value (stream-peek-char stream))) (if (eq value :eof) (return (report-eof stream eof-errorp eof-value)) (if (if (eq peek-type t) (not (member value '(#\space #\tab #\newline #\page #\return #\linefeed))) (char= peek-type value)) (return value) (stream-read-char stream)))))))) (defun UNREAD-CHAR (character &optional input-stream) (stream-unread-char (decode-read-arg input-stream) character)) (defun LISTEN (&optional input-stream) (stream-listen (decode-read-arg input-stream))) (defun READ-LINE (&optional input-stream (eof-error-p t) eof-value recursive-p) (declare (ignore recursive-p)) (let ((stream (decode-read-arg input-stream))) (multiple-value-bind (string eofp) (stream-read-line stream) (if eofp (if (= (length string) 0) (report-eof stream eof-error-p eof-value) (values string t)) (values string nil))))) (defun CLEAR-INPUT (&optional input-stream) (stream-clear-input (decode-read-arg input-stream))) (defun READ-CHAR-NO-HANG (&optional input-stream (eof-errorp t) eof-value recursive-p) (declare (ignore recursive-p)) (let ((stream (decode-read-arg input-stream))) (check-for-eof (stream-read-char-no-hang stream) stream eof-errorp eof-value))) ;;; Common Lisp output functions (defun WRITE-CHAR (character &optional output-stream) (stream-write-char (decode-print-arg output-stream) character)) (defun FRESH-LINE (&optional output-stream) (stream-fresh-line (decode-print-arg output-stream))) (defun TERPRI (&optional output-stream) (stream-terpri (decode-print-arg output-stream))) (defun WRITE-STRING (string &optional output-stream &key (start 0) end) (stream-write-string (decode-print-arg output-stream) string start end)) (defun WRITE-LINE (string &optional output-stream &key (start 0) end) (let ((stream (decode-print-arg output-stream))) (stream-write-string stream string start end) (stream-terpri stream) string)) (defun FORCE-OUTPUT (&optional stream) (stream-force-output (decode-print-arg stream))) (defun FINISH-OUTPUT (&optional stream) (stream-finish-output (decode-print-arg stream))) (defun CLEAR-OUTPUT (&optional stream) (stream-clear-output (decode-print-arg stream))) ;;; Binary streams (defun READ-BYTE (binary-input-stream &optional (eof-errorp t) eof-value) (check-for-eof (stream-read-byte binary-input-stream) binary-input-stream eof-errorp eof-value)) (defun WRITE-BYTE (integer binary-output-stream) (stream-write-byte binary-output-stream integer)) ;;; String streams (defclass string-input-stream (fundamental-character-input-stream) ((string :initarg :string :type string) (index :initarg :start :type fixnum) (end :initarg :end :type fixnum) )) (defun MAKE-STRING-INPUT-STREAM (string &optional (start 0) end) (make-instance 'string-input-stream :string string :start start :end (or end (length string)))) (defmethod stream-read-char ((stream string-input-stream)) (with-slots (index end string) stream (if (>= index end) :eof (prog1 (char string index) (incf index))))) (defmethod stream-unread-char ((stream string-input-stream) character) (with-slots (index end string) stream (decf index) (assert (eql (char string index) character)) nil)) (defmethod stream-read-line ((stream string-input-stream)) (with-slots (index end string) stream (let* ((endline (position #\newline string :start index :end end)) (line (subseq string index endline))) (if endline (progn (setq index (1+ endline)) (values line nil)) (progn (setq index end) (values line t)))))) (defclass string-output-stream (fundamental-character-output-stream) ((string :initform nil :initarg :string))) (defun MAKE-STRING-OUTPUT-STREAM () (make-instance 'string-output-stream)) (defun GET-OUTPUT-STREAM-STRING (stream) (with-slots (string) stream (if (null string) "" (prog1 string (setq string nil))))) (defmethod stream-write-char ((stream string-output-stream) character) (with-slots (string) stream (when (null string) (setq string (make-array 64. :element-type 'string-char :fill-pointer 0 :adjustable t))) (vector-push-extend character string) character)) (defmethod stream-line-column ((stream string-output-stream)) (with-slots (string) stream (if (null string) 0 (let ((nx (position #\newline string :from-end t))) (if (null nx) (length string) (- (length string) nx 1)) )))) Cost to Implementors: Given that CLOS is supported, adding the above generic functions and methods is easy, since most of the code is included in the examples above. The hard part would be re-writing existing I/O functionality in terms of methods on these new generic functions. That could be simplified if methods can be defined to forward the operations to the old representation of streams. For a new implementation, the cost could be zero since an approach similar to this would likely be used anyway. Cost to Users: None; this is an upward-compatible addition. Users won't even need to know anything about this unless they actually need this feature. Cost of non-adoption: Development of portable I/O extensions will be discouraged. Performance impact: This shouldn't affect performance of new implementations (assuming an efficient CLOS implementation), but it could slow down I/O if it were clumsily grafted on top of an existing implementation. Benefits: A broader domain of programs that can be written portably. Esthetics: This seems to be a simple, straight-forward approach. Discussion: This proposal incorporates suggestions made by several people in response to an earlier outline. So far, no one has expressed opposition to the concept. There are some differences of opinion about whether certain operations should have default methods or required methods: STREAM-LISTEN, STREAM-READ-CHAR-NO-HANG, STREAM-LINE-COLUMN, and STREAM-START-LINE-P. An experimental prototype of this has been successfully implemented on the Explorer. This proposal does not provide sufficient capability to implement forwarding streams such as for MAKE-SYNONYM-STREAM, MAKE-BROADCAST-STREAM, MAKE-CONCATENATED-STREAM, MAKE-TWO-WAY-STREAM, or MAKE-ECHO-STREAM. The generic function approach does not lend itself as well to that as a message passing model where the intermediary does not need to know what all the possible messages are. A possible way of extending it for that would be to define a class (defclass stream-generic-function (standard-generic-function) ()) to be used as the :generic-function-class option for all of the I/O generic functions. This would then permit doing something like (defmethod no-applicable-method ((gfun stream-generic-function) &rest args) (if (streamp (first args)) (apply #'stream-operation-not-handled (first args) gfun (rest args)) (call-next-method))) where stream-operation-not-handled is a generic function whose default method signals an error, but forwarding streams can define methods that will create a method to handle the unexpected operation. (Perhaps NO-APPLICABLE-METHOD should be changed to take two required arguments since all generic functions need at least one required argument, and that would make it unnecessary to define a new generic function class just to be able to write this one method.) Another thing that is not addressed here is a way to cause an instance of a user-defined stream class to be created from a call to the OPEN function. That should be part of a separate issue for generic functions on pathnames. If that capability were available, then PATHNAME and TRUENAME should be required to be generic functions. An earlier draft defined just two classes, FUNDAMENTAL-INPUT-STREAM and FUNDAMENTAL-OUTPUT-STREAM, that were used for both character and binary streams. It isn't clear whether that simple approach is sufficient or whether the larger set of classes is really needed.
HTML version Copyright 2001 by Kent M. Pitman. All Rights Reserved.