KMP's CL References


This is the writeup of failed issue STREAM-DEFINITION-BY-USER. Because it did not pass, it has no official standing other than as a historical document.


Click here to see my personal notes on this issue.
--Kent Pitman (10-Mar-2001)


References:	CLtL pages 329-332, 378-381, and 384-385.


Category:	ADDITION

Edit history:	Version 1, 22-Mar-89 by David N. Gray
Status:		For discussion and evaluation; not proposed for
		inclusion in the standard at this time.

Problem description:

  Common Lisp does not provide a standard way for users to define their
  own streams for use by the standard I/O functions.  This impedes the
  development of window systems for Common Lisp because, while there are
  standard Common Lisp I/O functions and there are beginning to be
  standard window systems, there is no portable way to connect them
  together to make a portable Common Lisp window system.

  There are also many applications where users might want to define
  their own filter streams for doing things like printer device control,
  report formatting, character code translation, or



  Define a set of generic functions for performing I/O.  These functions
  will have methods that specialize on the stream argument; they would
  be used by the existing I/O functions.  Users could write additional
  methods for them in order to support their own stream classes.

  Define a set of classes to be used as the superclass of a stream class
  in order to provide some default methods.


  The following classes are to be used as super classes of user-defined
  stream classes.  They are not intended to be directly instantiated; they
  just provide places to hang default methods.


    This class is a subclass of STREAM and of STANDARD-OBJECT.  STREAMP
    will return true for an instance of any class that includes this.  (It
    may return true for some other things also.)


    A subclass of FUNDAMENTAL-STREAM.  Its inclusion causes INPUT-STREAM-P
    to return true.


    A subclass of FUNDAMENTAL-STREAM.  Its inclusion causes OUTPUT-STREAM-P
    to return true.  Bi-direction streams may be formed by including both


    A subclass of FUNDAMENTAL-STREAM.  It provides a method for

    A subclass of FUNDAMENTAL-STREAM.  Any instantiable class that
    includes this needs to define a method for STREAM-ELEMENT-TYPE.


    It provides default methods for several generic functions used for
    character input.


    It provides default methods for several generic functions used for
    character output.





 Character input:

  A character input stream can be created by defining a class that
  includes FUNDAMENTAL-CHARACTER-INPUT-STREAM and defining methods for the
  generic functions below.

  STREAM-READ-CHAR  stream			[Generic Function]

    This reads one character from the stream.  It returns either a
    character object, or the symbol :EOF if the stream is at end-of-file.
    Every subclass of FUNDAMENTAL-CHARACTER-INPUT-STREAM must define a
    method for this function.

    Note that for all of these generic functions, the stream argument
    must be a stream object, not T or NIL.

  STREAM-UNREAD-CHAR  stream  character		[Generic Function]

    Un-does the last call to STREAM-READ-CHAR, as in UNREAD-CHAR.  Returns
    NIL.  Every subclass of FUNDAMENTAL-CHARACTER-INPUT-STREAM must define
    a method for this function.

  STREAM-READ-CHAR-NO-HANG  stream		[Generic Function]

    This is used to implement READ-CHAR-NO-HANG.  It returns either a
    character, or NIL if no input is currently available, or :EOF if
    end-of-file is reached.  The default method provided by
    is sufficient for file streams, but interactive streams should define
    their own method.
  STREAM-PEEK-CHAR  stream			[Generic Function]

    Used to implement PEEK-CHAR; this corresponds to peek-type of NIL.
    It returns either a character or :EOF.  The default method

  STREAM-LISTEN  stream				[Generic Function]

    Used by LISTEN.  Returns true or false.  The default method uses
    STREAM-READ-CHAR-NO-HANG and STREAM-UNREAD-CHAR.  Most streams should 
    define their own method since it will usually be trivial and will
    always be more efficient than the default method.

  STREAM-READ-LINE  stream			[Generic Function]

    Used by READ-LINE.  A string is returned as the first value.  The
    second value is true if the string was terminated by end-of-file
    instead of the end of a line.  The default method uses repeated
    calls to STREAM-READ-CHAR.

  STREAM-CLEAR-INPUT  stream			[Generic Function]

    Implements CLEAR-INPUT for the stream, returning NIL.  The default
    method does nothing.

 Character output:

  A character output stream can be created by defining a class that
  includes FUNDAMENTAL-CHARACTER-OUTPUT-STREAM and defining methods for the
  generic functions below.

  STREAM-WRITE-CHAR  stream character		[Generic Function]

    Writes character to the stream and returns the character.  Every
    subclass of FUNDAMENTAL-CHARACTER-OUTPUT-STREAM must have a method
    defined for this function.

  STREAM-LINE-COLUMN  stream			[Generic Function]

    This function returns the column number where the next character
    will be written, or NIL if that is not meaningful for this stream.
    The first column on a line is numbered 0.  This function is used in
    the implementation of PPRINT and the FORMAT ~T directive.  For every
    character output stream class that is defined, a method must be
    defined for this function, although it is permissible for it to
    always return NIL.

  STREAM-START-LINE-P  stream			[Generic Function]

    This is a predicate which returns T if the stream is positioned at the
    beginning of a line, else NIL.  It is permissible to always return
    NIL.  This is used in the implementation of FRESH-LINE.  Note that
    while a value of 0 from STREAM-LINE-COLUMN also indicates the
    beginning of a line, there are cases where STREAM-START-LINE-P can be
    meaningfully implemented although STREAM-LINE-COLUMN can't be.  For
    example, for a window using variable-width characters, the column
    number isn't very meaningful, but the beginning of the line does have
    a clear meaning.  The default method for STREAM-START-LINE-P on class
    that is defined to return NIL, then a method should be provided for

  STREAM-WRITE-STRING stream string &optional start end [Generic Function]

    This is used by WRITE-STRING.  It writes the string to the stream,
    optionally delimited by start and end, which default to 0 and NIL.
    The string argument is returned.  The default method provided by

  STREAM-TERPRI  stream				[Generic Function]

    Writes an end of line, as for TERPRI.  Returns NIL.  The default
    method does (STREAM-WRITE-CHAR stream #\NEWLINE).

  STREAM-FRESH-LINE  stream			[Generic Function]

    Used by FRESH-LINE.  The default method uses STREAM-START-LINE-P and

  STREAM-FINISH-OUTPUT  stream			[Generic Function]

    Implements FINISH-OUTPUT.  The default method does nothing.

  STREAM-FORCE-OUTPUT  stream			[Generic Function]

    Implements FORCE-OUTPUT.  The default method does nothing.

  STREAM-CLEAR-OUTPUT  stream			[Generic Function]

    Implements CLEAR-OUTPUT.  The default method does nothing.

  STREAM-ADVANCE-TO-COLUMN  stream column	[Generic Function]

    Writes enough blank space so that the next character will be written
    at the specified column.  Returns true if the operation is
    successful, or NIL if it is not supported for this stream.    
    This is intended for use by by PPRINT and FORMAT ~T.  The default
    method uses STREAM-LINE-COLUMN and repeated calls to
    STREAM-WRITE-CHAR with a #\SPACE character; it returns NIL if

 Other functions:
  CLOSE  stream &key abort			[Generic Function]

    The existing function CLOSE is redefined to be a generic function, but
    otherwise behaves the same.  The default method provided by class
    FUNDAMENTAL-STREAM sets a flag for OPEN-STREAM-P.  The value returned
    by CLOSE will be as specified by the issue CLOSED-STREAM-OPERATIONS.

  OPEN-STREAM-P stream				[Generic Function]

    This function [from proposal STREAM-ACCESS] is made generic.  A
    default method is provided by class FUNDAMENTAL-STREAM which returns
    true if CLOSE has not been called on the stream.

  STREAMP  object				[Generic Function]
  INPUT-STREAM-P  stream			[Generic Function]
  OUTPUT-STREAM-P  stream			[Generic Function]

    These three existing predicates may optionally be implemented as
    generic functions for implementations that want to permit users to
    define streams that are not STANDARD-OBJECTs.  Normally, the default
    methods provided by classes FUNDAMENTAL-INPUT-STREAM and
    FUNDAMENTAL-OUTPUT-STREAM are sufficient.  Note that, for example,
    (INPUT-STREAM-P x) is not equivalent to (TYPEP x
    'FUNDAMENTAL-INPUT-STREAM) because implementations may have
    additional ways of defining their own streams even if they don't
    make that visible by making these predicates generic.

  STREAM-ELEMENT-TYPE  stream			[Generic Function]

    This existing function is made generic, but otherwise behaves the
    same.  Class FUNDAMENTAL-CHARACTER-STREAM provides a default method
    which returns CHARACTER.

  PATHNAME and TRUENAME are also permitted to be implemented as generic
  functions.  There is no default method since these are not valid for
  all streams.

 Binary streams:

    Binary streams can be created by defining a class that includes either
    (or both) and defining a method for STREAM-ELEMENT-TYPE and for one or
    both of the following generic functions.

  STREAM-READ-BYTE  stream			[Generic Function]

    Used by READ-BYTE; returns either an integer, or the symbol :EOF if the
    stream is at end-of-file.

  STREAM-WRITE-BYTE stream integer		[Generic Function]

    Implements WRITE-BYTE; writes the integer to the stream and returns
    the integer as the result.


  The existing I/O functions cannot be made generic because, in nearly
  every case, the stream argument is optional, and therefore cannot be
  specialized.  Therefore, it is necessary to define a lower-level
  generic function to be used by the existing function.  It also isn't
  appropriate to specialize on the second argument of PRINT-OBJECT because
  it is a higher-level function -- even when the first argument is a
  character or a string, it needs to format it in accordance with

  In order to make the meaning as obvious as possible, the names of the
  generic functions have been formed by prefixing "STREAM-" to the
  corresponding non-generic function.

  Having the generic input functions just return :EOF at end-of-file, with
  the higher-level functions handling the eof-error-p and eof-value
  arguments, simplifies the generic function interface and makes it more
  efficient by not needing to pass through those arguments.  Note that the
  functions that use this convention can only return a character or
  integer as a stream element, so there is no possibility of ambiguity.

  STREAM-ADVANCE-TO-COLUMN may appear to be a reincarnation of the
  defeated proposal STREAM-INFO, but the motivation here is different.
  This interface needs to be defined if user-defined streams are to be
  able to be used by PPRINT and FORMAT ~T, which could be viewed as a
  separate question from whether the user can call then on
  system-defined streams.

Current practice:

  No one currently supports exactly this proposal, but this is very
  similar to the stream interface used in CLUE.

  On descendants of the MIT Lisp Machine, streams can be implemented
  by users as either flavors, with methods to accept the various
  messages corresponding to the I/O operations, or as functions, which
  take a message keyword as their first argument.


  ;;;; Here is an example of how the default methods could be
  ;;;; implemented (omitting the most trivial ones):

  (defmethod STREAM-PEEK-CHAR ((stream fundamental-character-input-stream))
    (let ((character (stream-read-char stream)))
      (unless (eq character :eof)
	(stream-unread-char stream character))

  (defmethod STREAM-LISTEN ((stream fundamental-character-input-stream))
    (let ((char (stream-read-char-no-hang stream)))
      (and (not (null char))
	   (not (eq char :eof))
	   (progn (stream-unread-char stream char) t))))

  (defmethod STREAM-READ-LINE ((stream fundamental-character-input-stream))
    (let ((line (make-array 64 :element-type 'string-char 
			    :fill-pointer 0 :adjustable t)))
      (loop (let ((character (stream-read-char stream)))
	      (if (eq character :eof)
		  (return (values line t))
		(if (eql character #\newline)
		    (return (values line nil))
		  (vector-push-extend character line)))))))

  (defmethod STREAM-START-LINE-P ((stream fundamental-character-output-stream))
    (equal (stream-line-column stream) 0))

  (defmethod STREAM-WRITE-STRING ((stream fundamental-character-output-stream)
				  string &optional (start 0) 
				  (end (length string)))
    (do ((i start (1+ i)))
	((>= i end) string)
      (stream-write-char stream (char string i))))

  (defmethod STREAM-TERPRI ((stream fundamental-character-output-stream))
    (stream-write-char stream #\newline)

  (defmethod STREAM-FRESH-LINE ((stream fundamental-character-output-stream))
    (if (stream-start-line-p stream)
      (progn (stream-terpri stream) t)))

  (defmethod STREAM-ADVANCE-TO-COLUMN ((stream fundamental-character-output-stream) 
    (let ((current (stream-line-column stream)))
      (unless (null current)
	(dotimes (i (- current column) t)
	  (stream-write-char stream #\space)))))

  (defmethod INPUT-STREAM-P ((stream fundamental-input-stream)) t)
  (defmethod INPUT-STREAM-P ((stream fundamental-output-stream))
    ;; allow the two classes to be mixed in either order
    (typep stream 'fundamental-input-stream))
  (defmethod OUTPUT-STREAM-P ((stream fundamental-output-stream)) t)
  (defmethod OUTPUT-STREAM-P ((stream fundamental-input-stream))
    (typep stream 'fundamental-output-stream))

  ;;;; Following is an example of how the existing I/O functions could
  ;;;; be implemented using standard Common Lisp and the generic
  ;;;; functions specified above.  The standard functions being defined
  ;;;; are in upper case.

  ;;  Internal helper functions

  (proclaim '(inline decode-read-arg decode-print-arg check-for-eof))
  (defun decode-read-arg (arg)
    (cond ((null arg) *standard-input*)
	  ((eq arg t) *terminal-io*)
	  (t arg)))
  (defun decode-print-arg (arg)
    (cond ((null arg) *standard-output*)
	  ((eq arg t) *terminal-io*)
	  (t arg)))
  (defun check-for-eof (value stream eof-errorp eof-value)
    (if (eq value :eof)
	(report-eof stream eof-errorp eof-value)
  (defun report-eof (stream eof-errorp eof-value)
    (if eof-errorp
	(error 'end-of-file :stream stream)
  ;;;  Common Lisp input functions
  (defun READ-CHAR (&optional input-stream (eof-errorp t) eof-value recursive-p)
    (declare (ignore recursive-p)) ; a mistake in CLtL?
    (let ((stream (decode-read-arg input-stream)))
      (check-for-eof (stream-read-char stream) stream eof-errorp eof-value)))
  (defun PEEK-CHAR (&optional peek-type input-stream (eof-errorp t) 
			eof-value recursive-p)
    (declare (ignore recursive-p))
    (let ((stream (decode-read-arg input-stream)))
      (if (null peek-type)
	  (check-for-eof (stream-peek-char stream) stream eof-errorp eof-value)
	  (let ((value (stream-peek-char stream)))
	    (if (eq value :eof)
		(return (report-eof stream eof-errorp eof-value))
	      (if (if (eq peek-type t)
		      (not (member value '(#\space #\tab #\newline
					   #\page #\return #\linefeed)))
		    (char= peek-type value))
		  (return value)
		(stream-read-char stream))))))))
  (defun UNREAD-CHAR (character &optional input-stream)
    (stream-unread-char (decode-read-arg input-stream) character))
  (defun LISTEN (&optional input-stream)
    (stream-listen (decode-read-arg input-stream)))
  (defun READ-LINE (&optional input-stream (eof-error-p t) 
			eof-value recursive-p)
    (declare (ignore recursive-p))
    (let ((stream (decode-read-arg input-stream)))
      (multiple-value-bind (string eofp)
	  (stream-read-line stream)
	(if eofp
	    (if (= (length string) 0)
		(report-eof stream eof-error-p eof-value)
	      (values string t))
	  (values string nil)))))
  (defun CLEAR-INPUT (&optional input-stream)
    (stream-clear-input (decode-read-arg input-stream)))
  (defun READ-CHAR-NO-HANG (&optional input-stream (eof-errorp t) 
				eof-value recursive-p)
    (declare (ignore recursive-p))
    (let ((stream (decode-read-arg input-stream)))
      (check-for-eof (stream-read-char-no-hang stream)
		     stream eof-errorp eof-value)))
  ;;;  Common Lisp output functions
  (defun WRITE-CHAR (character &optional output-stream)
     (stream-write-char (decode-print-arg output-stream) character))
  (defun FRESH-LINE (&optional output-stream)
    (stream-fresh-line (decode-print-arg output-stream)))
  (defun TERPRI (&optional output-stream)
    (stream-terpri (decode-print-arg output-stream)))
  (defun WRITE-STRING (string &optional output-stream &key (start 0) end)
    (stream-write-string (decode-print-arg output-stream) string start end))
  (defun WRITE-LINE (string &optional output-stream &key (start 0) end)
    (let ((stream (decode-print-arg output-stream)))
      (stream-write-string stream string start end)
      (stream-terpri stream)
  (defun FORCE-OUTPUT (&optional stream)
    (stream-force-output (decode-print-arg stream)))
  (defun FINISH-OUTPUT (&optional stream)
    (stream-finish-output (decode-print-arg stream)))
  (defun CLEAR-OUTPUT (&optional stream)
    (stream-clear-output (decode-print-arg stream)))
  ;;;  Binary streams

  (defun READ-BYTE (binary-input-stream &optional (eof-errorp t) eof-value)
    (check-for-eof (stream-read-byte binary-input-stream) 
		   binary-input-stream eof-errorp eof-value))
  (defun WRITE-BYTE (integer binary-output-stream)
    (stream-write-byte binary-output-stream integer))

  ;;;  String streams
  (defclass string-input-stream (fundamental-character-input-stream)
    ((string :initarg :string :type string)
     (index :initarg :start :type fixnum)
     (end :initarg :end :type fixnum)
  (defun MAKE-STRING-INPUT-STREAM (string &optional (start 0) end)
    (make-instance 'string-input-stream :string string 
		   :start start :end (or end (length string))))
  (defmethod stream-read-char ((stream string-input-stream))
    (with-slots (index end string) stream
      (if (>= index end)
	(prog1 (char string index)
	       (incf index)))))
  (defmethod stream-unread-char ((stream string-input-stream) character)
    (with-slots (index end string) stream
      (decf index)
      (assert (eql (char string index) character))
  (defmethod stream-read-line ((stream string-input-stream))
    (with-slots (index end string) stream
      (let* ((endline (position #\newline string :start index :end end))
	     (line (subseq string index endline)))
	(if endline
	    (progn (setq index (1+ endline))
		   (values line nil))
	  (progn (setq index end)
		 (values line t))))))
  (defclass string-output-stream (fundamental-character-output-stream)
    ((string :initform nil :initarg :string)))

    (make-instance 'string-output-stream))

  (defun GET-OUTPUT-STREAM-STRING (stream)
    (with-slots (string) stream
      (if (null string)
	(prog1 string (setq string nil)))))
  (defmethod stream-write-char ((stream string-output-stream) character)
    (with-slots (string) stream
      (when (null string)
	(setq string (make-array 64. :element-type 'string-char 
				 :fill-pointer 0 :adjustable t)))
      (vector-push-extend character string)
  (defmethod stream-line-column ((stream string-output-stream))
    (with-slots (string) stream
      (if (null string)
	(let ((nx (position #\newline string :from-end t)))
	  (if (null nx)
	      (length string)
	    (- (length string) nx 1))

Cost to Implementors:

  Given that CLOS is supported, adding the above generic functions and
  methods is easy, since most of the code is included in the examples
  above.  The hard part would be re-writing existing I/O functionality in
  terms of methods on these new generic functions.  That could be
  simplified if methods can be defined to forward the operations to the
  old representation of streams.  For a new implementation, the cost could
  be zero since an approach similar to this would likely be used anyway.

Cost to Users:

  None; this is an upward-compatible addition.   Users won't even
  need to know anything about this unless they actually need this feature.

Cost of non-adoption:

  Development of portable I/O extensions will be discouraged.

Performance impact:

  This shouldn't affect performance of new implementations (assuming an
  efficient CLOS implementation), but it could slow down I/O if it were
  clumsily grafted on top of an existing implementation.


  A broader domain of programs that can be written portably.


  This seems to be a simple, straight-forward approach.


  This proposal incorporates suggestions made by several people in
  response to an earlier outline.  So far, no one has expressed opposition
  to the concept.  There are some differences of opinion about whether
  certain operations should have default methods or required methods:

  An experimental prototype of this has been successfully implemented on
  the Explorer.

  This proposal does not provide sufficient capability to implement
  forwarding streams such as for MAKE-SYNONYM-STREAM,
  MAKE-ECHO-STREAM.  The generic function approach does not lend itself as
  well to that as a message passing model where the intermediary does not
  need to know what all the possible messages are.  A possible way of
  extending it for that would be to define a class 

    (defclass stream-generic-function (standard-generic-function) ())

  to be used as the :generic-function-class option for all of the I/O
  generic functions.  This would then permit doing something like

  (defmethod no-applicable-method ((gfun stream-generic-function) &rest args) 
    (if (streamp (first args))
	(apply #'stream-operation-not-handled (first args) gfun (rest args))

  where stream-operation-not-handled is a generic function whose default
  method signals an error, but forwarding streams can define methods that
  will create a method to handle the unexpected operation.  (Perhaps
  NO-APPLICABLE-METHOD should be changed to take two required arguments
  since all generic functions need at least one required argument, and
  that would make it unnecessary to define a new generic function class
  just to be able to write this one method.)

  Another thing that is not addressed here is a way to cause an instance
  of a user-defined stream class to be created from a call to the OPEN
  function.  That should be part of a separate issue for generic functions
  on pathnames.  If that capability were available, then PATHNAME and
  TRUENAME should be required to be generic functions.

  An earlier draft defined just two classes, FUNDAMENTAL-INPUT-STREAM and
  FUNDAMENTAL-OUTPUT-STREAM, that were used for both character and binary
  streams.  It isn't clear whether that simple approach is sufficient or
  whether the larger set of classes is really needed.

HTML version Copyright 2001 by Kent M. Pitman. All Rights Reserved.