Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!wuarchive!udel!rochester!pt.cs.cmu.edu!o.gp.cs.cmu.edu!ram
From: ram+@cs.cmu.edu (Rob MacLachlan)
Newsgroups: comp.lang.lisp
Subject: The future of Lisp
Message-ID: <1991Jan21.225418.22130@cs.cmu.edu>
Date: 21 Jan 91 22:54:18 GMT
Sender: netnews@cs.cmu.edu (USENET News Group Software)
Organization: School of Computer Science, Carnegie Mellon
Lines: 203
Well, since nobody else has taken on the mantle of Lisp visionary, I guess I
will. I'll start out with a few inflammatory claims:
-- More people than ever before are using "industrial strength" Lisp systems.
-- More people than ever before believe that "industrial strength" Lisp
systems are too big and complex.
-- "Industrial strength" Lisp systems are too small and simple.
I work for the CMU Common Lisp project. CMU Common Lisp is a high quality
public domain portable Common Lisp system. Although we have been developing
our Lisp since the early 80's, you probably haven't heard of it, since (up to
now) CMU Common Lisp only ran on strange hardware running CMU operating systems.
You will now be hearing more about CMU CL because we are running on MIPS and
SPARC boxes and because our O.S. requirements have gotten less exotic. More
about that later...
I'm going to be discussing the future of "industrial strength" Lisp here, since
that's my expertise. If your ideas about Lisp differ from mine, we might both
be right, since there are other niches for Lisp than the one occupied by
industrial-strength Lisp. An industrial-strength Lisp is an integrated
programming environment providing "power tools" for the evolutionary
development of complex programs. Such complex programs typically contain
several embedded extension languages and tens of thousands of lines of code.
Although Common Lisp was designed to be industrial-strength, there are also
industrial-strength Scheme systems. I'm not interested in starting a Common
Lisp v.s. anything-else flame war. What I want to do is clear up some
confusion resulting from the failure to distinguish industrial-strength Lisp
environments from other kinds of Lisp and other programming environments.
The main niche of industrial-strength Lisp is what I will call "development
prototyping." Development is the middle part of a three-stage model of
technical progress that goes from research to development to production.
Loosely speaking:
research is determining what is possible,
development is figuring out how to do it, and
production is making it cost-effective.
Development is really a process that ideas go through on the way from research
to production. Early development is indistinguishable from research, and late
development is indistinguishable from production engineering.
The prototyping part of "development prototyping" refers to constructing
programs that are not going to be mass-produced. A prototype is a functioning
model of a system under development. Lisp program prototypes are like real
engineering prototypes, in that they:
-- implement a solution to a problem well enough to prove or refute the
feasibility of that solution.
-- may have holes in functionality and rough edges that would be unacceptable
in a production system.
-- are repeatedly torn apart and put back together to add new functionality.
-- may be much more expensive to construct than a production system.
When I say that industrial strength Lisp is used for development prototyping, I
do not mean to imply that all Lisp programs should be thrown away. Although
catchy, I don't like the phrase "Plan to throw one away --- you will anyway."
Prototyping is an inherent part of the development process. A Lisp prototype
can easily be developed to have functionality and reliability comparable to a
production version. In research, feasibility studies and internal tool
development, it is usually unnecessary to code a production version, since the
cost of running the prototype is a small part of the cost of the total
development process.
I am claiming that development prototyping *is* and has *always been* the main
use of industrial-strength Lisp systems. This is entirely consistent with some
other ideas about Lisp, like:
-- Lisp is for extension languages, or
-- Lisp is for Artificial Intelligence.
What we are seeing here is the *history* of Lisp. MULTICS EMACS was written in
an industrial strength Lisp because the developers understood that Lisp
features would make it easy to implement a revolutionary text editor. Later
Unix EMACSes such as GNU EMACS were written mostly in C for efficiency,
but they retained a micro-Lisp interpreter for supporting simple
customizations. Because of the development done by ITS and MULTICS EMACS,
screen editors eventually reached the masses. Because of this development, we
know what primitives to design into a micro-Lisp for editor extensions.
At CMU, we still believe that it makes sense to write editors in
industrial-strength Lisp. It makes sense because editor development didn't
stop with MULTICS EMACS. We're doing new things with editors, so we still want
to be able rip our editors apart and stick them back together.
Similarly, Lisp is in the history of AI development. AI started as pure
blue-sky. People developed the first Lisp systems because they wanted to
prototype AI systems as part of their research, and existing languages were
grossly inadequate for the sort of complex symbolic operations the researchers
had in mind. It is history that AI was the first problem attacked by computer
science which clearly needed program prototyping tools. It is fact that
because industrial-strength Lisp was developed for prototyping AI systems, Lisp
became good for prototyping.
Some ideas developed in the early days of AI have now made it into production.
These production implementations are generally not Lisp-based, and this is
appropriate. Many of the early expectations for AI were wildly optimistic.
Lisp prototypes of simplistic "intelligence" models proved how complex
intelligence really is, and that is science. Negative results notwithstanding,
Lisp is still widely used by the AI researchers working on today's more
complicated models of intelligence.
Although people have been citing Symbolics's hard times as evidence of the
death of Lisp, I see it rather differently. Language-specific workstations are
now obsolescent, but the LISPM was not a dead end. The software development
done on the LISPMs has been tremendously influential, and the LISPM provided
the first industrial-strength Lisp workstation. It is not an accident that X
windows has "atoms" and "properties"; it is an accident that you can't run
Symbolics software on a Sun. I think that Symbolics would have done better if
they had more aggressively pursued compatibility with the emerging Unix-based
O.S. standards. As it is now, they have gobs of tremendously nifty software,
but they can't run it without their proprietary Lisp operating system.
One standard that Symbolics did get on top of was Common Lisp, and this
provided a good path off of the LISPM for many of their users. The Common Lisp
standardization effort of the early 80's happened because the core languages of
industrial-strength Lisp systems had become mature enough so that the arbitrary
decisions inherent in standardization could be made without fear of horrible
mistakes. Common Lisp has tremendously increased the popularity of Lisp
development prototypes. Much of this growth has been in areas outside of AI
and programming languages.
It is important to distinguish between industrial-strength Lisp and Lisp as a
subject of programming language research. Lisp was a breakthrough in
programming language design. Lisp is still at the cutting edge of programming
language design in the Scheme community. However, industrial-strength Lisp
systems were developed as research tools, not as research subjects. An
industrial-strength Lisp system does need to push the state of the art in
programming environment, but the language itself need not be cutting-edge.
My expectation is that for development prototyping, Scheme is not enough better
than Common Lisp to replace it any time soon. Scheme still has a great future
as a testbed for new programming language ideas, and it is possible that Scheme
will spin off new major languages that go beyond Common Lisp (which was
strongly inspired by Scheme.) However, I think that a major reason for
Scheme's popularity in programming language research communities is economic
rather than technical: there are Scheme systems with free source availability.
CMU Common Lisp is now making it possible for people to mess around with the
guts of Common Lisp as well.
There are a number of development prototypes written in Common Lisp at CMU.
One example is the Warp compiler. Warp is hardware research project to develop
a highly parallel number-cruncher. In addition to traditional
number-crunching, Warp is also being used for low-level vision processing and
connectionist AI. The compiler compiles programs for this special hardware.
Compiler development is a big part of the Warp project, and many new parallel
compilation techniques were developed in the Warp compilers. There's
a fair chance that when the Warp hardware makes it into production the compiler
will be reimplemented, but parallel compilation research will still be done in
Lisp.
Another development prototype is the user interface work being done by the
Garnet project. This is effectively a toolkit for constructing user
interfaces, but with the emphasis on doing r+d in user interfaces, rather than
production user interfaces. Although the landmark work in graphical user
interfaces was done in SmallTalk, industrial-strength Lisp systems were not far
behind. Now GUIs run on IBM PCs, and are written in assembler, and this is
appropriate. But Lisp is still a great language for user interface r+d.
CMU Common Lisp is itself something of a development prototype. From the
first, the goal of CMU Common Lisp was to provide a "LISPM quality" environment
with a powerful editor, debugger and coding tools. CMU CL is bigger and more
powerful than the commercial offerings of Lucid and Franz. These are the areas
where we have a significant technical lead:
-- Incremental programming environments (editors.) We have a full-function
EMACS-compatible text editor (written in Lisp) with many Lisp-specific
programming features. This was our first area of concentration; I think it
is pretty clear that we have the best non-Lispm environment, though the
current crop of sophisticated packages for GNU EMACS has cut our lead
here. We have some unusual support for distributed environments with
multiple (possibly different) CPUs. We are working on screen-oriented
debugging and integration of RCS version control.
-- Compiler optimization. The Python compiler (which I wrote) is by far that
most sophisticated Common Lisp compiler. Although much of Python's
implementation effort went into non-Scheme features such as type inference
and number crunching, in the areas where Common Lisp intersects with
Scheme, Python compares favorably to the best Scheme compilers. Python is
the first real compiler to apply flow analysis, global register allocation
and other mainstream optimization techniques to Lisp-specific problems like
type inference and representation selection.
-- Safety. We have always felt it was important to have complete and reliable
run-time error detection. We do *more* type checking than LISPMs.
Python's type support wins because:
- Type check optimization makes type checking less prohibitive on
conventional hardware.
- Compile-time type errors detect many trivial coding errors.
- Precise type checking allows more interesting consistency constraints to
be expressed.
-- Debugging. CMU CL allows source-level debugging of compiled code,
including precise display of the original source form responsible for a
run-time error. Although the current debugger user interface is plain,
this is a major "how did I ever live without it" capability.
-- Portability. We have a powerful and robust back-end definition mechanism
which generates top-quality native code. There is currently 9.5 thousand
lines of architecture-dependent code (11% of the total compiler source.)
Ports take about two (expert) man-months. You can get more portable than
this, but not without sacrificing efficiency.
Currently CMU CL only runs under Mach (on the SPARC, MIPS and IBM RT PC). CMU
CL is available through the Mach distribution mechanisms (requiring proof of
BSD license, lawyers, etc.) It should also run on OSF, and it would be
possible to port to SUNOS, Ultrix, etc. This post does not constitute a formal
announcement of CMU CL (we lack a distribution mechanism), but we can arrange
FTP access to the sources for people doing ports. Porting documentation is
still totally lacking.
Robert A. MacLachlan (ram@cs.cmu.edu)