Hunchentoot and CCL on Windows. Take 2.

I recently posted a brief workaround for a bug in CCL on MS Windows which prevented Hunchentoot from responding to HTTP requests. 

Shortly after posting this I, and everyone else on the openmcl-devel mailing list, received an email from R. Matthew Emerson who has informed us that this issue was resolved in their SVN repository over a week ago and that CCL no longer needs the workaround I posted.

So for all those CCL+Windows users, you can now

svn update

and

(rebuild-ccl :clean t)

and ignore my previous post.

Hunchentoot, CCL and Windows

It's been a couple of months now and I've been warming to CCL, so much so in fact, that it has replaced SBCL as my `other` CL implementation. If you've never used it, it is an Open Source Common Lisp which runs on a number of platforms, including Windows, Mac OS X and Linux in both 32 and 64 bit modes and supports threading on all 3 of these platforms.

There is however a small issue when running Hunchentoot with CCL on Windows; you will notice when connecting to a CCL+Hunchentoot server that it never responds with any content but hangs indefinitely. This is caused by a bug in the CCL Windows timeout handling which requires a small workaround by the user.

Fortunately the workaround is straightforward; simply start up Hunchentoot with the following extra parameters:

> (hunchentoot:start (make-instance 'hunchentoot:acceptor :port 8080 :read-timeout nil :write-timeout nil))

And you are on your way.

Mudballs 3

Well, I've finally pushed Mudballs 3 out the door and, while it contains a large number of changes and many improvements, this post isn't only about the work that has been done in the past few months but will also touch on where Mudballs will be going in the months to come.

The bad news is that, once again and for the very last time,  Mudballs users will be required to reinstall the mudballs distribution as various internals have been moved around, I'm terribly sorry about this and I've done my best to ensure that this is the last time that it will happen. Please note that nothing will break if you do not upgrade but the old repository will no longer be updated.

As always Mudballs is available from http://mudballs.com/#installing.

Lets first look at the changes that have made their way into the repository.

1. ASDF Support: This is a big one, Mudballs now includes a sysdef.asdf system which provides an ASDF package (or more precisely, a SYSDEF.ASDF package with the nickname ASDF) to provide ASDF support. To be more precise defsystem, operate and  the various component and action classes are exported and provide the same service you would normally expect from ASDF. The biggest drawback that it has at the moment is that when specializing methods on operate invoking `call-next-method` will have undefined consequences (read that as `Don't call call-next-method`).


2. Single File Systems: Mudballs can now be used to load single files on your computer. Mudballs will now search on the list *search-paths*   for systems when a relative portable pathname is supplied.  As an example, given a *search-paths* of '("~/Lisp/") and a file named #P"~/Lisp/workfiles/cc-validate.lisp" I can compile and load this file using the following form.  

   (mb:load ";workfiles;cc-validate")

    Internally a component is created in mudballs and is operated upon and can be looked up using find-system.   The component which is created can be customized by adding a (mb:component . options) form to the top of the file.  As an example, a small file which needs :cl-ppcre can be customized by adding the following form to the top of the file.  

   #+mudballs (mb:component (:needs :cl-ppcre))

    It is worth noting that SINGLE-FILE-SYSTEM is not, in fact, a subclass of SYSTEM but rather a subclass of LISP-SOURCE-FILE.


3. Multiple parent classes: DEFINE-SYSTEM now accepts any number of parent classes. This will create an anonymous class which extends the provided classes. eg.

   (define-system :my-system (core-system-mixin system)  .....)

   Will create a new system which extends both core-system-mixin and system.


4. FOR Option:  In an attempt to remove the need for #+ and #- reader macros in system definitions (which do not behave well with automating releases) a :FOR option has been added to components and to the syntax for defining dependencies which can be used to achieve that same goal as read time conditionalization. As an example, consider Edi Weitz's excellent :DRAKMA system which has a dependency on :USOCKET for all implementation other than Lispworks. Previously the needs form would have looked something like the following. 

    (:NEEDS #-lispworks :USOCKET)

       Using :FOR (which is now the preferred approach) the :NEEDS form will now look like this. 

    (:NEEDS (:USOCKET (:FOR (:NOT :LISPWORKS)))) 

   Granted this is somewhat more verbose (although somewhat less so when replacing a #+ conditionalization) it does mean that the system dependencies are now portably inspectable. This option is applicable to ALL instances of components, including modules and files. It is also worth noting the value for the FOR option can be anything which is suitable to sysdef::featurep.


5. NAMED-MODULE: The named-module is a new subclass of module which provides a way to specify the directory of the module per implementation,  this is designed to provide a more introspectable alternative to the reader conditionalization macros. As an example, CLOSER-MOP has the following NAMED-MODULE in it's COMPONENTS list. 
   

   (COMPAT NAMED-MODULE  
     (:NAMES :ALLEGRO :CLISP :LISPWORKS
             ((:OR :MCL :OPENMCL) "mcl")  
             ((:OR :CMU :SBCL) "pcl"))  
     (:COMPONENTS "closer-mop-packages" "closer-mop"))

   Each entry in the :NAMES option is expected to be either a symbol which, when matched using featurep, will use the string-downcase'd  version of the symbol as the directory name. When a list is provided then the first element in the list is the argument which will be tested using featurep and the second is the name to use if it matches.


6. REQUIRES Option: The requires option has gone through a number of changes and has now been resurrected from a deprecated nickname  for :NEEDS to portable way to `CL:REQUIRE` implementation specific libraries which you have a dependency on. Previously this was only handled for SBCL by defining SBCL systems in a system-definition file. This file has now been dropped and the :REQUIRES option is now the preferred  way to load implementation specific requirements.  As an example usocket has a dependency on SB-BSD-SOCKETS which used to be handled by having a

   (:NEEDS #+sbcl :SB-BSD-SOCKETS)

   option. This is now replaced by

   (:REQUIRES (:SB-BSD-SOCKETS (:FOR :SBCL)))



7. CONDUIT-SYSTEM's: A new subclass of System is now exported; this class, which performs the same task as asdf-system-connections,  provides a means to define a system which is loaded automatically when all of it's dependencies are loaded. A conduit system can be defined either by specifying CONDUIT-SYSTEM in the superclass list of define-system or by defining your system using DEFINE-CONDUIT-SYSTEM.


8. Portable Pathname Specifiers: Mudballs now provides a portable way of specifying pathnames when passing pathnames to the :config-file and :preferences-file options. In this naming scheme directory names are seperated by #\;'s (a semicolon) and mudballs will ensure that any path starting with #\~ (tilde) will have the result of (user-homedir-pathname) merged with the remaining namestring. Example. "~;mudballs;test.lisp" is a portable yet shorter way of specifying 
   

   
   (merge-pathnames (make-pathname :directory '(:relative "mudballs")
                                   :name "test" :type "lisp") 
                    (user-homedir-pathname))

Other Features and Changes

• FEATURE: system definitions files now are automatically compiled when mudballs changes. (this includes files added using wildcard-searcher)
  
• FEATURE: You no longer need to run sysdef:register-sysdefs when you add new systems. It is automatically run before find-system (some attempt has been made to ensure that it doesn't impact performance too heavily)
  
• BUG FIX: (:needs (:cl-ppcre :version "1.3")) is once again valid.
  
• UPDATE: The no-installed-component condition has been removed as it is a conceptual duplicate.
  
• UPDATE: redefining a system no longer creates a new instance of a SYSTEM but will reset and update the currently defined one if one exists.
  
• BUG FIX: redefining a system will not cause the system to be considered un-operated upon.
  
• NEWS: Mudballs now waits until the last possible moment before loading system definitions, this now results in faster startup times.
  
• And many other small changes and fixes.
  

The Future

Now that we've gotten the past and present out of the way lets look to the future. Mudballs, while having had a large portion of work dedicated to operating upon systems, does NOT have 'operating on systems' as it's primary goal. Mudballs is primarily about Distribution and distributing Common Lisp systems in particular, which is 

1. It's biggest difference to ASDF and 
   
2. a far more useful problem that needs solving.
   

So what does this mean? It means that mudballs users can expect, in the near future, a portion of mudballs.com dedicated to the creation of projects and the uploading of new versions of systems which will automatically provide new definition files and downloadable mudballs on mudballs.com.

This will allow any writer of Common Lisp code to upload their system and make it available (with correct version information) to the rest of the world, complete with automatically generated documentation and links to the various releases. This, not so small, feature should go along way to providing Common Lispers with something which is at least converging on CPAN.

This is all being done with one thing in mind: Repeatability.

This is currently the biggest shortcoming of the ASDF family, in that the installation of an arbitrary set of systems cannot be reliably reproduced, which makes troubleshooting library combination issues incredibly difficult; and of course the more systems you have installed the more likely it becomes that what you have will differ from a maintainer which makes having dependencies on systems an inherently risky affair.

I know I'm not the only person who's had a particular system fail only to be told by the maintainer that `It works for me` followed by a list of the versions he is using. This is typically followed by the `Vain attempt to find these exact versions` dance [1] which is all too often followed by the `I give up, hands in the air` shuffle. As it stands, ASDF and friends just do not help here and this is the situation Mudballs is intended to solve.

Mudballs primary purpose is not System Definition but Repeatable System Distribution.

As always comments and investments of time are welcome and you can reach myself and other Mudballs users at the Mudballs Google Group.

Enjoy,

 Sean.

1) Don't even get me started about systems whose primary form of distribution is `Fresh checkout from VC tool of choice`.

Mudballs Google Group

Not a full update I'm afraid, although one will be coming shortly.

In the interests of promoting discussion about Mudballs I have created a mudballs google group.

This can be found by pointing your browser at http://groups.google.com/group/mudballs/  

I hope to see you there.

sean.

A Mudballs Update

Yes, I know it's late but better late than never right.

Anyway, I have just uploaded the 'Christmas Cheer' release of mudballs which, unfortunately, will require a fresh install of mudballs for existing users as there was a rather egregious oversight regarding the updating of systems.

Noteworthy Features

• A publicly editable wiki has now been made available at http://redmine.mudballs.com/wiki/mudballs/   New pages can be created by opening the appropriate URL. For example, http://redmine.mudballs.com/wiki/mudballs/hunchentoot will  show an option for creating a new page. 
• The start of a mudballs FAQ can be found on it's wiki page: http://redmine.mudballs.com/wiki/mudballs/FAQ.
  
• down-casing of pathnames has been fixed. Down-casing will now only happen for components named with symbols.
  
• New versions of all systems, including core systems such as :cl-ppcre, can now be updated without having to reinstall mudballs.
  
• (mb:install :cl-ppcre :file "/path/to/file") now works.
  
• The output path of component is now calculated correctly (including honoring the :fasl-output-root preference).
  
• Uninstalled systems are no longer considered  for an operation if there is an installed version of the system present unless explicitly requested using the :version keyword.
  
• The fasl-path of system definition files is no longer the same as components of a system as this was conflicting with the output of compiling components.
  
• And other small fixes

As always mudballs is available for download from the releases directory and instructions can be found here.

Please let me know if any server on mudballs.com is not responding as my hosting provider is being a little flaky of late.

Thanks to the following people for reports and ideas.

- Magnus Malm

- Luke Renn

- Leigh Smith

Announcing Mudballs

 This post is to announce the release of Mudballs, a collection of Common Lisp code for managing systems.  It is currently in an early stage of development and still has a long way to go but is currently able to compile,  load, document, download and install systems across Windows, Linux and Mac OSX. Mudballs supports the following implementations 

• Lispworks on Mac OSX, Linux and Windows.
  
• SBCL on Mac OSX and Linux.
• CMUCL on Mac OSX and Linux.
• CLISP on Linux and Windows (Mac OSX as well provided  you have FFI).
• ClozureCL/OpenMCL on Mac OSX, Linux and Windows.
• AllegroCL on Mac OSX, Linux and Windows.

 

 As it currently stands it is not ASDF compatible (not ideal, I know) but ASDF support is planned for the future. It is available from http://mudballs.com  which contains enough information to get the intrepid lisper on his or  her way.

 

 As a small taste, the following is now possible from a freshly installed Lisp with Mudballs loaded.

> (mb:install :hunchentoot)
> (hunchentoot:start-server :port 8080)

P.S.  Prizes for anyone that can come up with a logo which doesn't look like a pile of excrement.

Renaming files in Common Lisp.

So I ran across this while going through my morning catchup routine and thought it could do with some clearing up.

The part of the post I'm actually interested is this.

When Haskell can compete on those types of problems, it'll be easier to induce people to learn it. (Same with CL, my fav language....)

Now, bonus points for proclaiming CL as being his favorite language, but minus 10 billion for continuing the meme that CL is somehow unsuited for writing[1] these kinds of simple software (for whatever definition of simple you have).

One of the examples given is "I have a bunch of files, and I want to rename them all according to some pattern." and as it so happens translate-pathname[2] makes this wonderfully simple.

(defun rename-files (from to)
 (dolist (file (directory from))
   (rename-file file (translate-pathname file from to))))

And thats it, 3 lines of code, or for our simple testing purposes

(defun show-rename-files (from to)
 (dolist (file (directory from))
   (format t "Renaming ~A to ~A~%" file
           (translate-pathname file from to))))

(show-rename-files "/usr/share/pixmaps/*.xpm" "/usr/share/pixmaps/backup-*.xpm")

The funny thing is that this isn't secret knowledge but is pulled straight from the Hyperspec (see the examples).

----
[1]: Please note, I said 'writing' them, not 'creating a 2k binary' of them, please people CL /= Unix.
[2]: Granted, the behavior of translate-pathname isn't specified in detail by the spec but that doesn't mean we cannot use it.

This post brought to you by Lispworks 5.1, clisp 2.41 and SBCL 1.0.12