Mudballs 3

Well, I've finally pushed Mudballs 3 out the door and, while it contains a large number of changes and many improvements, this post isn't only about the work that has been done in the past few months but will also touch on where Mudballs will be going in the months to come.

The bad news is that, once again and for the very last time,  Mudballs users will be required to reinstall the mudballs distribution as various internals have been moved around, I'm terribly sorry about this and I've done my best to ensure that this is the last time that it will happen. Please note that nothing will break if you do not upgrade but the old repository will no longer be updated.

As always Mudballs is available from http://mudballs.com/#installing.

Lets first look at the changes that have made their way into the repository.

1. ASDF Support: This is a big one, Mudballs now includes a sysdef.asdf system which provides an ASDF package (or more precisely, a SYSDEF.ASDF package with the nickname ASDF) to provide ASDF support. To be more precise defsystem, operate and  the various component and action classes are exported and provide the same service you would normally expect from ASDF. The biggest drawback that it has at the moment is that when specializing methods on operate invoking `call-next-method` will have undefined consequences (read that as `Don't call call-next-method`).


2. Single File Systems: Mudballs can now be used to load single files on your computer. Mudballs will now search on the list *search-paths*   for systems when a relative portable pathname is supplied.  As an example, given a *search-paths* of '("~/Lisp/") and a file named #P"~/Lisp/workfiles/cc-validate.lisp" I can compile and load this file using the following form.  

   (mb:load ";workfiles;cc-validate")

    Internally a component is created in mudballs and is operated upon and can be looked up using find-system.   The component which is created can be customized by adding a (mb:component . options) form to the top of the file.  As an example, a small file which needs :cl-ppcre can be customized by adding the following form to the top of the file.  

   #+mudballs (mb:component (:needs :cl-ppcre))

    It is worth noting that SINGLE-FILE-SYSTEM is not, in fact, a subclass of SYSTEM but rather a subclass of LISP-SOURCE-FILE.


3. Multiple parent classes: DEFINE-SYSTEM now accepts any number of parent classes. This will create an anonymous class which extends the provided classes. eg.

   (define-system :my-system (core-system-mixin system)  .....)

   Will create a new system which extends both core-system-mixin and system.


4. FOR Option:  In an attempt to remove the need for #+ and #- reader macros in system definitions (which do not behave well with automating releases) a :FOR option has been added to components and to the syntax for defining dependencies which can be used to achieve that same goal as read time conditionalization. As an example, consider Edi Weitz's excellent :DRAKMA system which has a dependency on :USOCKET for all implementation other than Lispworks. Previously the needs form would have looked something like the following. 

    (:NEEDS #-lispworks :USOCKET)

       Using :FOR (which is now the preferred approach) the :NEEDS form will now look like this. 

    (:NEEDS (:USOCKET (:FOR (:NOT :LISPWORKS)))) 

   Granted this is somewhat more verbose (although somewhat less so when replacing a #+ conditionalization) it does mean that the system dependencies are now portably inspectable. This option is applicable to ALL instances of components, including modules and files. It is also worth noting the value for the FOR option can be anything which is suitable to sysdef::featurep.


5. NAMED-MODULE: The named-module is a new subclass of module which provides a way to specify the directory of the module per implementation,  this is designed to provide a more introspectable alternative to the reader conditionalization macros. As an example, CLOSER-MOP has the following NAMED-MODULE in it's COMPONENTS list. 
   

   (COMPAT NAMED-MODULE  
     (:NAMES :ALLEGRO :CLISP :LISPWORKS
             ((:OR :MCL :OPENMCL) "mcl")  
             ((:OR :CMU :SBCL) "pcl"))  
     (:COMPONENTS "closer-mop-packages" "closer-mop"))

   Each entry in the :NAMES option is expected to be either a symbol which, when matched using featurep, will use the string-downcase'd  version of the symbol as the directory name. When a list is provided then the first element in the list is the argument which will be tested using featurep and the second is the name to use if it matches.


6. REQUIRES Option: The requires option has gone through a number of changes and has now been resurrected from a deprecated nickname  for :NEEDS to portable way to `CL:REQUIRE` implementation specific libraries which you have a dependency on. Previously this was only handled for SBCL by defining SBCL systems in a system-definition file. This file has now been dropped and the :REQUIRES option is now the preferred  way to load implementation specific requirements.  As an example usocket has a dependency on SB-BSD-SOCKETS which used to be handled by having a

   (:NEEDS #+sbcl :SB-BSD-SOCKETS)

   option. This is now replaced by

   (:REQUIRES (:SB-BSD-SOCKETS (:FOR :SBCL)))



7. CONDUIT-SYSTEM's: A new subclass of System is now exported; this class, which performs the same task as asdf-system-connections,  provides a means to define a system which is loaded automatically when all of it's dependencies are loaded. A conduit system can be defined either by specifying CONDUIT-SYSTEM in the superclass list of define-system or by defining your system using DEFINE-CONDUIT-SYSTEM.


8. Portable Pathname Specifiers: Mudballs now provides a portable way of specifying pathnames when passing pathnames to the :config-file and :preferences-file options. In this naming scheme directory names are seperated by #\;'s (a semicolon) and mudballs will ensure that any path starting with #\~ (tilde) will have the result of (user-homedir-pathname) merged with the remaining namestring. Example. "~;mudballs;test.lisp" is a portable yet shorter way of specifying 
   

   
   (merge-pathnames (make-pathname :directory '(:relative "mudballs")
                                   :name "test" :type "lisp") 
                    (user-homedir-pathname))

Other Features and Changes

• FEATURE: system definitions files now are automatically compiled when mudballs changes. (this includes files added using wildcard-searcher)
  
• FEATURE: You no longer need to run sysdef:register-sysdefs when you add new systems. It is automatically run before find-system (some attempt has been made to ensure that it doesn't impact performance too heavily)
  
• BUG FIX: (:needs (:cl-ppcre :version "1.3")) is once again valid.
  
• UPDATE: The no-installed-component condition has been removed as it is a conceptual duplicate.
  
• UPDATE: redefining a system no longer creates a new instance of a SYSTEM but will reset and update the currently defined one if one exists.
  
• BUG FIX: redefining a system will not cause the system to be considered un-operated upon.
  
• NEWS: Mudballs now waits until the last possible moment before loading system definitions, this now results in faster startup times.
  
• And many other small changes and fixes.
  

The Future

Now that we've gotten the past and present out of the way lets look to the future. Mudballs, while having had a large portion of work dedicated to operating upon systems, does NOT have 'operating on systems' as it's primary goal. Mudballs is primarily about Distribution and distributing Common Lisp systems in particular, which is 

1. It's biggest difference to ASDF and 
   
2. a far more useful problem that needs solving.
   

So what does this mean? It means that mudballs users can expect, in the near future, a portion of mudballs.com dedicated to the creation of projects and the uploading of new versions of systems which will automatically provide new definition files and downloadable mudballs on mudballs.com.

This will allow any writer of Common Lisp code to upload their system and make it available (with correct version information) to the rest of the world, complete with automatically generated documentation and links to the various releases. This, not so small, feature should go along way to providing Common Lispers with something which is at least converging on CPAN.

This is all being done with one thing in mind: Repeatability.

This is currently the biggest shortcoming of the ASDF family, in that the installation of an arbitrary set of systems cannot be reliably reproduced, which makes troubleshooting library combination issues incredibly difficult; and of course the more systems you have installed the more likely it becomes that what you have will differ from a maintainer which makes having dependencies on systems an inherently risky affair.

I know I'm not the only person who's had a particular system fail only to be told by the maintainer that `It works for me` followed by a list of the versions he is using. This is typically followed by the `Vain attempt to find these exact versions` dance [1] which is all too often followed by the `I give up, hands in the air` shuffle. As it stands, ASDF and friends just do not help here and this is the situation Mudballs is intended to solve.

Mudballs primary purpose is not System Definition but Repeatable System Distribution.

As always comments and investments of time are welcome and you can reach myself and other Mudballs users at the Mudballs Google Group.

Enjoy,

 Sean.

1) Don't even get me started about systems whose primary form of distribution is `Fresh checkout from VC tool of choice`.