diff options
Diffstat (limited to 'Lisp/asdf/README')
| -rw-r--r-- | Lisp/asdf/README | 752 |
1 files changed, 752 insertions, 0 deletions
diff --git a/Lisp/asdf/README b/Lisp/asdf/README new file mode 100644 index 0000000..7243844 --- /dev/null +++ b/Lisp/asdf/README @@ -0,0 +1,752 @@ +$Id: README 20 2005-12-27 15:21:23Z bjc $ -*- Text -*- + + +asdf: another system definition facility +======================================== + +* Getting the latest version + +0) Decide which version you want. HEAD is the newest version and +usually OK, whereas RELEASE is for cautious people (e.g. who already +have systems using asdf that they don't want broken), a slightly older +version about which none of the HEAD users have complained. + +1) Check it out from sourceforge cCLan CVS: + +1a) cvs -d:pserver:anonymous@cvs.cclan.sourceforge.net:/cvsroot/cclan login + (no password: just press Enter) + +1a.1) cvs -z3 -d:pserver:anonymous@cvs.cclan.sourceforge.net:/cvsroot/cclan + co -r RELEASE asdf + +or for the bleeding edge, instead + +1a.2) cvs -z3 -d:pserver:anonymous@cvs.cclan.sourceforge.net:/cvsroot/cclan + co -A asdf + +If you are tracking the bleeding edge, you may want to subscribe to +the cclan-commits mailing list (see +<URL:http://sourceforge.net/mail/?group_id=28536>) to receive commit +messages and diffs whenever changes are made. + +For more CVS information, look at http://sourceforge.net/cvs/?group_id=28536 + + +* Getting started + +- The single file asdf.lisp is all you need to use asdf normally. For +maximum convenience you want to have it loaded whenever you start your +Lisp implementation, by loading it from the startup script, or dumping +a custom core, or something. + +- The variable asdf:*central-registry* is a list of system directory + designators. A system directory designator is a form which will be + evaluated whenever a system is to be found, and must evaluate to a + directory to look in. For example, you might have + + (*default-pathname-defaults* "/home/me/cl/systems/" + "/usr/share/common-lisp/systems/") + + (When we say "directory" here, we mean "designator for a pathname + with a supplied DIRECTORY component") + + It is possible to customize the system definition file search. + That's considered advanced use, and covered later: search forward + for *system-definition-search-functions* + +- To compile and load a system 'foo', you need to (1) ensure that + foo.asd is in one of the directories in *central-registry* (a + symlink to the real location of foo.asd is preferred), (2) execute + ``(asdf:operate 'asdf:load-op 'foo)'' + + $ cd /home/me/cl/systems/ + $ ln -s ~/src/foo/foo.asd . + $ lisp + * (asdf:operate 'asdf:load-op 'foo) + +- To write your own system definitions, look at the test systems in + test/ , and read the rest of this. Ignore systems/ which is old + and may go away when next I clean up + +- Syntax is similar to mk-defsystem 3 for straightforward systems, you + may only need to remove the :source-pathname option (and replace it + with :pathname if the asd file is not in the same place as the + system sources) + +- Join cclan-list@lists.sf.net for discussion, bug reports, questions, etc + +- cclan.asd and the source files listed therein contain useful extensions + for maintainers of systems in the cCLan. If this isn't you, you + don't need them - although you may want to look at them anyway + +- For systems that do complicated things (e.g. compiling C files to + load as foreign code), the packages in vn-cclan may provide some + guidance. db-sockets, for example, is known to do outlandish things + with preprocessors + + http://ww.telent.net/cliki/vn-cclan + + + +* Concepts + +This system definition utility talks in terms of 'components' and +'operations'. + +Components form systems: a component represents a source file, or a +collection of components. A system is therefore a component, +recursively formed of a tree of subcomponents. + +Operations are instantiated then performed on the nodes of a tree to +do things like + + - compile all its files + - load the files into a running lisp environment + - copy its source files somewhere else + +Operations can be invoked directly, or examined to see what their +effects would be without performing them. There are a bunch of +methods specialised on operation and component type which actually do +the grunt work. + +asdf is extensible to new operations and to new component types. This +allows the addition of behaviours: for example, a new component could +be added for Java JAR archives, and methods specialised on +compile-op added for it that would accomplish the relevant +actions. + +* Inspiration + +** mk-defsystem (defsystem-3.x) + +We aim to solve basically the same problems as mk-defsystem does. +However, our architecture for extensibility better exploits CL +language features (and is documented), and we intend to be portable +rather than just widely-ported. No slight on the mk-defsystem authors +and maintainers is intended here; that implementation has the +unenviable task of supporting non-ANSI implementations, which I +propose to ignore. + +The surface defsystem syntax of asdf is more-or-less compatible with +mk-defsystem + +The mk-defsystem code for topologically sorting a module's dependency +list was very useful. + +** defsystem-4 proposal + +Marco and Peter's proposal for defsystem 4 served as the driver for +many of the features in here. Notable differences are + +- we don't specify output files or output file extensions as part of + the system + + If you want to find out what files an operation would create, ask + the operation + +- we don't deal with CL packages + + If you want to compile in a particular package, use an in-package + form in that file (ilisp will like you more if you do this anyway) + +- there is no proposal here that defsystem does version control. + + A system has a given version which can be used to check + dependencies, but that's all. + +The defsystem 4 proposal tends to look more at the external features, +whereas this one centres on a protocol for system introspection. + +** kmp's "The Description of Large Systems", MIT AI Memu 801 + +Available in updated-for-CL form on the web at +http://world.std.com/~pitman/Papers/Large-Systems.html + +In our implementation we borrow kmp's overall PROCESS-OPTIONS and +concept to deal with creating component trees from defsystem surface +syntax. [ this is not true right now, though it used to be and +probably will be again soon ] + + +* The Objects + +** component + +*** Component Attributes + +**** A name (required) + +This is a string or a symbol. If a symbol, its name is taken and +lowercased. The name must be a suitable value for the :name initarg +to make-pathname in whatever filesystem the system is to be found. + +The lower-casing-symbols behaviour is unconventional, but was selected +after some consideration. Observations suggest that the type of +systems we want to support either have lowercase as customary case +(Unix, Mac, windows) or silently convert lowercase to uppercase +(lpns), so this makes more sense than attempting to use :case :common, +which is reported not to work on some implementations + +**** a version identifier (optional) + +This is used by the test-system-version operation (see later). + +**** *features* required + +Traditionally defsystem users have used reader conditionals to include +or exclude specific per-implementation files. This means that any +single implementation cannot read the entire system, which becomes a +problem if it doesn't wish to compile it, but instead for example to +create an archive file containing all the sources, as it will omit to +process the system-dependent sources for other systems. + +Each component in an asdf system may therefore specify features using +the same syntax as #+ does, and it will (somehow) be ignored for +certain operations unless the feature conditional matches + +**** dependencies on its siblings (optional but often necessary) + +There is an excitingly complicated relationship between the initarg +and the method that you use to ask about dependencies + +Dependencies are between (operation component) pairs. In your +initargs, you can say + +:in-order-to ((compile-op (load-op "a" "b") (compile-op "c")) + (load-op (load-op "foo"))) + +- before performing compile-op on this component, we must perform +load-op on "a" and "b", and compile-op on c, - before performing +load-op, we have to load "foo" + +The syntax is approximately + +(this-op {(other-op required-components)}+) + +required-components := component-name + | (required-components required-components) + +component-name := string + | (:version string minimum-version-object) + +[ This is on a par with what ACL defsystem does. mk-defsystem is less +general: it has an implied dependency + + for all x, (load x) depends on (compile x) + +and using a :depends-on argument to say that b depends on a _actually_ +means that + + (compile b) depends on (load a) + +This is insufficient for e.g. the McCLIM system, which requires that +all the files are loaded before any of them can be compiled ] + +In asdf, the dependency information for a given component and +operation can be queried using (component-depends-on operation +component), which returns a list + +((load-op "a") (load-op "b") (compile-op "c") ...) + +component-depends-on can be subclassed for more specific +component/operation types: these need to (call-next-method) and append +the answer to their dependency, unless they have a good reason for +completely overriding the default dependencies + +(If it weren't for CLISP, we'd be using a LIST method combination to +do this transparently. But, we need to support CLISP. If you have +the time for some CLISP hacking, I'm sure they'd welcome your fixes) + +**** a pathname + +This is optional and if absent will be inferred from name, type (the +subclass of source-file), and the location of parent. + +The rules for this inference are: + +(for source-files) +- the host is taken from the parent +- pathname type is (source-file-type component system) +- the pathname case option is :local +- the pathname is merged against the parent + +(for modules) +- the host is taken from the parent +- the name and type are NIL +- the directory is (:relative component-name) +- the pathname case option is :local +- the pathname is merged against the parent + +Note that the DEFSYSTEM operator (used to create a "top-level" system) +does additional processing to set the filesystem location of the +top component in that system. This is detailed elsewhere + +The answer to the frequently asked question "how do I create a system +definition where all the source files have a .cl extension" is thus + +(defmethod source-file-type ((c cl-source-file) (s (eql (find-system 'my-sys)))) + "cl") + +**** properties (optional) + +Packaging systems often require information about files or systems +additional to that specified here. Programs that create vendor +packages out of asdf systems therefore have to create "placeholder" +information to satisfy these systems. Sometimes the creator of an +asdf system may know the additional information and wish to provide it +directly. + +(component-property component property-name) and associated setf method +will allow the programmatic update of this information. Property +names are compared as if by EQL, so use symbols or keywords or something + +** Subclasses of component + +*** 'source-file' + +A source file is any file that the system does not know how to +generate from other components of the system. + +(Note that this is not necessarily the same thing as "a file +containing data that is typically fed to a compiler". If a file is +generated by some pre-processor stage (e.g. a ".h" file from ".h.in" +by autoconf) then it is not, by this definition, a source file. +Conversely, we might have a graphic file that cannot be automatically +regenerated, or a proprietary shared library that we received as a +binary: these do count as source files for our purposes. All +suggestions for better terminology gratefully received) + +Subclasses of source-file exist for various languages. + +*** 'module', a collection of sub-components + +This has extra slots for + + :components - the components contained in this module + + :default-component-class - for child components which don't specify + their class explicitly + + :if-component-dep-fails takes one of the values :fail, :try-next, :ignore + (default value is :fail). The other values can be used for implementing + conditional compilation based on implementation *features*, where + it is not necessary for all files in a module to be compiled + +The default operation knows how to traverse a module, so most +operations will not need to provide methods specialised on modules. + +The module may be subclassed to represent components such as +foreign-language linked libraries or archive files. + +*** system, subclasses module + +A system is a module with a few extra attributes for documentation +purposes. In behaviour, it's usually identical. + +Users can create new classes for their systems: the default defsystem +macro takes a :classs keyword argument. + + +** operation + +An operation is instantiated whenever the user asks that an operation +be performed, inspected, or etc. The operation object contains +whatever state is relevant to this purpose (perhaps a list of visited +nodes, for example) but primarily is a nice thing to specialise +operation methods on and easier than having them all be EQL methods. + +There are no differences between standard operations and user-defined +operations, except that the user is respectfully requested to keep his +(or more importantly, our) package namespace clean + +*** invoking operations + +(operate operation system &rest keywords-args) + +keyword-args are passed to the make-instance call when creating the +operation: valid keywords depend on the initargs that the operation is +defined to accept. Note that dependencies may cause the operation to +invoke other operations on the system or its components: the new +operation will be created with the same initargs as the original one. + +oos is accepted as a synonym for operate + +*** standard operations + +**** feature-dependent-op + +This is not intended to be instantiated directly, but other operations +may inherit from it. An instance of feature-dependent-op will ignore +any components which have a `features' attribute, unless the feature +combination it designates is satisfied by *features* + +See the earlier explanation about the component features attribute for +more information + +**** compile-op &key proclamations + +If proclamations are supplied, they will be proclaimed. This is a +good place to specify optimization settings + +When creating a new component, you should provide methods for this. + +If you invoke compile-op as a user, component dependencies often mean +you may get some parts of the system loaded. This may not necessarily +be the whole thing, though; for your own sanity it is recommended that +you use load-op if you want to load a system. + +**** load-op &key proclamations + +The default methods for load-op compile files before loading them. +For parity, your own methods on new component types should probably do +so too + +**** load-source-op + +This method will load the source for the files in a module even if the +source files have been compiled. Systems sometimes have knotty +dependencies which require that sources are loaded before they can be +compiled. This is how you do that. + +If you are creating a component type, you need to implement this +operation - at least, where meaningful. + +**** test-system-version &key minimum + +Asks the system whether it satisfies a version requirement. + +The default method accepts a string, which is expected to contain of a +number of integers separated by #\. characters. The method is not +recursive. The component satisfies the version dependency if it has +the same major number as required and each of its sub-versions is +greater than or equal to the sub-version number required. + +(defun version-satisfies (x y) + (labels ((bigger (x y) + (cond ((not y) t) + ((not x) nil) + ((> (car x) (car y)) t) + ((= (car x) (car y)) + (bigger (cdr x) (cdr y)))))) + (and (= (car x) (car y)) + (or (not (cdr y)) (bigger (cdr x) (cdr y)))))) + +If that doesn't work for your system, you can override it. I hope +yoyu have as much fun writing the new method as #lisp did +reimplementing this one. + +*** Creating new operations + +subclass operation, provide methods for source-file for + +- output-files +- perform + The perform method must call output-files to find out where to + put its files, because the user is allowed to override output-files + for local policy +- explain +- operation-done-p, if you don't like the default one + +* Writing system definitions + +** System designators + +System designators are strings or symbols and behave just like +any other component names (including case conversion) + +** find-system + +Given a system designator, find-system finds an actual system - either +in memory, or in a file on the disk. It funcalls each element in the +*system-definition-search-functions* list, expecting a pathname to be +returned. + +If a suitable file exists, it is loaded if + +- there is no system of that name in memory, +- the file's last-modified time exceeds the last-modified time of the + system in memory + +When system definitions are loaded from .asd files, a new scratch +package is created for them to load into, so that different systems do +not overwrite each others operations. The user may also wish to (and +is recommended to) include defpackage and in-package forms in his +system definition files, however, so that they can be loaded manually +if need be. + +For convenience in the normal case, and for backward compatibility +with the spirit of mk-defsystem, the default contents of +*system-definition-search-functions* is a function called +sysdef-central-registry-search. This looks in each of the directories +given by evaluating members of *central-registry*, for a file whose +name is the name of the system and whose type is "asd". The first +such file is returned, whether or not it turns out to actually define +the appropriate system + + + +** Syntax + +Systems can always be constructed programmatically by instantiating +components using make-instance. For most purposes, however, it is +likely that people will want a static defystem form. + +asdf is based around the principle that components should not have to +know defsystem syntax. That is, the initargs that a component accepts +are not necessarily related to the defsystem form which creates it. + +A defsystem parser must implement a `defsystem' macro, which can +be named for compatibility with whatever other system definition +utility is being emulated. It should instantiate components in +accordance with whatever language it accepts, and register the topmost +component using REGISTER-SYSTEM + +*** Native syntax + +The native syntax is inspired by mk-defsystem, to the extent that it +should be possible to take most straightforward mk- system definitions +and run them with only light editing. For my convenience, this turns +out to be basically the same as the initargs to the various +components, with a few extensions for convenience + +system-definition := ( defsystem system-designator {option}* ) + +option := :components component-list + | :pathname pathname + | :default-component-class + | :perform method-form + | :explain method-form + | :output-files method-form + | :operation-done-p method-form + | :depends-on ( {simple-component-name}* ) + | :serial [ t | nil ] + | :in-order-to ( {dependency}+ ) + +component-list := ( {component-def}* ) + +component-def := simple-component-name + | ( component-type name {option}* ) + +component-type := :module | :file | :system | other-component-type + +dependency := (dependent-op {requirement}+) +requirement := (required-op {required-component}+) + | (feature feature-name) +dependent-op := operation-name +required-op := operation-name | feature + +For example + +(defsystem "foo" + :version "1.0" + :components ((:module "foo" :components ((:file "bar") (:file"baz") + (:file "quux")) + :perform (compile-op :after (op c) + (do-something c)) + :explain (compile-op :after (op c) + (explain-something c))) + (:file "blah"))) + + +The method-form tokens need explaining: esentially, + + :perform (compile-op :after (op c) + (do-something c)) + :explain (compile-op :after (op c) + (explain-something c))) +has the effect of + +(defmethod perform :after ((op compile-op) (c (eql ...))) + (do-something c)) +(defmethod explain :after ((op compile-op) (c (eql ...))) + (explain-something c)) + +where ... is the component in question; note that although this also +supports :before methods, they may not do what you want them to - a +:before method on perform ((op compile-op) (c (eql ...))) will run +after all the dependencies and sub-components have been processed, but +before the component in question has been compiled. + +**** Serial dependencies + +If the `:serial t' option is specified for a module, asdf will add +dependencies for each each child component, on all the children +textually preceding it. This is done as if by :depends-on + +:components ((:file "a") (:file "b") (:file "c")) +:serial t + +is equivalent to +:components ((:file "a") + (:file "b" :depends-on ("a")) + (:file "c" :depends-on ("a" "b"))) + + + +have all the + +**** Source location + +The :pathname option is optional in all cases for native-syntax +systems, and in the usual case the user is recommended not to supply +it. If it is not supplied for the top-level form, defsystem will set +it from + +- The host/device/directory parts of *load-truename*, if it is bound +- *default-pathname-defaults*, otherwise + +If a system is being redefined, the top-level pathname will be + +- changed, if explicitly supplied or obtained from *load-truename* +- changed if it had previously been set from *default-pathname-defaults* +- left as before, if it had previously been set from *load-truename* + and *load-truename* is not now bound + +These rules are designed so that (i) find-system will load a system +from disk and have its pathname default to the right place, (ii) +this pathname information will not be overwritten with +*default-pathname-defaults* (which could be somewhere else altogether) +if the user loads up the .asd file into his editor and +interactively re-evaluates that form + +* Error handling + +It is an error to define a system incorrectly: an implementation may +detect this and signal a generalised instance of +SYSTEM-DEFINITION-ERROR + +Operations may go wrong (for example when source files contain +errors). These are signalled using generalised instances of +OPERATION-ERROR + +* Compilation error and warning handling + +ASDF checks for warnings and errors when a file is compiled. The +variables *compile-file-warnings-behaviour* and +*compile-file-errors-behavior* controls the handling of any such +events. The valid values for these variables are :error, :warn, and +:ignore. + +---------------------------------------------------------- + TODO List +---------------------------------------------------------- + +* Outstanding spec questions, things to add + +** packaging systems + +*** manual page component? + +** style guide for .asd files + +You should either use keywords or be careful with the package that you +evaluate defsystem forms in. Otherwise (defsystem partition ...) +being read in the cl-user package will intern a cl-user:partition +symbol, which will then collide with the partition:partition symbol. + +Actually there's a hairier packages problem to think about too. +in-order-to is not a keyword: if you read defsystem forms in a package +that doesn't use ASDF, odd things might happen + +** extending defsystem with new options + +You might not want to write a whole parser, but just to add options to +the existing syntax. Reinstate parse-option or something akin + +** document all the error classes + +** what to do with compile-file failure + +Should check the primary return value from compile-file and see if +that gets us any closer to a sensible error handling strategy + +** foreign files + +lift unix-dso stuff from db-sockets + +** Diagnostics + +A "dry run" of an operation can be made with the following form: + +(traverse (make-instance '<operation-name>) + (find-system <system-name>) + 'explain) + +This uses unexported symbols. What would be a nice interface for this +functionality? + +** patches + +Sometimes one wants to + + +* missing bits in implementation + +** all of the above +** reuse the same scratch package whenever a system is reloaded from disk +** rules for system pathname defaulting are not yet implemented properly +** proclamations probably aren't +** when a system is reloaded with fewer components than it previously + had, odd things happen + +we should do something inventive when processing a defsystem form, +like take the list of kids and setf the slot to nil, then transfer +children from old to new list as they're found + +** traverse may become a normal function + +If you're defining methods on traverse, speak up. + + +** a lot of load-op methods can be rewritten to use input-files + +so should be. + + +** (stuff that might happen later) + +*** david lichteblau's patch for symlink resolution? + +*** Propagation of the :force option. ``I notice that + + (oos 'compile-op :araneida :force t) + +also forces compilation of every other system the :araneida system +depends on. This is rarely useful to me; usually, when I want to force +recompilation of something more than a single source file, I want to +recompile only one system. So it would be more useful to have +make-sub-operation refuse to propagate ":force t" to other systems, and +propagate only something like ":force :recursively". '' + +Ideally what we actually want is some kind of criterion that says +to which systems (and which operations) a :force switch will propagate. + +The problem is perhaps that 'force' is a pretty meaningless concept. +How obvious is it that "load :force t" should force _compilation_? +But we don't really have the right dependency setup for the user to +compile :force t and expect it to work (files will not be loaded after +compilation, so the compile environment for subsequent files will be +emptier than it needs to be) + +What does the user actually want to do when he forces? Usually, for +me, update for use with a new version of the lisp compiler. Perhaps +for recovery when he suspects that something has gone wrong. Or else +when he's changed compilation options or configuration in some way +that's not reflected in the dependency graph. + +Other possible interface: have a 'revert' function akin to 'make clean' + + (asdf:revert 'asdf:compile-op 'araneida) + +would delete any files produced by 'compile-op 'araneida. Of course, it +wouldn't be able to do much about stuff in the image itself. + +How would this work? + +traverse + +There's a difference between a module's dependencies (peers) and its +components (children). Perhaps there's a similar difference in +operations? For example, (load "use") depends-on (load "macros") is a +peer, whereas (load "use") depends-on (compile "use") is more of a +`subservient' relationship. |