Part II. Stratego/XT Development Manual (*)

Table of Contents

5. The Development of Stratego/XT
5.1. The Big Picture
5.1.1. Issue Tracker
5.1.2. Build Farm
5.1.3. Mailing Lists
5.2. Baseline development
5.2.1. What is a baseline and why do I need it?
5.2.2. Baseline restrictions
5.2.3. Baseline upgrading
5.2.4. Submitting your changes
5.3. Installation from Subversion
5.3.1. Requirements
5.3.2. How to checkout
5.3.3. How to install
5.3.4. Subversion Repoman
5.3.5. Autoconf Macros: aclocal, libtool, and autoconf
5.4. Contributing
5.4.1. Committing your changes

Chapter 5. The Development of Stratego/XT

This chapter will give you a brief introduction to the tools, services and conventions used in the development of Stratego/XT. If you want to contribute to the project, you should know this, but even if you do not intend to contribute, the information in this chapter may be of interest, as it explains how to get the latest development versions, and track the development in general.

5.1. The Big Picture

The development of Stratego/XT is completely in the open, as is fitting for an open-source project. This means that the services we use in our development -- the code repository, the issue tracker, the build farm and the mailing lists -- are all publicly readable. We will discuss each of these services in a bit more detail later on.

The development of Stratego/XT centers around the code repository, stored in Subversion at https://svn.strategoxt.org/repos/StrategoXT. This repository contains all the code that goes into Stratego/XT, with the exception of the ATerm library and the SDF2 tool bundle, which are available from CWI. From the source code repository, the project is rebuilt by the build farm whenever a commit is done to the Stratego/XT project. During a build, all the unit tests are also run. This improves stability, as we are able to find breaking changes shortly after they are introduced.

5.1.1. Issue Tracker

Our issue tracker is located at http://bugs.strategoxt.org/browse/STR. The tracker contains issues reported by our users, internally delegated tasks and requests for feature improvements, all placed into a release schedule for the coming releases. If you are looking for something to do, this is the place to start. Also, if you are interested in knowing where the project is headed, this is the best source of information.

5.1.2. Build Farm

The development of Stratego/XT relies heavily on continuous integration builds. The build farm will continously build Stratego/XT after each commit. Also, satellite projects, such as the BibTeX tools, Java-Front and even this manual, will be rebuilt whenever the code repository changes. A natural question to ask here is: should all the projects which use Stratego/XT also be rebuilt whenever Stratego/XT is changed? In the next section, we shall discuss this question.

The status of the build farm can be found at http://releases.strategoxt.org/. This page is updated every time the build farm completes a job. The result of a build (provided it suceeded) is downloadable, and you are free to download and test it. We do not, however, give any guarantees about the stability of these continuous release.

5.1.3. Mailing Lists

Disussions between the Stratego/XT developers, and also between Stratego users occur at our mailing lists, and . They are archived in the stratego archives and stratego-dev archives, respectively.

5.2. Baseline development

5.2.1. What is a baseline and why do I need it?

A baseline is a snapshot of the Stratego/XT source repository that only requires a C compiler to build itself. It is used for the traditional "bootstrapping" which is so common in the world of compilers.

The compiler for Stratego is implemented in Stratego. Also, many of the tools used to build Stratego/XT are written in Stratego and depend on themselves. Therefore, you will need a full Stratego/XT environment to build the Stratego/XT environment. This process is called bootstrapping. The baseline provides the initial Stratego/XT environment.

Unlike many bootstrapping compilers, the baseline provided for Stratego/XT is not a pre-built binary. It is a a source release of Stratego that requires a C compiler to build. We can therefore boostrap Stratego/XT on any of the supported platforms, with just one baseline.

Bootstrapping using the baseline is only necessary for developers of Stratego/XT itself. Normal users of Stratego/XT receive tarballs which do not need a baseline: they contain all generated and bootstrapped sources. See Section 5.3 for instructions on how to install Stratego/XT from Subversion.

The big advantage of a baseline is that the development process of Stratego/XT becomes cleaner and more robust. In the past, the bootstrapped sources were kept alongside the normal source code in the Subversion repository. This was very fragile and required great care on the part of the contributors. The baseline setup is much easier to use and a lot safer. We hope this will encourage contributions.

5.2.2. Baseline restrictions

Since we are using a baseline, you cannot use a new feature you have added to Stratego/XT in Stratego/XT itself until this feature has made it to the baseline. Remember that Stratego/XT must be built by the baseline.

This might sound difficult, but in practice it is easy to ensure that you don't do this. You should use the baseline we provide to compile Stratego/XT from Subversion and you should not change the configuration of Stratego/XT with this baseline after installation. The baseline now ensures that you are not using new features that you've added to Stratego/XT: it will just not compile.

5.2.3. Baseline upgrading

Baselines are taken always from the buildfarm. The baselines are thoroughly checked there. If the build job for a particular revision of Strateo/XT does not succeed, this revision cannot be promoted to a new baseline. This avoids us getting into nasty bootstrap problems because of uncaught build failures. In general, the release of a new baseline is careful process. Baselines will never be created in an ad-hoc way on machines of developers.

Picking a successful build from the buildfarm as a new baseline is done by hand. If you have added a feature that would you like to have in the baseline as soon as possible, then please request a baseline update at or the IRC channel #stratego on freenode.

5.2.4. Submitting your changes

We are fond of patches submitted by users. You don't have to fix a big issue: small fixes in error reporting and documentation of tools are very valuable as well. Please contribute, or at least report, if anything annoys you!

5.3. Installation from Subversion

In ??? in the Stratego/XT Tutorial, we explain how to download, compile and install stable versions of the Stratego/XT framework. In this section we will see how to download the bleeding edge sources from the Subversion repository. Downloading the development sources is only useful if you want to modify the sources of Stratego/XT and might want to commit your changes.

5.3.1. Requirements

Installation from Subversion requires some packages from the GNU build system. You need:

  • Autoconf 2.58 or newer (2.57 will not do)

  • Automake 1.7.9 or newer. Very recent versions (1.9.x) are recommended and required if you want to create a distribution

  • Libtool 1.5 or newer

  • pkg-config 0.15 or newer

Unfortunately, many Linux distributions contain old versions of these packages, so please check the versions that are installed on your system. Also, check that libtool is installed at the same location (prefix) as Automake, otherwise aclocal will not be able to find the Autoconf macros of Libtool (libtool.m4)

You now need a baseline installation of Stratego/XT to install Stratego/XT from Subversion. The latest baseline is always available at the ftp server of Stratego. The need for a baseline is explained in Section 5.2. For the installation of the baseline, you can follow the ordinary installation instructions for end-users (Chapter 2).

5.3.2. How to checkout

The Stratego/XT sources are kept in a source repository managed by Subversion. Before checking out, you need to install recent Subversion release. After you have done so, you may do a checkout using the following command:

  $ svn checkout https://svn.strategoxt.org/repos/StrategoXT/strategoxt/trunk/

Note that your Subversion client must have SSL support enabled. Install Subversion --with-ssl. This is the default installation for most distributions.

5.3.3. How to install

To bring the source tree into a state where you can follow the end-user installation instructions you must run the bootstrap script. (see ???).

  $ ./bootstrap

Note that on many distributions, the older versions of automake and autoconf are invoked by default. You can check which version is invoked by issuing automake --version and autoconf --version. Unfortunately, the mechanism for switching to a newer version is distribution-specific. We automatically figure this out for some distributions. Look inside the boostrap for details, adjust and provide us with a patch, if your distribution is not supported.

Now you can proceed with the installation instructions for a Stratego/XT distribution, with one exception: you must configure Stratego/XT with your baseline installation: --with-strategoxt=$baselineprefix. Section 5.2 explains why and how we use a baseline.

5.3.4. Subversion Repoman

The repoman gives an overview of possible ways to view the sources the StrategoXT repository:

https://svn.strategoxt.org/repoman/info/StrategoXT

5.3.5. Autoconf Macros: aclocal, libtool, and autoconf

GNU Autoconf needs definitions of macros that are used in configure.ac files. GNU Automake contains the tool aclocal, which installs the required macros. aclocal gets these macros from a directory with m4 files. aclocal --print-ac-dir reports this directory. Your libtool installation must install the libtool.m4 file in the ac-dir. If this is not the case, you will see a non-fatal error like this:

configure.in:9: error: possibly undefined macro: AC_PROG_LIBTOOL
       If this token and others are legitimate, please use m4_pattern_allow.
       See the Autoconf documentation.

Later on in the build proces a missing =Makefile.in= will be reported, after which the build terminates.

Usually the file libtool.m4 is installed in the right directory if you install Libtool and Automake at the same prefix. However, when using recent, non-standard, ports of these packages on a FreeBSD systems (and probably also some RPM based systems), this does not happen. If libtool.m4 is not in the right directory for some reason, then you must copy the libtool.m4 file by hand into the ac-dir.

5.4. Contributing

5.4.1. Committing your changes

Although it is time-consuming, it is strongly advised to run a make check on the complete source tree before committing anything.

Before committing you should review the files that you have modified. After a make clean and make bootclean the svn status will show these files. We have configured the source tree not to show files in this status overview that are not important. Please use svn diff to verify that your changes are as intended. Further, please try to keep the granularity of the commits to one fix per commit. Specifically, do not commit many, unrelated fixes in one go.

Please explain your changes in the ChangeLog of the subpackages. Also, use this changelog as your Subversion commit message.