Chapter 29. Building and Deploying Stratego Programs

Table of Contents

29.1. Building stand-alone artifacts
29.1.1. Static linking
29.2. Setting up your Project
29.3. Building Stand-alone Stratego Applications
29.4. Building Parse Tables, Tree Grammars and Stratego Signatures
29.5. Building Your Own Stratego Library
29.5.1. Compiling the Library
29.5.2. Using Your Library in Stratego Programs
29.6. Package Config Support
29.7. RPM Support
29.8. Summary

There are two typical scenarios for building Stratego programs. The first, and simplest, is to execute the compiler by hand on the command line in order to build one artifact (a program or a library). The second is to set up a full build system is based on the GNU Autotools. Both scenarios will be covered in this chapter.

29.1. Building stand-alone artifacts

Here we describe how to use the Stratego compiler for small projects and for compiling one-off examples on the command-line. We recommend that you use Autotools for larger projects, i.e. when you need to build multiple artifacts (see the next sections).

Invoking the compiler on simple programs which only depend on the Stratego library is straightforward:

$ strc -i myprog.str 

This produces the executable file myprog. When your program depends on other Stratego packages (libraries), you need to provide the compiler with proper include paths (for finding the module definitions) and linking arguments (for linking the libraries with the final executable). For convenience, you should define an alias called strcflags as follows:

$ alias strcflags="pkg-config --variable=strcflags "

By calling strcflags with the the name of a specific package, e.g. java-front, all necessary include paths and library arguments will be provided for you. This gives rise to the following idiom:

$ strc -i myprog.str $(strcflags dryad java-front)

Note that providing several arguments (packages) to strcflags is allowed.

29.1.1. Static linking

By default, the Stratego compiler will dynamically link all libraries. To enable static linking instead, you must add the command line options -Xlinker -all-static:

$ strc -i myprog.str -Xlinker -all-static

This ensures that the myprog executable is statically linked (and therefore has no external dependencies).

29.2. Setting up your Project

Setting up a build system for Stratego involves the Autotool programs automake, autoconf and libtool. In addition, Stratego provides a new tool called autoxt. If you are familiar with the Autotools, setting up a project for Stratego should be rather easy. If this is unfamiliar ground to you, don't fear. We will walk through it slowly in the next sections, but a full treatise is beyond the scope of this tutorial.

After creating your project directory, let's call it ogetarts, the first thing you should do is populate it with the basic build system files, namely configure.ac, bootstrap, Makefile.am. Additionally, you may want to add ChangeLog, AUTHORS, NEWS and README, but these are not essential. If you want to support the creation of RPMs, then you need to create a file ogetarts.spec.in.

For a normal Stratego project, with a syntax, some stand-alone tools, and a library, we suggest the project layout given below (directories end in /). We will discuss the all the components of this hierarchy in turn.

ogetarts/
  bootstrap
  configure.ac
  Makefile.am
  syn/
      Makefile.am
  lib/
      Makefile.am
  tools/
        Makefile.am

The build system is kickstarted by the bootstrap script. This script is responsible for generating a configure script. When run by the user of your package, the configure script will autodetect the Stratego compiler and other tools required to build your project. The concept of generation is very central to autotool-based build systems. The configure script is generated from a configure.ac declaration by the autoreconf tool. The Makefiles are generated from Makefile.in files by the configure script, and Makefile.in files are generated from Makefile.am files by automake. Simple, huh? Generally, the idea is that complicated scripts and makefiles can be generated from high-level declarations using tools. Let's start with the bootstrap script.

#! /bin/sh

autoxt || exit 1
autoreconf -ifv || exit 1

The bootstrap script should be an sh (or bash) shell script that takes care of running autoxt and autoreconf, as shown above. Note that we rely on reasonably recent versions of autoconf and automake.

Assume we are in a palindromic mood and want to name our project Ogetarts. The following file will then provide a reasonable starting point for the configure.ac file.

AC_PREREQ([2.58])
AC_INIT([ogetarts],[0.1],[ogetarts-bugs@ogetarts.org])
AM_INIT_AUTOMAKE([1.7.2 -Wall -Wno-portability foreign])

# set the prefix immediately to the default prefix
test "x$prefix" = xNONE && prefix=$ac_default_prefix

XT_SETUP
XT_USE_XT_PACKAGES

AC_PROG_CC
AC_PROG_LIBTOOL

### OUTPUT #############################
AC_CONFIG_FILES([
  Makefile
  syn/Makefile
  lib/Makefile
  tools/Makefile
])
AC_OUTPUT

Most of this is standard boilerplate. The foreign option specified in the arguments of AM_INIT_AUTOMAKE tells automake not the check that the package conforms to GNU package standards, which requires files such as ChangeLog, AUTHORS, COPYING, NEWS and README. For this small example, we do not want create all these files. You can leave out the foreign option if you want to make a complete GNU package.

The important line is XT_USE_XT_PACKAGES which is a macro invocation that will extend to shell script code that looks for the ATerm library, the SDF tools and Stratego/XT, respectively. These macros are provided by the autoxt tool, via a macro file called autoxt.m4. It provides the following macros.

XT_SETUP

Sets up a Stratego package by setting standard flags of the C compiler and linker for Stratego programs.

XT_USE_XT_PACKAGES

Adds configuration options to configure the package with the location of the ATerm library, SDF2 Bundle and Stratego/XT.

XT_PRE_RELEASE

Adds the suffix pre${SVN_REVISION} to the PACKAGE_VERSION and VERSION variables. This is a naming convention for unstable packages that we are using in our release management system. If you are not building your package in our buildfarm, then you do not need to invoke this macro.

XT_DISABLE_XTC_REGISTER

Disables the creation of an XTC repository. By default all programs and files are registered in an XTC repository.

XT_USE_BOOTSTRAP_XT_PACKAGES

Similar to XT_USE_XT_PACKAGES, this macro adds configuration options to configure the package with the location of the ATerm library, SDF and Stratego/XT. However, the macro will only check the existence of these packages if the option --enable-bootstrap is given to the configure script. In other case, it will only look for the Aterm Library and Stratego Libraries. Also, XTC registration is disabled. This macro is used for packages that need to be very portable, including native Microsoft Windows.

XT_SVN_REVISION

Determines the SVN revision and makes this number available to Stratego programs.

At the end of the configure.ac above, the invocation of the AC_CONFIG_FILES macro lists other important files of the build system, particularly the Makefiles. We must provide these, but remember that these are generated from .in files which in turn come from .am files. Hence, we need to provide some Makefile.am files. The Makefile.am for the root of the project should look like:

include $(top_srcdir)/Makefile.xt

SUBDIRS           = syn lib tools
BOOTCLEAN_SUBDIRS = $(SUBDIRS)
DIST_SUBDIRS      = $(SUBDIRS)
EXTRA_DIST        =

ACLOCAL_AMFLAGS = -I .

Again, most of this is boilerplate. The important point here is that SUBDIRS = syn lib tools will eventually result in rules that tell make to delve into these directories. We will explain below how the Makefile.ams for each of the source directories should look like. For now, you can just create empty Makefile.am files in the sub-directories syn/, lib/, and tools/. This allows you to bootstrap and configure the package:

$ mkdir syn lib tools
$ touch syn/Makefile.am lib/Makefile.am tools/Makefile.am
$ chmod u+x bootstrap
$ ./bootstrap
$ ./configure
$ make

The content of the empty Makefile.am files depends on whether you are building a parser, stand-alone Stratego programs, or a Stratego library. We will discuss each variant separately, but you are of course free to mix several of these in your project, like we do in this project: In lib lives the library parts of Ogetarts, in syn a parser is generated, and in tools we place the command-line programs.

29.3. Building Stand-alone Stratego Applications

In Chapter 11, we showed how to compile stand-alone Stratego programs using the strc compiler. This process is automated by the build system, provided you supply a suitable Makefile.am. Take the one provided below as a starting point.

include $(top_srcdir)/Makefile.xt
include $(wildcard *.dep)

bin_PROGRAMS = ogetarts
ogetarts_LDADD = $(STRATEGO_LIB_LIBS) $(STRATEGO_RUNTIME_LIBS) $(ATERM_LIBS)

STRINCLUDES    = -I $(top_srcdir)/lib -I $(top_srcdir)/syn
STRCFLAGS      = --main io-$*

EXTRA_DIST = $(wildcard *.str) $(wildcard *.meta)
CLEANFILES = $(wildcard *.c) $(wildcard *.dep)

This file should be placed in tools/. The following list explains the various parts occurring in the file.

include $(top_srcdir)/Makefile.xt

Includes the various Stratego/XT make rules in this Makefile, for example for the compilation of Stratego programs and parse tables.

include $(wildcard *.dep)

The Stratego compiler generates .dep files which contain information about module dependencies. When these .dep files are included, a rebuild is forced when a dependent file changes.

bin_PROGRAMS

A list of stand-alone Stratego programs that should be compiled and installed in the directory $prefix/bin. For each program, a corresponding .str file must exist. In this case, the most trivial ogetarts.str module could look like:

module ogetarts
imports libstratego-lib
strategies
  io-ogetarts = io-wrap(id)
program_LDADD

Stratego programs reuse various libraries, such as the ATerm library, the Stratego runtime and the Stratego library. This declaration tells the build system to link the program ogetarts with these libraries. If you use more libraries, such as the library libstratego-sglr for parsing, then you can add $(STRATEGO_SGLR_LIBS). All libraries follow this naming convention, for example the variable $(STRATEGO_XTC_LIBS) is used for the library libstratego-xtc. If you also have your own library in this package, then you can add $(top_builddir)/lib/libogetarts.la.

STRCFLAGS

Contains compiler flags passed to the Stratego compiler. A typical flags is --main. In this case, the declaration tells the Stratego compiler that the main stratego of a program foo is io-foo (instead of the default main). It is recommended that include directories are passed via the STRINCLUDES variable.

STRINCLUDES

A list of additional includes necessary for a succesful compilation, of the form -I <dir>. They will be passed unchanged to the strc compiler. The Stratego compiler will then look in these directories for Stratego modules.

EXTRA_DIST

Specifies which auxilary files have to be included in the distribution, when doing a make dist. In this case, we want to distribute all the Stratego modules and their .meta (which do not always exist)

CLEANFILES

Files to be deletes these files when the make clean command is issued. In this case we instruct automake to remove the .c and .dep files that are generated by the Stratego compiler.

BOOTCLEANFILES

Files to be deleted when make bootclean is issued. In addtion, the files specified in CLEANFILES will also be deleted.

As we can see from the list above, the bin_PROGRAMS is a list of the stand-alone programs that should be compiled in this directory. For each program, a corresponding .str must exist, in this case, a ogetarts.str file. For each program the Stratego compiler will be passed the STRINCLUDES and STRCFLAGS variables. The program_LDADD variable is used to add additional native libraries that should be linked as part of the C compilation process. STRINCLUDES and STRCFLAGS were explained above.

autoxt installs Makefile.xt, a collection of Automake rules for compiling Stratego programs and applying other XT tools, such as signature generation. Using this makefile, a makefile reduces to a declaration of programs to be compiled. The makefile automatically takes care of distributing the generated C code. The specification will only be compiled when it is newer than the C code.

29.4. Building Parse Tables, Tree Grammars and Stratego Signatures

In Chapter 6 we introduced you to the Syntax Definition Formalism SDF for specifying grammars, and showed you how to use pack-sdf and sdf2table to construct parse tables out of these definitions. The grammar can also be used to derive Stratego signatures and regular tree grammars. Not surprisingly, the build system is equipped to do this for you. Consider the Makefile.am provided below.

include $(top_srcdir)/Makefile.xt
include $(wildcard *.dep)

DEF_TBL = Ogetarts.def Ogetarts.tbl
RTG_SIG = Ogetarts.rtg Ogetarts.str

sdfdata_DATA = $(DEF_TBL) $(wildcard *.sdf)
pkgdata_DATA = $(RTG_SIG)

EXTRA_DIST   = $(DEF_TBL) $(RTG_SIG) $(wildcard *.sdf)
CLEANFILES   = $(DEF_TBL) $(RTG_SIG)

SDFINCLUDES   =
SDF2RTG_FLAGS = -m $*
PGEN_FLAGS    = -m $*

This file should be placed in syn/. The following list explains the various parts occurring in the file.

sdfdata_DATA

The important declaration in this file is sdfdata_DATA. They files listed for this variable will be installed as part of a make install into the directory $prefix/share/sdf/$packagename. For convenience, we have defines the .def and .tbl files, in a separate variable DEF_TBL. The build system knows how to generate .def files from .sdf using pack-sdf, and .tbl files from .def files using sdf2table. Note that there must be an SDF module Ogetarts.sdf, which is the main module of the syntax definition. For example:

module Ogetarts
hiddens
  context-free start-symbols Expr
exports
  context-free syntax
    IntConst -> Expr {cons("Int")}

  context-free priorities
      Expr "*" Expr -> Expr {left, cons("Mul")}
    > Expr "+" Expr -> Expr {left, cons("Plus")}

  lexical syntax
    [0-9]+ -> IntConst
    [\t\n\r\ ] -> LAYOUT
pkgdata_DATA

This variable defines a list of files that will be placed in prefix/share/package-name. This typically includes regular tree grammars, Stratego signatures and Stratego source code for libraries. In this makefile, we use pkgdata_DATA to tell the build system to build a Stratego signature (Ogetarts.str) and a regular tree grammar (Ogetarts.rtg) from the Ogetarts.def. Including signatures and source code with your program is useful when you want other projects to extend and compile against yours.

EXTRA_DIST

Similar to the makfile for building Stratego programs, we use EXTRA_DIST to define the files to distribute. In this case, we also distribute the derived .def, .tbl, .rtg, and .str, which is used to avoid a dependency on the full Stratego/XT. If this is not required, then we can leave these files out of the distribution.

CLEANFILES

In CLEANFILES we specify that we want make to remove the generated files when running a make clean.

PGEN_FLAGS

This variable is used to pass flags to the parsetable generator, sdf2table. The definition in this makefile defines the main module to be $*, which is the basename of the .def file that is used as the input to the parser generator. This is typical for most makefiles, since the default main module is Main.

SDF2RTG_FLAGS

Similar to PGEN_FLAGS, this variable is used to pass flags to the tool sdf2rtg, which generates a regular tree grammar for an SDF syntax definition. Again, we define the main module of the syntax definition to be the basename of the file.

SDFINCLUDES

Similar to SDFINCLUDES, you can define directories to search for SDF modules using the SDFINCLUDES variable. In addition to the option -I dir you can also include modules from an SDF definition: -Idef file.def.

29.5. Building Your Own Stratego Library

Stratego allows you to freely organize the Stratego modules of your project, but we recommend to have a separate library in the directory lib/. Each .str file placed inside this directory becomes a module for your library. For sufficiently large projects, it is recommended that you further organize your modules into subdirectories inside lib/. Each such subdirectory becomes a package.

Importing modules and packages in Stratego is done using the imports statement. Using imports, you specify which module from which package to import. See Chapter 11 for an introduction to modules. There is a direct mapping between the directory name and the package name, and also between file and module names. Consider the following main module lib/ogetarts.str of your library, which imports all the Stratego modules that constitute the library:

module ogetarts
imports
  ogetarts/main
  ogetarts/front/parse

The import declaration ogetarts/main states that we want to import the module main from the package ogetarts. This tells the Stratego compiler to look for the file main.str inside the ogetarts/ directory. The line ogetarts/front/parse will import the file parse.str in the ogetarts/extensions package. On disk, we will have the following layout:

lib/
    ogetarts.str
    ogetarts/
             main.str
             front/
                   parse.str

In this example, we will assume that the module ogetarts/main provides a simple evaluator for arithmetic expressions:

module ogetarts/main
imports libstratego-lib Ogetarts
strategies

  ogetarts-eval = bottomup(try(Eval))

rules

  Eval : Plus(Int(i), Int(j)) -> Int(<addS> (i, j))
  Eval : Mul(Int(i), Int(j))  -> Int(<mulS> (i, j))

The module ogetarts/front/parse provides a strategy for parsing ogetarts expressions:

module ogetarts/front/parse
imports libstratego-sglr libstratego-lib
strategies

  parse-ogetarts-stream =
    where(tbl := <import-term(Ogetarts.tbl); open-parse-table>)
    ; finally(
        parse-stream(strsglr-report-parse-error | tbl)
      , <close-parse-table> tbl
      )

So how does the compiler know where to search for the packages (directories)? This is specified by the -I option to strc. In our case, we did already specify lib/ as the basepath for our library in the section on compiling Stratego programs. Thus, all module and package references are looked up from there. For programs using several libraries installed at different locations, multiple base directories should be specified, each with the -I option.

29.5.1. Compiling the Library

In principle, it is possible to import full source of your library in your Stratego programs. In that case, there is no need to compile the library separately: it will be compiled by the Stratego compiler every time the library is included in a Stratego program. The compiler will act as a whole-program compiler and compile all the source code from scratch, including your library sources. This is wasteful, since the library is recompiled needlessly. To avoid this, the build system can be told to compile the library separately to a native library, e.g. a .so file. The creation of the native library is done using libtool, which takes care of creating both static and shared libraries. On most platforms, the linker prefers shared libraries over static libraries, given the choice. This means that linking to your Stratego library will be done dynamically, unless you or your library user take steps to enable static linking.

The code below is what goes into your Makefile.am for you library:

include $(top_srcdir)/Makefile.xt
include $(wildcard *.dep)

lib_LTLIBRARIES = libogetarts.la 
pkgdata_DATA    = libogetarts.rtree
EXTRA_DIST      = $(ogetartssrc) libogetarts.rtree
CLEANFILES      = libogetarts.c libogetarts.rtree

libogetarts_la_SOURCES  = libogetarts.c
libogetarts_la_LDFLAGS  = -avoid-version -no-undefined
libogetarts_la_CPPFLAGS = \
  $(STRATEGO_LIB_CFLAGS) $(STRATEGO_RUNTIME_CFLAGS) $(ATERM_CFLAGS)
libogetarts_la_LIBADD   = \
  $(STRATEGO_SGLR_LIBS) $(STRATEGO_LIB_LIBS) $(STRATEGO_RUNTIME_LIBS) $(ATERM_LIBS)

ogetartssrc = \
  ogetarts.str \
  $(wildcard $(srcdir)/ogetarts/*.str) \
  $(wildcard $(srcdir)/ogetarts/front/*.str)

STRINCLUDES = -I $(top_srcdir)/syn

libogetarts.c libogetarts.rtree : $(ogetartssrc)
    @$(STRC)/bin/strc -c --library -i ogetarts.str -o libogetarts.rtree $(STRINCLUDES)
    rm libogetarts.str
lib_LTLIBRARIES

pkgdata_DATA

libogetarts_LDFLAGS

libogetarts_CPPLAGS

libogetarts_LIBADD

Necessary for some platforms. Note that we add STRATEGO_SGLR_LIBS, since our module is importing libstratego-sglr.

STRINCLUDES

For finding the signature and parsetable

libogetarts.c : ...

Warning: use tab.

If you prefer static over dynamic libraries, you can enforce static linking by passing the -static option to the configure of your package via the LDFLAGS variable:

$ ./configure ... LDFLAGS=-static

29.5.2. Using Your Library in Stratego Programs

Add a program to the tools directory

module ogetarts-eval
imports
  libstratego-lib
  libogetarts

strategies

  io-ogetarts-eval =
    io-stream-wrap(
      ?(<id>, fout)
      ; parse-ogetarts-stream
      ; ogetarts-eval
      ; ?Int(<id>)
      ; <fputs> (<id>, fout); <fputs> ("\n", fout)
    )

Add to tools/Makefile.am:

bin_PROGRAMS = ogetarts-eval

ogetarts_eval_LDADD = \
  $(top_builddir)/lib/libogetarts.la \
  $(STRATEGO_LIB_LIBS) $(STRATEGO_RUNTIME_LIBS) $(ATERM_LIBS)

Invoke the eval tool:

$ echo "1+2" | ./tools/ogetarts-eval

29.6. Package Config Support

How to add a pkg-config file to the package.

29.7. RPM Support

Let's finish up the root build system first, closing with the ogetarts.spec.in file.

Summary: ogetarts
Name: @PACKAGE_TARNAME@
Version: @PACKAGE_VERSION@
Release: 1
License: LGPL
Group: Development/Tools/Ogetarts
URL: http://www.ogetarts.org/Ogetarts
Source: @PACKAGE_TARNAME@-@PACKAGE_VERSION@.tar.gz
BuildRoot: %{_tmppath}/%{name}-@PACKAGE_VERSION@-buildroot
Requires: aterm >= 2.5
Requires: sdf2-bundle >= 2.4
Requires: strategoxt >= 0.17
Provides: %{name} = %{version}

%description

%prep
%setup -q

%build
CFLAGS="-D__NO_CTYPE" ./configure --prefix=%{_prefix} --with-strategoxt=%{_prefix} \
       --with-aterm=%{_prefix} --with-sdf=%{_prefix}
make

%install
rm -rf $RPM_BUILD_ROOT
make DESTDIR=$RPM_BUILD_ROOT install

%clean
rm -rf $RPM_BUILD_ROOT

%files
%defattr(-,root,root,-)
%{_bindir}
%{_libexecdir}
%{_datadir}
%doc

%changelog

This file is not strictly necessary. It's a so-called .spec-file, which is a package descriptor for rpm-based distributions. When you provide it, the Stratego/XT build system can automatically make a .rpm package file for your project.

29.8. Summary

In this chapter we learned how to organize Stratego/XT projects, and how to set up the Stratego/XT build system. We saw how the build system, which is based on automake and autoconf, is used to generate many of the artifacts from the syntax definition, such as parse tables and signatures, in the way we discussed in Part II . We also learned how to make stand-alone programs, libraries and XT components from Stratego source code, and how modules and packages relate to files and directories.

More detail can be found by looking at the manual pages for the build tools, such as autoxt, strc, sdf2table, sdf2rtg and xtc. The detailed examples from the Stratego/XT Examples tutorial contain working build systems that can serve as a starting point.