Part IV. The Stratego Library

The Stratego Library was designed with one goal in mind: it should contain be a good collection of strategies, rules and data types for manipulating programs. In the previous part of this tutorial, we have already introduced you some of the specific features in the library for doing program manipulation. However, the library also contains abstract data types which are found in almost any library, such as lists, strings, hashtables, sets, file and console I/O, directory manipulation and more. In this chapter, we aim to complete your basic Stratego education by introducing you to how these bread-and-butter data types have been implemented for Stratego.

Online Library Documentation

Stratego and its library is a work in progress. New material is added to the library on a weekly basis. If you want to follow the progress, you should consult the latest version of the library documentation.

Beware that the online documentation will display strategies on the form apply-and-fail(Strategy s, ATerm name, ATerm in-term, ATerm out), whereas we adopt the more conventional format in this manual: apply-and-fail(s | name, in-term, out)

1. Anatomy of the Stratego Library

The organization of the Stratego library is hierarchical. At the coarsest level of organization, it is divided into packages, whose named as on a path-like form, e.g. collection/list. Each package in turn consists of one or several modules. A module is a leaf in the hierarchy. It maps to one Stratego (.str) file, and contains definitions for strategies, rules, constructors and overlays. The available packages in the library is listed below.

collection/hash-table
collection/list
collection/set
collection/tuple
lang
strategy
strategy/general
strategy/pack
strategy/traversal
system/io
system/posix
term
util
util/config

As an example, the collection/list package consists of the modules common, cons, filter, index, integer, lookup, set, sort, zip. Inside the sort module, we find the qsort strategy, for sorting lists.

In the remainder of this part of the tutorial, we will present the most important parts of the library, and show their typical usage patterns and idioms. If anything seems unclear, you are encouraged to consult the online documentation for further details.

Table of Contents

21. Arithmetic Operations
21.1. Basic Operations
21.2. Number comparisons
21.3. Other Operations
21.4. Random Numbers
21.5. Summary
22. Lists
22.1. Making heads and tails of it
22.2. Sorting
22.3. Associative Lists
22.4. Pairing Lists
22.5. Lightweight Sets
22.6. Transforming Lists
22.7. Folding from the Left and Right
22.8. Summary
23. Strings
23.1. Basic String Operations
23.2. Sorting Strings
23.3. Strings and Terms
23.4. Strings and Numbers
24. Hashtables and Sets
24.1. Hashtables
24.2. Indexed Sets
25. I/O
25.1. Console I/O
25.2. Path and Directory Operations
25.3. File and Text I/O
25.4. Term I/O
25.5. Logging
26. Command-line Options
26.1. Parsing Command-line Options
26.2. Adding Custom Options
26.3. Setting Description and About
26.4. I/O-less Programs
27. Unit Testing with SUnit
27.1. Setting up a test suite
27.2. Compare expected and actual output
27.3. Check for failure
27.4. Check arbitrary conditions on output
27.5. Unit testing with XTC
28. Transformation Tool Composition with XTC
28.1. Basic Mechanisms of XTC
28.1.1. Registration of Programs and Data
28.1.2. Importing other Repositories
28.1.3. Searching Repositories
28.2. Composing Tools in Stratego
28.2.1. Making an XT component
28.2.2. Invoking XT components
28.3. Summary
29. Building and Deploying Stratego Programs
29.1. Building stand-alone artifacts
29.1.1. Static linking
29.2. Setting up your Project
29.3. Building Stand-alone Stratego Applications
29.4. Building Parse Tables, Tree Grammars and Stratego Signatures
29.5. Building Your Own Stratego Library
29.5.1. Compiling the Library
29.5.2. Using Your Library in Stratego Programs
29.6. Package Config Support
29.7. RPM Support
29.8. Summary
30. Debugging Techniques for Stratego/XT
30.1. Debugging Stratego
30.1.1. Writing readable code
30.1.2. Debugging Stratego code
30.1.3. Common Pitfalls
30.2. Debugging XT compositions
30.3. Debugging SDF definitions

Chapter 21. Arithmetic Operations

In this chapter we introduce strategies for working with numbers. The Stratego runtime provides two kinds of numbers: real numbers and integers. They are both terms, but cannot be used interchangeably. The library strategies described in this chapter also maintain the distinction between real numbers and integers, but many may also be applied to strings which contain numbers.

21.1. Basic Operations

Stratego does not have the normal mathematical syntax for arithmetic operators, such as +, -, / and *. These operators are used for other purposes. Instead, the library provides the operators as the strategies, namely add, subt, div and mul. Further, there is convenience strategy for integer increment, inc and decrement, dec.

While the Stratego language operates exclusively on terms, there are different kinds of primitive terms. The runtime maintains a distinction between real numbers and integer numbers. The library mirrors this distinction by providing a family of strategies for arithmetic operations. Arithmetic strategies which work on real numbers end in an r, e.g. addr, and strategies working on integers end in an i, e.g. subti. For each arithmetic operator, there is also a type-promoting variant, e.g. mul, which will type-promote from integer to real, when necessary. Finally, there are convenience strategies for working on strings containing numbers. For each arithmetic operation, there is a string variant, e.g divS.

The full set of arithmetic operations in Stratego:

add,  addr,  addi,  addS
div,  divr,  divi,  divS
mul,  mulr,  muli,  mulS
subt, subtr, subti, subtS

Using these strategies is straightforward.

stratego> <addr> (1.5, 1.5)
3.000000000000000e+00
stratego> <subti> (5, 2)
3
stratego> <mul> (1.5, 2)
3.000000000000000e+00
stratego> <inc> 2
3

As we can see, the mul operator can be applied to a pair which consists of different terms (real and integer). In this case, type promotion from integer to real happens automatically.

Working on Strings.  The string variants, e.g. addS and divS work on strings containing integers. The result in strings containing integers.

     
stratego> <addS> ("40", "2")
"42"
stratego> <divS> ("9", "3")
"3"

21.2. Number comparisons

The strategies found in the library for comparing two numbers correspond to the usual mathematical operators for less-than (lt), less-than-equal (leq), equal (eq), greater-than (gt), greather-than-equal (geq). As with the arithmetic strategies, each of these operators comes in an integer variant, suffixed with i, a real variant (suffixed by r), a string variant (suffixed by S) and a type promoting variant without suffix. The full matrix of comparison functions thus looks like:

lt,  ltr,  lti,  ltS
gt,  gtr,  gti,  gtS
leq, leqr, leqi, leqS
geq, geqr, geqi, geqS

A few examples:

stratego> <lt> (1.0, 2)
(1.000000000000000e+00,2)
stratego> <ltS> ("1", "2")
("1", "2")
stratego> <geqS> ("2", "2")
("2","2")
stratego> <gtr> (0.9, 1.0)
command failed

The maximum and minimum of a two-element tuple of numbers can be found with the max and min strategies, respectively. These do not distinguish between real and integers. However, they do distinguish between numbers and strings; maxS and minS are applicable to strings.

stratego> <max> (0.9, 1.0)
1.0
stratego> <min> (99, 22)
22
stratego> <minS> ("99", "22")
"22"

Some other properties of numbers, such as whether a number is even, negative or positive, can be be tested with the strategies even, neg and pos, respectively.

21.3. Other Operations

The modulus (remainder) of dividing an integer by another is provided by the mod strategy. gcd gives the greatest common divisor of two numbers. Both mod and gcd work on a two-element tuple of integers. The log2 strategy can be used to find the binary logarithm of a number. It will only succeed if the provided number is an integer and that number has an integer binary logarithm.

stratego> <mod> (412,123)
43
stratego> <gcd> (412,123)
1
stratego> <log2> 16
4

21.4. Random Numbers

The library provides a strategy for generating random numbers, called next-random. The algorithm powering this random generator requires an initial "seed" to be provided. This seed is just a first random number. You can pick any integer you want, but it's advisable to pick a different seed on each program execution. A popular choice (though not actually random) is the number of seconds since epoch, provided by time. The seed is initialized by the set-random-seed strategy. The following code shows the normal idiom for getting a random number in Stratego:

stratego> time ; set-random-seed
[]
stratego> next-random     
1543988747

The random number generator needs only be initialized with a seed once for every program invocation.

21.5. Summary

In this chapter, we saw that Stratego is different from many other languages in that it does not provide the normal arithmetic operators. We saw that instead, strategies such as add and mul are used to add and multiply numbers. We also saw which strategies to use for comparing numbers and generating random numbers.

The module term/integer contains strategies for working with numbers. Refer to the library reference documentation for more information.

Chapter 22. Lists

This chapter will introduce you to the basic strategies for working with lists. The strategies provide functionality for composing and decomposing, sorting, filtering, mering as well as constructing new abstractions from basic lists, such as associative lists.

Every value in Stratego is a term. This is also the case for lists. You can write the list 1, 2, 3 as Cons(1,Cons(2,Cons(3,Nil))), which is clearly a term. Fortunately, Stratego also provides some convenient syntactic sugar that makes lists more readable and easy to work with. We can write the same list as [1,2,3], which will be desugared internally in the the term above.

22.1. Making heads and tails of it

The most fundamental operations on lists is the ability compose and decompose lists. In Stratego, list composition on "sugared" lists, that is, lists writen in the sugared form, has some sugar of its own. Assume xs is the list [1,2,3]. The code [0|xs] will prepend a 0 to it, yielding [0,1,2,3]. List decomposition is done using the match operator. The code ![0,1,2,3] ; ?[y|ys] will bind y to the head of the list, 0, and ys to the tail of the list, [1,2,3].

The module collection/list contains a lot of convenience functions for dealing with lists. (collection/list is contained in the liblib library.) For example, the strategy elem will check if a given value is in a list. If it is, the identity of the list will be returned.

stratego> import liblib
stratego> <elem> (1, [2,3,1,4])
[2,3,1,4]

Continuing on the above Stratego Shell session, we can exercise some of the other strategies:

stratego> <length> [1,2,3,4,5]
5
stratego> <last> [5,6,7,8,9]
9
stratego> <reverse> [1,2,3,4,5]
[5,4,3,2,1]

There are two strategies for concatenating lists. If the lists are given as a tuple, use conc. If you have a list of lists, use concat:

stratego> <conc> ([1,2,3],[4,5,6],[7,8,9])
[1,2,3,4,5,6,7,8,9]
stratego> <concat> [[1,2,3],[4,5,6],[7,8,9]]
[1,2,3,4,5,6,7,8,9]

The sublist of the first n elements can be picked out with the take(|n) strategy:

stratego> <take(|3)> [1,2,3,4,5]
[1,2,3]

Finally, the fetch(s) strategy can be used to find the first element for which s succeeds:

stratego> <fetch(?2)> [1,2,3]
2

The Stratego library contains many other convenient functions, which are documented in the API documentation.

22.2.  Sorting

The list sorting function is called qsort(s), and implements the Quicksort algorithm. The strategy parameter s is the comparator function.

  
stratego> <qsort(gt)> [2,3,5,1,9,7]
[9,7,5,3,2,1]

22.3.  Associative Lists

Stratego also has library support for associative lists, sometimes known as assoc lists. There are essentially lists of (key, value) pairs, and work like a poor man's hash table. The primary strategy for working with these lists is lookup. This strategy looks up the first value associated with a particular key, and returns it.

stratego> <lookup> (2, [(1, "a"), (2, "b"), (3, "c")]) => "b"

22.4.  Pairing Lists

The library also contains some useful strategies for combining multiple lists. The cart(s) strategy makes a cartesian product of two lists. For each pair, the parameter strategy s will be called. In the second case below, each pair will be summed by add.

stratego> <cart(id)> ([1,2,3],[4,5,6])
[(1,4),(1,5),(1,6),(2,4),(2,5),(2,6),(3,4),(3,5),(3,6)]
stratego> <cart(add)> ([1,2,3],[4,5,6])
[5,6,7,6,7,8,7,8,9]

Two lists can be paired using zip(s). It takes a tuple of two lists, and will successively pick out the head of the lists and pair them into a tuple, and apply s to the tuple. zip is equivalent to zip(id).

stratego> <zip> ([1,2,3],[4,5,6])
[(1,4),(2,5),(3,6)]
stratego> <zip(add)> ([1,2,3],[4,5,6])
[5,6,7]

The inverse function of zip is unzip.

stratego> <unzip> [(1,4),(2,5),(3,6)]
([1,2,3],[4,5,6])

There is also unzip(s) which like unzip takes a list of two-element tuples , and applies s to each tuple before unzipping them into two lists.

22.5.  Lightweight Sets

In Stratego, lightweight sets are implemented as lists. A set differs from a list in that a given element (value) can only occur once. The strategy nub (also known as make-set) can be use to make a list into a set. It will remove duplicate elements. The normal functions on sets are provided, among them union, intersection, difference and equality:

stratego> <nub> [1,1,2,2,3,4,5,6,6]
[1,2,3,4,5,6]
stratego> <union> ([1,2,3],[3,4,5])
[1,2,3,4,5]
stratego> <diff> ([1,2,3],[3,4,5])
[1,2]
stratego> <isect> ([1,2,3],[3,4,5])
[3]
stratego> <set-eq> ([1,2,3],[1,2,3])
([1,2,3],[1,2,3])

22.6. Transforming Lists

Elementwise transformation of a list is normally done with the map(s) strategy. It must be applied to a list. When used, it will apply the strategy s to each element in the list, as shown here. It will return a list of equal length to the input. If the application of s fails on one of the elements map(s) fails.

stratego> <map(inc)> [1,2,3,4]
[2,3,4,5]

mapconcat(s) is another variant of the elementwise strategy application, equivalent to map(s) ; concat. It takes a strategy s which will be applied to each element. The strategy s must always result in a list, thus giving a list of lists, which will be concatenated. A slightly more convoluted version of the above mapping.

If we want to remove elements from the list, we can use filter(s). The filter strategy will apply s to each element of a list, and keep whichever elements it succeeds on:

stratego> <filter(?2 ; !6)> [1,2,3,2]
[6,6]
stratego> <mapconcat(\ x -> [ <inc> x ] \)> [1,2,3,4]

22.7. Folding from the Left and Right

List folding is a somewhat flexible technique primarily intended for reducing a list of elements to a single value. Think of it as applying an operator between any two elements in the list, e.g. going from [1,2,3,4] to the result of 1 + 2 + 3 + 4. If the operator is not commutative, that is x <op> y is not the same as y <op> x, folding from the left will not be the same as folding from the right, hence the need for both foldl and foldr.

The foldr(init, oper) strategy takes a list of elements and starts folding them from the right. It starts after the rightmost element of the list. This means that if we use the + operator with foldr on the list [1,2,3,4], we get the expression 1 + 2 + 3 + 4 + , which obviously has a dangling +. The strategy argument init is used to supply the missing argument on the right hand side of the last plus. If the init supplied is id, [] will be supplied by default. We can see this from the this trial run:

stratego> <foldr>(id, debug)
(4,[])
(3,(4,[]))
(2,(3,(4,[])))
(1,(2,(3,(4,[]))))
(1,(2,(3,(4,[]))))

With this in mind, it should be obvious how we can sum a list of numbers using foldr:

stratego> <foldr(!0, add)> [1,2,3,4]
10

The related strategy foldl(s) works similarly to foldr. It takes a two-element tuple with a list and a single element, i.e. ([x | xs], elem). The folding will start in the left end of the list. The first application is s on (elem, x), as we can see from the following trial run:

stratego> <foldl(debug)> ([1,2,3,4], 0)
(1,0)
(2,(1,0))
(3,(2,(1,0)))
(4,(3,(2,(1,0))))
(4,(3,(2,(1,0))))

Again, summing the elements of the list is be pretty easy:

stratego> <foldl(add)> ([1,2,3,4], 0)
10

22.8. Summary

In this chapter we got a glimpse of the most important strategies for manipulating lists. We saw how to construct and deconstruct lists, using build and match. We also saw how we can sort, merge, split and otherwise transform lists. The strategies for associative lists and sets gave an impression of how we can construct new abstractions from basic lists.

More information about list strategies available can be found in the collections/list module, in the library reference documentation.

Chapter 23. Strings

23.1. Basic String Operations

Strings, like all other primitive data types in Stratego, are terms. They are built with the build (!) operator and matched with the match (?) operator. Additional operations on and with strings are realized through strategies provided by the Stratego library. The most basic operations on strings provided by the library are concatenation, length computation and splitting. We will discuss operation each in turn.

The library provides two variants of the string concatenation operation. The first, concat-strings, takes a list of strings and returns the concatenated result. The second, conc-strings takes a two-element tuple of strings and returns the concatenated result:

stratego> <concat-strings> ["foo", "bar", "baz"]
"foobarbaz"
stratego> <conc-strings ("foo", "bar")
"foobar"

Once you have a string, you may want to know its length, i.e. the number of characters it contains. There are two equivalent strategies for determining the length of a string. If you come from a C background, you may favor the strlen strategy. If not, the string-length strategy may offer a clearer name.

The final basic operation on strings is splitting. There is a small family of closely related strategies for this, which all do simple string tokenization. The simplest of them is string-tokenize(|sepchars). It takes a list of characters as its term argument, and must of course be applied to a string.

stratego> <string-tokenize(|[' '])> "foo bar baz"
["foo","bar","baz"]

Another strategy in the tokenizer family is string-tokenize-keep-all(|sepchars). It works exactly like string-tokenize(|sepchars), except that it also keeps the separators that were matched:

     
stratego> <string-tokenize-keep-all(|[' '])> "foo bar baz"
["foo"," ","bar"," ","baz"]

23.2. Sorting Strings

Even if you don't maintain a phone directory, sorting lists of strings may come in handy in many other enterprises. The strategies string-sort and string-sort-desc sort a list of strings in ascending and descending order, respectively.

stratego> !["worf", "picard", "data", "riker"]
["worf", "picard", "data", "riker"]
stratego> string-sort
["data","picard","riker","worf"]
stratego> string-sort-desc
["worf","riker","picard","data"]

If you only have two strings to sort, it may be more intuitive to use the string comparison strategies instead. Both string-gt and string-lt take a two-element tuple of strings, and return 1 if the first string is lexicographically bigger (resp. smaller) than the second, otherwise they fail.

stratego> <string-gt> ("picard","data")
1
stratego> <string-lt> ("worf","data")
command failed

Not directly a sorting operation, string-starts-with(|pre) is a strategy used to compare prefixes of strings. It takes a string as the term argument pre and must be applied to a string. It will succeed if pre is a prefix of the string it was applied to:

stratego> <strings-starts-with(|"wes")> "wesley"
"wesley"

23.3. Strings and Terms

We already said that strings are terms. As with terms, we can also deconstruct strings, but we cannot use normal term deconstruction for this. Taking apart a string with explode-string will decompose a string into a list of characters. We can then manipulate this character list using normal list operations and term matching on the elements. Once finished, we can construct a new string by calling implode-string. Consider the following code, which reverses a string:

stratego> !"evil olive"
"evil olive"
stratego> explode-string
[101,118,105,108,32,111,108,105,118,101]
stratego> reverse
[101,118,105,108,111,32,108,105,118,101]
stratego> implode-string
"evilo live"

This explode-string, strategy, implode-string idiom is useful enough to warrant its own library strategy, namely string-as-chars(s). The code above may be written more succinctly:

stratego> <string-as-chars(reverse)> "evil olive"
"evilo live"

Sometimes, in the heat of battle, it is difficult to keep track of your primitive types. This is where is-string and is-char come in handy. As you might imagine, they will succeed when applied to a string and a character, respectively. A minor note about characters is in order. The Stratego runtime does not separate between characters and integers. The is-char must therefore be applied to an integer, and will verify that the value is within the printable range for ASCII characters, that is between 32 and 126, inclusive.

Finally, it may be useful to turn arbitrary terms into strings, and vice versa. This is done by write-to-string and read-from-string, which may be considered string I/O strategies.

stratego> <write-to-string> Foo(Bar())
"Foo(Bar)"
stratego> read-from-string
Foo(Bar)

23.4. Strings and Numbers

Another interplay between primitive types in Stratego is between numbers and strings. Converting numbers to strings and strings to numbers is frequently useful when dealing with program transformation, perhaps particularly partial evaluation and interpretation. Going from numbers to strings is done by int-to-string and real-to-string. Both will accept reals and integers, but will treat is input as indicated by the name.

stratego> <int-to-string> 42.9
"42"
stratego> <real-to-string> 42.9
"42.899999999999999"

The resulting number will be pretty-printed as best as possible. In the second example above, the imprecision of floating point numbers results in a somewhat non-intuitive result.

Going the other way, from strings to numbers, is a bit more convoluted, due to the multiple bases available in the string notation. The simplest strategies, string-to-real and string-to-int, assume the input string is in decimal.

stratego> <string-to-real> "123.123"
1.231230000000000e+02
stratego> <string-to-int> "123"
123

For integers, the strategies hex-string-to-int, dec-string-to-int, oct-string-to-int and bin-string-to-int can be used to parse strings with numbers in the most popular bases.

Chapter 24. Hashtables and Sets

24.1.  Hashtables

The rewriting paradigm of Stratego is functional in nature, which is somewhat contradictory to the imperative nature of hashtables. Normally, this doesn't present any practical problems, but remember that changes to hashtables "stick", i.e. they are changed by side-effect.

The Stratego hashtable API is pretty straightforward. Hashtables are created by new-hastable and destroyed by hashtable-destroy.

stratego> import lib	
stratego> new-hashtable => h
Hashtable(136604296)

The result Hashtable(136604296) here is a handle to the actual hashtable. Consider it a pointer, if you will. The content of the hashtable must be retrieved with the hashtable-* strategies, which we introduce here. The strategy hashtable-copy can be used to copy a hashtable.

Adding a key with value to the table is done with hashtable-put(|k,v), where k is the key, v is the value. Retrieving the value again can be done by hashtable-get(|k).

stratego> <hashtable-put(|"one", 1)> h
Hashtable(136604296)
stratego> <hashtable-get(|"one")
1

The contents of the hashtable can be inspected with hashtable-values and hashtable-keys.

Nesting is also supported by the Stratego hashtables. Using hashtable-push(|k,v), a new "layer" can be added to an existing key (or an initial layer can be added to a non-existing key). Removing a layer for a key can be done with hashtable-pop(|k).

stratego> <hashtable-push("one",2)> h
Hashtable(136604296)
stratego> <hashtable-get("one")> h
[2,1]
stratego> <hashtable-pop(|"one")> h
Hashtable(136604296)
stratego> <hashtable-get(|"one")> h
[1]
stratego> <hashtable-remove(|"one")> h
Hashtable(136604296)
stratego> <hashtable-values> h
[]

24.2.  Indexed Sets

The library provides a rather feature complete implementation of indexed sets, based on hashtables. A lightweight implementation of sets, based on lists, is explained in Chapter 22.

Similar to hashtables, indexed sets are created with the new-iset strategy, copied with iset-copy and destroyed with iset-destroy.

stratego> new-iset => i
IndexedSet(136662256)

The resulting term, IndexedSet(136662256), is a handle to the actual indexed set, which can only be manipulated through the iset-* strategies.

Adding a single element to a set is done with iset-add(|e), whereas an entire list can be added with the iset-addlist(|es). Its elements can be returned as a list using iset-elements.

stratego> <iset-addlist(|[1,2,3,4,4])> i
IndexedSet(136662256)
stratego> iset-elements
[1,2,3,4]

Notice that the result is indeed a set: every value is only represented once.

Using iset-get-index(|e), the index of a given element e can be found. Similarly, iset-get-elem(|i) is used to get the value for a particular index.

stratego> <iset-get-index(|3)> i
2
stratego> <iset-get-elem(|3)> i
4

Note that the indexes start at 0.

The set intersection between two sets can be computed with the iset-isect(|set2) strategy. The strategy iset-union(|set2) calculates the union of two sets, whereas iset-subset(|set2) checks if one set is a subset of another. Equality between two sets is checked by iset-eq(|set2). These strategies are all used in a similar way:

stratego> <iset-eq(|set2)> set1

A single element can be removed from the set with iset-remove(|e). iset-clear will remove all elements in a set, thus emptying it.

Chapter 25. I/O

This chapter explains the strategies available in the library for controlling file and console I/O.

The need for traditionally file I/O is somewhat diminished for typical applications of Stratego. Normally, Stratego programs are designed to worktogether connected by Unix pipes. The programs employ io-wrap (or similar strategies) that automatically take care of the input and output. See Chapter 26 for details.

The primitive layer of Stratego I/O inherits its characteristics from Unix. The basic I/O strategies recognize the special files stdout, stdin and stderr. Streams are opened by fopen and closed with fclose On top of this, a collection of more convient strategies have been built.

25.1. Console I/O

The basic strategies for console I/O print and printnl are used to write terms to stdout or stderr (or any other opened file). They both take a tuple. The first element of the tuple is the file to write to, the second is a list of terms. Each term in the list be converted to a string, and and these strings will be concatenated together to form the resulting output. The printnl will also append a newline to the end of the resulting string.

The following module should be compiled with strc, as usual.

module example
imports liblib
strategies
  main = 
    <print> (stdout, ["baz"])
    ; <printnl> (stdout, [ "foo", 0, "bar" ])

After compiling this file, running it will give the following result:

$ ./example
bazfoo0bar
$

Notice how the string baz will be written without a newline (or other space). Also, notice how the terms in the list argument were concatenated.

When using these strategies in the Stratego Shell, some care must be taken when using the std* files, as the following example shows.

stratego> <printnl> (stdout(), [ "foo", 0, "bar" ])
foo0bar

The shell requires that you put an extra parenthesis after the stdout.

The debug and error are convenience wrappers around printnl. They will always write their result to stderr. The error strategy is defined as:

error =
  where(<printnl> (stderr, <id>))

It is used similarly to the printnl strategy:

stratego> <error> ["foo", 0, "bar"]
foo0bar

The debug strategy accepts any term, i.e. not only lists of terms. The term will be written verbatim:

stratego> <debug> [ "foo", 0, "bar" ]
["foo",0,"bar"]

25.2. Path and Directory Operations

The library provides a small set of simple file and directory manipulation operations. Assume the directory /tmp only contains the files foo, bar, baz. Elementary directory operations can be done as illustrated below:

stratego> <readdir> "/tmp"
["foo","bar","baz"]
stratego> <rename-file> ("/tmp/foo", "/tmp/bax")
"/tmp/bax"
stratego> <remove-file> "/tmp/baz"
[]
stratego> <link-file> ("/tmp/bar", "/tmp/foo")
"/tmp/foo"
stratego> <link-file> ("/tmp/bar", "/tmp/foo")
"/tmp/foo"
stratego> <new-temp-dir> "/tmp"
"/tmp/StrategoXTnsGplS"

The library contains a family of strategies which must be applied to a File, and will return information about it. these include isdir, isatty, isfifo and islnk which are predicates checking if a file is a directory, TTY, FIFO or a symbolic link, respectively. To obtain a File object in the first place, we should call file-exists followed by filemode. Thus, checking if /etc is a directory is done as follows:

stratego> <file-exists ; filemode ; isdir> "/etc"

The library also has another family of strategies for getting information about files. These must be applied to a string containing the filename. The family includes is-executable, is-readable and is-writeable.

stratego> <is-executable> "/bin/bash"
"/bin/bash"

Finally, the directory strategies also include the usual suspects for dealing with paths.

stratego> <is-abspath> "../foo"
command failed
stratego> <dirname> "/foo/bar/baz"
"/foo/bar"
stratego> <base-filename> "/foo/bar/baz"
"baz"
stratego> <get-extension "/tmp/foo.trm"
"trm"
stratego> <abspath> "../foo"
/home/karltk/source/oss/stratego/strategoxt-manual/trunk/../foo

There are also a few strategies for finding files. We shall describe find-file(s). The other variants of find-file are described in the library documentation. The strategy find-file(s) finds one file with a specific file extension in a list of directories. It takes a two-element tuple. The first element is a file name as a string, then second element is a list of paths, i.e. (f, [d*]). The extension of f will be replaced by what is produced by s, and the directories given in [d*]. Consider the code below.

stratego> <find-file(!"rtree")> ("file.str", ["."])

This snippet will consider the filename file.str, replace its extension with rtree and look through the directories in the list ["."]. Effectively, it will search for file.rtree in the current directory.

25.3. File and Text I/O

Opening a file is done with the fopen strategy. It takes a two-element tuple, the first element is the filename as a string, the second is the open mode, which is also a string. The most important modes are read (r); write ("w") which opens and empty file for writing, truncating any existing file with the same name; and append (a) which appends to the file if it already exists. After all file operations stream have been finished, it should be closed with fclose, which will flush and close the file. Explicit flushing can also be done with fflush.

It should be pointed out that reading and writing text files with Stratego is rather rare. Normally, text files are read with a parser generated from an SDF description and written using a pretty-printer defined in the Box formalism. In rare cases, this may turn out be too heavy handed, especially if the file format is simplistic and line-based. In this instance, we can come up with an easier solution using read-text-file and read-text-line.

Assume the file /tmp/foo contains the following lines:

one
two
three

We can read this file in one big chunk into a string with the read-text-file strategy, which must be applied to a filename:

stratego> <read-text-file> "/tmp/foo"
"one\ntwo\nthree\n"

Alternatively, for example if the file is large, we can read it line by line. In this scenario, we must open the file and get a handle to a stream.

      
stratego> <fopen> ("foo.txt", "r") => inp
Stream(136788400)
stratego> <read-text-line> inp
"one"

25.4. Term I/O

The primary form of file I/O you will be using in Stratego is reading and writing terms. As explained earlier, the terms are stored on disk as either binary, compressed text or plain text ATerms. Reading a term, no matter which storage format, is done with the ReadFromFile strategy. It is applied to a filename.

stratego> <ReadFromFile> "/tmp/foo.trm"
Foo(Bar)

To write a term to file, you can use WriteToTextFile or WriteToBinaryFile. The binary format is approximately eight times more space-efficient on average. Both strategies take a two-element tuple where the first element is the filename and second is the term to write. Writing the current term requires a minor twist, which is shown here:

stratego> <WriteToBinaryFile> ("/tmp/bar.trm", <id>)
Foo(Bar)

It is also possible to read and write terms from and to strings, using read-from-string and write-to-string. Chapter 23 contains explanation of how these strategies work.

25.5. Logging

The strategies for logging are used pervasively throughout the Stratego toolchain. They are easy to use in your own applications, too. The logging system is built on top of the log(|severity, msg) and log(|severity, msg, term) strategies. It is possible to use these directory, as the following example demonstrates.

stratego> import util/log
stratego> log(|Error(), "my error")

However, it is preferrable to use the high-level wrapper strategies fatal-err-msg(|msg), err-msg(|msg), warn-msg(|msg) and notice-msg(|msg). Except for fatal-err-msg, these strategies will return with the current term untouched, and write the message as a side effect. The fatal-err-msg strategy will also terminate the program with error code 1, after writing the message.

Chapter 26. Command-line Options

As Part II explained, the world of Stratego is one of small programs tied together using Unix pipes. The pipes carry the data, while configuration and control is passed between programs in the form of command line arguments. Incidentally, this is the same mechanism used by humans to invoke programs, and this eases understanding and debugging of XT compositions tremendously. Details about debugging is covered in Chapter 30. In this chapter, we will cover the mechanism available in Stratego for working with command line arguments.

26.1. Parsing Command-line Options

When a compiled Stratego program is first started, the initial value of the current term is a list containing the command line arguments to the program. Fhe following program, foo, is a genuine "do nothing" program.

module foo
imports liblib
strategies
  main = id

In the course of this chapter, we will extend this program with new options and even a help screen for the user to enjoy. But first, let us compile and run foo with some arguments, to get an idea of where we stand.

$ strc -i foo.str
...
$ ./foo --help
["./foo","--help"]
$ ./foo -i foo.str --extra --option -s
["./foo","-i","foo.str","--extra","--option","-s"]

From this interaction, we see that a list of the arguments provided on the command line becomes the initial term of the program. Each command line argument becomes its own string element in the list, and the first element of the list is the command used to invoke the foo itself. Clearly, this list must be interpreted somehow, for the arguments to have any meaning. The library contains a collection of robust strategies that deal with just this. The option handling strategies will parse the argument list, let you set default values for options and transparently deal with long and short forms of the same option. Even more interesting, the library provides so-called wrap strategies that abstract away all of the dreary details of this option handling, and also provide a default set of options with some basic functionality.

Perhaps the most frequently used wrap strategy is io-wrap (or its XTC equivalent, xtc-io-wrap, which is not covered here). In fact, io-wrap is a family of strategy, which includes io-wrap(s), io-wrap(opts, s) and io-wrap(opts, usage, about, s). All of these variants provide the same basic functionality. The increasing number parameters allows you to override default bevhavior, when you need to. When using one of these strategies, a standard package of command line handling will be provided to your users through your program. Let us start with the simplest case, io-wrap(s) and work our way from there. Consider a revised edition of foo, from above:

module foo
imports liblib
strategies
  main = io-wrap(id)

Here, we wrap the core logic of our program (just id in our case) inside the io-wraper. If we run foo with the --help this time around, we will get a more instructive reply than previously:

$ ./foo --help
Options:
   -i f|--input f   Read input from f
   -o f|--output f  Write output to f
   -b               Write binary output
   -S|--silent      Silent execution (same as --verbose 0)
   --verbose i      Verbosity level i (default 1)
                    ( i as a number or as a verbosity descriptor:
                      emergency, alert, critical, error,
                      warning, notice, info, debug, vomit )
   -k i | --keep i  Keep intermediates (default 0)
   --statistics i  Print statistics (default 0 = none)
   -h|-?|--help     Display usage information
   --about          Display information about this program
   --version        Same as --about

Description:

All of a sudden, our program is a well-behaved citizen in the command line world. It answers to --help, and appears to have a few other tricks up its sleeve to. For example, the input term to foo can now be specified using the -i option, and output can be stored to a file using the -o option.

So does this actually work? All for free? Let's test with putting the following term into the file term.trm:

Yes(It(Works))

Bursting with excitement, we try:

$ ./foo -i term.trm
Yes(It(Works))

And if that's not enough, there is even a bit of extra convenience provided by io-wrap: If -i is not provided, stdin is read. Similarly, stdout is written to if -o is not specified.

So, using io-wrap is all you have to do for your program to gain a minimal, but functional set of command line options. As a bonus, these options also make your program compatible with XTC; it can be composed with other XTC components.

26.2. Adding Custom Options

It is often necessary for programs to expose switches to turn functionality on and off, or to read extra configuration knowledge from external files. All these cases require additional command line options, so we need a mechanism for extending the basic io-wrap(s) strategy. The preferred way of adding new options is to use the io-wrap(opts,s) strategy, providing it with a strategy encoding the options.

When adding a new option, we must decide whether this option will require a argument of its own, or not. The term ArgOption is used to construct options that take arguments and Option is the term used for on/off switches. Suppose we want to expose an option --verify that enables the user to run our transformation in a self-verifying mode. This is clearly an on/off switch, and therefore a job for Option.

Adding this option to our program foo gives us the following code:

module foo
imports liblib
signature
  constructors
    Verify : Option

strategies

  main =  io-wrap(
    Option(
        "--verify"
      , <set-config> (Verify(), "on")
      , !"--verify         Turn on verification")
    , do-foo)

  do-foo = ...

Note that we made a new term type, Verify, to serve as our switch symbol. Inside the real logic of program, given by do-foo, we would write <get-config> Verify to get the state of the Verify switch. In the case where the user had specified --verify on the command line, get-config would result in the term "on", taken from the declaration of our verify Option. If the did not add --verify to his command line arguments, <get-config> Verify will fail.

Options with arguments is provided through ArgOption. The use is pretty much identical to that of Option. Assume our transformation needs a bit of help from configurable processing tables (whatever that might be), and that we want these tables configured at runtime, using the -p option. We want to add another alternative to the opts argument of io-wrap:

  main = io-wrap(
     Option(....)
     + 
     ArgOption(
        "-p"
       , where(<extend-config> ("-p", [<id>]))
       , !"-p file   Use processing table in file")
     , id)

Note how we compose options with the choice operator (+). With this addition, our users can now specify their elusive processing tables with the -p. The arity of this option is checked automatically. That is, if the user adds -p to his argument list without specifying a file afterwards, this will be signaled as an error, and the usage screen will be printed. Once the user has read this and corrected the error of his ways, the value (the filename) provided via -p can be obtained using the get-option strategy. If no -p was specified, get-option will fail. Sometimes, this failure may be inappropriate, and a default value is desired instead. If you browse through Stratego code, you may come across the following idiom for dealing with this situation:

  get-config-p =
    <get-config> "-p" <+ ![]

This is all you need to know about basic command-line option processing. When in doubt, you are advised to refer to the detailed API documentation is available in the API reference.

26.3. Setting Description and About

If your program is primarily intended for human use, you are encouraged to complete your program's option configuration with a short description of what your tool does.

We can easily add a short description and also an about section. The description is shown as part of the help screen (displayed with --help), whereas the about section is displayed when the arguments to foo contain --about. It is customary for the about screen to contain copyright notices and credits.

  
  main = io-wrap(
        ...
      , foo-tool-usage
      , foo-tool-about
      , id)

  foo-tool-usage =
    default-system-usage(
      !["Usage: foo -p proctbl [options]"]
    , ![ "This program verifies the input against a processing table.\n"]
    )

  foo-tool-about =
     <echo> "Written by Alan Turing <alan@turing.org>"

After compiling this version of foo, invoking it with --help give the following help screen:

$ ./foo --help
Usage: foo -p proctbl [options]

Options:
   --verify         Turn on verification
   -p file          Use processing table in file
   -i f|--input f   Read input from f
   -o f|--output f  Write output to f
   -b               Write binary output
   -S|--silent      Silent execution (same as --verbose 0)
   --verbose i      Verbosity level i (default 1)
                    ( i as a number or as a verbosity descriptor:
                      emergency, alert, critical, error,
                      warning, notice, info, debug, vomit )
   -k i | --keep i  Keep intermediates (default 0)
   --statistics i  Print statistics (default 0 = none)
   -h|-?|--help     Display usage information
   --about          Display information about this program
   --version        Same as --about

Description:
This program verifies the input against a processing table.

If we invoke our splendid program foo with the --about option, we can now observe:

$ ./foo --about
Written by Alan Turing <alan@turing.org>

26.4. I/O-less Programs

Not all programs written in Stratego intended for processing ATerms. For these programs, the full I/O functionality provided by io-wrap may often turn out to be inappropriate. An alternative way of dealing with parameters is also provided by the library, centered around the option-wrap family of strategies. option-wrap works analogously to io-wrap, but does not provide the -i and -o options, nor does it read anything from stdin automatically.

The default set of options provided by option-wrap is shown below:

    
   --verify         Turn on verification
   -h|-?|--help     Display usage information
   --about          Display information about this program
   --version        Same as --about

Adding new options is identical to what we already explained for io-wrap.

Chapter 27. Unit Testing with SUnit

StrategoUnit or SUnit is a Unit Testing framework for Stratego inspired by JUnit. The idea is to specify tests that apply a strategy to a specific term and compare the result to the expected output. The tests are combined into a test suite, which runs all tests and reports the number of successes and failures. When all tests pass, the program exits with status 0, otherwise it exists with status 1 to flag the error to the calling program, which is typically a makefile.

27.1. Setting up a test suite

Module list-zip-test.str in the Stratego Standard Library is an example test suite. SUnit test suites can be compiled to an executable program. When run the program gives the following output:

test suite: list-zip-test
nzip0-test
zip-test1
zip-test2
successes: 3
failures: 0
(3,0)

An extract from the module shows how a test suite can be set up:

module list-zip-test
imports list-zip sunit
strategies

  main =
    test-suite(!"list-zip-test",
        nzip0-test;
        zip-test
     )

  nzip0-test =
    apply-test(!"nzip0-test"
        ,nzip0(id)
        ,!["a","b","c"]
        ,![(0, "a"), (1, "b"), (2, "c")]
        )

The test-suite strategy takes a string and a strategy. The strategy is typically a sequential composition of calls to unit tests. Unit tests such as nzip0-test are defined using one of the unit-test strategies from module sunit, which is in the standard library.

27.2. Compare expected and actual output

The prototypical unit test is composed with the apply-test strategy, which takes a name, the strategy that is being tested, the input term, and the exptected output term. Note that the name, input, and output term must be specified in strategies, which means that if literal terms are used then you must build them using the ! operator.

apply-test(!"test3"
, ltS
, !("3","5")
, !("3","5")
)

27.3. Check for failure

It is also useful to test that a strategy fails if applied to a certain input. The ordinary apply-test cannot be used for this because it requires an output term to which the result will be compared. For testing failure the apply-and-fail strategy is available in sunit. It ensures that the strategy fails if applied to a certain input.

apply-and-fail(!"is-double-quote 3"
, is-double-quoted
, !"\"fnord"
)

27.4. Check arbitrary conditions on output

Sometimes a test just want to check some condition on the output of a strategy application, without actually specifying the output itself. The apply-test strategy cannot be used for this because it performs an equivalence test of the required output and the actual output. In this case apply-and-check can be used. This strategy takes 4 arguments: a name, a strategy (s) to test, an input and a strategy that will be applied to the result of applying s to the input. This strategy must succeed if the output is correct, or fail if the output is incorrect.

new-test =
  apply-and-check(!"new test",
    (new, new)
  , !(1, 1)
  , not(eq); (is-string, is-string)
  )

27.5. Unit testing with XTC

XTC is the preferred system for writing components in Stratego/XT. Testing that an XTC component acts as intended is not much more difficult than for individual strategies. As the semantics of the interfaces to XTC programs is open-ended, there is currently no generic testing framework for handling all the details.

Nevertheless, adapting the unit testing framework explained above for testing your components is easy. Consider the strategy xtc-apply-test, given below.

xtc-apply-test(|nm, prog, args, inp, outp) =
  xtc-temp-files(
    apply-test(!nm,
        write-to
      ; xtc-transform(!prog, !args)
      ; read-from
    , !inp
    , !outp
    )
  )

This is a straightforward extension of apply-test. It will invoke the XTC program given in the prog term argument with the list of command line arguments in args. The input to the program is the term given by inp. The resulting term after prog has been run is checked against outp. If they match, the test succeeds.

The simple code above does not account for additional file arguments to prog. This may be useful if prog if prog is, say, a pretty-printer which takes both a document and a style sheet as arguments. The code below provides an example of how this may be handled.

xtc-apply-pp-test(|nm, prog, ssheet, args, inp, outp) =
  xtc-temp-files(
    apply-test(!nm,
      where(<print-to> [ssheet] => FILE(f))
      ; write-to
      ; xtc-transform(!prog, <concat> [ args, "--stylesheet", f])
      ; read-from
    , !inp
    , !outp
    )
  )      

Chapter 28. Transformation Tool Composition with XTC

In Chapter 3, we explained how the Stratego/XT universe is built from small components, each a separately executable program. Many such components are provided for your reuse by the Stratego/XT distribution. We have already seen some of these in action throughout Part II. In this chapter, we will first explain how to compose existing components together, and then proceed to explain how you can make your own Stratego programs into reusable XT components.

28.1. Basic Mechanisms of XTC

Before we can compose XT components, we must place them in a component repository. This is referred to as registration. The registration associates each component with a name, a path and a version. The name is used later to look up components, and map their names to their actual location in the file system. This is very handy when you program with XT components. Inside our Stratego programs, we only have to specify the name of the program we want executed, and the XTC machinery will automatically figure out which program to execute, set up the piping, deal with temporary files, and even perform the execution and parameter passing for us.

In this section, we will cover the registration and lookup mechanisms in some detail.

28.1.1. Registration of Programs and Data

An XTC repository is a collection of programs and data files. The programs, which we usually refer to as XT components, must be registered explicitly using the xtc tool before they can be used. This is also the case for the data files. A normal program transformation system built with Stratego/XT will contain an XTC repository, where all its components have been registered. Though this registration is done automatically for you by the Stratego build system, it will prove instructive to know what takes place behind the scenes, by the understanding the xtc command (refer to Chapter 29 for an explaination of how to configure the Stratego/XT build system to automatically register your XT components).

Suppose our project, called Ogetarts, has been installed into .../ogetarts (where ... is a path of your choice). A typical directory layout for this project would include the following directories:

.../ogetarts/
             share/
                   ogetarts/
             bin/
             libexec/
             lib/

The part we are interested in now is the directory .../ogetarts/share/ogetarts/. This is where the registry for the XTC repository is normally placed, in the file XTC. Assuming we already have the XT component foo2bar placed in .../ogetarts/libexec/. The following xtc invocation will register it with the Ogetarts registry:

$ xtc -r .../ogetarts/share/ogetarts/XTC register -t foo2bar -l .../ogetarts/libexec -V 1.0

This command will also take care of creating a fresh XTC registry file if it does not already exist. The -l specifies the path to the component being registered, and -t specifies its name. This name is used for lookup purposes later. The -V is used to associate a version number with a given program. It is possible to include the version number in the lookup of a component, to find a particular version of a particular component, but this is not common practice, and should be avoided. However, every registration must have include a version number.

28.1.2. Importing other Repositories

The XTC system provides a form of inheritance, whereby one repository can refer to another. You may also think of it as a scoping mechanism on the namespaces of repositories. When searches for components fail at the local repository, the imported repositories will be searched in turn.

It is practically always necessary for your project to import the Stratego/XT repository, as most transformation systems built with Stratego/XT reuse many of the XT components. The following command adds the Stratego/XT repository to our project:

$ xtc -r .../ogetarts/share/ogetarts/XTC import /usr/share/StrategoXT/XTC

In case you wondered, the current version of XTC has no mechanism for declaring some components as private, and others public. Once a component c is registered in an XTC repository r, all other repositories importing r can ask for c.

28.1.3. Searching Repositories

When you have registered your components or imported other repositories, you may inspect your repository to see that everything looks good. This is done using the query option to xtc. Using query, you can either look for a particular component, or list all registrations in a given repository.

The following command will search the Ogetarts repository for the sglr component. This component was not registered by us, but is inherited from the Stratego/XT repository.

$ xtc -r .../ogetarts/share/ogetarts/XTC query -t sglr 
sglr (3.8) : /usr/bin/sglr

Alternatively, we can list all registrations, in one go:

$ xtc -r .../ogetarts/share/ogetarts/XTC query -a
foo2bar (1.0) : .../ogetarts/libexec/foo2bar
stratego-lib (0.16M1) : .../stratego-lib
...

The format of this list is name (version) : path, where name, we remember, is the handle used to look up the given component. The list follows a pre-determined order. First, all registrations in the Ogetarts will be displayed. Here, foo2bar is our only component. After the local components, the contents of each imported repository will be displayed. In our case, we only imported Stratego/XT, and the first component in Stratego/XT is stratego-lib. The other 490 registrations have been omitted, for the sake of clarity.

28.2. Composing Tools in Stratego

Much of the scalability of Stratego/XT comes from its component model, thus it is important to know how to design your own programs to take advantage of this infrastructure. Fortunately, the programming interface of XTC consists of a small set of clearly defined strategies. If you are already familiar with io-wrap, as introduced in Chapter 26, this will be even easier to comprehend.

There are three main usage scenarios for the XTC API. Either you use the API to create a new XT component of your own, or you use it to invoke an XT component from your Stratego program (which need not be an XT component), or both; you are writing an XT component which calls other XT components.

28.2.1. Making an XT component

The simplest way to make an XT component is to wrap your top level strategy with the xtc-io-wrap wrapper. This automatically bestows your program with basic command line option parsing capabilities, and also basic I/O.

The following is an example of a trivial XT component which just passes through the term passed to it.

module xtcid
imports liblib
strategies
  main = xtc-io-wrap(xtcfoo)
  xtcfoo = id

As with the io-wrap strategy explained in Chapter 26, a default set of command line options is provided by xtc-io-wrap. After compiling the xtcid.str, we can run it to inspect the default set of options.

$ ./xtcid --help

Options:
   -i f|--input f   Read input from f
   -o f|--output f  Write output to f
   -b               Write binary output
   -S|--silent      Silent execution (same as --verbose 0)
   --verbose i      Verbosity level i (default 1)
                    ( i as a number or as a verbosity descriptor:
                      emergency, alert, critical, error,
                      warning, notice, info, debug, vomit )
   -k i | --keep i  Keep intermediates (default 0)
   --statistics i  Print statistics (default 0 = none)
   -h|-?|--help     Display usage information
   --about          Display information about this program
   --version        Same as --about

Description:

This is normally all you have to do in order to have a working XT component. You can add additional options using the Option and ArgOption, as explained in Chapter 26, by wrapping your toplevel strategy with xtc-io-wrap(extra-opts, s) instead of xtc-io-wrap(s). In either case, your program will now automatically read the input term from the file specified with -i, pass this to s, then write the result of s to the file specified by -o. When -i or -o are not specified, stdin and stdout will be used instead, respectively.

$ echo "My(Term)" | ./xtcfoo -o myterm.trm
$ cat myterm.trm
My(Term)
$

In some situations, it does not make sense for your component to accept an input term, or to generate and output term. That is, your component may be a data generator, or a data sink. In these cases,your component should rather use xtc-input-wrap, in the case of a sink, or xtc-output-wrap, in the case of a generator.

The following programs shows a trivial generator, which produces the term "foo" when invoked.

module xtcfoo
imports liblib
strategies
  main = xtc-output-wrap(xtcfoo)
  xtcfoo = !"foo" ; write-to

xtcfoo strategy is our top level strategy. Note how it ends in a call to write-to. The argument s to xtc-output-wrap(s) must result in a file, not a term. Copying the current term to a file, is taken care of by the write-to strategy. write-to will create a fresh temporary file and write the current term to it. The inverse of write-to is read-from which reads a term from a file. This latter strategy is used together with xtc-input-wrap, analogously to the example above.

In both cases (xtc-input-wrap and xtc-output-wrap), you may add additional command line options, by using variants of the wrappers which accept extra options, xtc-input-wrap(extra-options, s) and xtc-output-wrap(extra-options, s), respectively.

Checking Component Dependencies.  If you write a component which depends on other components as part of its operation, you are encouraged to add dependency checking using the xtc-check-dependencies. This is a two step procedure: First, you add the --check option to your components command-line options, by adding check-options(deps) to the extra-opts argument of xtc-io-wrap. Afterwards, you should call xtc-check-dependencies as part of your option-processing.

Going back to our simple xtcid program, we would add the --check with functionality, as follows:

module xtcid
imports liblib
strategies
  main = xtc-io-wrap(check-options(!["sglr", "foo2bar"]), xtcid)
  xtcid = xtc-check-dependencies ; id

After recompiling xtcid, the user can now ask your component to do a self-check of its dependencies, by calling it with the --check. You may want to piggy-back on this option, adding other kinds of self-checks, such as data consistency checking.

Graceful Termination.  As you have witnessed, the implementation of the XTC model makes heavy use of temporary files. It is important that these files are cleaned up after exectuion of your program. In all programs we have seen so far, this was taken care of automatically by the wrappers. But what happens if you want to terminate your program in the middle of execution, by calling exit, for example? In that case, the temporary files will not be removed properly. A safe way of exiting XTC programs is provided by the xtc-io-exit strategy. This strategy is a plugin replacement for exit, but takes care of cleaning any and all temporary files created by your program before terminating.

28.2.2. Invoking XT components

Now that we know how to create XT components, and we also know that the Stratego/XT environment provides many reuseable XTC programs, we should take a bit of time to explain how we can invoke these inside our Stratego programs. It is important to realize that your Stratego program need not itself be an XT component before it can call out to other XT components. Both XTC programs and normal Stratego programs call XT components in the same way.

Before we proceed, we need to create a small XTC program that we can call. The following is an XTC version of the inc strategy. When invoked on an integer n, it will return n + 1.

module xtcinc
imports liblib
strategies
  main = xtc-io-wrap(read-from ; <add> (<id>, 1) ; write-to)

Let us quickly whip up a client for this component, that invokes xtcinc on the integer 1:

module xtcclient
imports liblib
strategies
  main = !1 ; xtc-transform(!"xtcinc")

Let us a compile and run this program.

$ ./xtcclient
[ identity crisis | error ] No XTC registration for xtcinc found
./xtcclient: rewriting failed

Harking back a few sections, we should now realize that we forgot to register xtcinc in a repository. We already know how to do this. For the sake of this demonstration, let us create the XTC registry in the same directory as the source code:

$ xtc -r ./XTC r -t xtcinc -V 1.0 -l `pwd`

If we were to run our program xtcclient again at this point, we would still get the same error. This is because we have not told xtcclient which repository it should use. This is done by the --xtc-repo option to the Stratego compiler.

$ strc --xtc-repo ./XTC -i xtcclient.str
....

This should complete the necessary steps: We have created the XTC program xtcinc, we have registered it in the XTC registry in ./XTC, and we have told strc to compile our client program, xtcclient against this repository. We should be all set to run our composition. Let's try.

$ ./xtcclient
2

At last, it works. We have connected two components. By changing the internals of our components, we can change them to process any terms we want, and we can of course also add additional components into the mix, all using the same basic steps we have explained here. Having to remember the --xtc-repo option to strc all the time is a bit annoying, but as we shall in Chapter 29, this will be taken care of automatically by the Stratego/XT build system.

Finding XTC Data Files.  We have just seen how to use XTC programs from a repository, but so far, we have said nothing about any data files we have registered. This is where the xtc-find-file strategy comes in handy. Let us go through the process of creating a term, registering it, and then using it inside our program.

Suppose the file myterm.trm contains the following term, written in plain text:

My(Term)

Calling xtc as follows, will register it in the local repository we have already created.

$ xtc -r ./XTC r -t myterm.trm -V 1.0 -l `pwd`

This will register the file myterm.trm under the name myterm.trm, which we can use to look it up from inside our programs later. Let us make a new program that does just this.

module xtcload
imports liblib
strategies
  main = <xtc-find-file> "myterm.trm" ; read-from

xtc-find is applied to a string, which must be the name of an already registered data file in the repository. As before, we have to compile our program using strc, and remember to include the --xtc-repo option.

$ strc -i xtcload.str --xtc-repo ./XTC

We can now run our freshly compiled program.

$ ./xtcload
My(Term)

The result is as we anticipated. Congratulations! You have now mastered the basics of the XTC mechanics.

28.3. Summary

This chapter introduced you to the mechanics of the XTC model. We saw how to maintain a registry of XT components using the xtc tool, and also how to write XTC compositions in Stratego. Additional detail about the XTC API can be found in the API reference documentation. The complete documentation for the xtc tool is given on the manual page (xtc).

Chapter 29. Building and Deploying Stratego Programs

There are two typical scenarios for building Stratego programs. The first, and simplest, is to execute the compiler by hand on the command line in order to build one artifact (a program or a library). The second is to set up a full build system is based on the GNU Autotools. Both scenarios will be covered in this chapter.

29.1. Building stand-alone artifacts

Here we describe how to use the Stratego compiler for small projects and for compiling one-off examples on the command-line. We recommend that you use Autotools for larger projects, i.e. when you need to build multiple artifacts (see the next sections).

Invoking the compiler on simple programs which only depend on the Stratego library is straightforward:

$ strc -i myprog.str 

This produces the executable file myprog. When your program depends on other Stratego packages (libraries), you need to provide the compiler with proper include paths (for finding the module definitions) and linking arguments (for linking the libraries with the final executable). For convenience, you should define an alias called strcflags as follows:

$ alias strcflags="pkg-config --variable=strcflags "

By calling strcflags with the the name of a specific package, e.g. java-front, all necessary include paths and library arguments will be provided for you. This gives rise to the following idiom:

$ strc -i myprog.str $(strcflags dryad java-front)

Note that providing several arguments (packages) to strcflags is allowed.

29.1.1. Static linking

By default, the Stratego compiler will dynamically link all libraries. To enable static linking instead, you must add the command line options -Xlinker -all-static:

$ strc -i myprog.str -Xlinker -all-static

This ensures that the myprog executable is statically linked (and therefore has no external dependencies).

29.2. Setting up your Project

Setting up a build system for Stratego involves the Autotool programs automake, autoconf and libtool. In addition, Stratego provides a new tool called autoxt. If you are familiar with the Autotools, setting up a project for Stratego should be rather easy. If this is unfamiliar ground to you, don't fear. We will walk through it slowly in the next sections, but a full treatise is beyond the scope of this tutorial.

After creating your project directory, let's call it ogetarts, the first thing you should do is populate it with the basic build system files, namely configure.ac, bootstrap, Makefile.am. Additionally, you may want to add ChangeLog, AUTHORS, NEWS and README, but these are not essential. If you want to support the creation of RPMs, then you need to create a file ogetarts.spec.in.

For a normal Stratego project, with a syntax, some stand-alone tools, and a library, we suggest the project layout given below (directories end in /). We will discuss the all the components of this hierarchy in turn.

ogetarts/
  bootstrap
  configure.ac
  Makefile.am
  syn/
      Makefile.am
  lib/
      Makefile.am
  tools/
        Makefile.am

The build system is kickstarted by the bootstrap script. This script is responsible for generating a configure script. When run by the user of your package, the configure script will autodetect the Stratego compiler and other tools required to build your project. The concept of generation is very central to autotool-based build systems. The configure script is generated from a configure.ac declaration by the autoreconf tool. The Makefiles are generated from Makefile.in files by the configure script, and Makefile.in files are generated from Makefile.am files by automake. Simple, huh? Generally, the idea is that complicated scripts and makefiles can be generated from high-level declarations using tools. Let's start with the bootstrap script.

#! /bin/sh

autoxt || exit 1
autoreconf -ifv || exit 1

The bootstrap script should be an sh (or bash) shell script that takes care of running autoxt and autoreconf, as shown above. Note that we rely on reasonably recent versions of autoconf and automake.

Assume we are in a palindromic mood and want to name our project Ogetarts. The following file will then provide a reasonable starting point for the configure.ac file.

AC_PREREQ([2.58])
AC_INIT([ogetarts],[0.1],[ogetarts-bugs@ogetarts.org])
AM_INIT_AUTOMAKE([1.7.2 -Wall -Wno-portability foreign])

# set the prefix immediately to the default prefix
test "x$prefix" = xNONE && prefix=$ac_default_prefix

XT_SETUP
XT_USE_XT_PACKAGES

AC_PROG_CC
AC_PROG_LIBTOOL

### OUTPUT #############################
AC_CONFIG_FILES([
  Makefile
  syn/Makefile
  lib/Makefile
  tools/Makefile
])
AC_OUTPUT

Most of this is standard boilerplate. The foreign option specified in the arguments of AM_INIT_AUTOMAKE tells automake not the check that the package conforms to GNU package standards, which requires files such as ChangeLog, AUTHORS, COPYING, NEWS and README. For this small example, we do not want create all these files. You can leave out the foreign option if you want to make a complete GNU package.

The important line is XT_USE_XT_PACKAGES which is a macro invocation that will extend to shell script code that looks for the ATerm library, the SDF tools and Stratego/XT, respectively. These macros are provided by the autoxt tool, via a macro file called autoxt.m4. It provides the following macros.

XT_SETUP

Sets up a Stratego package by setting standard flags of the C compiler and linker for Stratego programs.

XT_USE_XT_PACKAGES

Adds configuration options to configure the package with the location of the ATerm library, SDF2 Bundle and Stratego/XT.

XT_PRE_RELEASE

Adds the suffix pre${SVN_REVISION} to the PACKAGE_VERSION and VERSION variables. This is a naming convention for unstable packages that we are using in our release management system. If you are not building your package in our buildfarm, then you do not need to invoke this macro.

XT_DISABLE_XTC_REGISTER

Disables the creation of an XTC repository. By default all programs and files are registered in an XTC repository.

XT_USE_BOOTSTRAP_XT_PACKAGES

Similar to XT_USE_XT_PACKAGES, this macro adds configuration options to configure the package with the location of the ATerm library, SDF and Stratego/XT. However, the macro will only check the existence of these packages if the option --enable-bootstrap is given to the configure script. In other case, it will only look for the Aterm Library and Stratego Libraries. Also, XTC registration is disabled. This macro is used for packages that need to be very portable, including native Microsoft Windows.

XT_SVN_REVISION

Determines the SVN revision and makes this number available to Stratego programs.

At the end of the configure.ac above, the invocation of the AC_CONFIG_FILES macro lists other important files of the build system, particularly the Makefiles. We must provide these, but remember that these are generated from .in files which in turn come from .am files. Hence, we need to provide some Makefile.am files. The Makefile.am for the root of the project should look like:

include $(top_srcdir)/Makefile.xt

SUBDIRS           = syn lib tools
BOOTCLEAN_SUBDIRS = $(SUBDIRS)
DIST_SUBDIRS      = $(SUBDIRS)
EXTRA_DIST        =

ACLOCAL_AMFLAGS = -I .

Again, most of this is boilerplate. The important point here is that SUBDIRS = syn lib tools will eventually result in rules that tell make to delve into these directories. We will explain below how the Makefile.ams for each of the source directories should look like. For now, you can just create empty Makefile.am files in the sub-directories syn/, lib/, and tools/. This allows you to bootstrap and configure the package:

$ mkdir syn lib tools
$ touch syn/Makefile.am lib/Makefile.am tools/Makefile.am
$ chmod u+x bootstrap
$ ./bootstrap
$ ./configure
$ make

The content of the empty Makefile.am files depends on whether you are building a parser, stand-alone Stratego programs, or a Stratego library. We will discuss each variant separately, but you are of course free to mix several of these in your project, like we do in this project: In lib lives the library parts of Ogetarts, in syn a parser is generated, and in tools we place the command-line programs.

29.3. Building Stand-alone Stratego Applications

In Chapter 11, we showed how to compile stand-alone Stratego programs using the strc compiler. This process is automated by the build system, provided you supply a suitable Makefile.am. Take the one provided below as a starting point.

include $(top_srcdir)/Makefile.xt
include $(wildcard *.dep)

bin_PROGRAMS = ogetarts
ogetarts_LDADD = $(STRATEGO_LIB_LIBS) $(STRATEGO_RUNTIME_LIBS) $(ATERM_LIBS)

STRINCLUDES    = -I $(top_srcdir)/lib -I $(top_srcdir)/syn
STRCFLAGS      = --main io-$*

EXTRA_DIST = $(wildcard *.str) $(wildcard *.meta)
CLEANFILES = $(wildcard *.c) $(wildcard *.dep)

This file should be placed in tools/. The following list explains the various parts occurring in the file.

include $(top_srcdir)/Makefile.xt

Includes the various Stratego/XT make rules in this Makefile, for example for the compilation of Stratego programs and parse tables.

include $(wildcard *.dep)

The Stratego compiler generates .dep files which contain information about module dependencies. When these .dep files are included, a rebuild is forced when a dependent file changes.

bin_PROGRAMS

A list of stand-alone Stratego programs that should be compiled and installed in the directory $prefix/bin. For each program, a corresponding .str file must exist. In this case, the most trivial ogetarts.str module could look like:

module ogetarts
imports libstratego-lib
strategies
  io-ogetarts = io-wrap(id)
program_LDADD

Stratego programs reuse various libraries, such as the ATerm library, the Stratego runtime and the Stratego library. This declaration tells the build system to link the program ogetarts with these libraries. If you use more libraries, such as the library libstratego-sglr for parsing, then you can add $(STRATEGO_SGLR_LIBS). All libraries follow this naming convention, for example the variable $(STRATEGO_XTC_LIBS) is used for the library libstratego-xtc. If you also have your own library in this package, then you can add $(top_builddir)/lib/libogetarts.la.

STRCFLAGS

Contains compiler flags passed to the Stratego compiler. A typical flags is --main. In this case, the declaration tells the Stratego compiler that the main stratego of a program foo is io-foo (instead of the default main). It is recommended that include directories are passed via the STRINCLUDES variable.

STRINCLUDES

A list of additional includes necessary for a succesful compilation, of the form -I <dir>. They will be passed unchanged to the strc compiler. The Stratego compiler will then look in these directories for Stratego modules.

EXTRA_DIST

Specifies which auxilary files have to be included in the distribution, when doing a make dist. In this case, we want to distribute all the Stratego modules and their .meta (which do not always exist)

CLEANFILES

Files to be deletes these files when the make clean command is issued. In this case we instruct automake to remove the .c and .dep files that are generated by the Stratego compiler.

BOOTCLEANFILES

Files to be deleted when make bootclean is issued. In addtion, the files specified in CLEANFILES will also be deleted.

As we can see from the list above, the bin_PROGRAMS is a list of the stand-alone programs that should be compiled in this directory. For each program, a corresponding .str must exist, in this case, a ogetarts.str file. For each program the Stratego compiler will be passed the STRINCLUDES and STRCFLAGS variables. The program_LDADD variable is used to add additional native libraries that should be linked as part of the C compilation process. STRINCLUDES and STRCFLAGS were explained above.

autoxt installs Makefile.xt, a collection of Automake rules for compiling Stratego programs and applying other XT tools, such as signature generation. Using this makefile, a makefile reduces to a declaration of programs to be compiled. The makefile automatically takes care of distributing the generated C code. The specification will only be compiled when it is newer than the C code.

29.4. Building Parse Tables, Tree Grammars and Stratego Signatures

In Chapter 6 we introduced you to the Syntax Definition Formalism SDF for specifying grammars, and showed you how to use pack-sdf and sdf2table to construct parse tables out of these definitions. The grammar can also be used to derive Stratego signatures and regular tree grammars. Not surprisingly, the build system is equipped to do this for you. Consider the Makefile.am provided below.

include $(top_srcdir)/Makefile.xt
include $(wildcard *.dep)

DEF_TBL = Ogetarts.def Ogetarts.tbl
RTG_SIG = Ogetarts.rtg Ogetarts.str

sdfdata_DATA = $(DEF_TBL) $(wildcard *.sdf)
pkgdata_DATA = $(RTG_SIG)

EXTRA_DIST   = $(DEF_TBL) $(RTG_SIG) $(wildcard *.sdf)
CLEANFILES   = $(DEF_TBL) $(RTG_SIG)

SDFINCLUDES   =
SDF2RTG_FLAGS = -m $*
PGEN_FLAGS    = -m $*

This file should be placed in syn/. The following list explains the various parts occurring in the file.

sdfdata_DATA

The important declaration in this file is sdfdata_DATA. They files listed for this variable will be installed as part of a make install into the directory $prefix/share/sdf/$packagename. For convenience, we have defines the .def and .tbl files, in a separate variable DEF_TBL. The build system knows how to generate .def files from .sdf using pack-sdf, and .tbl files from .def files using sdf2table. Note that there must be an SDF module Ogetarts.sdf, which is the main module of the syntax definition. For example:

module Ogetarts
hiddens
  context-free start-symbols Expr
exports
  context-free syntax
    IntConst -> Expr {cons("Int")}

  context-free priorities
      Expr "*" Expr -> Expr {left, cons("Mul")}
    > Expr "+" Expr -> Expr {left, cons("Plus")}

  lexical syntax
    [0-9]+ -> IntConst
    [\t\n\r\ ] -> LAYOUT
pkgdata_DATA

This variable defines a list of files that will be placed in prefix/share/package-name. This typically includes regular tree grammars, Stratego signatures and Stratego source code for libraries. In this makefile, we use pkgdata_DATA to tell the build system to build a Stratego signature (Ogetarts.str) and a regular tree grammar (Ogetarts.rtg) from the Ogetarts.def. Including signatures and source code with your program is useful when you want other projects to extend and compile against yours.

EXTRA_DIST

Similar to the makfile for building Stratego programs, we use EXTRA_DIST to define the files to distribute. In this case, we also distribute the derived .def, .tbl, .rtg, and .str, which is used to avoid a dependency on the full Stratego/XT. If this is not required, then we can leave these files out of the distribution.

CLEANFILES

In CLEANFILES we specify that we want make to remove the generated files when running a make clean.

PGEN_FLAGS

This variable is used to pass flags to the parsetable generator, sdf2table. The definition in this makefile defines the main module to be $*, which is the basename of the .def file that is used as the input to the parser generator. This is typical for most makefiles, since the default main module is Main.

SDF2RTG_FLAGS

Similar to PGEN_FLAGS, this variable is used to pass flags to the tool sdf2rtg, which generates a regular tree grammar for an SDF syntax definition. Again, we define the main module of the syntax definition to be the basename of the file.

SDFINCLUDES

Similar to SDFINCLUDES, you can define directories to search for SDF modules using the SDFINCLUDES variable. In addition to the option -I dir you can also include modules from an SDF definition: -Idef file.def.

29.5. Building Your Own Stratego Library

Stratego allows you to freely organize the Stratego modules of your project, but we recommend to have a separate library in the directory lib/. Each .str file placed inside this directory becomes a module for your library. For sufficiently large projects, it is recommended that you further organize your modules into subdirectories inside lib/. Each such subdirectory becomes a package.

Importing modules and packages in Stratego is done using the imports statement. Using imports, you specify which module from which package to import. See Chapter 11 for an introduction to modules. There is a direct mapping between the directory name and the package name, and also between file and module names. Consider the following main module lib/ogetarts.str of your library, which imports all the Stratego modules that constitute the library:

module ogetarts
imports
  ogetarts/main
  ogetarts/front/parse

The import declaration ogetarts/main states that we want to import the module main from the package ogetarts. This tells the Stratego compiler to look for the file main.str inside the ogetarts/ directory. The line ogetarts/front/parse will import the file parse.str in the ogetarts/extensions package. On disk, we will have the following layout:

lib/
    ogetarts.str
    ogetarts/
             main.str
             front/
                   parse.str

In this example, we will assume that the module ogetarts/main provides a simple evaluator for arithmetic expressions:

module ogetarts/main
imports libstratego-lib Ogetarts
strategies

  ogetarts-eval = bottomup(try(Eval))

rules

  Eval : Plus(Int(i), Int(j)) -> Int(<addS> (i, j))
  Eval : Mul(Int(i), Int(j))  -> Int(<mulS> (i, j))

The module ogetarts/front/parse provides a strategy for parsing ogetarts expressions:

module ogetarts/front/parse
imports libstratego-sglr libstratego-lib
strategies

  parse-ogetarts-stream =
    where(tbl := <import-term(Ogetarts.tbl); open-parse-table>)
    ; finally(
        parse-stream(strsglr-report-parse-error | tbl)
      , <close-parse-table> tbl
      )

So how does the compiler know where to search for the packages (directories)? This is specified by the -I option to strc. In our case, we did already specify lib/ as the basepath for our library in the section on compiling Stratego programs. Thus, all module and package references are looked up from there. For programs using several libraries installed at different locations, multiple base directories should be specified, each with the -I option.

29.5.1. Compiling the Library

In principle, it is possible to import full source of your library in your Stratego programs. In that case, there is no need to compile the library separately: it will be compiled by the Stratego compiler every time the library is included in a Stratego program. The compiler will act as a whole-program compiler and compile all the source code from scratch, including your library sources. This is wasteful, since the library is recompiled needlessly. To avoid this, the build system can be told to compile the library separately to a native library, e.g. a .so file. The creation of the native library is done using libtool, which takes care of creating both static and shared libraries. On most platforms, the linker prefers shared libraries over static libraries, given the choice. This means that linking to your Stratego library will be done dynamically, unless you or your library user take steps to enable static linking.

The code below is what goes into your Makefile.am for you library:

include $(top_srcdir)/Makefile.xt
include $(wildcard *.dep)

lib_LTLIBRARIES = libogetarts.la 
pkgdata_DATA    = libogetarts.rtree
EXTRA_DIST      = $(ogetartssrc) libogetarts.rtree
CLEANFILES      = libogetarts.c libogetarts.rtree

libogetarts_la_SOURCES  = libogetarts.c
libogetarts_la_LDFLAGS  = -avoid-version -no-undefined
libogetarts_la_CPPFLAGS = \
  $(STRATEGO_LIB_CFLAGS) $(STRATEGO_RUNTIME_CFLAGS) $(ATERM_CFLAGS)
libogetarts_la_LIBADD   = \
  $(STRATEGO_SGLR_LIBS) $(STRATEGO_LIB_LIBS) $(STRATEGO_RUNTIME_LIBS) $(ATERM_LIBS)

ogetartssrc = \
  ogetarts.str \
  $(wildcard $(srcdir)/ogetarts/*.str) \
  $(wildcard $(srcdir)/ogetarts/front/*.str)

STRINCLUDES = -I $(top_srcdir)/syn

libogetarts.c libogetarts.rtree : $(ogetartssrc)
    @$(STRC)/bin/strc -c --library -i ogetarts.str -o libogetarts.rtree $(STRINCLUDES)
    rm libogetarts.str
lib_LTLIBRARIES

pkgdata_DATA

libogetarts_LDFLAGS

libogetarts_CPPLAGS

libogetarts_LIBADD

Necessary for some platforms. Note that we add STRATEGO_SGLR_LIBS, since our module is importing libstratego-sglr.

STRINCLUDES

For finding the signature and parsetable

libogetarts.c : ...

Warning: use tab.

If you prefer static over dynamic libraries, you can enforce static linking by passing the -static option to the configure of your package via the LDFLAGS variable:

$ ./configure ... LDFLAGS=-static

29.5.2. Using Your Library in Stratego Programs

Add a program to the tools directory

module ogetarts-eval
imports
  libstratego-lib
  libogetarts

strategies

  io-ogetarts-eval =
    io-stream-wrap(
      ?(<id>, fout)
      ; parse-ogetarts-stream
      ; ogetarts-eval
      ; ?Int(<id>)
      ; <fputs> (<id>, fout); <fputs> ("\n", fout)
    )

Add to tools/Makefile.am:

bin_PROGRAMS = ogetarts-eval

ogetarts_eval_LDADD = \
  $(top_builddir)/lib/libogetarts.la \
  $(STRATEGO_LIB_LIBS) $(STRATEGO_RUNTIME_LIBS) $(ATERM_LIBS)

Invoke the eval tool:

$ echo "1+2" | ./tools/ogetarts-eval

29.6. Package Config Support

How to add a pkg-config file to the package.

29.7. RPM Support

Let's finish up the root build system first, closing with the ogetarts.spec.in file.

Summary: ogetarts
Name: @PACKAGE_TARNAME@
Version: @PACKAGE_VERSION@
Release: 1
License: LGPL
Group: Development/Tools/Ogetarts
URL: http://www.ogetarts.org/Ogetarts
Source: @PACKAGE_TARNAME@-@PACKAGE_VERSION@.tar.gz
BuildRoot: %{_tmppath}/%{name}-@PACKAGE_VERSION@-buildroot
Requires: aterm >= 2.5
Requires: sdf2-bundle >= 2.4
Requires: strategoxt >= 0.17
Provides: %{name} = %{version}

%description

%prep
%setup -q

%build
CFLAGS="-D__NO_CTYPE" ./configure --prefix=%{_prefix} --with-strategoxt=%{_prefix} \
       --with-aterm=%{_prefix} --with-sdf=%{_prefix}
make

%install
rm -rf $RPM_BUILD_ROOT
make DESTDIR=$RPM_BUILD_ROOT install

%clean
rm -rf $RPM_BUILD_ROOT

%files
%defattr(-,root,root,-)
%{_bindir}
%{_libexecdir}
%{_datadir}
%doc

%changelog

This file is not strictly necessary. It's a so-called .spec-file, which is a package descriptor for rpm-based distributions. When you provide it, the Stratego/XT build system can automatically make a .rpm package file for your project.

29.8. Summary

In this chapter we learned how to organize Stratego/XT projects, and how to set up the Stratego/XT build system. We saw how the build system, which is based on automake and autoconf, is used to generate many of the artifacts from the syntax definition, such as parse tables and signatures, in the way we discussed in Part II . We also learned how to make stand-alone programs, libraries and XT components from Stratego source code, and how modules and packages relate to files and directories.

More detail can be found by looking at the manual pages for the build tools, such as autoxt, strc, sdf2table, sdf2rtg and xtc. The detailed examples from the Stratego/XT Examples tutorial contain working build systems that can serve as a starting point.

Chapter 30. Debugging Techniques for Stratego/XT

Even in Stratego/XT, it is not entirely uncommon for developers to produce erroneous code from time to time. This chapter will walk you through the tools and techniques available for hunting down bugs, and tips for how to avoid them in the first place.

30.1. Debugging Stratego

Both the Stratego language paradigm and its syntax are rather different from most other languages. Knowing how to use the unique features of Stratego properly, in the way we have described in this manual, goes a long way towards avoiding future maintenance problems.

30.1.1. Writing readable code

One important practical aspect of using language constructs is expressing their syntax in a readable manner. The intention behind the code should be apparent for readers of the code. Judicious use of whitespaces is vital in making Stratego code readable, partly because its language constructs are less block-oriented than most Algol-derivates.

The most basic unit of transformation in Stratego is the rewrite rule. The following suggests how rules should be written to maximize readability.

  EvalExpr:
  Expr(Plus(x), Plus(y) -> Expr(z)
  where
    <addS> (x,y) => z

Rules are composed using combinators in strategies. One of the combinators is composition, written ;. It is important to realize that ; is not a statement terminator, as found in imperative languages. Therefore, we suggest writing a series of strategies and rules joined by composition as follows:

  
  eval-all =
      EvalExpr
    ; s1
    ; s2

Both rules and strategies should be documented, using xDoc. At the very least, the type of term expected and the type of term returned should be specified by the @type attribute. Also take care to specify the arity of lists and tuples, if this is fixed.

 /**
  * @type A -> B
  */
  foo = ...

Inline rules are handy syntactic sugar that should be used with care. Mostly, inline rules are small enough to fit a single line. When they are significantly longer than one line, it is recommended to extract them into a separate, named rule.

  strat = 
    \ x -> y where is-okay(|x) => y \

Formatting concrete syntax depends very much on the language being embedded, so we will provide no hard and fast rules for how to do this.

Formatting of large terms should be done in the style output by pp-aterm.

30.1.2. Debugging Stratego code

The Stratego/XT environment does not feature a proper debugger yet, so the best low-level debugging aids are provided by the library, in the from of two kinds of strategies, namely debug and a family based around log.

The debug strategy will print the current term to stdout. It will not alter anything. While hunting down a bug in your code, it is common to sprinkle debug statements liberally in areas of code which are suspect:

  foo = 
      debug
    ; bar
    ; debug
    ; baz

Sometimes, you need to add additional text to your output, or do additional formatting. In this case, an idiom with where and id is used:

  foo = 
     where(<debug> [ "Entered foo : ", <id> ])
     ; bar
     ; where(<debug> [ "After bar : ", <id> ])
     ; baz

The where prevents the current term from being altered by the construction of your debugging text, and id is used to retrieve the current term before the where clause. If, as in this example, you only need to prepend a string before the current term, you should rather use debug(s), as shown next.

  foo =
    debug(!"Entered foo : ")
    ; bar
    ; debug(!"After bar : ")
    ; baz

The use of debug is an effective, but very intrusive approach. A more disciplined regime has been built on top of the log(|severity, msg) and lognl(|severity, msg) strategies. (See Chapter 25 for details on log and lognl). The higher-level strategies you should focus on are fatal-err-msg(|msg), err-msg(|msg), warn-msg(|msg) and notice-msg(|msg).

It is recommended that you insert calls to these strategies at places where your code detects potential and actual problems. During normal execution of your program, the output from the various -msg strategies is silenced. Provided you allow Stratego to deal with the I/O and command line options, as explained in Chapter 26, the user (or the developer doing debugging) can use the --verbose option to adjust which messages he wants to be printed as part of program execution. This way, you get adjustable levels of tracing output, without having to change your code by inserting and removing calls to debug, and without having to recompile.

30.1.3. Common Pitfalls

Some types of errors seem to be more common than others. Awareness of these will help you avoid them in your code.

Strategy and Rule Overloading.  The way Stratego invokes strategies and rules may be a bit unconventional to some people. We have already seen that the language allows overloading on name, i.e. you can have multiple strategies with the same name, and also multiple rules with the same name. You can even have rules and strategies which share a common name. When invoking a name, say s, all rules and strategies with that name will be considered. During execution the alternatives are tried in some order, until one succeeds. The language does not specify the order which the alternatives will be tried.

 Eval:
 If(t, e1, e2) -> If(t, e1', e2')
 where
     <simplify> e1 => e1'
   ; <simplify> e2 => e2'
 
 Eval:
 If(False, e1, e2) -> e2

When Eval is called, execution may never end up in the second case, even though it the current term is an If term, with the condition subterm being just the False term.

If you want to control the order in which a set of rules should be tried, you must name each alternative rule differently, and place them behind a strategy that specifies the priority, e.g:

  SimplifyIf
  If(t, e1, e2) -> If(t, e1', e2')
  where
     <simplify> e1 => e1'
   ; <simplify> e2 => e2'

  EvalIfCond:
  If(False, e1, e2) -> e2
  
  Eval = EvalIfCond <+ SimplifyIf

Combinator Precedence.  The precedence of the composition operator (;) is higher than that of the choice operators (<+,+, >+). This means that the expression s1 < s2 ; s3 should be read as s1 < (s2 ; s3), and similarly for non-deterministic choice (+) and right choice (>+). See Section 15.3 for a more detailed treatment.

Guarded Choice vs if-then-else The difference between if s1 then s2 else s3 end and s1 < s2 + s3 (guarded choice) is whether or not the result after s1 is passed on to the branches. For if-then-else, s2 (or s3) will be applied to the original term, that is, the effects of s1 are unrolled before proceeding to the branches. With the guarded choice, this unrolling does not happen. Refer to Section 15.3.2 for details.

Variable Scoping.  Stratego enforces a functional style, with scoped variables. Once a variable has been initialized to a value inside a given scope, it cannot be changed. Variables are immutable. Any attempt at changing the value inside this scope will result in a failure. This is generally a Good Thing, but may at times be the cause of subtle coding errors. Consider the code below:

stratego> <map(\ x -> y where !x => y \)> [1]
[1]
stratego> <map(\ x -> y where !x => y \)> [1,1,1,1]
[1,1,1,1]
stratego> <map(\ x -> y where !x => y \)> [1,2,3,4]
command failed

Apparently, the map expression works for a singleton list, a list with all equal elements, but not lists with four different elements. Why? Let us break this conondrum into pieces and attack it piece by piece.

First, the inline rule \ x -> y where !x => y \ will be applied to each element in the list, by map. For each element, it will bind x to the element, then build x and assign the result to y. Thus, for each element in the list, we will assign this element to y. This explains why it works for lists with only one element; we never reassign to y. But why does it work for lists of four equal elements? Because the rule about immutability is not violated: we do not change the value of y by reassigning the same value to it, so Stratego allows us to do this.

But why does this happen? We clearly stated that we want a local rule here. The gotcha is that Stratego separates control of scopes from the local rules. A separate scoping construct, {y: s} must be used to control the scoping of variables. If no scoping has been specified, the scope of a variable will be that of its enclosing named strategy. Thus, the code above must be written:

stratego> <map({y: \ x -> y where !x => y \})> [1,2,3,4]
[1,2,3,4]

It may be a bit surprising that this works. We have not said anything about x, so logically, we should not be able to change this variable either. The difference between x and y is that x is a pattern variable. Its lifetime is restricted to the local rule. At first glance, this may seem a bit arbitrary, but after you code a bit of Stratego, it will quickly feel natural.

30.2. Debugging XT compositions

The XT component model is based on Unix pipes. Debugging XT compositions can therefore be done using many of the familiar Unix command line tools.

Checking XTC registrations.  Whenever you call XTC components using xtc-transform, the location of the component you are calling is looked up in a component registry. When invoking a component fails, it may be because the component you are calling has been removed. Checking the registrations inside a component registry is done using the xtc command:

# xtc -r /usr/local/apps/dryad/share/dryad/XTC q -a
dryad (0.1pre11840) : /usr/local/apps/dryad/dryad
dryad.m4 (0.1pre11840) : /usr/local/apps/dryad/share/dryad/dryad.m4
...

The -r option is used to specify which registry you want to inspect. The path given to -r must be the XTC registry file of your installed program transformation system that you built with Stratego/XT. By default, xtc will work on the Stratego/XT XTC repository, and only list the components provided by Stratego/XT. This is seldom what you want.

XTC registries are hierarchical. The XTC repository of your project imports (refers back to) the other projects you used in your build process, such as Stratego/XT itself. The component list you get from xtc when giving it your repository is therefore a full closure of all components visible to transformations in your project.

Now that you know how to obtain the paths for all XT components, it is easy to determine that they actually exist at the locations recorded, and that the access rights are correct.

Programs such as strace may also be useful at the lowest level of debugging, to see which parameters are passed between components, whether a given component is located correctly, and whether execution of a given component succeeds.

Format Checking.  Each component in a system built with Stratego/XT accepts a term, definable by some grammar, and outputs another term, also definable by a (possibly the same) grammar. During debugging of XT compositions, it is useful to check that the data flowing between the various components actually conform to the defined grammars. It is not always the case that the grammar in question has been defined, but you are highly encouraged to do so, see Chapter 8 for how to define regular tree grammars.

Once you have a formal declaration of your data, in the form of a regular tree grammar, you can insert calls to the format-check between your XT components to verify data correctness, i.e. the correctness of the terms.

  ast2il = 
      xtc-transform(!"format-check", !["--rtg", "language-ast.rtg"])
    ; xtc-transform(!"ast2il")
    ; xtc-transform(!"format-check", !["--rtg", "language-il.rtg"])

The ast2il component transforms from the abstract syntax tree representation of a given language to an intermediate language (IL). format-check is used to verify that the AST passed to ast2il is well-formed, and that the result obtained from ast2il is also well-formed.

Tool Debugging Options.  Most of the XT tools accept a common set of options useful when debugging. These include --keep, for adjusting the amount of intermediate results you want to keep as separate files on disk after transformation, --verbose for adjusting the level of debugging information printed by the tool, and --statistics for displaying runtime statistics.

30.3. Debugging SDF definitions

The SDF toolkit comes with some very useful debugging aids. The first is the sdfchecker command, which will analyze your SDF definition and offer a list of issues it finds. You do not need to invoke sdfchecker directly. It is invoked by the sdf2table by default, whenever you generate a parse table from a syntax definition. Be advised that the issues pointed to by sdfchecker are not always errors. Nontheless, it is usually prudent to fix them.

The other SDF debugging tool is the visamb command. visamb is used to display ambiguities in parse trees. Its usage is detailed in the command reference (visamb).

Pitfalls with Concrete Syntax.  Doing transformations with concrete syntax in Stratego, as explained in Chapter 19 depends in the correct placement of .meta files. When creating, splitting, moving or removing Stratego source files (.str files), it is important that you bring along the accompanying .meta files.

Another thing to be aware of with concrete syntax, is the presence of reserved meta variables. Typically, x, xs, e, t and f have a reserved meaning inside the concrete syntax fragments as being meta variables, i.e. variables in the Stratego language, not in the object language.

A final stumbling block is the general problem of ambiguities in the syntax definition. While SDF allows you to write ambiguous grammars, and sglr accepts these gracefully, you are not allowed to have ambiguous syntax fragments in your Stratego code. In cases where the Stratego compiler (strc) fails due to ambiguous fragments, you can run parse-stratego on your source code to see exactly which parts are ambiguous. The visamb tool should then be applied to the output from parse-stratego to visualize the ambiguities.