Part 3 will be a general post about declarative systems, not directly related to build automation. Part 4 should be about auto-generating the make files (which is part of the motivation for writing about declarative systems first).

Fundamentals

The original “insight” of make is that whatever we want executed can be considered a goal and:

1. Each goal is represented by exactly one file.
2. Each dependency of a goal is itself a goal.
3. A goal is outdated when the represented file does not exist or is older than at least one of its depenencies.
4. A goal can be brought up-to-date by one or more shell commands.

This is all there is to it. By linking the goals (via depenencies) we get the aforementioned DAG, and with this simple data structure we can model all our processes as long as the four criteria above are met, which they generally are, at least on unix where “everything is a file” :)

Extending the Graph

One of the reasons I like to view the process as a directed graph is that it becomes easy to see how we need to “patch” it to add our own actions. Yes, I said patch, because we can actually do that, and quite easily, even if we can’t edit the original make file.

Imagine we are building Lunettes (a new UI for the VLC media player) which depends on VLCKit.

Considering the graph there must be some goal of Lunettes that depend on the VLCKit, in Makefile syntax this could simply be:

APP_DST=Lunettes.app/Contents

$(APP_DST)/MacOS/Lunettes: $(APP_DST)/Frameworks/VLCKit.framework

This syntax establish a connection (dependency) between the executable and the framework. Here I made it depend on the framework’s root directory, of course it should depend on the actual binary in the framework (but then my box will overflow).

What this means is that each time the framework is updated, the executable is considered out-of-date and as a result, will be relinked (with the updated framework).

Unit Tests

The reason I mentioned the above link between the application and its framework is because this is where we want to insert new nodes (goals) in the graph incase we want to add unit tests to the VLCKit framework.

So the scenario is this: We write a bunch of unit tests for the VLCKit framework and we want these to run every single time the framework is updated, not only when we feel like it, but at the same time, since we probably spend most time developing on the application itself, we do not want the tests to run each time we do a build.

What we do is mind-boggling simple, we introduce a file to represent the unit test goal and we touch this each time the test has been successfully run:

vlckit_test: $(APP_DST)/Frameworks/VLCKit.framework
    if «run test»; then touch '$@'; else false; fi

We can now make vlckit_test to run the test, and if the test has been run (succesfully) after last build of the framework, then it will just tell us that the goal is up-to-date.

To avoid running this manually, we add the following to our make file:

$(APP_DST)/MacOS/Lunettes: vlckit_test

Now our application depends on having succesfully run the unit test for the used framework.

This is all done without touching any of the existing build files, we simply extend the build graph with our new actions.

And the result is IMO beautiful in the sense that the unit tests are only run when we actually change the framework, and failed unit tests will cause the entire build to fail.

As a reader exercise, go download the actual build files of the Lunettes / VLCKit project (much of it is in Xcode) and add something similar. What you will end up with is Xcode’s answer to the problem of extensibility: “custom shell script target” which will run every single time you re-build your target, regardless of whether or not there actually is a need for it.

This might be ok if you only have one thing that falls outside what the system was designed to handle, but when you have half a dozen of these…

Build Numbers

Another common build action these days is automated build numbers. Say we are going to do nightly builds of Lunettes and want to put the git revision into the CFBundleVersion.

You remember how everything is a file on unix? To my great delight, git conforms quite well to this paradigm and we can find the current revision as .git/HEAD, although this file contains a reference to the symbolic head which likely is .git/refs/heads/master.

For simplicity let us just assume we always stay on master (and we don’t create packs for the heads). The file is updated each time we make a commit, bumping its date, so all we need to do is have our Info.plist depend on .git/refs/heads/master and let the action to bring Info.plist up-to-date insert the current revision as value for the CFBundleVersion key.

Again make’s simple axiomatic system makes it a breeze to do this, and “do it right”, that is, do it in a way that limits computation to the theoretical minimal, rather than update the Info.plist with every single build or require it to be manually updated.

External Dependencies

I have used Lunettes as example in this post so let me continue and link to the build instructions.

Here you see several steps you have to do in order to get a succesful build, additionally if you look in the frameworks directory of Lunettes you’ll find that it deep-copied these from other projects.

Since every single person who wants to build this has to go through these steps, we should incorporate it in the build process, and it is actually quite simple (had this project been based on make files), for example we need to clone and build the VLC project which can be done using:

vendor/vlc:
    git clone git://git.videolan.org/vlc.git '$@'
    $(MAKE) -sC '$@'

So if there is no vendor/vlc then we do a git checkout and call make afterwards. In theory we can also include the make file from this project so that we can do fine-grained dependencies, but since this is not our project we do not have control over its make file and can’t fix any potential clashes, so it’s safer to simply call make recursively on the checked out project.

We need to setup a link between Lunettes and vendor/vlc so that the checkout will actually be done (without having to make vendor/vlc), but that is just a single line in our make file.

Other Actions

If it isn’t clear by now, make files is what drives my own build process when I build TextMate. I run the build from TextMate itself, and the goal I ask to build is relaunching TextMate on a successful build.

This isn’t always desired, as I am actually using the application when it happens, so what I have done is rather simple and mimics the unit test injection shown above.

Let me start by quoting from my make file:

$(APP_NAME)/run: ask_to_relaunch

ask_to_relaunch: $(APP_PATH)/Contents/MacOS/$(APP_NAME)
    @[[ $$("$$DIALOG" alert …|pl) = *"buttonClicked = 0"* ]]

.PHONY: ask_to_relaunch

This introduces a new goal (ask_to_relaunch), it is declared “phony” so it is not backed by a file on disk (and therefor, always considered outdated). It depens on the actual application binary, so it will never be updated before the application has been fully built.

I use phony goals like «app»/run, «app»/debug and similar. When I build from within TextMate it is the «app»/run goal that I build, and I have set this to depend on my (phony) ask_to_relaunch goal.

As this goal is always outdated, it will run the (shell) command to bring it up-to-date. The shell command opens a dialog (via the "$DIALOG" alert system) which asks whether or not to relaunch. If the user cancels the dialog, the shell command will return a non-zero return code and make will treat that as having failed updating the ask_to_relaunch goal which in turn will cause the «app»/run goal to never be updated (have its (shell) commands executed), as one of its dependencies failed.

Simple yet effective.

Conclusion

This has just been a bunch of examples, what I hope to have shown is how simple the basic concept of make is, how easy it is to extend an existing build process, and how flexibile make is in what it can actually do for us.

Of the many build systems I have looked at, I don’t see anything which has this simple axiomatic definition nor is actually very versatile. A lot of build systems have been created because make files are ugly/complex/arcane/etc., and I agree with that sentiment, but it seems like many of the replacements are systems hardcoded for specific purposes which simplify the boilerplate but make them inflexibile, or they are actual programming languages, which makes the build script only marginally better than a custom script, for example some, but not all, of the systems which takes the “programming language route” lack the ability to execute tasks in parallel, which, with 16 cores and counting, is a pretty fatal design limitation.

I have looked at a lot of build systems myself, and while I agree that the best build system is the one you create yourself I am also a big fan of make and believe that the best approach is to use generated Makefiles.

This post is a “getting started with make”. I plan to follow up with a part 2 about how to handle auto-generated self-updating Makefiles.

Concept

The UNIX philosophy is to have small tools (commands) which solve a well defined problem. These can then be combined to build more complex systems.

While each build process is different, the common denominator is that we should be able to represent our target(s) as nodes in a directed acyclic graph where each node represents a file and each edge represents a dependency.

This is what a Makefile captures, i.e. a Makefile should be a declaration of the dependency graph with actions per node to create it if (the file it corresponds to on disk) is missing or older than its dependencies, i.e. the nodes we can reach from the (directed) edges.

By keeping the dependency information declarative we let make figure out which files are outdated and need to be rebuilt plus give it freedom to pick a strategy to rebuild files which may include running jobs in parallel.

Example

To give an example let us look at the generate_keys script which is part of Sparkle and can generate a public and private key file.

The public key is extracted from the private key and the private key requires a DSA parameter file (we’ll ignore the -genkey flag to dsaparam).

So our (simple) graph looks like this:

pubkey → privkey → dsa_parameters

A Makefile “rule” is effectively one node in our graph and looks like:

«goal»: «dependencies»
    «action»

Here «goal» is the node itself, that is, the file it represents. The «dependencies» is the nodes it depends on and «action» is the command(s) to execute to generate/update the node/file (interpreted by the shell).

Using the generate_keys script as source our Makefile ends up like this:

pubkey: privkey
    openssl dsa -in '$<' -pubout -out '$@'

privkey: dsa_parameters
    openssl gendsa '$<' -out '$@'

dsa_parameters:
    openssl dsaparam 2048 < /dev/urandom -out '$@'

In the above I have used two variables. The variable $@ expands to the goal (i.e. the file we are generating) and $< expands to the first dependency.

If you save the above as Makefile and run make then it will generate 3 files: pubkey, privkey, and dsa_parameters. By default calling make without arguments will ensure the first goal in the Makefile is up to date. If you re-run make it should say:

make: `pubkey' is up to date.

You can also run make privkey to ensure (only) privkey is up to date (which then won’t extract the public key).

Intermediate Files

The above Makefile reproduce the script except that we are not removing the temporary dsa_parameters file after having generated the keys. We can fix this by making dsa_parameters a dependency of the fake .INTERMEDIATE goal by adding this line:

.INTERMEDIATE: dsa_parameters

If we now run make it will automatically remove the dsa_parameters file after it has been used.

We probably want to use our public key from C so let us add another goal (node) namely pubkey.h. This goal will create a C header from the pubkey file, so it will depend on it. This goal can be handled by adding the following rule:

pubkey.h: pubkey
    { echo 'static char const* pubkey ='; \
      sed < '$<' -e $$'s/.*/\t"&\\\\n"/'; \
      echo ';'; } > '$@'

Perhaps not the nicest way to generate the pubkey.h file but what is nice about this is that whatever application needs to use this header can declare it as a dependency, and it will be generated when needed, including extracting the public key if not already done.

Includes

To keep things modular we can save our Makefile as Makefile.keys and include it from our main Makefile using:

include Makefile.keys

If we go back to the Sparkle distribution there is also a sign_update script which signs an update using the private key.

We can add this as another goal to our Makefile, e.g. using:

archive.sig: privkey archive.tbz
    openssl dgst -dss1 -sign privkey archive.tbz

Here the archive signature depends on both having a private key and an archive. The private key will be generated if not already there, the archive we of course need to add another goal to create. The archive goal will depend on our actual binary which will depend on its object files which will depend on the sources (where one source is likely going to depend on pubkey.h).

Phony Targets

In addition we probably want to add another goal to construct an RSS feed (or similar) which include the archive signature and eventually we will want a deploy goal which will depend on the RSS feed and the archive. The action for this goal will likely be using scp to copy the files to the server and the goal itself will not be a file, i.e. when we run make deploy we do not expect an actual deploy file to be generated. While there is little harm in declaring a goal with actions that do not generate the file, we could risk getting a:

make: `deploy' is up to date.

If there actually is a deploy file which is newer then the dependencies of the deploy goal. To avoid this we make the fake goal named .PHONY depend on deploy similar to what we did with the .INTERMEDIATE goal:

.PHONY: deploy

Closing Words

This post is just a mild introduction to make. I have deliberately picked something that does not involve building C sources as the example to show that make is a versatile tool.

Whenever you have a set of actions that need to be run in a specific order then consider if a Makefile can capture the dependency graph.

When you do write a Makefile aim for having a rule only do one thing. For example imagine we are writing a manual and store each chapter as Markdown. Rather than do something like this:

chapter.html: header.html chapter.mdown footer.html
    { cat header.html; \
      Markdown.pl < chapter.mdown; \
      cat footer.html } > '$@'

We can instead do:

chapter.html: header.html cache/chapter.html footer.html
    cat > '$@' $^

cache/chapter.html: chapter.mdown
    Markdown.pl < '$<' > '$@'

The new $^ variable expands to all the dependencies.

There are a few reasons to favor this approach. In this concrete example we have the advantage of not needing to pipe all the chapters through Markdown.pl if we change the header or footer. But in general it just makes things more flexible, easier to re-use goals, faster to restart a failed build, it may improve the number of jobs that can run in parallel, etc.

Few need to implement their own self-balancing trees, but since two previous comments referred to AVL and red/black trees respectively, I should give a shout-out to Arne Andersson and his paper titled Binary Search Trees Made Simple (PDF).

The paper introduces AA trees which are simple to implement but understanding the logic for when to skew/rotate is not clear from the paper. Julienne Walker filled that hole with a great tutorial about AA trees and how they (like red/black trees) stem from B-trees of order 3 (PDF).

The fallback strategy can affect lookup time since we need to do the same probing when a lookup results in an entry with wrong key, turning the nice O(1) time complexity into (worst case) O(n).

Of course the O(n) time is pessimistic as we will rehash to a larger table size when we reach a certain threshold, though from a theoretical point of view an intriguing approach to handling collisions is cuckoo hashing which guarantees O(1) lookup time (insertion can still be worse).

Quoting the Wikipedia page:

The basic idea is to use two hash functions instead of only one. This provides two possible locations in the hash table for each key.

When a new key is inserted, a greedy algorithm is used: The new key is inserted in one of its two possible locations, “kicking out”, that is, displacing, any key that might already reside in this location. This displaced key is then inserted in its alternative location, again kicking out any key that might reside there, until a vacant position is found, or the procedure enters an infinite loop. In the latter case, the hash table is rebuilt in-place using new hash functions.

This means that no additional probing is required during lookup as an element will always be in one of the two slots given by the hash functions (if it is in the table).

In practice linear probing with proper thresholds and a good hash function may perform better (due to locality of reference) plus insertion using cuckoo hashing can be more expensive (as we do more memory writes on collisions than linear probing), still, I love the theoretical property of this collision strategy :)

Though for version 2.0 I want it to do a richer layout, e.g. larger headings in markup languages, indented soft wrap, proper support for unicode, etc. So I had to bite the bullet and figure out how to allow this with reasonable performance, this article explains the problem and data structure I picked.

Problem

The main problem is variable height of lines. If we make a change to line 3,128 then what pixel position does that correspond to (for redrawing) when lines preceding it can have an arbitrary height?

Naive Solution Methods

Two solution methods exist with different characteristics:

1. Use an array of lines where each line knows its Y position.
2. Use a linked list of lines and let each line know its height.

If we pick the first we can find lines and line positions in constant time, but changing a line’s height or inserting/removing lines is linear in time.

Solution number two allows us to insert/remove lines in constant time (if we know where to insert) and also to update a line’s height in constant time. Finding the position of a line (or the nth line) is however linear in time.

Pick Your Poison

Both the solution methods are unusable as explained above, but unlike the first one, the latter one has potential. The reason for this is that the first solution method requires the entire data structure to be updated, the data structure is invalid if we do not perform this update, it is unlikely we will be able to improve on that (and updating is a common operation).

The latter solution method has expensive queries, but there is almost always a way to speedup a query, for example by using a cache.

Optimizing Queries

If we go with the second solution method our problem is now how to speedup queries in a linked list.

One way to improve it is by using an index, for example a skip lists. We have two different query keys, line number and Y position. I.e. given a layout, we may want to ask for the node corresponding to line number 32, or we may want to ask which node is at Y position 27,423.

For now, let us only focus on the first type of queries, so query key is the line number. The problem is the same because if we create an index for the line number and insert a new line, our line numbers change, and the index becomes void.

Rather than work with a skip list, let us use a binary search tree and let me draw how a (perfectly balanced) one looks for a layout with 7 lines:

     4
   /   \
  2     6
 / \   / \
1   3 5   7

So with this tree we can find the node for line 3 by first visiting node 4 and 2.

If we insert a new line after line 2 then it becomes:

     4
   /   \
  2     6
 / \   / \
1   3 5   7
   /
  ×

After the insertion all line numbers (after line 2) have changed, so we need to update the keys in our search tree:

     5
   /   \
  2     7
 / \   / \
1   4 6   8
   /
  3

So we are back to the original problem of having to update the entire data structure. But there is a way around this.

Instead of storing the actual line number in each node, we store number of nodes in the sub-tree rooted at the node in question (which I think is called a finger tree), so the initial tree instead becomes:

     7
   /   \
  3     3
 / \   / \
1   1 1   1

The way we use this tree is slightly different than when we had the actual line numbers.

We need to always start at the root and set: skipped_lines = 0.

Whenever we go right in the tree, we add the number of children in the left subtree plus one (for the node itself): skipped_lines += node.left.count + 1.

The line number for a node is then: skipped_lines + node.left.count + 1.

So the transformation has not made queries more expensive. Let’s now look at an insertion (again after line 2):

     7
   /   \
  3     3
 / \   / \
1   1 1   1
   /
  ×

The only nodes which need updating are those on the path from the root down to the inserted node, i.e. we never update more nodes than the height of the tree which is log(n) (assuming our binary search tree is self-balancing), the updated tree looks like this:

     8
   /   \
  4     3
 / \   / \
1   2 1   1
   /
  1

Back to Pixels

We have solved the problem of storing n items in a binary search tree and finding items based on their index (line number) in log(n) time (which is different than regular binary search trees where each item has a fixed search key).

This is no different than using Y positions. Instead of storing number of children in each tree, we store the (pixel) height of the (sub)tree which is node.height = node.lineHeight + node.left.height + node.right.height. When we traverse the tree (as loosely described above) we add node.lineHeight to skipped_lines instead of 1 (and probably rename the variable to skipped_pixels).

I host two blogs, a wiki, and a ticket system, all targets for spam, so I have since generalized the system by using mod_rewrite to redirect all POSTs without a cookie to a page which uses JavaScript to set this cookie and resubmit the request (which is then no longer catched by mod_rewrite due to the cookie being set). This means “blocking” spam doesn’t require a plug-in written specifically for the particular web application.

Despite this JS challenge some spam still gets through, and that’s what this post is about.

Spam Not Caught

Until recently I deleted all spam that fooled the JS challenge but I did look through the logs for some of them to look for patterns, and until recently it was hard to find one since:

1. The log entries look like a human (and it might be), e.g. loading all images and CSS and generally taking a few minutes from first hit to the POST.
2. The comment says “thanks for this article”, “very interesting”, or something along those lines. I.e. just one or two lines of praise (and often slight variations).
3. Even the URL provided (for author) can look very non-spammy (including the landing page).

Patterns

Given the above, I started to consider other things that could be used for filtering and arrived at the following list:

1. While comment has no spammy content it has also no actual content, so comparing it to legit content (the previous comments + post) might provide a score for how likely it is a valid comment.
2. User agent generally include Windows, yet all the cool kids are on Mac :)
3. IP is often from Central America, sometimes Eastern Europe.
4. URL referrer is often Google (searching for “blog” or similar).
5. Comment is often added to an old post. While legit comments are also added to old posts, these tend to be long.

As said above, so far I have just thrown spam comments away, so I don’t have a corpus to test the above on, so take it only as anecdotal.

I plan to archive all spam from this point on so that I can later experiement with filters based on the above list. Though my other project might not allow me to get around to experiment with this anytime soon, perhaps someone else will find inspiration in the above.

One BSD command to read OS properties is sysctl but prior to Leopard it didn’t offer kern.osrelease making it a bit awkward, as you had to keep track of how kern.osversion maps to a more humane number.

Lo and behold! Today I stumbled over sw_vers (x-man-page links fail when there is an underscore in the URL, so type man sw_vers in Terminal, rdar://7111174).

The manual page has no history, but the date is 2003 and an example in the manual page has 10.2.4 as output, so it goes way back.

A lot of other path functions use or rely on normalize, for example my version of dirname() is simply: return normalize(path + "/..");.

I was recently tasked with rewriting normalize to be more efficient and it proved to be a bit of a challenge, so I’ll share what I came up with.

Motivation / Background

In TextMate I index all bundles and store this index on disk. This is what gets loaded on startup, and in 1.x I stat() directories to see if part of it needs updating.

Starting with Leopard it is possible to use FSEvents rather than having to stat each directory (which is not fast when you have a gazillion bundles). The API of FSEvents is so that you tell what directory you want events for. It may seem like I should just watch ~/Library/Application Support/TextMate/Bundles but it is unfortunately not that simple, since this directory can contain symbolic links pointing outside the directory for which I would then not see events.

Both for this reason, to minimize storage, and to easily figure out what has been added/removed when getting a file system event, I store a lot of relative paths in the index.

I convert all the relative paths to absolute paths for the memory cache (created from the index) and with almost 10,000 bundle items, and join being called multiple times for each item (as we go from bundles directory → bundle → kind folder → actual item) the normalize function was responsible for 55% of the time spent creating this memory cache.

To be exact, it took a tenth of a second, which isn’t much, but in 2.0 editing a bundle does not edit any in-memory data structures, it only saves the new item to disk, which causes a file system event that triggers updating the index, which in turn regenerates the memory cache. So I wanted this to be as close to instant as possible.

Another solution would be to make the paths absolute only when needed. The reason I am presently not doing this has to do with code abstraction (not having the bundle query system know about the index).

The Problem: Resolving Parent References

The difficulty with optimizing normalize is the step which removes parent references. It sounds simple, i.e. each time we see ‘..’ then we backtrack one component in the path.

The complexity comes from allowing ‘..’ as the component preceding ‘..’. So we can’t backtrack one component, we need a stack of backtrack points. The pseudocode for the algorithm is as follows:

stack = [ ]

for component in path

   if component == ".."
         stack.pop
   else  stack.push(component)

return stack.join('/')

This code assumes that the first component of an absolute path is the empty string, e.g. /path/to/foo → [ "", "path", "to", "foo" ] and it doesn’t handle relative paths which (possibly indirectly) start with a reference to parent (e.g. ../foo).

Regardless, this algorithm is not going to work, the problem is that we split a path into an array of strings and use a dynamic stack to handle the backtracking. These things involve memory allocation which makes the code magnitudes slower than a potential in-place algorithm.

We could reduce the number of memory allocations by working with string indices, but it is still based around a dynamic stack and backtracking.

Avoiding the Stack

Turns out getting rid of the stack-based approach is fairly simple if we iterate the path right-to-left, that is, backwards.

If we move backwards then when we see ‘..’ we skip the next component (rather than previous). If next component is also ‘..’ then we skip the two next components, etc. So we can manage this with a simple counter instead of a stack. Here is the code from above, but rewritten to iterate the path backwards:

res  = [ ]
skip = 0

for component in path.reverse

   if component == ".."
      skip += 1
   else if skip > 0
      skip -= 1
   else
      res.push(component)

return res.reverse.join('/')

I kept it close to the stack-based approach, but the above can be converted to an in-place version which simply overwrites path, so we can write the above without any memory allocations at all.

The skip counter also tells us how many parent references we weren’t able to resolve, so we can add (what we didn’t handle in the stack-based version):

while skip > 0
   res.push("..")
   skip -= 1

Absolute versus Relative Paths

I mentioned that for absolute paths the first component should be the empty string, for the stack-based algorithm to work.

This isn’t ideal, for example an (ill formed) path like /foo/../../bar will be converted to bar, i.e. a relative path.

To avoid this, a simple approach is to check if the received path is absolute, if so, remove the first slash to make it relative. Our code then only needs to handle relative paths, and by re-adding the slash last, we ensure the result is absolute.

In the in-place version removing and adding that slash is much simpler than it sounds, at least if we use the calling conventions of the C++ standard template library and takes a first/last iterator as argument and return the new end-point of the sequence, since with such API we can write the entire function like this:

char* remove_parent_references (char* first, char* last)
{
   if(first != last && *first == '/')
      ++first;

   std::reverse(first, last);

   char* src = first;
   char* dst = first;

   size_t skip = 0;
   while(src != last)
   {
      char* from = src;
      while(src != last && (src == from || *src != '/'))
         ++src;

      if(is_parent_meta_entry(from, src))
         ++skip;
      else if(skip)
         --skip;
      else
         dst = std::copy(from, src, dst);
   }

   static char const parent_str[3] = { '/', '.', '.' };
   while(skip--)
      dst = std::copy(parent_str, parent_str + sizeof(parent_str), dst);

   std::reverse(first, dst);
   return first != dst && dst[-1] == '/' ? --dst : dst;
}

This rule of thumb lowers complexity and makes both refactoring and re-use of code easier.

One scenario where it might be appealing to ignore this rule is when outsourcing computation to a worker thread, but here it is actually more important to stick with it.

Let us say we want to search folders recursively and provide the user with status about where we are in the process.

To provide this status we can have the worker thread send a message back to the main thread to let it know which folder it is presently searching, but this breaks the rule! The main thread sets up the worker thread and will also terminate it, should the user abort the search, so the main thread clearly knows about the worker thread (and need to). If the worker thread sends back messages, then it knows about the main thread.

Synchronous Message Deadlock

If we do synchronous message passing then this simple design can lead to a deadlock. E.g. if the worker thread sends back a status update and at the same time, the main thread sends a terminate message to the worker then both threads are stuck waiting for the other to acknowledge the message.

Asynchronous Message Race Condition

By using asynchronous message passing we avoid the deadlock but instead introduce potential race conditions. The main thread may send a terminate message to an already completed worker thread, because it hasn’t received a “did terminate” message yet, or the worker thread may send a status update to the main thread after the main thread sent a terminate message.

This may lead to messages sent to disposed objects or resources being leaked, it is not impossible to “get right” but it is definitely not a simple problem.

The Solution: Polling

While polling in general should be avoided, it fits this problem very well. Our search code will look something like the following (C++):

class searcher
{
    volatile bool keep_running, done;
    std::vector<std::string> results;

public:
    searcher () : keep_running(true), done(false) { }

    void start_search (std::string const& src)
    {
        std::vector<std::string> toSearch(1, src);
        while(keep_running && !toSearch.empty())
        {
            std::vector<std::string> tmp;

            // pseudo-code:
            folder = toSearch.pop()
            for each file in folder
               tmp.push(file)      if file matches criterion
               toSearch.push(file) if file.type == folder

            lock(mutex);
            results.insert(results.end(), tmp.begin(), tmp.end());
            unlock(mutex);
        }
        done = true;
    }

    void stop_search ()
    {
        keep_running = false;
    }

    std::vector<std::string> get_results ()
    {
        std::vector<std::string> res;
        lock(mutex);
        res.swap(results);
        unlock(mutex);
        return res;
    }

    bool is_done () const
    {
        return done;
    }
};

This encapsulates the searching, but does not use a thread itself. The get_results member function though is thread safe, so a user can spawn a thread, call start_search in that thread. In the main thread a timer is started, and get_results is periodically called (together with is_done).

When is_done returns true, the main thread knows that the search is done and can stop the timer (and delete the searcher object).

Advantages

In addition to avoiding the potential deadlock and/or race conditions, two other advantages with this approach is:

1. Separation of concerns. The search code is completely self-contained and does not need to incorporate knowledge about threads or message passing. This makes it easier to re-use it, e.g. if we are making unit tests we can test the code without needing to involve an actual worker thread.
2. Free throttling! In a user interface we don’t want to refresh the progress more than a few times per second, so we simply set the timer to fire e.g. 5 times per second. Had we instead made the worker thread signal the main thread, it would be difficult to control the number of messages sent. For example searching a 4600 RPM disk might only produce one new result per second, so here it might be ideal to signal the main thread whenever we get a new result, but say we are searching an SSD disk, the disk cache is hot, or there are hundreds of matches per folder, then we flood our main thread to a point where it may affect perceived performance.

Closing Remarks

I started by writing that if A knows about B, B should not know about A. When deciding which of the two should know about the other, it should be the component most likely to be re-used, which should not know about the other component.

In the above example we made the search code be the candidate for re-use by not letting it have any dependencies (knowledge about other objects), in a MVC pattern it is the view and model we want to re-use, and so, these do not know about any of the other parts.

a || (x && b) || (x && y && c) || (x && y && z && d)

It looked redundant with 10 instances of only 7 different variables.

Pretty much the only rule I know for manipulating boolean logic is how to change AND to OR so I didn’t have much luck working on the above until I realized that AND can be seen as multiply and OR as plus (with false being 0 and everything else being true).

Using that insight we instead have:

a + (x · b) + (x · y · c) + (x · y · z · d)

It is now possible to use the algebraic properties with which most of us should be somewhat familiar. For example we can use the distributivity property to only have one instance of x:

a + x · (b + y · c + y · z · d)

We can do exactly the same with y:

a + x · (b + y · (c + z · d))

We now only list each variable once, so this is optimal (assuming that all variables can affect the result of the expression). Going back to boolean operations we end with:

a || x && (b || y && (c || z && d))

Another thing I gained from this exercise is that now I will always be able to remember that AND has higher precedence than OR which is useful if you want to cut down on your use of parenthesis.