Development diary, part eight

As summer gives way to autumn here in London, it's time for another update on our progress. As the photo above shows, there is great satisfaction to be gained from lining things up just so. In this post, I'm going to discuss two of the areas in which we have expended tremendous effort in ensuring our automatic engraving produces as many of these moments of satisfaction as possible.

Aligning voices

When music for multiple instruments or singers is written on the same stave, or when writing for instruments capable of polyphony, such as the piano or guitar, it's sometimes necessary to write notes that sound at the same time on separate stems, rather than as simple chords; for example, when the durations of the notes are different, or when writing contrapuntal textures.

In Behind Bars, Elaine Gould calls this "double-stemmed writing", while different applications have different terminology (for example, "voices" or "layers"). In our application, we are calling each individual stream of notes belonging to an instrument a voice, so two simultaneous voices result in double-stemmed writing.

A voice has a nominal stem direction, which it uses in the event that multiple voices are laid out on the same staff at the same time; when a voice appears on a staff on its own, it uses what we call its natural stem direction (if unbeamed, down if the note is above the middle stave line, up if it is below the middle stave line, and either up or down if it is on the middle stave line depending on the context of the surrounding notes; if beamed, then determined by the balance of the notes under the beam). The first voice belonging to an instrument has a nominal upward stem direction, the second a nominal downward stem direction, the third upward, the fourth downward, and so on. It's very rare for a single instrument to require more than three voices simultaneously, but it's not uncommon for three voices to be needed in complex piano and guitar music.

Working out how to position notes in multiple voices is a non-trivial problem. You can't change the vertical positions of any of the notes (as that would obviously change their pitches), which means that you have to determine whether or not the voices fit directly above and below each other, or if they overlap or interlock, how they should be horizontally offset in order not to collide. If any of the voices have dotted note values, an additional complexity is introduced by determining where best to position all of the dots.

In our application, several little engines conspire together to solve all of these problems. Firstly, the stem directions are calculated; next, the notes in each voice are then positioned on the correct side of the stem (normally to the right of a down-stem, and to the left of an up-stem, unless there are notes in adjacent staff positions, in which case one of the notes must appear on the opposite side of the stem, with complex considerations about ensuring that the majority of notes appear on the normal side of the stem, and so on); next, the pitches of the notes in all of the voices are considered, and voices are assigned to columns. A voice column is exactly what its name suggests: a bunch of notes positioned at the same horizontal position, i.e. directly above or below each other.

The rules for whether or not two voices can occupy the same voice column are complicated: for example, if the notes in two voices are in unison, they can share the same column if their duration is the same, unless they are a whole note (semibreve) or longer; if the pitches of the notes do not overlap or are not adjacent, they can share the same column; and so on.

In music with only one voice, there can only ever be a single voice column at each rhythmic position; in music for two voices, there will often be a single voice column but may be two, if the voices cannot occupy the same horizontal position; in music for three voices, there will normally be at least two voice columns, unless the pitches of the notes are sufficiently widely spaced that they can all occupy the same horizontal position without any collisions. Where there are two or more columns, as important as which voices are assigned to which columns is which order those columns should appear.

Once all of this has been worked out, it's finally the job of the voice positioner to work out how close together multiple voice columns can be positioned, and to position all of the rhythm dots carefully. Let's take a look at a few simple cases, all of which involve two voice columns:

This is the default result from our application today. (There are two different default gaps in use here depending on the situation – 1/16 space, and 7/10 space – and both of them are editable.)

Update (2014-10-01): The image above has been updated today based on the feedback from my readers, who let me know (in the comments below) that they felt the default gaps were too large. I'm grateful for this feedback, and have updated the defaults used by our application to match the preferences expressed by the clear majority. The original image can still be found here if you want to see yesterday's results.

By way of comparison, let's take a look at the default results from a number of other scoring applications. We'll stick with the same identifiers we used in our previous diary instalments, so here's Product A:

The most obvious differences are found in the second chord, where a different decision about in which order the columns should be shown has been made; Product A prefers that notes are always positioned stem-to-stem when they are adjacent or overlap, which is indeed favoured by some publishers, but in cases where the notes are truly overlapping (i.e. the stem-down note is more than a second higher than the stem-up note), the stem-to-stem arrangement occupies more space than the head-to-head arrangement used by our application.

In the sixth, seventh and eighth chords, there is no offset at all between the voice columns, so the stems overprint, and it's hard to figure out what is really going on.

Here's the other main commercial scoring program, Product B:

Again, the voice columns in the second chord are arranged in the opposite order, but here the stems actually collide. In the third and fifth chords, the offset is too large; in the fourth and ninth, it is too small. In the sixth, seventh, and eighth chords, Product B produces the same unsatisfactory results as Product A.

Let's look at the results from the leading open source music engraving program, Product C:

Update (2014-10-01): I'm grateful to readers with more experience than me in the use of this product for pointing out an error I had made in the way the seventh and eighth chords in the above example were input. Having corrected that input, the results in Product C are practically identical to the results in our own application.

Finally, let's look at the most popular GUI-based open source scoring program, Product E:

The results are almost identical to those of Product C, with the exceptions of incorrectly superimposing the voices in the seventh and eighth chords, and of a larger gap between the unison noteheads of differing durations in the ninth chord.

Of course, all of these programs provide the means to adjust these results, typically on a case-by-case basis, but once again our focus is on producing excellent results by default, with simple means of adjusting those defaults, plus the ability to override those results as desired.

Tied up in knots

Much of the art in music engraving is in solving tricky graphical problems within a given set of constraints. This is true in the case of figuring out how to align voices, and it is also certainly true in the case of drawing ties between notes, which have many constraints: their ends should not start and stop directly on a stave line; the middle of their curvature should likewise not touch a stave line; when a chord has multiple ties, they should be spaced just so and not touch each other; the direction of the curvature is determined by the stem direction of the tied-from and tied-to notes; and on, and on, and on.

We are spending a lot of time on ensuring both that the default results of our automatic engraving algorithms are as good as possible, and that you as the user will have appropriate control over those algorithms, so that you can get the results you want with as little effort and time expended as possible. Although it will be possible to adjust the appearance of individual elements in the score, such as ties, our intention is that you should try changing the default settings first: most sensible options will be available as choices at the click of a couple of buttons, and will then apply from that point onwards. You will even be able to choose whether or not to make those choices apply to future projects, so you don't have to review them again.

This effort to produce not only great default results but also to provide extensive options for customising those defaults has been particularly extensive with ties. In fact, there are currently 45 discrete options relating to ties alone... and there may well be more before we're finished with them.

Ties are one of the most common graphical elements in music notation, so it's important that their default appearance is as good as possible: if it isn't, then an experienced engraver is going to need to spend a lot of time editing them, and a user with less experience may not even realise what needs to be changed, apart from perhaps having a subconscious feeling that something about the ties is not quite right. Getting the most common things right (accepting that there is no single, objective right solution to any given music engraving problem) is where some of the biggest time-savings for professional users will come from; and for those users who lack the experienced eye to be able to fix problems at all, it also ensures that the baseline graphical quality of the music being produced increases regardless.

Let's take a look at some of the situations that can arise when working with ties, and how different applications handle them. We'll start with something simple, such as determining correct tie direction for a tie on a single note, in a single voice. Provided the stem direction of the notes at either end of the tie is the same, the tie should curve away from the notehead. If the stem direction differs, however, there are two possible rules: the most common is that the tie should always curve upwards; the other, less common (but advocated in Behind Bars), is that the tie should curve away from the middle line of the staff.

Our application provides the option to use either rule, defaulting to the more common one:

Product B and Product E both follow the "always curve upwards" rule, Product C follows the "curve away from the middle stave line" rule, while Product A follows neither: the direction of the tie is determined solely by the stem direction of the tied-from note, which gives an incorrect result as often as not.

In chords, it's recommended that ties be separated vertically, ideally by one stave space, so that it's easy to read the chord. For chords with many tied notes in close position, this can be difficult to achieve, and even more so if one or more of the notes in the chord is not tied: a tie should never be positioned close to a notehead that is not tied, since it would create ambiguity about which notes should be held and which restruck.

Here is a slightly unlikely example of two chords in close position, with all noteheads tied except for a single moving note (B4 to C5):

You can see that our default appearance tries to keep the area around the middle stave line clear, to make it obvious that a note is moving there. Product A has the next best solution, but the tie for the D is positioned in the third space, and is a little ambiguous as a result. Product C positions the tie on the A4 too high, creating similar ambiguity. The results produced by Product B and Product E are far from ideal, each requiring substantial tweaking to produce something legible.

Just as important as unambiguous vertical positioning of ties is the positioning of the ends of ties horizontally. Generally speaking a tie should be positioned as close as possible to the notes it joins, but just how close it is appropriate for a tie to start and stop is highly contextual. For example, if necessary a tie can bisect the stem of a flat or even a natural, and if unavoidable a tie can also bisect the stem of a note (e.g. for another voice moving more quickly than the tied notes or chords), but ties should not collide with note flags or with the bowl of a flat, main void of a natural, or any part of a sharp. Here are two simple chords that demonstrate both potential problems:

Only our application and Product C produce default results that avoid collisions with flags and accidentals, including changing the direction of the tie in the first bar such that it curves upwards rather than downwards (because this situation is correctly treated like a chord, rather than like a single note). Each of the other applications fail to avoid either the flag at the start of the tie or the sharp at the end.

As ever, the results shown here are merely the defaults: all of the applications provide tools of various kinds to adjust the default results.

We could spend thousands more words looking at various problem cases with ties, but the foregoing is probably sufficient to illustrate the true depth of something as conceptually simple as drawing a curved line between two fixed points!

We continue to sweat these details because we want to make our application produce the best default appearance, and to provide flexible tools of the appropriate level of granularity to achieve different results, if that's what you want.

More to come

That will have to do for this instalment of the diary. We are continuing to work hard towards getting our product into the hands of users as quickly as possible, but there remains a great deal to be done. I hope that these diary entries give you an insight into the level of detail required to produce a truly next-generation scoring program. I always enjoy reading your feedback on these posts, so please have at it.