Development diary, part nine

As the year draws to a close, thoughts naturally turn to the events of the past twelve months, and indeed to the promise of the year ahead. For us, the past year represents a lot of forward progress on our project, but the year ahead will bring more milestones on the road towards the release of our new application.

A year ago, we were around nine months into the development of the music engine at the heart of our application, but really had no functional application to speak of: we had a test harness that could display only (unbeamed) notes and rests imported from MusicXML, but nothing in the way of interaction. Over the past year, we have chosen and adopted a new application framework, introduced the basic editing loop, built the skeleton of a functional application, developed our step-time input method, added grace notes to our music model, added ties, articulations, ledger lines, time signatures, key signatures, clef changes, accidentals, beams, and so on, and so on.

We are still building up the core of our application's functionality, but our approach remains to focus above all on quality, rather than speed. If our application is going to be worth the time and attention of professional composers, arrangers, engravers, copyists, and teachers, then it must not only have a critical mass of features, but it must also have both a more efficient workflow and a faster path to excellent results than the tools professionals are already using. That's a high bar for us to meet, but we are determined to clear it.

Don't get me wrong: we are going as fast as we can. But we are committed to developing an application that will redefine the state of the art in scoring software, and that can't be done overnight.

Enough with the ghost of Christmas future. Let's talk about the ghost of Christmas past – what we have been working on since the last instalment of this diary at the end of September.

Grace notes

Grace notes are tricky creatures, because they are arrhythmic, i.e. they do not follow the same rules for duration as normal, or rhythmic, notes. For a long time our application's music model didn't support them at all, because it wasn't clear how best to support them in such a way that didn't cause problems elsewhere.

The simplest, most fundamental representation of music in our application's music model is analogous to how a human listener perceives it, i.e. a stream of notes with pitches and durations, in time. Considered in these terms, grace notes are just the same as other notes, and you could choose to write them either as grace notes or in the same way as all other rhythmic notes, using very short note values. So our initial thought was that grace notes should indeed just be the same as rhythmic notes, and that these very short notes could be rendered either as grace notes or as rhythmic notes of very short durations, as desired: in other words, it would be easy to convert to and from grace notes.

However, once we started considering this in earnest, a number of problems began to rear their heads. The most serious turned out to be that if you wanted to edit a grace note, you would also have to edit either the rhythmic note before or the note following the grace note: inserting, say, another grace note before a rhythmic note would require the duration of the rhythmic note to be reduced by the duration of the new grace note. And, furthermore, the rhythmic note would then have to be displayed using a note value longer than its played duration, since the played duration of the grace notes would have to be subtracted from its notated duration. In other words, it would be convenient for playback, but inconvenient for more or less everything else!

On balance, this no longer seemed like a good idea, so we came up with another approach: because grace notes always precede a rhythmic note1, they should be stored with that rhythmic note. We decided that the grace notes that precede a rhythmic note would be stored separately from the rhythmic notes, like their own little mini-score of all of the grace notes in all of the voices at a given rhythmic position. This was important to allow multiple simultaneous grace notes in different voices at the same rhythmic position to be spaced correctly relative to each other, and to the rhythmic notes they precede.

The real benefit of this approach turned out to be that, because grace notes are stored in their own mini-scores, the existing engines – that determine staff position, beaming, accidentals, voice ordering, etc. – could all be modified to work with runs of grace notes relatively easily, so that grace notes are, where possible, treated the same way as rhythmic notes throughout the application.

The major difference between grace notes and rhythmic notes, of course, is how they are drawn, and how they are spaced. Grace notes are drawn at two-thirds the size of rhythmic notes, and they are spaced more closely than rhythmic notes. Furthermore, the space between grace notes should not be justified (stretched) when the music is spread out to fill the width of a system.

Grace notes should also be positioned as close as possible to the rhythmic note they precede, which is surprisingly tricky in itself. Consider the following simple case, where there is space for a grace note to tuck in over an accidental belonging to the rhythmic note:

As a reminder, Product A and Product B are the leading commercial scoring programs, Product C is an open source music engraving program that uses text-based input files, and Product E is an open source scoring program. Only our application and Product C correctly tuck the grace note in here by default2.

Other situations also require manual adjustments in most other scoring programs. For example, to have one or more grace notes before a barline, and one or more grace notes after the barline and preceding the rhythmic note, is problematic: in Product A, a grace note can be dragged before a barline and will in general stay put, while in Product B and Product E it's not obvious how to achieve this at all. Placing a clef change in the middle of a run of grace notes is even more problematic: neither Product A, Product B, nor Product E support this at all.

Our application does its best to handle all of these situations with grace (if you'll pardon the pun), so that even the most demanding use of grace notes, as found in particular in keyboard literature of the 19th century onwards, is accommodated3.

We still have more work to do on grace notes: at the moment, we are not drawing the slash across the flag or beam that denotes an acciaccatura rather than an appoggiatura, and a couple of the existing engines that run on rhythmic notes have not yet been updated to run on grace notes too, but in general we are satisfied that our approach to handling grace notes is sound, and above all will produce stable, well-spaced music that requires a minimal amount of manual adjustment to achieve excellent results.

Articulations

In other applications, the term articulation is used to cover a multitude of things, spanning ornaments (turns, mordents, and the like), holds and pauses (fermatas, caesuras, etc.), playing techniques (harmonics, snap pizzicato, etc.), as well as actual articulations.

Those other applications presumably group these disparate elements together because they nominally share characteristics, such as how they are positioned relative to notes and chords, and perhaps how they are drawn by the software (typically using one or more glyphs or symbols from a music font). In our application, by contrast, we are trying to keep musical concepts (which are in the domain of the user) separate from implementation details (which should remain fully in the domain of our engineers, and not leak out such that the user needs to understand how the program works).

When we say articulation, we mean the following definition, taken from Gerou and Lusk's excellent pocket-sized Essential Dictionary of Music Notation4:

Articulations are symbols used to indicate how a note or chord is played, alone or in relation to other notes and chords.

There are articulations of duration (from shortest to longest, staccatissimo, staccato, tenuto-staccato, tenuto), articulations of force (accent, marcato5), and, less commonly, articulations of stress (stress, unstress).

A given note can only have one of each of these kinds of articulations (e.g. it can have staccato and stress, but not staccato and staccatissimo, or stress and unstress), so if a note or chord already has an accent and you specify that it should have a marcato, the accent is removed and replaced with the marcato:

Editing articulations on notes

This also extends to chains of tied notes. In our application, tied notes are treated as single events, regardless of their duration. This means that when you edit the pitch of one note in a chain of tied notes, the pitch of the entire tie chain is changed; likewise, adding, removing or changing the accidental on a chain of tied notes affects the whole tie chain. Indeed, when you select any note or chord in a tie chain, the whole chain is selected, to reinforce that this really is a single event. Most other scoring programs store each tied note as a separate entity, which means that as you edit the pitch, accidental, or articulation of one of those notes, odd things can happen to other notes in the tie chain (e.g. changing the pitch of the first note in a tie chain will not change the pitch of other notes in the tie chain). The particular benefit that this gives in the case of articulations is that you can select any note or chord in a tie chain, and when you add or remove an articulation, it will automatically appear on the correct note in the tie chain (by default, articulations of force and stress appear at the start of the tie chain, i.e. at the attack of the note, while articulations of duration appear at the end of the tie chain, though you can of course change which end of the tie chain each type of articulation should appear). Our application makes sure that the notation always makes sense to a human reader.

Taken in isolation, articulation positioning is relatively straightforward to handle: articulations of duration go closest to the note or chord, followed by articulations of force, with articulations of stress furthest away. In single-voice situations, articulations go as close as possible to the notehead (except for marcato, which can optionally be made to always be positioned above the stave), which means they are allowed to go inside the stave if they fit between the stave lines. In multi-voice situations, articulations go to the stem side of the note or chord instead, and are centred on the notehead (unless the only articulation present is a staccato dot, in which case that can be centred on the stem instead).

Where things get interesting is when articulations are in the mix with tuplet brackets, slurs, and octave lines. The rules for how these four distinct types of notation interact are complex, to say the least: for example, articulations of duration should be positioned inside slurs, while articulations of force should be positioned outside slurs if they are located on the note or chord where the slur starts or ends, but may go inside the curve of the slur on intervening notes, if there's sufficient room; articulations generally go inside tuplet brackets if they are positioned on the same side of the notes and chords, but tuplet brackets are allowed to nest inside slurs, if the duration of the tuplet is completely encompassed by the slur, and so on.

Our application draws tuplets, but does not yet draw slurs or octave lines, though both of those notations are currently in development. Once all of them are supported, the fun of figuring out how to position them correctly in relation to each other will be able to begin.

Rhythmic spacing

The tucking behaviour I described earlier on concerning grace notes is part of the more general approach to spacing used in our application.

The first step in spacing music is to create what Ross6 calls punctuation, which involves creating a rhythmic condensation of the music, as illustrated here:

In the above image, the single-line stave represents the rhythmic condensation of the music (I use the term loosely) on the lower two staves. The condensation provides the list of all of the columns that must be spaced. The minimum amount of space allocated to a given column is determined by its rhythmic value, but this space can be increased if required, e.g. taking into account the width of accidentals to the left of a note, and flags and rhythm dots to its right.

Most scoring programs treat each column as if it is made up of a rectangular block that extends the height of the stave, almost as if each column is on its own piece of movable type (as indeed once upon a time it would have been). Two adjacent columns cannot overlap, so if there is something to the left or right of those columns, they are pushed further apart, regardless of whether the elements causing the additional space to either side are likely to collide. The grace notes image above illustrates this quite clearly – the grace note can tuck in above the accidental on the rhythmic note, but most scoring programs do not do this, because the grace note and accidental are in separate columns that cannot overlap – and the image below illustrates the same principle:

The areas shaded in green show where two successive columns are allowed to be separated only by the minimum space required for their rhythmic durations, despite the presence of accidentals to the left of the next column. Only our application and Product C7 allocate only the minimum space to each column: all the other applications effectively draw a vertical line extending the height of the stave between the columns, and only allow the edges of those columns to abut, not to overlap, as shown by the areas shaded in red.

This approach to spacing has some parallels with kerning in typography, whereby the spacing between characters is adjusted to ensure a pleasing appearance. Our application similarly kerns adjacent spacing columns to eliminate rhythmic distortion, producing an even, pleasing result automatically and removing the need for time-consuming adjustment of rhythmic spacing.

I will undoubtedly return to rhythmic spacing again in future instalments of this diary, since there is a lot more to say about it.

What's next?

If you're still with me after all of that nerdy detail, thank you for reading, and thank you for taking an interest in the development of our application. There are lots of other things going on in areas like the visual design of the application, key interactions in setting up new projects, and inputting and editing music. No doubt I will cover some or all of these things in future instalments.

In the meantime, on behalf of all of us in the scoring team here at Steinberg, I wish you a very merry Christmas and a happy and prosperous New Year.

You might be asking yourself about grace notes at the ends of trills or glissandi, or grace notes at the end of a bar (arguably terminal notes at the end of ornaments are not grace notes, but let's not quibble). These grace notes can still be considered as belonging to the next rhythmic position, but they steal their played duration from the notes they follow, rather than the notes they precede. This distinction is necessary for playback, but not for drawing and spacing the grace notes. ↩

Of course, it's possible to adjust the spacing of the grace note relative to the rhythmic note in all of these programs, but, for example, in Product A there is always extra space to the left of the grace note's tucked position that can only be closed manually, and everything will be undone if the user has to reset the note spacing for any reason, e.g. by changing the time signature at that location. ↩

Admittedly, you cannot insert a key change in the middle of a run of grace notes in the middle of a bar in our application, but we have not yet found such an example in real music; it is, in any case, very rare to change key in the middle of a bar, let alone happening to have grace notes at exactly the same spot. You can, however, insert a change of key at a barline and have grace notes both before and after that barline, and they will show the correct accidentals as required. ↩

In Behind Bars, Elaine Gould includes slurs in her definition of articulation, which is definitely a reasonable thing to do, since slurs certainly fit Gerou and Lusk's definition. However, slurs are also used as phrase-markings, which denote how a longer span of music than a single note or chord should be played, and as such we felt they are sufficiently distinct to be considered separately to note- or chord-bound articulations. ↩

Tenuto is sometimes considered to be an articulation of force, rather than articulation of duration, but we decided to come down on the side of duration, because it is possible to mark a note with e.g. tenuto-accent, which would not be possible in our application if tenuto and accent were both considered articulations of force, unless we included a specific combined tenuto-accent articulation, as we have done for tenuto-staccato. ↩

Ted Ross, Teach yourself the Art and Practice of Music Engraving, 3rd edition, 1987; these days, only available to buy on CD-ROM from NPC Imaging. His section on punctuation begins on page 73. ↩

I believe that Product D, not represented in the images here, but discussed in a previous diary entry, is also capable of this kind of tucking, provided you have the right third-party utility available to you. ↩