One year ago today, my colleagues and I began work on our new scoring application in our new home at Steinberg. To mark this anniversary, I thought I would bring you right up to date with our progress. My last update took us up to the end of July, so there are three whole months to catch you up on. We’ve been continuing to work hard on getting our application’s understanding of fundamental musical concepts to be really solid, so that we can build powerful and flexible features on top of those foundations. Read on.
When starting a new project from scratch, it’s wise to think carefully about every decision, even the deceptively simple ones. For example, there can be few things more fundamental to an application that works with music notation than how to determine the position on the staff to draw a given note. Such a simple concept actually depends on three or more factors: its sounding pitch, the prevailing clef, the instrument’s transposition, and potentially a further transposition modifier, such as an octave (ottava or 8va) line.
And even these three or four factors represent an over-simplification: how do you even know which staff a given note should appear on? In most scoring programs, there is a tightly defined relationship between an instrument (if such a concept even exists in the application’s model), its staff or staves, and the notes it plays. Aside from special cases like being able to cross a note to the staff above or below for keyboard music, notes stay put on the staff on which they were input.
Our new application works a little differently: the relationship between notes and staves is somewhat more flexible, which will allow a number of powerful features to be built. So before a note can be positioned on a staff, it first has to be assigned to the right staff, and such assignment can change over the course of a span of music.
By the time August rolled around, we were starting to get a handle on exactly how flexible the design of our application’s musical brain would really have to be, and the possibilities that this flexibility opens up are really exciting.
Another basic component of music notation is how key and tonality are represented. Most music notation programs present a simple choice of major and minor keys up to seven sharps and seven flats, which is flexible enough for lots of music, but of course there are other mode and key systems beyond the traditional western circle of fifths, and where existing applications address these requirements at all, they do so in a bafflingly complicated way.
Our goal is to make working with keys and modes as straightforward as possible, while retaining significant flexibility. This boils down to allowing you to define a tuning system – in terms of into how many equal divisions an octave should be divided, and how many and which of those divisions should be given a specific accidental symbol – and a set of keys or modes using that tuning system – in terms of which accidentals should be included in the key signature, if any, and how transposition should work.
As the month closed, our developers had pushed the number of unit tests (which are self-contained tests to verify that the program continues to behave as expected, even as more and more features and functions are added) to more than 500.
As the weather finally started to cool off, we were blessed with the arrival of a fancy coffee machine to keep our developers fully caffeinated as they continued to do battle with notation fundamentals.
We’re working hard to make our new program as efficient as possible. It’s going to be doing a lot of work to ensure that the resulting notation looks as good as possible after every input and edit, including laying the music out on the page and spacing it, ensuring accidentals appear as they should, making sure objects are positioned correctly, and so on.
One of the potential weaknesses of any application that has to do a lot of processing after every edit is that the program can get slower and slower as the size of the data it has to work on gets larger (like e.g. Sibelius). You can guard against this problem by doing less work for the user automatically, leaving them to fix engraving problems for themselves (like e.g. Finale), or you can sidestep the problem entirely by working slower than real time and processing the whole document in a single, non time-sensitive operation (like e.g. LilyPond).
Our approach, on the other hand, is to try to be as smart as possible about figuring out the extent of the effect of any given edit, so that the program is doing as little recalculation as possible each time, and to carry out as much of the resulting recalculation as possible in parallel, taking advantage of the multi-core CPUs in today’s computers.
So our application works like a series of interconnected small engines, each doing a specific small job and passing its output on. The end result should be a program that stays sprightly even as the size of the score increases, and still does more work on behalf of the user to produce a beautiful result than all of the other real-time scoring programs.
In September, the team designed how another class of these little engines would work, and finished building the first two: one to determine where on a staff a note should be positioned, and another to determine the stem direction for each note or chord (notwithstanding how that may be modified later as a result of being grouped together by a beam, etc.).
Stem direction is another deceptively simple decision: stems should point up if notes are below the middle line of the staff, or down if notes are above the middle line of the staff, with notes on the middle line of the staff having a neutral stem direction – i.e. they can point either up or down, depending on what looks right in the musical context. But even this simple rule is quite difficult for a computer to process in real time, since the stem direction of notes on the middle line (or for chords that are equally balanced either side of the middle line) should change according to notes on either side.
This is where our application’s careful design of small, self-contained engines that can perform specific jobs in parallel should come into its own. We may be dealing only with the most simple and fundamental concepts today, but this kind of architecture should make it possible to handle more complex concepts in the same way, each with only marginal impact on the performance of the whole program.
On the face of it, you might wonder why it should have taken so long to get to the point where our application’s musical brain can determine the position of notes on staves and their stem direction, but large-scale software development is very much analogous to an iceberg – only around 10% of its mass is visible above water. More exciting, but less visible, is establishing a design and architecture that will allow each subsequent notation feature to be developed more quickly and easily, and with marginal performance impact.
Over the summer, we continued working on prototypes to define the most important interactions with the program. As Autumn arrived, we started working on a more sophisticated prototype written in native code to model how you will set up a project to include multiple instruments, multiple spans of music (e.g. movements, songs, etc.), and then create score and part layouts to bring them together. Our application introduces several important new abstractions, to bridge the gap between how scoring programs typically work – with their fixed relationships between instruments and staves, and enforced hierarchy between full score as master and instrumental part as linked slave – and how music works in the real, more complicated world.
To make these relationships more flexible and powerful, we have to make them more indirect, which runs the risk of making the program hard to understand. As a result, we are iterating the design of this part of the program particularly intensively, so that the way the application presents these concepts to you is as easy as possible to understand.
Another important prototype is a means to visualise music on staves. Several months ago, a very simple visualiser was written that shows rhythms, but not pitches, of notes belonging to a single voice on a single-line staff. Since then, we’ve done work on determining staff position and stem direction for notes and chords, and also have the capability to assign multiple voices to the same staff, but we’ve had no way to visualise the result on a staff. Now our test application can optionally show music for multiple voices on a five-line staff, and can display multiple staves together.
It’s still very crude: notes are not beamed together, the spacing is pretty terrible, and things like ties are drawn very simplistically. This is not by any means the basis for how music will eventually appear in our application. But it is an important diagnostic tool as we continue to add more and more musical building blocks.
Two new building blocks are now in the process of being added: an engine to work out how to group notes and rests according to beat divisions, for example those provided by a time signature; and an engine to position rests in multiple voices according to engraving conventions.
Again, each of these problems is deceptively simple, but as many wise men have remarked, the individual rules of music notation are simple, but when used in combination they become as complex as any layout problem in the fields of text and mathematical layout, if not more so. There is still lots of work to do, but the end result will be a default appearance that goes far beyond that provided by other real-time scoring applications, with lots of control available for those users who want to take it.
Our ethos is that our application will be most useful if it does automatically what an experienced engraver or copyist would do anyway. If an engraver and copyist can trust the musical intelligence built in to our application to make the right decisions, it will become a truly fast and efficient tool, and hopefully the one they will come to prefer over and above the others at their disposal.
We still have a way to go before our application is ready for everybody to use, but one year in to our journey, we are making great strides, and are looking forward to making greater strides still as we get closer to unveiling the fruits of our labours.
Stay tuned for further updates!