Development diary, part 15

The light at the end of the tunnelIt feels quite strange to say, but this is the final instalment of the Dorico development diary before the first version of the software is released. As October begins, so too does the fourth quarter of the year, and Dorico will be available to buy before the winter solstice. We can see the light at the end of the tunnel.

As I stand here on the threshold, I find myself reflecting on what we have achieved since the last diary instalment at the end of June, and on what lies ahead.

What's hot, and what's not

Once Dorico is released, there will probably be a great deal of discussion about the features that are not included, perhaps even more than discussing the features that are included. The team is certainly keenly aware of the fact that each feature that is not yet included — be it chord symbols, or repeat ending lines, or guitar tab, or jazz articulations, or whatever it might be — represents some number of musicians for whom Dorico is less useful than it otherwise might be, possibly even to the point of making it not yet worthwhile for some of those musicians to give it a try.

I hope it is obvious that we have not excluded these features from the initial release of Dorico out of callousness, or a lack of understanding of the needs of different groups of musicians, but perhaps it bears saying. No, we have agonised over which features to include and what would consequently be excluded, and if you had been a fly on the wall during our bi-weekly sprint planning meetings in which we try to figure out what each team member will be working on for the next two weeks, or indeed during our individual daily conversations, you would have seen a lot of spirited discussion about how to prioritise and how to try to ensure that we get the best balance of functionality for the widest possible cross-section of musicians in the time available.

On the left, a rock. On the right, a hard place. Not pictured: Mr Hobson.

We believe that to make great software, you have to work methodically, starting with a clear understanding of the requirements, working through a collaborative process to come up with the best design that balances those requirements with what is practical to achieve in the existing architecture, and then implementing the features to satisfy those requirements carefully, taking time to ensure both that the new requirements are met and that existing functionality is not adversely affected by the changes.

In many ways, Dorico's architecture reflects this philosophy. Its modular, multi-threaded core with its dozens of interdependent engines performing individual tasks requires that features are added carefully, working through the various layers from the lowest-level event model all the way up to what is eventually drawn in the score, and its presentation in the user interface. This architecture actively mitigates against "hacking something in" quickly.

The fact that it is hard to hack features in to Dorico quickly is actually a very good thing for the long-term future of the application. The architecture that the team has built provides a solid technical foundation for all future feature development. If we continue to refine this architecture and fit features in carefully and logically, the software will hopefully grow in sophistication and capability in a controlled manner that will allow us to keep technical debt to a reasonable level.

All of this is a long way of saying that, whatever your own personal pet feature that's not included in the first release, the reason it's not there is because we want to take the time to implement things properly. We hope you will agree that it is better to have software with a smaller number of deeper, better implemented features than it is to have a larger number of shallower, overall less useful features.

Likewise, we hope you will look at the first release of Dorico and see past the things it does not yet do, and look at all of the things it can do that other scoring software cannot do — that list is already pretty long, and we plan to make it ever longer. This is only the beginning, and with your support, I am confident that great things lie ahead.

Enough philosophising — what are some of the things we've been working on since the end of June?

Play mode

I promised in the last instalment that I would tell you something about Play mode and how playback will work more generally in Dorico. As we approach the initial release, we now know where the functional baseline will be, the foundation upon which future development will be built.

Dorico's playback functionality is all built upon the audio engine from Cubase and Nuendo. In a real sense, Dorico has received a heart transplant. Heart transplants are difficult and time-consuming operations and require enormous amounts of expertise. We have been fortunate to have some of our best engineers working on this project for many months now, and I would like to take this moment to thank them. On the Hamburg side, the project has been led by Ulf Stoermer, and the bulk of the transplant work has been done by Yvan Grabit, Stefan Scheffler, and Bernd Schmidt. Of course, behind Ulf, Yvan, Stefan and Bernd are dozens more skilled engineers who have been building and refining the audio engine over many years. Transplanting the audio engine into Dorico has been both a technical and a logistical challenge, with many difficulties along the way, but the Hamburg team, along with my colleague Paul Walmsley in the London team, have worked tirelessly to make the integration a success.

One of the many cool things about the way Dorico and the audio engine are integrated is that the audio engine runs as a separate process. This both ensures that the engine will run on separate CPU cores to Dorico itself, but also provides some measure of protection against crashes: if the audio engine crashes, for example because of a badly-behaved VST plug-in, then generally Dorico can keep running, although of course it will no longer be able to playback or indeed receive MIDI from your MIDI keyboard. At least you should be able to save your project and safely quit and restart Dorico, which will restart the audio engine.

Dorico in Play mode, with three instances of HALion Sonic playing some Bernstein, yesterday.

By default, Dorico plays back through HALion Sonic SE 2, which is the version of the HALion sample workstation included with Cubase. HALion Sonic SE 2 comes with its own pretty extensive library of more than 1500 sounds, which includes some decent acoustic instruments alongside more production-focused sounds. There's a nice Yamaha piano, for example, and some decent guitars, among others. It's pretty easy to try out different sounds in HALion: show its interface, then click on the little arrow at the right-hand side of one of the channels to open the MediaBay interface, into which you can type a search term. Click one of the results to load that patch into the selected channel, and then you can try it out using the on-screen keyboard. If you load a new patch into one of the channels that Dorico has already loaded, when you save your project, Dorico will retain your choice there until you explicitly instruct it to reset the loaded sounds back to the default choices we have picked for each instrument.

In addition to the 1500 sounds from HALion Sonic SE 2's own library, Dorico also includes the whole HALion Symphonic Orchestra library, with more than 100 high-quality orchestral patches. HSO has been around for 10 years now, and of course there are newer orchestral libraries with a wider range of sounds and many powerful possibilities for automation and expression, but HSO still sounds good and is a reasonable default set of sounds to get started.

We expect that you will be able to make basic use of third party virtual instruments in the first release of Dorico, routing individual voices from each instrument to the appropriate channel in your chosen plug-in. In the initial release, support for arbitrary VST Expression Maps will not yet be there, and the ability to edit individual events and record controllers to enliven the default playback will be very limited, but the foundations are in place and we expect to be able to extend this functionality after release reasonably quickly.

Because support for VST Expression Maps is at its early stages, so too is interpretation of markings written in the score. Tempo marks are interpreted, and so too are dynamics (including support for instruments that use continuous controllers to produce gradual changes in dynamics), and other markings will follow soon. We will continue to work on functionality to make score playback both more expressive by default and also more tweakable in Play mode as quickly as we reasonably can.

REVerence. My colleagues are sick of me talking about how much I like its shiny silver knob. Oh, and it sounds OK, too.

In addition to playback through VST instruments like HALion Sonic SE 2 and Kontakt, Dorico can also handle VST effects processors. The Mixer window provides you with an individual channel for every output of every instrument plug-in, and includes both a built-in four-band parametric EQ in each channel, but also slots for inserts and sends, into which you can load one of the 30 included effects plug-ins, or indeed load your own VST 3 effects. The bundled effects include things like compressors, limiters, guitar amp simulators, filters, gates, and more besides. The state of the Mixer is saved with your project, and with this kind of flexibility it starts to become possible to actually mix with Dorico, rather than merely balance sounds that you cannot reasonably modify.

Holds and pauses

One of the panels in Write mode that had sat empty until recently is Holds and Pauses, from where you create fermatas, caesuras, and breath marks. On the face of it the notational requirements for, say, a fermata seems pretty straightforward: draw the fermata centred on the notehead or rest, position it above the stave, and allow another to appear below the staff in the case of multiple voices. So far so good: and indeed in other scoring software this is more or less all there is to it. You select the note or rest on each staff on which you want the fermata to appear, and create each one in turn.

However, from a purely musical point of view this approach doesn't make a lot of sense. A fermata indicates a pause that applies to the music being played by all of the musicians in the ensemble — even in a cadenza passage, in which a soloist extemporises while the rest of the ensemble holds on, there is normally a pause written at the end of the cadenza as well, so that conductor, soloist and ensemble can move on together — so really if you create a fermata, it should be created on all instruments together. If you have to add the fermata one by one to each instrument, you could easily miss one out, with the attendant possibility of a mistake in rehearsal or, worse, in performance.

Even if you remember to add the fermata correctly to each instrument, deciding exactly where to put the fermata on each instrument requires some careful thought and could lead to the odd mistake. If each instrument is playing a different rhythm, then you have to be sure to put the fermata on the last note or rest that coincides with the instrument or instruments that have the note at the rhythmic position at which the pause takes effect. A single played pause could result in the fermata needing to be shown at a different rhythmic position in every instrument!

Dorico eliminates both problems at once: to create a fermata in Dorico, you simply find that last note, the one at which the pause actually takes effect, and add the fermata there. Dorico then creates the appropriate fermata for each voice in every staff in the ensemble, working out which note or rest it should appear on in each case. You can be sure that every instrument will show the fermata, and in the right place.

Adding a fermata to a bar from a Villa-Lobos guitar study.

You might not want to show the fermata for every voice in complex multi-voice keyboard or guitar music, particularly if you have more than one voice with the same stem direction, e.g. two up-stem voices. In that case, you can choose, either by default or on a pause-by-pause basis, whether to show one pause per voice, one pause per side of the staff (so for two up-stem voices, you would show the pause only on the later of the two notes), or one pause per staff (in which case the pause would be drawn on the latest sounding up- or down-stem voice).

Compared to other scoring software, it's really great to be able to create a pause in exactly the right place for every instrument with a single operation. It's another example of how we try to think about each notation from musical first principles and then shape the implementation of the feature around the musical requirement, rather than what might be quickest to implement.

Oh, and if you want to display a fermata on a barline, no time-consuming workarounds are required: simply place the fermata on the first note in the bar following the barline on which you want the fermata to appear, and switch on the Attach to barline property in the Properties panel. The fermata then appears above each barline section (i.e. above the top staff of the system, and at any breaks in the barline due to bracketing further down the system) instead of on notes and rests.

Dorico provides seven different types of fermata, from very short to very long, including the half fermata used by Henze to mean a short pause, and the one with two dots that he used to mean a long one.

Caesuras are implemented in a similarly simple to use fashion. Simply select the note before which you want the caesura to appear (e.g. the pick-up into a new phrase), and choose one of the four types of caesura provided (normal, thick, straight, or curved). Dorico creates a little space before the selected note and places the caesura on every staff, correctly straddling the top staff line.

A picture paints a thousand words

If you add up all the words I've written about Dorico in the fourteen previous instalments of this diary, I've written 35,241 words — that's more than 5,000 more than in George Orwell's Animal Farm, but approximately 130,000 fewer than in Elaine Gould's Behind Bars. It's appropriate, then, that the last thing I want to write about today is about how we can avoid using so many words in Dorico's user interface.

Speaking of Behind Bars, that book has been more than just a valuable resource during the development of Dorico: it also quite directly inspired the approach we have taken to presenting the hundreds of options found inside Dorico. When I first started reading Behind Bars five years ago, the thing that struck me most in comparison to the other well-thumbed reference books on my shelf was just how generously illustrated it is. Unlike Read or even Stone, you can find things in Behind Bars simply by flicking through and stopping when you come to an illustration that looks like the sort of thing you were thinking about, then reading around that page.

Dorico has literally hundreds of options that allow you to control the appearance of your music, split across three main dialogs: Layout Options, Notation Options, and Engraving Options. Each dialog groups its various options in logical categories, to try to chart a course through the vast ocean of possibilities, but with a single kind of notation having dozens of options on its own (slurs, dynamics, tempo, and ties each have more than 50 distinct options each, for example), you need to do more than to simply group options together.

Inspired by the generous use of illustration in Behind Bars, Dorico's options dialogs are likewise full of illustrations that will hopefully help you find the option you want right away. As a bonus, we don't have to try to compress complex concepts down to a few words that end up reading like a nonsense non-sentence out of Lewis Carroll (what exactly does "spaces concavity forces flat" mean, anyway? Don't write in — I do actually know).

So to spare you any further of my ramblings, here are a few pictures drawn from the Notation Options and Engraving Options dialogs, to give you a taste of the kinds of flexibility and power to be found in Dorico behind its friendly face.

Choosing between the three basic options for accidental duration rule, in Notation Options.

Dorico provides a number of options for how to notate syncopated rhythms; Notation Options can be set independently for each flow if necessary.

A few of the options describing the strategies Dorico can use to avoid collisions between slurs and the notes, accidentals and articulations within their arc, taken from Engraving Options.

The end of an era

And with that, dear reader, we reach the end of the first phase of Dorico's development. Soon, you will be able to try Dorico for yourself. On behalf of the whole team, who have shouldered the hard work and built Dorico with the sweat of their brows, thank you for reading, and for following the story so far.

It may be the end of one era, but it is the beginning of another more exciting one. This blog will continue, with what will I hope be more regular posts that can talk in more concrete terms about Dorico once it is out there in the world, with news of updates, with stories about projects for which Dorico is being used, and who knows what else. I hope you will continue to come along with me — things are about to get interesting around here.