All of a sudden, over the past week I have received lots of requests for an update on our progress, so here we go. My last update was at the start of November, on the occasion of the first anniversary of our team joining Steinberg. This new update comes more or less on the occasion of the first anniversary of us starting to write our new scoring application. We’ve made a lot of progress in a year, but we still have a long way to go before the product will be in your hands.
In my last update I described the high-level design of the musical brain at the heart of our application: lots of individual engines performing single tasks, some dependent on one another, others running independently. This design approach allows us to break down the immensely complex problem of how to correctly notate and lay out music into smaller, more manageable chunks, and also provides the opportunity for these little engines to be designed and implemented independently, and eventually to run in parallel (unless an engine is dependent on the output of another engine, of course).
I described in broad terms the first couple of such engines that we had implemented already: determining the staff position of a note (taking into account clef, transposition and octave shift), and determining the stem direction of a note (taking into account staff position and musical context). We were also starting to embark on engines to handle note and rest grouping, and to position rests in multiple voices correctly. Over the next couple of posts, I’ll talk about these engines, together with a few other new ones we have developed since the last update.
Note and rest grouping
How notes and rests are rhythmically grouped, based on the prevailing metrical grouping or time signature, has a fundamental impact on how music is apprehended at sight. As with many of the rules of music notation, an experienced musician will instantly recognise if it is wrong, but may not always be able to articulate exactly what is wrong.
Consider the very simple case of a note on the final eighth note subdivision of an otherwise empty bar of 3/4, and a bar of 6/8. You can instantly see which is which:
Now consider the same bar, but with a note of five eighths in duration at the start of the bar. Again, it’s obvious which bar is in 3/4 and which is in 6/8 by the way the long note is divided into smaller parts:
Most scoring programs take quite a simplistic approach to note and rest grouping. Typically, the user inputs the note durations exactly as they want them to appear, and the software only has to worry about creating rests to pad out the bar; in the example above, neither of the two most mature commercial scorewriters provides a means for the user to enter a note of five eighths in duration using the mouse or keyboard, so the user has to choose how to split that note of five eighths himself. The exception to this, of course, is when the user plays notes using a MIDI instrument in real time, or imports a MIDI file – but you might be surprised at some of the choices that the established scoring programs make with relatively simple input.
It seems that, on the whole, existing scoring programs have a simple algorithm for note and rest grouping according to the time signature, with a few common edge cases given special handling.1
Our goal is to handle note and rest grouping completely automatically, while providing a small number of global options to account for some variation in conventions, and of course allowing editing on a case-by-case basis. For example, it’s normal modern practice to use dotted rests in compound time signatures, but if you are reproducing an earlier edition, you might prefer not to use dotted rests; you might not ever want to see double dotted rests; and so on.
To this end, we have devised an algorithmic approach to note and rest grouping that is fully general, so it works satisfactorily in any time signature or beat grouping. This has taken time to get right, but it will be used considerably more in our application than in other scoring programs, because the notation in our application is considerably more dynamic than in most other scoring programs.
Positioning rests
When two or more voices (independent streams of notes) are rendered on the same staff, it’s important that rests in both voices are vertically positioned in such a way as to remove ambiguities in the rhythm of the music. Elaine Gould’s book Behind Bars has some great examples of the kinds of ambiguities that can occur, like this doozy2:
Obviously this example is somewhat contrived, designed to make a point: Gould states that when the notation gets this complicated, each voice should really be given its own staff. But if you’re going to put it on a single staff, you need to ensure not only that rests do not collide with notes (notes obviously can’t move vertically to avoid rests, so rests have to move vertically to avoid notes), but that the rests are placed generally in line with the voice to which they apply. Look, for example, at the 32nd rest at the end of the second bar: this rest is in the lower voice (whose notes are the C5 quarter note and the D6 double-dotted eighth). It could physically be drawn in its nominal normal position, i.e. centered around the middle staff line, as it wouldn’t risk overprinting with a note there: however, it would then look as if it were following the stem-up eighth note in the upper voice, which is ambiguous, if you’re generous, or just plain wrong, if you’re not.
How do the main scoring programs handle this example by default? Here’s Product A, one of the two most mature commercial scorewriters:
Ignoring the issues with stem-up and stem-down notes colliding in the first bar, you can see that a simplistic approach is taken, simply offsetting the rests in each voice outwards, away from their normal position in the middle of the staff, by a space. This isn’t enough to avoid the notes, and only one rest (the 16th rest in the lower voice in the final bar of the example) is positioned satisfactorily.3
Product B, the other of the two main commercial products, fares somewhat better:
This takes a similarly simplistic approach to Product A, offsetting the vertical position of the rests in each voice by a fixed amount, but the decision to offset by three spaces rather than Product A’s one space definitely produces a more legible result.
The open-source music engraving program, which we’ll call Product C, fares better still:
Here all of the differences from Gould’s preferred rendering really come down to a matter of taste: it would be neater to align the eighth rests in the lower voice in the first bar, and it would be clearer to place the lower-voice bar rest in the fourth bar and the upper-voice half rest in the fifth bar on leger lines further offset by one more space, but it certainly provides a legible result by default.
Finally, here’s how the rest positioning algorithm in our application handles this case:
In the interest of full disclosure, the above example shows how the rests are positioned by our algorithm, but is not a direct screengrab from our application, because the prototype score rendering technology in our test harness doesn’t produce such a beautiful result just yet.
While it’s relatively quick to adjust the vertical position of rests in both of the main graphics-based scoring programs (and can be done in Product C by specifying a “pitch” for the rest), in our application it’s our goal that you simply won’t have to do so – even under extreme circumstances such as this contrived example.
More to come
In the next instalment in this series (coming soon!), I’ll describe a couple more of the engines we’ve been working on recently, focused in particular on accidentals.
- It must be said that the two most mature commercial scoring products fare much better in this regard than, say, the leading open-source music engraving program, which takes the most simplistic approach possible when converting MIDI data into notation, simply biting off as much of a long note as possible as can be represented by a single note value (with augmentation dot if necessary), and then handling any remaining duration in the same way, so every long note is always split up using durations that proceed from longest to shortest, completely ignoring the beat divisions implied by the prevailing time signature. Unfortunately for users of text input-based engraving programs, they probably also have the hardest job of fixing up the result, given that they have to manually edit the resulting text input file. In fairness, it is presumably not often that users of these kinds of programs choose to begin a project by importing a MIDI file. However, even the major graphic-based scoring programs offer relatively little control over the rules to be used. ↩
- This example is found at the bottom of page 37. ↩
- Improving rest positioning in Product A is, at the time of writing, its third highest-voted feature request. ↩
Thanks for these updates and responding to our comments, Daniel; we know you’re a busy chap! I’m very much looking forward to getting to try this new application, as it sounds as though it’s going to make my job as an arranger and typesetter much easier.
Product A sneakily identified in Footnote 3…
Well, _all_ products implicitly identified by their file names 😉
One of the things that Finale and Sibelius lack is the ability to score microtonal music, especially Turkish & Arabic notation, in an easy way. It can be done, but they make the user jump through hoops with all kinds of silly setups, plugins, and workarounds. There are plenty of non-Western users around the world who need microtonal accidentals and alternative key signatures that can automatically detune the MIDI notes. Would your team consider implementing these features?
The other blog posts allude to such improvements so that the user will be able to define a set of accidentals and their semantic uses.
good on ! please inclus the individual tunning by instrument ! or multiple player .
as it was possible in sibelius before the 7 .
will your engine automatically choose to lengthen the stems?
@Joseph: A very good and fair question! We haven’t yet written the engine that handles the interlocking of opposing voices (so our crude test renderer just draws them on top of each other, without regard for collisions), but my hope is that this should be one of the possible outcomes, yes; others include increasing the x offset, or reversing the order of the notes, such that the notes go stem-to-stem rather than head-to-head.
I am hoping that the nomenclature in use in this post is a hint that Steinberg has decided to call the new software ‘Product D’.
Sounds like we need a t-shirt contest… And, of course, a theme song.
Ha!
Now we all know the name of your new scoring application! 😉
Daniel–
As you discuss the architectural strategy of loosely-coupled engines operating on multiple threads, how does this integrate with Steinberg’s existing assets? I think it’s been suggested once or twice that playback in Product D would leverage existing Steinberg technology–are your colleagues in Germany thinking along the same lines as your team, or will your approach take you down the path toward developing new playback tools as well?
@John: Our plan is to use the core audio engine technology from Cubase and Nuendo: it’s battle-tested, high-performance code and far more capable than what we could come up with on our own. The nitty-gritty of exactly how this technology will be integrated into our application is still being worked out in collaboration between the engineers here in London and in Hamburg. To my knowledge, Cubase is already multi-threaded in any case.
Cubase is multithreaded and the audio engine is pretty damn stable, experienced in my 10 years of working with it.
Tech-wise, this design collaboration has great potential.
PLEASE, PLEASE!, make available ‘playback capability and spacing distribution for simultaneous time signatures (different lengths) ! ! ! ! ! ! ! !
absolutly !!!!!!
Lucas, I seem to remember this is aready agreed …
Daniel, am I right?
I don’t know if this is the right place to simply list requests (and please tell me if there is a better place, or if you don’t need any right now!) but I sure would like to see a way to quickly generate common rhythmic groupings (and the ability to define what is “common”) such as permutations of a quarter beat broken into 8ths and 16ths, triplets, etc. Rhythm is the most difficult part of note entry, and it would be very nice not to have to always copy-paste from an earlier example, or use tricks to get, say, an eighth note triplet written as quarter-eighth (in Sibelius).
And don’t get me started about hyphens in lyrics. They need a lot more attention than the standard packages have given them.
Exciting! I love the way you’re willing to discuss the inner workings and thought processes… I *so* wish I could move to England to write C++ code with you! I don’t think I’ve been this enthused since I encountered the first advert for Sibelius 1.0 in EM years ago. So long ago that I had request the demo on CD because downloads were still unheard of!
Have you put similar thought into lyrics spacing? (The bane of my recent work…) I’d love to see much more intelligence about when and how extra spacing is added versus strategic moving of the syllables, etc. Sign me up for any help testing you want (when you get to that point)!
Cheers!
@Chris: We haven’t actually started on lyric spacing yet, but we definitely have some ideas about how to handle it better than in the major existing scoring programs.
Yes re lyrics! I deal with entering or scanning and then setting vocal music almost every day, and lyrics can be a problem. Here’s another hyphen issue that drives me nuts: hyphens in Photoscore go into Sibelius as hard hyphens. Don’t know in which system the problem lies, but in what world do hard hyphens dominate in lyrics!
I also would love to be part of any testing that you need.
Cheers!
Ed
A clarification on your first footnote: MIDI import in MuseScore was improved last summer, so it ties according to time signature in the nightly builds.
@exa: Product C isn’t MuseScore. But I saw the Google Summer of Code project for MuseScore’s MIDI import last year, and it’s great to see that area getting some attention.
In the very first pair of examples i would suggest that the correct notation for the 6/8 example would be a dotted crotchet rest followed by a crotchet rest followed by the quaver note. Maybe it was written the way it is for the sake of example, but in compound time the division is usually 2+1, not 1+1+1.
@Derek: That’s a matter of taste, I think. One of the small number of options that our note and rest grouping algorithms provide is whether or not rests at the start of compound beats should be consolidated. I suspect that the default setting will indeed be to enable consolidation of rests at the start of compound beats, i.e. to show a quarter (crotchet) rest in that specific situation.
@ Derek – that concurs with what I’m expected to teach my abrsm music theory students too.
This is a fantastically insightful blog, really enjoying reading it. Thanks Daniel! On the (sort of) related topic of multiple instruments on one staff, I was wondering if you guys were planning on providing a technique to input instrument numbers vertically at the start of the staff, as you might see in a conductors score. I am yet to discover a technique to do this on Sibelius without doing the whole thing manually which is a real pain!
@George: We do have plans in this area, but I don’t want to give away too much too soon. Perhaps I will be able to talk about this more in a future post. I’m glad you’re finding the blog interesting!
Can’t wait to try out this new software, Daniel. I don’t suppose there is an ETA yet??? 🙂
@Engela, Daniel will not offer any ETA on the new app. But after reading between the lines Daniel’s various public statements to date and after consulting with a top music PSYCHIC, I believe the app not appear for public consumption until sometime in 2015 or 2016.
I started music processing on HB Engraver eons ago, progressed through Nightingale (dismissed Finale) then on to Sibelius on the Acorn, and so on to 6 (OK) and 7 (dislike) – I SO hope this will be available before I finally throw in the towel. Keep up the good work, Daniel and Team – if anyone knows what’s needed, you do, and I’m sure you’ll do your darnedest to make it work. Can’t (but have to) wait! Blog is fascinating – thank you.
Great article and comparison.
Indeed surprising, that the market leaders do not even completely prevent rest/note or note/not collisions.
Lilyond obviusly does the best job for the existing programs.
Why that? I think one reason for lilyponds superior engraving results is, that due to its nature as a compiler it has “all time of the world” to format the score. Whether it takes one second or two, that doesn’t matter. The user has to wait anyway.
This is different from graphical interactive notation editors. In those editors all formatting has to be done in realtime-like performance.
Imagine what happens, if you move a note up or down and it hits a note in another voice. The note has to move left or right, it will take more space, the bar will grow wider, the linebreaks might change, and finally you end up with a orchestra score taking one page more of space than before. Simply by dragging an innocent note!
This reformatting has to be performed in milliseconds, otherwise you guarantee total annoyance for the user.
This is the fundamental difference to compilers like lilypond. Compilers can simply choose the best algorithm with arbitrary complexity. Editors in contrast alwas have to care for highest performance and therefore have to undergo compromises.
However, it can be done better than shown by products A and B. For example PriMus does a better job as you can see here:
http://www.columbussoft.de/img/xref/PolyphonicRestsInPriMus.PNG
Delegating the task to threadable agents – as Daniel mentioned – looks like a promising approach to allow more complex algorithms. The result confirms this.
In an earlier post (or comment to a post, don’t recall) Daniel wrote that they are considering an algorithm to determine the smallest portion of music that has to be recalculated after a change. So “dragging an innocent note” would have the least possible impact while still allowing for a thorough layout decision.
I think this is quite promising (given all that seriousness that speaks through this blog).
Great article! I fight this battle every day, since I tend to work on complicated large scores. Cannot wait for product release!
Daniel, thanks for your insights on the process and your accessibility. I see that there will be a quality playback engine, taken from Cubase/Nuendo. Will the full suite of MIDI editing tools from Cubase be incorporated into the scoring program as well? Currently Cubase is the closest tool for my needs, but I don’t need the extensive audio editing tools.
A powerful MIDI editor/Score notator with the ability to sync via Asio Positioning Protocol and to import/playback wav files would be fantastic. Add in some simple recording features, and it would be my dream music app!
@Cory-Paul: We don’t intend to get too close to Cubase in terms of DAW-like functionality, but the precise extent to which we will include those kinds of features is yet to be determined, and it may change over time as we move from version to version.
Thanks Daniel, maybe I made too strong of a request for the audio side. The market seems to lack a “MIDI Workstation” that isn’t overwhelmed with the DIgital Audio side of things. If it can sync via APP, all sorts of external options open up, and it isn’t necessary to include the audio-related features.
@Claude: what software are you using? In Sibelius it is extremely simple to set stem directions for up to 4 separate voices on one staff. Note the numbers at the bottom of the keypad window; these correspond with the voice that each note (or rest) is set to. So if you have two voices on one staff set to numbers 1 and 2, the stem direction will correspond to which voice it is, regardless of vertical staff position.
Daniel,
I think I placed too strong of a request also before (as you probably remember). I was unfair to you.
But if possible, is it likely we’ll at least see a better method of editing volume/modulation changes than Sibelius had? Instead of a set of numbers (practically impossible to really edit), might we see some level of midi-performance editing that is different?
1) I assume you’ll have some form of editing the midi performance, no?
2) If yes, will it at least be more intuitive than Sibelius’s methods?
I won’t expect Cubase flexibility, but could you possibly answer those questions?
With respect, thank you…
-Sean
(Great job so far, things look great!)
@Sean: I can’t make any promises about any feature in our program, whether related to playback or engraving, or whatever. But I can promise you that I am fully aware of your request!
… and for me the other way round please, or rather: Combine a fully fledged Cubase with a Sibelius Plus. That would be the ultimate DAW!
OH YES, can’t agree more – this would be the film composers wet dream!!
At least a better integration with Cubase/Nuendo going from mockup to notation
would be superb – and finally appears to me as most doable, now that Steinberg teamed up with your team, Daniel.
This is an absolutely unique chance, isn’t it?
Are there any plans in that direction (i.e. –> better integration/workflow between MIDI-sequencing and notation)
Fascinating stuff! As a publisher the way I deal with the problem of multiple voices in one staff is to put all the stems in one voice in one direction, even if they cross over to each other’s range. Unfortunately in current software you have to do this manually one by one (you can only flip), since some will be up pr down depending on range. It would be a huge time saver for those of us in publishing to simply have one shortcut: select voice 1 and flip all stems down
Thanks for including Product C in the comparison, Daniel 🙂
Btw, why there’s a slur in the product C rendering? I know that product C doesn’t add slurs to the music unasked 🙂
A small explanation concerning “pitched rests” in product C: the “pitch” simply tells the program where the rest would be placed if it were a note, so b’ \rest results in a rest that is placed exactly in the middle of the staff, while g’ \rest would give a rest exactly one staffspace below that.
Note that such positions would be automatically adjusted if the music is transposed. So, in treble clef, if you have a d” \rest (i.e. a rest on the 2nd to top staff line) and transpose the music up a third, you’ll get a rest placed on the highest staff line.
I appreciate the intention of trying to get the notation program to encourage best practices in the copyist process. It is a huge and growing problem. Many of the works being published today would get anywhere from a D- to an F- from me as far as their notation is concerned. A brilliant composition notated badly will probably be played badly, At minimum it will waste a lot of valuable rehearsal time.
I understand the point of looking at those cases where two utterly incompatible parts are placed on the same staff. Frankly I don’t consider any of the examples anywhere near acceptable. I know you are just looking at what the software can do to improve on a terrible situation. But really, those parts should never, under any circumstances be placed on the same staves if two different musicians are expected to perform that correctly.
I admire your resolve to improve this situation, but I would suggest that in the end, incompetence will overcome anything you try to do in this area. I’m not saying you shouldn’t try. Indeed, if a few pieces of music emerge in a more playable condition because of your work, that is something. But really, maybe a direction for your tool could be identify the impossibly inept scores and present a wizard that suggests how the job could be done properly.
Thanks a lot for this nice comparison. If you achieve to combine the automatic positioning like product C is able to do with a user interface like product A or B, I will be impressed.
Concerning the examples: For me, A and B do not need to be discussed. For C, I am not sure whether I prefer Gould’s example over product C’s with respect to the half and whole rests “flying” outside the staff. But the alignment of the rests in the first bar, done with your algorithm, is nice, indeed.
Cory-Paul Allen wrote something about midi workstation. Im working with Eastwest hollywood libraries and constantly need to change at least 2-3 midi CC on each instrument. I recall using Cubase some years ago it was easy to change the modwheel/expresion CCs on each midichanel. It really would make me a happy lad if I could do that while Im composing without using a line and a hard working plugin for every CC change I want to make. Keep up the good work
Jesper,
I requested the same thing of Daniel, but he isn’t promising anything yet. He said it’s worth looking at in future versions (if it isn’t done early on anyway).
To me, that says this is more of a publishers program, As a composer, I need to make legit sounding mock-ups first because directors don’t like waiting. With this new program, I anticipate I’ll still have to use Cubase first, then move to notation. I’m perfectly fine by that if the transition is smooth.
I’m pretty sure the team will have the transition smooth, whether in v1 or later… it’s an obvious enough need. Better handling of midi and video would still be very nice. My hope is that they’ll employee some of Cubase’s methods (lanes for CC, lanes for tempo, lanes for video. LANES!! lol) rather than reinvent the wheel. If I had lanes on bottom, notation on top… I’d die a happy man.
-Sean
Thanks for the update Daniel, this looks great! I’m very looking forward to see more of this software. By the way, when do you think the software could possibly be released? I’m currently looking at Finale 2014, as making orchestral scores with Sibelius is quite time consuming… However if this new software would come in the next two years, maybe I could wait 😉
Thumbs-Up!
@Max: If you’re already using Sibelius, in my opinion you won’t gain a great deal by switching to Finale (and the converse is also true), so I would definitely recommend waiting for our application, though I can’t make any promises about the schedule for its release.
Every time Finale & Sibelius come out with a new underwhelming (and/or disastrous) upgrade, I wish we could fast forward a couple of years until this Steinberg version is released. Fingers crossed. (No pressure!)
Thanks George, I do know and I appreciate you taking the time. The problem is when Sibelius decides, for one reason or another, to change the stem direction of one particular voice. I do normally use the ‘shortcut’ for select voice 1 or 2, etc. and then filp, but I would like a command that specifies up or down, not just flip.
Are you able to see the farther you plunge into development whether or not a score is defined by measures? One thing that I find SUPREMELY frustrating with Finale and Sibelius is the issues with “filling in” the measures with idiosyncratic voice structures and rests. This is one reason (and I don’t mean to advertise) why I have switched to LilyPond.
I guess I would be interested in a program that could score like SCORE (no pun intended), where the information can be quickly introduced by text or alternatively by “click-and-drag.” I’m also interested in processes being LESS automated (and by automated, I mean, micromanaged) in favor of macro tools (such as beaming applications, rhythmic spacing, etc). Though I have never had the pleasure of using SCORE, I wonder if this is the direction this scoring application is going?
@Joshua: I certainly admire SCORE in many ways. You can’t argue with the results that engravers are able to obtain with SCORE (and the ecosystem of utilities and add-ons that have developed around it), though of course most folks these days would prefer an easier way to get those same kinds of results. It’s definitely our intention to build an application that produces results with the beauty and flexibility of SCORE but with the ease of use and learning of a modern graphical application.
Wow, I love the way the test output looks! It reminds me of the look of the Neue Mozart Ausgabe put out by Barenreiter (which probably contains some of the most gorgeous & readable scores I’ve seen to date). If it became possible to get scores of contemporary music to look close to that, it would be incredible. Did your team draw inspiration from the NMA?
@Paul: Although I think there’s a lot to admire in the Neue Mozart Ausgabe, in that it is beautifully clear and precise, I also find it a little more clinical than the classic engravings of the late 19th century and first half of the 20th century. That’s just my personal opinion, of course! However, it is my hope that you will be able to achieve any kind of published look you are after with our new application, and if you want your own scores to look like the NMA, our application should give you the tools to achieve it quickly and easily.
Sorry to repeat myself here: But I really hope, that this synergy between the former Sibelians and Steinberg will result in a quantum leap for Cubase Score. That would be a real reason for me to by a new version. It would probably dwarf the competition for the Status of the ultimate DAW. And since my work bridges Midi composing and notation scoring what better gift could there be, than a Cubase with a notation section that is even better than Sibelius?
Hi Daniel. This is all very interesting to read, and I’m excited about the new product.
The vertical positioning of rests is also an important issue in music that makes use of beams over rests and stemlets. Does the rest positioning algorithm in your application address this case as well, or do you intend/hope it will?
Thanks,
Dave
@Dave: It’s not yet implemented, but it’s definitely our intention to handle beamed rests in a way that is as respectful of engraving conventions as the other engraving features we have worked on so far.
I had been hoping the scoring functionality inside Cubase itself would get an overhaul along these lines. I would much rather do all my composing and arranging right there so I am little saddened to see it will be it’s own application. Perhaps a lite version of this could makes it way into Cubase? In any case, will the software have many of the guitar niceties found in something like Guitar Pro 6?
@Greg: The long-term plan is to migrate some of the new technology we are building into Cubase, where it makes sense. The Score Editor functionality in Cubase is quite deep and sophisticated, so it wouldn’t make sense for us to try to replace it with our new technology until it is at least as sophisticated. But over time the plan will be to bring the two applications closer together, without trying to merge them completely: Cubase will always be DAW-first, and the scoring application will always be score-first.
On another topic, slash chord staff chord analysis. Finale analyzes the F/G chord as Am7(#5)/G; G/C as Bmb9#5/C; and A13 as Gbmb9/A. Sad. Very sad. Sibelius gets it right a good part of the time, and I am hopeful that the Steinberg Notation program will get slash chord analysis right as well.