EvoGrain: Algorithmic Sound Design with L-Systems and Granular Synthesis

Nathan Ho — Sat, 17 Oct 2026 00:00:00 +0000

Grain Set #6 from further down this article.

A few years back I wrote an article here on L-systems with granular synthesis. I was rather unsatisfied with it at the time, and today I wince even looking at it. The current article is an attempt to correct this injustice with a far more ambitious project which generates audio with L-systems and granular synthesis. In this article, I will give a detailed walkthrough of the entire construction of the system, called EvoGrain, and discuss what I went through to build it. You can hear a sample of its output above, and more examples further down the article.

Source code for EvoGrain is on GitHub.

Background

I have a few sources of inspiration to talk about here.

Iamus

All computer music projects should have a dedicated egg.

Iamus,¹ created circa 2010 by a team of researchers at the University of Málaga, was one of the more compelling projects from an older era of algorithmic composition (before algorithmic art suddenly entered the public consciousness for all the wrong reasons). Its creators claim that scores are generated with no human intervention, and that Iamus is “the first computer that has developed its own original style.” The style in question is under the umbrella of post-tonal modern classical music.

The marketing copy for Iamus buries what I find to be the most compelling part of the project, which is that Iamus is not trained on existing human compositions, and it is entirely rule-based. With that in mind, take a listen to Hello World (2011), its first piece to be premiered. Any judgment of this recording has to take into account the obvious fact that it’s performed by human musicians, and they’re adding an unavoidable interpretative touch to it. Compositionally, it’s maybe a bit aimless, but it’s still pretty impressive work for something made without a corpus. The Iamus website apparently once hosted a large number of generated scores for free download, but I’m not sure where to find them now.

Naturally, I was curious how it worked. Neither the source code nor a detailed technical description of Iamus were ever made available; the project was tied to a commercial product by a company by Melomics Spain SL, which does not seem to exist anymore.² The most detailed description of the project’s approach is in a chapter by Vico et al. of the book Handbook of Artificial Intelligence for Music;¹ there are also a few shorter papers from the same team worth looking at.³ ⁴ The gist of these papers, assuming they give a reasonably accurate picture of the project, is that Iamus uses Lindenmayer systems (L-systems) and evolutionary algorithms. An L-system generates a finite string of symbols, which then encode a score of music with a sequence of commands like “increase pitch by one semitone,” “output a note,” “change instruments,” etc. The L-system is deterministic and encodes exactly one score.

The evolutionary algorithm treats the rules of the L-system as the genotype (genetic material) and the generated scores as phenotypes (physical traits of organisms), with the scores evaluated against a fitness function, with typical selection, crossover, and mutation operators. The evolutionary system also apparently included human-guided evolution.⁵

The most useful takeaway I had from the papers on Iamus is that of indirect encoding. The relationship between the genotype and the phenotype is very complex here, determined by the meanings of the symbols in the L-system and how they produce a score. These decisions are really critical to get musically interesting results, perhaps even more important than the fitness function and the evolutionary computation parts.

Other Musical L-Systems

There are many resources on musical L-systems for algorithmic composition. A good one for a lay audience is Tom Johnson’s Self-Similar Melodies,⁶ which uses very simple L-systems which are meant to be highly transparent to a listener, i.e. so-called “process music.” A paper by Mason and Saffle⁷ uses a similar method but also shows how the generated melodies can mimic statistical properties of other composers’ melodies. A very different approach by Jones⁸ uses the L-systems to partition time and pitch space. A dissertation by DuBois⁹ points to some other early examples and describes his own system, which involves real-time interaction with performers.

The combination of L-systems and sound synthesis is relatively underexplored, and the best resource on this I’ve found is Stelios Manousakis’ thesis on Musical L-Systems.¹⁰ A major part of his research is using L-systems to directly generate waveforms, which is a bit different from what I’m doing here, but he also uses an L-system–granular synthesis combination L-systems with granular synthesis that seems quite similar to what I’m doing in this project. One difference, if I am reading it correctly, is that his system is designed for real-time, while the one I’m about to describe is offline and tries to take advantage of the non-real-time medium.

EvoGrain

In this section, I’ll walk you through my own L-system granular synth, working title EvoGrain. EvoGrain is an offline program that synthesizes an audio buffer between a few seconds to 30 seconds long. The result is not meant to be a complete work of music but rather raw material for sampling and transformation. Still, it’s a plus if the results are interesting to listen to in their own right.

I view EvoGrain as living in a middle ground between a synthesizer and an algorithmic composition system. (For prior work in algorithmic sound design systems, see e.g. ¹¹.) It is also intended to be effectively autonomous. Very few parameters are exposed to the user, and most of the behavior is random. Re-running EvoGrain repeatedly should generate a wide variety of results.

EvoGrain is designed as an entirely rule-based system. There is no external training data, and there is no specific attempt to mimic existing sounds or musical styles.

The architecture of EvoGrain is as follows:

The L-system randomizer produces the rules of an L-system and an initial turtle state. The probability distributions and behavior of the random L-system generator dramatically affect the output.
The L-system is run, creating a turtle command sequence.
The turtle interprets the command sequence and produces a set of grains. Each grain is key-value data comprising start time, duration, pitch, and synthesis parameters.
The grain set is “adjusted” using some ad-hoc global methods to improve its musicality.
The grain set is synthesized to an audio file, and plotted for visualization.
The grain set information is analyzed with some statistics to produce a fitness value.
The entire process is run many times for different L-systems and a basic optimization procedure picks the best ones.

Here is a dataflow diagram:

In the following sections, I will explain each individual step. It’s easiest to start from the bottom and work to the top.

Synthesis: from grain sets to audio

EvoGrain contains a basic granular synthesis engine. I kept the DSP and the synthesis simple and conventional, because I want the placement and parameters of the grains to be the focus of the project.

The synthesizer’s job is to produce a stereo audio file from a grain set, a collection of grains. Typically there are about 20,000 grains in a set. Each grain is synthesized from the following key-value data:

Start time of the grain.
Duration.
Amplitude.
Stereo panning.
Distance: reverb send.
Amplitude envelope: rectangular, decaying, reversed, or Hann window.
Pitch.
Synthesis engine: either “Basic” for basic shapes, or “PM,” for 2-op sine wave phase modulation.
Timbral parameters:
- Basic engine: choice of waveform, sine wave or triangle. I originally added square and saw waveforms to the Basic engine, but decided I didn’t like the result much.
- PM engine: index and ratio.
Inter-onset interval (IOI): nominal distance to the start time of the next grain. (Not actually used by the synthesizer, but convenient to treat as a grain parameter.)

I didn’t bother with antialiasing in the oscillator, because who cares, it sounds fine.

There is some amplitude compensation so that the volume tapers off if the pitch exceeds a fundamental of 1000 Hz, which mitigates ear-piercing highs. I also reduce amplitude a little bit for longer durations.

The “distance” parameter’s main function is to send the grain to a reverb. I implemented a stereo-to-stereo 16x16 FDN reverb with a mixing matrix inspired by a paper by Menzer and Faller¹² and energy-preserving LFO-modulated allpass filters.¹³ It’s not a great reverb and I spent little time tuning it, but it gets the job done. I was conflicted on whether I wanted to add reverb at all, but after implementing it, I found it makes the results easier to listen to. That is rather important given that I’m on headphones and auditioning dozens of these during the development process. EvoGrain outputs a mixed-down stereo file, but also separate files for dry, wet (before reverb), and wet (after reverb) so that the user can do their own mixing.

Several grain parameters are encoded using the logarithm of the parameter instead of the actual value: amplitude (dB units), pitch (MIDI note numbers), duration, IOI, and PM ratio. Other nonlinear mappings of grain parameters to synthesis parameters are possible; one example would be quantizing the pitch parameter to a scale.

Adjustment: improving the grain set

Adjustment is my term for editing the grain set after it is generated. Adjustment is not a standard part of the L-system pipeline, but it improves the musicality of the result. Currently I have the following adjustment steps:

The entire time span of the grains, from beginning to end, is scaled so that it is within a target range of durations.
The durations of the grains are all scaled together so that their median is 20 milliseconds (exact value tunable by user).
If a lot of grains overlap each other, their amplitudes are reduced slightly to avoid excessive dynamic range, a bit like a compressor.

More complex adjustments are possible — correcting panning so it’s centered, etc. I recommend making all adjustment steps deterministic.

Interpretation: from command sequences to grain sets

A quick search for L-systems shows various nice fractals created with them. The step between the string rewrite system and the graphical output is turtle graphics. In the textbook example of this model, you have a turtle in 2D space with a state comprising position and orientation and possibly other things, and you send it commands like “move forward 1 unit” or “turn 30 degrees” or “set color to green.” Interpreting a series of commands alters the state and creates a final drawing.

While not inherently graphical, EvoGrain’s method is pretty similar to turtle graphics. The EvoGrain turtle consists of:

The “turtle state,” comprising the parameters of a grain: start time, IOI, duration, panning, amplitude envelope, etc.
A FIFO stack of turtle states, which is initially empty.

The commands of the turtle are as follows:

Grain (g): use all the parameters in the turtle’s current state to produce a grain, then advance the start time forward by the IOI.
MoveParam(param, delta) (m): move a numerical parameter by a relative amount. For example, MoveParam(Pitch, +2) increase pitch by two semitones. In the next section, this command will be revised.
SetParam(param, value) (s): set a numerical parameter to an absolute value.
FlipPan (p): mirror the pan parameter from left to right.
SetEnvelope(type) (e): set the amplitude envelope.
SetSynthEngine(engine) (y): set the synthesis engine.
SetBasicWaveform(waveform) (w): set the waveform for the Basic engine.
Push ([): save all parameters, including time, to the stack (FIFO).
Pop (]): remove the last entry from the stack and set the current turtle’s parameters to the entry’s parameters.
PushSequence (<): same as “Push.”
PopSequence (>): remove the last entry from the stack and set the current turtle’s parameters to the entry’s parameters, but do not change the current turtle’s time parameter.

Clearly, Grain, MoveParam, and SetParam are the most important here. If you interpret 10 Grain commands in a row, you’ll produce 10 identical grains evenly spaced in time. If you insert MoveParam commands in between, with small deltas, the grains will be different from each other, creating smooth gestures. There’s also the balance between absolute (SetParam) and relative (MoveParam) parameter changes; if it’s mostly MoveParams, then the movements of parameters will be incremental, and if it’s mostly SetParams, the grains will jump around more (and in the context of L-systems, often jump to consistent and specific values). The relative occurrences of these three commands are musically relevant, and the effects of changing them are easy to hear.

The four “bracket commands” Push/Pop/PushSequence/PopSequence are important for creating repetitive structures (although the L-system itself can also create repetition). Push and Pop, crucially, allow the turtle to teleport backward in time and create polyphony.

Commands can be as ad hoc as desired. For example, I added FlipPan because bouncing back and forth between the L and R channels just sounds nice. A well-designed command set will bias the system towards better results. You might be able to think of more turtle commands — some ideas I had but didn’t try include: velocity setting that multiplies the delta of future MoveParam commands allowing slowing down or speeding up or mirroring the turtle movement; a variant of SetParam that moves to the average of the current value with the new value rather than a hard jump; using the current size of the stack to influence grain parameters; commands to disable or enable pitch quantization to a scale; a command that creates a marker in time that causes nearby grains to be modified (attracting or repulsing them, etc.).

Dealing with boundaries

MoveParam has a major issue: there is no particular logic to prevent a turtle from walking over to a wacky parameter value and possibly rocketing off to infinity. Using more SetParam and Push/Pop commands can help mitigate this, but are insufficient to prevent it. A naive solution is to clip out-of-bounds parameters so they are back in bounds, but this often results in the turtle sticking to the boundary, which isn’t great especially for pitch. Alternatively you can mirror the turtle between the two bounds, but this feels like a cop-out, and the behavior is unmusical as often results in the turtle zigzagging between the two boundaries. You can also let out-of-bounds parameters happen and address them at a higher level by discarding or penalizing L-systems with such problems.

But my favorite solution was inspired by Manousakis, where he describes “threshold” operators. The idea here is to make the logic of relative parameter settings more complex. We add a new field to MoveParam like this:

MoveParam(param, delta, reset): move the parameter by relative amount delta. If the resulting value is out of bounds, reset the parameter to absolute value reset.

The “reset” parameter should be selected so that it is always within bounds. Although ad hoc, this makes sense if you think of how a jazz improviser might operate — if you do an upward run to the very highest register of your instrument, jumping back to a normal range is a natural thing to do, especially if extreme registers cause fatigue.

But still, the turtles consistently run up to the extreme edges of the boundaries, which I still find unmusical. To fix this, I used fuzzy logic. Each parameter is assigned a trapezoidal function f(x) which is 1 when we are completely in bounds, 0 if we are totally out of bounds, and ramps linearly between them:

MoveParam(param, delta, reset): move the parameter by relative amount delta. Let x be the resulting value. Set the parameter to x f(x) + reset (1 - f(x)).

Thus if f(x) = 0 (in bounds), it moves the parameter normally; if f(x) = 1 (out of bounds), we reset; otherwise, we linearly interpolate between those behaviors. I like this approach as it causes the marginal distribution of the grains for each parameter to taper off as they approach the bounds.

“Boundary hugging” problems are a well-known in algorithmic music. A paper on algorithmic melody generation by Brown et al.¹⁴ discusses this issue and using an “elastic band” in melody generation, where with each new note we average the pitch with the center of the range. This “elastic band” helped inspire the above fuzzy logic approach. Remember, setting boundaries with your turtle isn’t selfish, you’re just practicing self-care.

Expansion: from L-systems to command sequences

Many good resources introduce L-systems, such as a 1986 paper by Prusinkiewicz.¹⁵ I will go over L-systems here if only to disambiguate terminology. An L-system consists of an alphabet of symbols which are either variables to be rewritten, or terminals which are left untouched by the rewriting system. In this case, terminals are precisely the commands given to the turtle. The L-system also has one variable designated as the axiom, and a set of rules so that each variable is mapped to exactly one string of symbols (the successor). The L-system is run by starting with a string consisting only of the axiom, and rewriting variables according to rules to produce successive generations. The L-systems I talk about here are deterministic and context-free (“DOL-systems”).

Many L-systems will keep generating longer and longer strings infinitely, but we need a finite result here. One option is to only iterate a finite number of generations. I ended up using a technique inspired by the Iamus papers, where I assign each rule a budget, which is a positive integer defining how many times the rule can be invoked. After a rule is used that many times, its associated variable is replaced with an empty string.

It’s worth noting that while many examples of L-systems use a finite alphabet, this one doesn’t, as commands allow parameters like SetParam(Pitch, +60.234). However, since commands are terminals, the rules don’t know about them.

All steps described so far — expansion, interpretation, adjustment, and synthesis — are all deterministic, and the L-system unambiguously encodes the audio file it generates. Although there is considerable literature on stochastic L-systems, determinism is best in this context. It’s nice to be able to tweak these steps and see how the changes impact the result without a confounding factor of randomness in between.

Randomization: generation of L-systems

I’ve walked backward through the entire process of going from an L-system to a final audio file. Now for the first step in the process, which is producing the L-systems themselves. This step is very open-ended and I’m sure you can find ways to improve and expand on what I’ve done here.

The main way that L-systems shine is their ability to model hierarchical outputs. A lot of music is hierarchical in some way — an entire piece divides into sections, sections divide into phrases, phrases divide into gestures, gestures divide into notes. Although I’m not trying to generate entire pieces, I find EvoGrain’s output most personally appealing when it has a clear fractal-like shape to it, just like an entire composition might.¹⁶

Many examples of L-systems are handwritten ones with only a few rules and a few symbols. Here the L-systems are larger, with 50 rules and up to hundreds of symbols in each rule. Here the 50 variables — the symbols that only exist in the L-system to get replaced with other symbols — are given only numbers and labeled #0, #1, #2, … with #0 being the axiom. The symbols are divided into two groups: #0-#19 are macro variables, which are viewed as creating the largest structures in the grain set. #20-#49 are the micro variables, which produce the small gestures of maybe a few grains. Although I found 20–30 to be a nice split, the number of macro variables and micro variables can be changed.

The randomizer uses the following logic to generate L-system rules:

For a macro variable #n, the successor is a string with about 15 symbols, and each one is:
- 90% chance of macro variable randomly selected from #(i + 1) to #(i + 4) inclusive.
- 10% chance of a random turtle command.
For a micro variable, the successor is a string of 50–500 symbols, and each one is:
- 95% of a random turtle command.
- 5% chance of a micro variable.

So #0 is the entire piece, #1 is the highest-level structures, #2 the next level, etc. But they do not perfectly nest, as #0 can jump down the hierarchy and invoke #4. The later macro variables can expand to micro variables, and the micro variables can reference each other ad infinitum.

The numbers of symbols in the the successors affect the length of the generated command sequence, and ultimately the number of grains in the final result, so it is quite important to tune it correctly.

To generate a random turtle command, we just pull commands out of a hat. The probabilities are selected so that MoveParam and Grain are most common, SetParam is relatively rare, and other commands even rarer. The bracket commands [ ] < > are inserted after all the others in a way that guarantees that they are always matched. This is done by selecting a random set of insertion points and generating a random Dyck word.¹⁷ I don’t know how necessary that is, but it bugs me if they aren’t.

SetParam and MoveParam commands have numeric fields that the randomizer must supply. To set these values, I have manually defined two probability distributions for each synthesis parameter: an absolute distribution and a delta distribution for each grain parameter. For example, for pitch the absolute distribution is a uniform one from MIDI note numbers 30 to 120, and the delta distribution is a normal distribution with standard deviation 3.0. When the randomizer creates a SetParam(Pitch, <value>), the value is sampled from the absolute distribution, while the delta in MoveParam(Pitch, <delta>, <reset>) is sampled from the delta distribution (and the reset value from the absolute one). Most of the delta distributions are symmetrical around 0, but the distribution for amplitude (in dB) is skewed so that amplitude is more likely to decrease than increase from grain to grain, which sounds nice since it vaguely mimics decaying echoes. These distributions are only known to the L-system randomizer to produce these commands; they are not sampled by the turtle, because interpretation of the command sequence is deterministic.

As an example, here are the rules of an L-system randomly generated by EvoGrain, which is at the top of this article and below as Grain Set #6. Each command is represented with a single letter, so for brevity I am not showing the arguments for SetParam (s), MoveParam (m), etc. Viewing these rules directly is not very illuminating for understanding the grain set, only for illustrating the behavior of the randomizer.

#0 [budget 5] -> #1<#1#2<#4#2>m#1>#4#4#1#4[#4]#1
#1 [budget 6] -> #5<g[#5]#2g<#3>#3>
#2 [budget 9] -> [#3#3[#4]#5[m]#4<#3<#4[g]#3>#3>#3]
#3 [budget 11] -> m<#6#7>#7<#6>#6#7#4#4<#6#6[#6#7]#4#7>#6
#4 [budget 10] -> #7#7[#7#7]#6<#6#5<#5[#5m]#8#7[#7]#6>#7[#7]#6#8>#8#8
#5 [budget 13] -> [#6[#6<m>#8[#6]#7[#9#7]#8]#7]
#6 [budget 14] -> #10<#10<#8#8>#10#7#8#8<g#8gg#9>#8#9>
#7 [budget 12] -> <#9#9<#10m[#9]#9#9#9>#9[#11]#9#10>#8#8#9
#8 [budget 9] -> [#10[#12[m]#9]#10#9#11]#11#10#10[#12]#11<#12>#10
#9 [budget 10] -> #11#10<#11#10#10#12#10#11#13#10#12#13#12m>#11#13m#12g#10
#10 [budget 8] -> <#14#14[#13]#11<#13>#13>#14#14<#11[#12]#13>#13[#13#12[m]#12]
#11 [budget 10] -> #15#15#13#15#12[#15#15#12]#13#15#12#14#15#14[#14#12<#12>#12]
#12 [budget 9] -> #15#15[#15]#15<#15>#16<#14#13>
#13 [budget 13] -> <#16>#16#16[#16<#16[m]g>#17]
#14 [budget 8] -> <#16#15<#16>#18>#16[#17]#15
#15 [budget 11] -> <#18m<#18>#18[#17[#18[#16<#17#17>#16]#17]#16]#16>
#16 [budget 8] -> [#18#19#18<#17#17#19>#18]#19#17m#17#17#19#19#17#18[#17]#19#19#19
#17 [budget 17] -> #20<g#19#18#21>#21[#18]
#18 [budget 15] -> #19g#20<#22>#20#22[m#20#20#20]#20#20
#19 [budget 17] -> #20<#20#23#22#22#21#21#20#21#23[#22#23[#23]#20#20]#23#21#20>
#20 [budget 11] -> #21gmmmggmmggmmmmmmm#36gm#22mgggmmgm#25#26mm#31<#35[#30mg]m#23#33gmgmmmmmmgmmmmmmmm#34mgmmmmmm<mm[mmmmgmmmgmgm]mgmmgmm#21mmmggmmmmmgmggmm#32gmmgmgmmgmgggmmgmmmm[g#43mmgmmmmgmmgmmmgmm#46mggggmm#20#27gmmggmgmmggmwymmmmmm]mmmgmmmmmmmmgmm>wgmmmmmmmmmm#41m#31mmmmggggmggmy#37>mmmmmmmmgmmgmmgmgmmmggg#35gmgm
#21 [budget 17] -> mggmmgmmmgmmmmm<mmgmmmmgmgmmmmgmmmmgmmg<mmmmmmmmmmmgmmmmm[gm]gmgmgmmmmmgmmm#36mmmmggmm>mgmmmmggmmmmmgmmmm#30m>gmmmgmmmmmgmgmmmmmmmmgmgmgmgmgmmmmmmmmmmmmgmmmmgmmmmmgm#24mg#24mmmmmmmmg#40mmmmgmmmgmmmmmmgm#42g#43#47mmmmgmmy#22wggmgmgggmmmgmgmmmmgmmmmmgmgmmgmmgwmmmmmmmmgmmgmmmggmmgmmmgmmmmgmgg
#22 [budget 13] -> m[mmgmggmmgmmmmmmmmmgmmmmgmmggs]mgmmmgm[mmgmg#37mmgggmmmmgmmmmmmmmmmmmmgmmmmmmmgmmmmggemmmggmgm#42mmsggmmmgmmmggggmmmpmmgg#22gmmmm[mmmmmmm]#34gmmmgmgmmmggggmg#31m]mmmgmmmmmmmmmmgm#28mmmgmmggmggmmmmggmmmmmm
#23 [budget 8] -> mmmgmm#42ggggmmmmmgmmmsmmmmmmmgggmmmmmmm#31#20ggmmg<mmmgmmgmmmmggmm#27mmpmmm#32mmgmggmmm#37mmmggggmmgmgmmgmmmmmgg#41mmmmmggmgm#46gmgmgmmmmggmgmgsmmmmgmmgmgmmmmmmmmp#26mgmmmmmgggmmgm#22mgmmmmgmmgmmmmgmgmggggmg<gmgmmggmmmmgmm>#36mmggmmmmmgmmmmmmmmgmmmmmmmmmgmggmmmgmmmggmggm>mmggmmmgmmmgmggm#43gmmmmmmmmmgg#43mggmgmmm#39mggmmmmmmgmmmmmmmmmgmmmmgmmgmgmm
#24 [budget 9] -> mmmm<mmmggmmmgmmggmgmm#37mgmgmmmm[mmgmmmmggmmgm]gmmmmmmmmmmmg[mgmmmmmmygmmgmmmmmgmmmg#31m#32ggmmm#47m<gm>m#35gmggmmm#40gmgpmmmmmgmgmmm#34mmmmmm#37#42sm#36mmmgggmmgmgm#47m[ggmmmmmm#26mggmmmmgmmm]mmg[mgm#47mmmmmmm#40mmggmmmm#31gmgmgmmmmmmmg#29gmmmmmmgmmmgmmg#45mmm#44#36mgmmmmm<mmm#38<mgggmgmmgmmmm#25mmmmmggmmmmmmmmmmm#44mmggmg#33ggmmm#46gmggmg>mm#28#45gmmgmgmggmmmmmmgmggmgmmmm#27mgmmmmmmm#30mm#37gmgmmmmmgmmmgmgggm>mgmmmmmmmmmmgmmggggymmgmgmmgmmmmmmggmgmmmmmmmggmmmmmmmmmmgmmm]ggm#48mmmmgmgmmmmmmsmymmmeggmmgmmmmmmmmggmgmmmmgmmmmmm]mmmm>gm#33megmmgmmmmmmmmmgmmmgmmmmgmmmmmmm
#25 [budget 14] -> gmmmmmmggmmmgm<gmggmmmmmgmgmmmmmmmmmm#22mmmmmgmmgmmmmmg#34>mmgmgygmmgmegmmmmg<mm<m#25mmmmgg>m<mmm>g>mgmm[mgmm]gmgmmm[mgmm]mm<mgmggmgmmmm>gymmggm
#26 [budget 13] -> ggmmmmmgmgggmmmm<mgmgmggmmmmmgmmmmmmmmmgmm#33mggg[mmgmmm#37mmgmmgmmgg#25]#44gmm#40<mmmm#34gmmmggmggmmmgm#41mgmmgmmgmmmgmmmmmgg>mgmmmm>gmgmggmgmmgggmgw<gmmgmmgg>mmmgmgwgmmggm[mmgsmmgmgm]mmmm#38gmmmmmmmgm#27mmggmmmmmmmmgmmmgmgmggmg#44mgmmmmmmmmmpmggmmmgmmmmmmmmg#47mgmmmm#28mmmmmgymmmgmmmgmgmgmmmmmmmmmmgmmmmygmmmmgmmmggmgmm#43gg
#27 [budget 7] -> ggmm[mggmmmmgm#27ggmgmmmmmmm]#28mgmmmmmmmmmmmmmmmmmgmmmg<gmmmgmmggmmmmggmmggmmm#26#36mmggmmmp#37mmmmmm#31ggmmmmmmmmwmggmgggmmmmgmmgmmmgmmmggmmg#24mmmmmmmmmgm#20m#38gmmg#25#22mmmmgm>mmmmmmmgmmmmmmgmmgmgmgggmg#38#44mgmmgmmmmmgmm#44mmmmmmmmmgmgmmmgm#47mmmgmm[gmmmmg#47mmmgggggmmmgmm[mmmmmpmmgmgmmmmmgmgmmmgggmm#27m]mmmmmmggmgwgg#42mm#31ggmmmgmmgm]gmmmmggmmmmmmmgggmg#24ggmmm
#28 [budget 11] -> mmmmgmmgmmmmgmmg#41mmmmgmmmmmmmmmmmgmgmmgmgmmmmmmgmmmmmmmmgmmgmggmgmgmgmgemmmmgpmmmggmgggmg#39<mmmgggymmmgmgggmmmg#40mmmgmmgsmmgmggmmgmmgmmmmmgm#47mgmmmmgmmm<gmggmgm[mmmmmgmgg#44mmg]m>mgmmmggmgym>mmgggmgmmmm#42ggmgmmg<mmmmggmggmmyggmmggmmmmmggmmmmmmggggmmggggggmmgmmggmmmgmmgmgsggmmmmgmmmmgmm#32m#20gmmmgmmmgm<gmgmmmym#48mmggg>mmgmmmgmmmmmmmmmmmmgmmmmggmmmmmmmmmmm#32mmgmgmmmmg#31mgmmmmmmggggmggggmmgmmmmgmmgggggmmmgmmmm>mgmmmmmgmmmmmmggmmmgmggmmmmmmmmm#30g#29gmm#31
#29 [budget 14] -> <mmmmmmmmmmmmmg[mm<mmmgmgmmgmmmgm>m]ygmmmmwgmmmmmgmmmmgmmmmmmmmmgmmmmmmmg#31ggmggmgggmg#41<mmmmmmmmmggmggwgmmg#20mggmmgmmmgmmggemgmggmgmmmmmmmgmmgmmm#39ggmmgmgmgmmm#20mgmmgy#49mmm#20gmmgmmmmgmmmgggmmmgm#22mmmmgmmggmggmmgmmgmgm#40gmgm#21gmm>mmmmmgmmmmgmmymggmmggggg>gmg#29gymmygmmmm#27#37m#44m#33mmmgmm
#30 [budget 7] -> mmmm<mmmmmgggm[mmmmggmgmgmmmmgmmm]gmmmmmgmmmmgmggmmmgmmmgmmgmmmgmmm>g#40gmmmmsgmmm#49mmmmm#21gmmmmemmmmmgmmmmgmmmgmgmgmggmmmm#37#45mmm#25mmmgmm#28wmmmggg#29#36ggm#20mm
#31 [budget 9] -> mggmgmggmm[mmm#32mmgmmgmgg#39ggmmmmmm#22mmm#21gmggmmgm#21mmgmggm#38ggmmmg#27mmmm#25gmgmmggmmmmgm#39ggmmmmmmmmmmmmg#35mmmwgmmmmmmmmmmsmmgm#24mmmmmgmmggmmm#49gmmmgmmmggmmgmmgmmmygmgm#48m#41gmggggwmmmgggmmmgmmmgmmmmmgm#46mgmgmmmmmmmmmmmgmmmmmmmmmmmmmmmsmmgm#27mmg[mgmggggmgwgmmmggmg#22mggmgmmmmgmmmmmmggmm#23gmgmmmmmgmmmmmgg]mmmmmgmm#24mg]#23gmgmmggmmmmm#48
#32 [budget 13] -> mmmmgggg#39mmmmmmmgmgmmmmgmggmgmmmmmgmm#25g#48mmgmmm[mmmmgggggggmmm#34gmm#34mm#28mmmmmmgmmmmgmm#31mmmmmmmg#35mmgmmm#46mmmmmmgmmmmgmmmm#42gmmmmmmggmmmmgggmmmm]ymgmmmwggmmmmmm#35mmemmmsmmmm#23mgmmmmm[gmmgmmgmm#27mmm#44mgmmmg#44#44mmgmgmmmmgmgg#28mmmmgmgmmmmmgmmgmmgmgmgmgmmmgmgmmgmmmmg]gmgmgmmmm<#47mgmmgmmmgmmmmmmmm#35mmmmmmgm#26mgmmgmgmmgmgmmmmgmmgmgmmmmmm#36mmgmgmmgmmmmgmmym#23gwgmmmmgmmmggm#40mmmmgmmgm#40mmmmmgmmmgmgmmgmgmmmgmmgmmmmmmgmg#42mgmmmmggmmmmm#25mmmmmmgmm>mmmmmmmmmmmmgggmgmmggmm#28gmmgmg
#33 [budget 5] -> gmggmggmmm[m#42g<mgmgmmgmgmmmm#27m>ggm#22mmmgggggmmgmggmggm#27#42mmmmmggmggmmgmmg#29mmggmmmmgmgm#47mmmmmmmmmmmmgmg#22gmm#21mmmgmggmmmggmmmmgmmmmgp#28#33gmgmmwmgmmmmggmgmgmgmmmmgmmmggmmmmmmmmm#24mmpmmgmgmgmmmmmggmmgmgmmgmmmmgmgmmgmggmgmgwmmmmmmmmmmmmm]mm#33mgmmmmmmmggmgmmmgemg#33gmmmmmmgmmgmmmmgg
#34 [budget 10] -> mggg#24gmmm#28mmggmmgmgmmmmgmmgmggmmmgm#33ggmmggmmmmmmmgm#39mmmmgmmmggmmmmg#49mmg<mgsmmmmggmmmmmmmmmgmmmgmmmmmm>mmg#43gmmm#49mm<mmmgmmmggmmmmmgmmegmmgg#36mgmmmm>mmmgmmm<g>mmmmgm#46m[ggmmgggmggmm#49mmmmmmmmmmmmgmm#22gmggmmmgmmmgmmmmmmmm#31ggmmggmmmgmmmmmmg#26mmgmgmggmmgmmmmmmmmm[mm]mmmmg]msmggmmwm[ge]m#40mgmmmggmgmmmgmmmmmmmgmmmgm
#35 [budget 13] -> gmm[gm<mgmgmm#28gmmmm[mgm]mmgm#46>mmmgmgmgmg#36mgm#48m#24gggmm<mmmmmggm[mmg[mmg]gm]m#41>m]gm[gmmm#22mmmmmmmmm]mmygmm#44mmmgmggmmmm
#36 [budget 13] -> mmmgmmm[gmm]gmmm#42mmmm[ggggmm<gmmmg>gp#26mmggg<m>mm]mmg[g#42<mg>mmmm]mwg
#37 [budget 11] -> ggmmgmg#38g#43gmgm#26mmmmgmmmmgm#29#28ymmmmmgggmmmggggmmmmgw#20mgmmm[m[mgmgm#24mmmmggggmmmgmmmmmmgmmmgmm[g#49mmmmgmggmmmg#21]mmmgmmmmmmmggmmmmmmmgpmgmgmmmgmpgmmm[gs#30mmmm]gmmgm]mmgmm#33mmm[#35mmmmmm]gmgme]mmmmmmgggggmmm#45gmmm
#38 [budget 9] -> mmmgg#26mps#46mmmmsgggmgg#45gm[gm[mmgmmmggmmmgmmmggmgm#35mmwgmm#34mm[mmmgmmmmm#33mmmmmm#25m]mmgmgmmmggm<gggm<gg>mmmmmmgmmmgmmggmmggg[gmgmmgmmmmmgmm<mgg>mmgmgmgmggm]mmmmmmgm<mgmmgm>mgmmggm#32mgmmmmmmmm>mgmgmmgmmgmmmmmmmmmmmmmg#32mmmmgg<mmmgmmmem>mm<mmgmmmgmm>mmg#41gg#40mmmmmmmmmmmggmmmm#28mmgmm#29m#32g]m]mmmmgg
#39 [budget 10] -> mmgmmmgmgmgmmmmmmgmmmmmmmmmmgmmggm#33gmmg[ggmmmmmmmgmmgmmmmgmmggmm#21<m#30#32mgmmmgmmmmgmgmgggm<#29#35mmgmmmgmmmmgm#41mmmmggggmmmmggmmgmgmgmmggmgggmmmmmggmmmmmmmmmmmmgmm>mgmmgggmmgmmwmmmmg#24mmmmmmmgmgmggmmmmmgmmmmmmmgp#45ggmmmggmmmmgmm#40gmgmmmmmmm#21mgggmmmgm#47mmgmmgmmgm#25mmmmmmmmggmmmmmmmgmggmmggmmmmmmmymgggmmmgmggmmmm>mmmmmgmmmmmmggmmmmggmmmmgmgggmg]mmmggmmggmgmm
#40 [budget 15] -> #21mmm<gmggmgm[ggggmmgmg#28mgmgmm]wgmmmmmgggmmmmgm#23g>mmmgmmm#42#23mmmsmgmmmgggmmmmmmgm#42mmm<m#48ggmmggmgmgmmgm>m#38mm<mmsmmgmmmg<mgmgmmmmggmmgmm>mmg#24gmmm#33ggmg#31m>mmmgmggmmmmm<mmmmgmmmgmgmmmgmmmmm>mmggmmmmgmgmmm#32ggg#23mmmgmgmmgmgmmmmmgmmmggmggmmmmgmmmmmggmgmmgmg#32mmgmmmggmmmmg
#41 [budget 18] -> mwpmmm#45gmgg[mmg#35m[mmgms#23mmmmm<mggggmgmgmmmmmmgmmmmmmmmmggmmmgmggmmmmgmmemmmggmmmg>mmmgmmmmmmm#43gmmmmmgmmm#27mgmmmmmmmm#37mmmmmgm#41mmmmmmmmmmm#48mmmmmgm#22mmgmgmmmgmggmmmmg#21mgmmmmmgmmmmgmm#20mgmmmmmmmmmm#44mmgm]gg#29gmmmmgmgmm[mmmmgmgmmgmmmmmmmmmwmm<gmmmmmmggmmmmgmmmmgmmgmmmw>mmmgg]mmgmmgmmmgmmmgmmmm<egmmymmmm#23mmgmgmmmggmmmwgmmmmmmmgmmm#23ggmgmwgmmmmmggmm[mmmmm]msg#32mgm>mmmmg#22mgmgmmmmpwg#45gmg]mmm#37mgggggmmmgg#41mmgmmm#31mgg
#42 [budget 9] -> mmmmmmmm#44mggmgmmmgmmmmggmmgmm#35[mgmmmm#22mmgmmw#36gmmmmmmmmgmm#34gmmggmmmgmmmmg<ggmmmggmg#27mmgmpmgmmmm#36mmggmym#34gmmmmgmgmgmmmmmm[mmgmmwgmmmgmgmgmmmgmmmgmmm#31mmmggmmgggmmmmmmggmmmmmmgggmmmmmgm]mgmmmmmmgg#45mmgmmgmmmgmmmmgmm#20mgmmggmmmmm[mmgggmmgmmgmmgmmmmgmmmmmmm#33gm[gmmmm#30ggmggmgmmggmgpmg#43mmmmmgmmmmgmmmgmmmmymmm#44gmmg#47mgmmm#28#40mmmmmmmgmgmmmgmmmmmmgmmmgmggmmgm]mm#36mgmgmgmgmmg#49mggmmmmmgmgggmmm]gm#34mgmgmmm>m#29mmmmm]mmmggmmmmmmgmgmmgmmmm#20mmmgggmgmggymgmmgmggmm#21gmgmmmmmmmmmmmm
#43 [budget 8] -> #21m[gmmmgmgmpgmmmgemmmgmm[m#33g#35mggmmmmm]sgmmmgggmmmgmmgggmgggmmmmmmgmmmmmmmmggmmmmmmgmgggmm#35mggmmmmgmm#39gmg#21wmmmmgmgmmmmmmpgmmg#33mmmmgmg[mmmggmmgmgmmmmmgmgmmmmmmmmmgmg#38mmmm#22mmmmmgmmmgmmmmmgmgwmmmmmm<mmmmmmpmmmmggmemmmmmwmgmmmmmgmmgmmmmm#48mmmmmgmmmmmmmgmm#49mmgmmgmmmgmm#26mmmm[sm#44gmmmmmggmggmmm]mm#29mmg<mmmmgmmgmmggmg#22mmmmgmmggmmmmgmmmmgmmmmmmmgmmmgmwmgmmmggmmmmmmm#33#45gmm<mmmm#43mgmmmmmmm#25mmmmmmg#37mmmmmmmmmmmggggmgmmm#48mmmm#20mmmm#38mm#42mgmm>mg>mgmmmgmmmmmmgmggmm#40>gggmmgmyg#36mmgmmmmmpmmmmmmmmgmgmggmmggmgmmmmgpgmm#47mmmm]#30mm]gmmmgggmmgmmgmgmmmggmmgm#36ggmmgmmmmmmmmmgmmgmmmggmm
#44 [budget 18] -> gmgm[mmmmmgmmgmmmmmmg#33gmmmgmgmmmgmggg#34mggmmmgmmgmgmgmmmm#39mmmmgmmgmmmmmmmmmmmmmggmmmmmmmmmmmmmmmmmgmmmm<ggmmmgmmmmmmmmgggmgmm<mmgmggmmgmmgmggmmggmmmmgmgmmg#43mmgmgm>mmgmg#26mmmmgmmmmmmmmmmgm>mmmmpmgmmmgmggmmmggm#24#22mmmmggmmmgmmmmmmgmgmmmmggmgmmgmgmgg<mmmgmmpgmmmmmmmggmmmgmmmmgmmgmmmmmggmmmg>mmmggmmmsgmmmgmmmmmmgmgmmmmygmmmgmmmmmmg#35g#20mmmgmmmmggmmgmgmmwmmmmmmmmmmgmmmm#33mmmmmmgmmmmgmgmm#34mmmmmm]mmmmmggpmggmmsmmmmgmmgg#43m#39mmggmmg#37m#35mgmmmmgmmggmgm#28mgmggmmmgmgmgggmmgmgmgm
#45 [budget 10] -> g[gm#24gggg#21gmmmmmggggmmmg#21m#49<gmmmg#43mmgmg<mmmmgmgmmmmmmm#42gggmmm#49m>gmmmmmgmmmggggggmmgmmmmmmmg#49mmggm[mmgm]gm#46gmmggm#32mm>g]mmegmsg[gmgmmgmmmggmmmgggmgmmgm]mmg#28mmmggggmgmmmmgmgmmm[gmgmmm]mmmm#48mgggmgmmgmmmggmmmmmgyg
#46 [budget 12] -> mmmmmm#36mmmmmmgmgmmmmgg#39mmmgmmmmmmmmmmmg[gmgmmmg]mggmgggmmmggmmwmmmgggmggmmmmmmg#32gmmgmmgm<mmmgmmmm>pggmgmmmm#42g#42gmmmggggmmmmmmgmmmgmmggmggm#25m<ggmmmmmmgmmmmmm>gmmmmmmmgmgmgmmmm#20ggmggmgm#21mmgggmgggmmmmmmm#42gmmmmgmgmgmmmmg#30m
#47 [budget 10] -> mgmmmmggg#39gmgmmmmmmgmmgmmggmmmmmmmmgmmmggggmmgmgmmmgmgmmmggm[gmgmgmggm#47#30mmmggmggmmmmmmmmgm]mmmmmm#39ggmm#40mgggmmmmgmgmmwmggmmmm#30m<mmmmmmmggmg#24mm#49mmmmmgmmmmmgemgmmmgggmgmgmgmmmg>#40ggmmm#44mmsmgwmmsmm#22mggmmmmmggm#31mmmmmmmgmgggggmmggmmmmmmmgmmmgmmmgmmgmmmmgmmmmgmgmggmgmy#39gmmmmmmggmmmmmgmmmmmmgmmmgmmg#38mmmmmmgmm#31mmmgmm
#48 [budget 11] -> mmmmmg#35gmgmmmmgmgmmmmsggmmmm#28gmgmmmmmmmmmm#32gmmggmgmmmg#43gmmgggmmmmmgmgggmmgmmgmym#20mgmmgmmgmmgm[ggmmmmmmmmmmm#21mggmmmmmmmmmmgmgm#29mmggmmgmgmmgmmgmmmmmmmmmggmgmmmmmmmmgggg#32m#22gmmmmmmmmwgmggpmmggmmggmmmgmmmmmmmgggmgmm<mgmmmmmgmmggmmmmmmmgmmmmmmgmmmgmmmgggmmmmmmmmmmmmgg#32mm>gmmmggmgmmmgmmgmmmmgmmmgmmmggmm#46mmgmgmm#41mmmgmmmgmmmgmsmmmmmgmggmm#24mgmmmmgmmgmmmmmmgmgmmmmmmmm#28mgmmm#38gmgmgmmmmmymgmmmmgm<m#31g>#36mmgmmmmmmmmgmgmmgmmgmmggmmmmmggmmmggmmgmgmmmmmmmgymmmmmmggmmmymmmmgmmmmmmmg]gmmggmmmmmmmmmmmmmmmmmm#23mm#26gmmgmmmmgg
#49 [budget 7] -> mmmmggm[mmmmmmmgmgmmmmmmmm#28mgmmggmgmmmmggmmm#44mmgmmgmmmmgggggmmgmmmpmmmgmmmm<m#21mgmmgggmmmmmmgmmmgmm[mgmgm]mmmgmgmmgmmmgmm#43mmmm#46gmmggg#23mgmegg#21m#28m#23mgm>mmmg#49mmmgmgmmmmmgmm#30gmmggmmmgmm[m]mmm#34mgggmg#49g#41gmmgm#43mgm]mmgmmmm#35mmmmmgmmggmm

Evaluation: the fitness function

It’s quite difficult to predict what the final grain set will be from an L-system aside from just running it. As a result, it is entirely possible that the grain set will be basura: it might have too few grains or too many. This is where we introduce the fitness function, which closes the loop by evaluating how good a given grain set is.

To compute the fitness, first I compute some descriptive statistics:

Average number of grains per second (computed as total number of grains / length in seconds)
Median and 90th percentile of pitch
Median “overlap factor” (defined as duration / IOI for each grain)

For each of these features, I define a trapezoidal function which assigns higher scores to values I like better. For example I like the median pitch to be around C4 (middle C) but no higher than F#6, so I define a trapezoidal function that starts at 0 at 60, ramps to 1 at 70, stays at 1 until 80, and ramps down to 0 at 90. I take the actual median pitch from the grains and plug it into this trapezoidal function to get a score ranging fom 0 to 1. I compute similar scores for other stats, and do a weighted sum to get the final fitness metric.

In designing this fitness function, I prioritized metrics that were easy to implement and were useful for avoiding the types of outputs I didn’t like. Currently, the fitness function is built just to reject obvious junk rather than do any kind of aesthetic evaluation. More sophisticated metrics are possible which add to this by using computational aesthetics methods to find subjectively better grain sets. I’m not entirely sure what those metrics would be. Either way, for those exploring that direction, I would recommend multi-objective optimization: one fitness value identifies obviously bad outputs to be discarded, and another does a beauty contest.

The fitness function only requires a grain set. You could synthesize the audio and then compute amplitude, spectral centroid, etc. and do time-series analysis on that. I actually did this in an earlier, failed attempt at this project about two year ago, but found that the fitness functions were difficult to design and unintuitive to tune, and also the synthesis and analysis steps made iteration much slower. Still, the aesthetic potential of having the synthesis algorithm in the feedback loop is tempting.

Search: finding L-systems with the best fitness

Although I gave my software the working title “EvoGrain” and it was mostly inspired by Iamus, an evolutionary computing project, I ended up not using genetic algorithms. The Algorithm Design Manual flatly discourages the reader from using GAs for optimization, arguing that the added complexity doesn’t pay off. That is just one opinion, but one I take seriously. Plus, it isn’t necessarily EvoGrain’s goal to find an optimal solution, but rather to just produce a diverse set of results that sound good. This suggests that very basic optimization algorithms are OK here, and in practice that seems to be the case.

The search algorithm I ended up with is to generate a pool of 100 random L-systems, evaluate their fitnesses, and pick the L-system with the highest fitness. I then perform a basic Random Search by doing the following 10 times: pick a random rule of the L-system, re-randomize it, check to see if the new L-system has better fitness than the old one, and replace if so. Because the fitness function is the sum of trapezoidal fitnesses, there is a maximum fitness which can be achieved, which allows us to quit early if we reach it. (I have a hunch that trapezoid-shaped fitness functions are good building blocks for evaluation of computational art — often in artistic decisions, there’s a range of pleasing values rather than a single best one.)

My takeaway from the earlier, abandoned prototype of EvoGrain is not to overengineer the fitness function or the evolution side of things. These parts of the system are not really where the magic happens, and the bulk of effort in this project went into the randomizer, the turtle command set, and the synth.

Implementation & Results

EvoGrain is an offline, command-line Rust program, about 2,000 LOC. There are no special DSP libraries, just the hound crate for WAV output. The output files are:

The L-system, serialized as JSON, providing a complete recipe for all other files.
Four stereo WAV files: dry, reverb input, reverb output, and mixed dry + reverb output.
A textual “report card,” displaying human-readable stats like grains/sec and fitness to help with tuning and debugging.
A scatterplot (using the plotters crate) of the grains with time on the horizontal axis and pitch on the vertical. Color indicates synth engine.

The grain sets are unpredictable. Sometimes they generate audible and visible patterns with repetitive structures, other times they produce blobs and clouds. Sometimes a bunch of grains clump up at specific vertical moments. Either way, there is always some kind of shape. This is not an endless texture started and stopped at arbitrary points; there is always global structure, and it anticipates the end of the audio (or rather, it decides when the audio file ends).

I would say EvoGrain succeeds in all the goals I set for it, except for the one of producing diverse outputs, which is arguable. I’ve listened to a lot of EvoGrain outputs and it’s turned me into somewhat of a connoisseur, with the results clearly quite different to me. However, I would not blame someone else for saying the output sounds like indistinct bleep bloops. You be the judge.

Examples

Here are ten uncurated examples of EvoGrain’s output, all using the same default parameters. I have plotted time on the horizontal axis and pitch on the vertical one and used pink for the Basic synth engine and blue for the PM engine.

It takes only a few seconds to run the entire system, from repeated randomization and search to producing audio output, so all these examples together took under a minute.

Grain Set 0

Grain Set 1

Grain Set 2

Grain Set 3

Grain Set 4

Grain Set 5

Grain Set 6

Grain Set 7

Grain Set 8

Grain Set 9

Conclusion

EvoGrain is a research project. The point here is to demonstrate what is possible with this premise, not to pitch you a new sound design secret weapon. I have made the source code available, and if you want to use it you can. Source code.

I was able to code up a quick prototype of the EvoGrain system that produced decent sounds in about half a day of work, and good sounds in about a weekend. This is an invitation to try building this yourself. There are a lot of internal parameters to tune, and many big and small decisions, and every implementation will produce a different sound palette.

A few thoughts on further directions:

Can this system be simplified while producing output which is no less musical? By “simplified,” I mean easier to describe in words or implement in code.
Any synthesis engine could be used for the grains. Scrubbing a sample buffer would definitely sound great. I avoided doing this for EvoGrain because it’s too flattering — it’s very easy to get cool sounds with buffer granulation, and I wanted to do my best to get good results in the absence of samples. The possibilities with per-grain spatialization can be expanded; even with just stereo output there is a lot you can do with Haas effect panning or binaural virtualization.
More interactivity. Maybe allow the user to edit the L-system directly. Or given an L-system, generate a handful of random mutations by making small changes to the rules, and let the user select their favorite.
Inverse modeling. A paper¹⁸ describes taking an SVG image as input and reverse-engineering an L-system that produces an approximation of the image, which is honestly black magic to me. The system even lets the user modify the rules of the graphics to edit the image (!). Would this work at all for music or audio? Only one way to find out…
Adopt genetic algorithms. Not really because they’re necessarily the best way to optimize the fitness function, but because they’re fun and cool. If you’re going to implement a genetic algorithm, why not go all in and implement a full artificial life simulator with nucleotide bases, codons, reading frames, and genomics?¹⁹

Footnotes

According to the Wayback Machine, melomics.com was last operational in early 2018. The current iteration of the site is plastered with ads for casinos. Interesting brand pivot.

¹⁶

Of course, the comparisons between fractals and music are highly dependent on musical style and culture. While I’m here, please be advised that fractals are not necessarily self-similar — a fractal need not look the same at every level of detail.

References

⁴

Casas, María J., and Francisco J. Vico. 2009. “On the Performance of Some Bioinspired Genetic Operators in Complex Structures Evolution.” Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation, July 8, 1841–42.

Albarracín-Molina, David Daniel, Juan C. Moya, and Francisco J. Vico. 2016. “An Evo-Devo System for Algorithmic Composition That Actually Works.” Proceedings of the 2016 on Genetic and Evolutionary Computation Conference Companion (New York, NY, USA), GECCO ’16 Companion, July 20, 37–38.

Vico, Francisco, David Albarracin-Molina, Gustavo Diaz-Jerez, and Luca Manzoni. 2021. “Automatic Music Composition with Evolutionary Algorithms: Digging into the Roots of Biological Creativity.” In Handbook of Artificial Intelligence for Music, edited by Eduardo Reck Miranda. Springer International Publishing.

¹⁴

Brown, Andrew R., Toby Gifford, and Robert Davidson. 2015. “Techniques for Generative Melodies Inspired by Music Cognition.” Computer Music Journal 39 (1): 11–26.

¹⁵

Prusinkiewicz, Przemyslaw. 1986. “Graphical Applications of L−systems.” Graphics Interface.

¹⁸

Št’ava, O., B. Beneš, R. Měch, D. G. Aliaga, and P. Krištof. 2010. “Inverse Procedural Modeling by Automatic Generation of L‐systems.” Computer Graphics Forum 29 (2): 665–74. https://doi.org/10.1111/j.1467-8659.2009.01636.x.

¹⁹

Burke, Donald S., Kenneth A. De Jong, John J. Grefenstette, Connie Loggia Ramsey, and Annie S. Wu. 1998. Putting More Genetics into Genetic Algorithms.

¹⁷

Arnold, D. B., and M. R. Sleep. 1980. “Uniform Random Generation of Balanced Parenthesis Strings.” ACM Trans. Program. Lang. Syst. 2 (1): 122–28. https://doi.org/10.1145/357084.357091.

¹²

Menzer, Fritz, and Christof Faller. 2010. “Unitary Matrix Design for Diffuse Jot Reverberators.”

¹³

Werner, Kurt James. 2020. Energy-Preserving Time-Varying Schroeder Allpass Filters.

⁶

Johnson, Tom. 1994. Self-Similar Melodies. Editions 75.

⁷

Mason, Stephanie, and Michael Saffle. 1994. “L-Systems, Melodies and Musical Structure.” Leonardo Music Journal 4: 31. https://doi.org/10.2307/1513178.

⁸

Jones, Kevin. 1989. “Generative Models in Computer-Assisted Musical Composition.” Contemporary Music Review 3 (1): 177–96. https://doi.org/10.1080/07494468900640121.

⁹

DuBois, Roger Luke. 2003. “Applications of Generative String-Substitution Systems in Computer Music.”

¹⁰

Manousakis, Stelios. 2006. “Musical L-Systems.”

¹¹

Miranda, Eduardo Reck. 1995. “An Artificial Intelligence Approach to Sound Design.” Computer Music Journal 19 (2): 59. https://doi.org/10.2307/3680600.

⁵

Diaz-Jerez, Gustavo. 2011. “Composing with Melomics: Delving into the Computational World for Musical Inspiration.” Leonardo Music Journal 21 (December): 13–14. https://doi.org/10.1162/LMJ_a_00053.

Nathan Ho - Steganography

Nathan Ho — Sun, 10 May 2026 00:00:00 +0000

Steganography_ is my second album. Eleven tracks, 30 minutes. Fully synthesized. Distributed by 3OP. Digital download on Bandcamp.

Vertex Figurine
Growing Plums in the Desert
Why Becoming Selfish is the Best Thing I Ever Did
Unmaker Process
Despite Our Best Efforts
Warring Factions
Irreversible Information Acquisition
Things Toxic Friends Say
Systematic Instinct
Stab Variation
Treachery

Mastering by Nil Hartman.

Available as a hand-bound 250-page hardcover book containing the complete SuperCollider source code to the album. A hole cut in it contains a USB stick with 48k/24 audio files. Limited run of five books. I am making them all by hand. (I may make another batch if they sell out, but no promises!)

The source code is not available digitally. These books are the only place to get it.

Lime68k & Nathan Ho - Striations of Grace

Nathan Ho — Tue, 30 Sep 2025 00:00:00 +0000

Striations of Grace_, a collaborative album on by Lime68k and me, has been announced on EVEL Records, releasing October 3rd on digital, CD, and a 7“ vinyl edition. Seven tracks, 53 minutes of cosmic horror synthesis mutations.

Lime68k & Nathan Ho - SOLVE
Nathan Ho - The Kraken
Lime68k - Out to Own Concern
Lime68k & Nathan Ho - We Regret the Error I
Lime68k - Who’s Out Concerns [vinyl release side A]
Nathan Ho - Exterior Algebra and Heroin [vinyl release side B]
Lime68k & Nathan Ho - We Regret the Error II

Lime68k used Max/MSP, I used SuperCollider. One year in the making.

Mastering by Alfonso. Cover art by Aplauso.

DM Synthesis

Nathan Ho — Sun, 24 Nov 2024 00:00:00 +0000

A while back, I tried recreating the sound of a dialup modem from scratch, which involved a dive into multiple PDFs of telecommunication standards. (I am not the first person to do this.) Very helpful was Windy Tan’s beautiful analysis breaking down a spectrogram of a real dialup handshake. A particularly famous bit is the nasal mid-low “whine” at the end of the handshake, and if you’ve ever wondered what that is, you’re hearing line probing signals L1 and L2, which are synthesized using additive synthesis. The exact partial frequency, phases, and amplitudes are specified in section 10.1.2.4 of ITU-T V.34. They’re at 9 seconds in this recording of a modem by thearchiveguy99 on Freesound:

WARNING: Audio files in this post are loud and piercing. Please turn up your speakers to extremely painful levels for optimal telecommunication.

The whine is legendary, but I want to turn your attention to high-frequency gurgles before it, starting at about 4 seconds in, some of them resembling slurping a last bit of soda through a straw in a McDonald’s cup. These tones, specified at a high level in ITU-T V.8bis, are encoded using frequency-shift keying (FSK) as given in the lower-level ITU-T V.21 spec. In its simplest form, FSK uses a single sine wave to encode a binary signal by modulating its frequency. Following V.21’s description of a “channel 1” signal, at regular intervals of 1/300 seconds, a single bit is transmitted. If the bit is 0, the sine wave’s frequency is set to 1180 Hz. If the bit is 1, the frequency is set to 980 Hz.

Yep, that’s a dialup gurgle. The graph below demonstrates this visually (for different parameters — I had to lower the carrier for the 1 bits so that the distinction between the two is obvious). Above we have the audio waveform, below it the data signal.

Designing that dialup imitation made me realize that I really like how FSK sounds purely as a sound design tool. This got me looking into its various siblings, which are known as the “digital modulation” schemes for transmission of digital signals over an analog medium such as radio. In contrast, in “analog modulation” the signal being transmitted is continuous, encompassing familiar techniques like FM, AM, and single-sideband modulation. Today we’re going to honor the long tradition of adapting telecommunications methods for creative use by seeing what kind of sounds we can design by directly listening to signals produced with FSK and related methods. I call this approach Digital Modulation Synthesis, a name I’m not in love with, but I couldn’t think of a better one. It may be abbreviated to the cooler-sounding “DM Synthesis.”

In telecommunications, modulation is only half the story, and you also need a demodulator which takes the transmission signal and gets the data back. Engineering modulators and demodulators in a way that’s robust to noise is an interesting problem subject to nearly a century of study, but here we’re going to be the weirdos who just want to listen to the modulated signal itself. I will spend no time here discussing the engineering tradeoffs of these digital modulation schemes in a telecommunication settings; those problems are very interesting and deep but not relevant to this post.

This post is pretty light on math, and the DSP isn’t fancy here, as our concentration is more on sound design and synthesis. All this should be doable in SuperCollider, Pd, Max, etc. with pre-existing units.

I didn’t research this post as thoroughly as I would have liked, so it’s not impossible that I have misrepresented things from the telecommunications literature. If there are mistakes, sorry about that.

On-off keying (OOK)

To introduce some terminology, our digital modulation schemes will have at least one carrier frequency which is just the frequency of the synthesized oscillator, and a signaling rate which is the number of bits per second. The reciprocal of the signaling rate, and therefore the time interval between bits, is notated (T) and given in seconds. In our above example, (T = 1/300).

Probably the simplest form of digital modulation is on-off keying (OOK), which encodes a binary signal by turning a sine wave on or off. You start by synthesizing a sine wave at the carrier frequency, and every (T) seconds, set the amplitude to either 1 or 0 as determined by the input bit.

Here I’ve set the carrier frequency to 1000 Hz and the signaling rate to 300 Hz. The modulation parameters available to us are the carrier frequency and signaling rate. In particular, the ratio between them is of importance; if the carrier frequency is an integer multiple of the signaling rate, then the sine wave will always resume at the same phase. (This is directly analogous to inharmonic vs. harmonic FM synthesis.) OOK has a pretty harsh sound here due to discontinuities in the final signal; you can also add some smoothing to the on-off transitions if you want to change that.

This is a good time to discuss the input bit stream. This is art, so we have the liberty of making up whatever data we want:

An independent random choice for each bit. The probability could be 50% or controlled as a synthesis parameter.
A periodic bitstream, like 0, 1, 1, 0, 1, 1, 0, 1, 1, … These are audibly different from aperiodic signals.
Some mathematical pattern like the Thue-Morse sequence, Fibonacci word, characteristic functions of any integer sequence, etc.
A random bit stream but the probabilities are non-independent, e.g. Markov chains.
1-bit audio signal from anywhere.
Raw data from somewhere else, such as ASCII text or a random file you have sitting around.
Switching among any subset of the above.

Amplitude-shift keying (ASK)

OOK is a special case of amplitude-shift keying (ASK), where instead of just amplitudes 0 and 1 you can have any fixed number of amplitudes. As such the input signal is no longer a bit stream, but a discrete signal with a fixed number of values.

Here, the data signal takes on eight evenly spaced values from 0 to 7 inclusive, with values streamed with a 300 Hz frequency. The carrier is 1000 Hz again.

Frequency-shift keying (FSK)

I have already introduced FSK, which modulates a bit stream using a single sine wave that jumps between two carrier frequencies depending on the input bit. Let’s see that again. (Again the image and audio do not match here; I spaced the carriers in the graphic farther apart for visual clarity.)

To clarify, the transitions between the frequencies are immediate, but the sine wave’s phase is continuous and the jumps do not actually create a discontinuity in the signal. As such, FSK is essentially FM synthesis where the modulator signal is 1-bit.

FSK vowel synthesis: Since it generates two frequency peaks, FSK can be used as a formant synthesizer. The human ear is incredibly sensitive to audio with speech-like qualities, and in my experience only needs to hear two formants moving in opposite directions in the 500-3000 kHz range to start getting the impression of something vocal. Long ago I learned that you can make extremely bright growls by FMing a sine wave with a square wave and messing with the modulation index so that the sine wave alternates between two frequencies that move in opposite directions. Here’s what that sounds like with the data signal at 100 Hz, the lower frequency sliding from 500 to 800 Hz, and the upper one sliding from 2000 Hz to 1000 Hz:

The FSK generalization of this is that now the alternation between the two carrier frequencies is controlled by an arbitrary bit stream. Here’s the same idea with random bits (I raised the data signal rate to 300 Hz since it just sounded better):

If you need more formants, you can either mix together multiple FSK instances, or use three or more frequencies (see next section).

Multiple frequency-shift keying (MSFK)

FSK usually refers to only two carrier frequencies, but you can use as many as you want to modulate an integer signal with 3 or more discrete values. While straightforward to explain, consider the musical options — you can tune these frequencies to vocal formants, or to a scale or chord, or harmonics of a fundamental, or to peak frequencies analyzed in an input signal. I’ve just used 4 stacked octaves with the bottom one at 250 Hz for this example.

Binary phase-shift keying (PSK)

Phase-shift keying methods use only a single carrier frequency, synthesized with a sine wave, but its phase is modulated in response to the input signal.

There are two ways to think of binary phase-shift keying (BPSK). One way to think of it is as flipping the carrier sine wave’s phase by 180 degrees for 1 bits. That is, the signal is synthesized as (x = \sin(2\pi f_c t + \phi(t))), and every (T)-second interval, if the input bit is 0 then set (\phi(t) = 0); if it is 1 then set (\phi(t) = \pi). Alternatively, BPSK is ASK where the two amplitudes are (\pm 1). That is, the signal is synthesized as (x = A(t) \sin(2\pi f_c t)), and every (T)-second interval, if the input bit is 0 then set (A(t) = 1); if it is 1 then set (A(t) = -1). These are equivalent.

It seems most common for the signal period (T) to be an integer multiple of the carrier’s period (1/f_c), so I have here set (T = 4/f_c).

It is common to smooth out the modulation signal using a lowpass filter prior to modulation.[1] This is “pulse shaping” (although I am oversimplifying since pulse shaping is actually done on an impulsive signal rather than the “latched” one we use here, see[2]). To my understanding, pulse shaping of BPSK is always done in the “ASK domain” by filtering the amplitude signal (A(t)), not the phase signal (\phi(t)).

Quadrature phase-shift keying (QPSK)

In quadrature phase-shift keying (QPSK), the input signal is no longer binary but has four different values. Let’s say the input signal is (b(t)) which takes on values 0, 1, 2, or 3 and is piecewise constant at intervals of (T) seconds. We use four sine waves spaced at 90-degree phase offsets and multiplex them:

[\begin{aligned} \begin{align*} x_0(t) &= A_0(t) \sin(2\pi f_c t + \pi / 4) \ x_1(t) &= A_1(t) \sin(2\pi f_c t + 3\pi / 4) \ x_2(t) &= A_2(t) \sin(2\pi f_c t + 5\pi / 4) \ x_3(t) &= A_3(t) \sin(2\pi f_c t + 7\pi / 4) \ x(t) &= x_0(t) + x_1(t) + x_2(t) + x_3(t) \end{align*} \end{aligned}]

where (A_0(t)) is set to 1 whenever (b(t) = 0) and set to 0 otherwise, and (A_1(t)) is set to 1 whenever (b(t) = 1) and 0 otherwise, and so forth. Effectively this is multiplexing the four sine waves, and only one is on at a time. You can also write all this with phase modulation where the phase offsets may only take on values (\pi/4, 3\pi/4, 5\pi/4, 7\pi/4), but we’ll see why the above is better in a moment.

As with all PSK schemes, sources recommend (T) being a multiple of (1/f_c). In these examples I have set (T = 2/f_c).

Offset quadrature phase-shift keying (OQPSK) is a further variant of QPSK with the constraint that the phase may only jump in 90-degree increments. That is, (b(t)) may not jump directly between 0 and 2 or between 1 and 3. So in reality, each interval is encoding a single bit – as each sample (b(t)) either steps up or down by 1 (modulo 4).

It doesn’t sound or look radically different but I swear it’s a little smoother to the ears. One way to synthesize the phase signal in a modular-like environment is to sum two 1-bit signals, each with a signaling rate of (2T) and one of them delayed in time by (T).

It is also common to pulse-shape QPSK. The way to do so is to filter the four amplitude signals (A_i(t)) with the same lowpass filter. Here, below the waveform and the data signal, I’ve plotted these four signals.

Miscellany

This isn’t the full gamut of digital modulation techniques that have appeared in the literature. I’ve left off Gaussian Frequency-Shift Keying (GFSK) which is FSK with a lowpass filter on the modulator, the various other PSK variants like 8-PSK or (\pi/4)-PSK, and the digital variant of Quadrature Amplitude Modulation (QAM). But it’s enough to get some cool sounds, I figure.

(Minimum-shift keying or MSK is just a special case of FSK, and DPSK appears to be a preprocessing stage, so nothing too interesting for sound design applications.)

Instead of piling on more established methods, I’ll throw in some wilder ones that we have the freedom to do in a synthesizer:

In all of these methods, the sine wave carrier can be swapped out for a non-sine-wave carrier.
In FSK-like methods, the “carrier” can be not an oscillator but the cutoff of a filter.
In methods like BPSK/QPSK/OQPSK, in the absence of pulse shaping, you can think of them as sequencing (say) 4 different sinusoids depending on the input data. Instead of using 4 different sinusoids of different phases, try 4 arbitrary segments of audio sourced from anywhere.

Discussion

I have felt that a lot of writing on this blog has lacked good visual aids to explain DSP algorithms. The main reason is that I’m lazy and it really takes a lot of work to make them. Hopefully the amount of effort here pays off, even for these fairly simple DSP algorithms. I would liked to have added some block diagrams to demonstrate how to implement these in synthesizers, but unfortunately I don’t have the energy for that now.

I mentioned in the intro that there’s a long history of creative music DSP arising from telecommunications. That’s an understatement, because the entire field of signal processing basically originated from people designing communication systems, and most effects used in music production originated in that field. FM synthesis of course developed from FM radio, but we also have frequency shifters using tech from SSB radios. The first vocoder, SIGSALY, was developed for the military. Analog filters first found practical use in telegrams.

I can envision four different broad approaches for employing codecs in musical applications:

Type 1: data → encode as an audio signal.
Type 2a: input audio → encode → decode.
Type 2b: input audio → encode → add noise or other effects → decode.
Type 3: input audio → decode as another audio signal.
Type 4: input audio → decode to some signal → encode as audio signal.

Here I’ve explored type 1. Types 2a and 2b would describe things like bitcrushers (there’s a surprising number of ways to bitcrush audio, possibly the subject of a future post), linear predictive coding, or audio effects that simulate MP3 artifacts – you encode the signal, usually in some lossy way, and then decode it. Types 3 and 4 would involve demodulating audio signals that were not necessarily intended to be demodulated.

Footnotes

Steer, Michael. “Microwave and RF Design I - Radio Systems,” chapter 2.
Gilley, James E. “Digital Phase Modulation: A Review of Basic Concepts.”

System-Building Syndrome

Nathan Ho — Wed, 05 Jun 2024 00:00:00 +0000

Synthesizer forums have a long-running joke about “Gear Acquisition Syndrome” that pictures the stereotypical hobbyist with all their Hainbach-endorsed gear, seemingly oblivious to the fact that they’re intended for making music. Trite as the joke is, the culture that reduces synthesizers to boutique furniture deserves all the mockery it gets. I’m a bit annoyed that I don’t remember who said this: Eurorack is millennials’ equivalent of model trains.

A lot of the audience of this blog is electronic music DIYers and open source fanatics, and many of us have good reasons to sneer at this culture. I myself own very little music hardware and basically never use VSTs, and I admit I feel a little smug when I read about how much money people can blow on synths and even plugins. Then I remember my own history with electronic music.

When I was first getting into SuperCollider — my first ever experience making electronic music — I found my way to DSP conference papers and became totally obsessed. I read every DAFx paper that seemed remotely interesting to me; I learned C++ to implement these in my own UGens. I was pretty certain that with enough effort I could replace any VST plugin I read about.

I thought I was being clever circumventing GAS by building everything myself, but what really happened is that I had fallen into my own version of GAS: I had System-Building Syndrome. System-building can be personally enriching, and no doubt more cost-effective than buying roomfuls of synths, but if your mission is to produce and release music then getting into engineer mode will actively hinder your efforts to do so. When I sat down to make music with all the nice fancy UGens I made, my music sucked. It discouraged me from making more music, so I’d spend more time building UGens since clearly I didn’t have enough of them.

Ultimately, it’s a success story, since that obsession led me to a career in signal processing and an audience for my writing. You learn a lot in the process of building systems, and it’s hard to see how building cool stuff can be a problem. But I’m one of many people who feel spiritually compelled to make music; if I don’t make it I will likely go insane, and as much as I enjoy programming and DSP they are not wired into me on that level. In my mid-20’s I slowly realized that the only way to make music I liked was to stop trying to engineer software, and simply write the most basic SuperCollider code that gets me sounds. At a high level, this is how you make music:

Make music.
Listen to and evaluate music.
Improve music.
Repeat.

Your music improves if and only if you’re spending time in the Core Loop. This is obvious enough, surely tautological for those who have learned instruments, but System-Building Syndrome and its many comorbid afflictions make it easy to lose sight of this. Here are some activities that have nothing to do with the Core Loop:

Buying synths or plugins.
Watching synth reviews.
Watching documentaries on electronic music.
Watching interviews with your favorite musicians.
Promoting your work on social media.
Reading music theory textbooks.
Reading artist manifestos.
Reading books or papers on the philosophy of music composition.
Reading or writing about tuning systems.
Reading DSP papers.
Implementing DSP algorithms out of papers.
Building live performance systems.
Building algorithmic composition systems.
Reading ModWiggler, Gearspace, CDM, Lines, or nathan.ho.name.

Most these activities are benign, some personally enriching or professionally valuable. If you’re an academic then I’m sure you have specific research topics in mind; if you’re a hobbyist and just want to putter around with tech, then obviously you can do whatever you want. But if you’re a serious musician and you’re committed to making the best music you can, then you have to internalize that the above pursuits are at best side projects or supplements to musical growth, and at worst tactics of procrastination. Unchecked, they can become ways to avoid confronting how your own music sounds.

People who joke about GAS are certainly self-aware about its distant relationship to music, but the DIY projects — audio effects, live performance systems — have a sneaky way of giving you the impression that they’re part of the Core Loop when they’re not. When I was busily coding up virtual analog filters in SC, I really did believe they would make my music more cool and unique than if I was using the vanilla SuperCollider installation. These kinds of projects are especially tempting to musicians stuck in a creative rut or feeling dissatisfied with their own work. We have to understand that these activities will not dig us out of such situations. There is no “one weird tip” to music. You get into the Core Loop, that’s the only way to do it.

I want to elaborate particularly on algorithmic/procedural/generative composition, since it’s a useful case study and one that I have considerable experience with.[1] The biggest misconception about algorithmic composition is that it saves time compared to composing manually. I’ll just build this system to write melodies for me, perhaps even write entire tracks autonomously, and I won’t have to put in the effort for the little details ever again.

Of course, the reality is that after building what you envisioned, ultimately you’re left with the process of actually setting the parameters of your algorithm, auditioning various points in the parameter space by hand to generate the most compelling music. We’ve arrived back at the Core Loop. You can’t avoid it. Regardless of the level of abstraction or autonomy your system has, you have to set aside ample time to experiment with the system that you’ve already built. William Fields’ system allows him to randomly generate patches in a variety of musical styles, but he also spends a lot of time curating and tweaking its output; it’s nothing like the dream of “set it and forget it” that many people have when they start with algorithmic composition.

FieldsOS is a successful system by any measure, but here’s a scenario that I suspect is more common. Someone — perhaps a young SuperCollider user with a lot more confidence than production ability — starts building an elaborate algorithmic live performance system, finishes most of the features he wants two weeks before his undergrad final show, tries to make music with it, and belatedly realizes “well, damn, I still actually have to sit down and learn how to use this thing.” Insufficient time has been left for making and evaluating music, and he’s too exhausted by all the software engineering work to actually sit down and focus on the art, and the results are audibly half-assed. The cliche of art being 1% inspiration, 99% perspiration rings true; if the inspiration eats the perspiration, then the art usually sucks.

Letting algorithmic composition predominate actual composition is a trap that often catches musicians with a software background. I understand why. If I have dozens of files that I want to rename in a systematic manner, I’ll cook up a shell script to get it over with. It seems logical enough that if I want to make dozens of tunes, I just need to write the correct automated tool to take care of that. But this reasoning falls apart — creating music is nothing like renaming files. Art as a whole isn’t a computational process, even when it employs systems at the surface level. To think that I can somehow be more artistically prolific by putting in less effort is an oxymoron and a programmer-brain delusion; in fact, algorithmic composition often seems to require more effort than manual composition of comparable quality. I’m down with algorithmic composition as much as anyone, I use algorithmic techniques all the time in my work, but over time I’ve learned to keep realistic expectations: algorithms cannot reduce the amount of time you need to spend in the Core Loop, it doesn’t make the Core Loop more efficient. More often than not, it delays starting the Core Loop in the first place.

I focus on algorithmic composition because it’s a clear example, but so many hobbyhorses typical of “wonk” musicians are similar. You can obsess all you want over nonstandard synthesis or cybernetics or wavelet transforms, but none of them will do your sound design for you, and if you want good sounds you’re still tweaking envelopes and EQ settings for hours. You can build the ultimate live electronics performance system you’ve been dreaming of, but you’ll also have to spend a year learning how to use it. You can edit Xenharmonic Wiki all day, but tunings on paper bear little relation to tunings in tunes. I stress again that these kinds of diversions can be as gratifying as making music — and yes, they can impact your music, sometimes dramatically — but they should never be confused with actual music-making.

If your mission is to perform or release music, know what you’re getting into if you plan on adding side quests. The more ambitious and far-reaching a system-building project is, the more such projects are inherently distant from basic process of iteration on sounds. Overgeneralizing too early is a great way to ensure that this kind of project fizzles out. Successful system-building integrated into a musical practice is usually a highly incremental process, and one backed with experience; always build yourself tiny systems before you start building the big one you envision. (You may be surprised how musically powerful even the most elementary works of software can be.) And never, ever build systems with the expectation that they will develop your musicianship for you.

Some might find my assertions about how music should be made too reductive or binaristic. There are certainly musicians and artists throughout history where their systems seem more of a centerpiece than actual sounds — I’m thinking in particular of serial composers, whose work I enjoy especially when I read about its construction. And maybe you find that you don’t want to make music anymore and you’re perfectly content in music tech; I have known many people who followed that trajectory. It’s your choice in the end, you have agency over how you want to allocate your time for creative and technical work, it’s all cool. I’m just here to give practical advice: try taking the simplest path to making it sound good. Contain the distractions and big visions, keep asking yourself and others how it sounds, and you might get there.

Footnotes

This is directed at people designing their own algorithmic composition systems and tools where the primary audience is themselves. Someone typing prompts into a trendy off-the-shelf service built by some startup is pro [OPINION REMOVED - PERPETRATOR OBLITERATED - BODY INCINERATED TO POWER GPU FARM]

Audio Effects with Wavesets and K-Means Clustering

Nathan Ho — Tue, 09 Apr 2024 00:00:00 +0000

(original)

(processed)

I would like to congratulate wavesets (not wavelets) for entering their 30th year of being largely ignored outside of a very small circle of computer music nerds. Introduced by Trevor Wishart in [Wishart1994] and popularized by Microsound [Roads2002] and the Composers Desktop Project, a waveset is defined as a segment of an audio signal between two consecutive upward zero crossings. For simple oscillators like sine and saw waves, wavesets divide the signal into pitch periods, but for general periodic signals there may be any number of wavesets per period. For signals containing noise or multiple pitches at once, waveset segmentation is completely unpredictable.

Many simple audio effects fall out of this idea. You can reverse individual wavesets, omit every other waveset, repeat each waveset, sort them, whatever.

I like waveset-based effects best on input signals that are monophonic (having only one pitch) and low in noise. Synthetic signals can make for particularly interesting results. Much as the phase vocoder tends to sound blurry and phasey, waveset transformations also have their own “house style” in the form of highly digital glitches and crackles. These glitches are particularly pronounced when a waveset-based algorithm is fed non-monophonic signals or signals containing strong high-frequency noise. Wavesets are extremely sensitive to any kind of prefiltering applied to the input signal; it’s a good idea to highpass filter the signal to block dc, and it’s fun to add pre-filters as a musical parameter.

Today, we’re putting a possibly new spin on wavesets by combining them with basic statistical learning. The idea is to perform k-means clustering on waveset features. The steps of the algorithm are as follows:

Segmentation: Divide a single-channel audio signal into (N) wavesets.
Analysis: Compute the feature vector (\mathbf{x}_i) for the (i)-th waveset. I use just two features: length (\ell_i), or number of samples between the zero crossings, and RMS (r_i) of the waveset’s samples. All the lengths are compiled into a single size-(N) vector (\mathbf{\ell}) and the RMSs into (\mathbf{r}).
Normalization: Scale (\mathbf{\ell}) and (\mathbf{r}) so that they each have variance 1.
Weighting: Scale (\mathbf{\ell}) by a weighting parameter (w), which controls how much the clustering stage emphasizes differences in length vs. differences in amplitude. We’ll talk more about this later, but (w = 5) seems to work as a start.
Clustering: Run k-means clustering on the feature vectors (\mathbf{x}), producing (k) clusters.
For each cluster, pick one representative waveset, the one closest to the centroid of the cluster.
Quantization: In the original audio signal, replace each waveset with the representative waveset from its cluster.

Implementation is very lightweight, clocking in at about 30 lines of Python with scikit-learn. There are only two parameters here other than the input audio: (w) and (k).[1] The length of the input audio signal is important too, so for musical reasons let’s not think in terms of (k) but rather “clusters per second” (c), which is (k) divided by the signal length in seconds. As we will see, with only two-dimensional control we can produce a tremendous variety of sounds.

Exploration

I started with an a cappella cover of “Stay With Me” by a singer named Aiva, which I use often for testing monophonic algorithms. This segment is 17 seconds long, 44.1 kHz — as I mentioned, the length of the audio file is important.

This file splits into about 26,000 wavesets. Running the algorithm with (w = 5.0) and (c = 4.0):

The pitch content is retained, but the voice is now speckled with all kinds of glitchy mutations. (As an aside, sklearn.cluster.KMeans crunches it in a fraction of a second, and I have no concerns with efficiency.)

It is always instructive to look at plots if we can, and in this case it is quite easy given that we have only two-dimensional data to plot. I’ve placed waveset duration on the X-axis and waveset RMS on the Y-axis, and assigned each cluster to a random color. (Sorry, some different clusters might have similar-looking colors.)

(c = 10.0):

(c = 5.0):

(c = 2.0):

(c = 1.0):

(c = 0.5):

We also have (w) to tune, which affects the “warbliness” of pitch. Higher (w) values result in clusters forming more vertically, while lower values result in more horizontal stripes. For the sake of space, I will not demonstrate this tuning here; I don’t find it a particularly musical parameter to tune and prefer to leave it at 5.

Trying on different audio files

This algorithm is highly signal-dependent, and worth trying on a wide variety of audio files. Here are just a few. The morphology of the 2D plot is particularly interesting.

Speech

Singing tends to hover around quantized pitches, but speech does not, and the clustering algorithm is aware of this. The effect on pitch is particularly interesting here, and very squelchy. (This is perhaps not a fair comparison because this file is considerably shorter than the sung example.)

(original)

(processed)

Percussion

Manhattan Transfer’s “Soul Food to Go” sounds like a Clipping beat.

(original)

(processed)

Full mix

Scott Krippayne’s “I’m Not Cool” turns into digital soup.

(original)

(processed)

Discussion

No question this could be done in real time, by performing k-means on a sliding window of the last N seconds of wavesets. There is considerable literature on “streaming k-means” algorithms [Ailon2009], but for our application the accuracy of clustering is not too important and simply re-running an iteration step of the standard k-means algorithm might work fine. Now imagine filtering it with EQ to screw up the zero crossings, and routing it back into itself with a delay.

I messed a bit with expanding the contextual information of the feature vectors, adding the features of nearby wavesets (with reduced weights so they have less impact on the clustering). However, I was pleased enough with the context-free wavesets that I decided not to push this. I also feel there are other approaches to context that are perhaps more immediately musical. You could for example train a Markov chain on the sequenced wavesets.

This use of clustering is known as vector quantization (VQ), where a vector is replaced with the closest one in a “codebook” of vectors. A very important application of VQ is compression, so it appears commonly in lossy codecs, including audio codecs where it is often used to quantize speech parameters (see Code-Excited Linear Prediction).

Another precedent for what we’re doing here is concatenative synthesis[2], whose standard pipeline involves segmenting audio, analyzing the segments to produce feature vectors, and performing any manner of statistical operations on said feature vectors. Maybe I didn’t look hard enough, but I was unable to find anyone combining wavesets with concatenative synthesis. I did find that a lot of concatenative synthesizers pile on all sorts of fancy machine listening features. There’s nothing wrong with doing that, but the success of the method presented here demonstrates that a “dumb” algorithm — in this case an elementary segmentation method and just two features — is fully capable of sounding distinctive and musical.

Footnotes

References

Ailon2009
Ailon, Nir et al. 2009. “Streaming k-means approximation.” https://proceedings.neurips.cc/paper/2009/hash/4f16c818875d9fcb6867c7bdc89be7eb-Abstract.html
Roads2002
Roads, Curtis. 2002. Microsound.
Wishart1994
Wishart, Trevor. 1994. Audible Design.

A different clustering algorithm, affinity propagation, can automatically determine the number of clusters. I tried it but it was slow enough to annoy me despite running only 25k 2D veature vectors.
Concatenative synthesis has a different but closely related meaning in speech synthesis, where it simply means approaches based on splicing together transformed fragments of recorded speech.

Cybernetic Synthesis and Roland Kayn

Nathan Ho — Tue, 19 Dec 2023 00:00:00 +0000

Let’s open this post with music. If you aren’t familiar with the late composer Roland Kayn’s work, listen to one of his Electronic Symphonies, or at least sample a few minutes. The album cover links to his official Bandcamp.

I found out about Kayn’s music through a thread on Lines, and it gets only more interesting as I looked into his story. Kayn was an exponent of combining music with cybernetics, a field I gloss as “the study of feedback systems.” At its heart, it’s a mathematical study; Norbert Wiener’s original text Cybernetics is chock full of integrals and differential equations. As the field developed, it overlapped significantly with medicine, social sciences, humanities, and especially critical theory. I do hope to read Wiener’s Cybernetics and The Human Use of Human Beings, but haven’t gotten around to them, so consider my understanding of the field surface-level — I’m mainly concerned with cybernetics as it relates to Kayn’s music.

Kayn began making cybernetics-influenced electronic works in university electronic music studios in the 60’s. We don’t know what exact tools, but Pickles’ 2016 dissertation Cybernetics in music surmises that the equipment back then was something along the lines of tape machines and basic analog processors and generators: oscillators, noise, filters, reverbs, envelope generators, ring modulators. More intriguingly, recordings of his works were entirely real time — a stark contrast to the offline tape-based processes of the contemporary musique concrète movement. Critic Frans van Rossum wrote in 2011 that Kayn worked with a “network of electronic equipment” and a “a system of signals or commands that it can obey and execute” (quote also via Pickles).

EDIT 2023-12-20: An earlier version of this post credited the technology to Kayn alone, but I have since been properly schooled: it was Jaap Vink who actually built the system, as Sascha Frere-Jones wrote in 4Columns. Given the intimacy that cybernetic musicians necessarily have with their tools, his fingerprints are deeply embedded in Kayn’s electronic works. The sources I looked at didn’t make this clear to me, and most don’t even mention Vink by name. Vink referred to himself as a technician, not a composer, and it’s possible that the concealment of his status was his preference.

That these works were likely made on 60’s equipment, were recorded in real time, and sound varied and dynamic over long stretches of time should impress anyone familiar with modular synths. The implications that his musical systems had some kinds of self-generating and autonomous properties makes this even more remarkable. Now Kayn’s output is more than just great music, it’s now a puzzle. How did he make this music? Can we recreate part of his process?

I don’t think we’ll ever get definitive answers. Kayn died in 2011, Vink in January 2023. While there’s a small body of writing about them, little of it is technical. We can only guess, but we have some resources that helps us with the guessing process — the Lines thread I mentioned, and also La Synthèse Humaine, a YouTube channel run by musician Émile Zener a.k.a. Gunnar Haslam. He takes direct inspiration from Kayn, and unlike me has extensively studied the cybernetics literature. I won’t take the time to resummarize the Lines thread or Zener’s work — just go check them out for yourself — instead I am interested in expanding on them.

Making a cybernetic LFO

It’s perhaps easiest if I first walk you through a concrete example of building a cybernetic patch. We begin with the simple harmonic oscillator, which can be implemented with the system of ordinary differential equations

[\begin{aligned} \begin{align*} \dot{a} &= -kb \ \dot{b} &= ka \ \end{align*} \end{aligned}]

and the following block diagram:

This comprises two integrator stages in a feedback loop. An integrator is a unit that continuously accumulates the input signal (you can get something like a leaky integrator with a one-pole lowpass filter with a very low cutoff). The triangles are simple amplifiers multiplying by constants; one must be negative and the other positive. There are two outputs, A and B, which produce a sine and cosine wave respectively. The system does need an initial “kick” to generate sine waves, by e.g. initially setting (a(0) = 1) and (b(0) = 0).

Assume that the oscillator is tuned to oscillate at a slow 1 Hz. It is an “LFO” in the literal sense that it oscillates at a low frequency, but critically we aren’t generating it with an actual oscillator module, and the oscillation is an emergent property of the system that we built. We can turn this into an audible synthesis patch by using the two outputs, A and B, as amplitudes, multiplying them with VCAs by two saw oscillators with static frequencies. Mixing the oscillators together, we have a simple looping alternation between two “notes” even though we have no actual sequencer here.

However, the oscillators and VCA’s aren’t part of a loop, and their output does not influence the rest of the system. For reasons that I’ll justify later, we loop in their VCAs by running an envelope follower on the output of the VCAs, and using it to modulate the coefficients of other VCAs that are in the loop.

(I’m misusing notation here a bit; the arrows into the loop VCAs are not actually setting the gain, but wiggling around the gain it had in the previous diagram. The intentions of this diagram are more romantic than schematic — the point is to illustrate the closing of loops, not to specify an exact algorithm.)

This isn’t an actual simple harmonic oscillator anymore, but it mimics the core behavior of one. However, if we start changing any parameter, the core behavior of the “LFO” starts changing too. Everything interacts with everything, and nothing is isolated or purely upstream or purely downstream.

We’ll return to this patch and expand it into something more compelling in sound design terms. Let’s go theoretical for a bit and explain what we’re actually trying to do here.

Rules for Cybernetic Synthesis

Building off La Synthèse Humaine, Lines, and my own experiments, I suggest the following rules as my own interpretation of cybernetic synthesis. I’m not proffering a serious hypothesis on how Kayn and Vink worked — that’s more fun as secrets — instead, I’m trying to impose cybernetic-inspired constraints on myself to see if any interesting musical ideas emerge, especially ones that resemble their music. In this vein, I offer the following rules.

Rules for Cybernetic Synthesis

Avoid open loops. Every part of the signal chain should be in some kind of loop, in a way that significantly impacts the whole system’s behavior.
Avoid explicit LFOs, envelope generators, randomization, and explicit sequencing (piano rolls, MIDI-like sequencers, etc.). Oscillatory behavior and other gradual modulations should be emerging byproducts of the system feeding back into itself. Audio-rate oscillators are permitted, as it’s too impractical to go without them.
The system should be able to sustain itself and produce material without needing constant intervention from a human player. Human interaction is permitted, but the dynamic should be an artist conversing with a system, not a performer playing an instrument.

These don’t have precise mathematical meanings, and they’re about attitude and philosophy over signal processing. Even then, they’re restrictive: the vast majority of electronic music production workflows, including my typical process, are clearly not cybernetic. Top-down sequencing of musical events, whether manually entered or generated by algorithms, is entirely disallowed in this paradigm. Most of us employ a separation of concerns between sequencing and synthesizer; in cybernetic synthesis, my experience is that these systems necessarily have to be blurred, with the synthesis and sequencing “melting” into each other. We saw in the above example that we didn’t make explicit note events to speak of, but rather they emerged from the system.

It’s important to note here that the mere presence of feedback isn’t sufficient for a system to be cybernetic. Cybernetic synthesis is more radical than that: it demands that everything ultimately affects everything else in feedback. It could be one big loop, it could be multiple interacting loops, but there cannot be any signal flow that’s completely unidirectional. In graph theory terms, we could say that the directed graph of signal flow should be strongly connected, meaning there is a directed path from every node to every other node.

This feedback has to actually matter as well — you can multiply an otherwise open-loop signal by -100 dB and feed it back into the system, and you could technically lawyer your way into saying it’s closed-loop, but it won’t change much and therefore doesn’t really qualify as a cycle. (Switching, amplifying, or multiplexing can disable a loop temporarily, I consider this okay as long as it gets enabled and impacts everything else at some point.) Turning almost any knob should have a pretty major impact on the whole patch, an impact that is hopefully unpredictable.

Extending the cybernetic oscillator

That’s all nice, but I’m itching to get some legitimate sounds here. Let’s start by extend the signal path. Kayn uses a lot of reverb, so an obvious choice would be to throw some ’verb on those oscillators, and reduce the highs. I’ve also changed the oscillator frequencies. Both the reverbs and the filters are in the feedback loop, so they also impact the operation of the core simple harmonic oscillator.

I would like some contrast with static and dynamic pitches for the oscillators. So let’s conditionally shut off the frequency modulation. A cybernetic way to do this would be to take the integral of some envelope follower, and switch off the modulation if the integral exceeds a threshold. To turn it back on again, the integral can wrap around back to 0 if it exceeds another threshold.

I also changed a bunch of other parameters here, but the little “whines” that start happening later were not something I anticipated.

The undulation that the core SHO created is very hard to make out now. That’s totally fine — cybernetic patches can go in completely different directions from what you intended. It calls into question whether it was worth starting with an SHO in the first place, but hindsight 20/20.

After an additional hour or two of throwing things at the wall, adding various areas of feedback and EQ and filtering and switches controlled by integrated envelope followers, and I got this:

This is completely autonomous, there is no live interaction in this recording. (It is also deterministic, there are no noise sources.) Don’t know about you, but I think it sounds reasonably Kaynian. True Kayn fans are free to roast me on the inaccuracy, but more importantly it’s a sound I like and would not have gotten otherwise, which is all I need for a successful experiment.

Part of the whole cybernetics deal, at least as I understand it, is that the human musician is also an integral part of the system — involved in its construction and usage in a bidirectional feedback loop. It’s for this reason that I don’t plan on sharing the SuperCollider source code here. To really learn how this process works, you have to just dig in and build your own patch from scratch. Having now spent a small amount of time on the builder’s side of things, the relative mystery around Kayn and Vink’s technical methods starts to make more sense to me. It’s hard to explain a system that grows organically and that I don’t fully understand myself!

Discussion

Previous work of mine on intentionally unpredictable audio systems include Feedback Integrator Networks and Matrix Modular Synthesis.

My typical process as a sound designer is as “top down” as it can be. I conceptualize a sound (or listen to a reference) and realize it as a patch, and while spontaneous and exploratory moments happen when working with synthesizers, full-on surprises have happened less and less often as I develop specific habits and go-to sound design patches. The “top down” nature goes out the window in cybernetic synthesis — with these rules, I’ve specifically denied myself the ability to know exactly what’s going to emerge. For a cybernetic synthesizer patch to do anything interesting, it has to be unpredictable at a fundamental level. It’s about receding control to emergent behavior, and this is what makes cybernetic sound design both rewarding and frustrating.

The very mention of chaos theory will certainly make a subset of my reader base perk up. I didn’t touch on chaos here because I didn’t want to imply that cybernetic systems have to exhibit chaos, but it’s certainly true that chaos is cybernetics’ rockstar nephew, and it’s perfectly applicable here. (The system that I built for this might already be chaotic, but I’m reluctant to use the term without checking things like the Lyapunov exponent.) If you’re really set on chaos, the core network of the simple harmonic oscillator can be replaced with a Duffing oscillator or a Lorenz attractor, but remember that these systems are only chaotic with certain parameter settings and some can be easily knocked into periodic or static behavior. It’s chaos, the entire point is that it’s difficult to predict.

State of the Blog, 2023

Nathan Ho — Sat, 02 Dec 2023 00:00:00 +0000

As 2023 ends, it seems like a good time to reflect and allow myself some vanity. It’s also a few days shy of the 15th anniversary of the first thing I created on the Internet (a certain wiki I have mentioned before). Amusingly, I’m really doing the same thing as I was back then: publicly writing about niche technical subjects. Back then it was mathematics, now music technology.

I set a goal for myself to release at least one substantial technical blog post every month this year, and succeeded:

Outside of writing, I also released a full-length album, began performing live, and gave some talks and guest lectures.

Meeting the monthly deadline required pushing myself a little, but I’m glad I set this schedule, because the self-imposed time constraints forced me to scope projects realistically and suppress perfectionism. Often all I needed to do for a blog post was to find an exciting topic and demonstrate its value with some nice sounds; there’s a time and place for thorough research and literature review, but I’m okay without it for most projects here. I also felt the variety is beneficial, since it helped me gauge my level of interest in various music tech topics and, more importantly, my audience’s. I don’t have analytics or a comment section, but I do hear from readers by email occasionally (thank you!), and this helps me understand what kind of writing is most wanted in the music tech community.

Speaking of engagement, now’s a good time as any to announce that I will be pivoting to topics suggested to me by this free online Blog Idea Generator:

While I expect the general topic to continue, I won’t try to hold myself to this schedule next year. This is so that I can shift towards a smaller number of projects that are more ambitious in scope. The goal I’m most jazzed about for 2024 is submitting an academic paper to a peer-reviewed conference as an independent researcher. I have a specific topic and a specific conference in mind, but I won’t discuss it until it’s in the reviewers’ mailbox.

Another big project I’m hoping to release is the Rust-based SuperCollider successor I’ve been building. While significant, it’s not as crazy as it sounds — my system is quite compact compared to SC, as it only replaces the small sliver of the platform that I personally use. A portion is already built and I’ve been using it in live performance, but it needs considerable refinement before others can use it.

I have some other projects lined up too. I have new music in the oven as usual, and I’ve been working sporadically on a mathematical visualization project.

Thoughts on creativity

I have been described as “prolific.” The main factors here are:

I have a naturally hyperactive personality and an overwhelming urge to make things.
I enjoy the privilege of supporting myself through freelancing, giving me a relaxed work schedule.
I slashed Internet surfing and phone use a few years ago, which helped my natural state shift over from consumption over to creativity.
Most of my social circle is creative types, and the element of social contagion is motivating, especially since I started connecting in person with the local art community.

I’m not a workaholic in any sense. My days are pretty relaxed and I go with the natural ebb and flow of my creative motivation. Between everything, I nap and watch YouTube and chat with friends and walk around outside (not at the same time).

I usually have one primary project going, one that I try to progress on regularly even when I don’t particularly feel like it. If I really run out of steam on the main project, I have a rotation of a few other secondary projects so I can stay stimulated. Some of these projects require intense focus, others are more low-key and leisurely, and it depends on how I’m feeling. I’m increasingly trying to incorporate computer-free projects into that rotation to reduce screen time while still fulfilling my creative needs. (I got into drawing and bookbinding recently.)

To avoid wandering between numerous unfinished projects and never completing any, I find it’s critical to give every project a definition of done. For software and informal scientific research, “completion” is a slippery concept, so I find that blog posts have been a consistently good way to give something a stamp of finality. I document my work, put it on the blog, send a celebratory email to the newsletter, and I can move on.

For music and other art forms, the definition of done is usually more straightforward, such as a release or a premiere. That doesn’t make the actual process of completing musical projects easy though.

Releasing an album

Multiple friends asked me about what the process is like for releasing an album. I feel I’m qualified to give an answer now that, well, I have an album out. I’ve discussed some aspects of the album musically, and I might do so more, but here I’m purely discussing logistics.

Haywire Frontier took about six months. I committed to making an album in February 2023, and I shipped off the premasters in August. Almost all tracks were started from scratch in that span, although “Acetone Splash” and “Immodality” started as earlier sketches. The work schedule was somewhat irregular. I’d have a week where I made a lot of progress, and a week where I didn’t touch the album at all. However, I tried not to initiate any other really big projects while the album was in progress.

Each of the tracks averaged about 3–6 hours of work total, usually split into one or two sessions where I composed and designed sounds, and one session dedicated to mixing and loudness. For me, it’s crucial to work as rapidly as possible, especially during the very first stages of a piece, because I get fatigued and frustrated if I chew on a single track for too long. (To get an idea of the pace that I work at, I’ve documented my compositional process extensively with session videos.) If I find myself spending a lot of time trying to “fix” a patch that doesn’t work, I let it go and try something else. I had a few such dead ends. The outro to “The Calico Trials” was originally much softer and prettier but didn’t hit the spot the way I wanted, and “The Good in You” was 90% rewritten, with only the title and the bookends retained.

A tool that proved invaluable for scaling the day-to-day production process up to a 40-minute musical project was this spreadsheet:

I just chipped away at this, checkmark by checkmark, until the album was done. The “Form” and “Patches” stages are a little amorphous and not necessarily in that order, but otherwise it’s left to right. A formula tracked the completion of the entire album, which was a nice little motivator. I bookmarked this Google Sheet in my browser toolbar so I couldn’t avoid the reminder that I had an album in the works.

Overall, my advice for tackling ambitious musical projects is to think like an accountant or project manager. If you’re down with agile or agile-like methods, use them. Really, go with any strategy that helps you take the musical process from liquid to solid, and break it into actionable chunks. It isn’t possible to decompose the entire creative process into an assembly line — we aren’t computers, after all — but these kinds of tools give us the peace of mind and help us focus on the actual artistry.

Finally, I’ve been asked about what it’s like to find a label. I’m not sure I can give useful advice here, since it’s a fickle process, and for better or worse mostly one of cronyism. I reached out and groveled to people who knew who I was (if you’re one of those people, I am very sorry, and thank you); many couldn’t help me but a few were kind enough to pass on demos, totaling five labels. I had this network in the first place not so much because of my music, but because of the audience for this blog and my contributions to SuperCollider.

It’s a common thread in the online experimental music community: quality sounds are still a requirement to get heard at all, but you really draw people in when you pull back the curtain and talk about processes and tools in a detailed manner. This wasn’t a strategic mastermind decision on my part, rather, I have always obsessively documented the technical side of my music, and it turns out people enjoy that. Process-sharing can also be taken to a cynical end, such as in the Synthfluencer–Industrial Complex, but that only makes it more important to try to build resources for electronic musicians that are genuine, thoughtful, and informative. If you make those kinds of resources, people will take notice and they’ll check out your music, too.

Writing, note-taking, organizational tools

I’m not quite sure how it happened, but at some point in the past few years I became an absolutely compulsive writer. I regularly open text files just to type and type and type. I just use Markdown files in Dropbox, with Obsidian as the editor. I have found no use for its linking or tagging features, I just prefer it to VSCode for editing text. Central to this repository is one long-running file, ideas.md, where I jot down anything that could develop into an interesting project. I go over it every now and then, reordering it to put the most tempting ideas near the top, pruning out anything that I’ve lost interest in.

I also carry a pen-and-paper journal around, and I write in it daily. Handwriting is inefficient for writing and editing text, so it’s intended more for personal reflection where efficiency doesn’t matter.

For reference management, I used a disorganized folder of PDFs for a long time, and I’m mad at myself for not picking up Zotero earlier. I definitely would have learned this had I gone to grad school, but alas I had to learn this the hard way.

For to-do lists, I use Trello, tracking all personal tasks, creative projects, and (non-NDA) freelance work on one huge board. If I divide my attention between multiple boards I will forget to look at one. My philosophy towards organizational tools is that they should be as simple and few in number as possible, and are only there to support the work.

I have little to say on the writing process itself, because it’s an atomic, indecomposable task. I just write, I’m not sure if I can explain it.

Conclusion

Maybe this is an interesting look behind the curtain, maybe this comes off as pompous self-indulgent rambling, I don’t know. In any case, I appreciate all who pay any attention to what I’m doing.

Audio Effects with Cepstral Processing

Nathan Ho — Thu, 23 Nov 2023 00:00:00 +0000

Much like the previously discussed wavelet transforms, the cepstrum is a frequency-domain method that I see talked about a lot in the scientific research literature, but only occasionally applied to the creative arts. The cepstrum is sometimes described as “the FFT of the FFT” (although this is an oversimplification since there are nonlinear operations sandwiched in between those two transforms, and the second is really the Discrete Cosine Transform). In contrast to wavelets, the cepstrum is very popular in audio processing, most notably in the ubiquitous mel-frequency cepstral coefficients (MFCCs). Some would not consider the MFCCs a true “cepstrum,” others would say the term “cepstrum” is broad enough to encompass them. I have no strong opinion.

In almost all applications of the cepstrum, it is used solely for analysis and generally isn’t invertible. This is the case for MFCCs, where the magnitude spectrum is downsampled in the conversion to the mel scale, resulting in a loss of information. Resynthesizing audio from the cepstral descriptors commonly used in the literature is an underdetermined problem, usually tackled with machine learning or other complex optimization methods.

However, it is actually possible to implement audio effects in the MFCC domain with perfect reconstruction. You just have to keep around all the information that gets discarded, resulting in this signal chain:

Take the STFT. The following steps apply for each frame.
Compute the power spectrum (square of magnitude spectrum) and the phases.
Compute a bank of bandpass filters on the power spectrum, equally spaced on the mel-frequency scale. This is the mel spectrum, and it downsamples the magnitude spectrum, losing information.
Upsample the mel spectrum back up to full spectral envelope. Divide the magnitude spectrum by the envelope to produce the residual spectrum. (You have to add a little epsilon to the envelope to prevent zero division.)
Compute the logarithm and then the Discrete Cosine Transform of the mel spectrum to produce the MFCCs.
Perform any processing desired.
Invert step 5: take the inverse DCT and then the exponent to produce the mel spectrum.
Invert step 4: upsample the mel spectrum to the spectral envelope, and multiply it by the residual spectrum to produce the power spectrum.
Recombine the power spectrum with the phases to produce the complex spectrum.
Inverse FFT, then overlap-add to resynthesize the signal.

It’s a lot of steps, but as an extension of the basic MFCC algorithm, it’s not that much of a leap. I would not be surprised if someone has done this before, storing all residuals when computing the MFCCs so the process can be inverted, but I had difficulty finding prior work on this for the particular application of musical effects. Something similar is done in MFCC-based vocoders, where the “residual spectrum” instead replaced with speech parameters such as pitch, but I haven’t seen this done on general, non-speech signals.

I will be testing on the following mono snippet of Ed Sheeran’s “Perfect.” (If you plan on doing many listening tests on a musical signal, never use a sample of music you enjoy.)

As for the parameters: mono, 48 kHz sample rate, 2048-sample FFT buffer with Hann window and 50% overlap, 30-band mel spectrum from 20 Hz to 20 kHz.

Cepstral EQ

Because of the nonlinearities involved in the signal chain, merely multiplying the MFCCs by a constant can do some pretty strange things. Zeroing out all MFCCs has the effect of removing the spectral envelope and whitening the signal. The effect on vocal signals is pronounced, turning Ed into a bumblebee.

Multiplying all MFCCs by 2 has a subtle, hollower quality, acting as an expander for the spectral envelope.

MFCCs are signed and can also be multiplied by negative values, which inverts the phase of a cosine wave component. The effect on the signal is hard to describe:

We can apply any MFCC envelope desired. Here’s a sine wave:

Cepstral frequency shifting

Technically this would be “quefrency shifting.” This cyclically rotates the MFCCs to brighten the signal:

And here’s the downward equivalent:

Cepstral frequency scaling

Resampling the MFCCs sounds reminiscent of formant shifting. This is related to the time-scaling property of the Fourier transform: if you resample the spectrum, you’re also resampling the signal. Here’s upward scaling:

Here’s downward scaling:

Cepstral time-based effects

Here’s what happens when we freeze the MFCCs every few frames:

Lowpass filtering the MFCCs over time tends to slur speech:

Stray thoughts

I have barely scratched the surface of cepstral effects here, opting only to explore the most mathematically straightforward operations. That the MFCCs produce some very weird and very musical effects, even with such simple transformations, is encouraging.

In addition to playing with additional types of effects, it is also worthwhile to adjust the trasforms being used. The DCT as the space for the spectral envelope could be improved on. One (strange) possibility that came to mind is messing with the Multiresolution Analysis of the mel spectrum; I have no idea if that would sound interesting or not, but it’s worth a shot.

It’s possible to bypass the MFCCs and just do the DCT of the log-spectrogram. I experimented with this and found that I couldn’t get it to sound as musical as the mel-based equivalent. I believe this is because the resolution of the FFT isn’t very perceptually salient. The mel scale is in fact doing a lot of heavy lifting here.

Making Synthesized Sounds More Acoustic

Nathan Ho — Mon, 16 Oct 2023 00:00:00 +0000

I have been experimenting a lot with finding ways to get more acoustic sounds out of synthesizers. These sounds don’t need to be perfect recreations of any particular real instrument, but I want a piece of the complexity and depth that those have, and also to investigate “Pinocchio” synth patches that fall short of becoming a real boy in hopefully interesting ways.

Computer music types often jump to physical modeling, a field that I adore and have casually researched. But with the exception of modal synthesis, most physical modeling paradigms take considerable software engineering tenacity to get good results — especially the finite difference models, but waveguides too. I do intend to explore them further, but also I believe that some cool sounds can come out of chains of completely ordinary oscillators and effects. In my experiments, I’ve come across a bunch of little tricks that can help lend some more realism to those kinds of synth patches. Many of these also apply to sophisticated physical models too (after all, physical models can’t deliver you from having to do sound design).

In general, randomize everything a little bit and modulate everything with a slow, smooth random LFOs.
Real percussive envelopes have a very tall initial spike. Inspecting the waveform of an unprocessed xylophone hit, I was surprised by how loud the transient of a typical percussive instrument is compared to its resonating tail.
The high dynamic range can make such sounds tough to bring up in the mix, and can often be addressed by adding clipping or similar distortion that only gets driven during the initial transient. This improves “bite” as well. However, acoustic sounds by nature have higher dynamic range, and clipping and compression can take away from that. Find a balance that works for you.
Key tracking (modifying synth parameters based on register) is essential. No acoustic instrument has the same physics in every register, and some have very limited pitch range in the first place. I usually at least key track amplitude and the cutoff of a lowpass filter. Don’t get discouraged if something sounds good in one octave but bad if you transpose it. You may even need an entirely different patches for different octaves.
In a tonal percussive synth, it’s essential that partials decay at different rates. A rule of thumb for damped physical resonators is that the decay time is roughly proportional to the inverse of the square of frequency. For example, going up an octave will multiply the decay time by about 0.25. This is not only true of partials within a note, but even of different keys of many mallet instruments. (In a piano, different registers have different physical constructions including different numbers of strings per key, which I believe is specifically compensating for this phenomenon.)
Synthesized drums made using oscillators benefit from some subtle stereo detuning.
You can spruce up standard ADSR or percussive envelopes by using multiple little spikes in sequence, a bit like an 808 clap envelope. These spikes can be obvious or subtle.
Add little noise bursts and puffs to every sharp transient. Delay the noise bursts relative to each other a little bit and randomize all properties slightly. Even if the bursts are subtle, the effect will add up tremendously. Noise doesn’t need to be white or pink; crackly impulsive noise is fun too, and more metallic noise is possible using banks of bandpass filters or inharmonic FM.
Adding a little puff of noise before the transient can sound really nice for reed instruments, and I’ve gotten a pretty decent thumb piano sound with it by simulating the thumb scraping against the key. Watch yourself, Four Tet.
Add “box tone” to every sound source using a bunch of random peaking filters that boost different bands by +-2 dB (maybe more if you’re feeling adventurous!). Wiggle the parameters around slowly if desired, which is pretty ad hoc but might mimic physical changes in temperature, posture, grip of the instrument, etc. Box tone is a good idea in general to compensate for the relative cleanliness of an all-digital signal path. You can even use multiple layers of wiggly EQ with gentle nonlinearities sandwiched between them.
This is an obvious one, but almost all acoustic instruments have resonating bodies, so reverb can make or break the realism of a sound. Use slightly different subtle reverbs on every instrument, and prefer “weird” and metallic reverbs with short decay times. I often just use a bank of parallel comb filters; you can also use short multitap delays. The lush Alesis/Lexicon sound has its place in mixing, but I find that sound a little too smooth to work as an instrument body. Obviously, reverb on master is pretty essential if your instruments are in a concert hall.
Inharmonic modal synthesis (either with actual resonance or with decaying sine waves) can be enhanced with parallel ring modulation with one or more sine waves. This greatly multiplies the number of partials. I like to add a decay on the “modulator” sine waves. This works best for a grungy junk percussion sound, banging on pots and pans that aren’t carefully tuned by instrument builders.
It’s not just the patch, it’s the sequencing. Humanize velocity; don’t just completely randomize it, make the velocity follow the musical phrasing. Also, louder playing is correlated with higher timing accuracy, and conversely softer playing benefits from humanization in arrival times.
String players performing in detache style tend to crescendo a little bit when they anticipate the bow change for the next note. I’ve never played winds, but I wouldn’t be surprised if they did something similar.
For instruments that comprise a different physical source for every pitch (pianos, mallet instruments, pipe organs, harmonicas), try detuning each key by a tiny, fixed amount to emulate imperfections in the instrument. You can use a lookup table, but my favorite approach is to use the pitch to seed a random number generator; I use the Hasher UGen in SuperCollider a lot for this. Timbral parameters can be similarly randomized in this manner.
Haas effect panning: delay one channel and put it through a random EQ that’s sloped for high-frequency loss.
SuperCollider’s PitchShift is really great for adding a weird metallic “splash” which I mix back into the signal, sometimes fairly subtly. In general, find weird effects and use them in parallel at a low volume to simulate little imperfections that add up.
The notion that bass has to be mono is a total myth, and only matters today if your music is being pressed to vinyl. (Trust me, nobody is summing your stereo channels.) Low instruments can absolutely have stereo image and using that can make a mix sound less electronic.
If simulating an ensemble, some instruments will be further away than others, which can be mimicked with higher wet-dry ratio in a reverb and some high-frequency loss. This helps particularly for homogeneous ensembles like string orchestras.
Wind and string instruments require significant dexterity or physical exertion to reach the higher notes in their registers. To mimic this, random detuning should be more dramatic for higher notes.
Winds and strings playing legato are typically done with rapid portamento, which sounds fine. Realism can be further improved by briefly fading in some noise and/or high passing the source, simulating instability as the instrument physically transitions between consecutive notes.
Saw and pulse waves are pretty obviously synthetic and often need considerable massaging to sound remotely acoustic. Consider other sources if you aren’t getting success with those. Breakpoint synthesis (GENDY) is a favorite of mine.
Common mixing advice states that you should avoid putting effects or processing on anything unless necessary as demonstrated by A/B test. This is a wise idea for recordings, but in mostly synthesized music, a long chain of subtle effects can create an “imperfection cascade” that help get an extra 5% of realism. This only really helps if the sounds are already good enough to stand on their own.
Unisons of multiple instruments can sound more realistic as a whole than exposed instruments, since they can mask each other’s imperfections, especially if those instruments are very different from each other.
Slow parameter fluctuations happen not only for individual instruments, but for the entire ensemble, especially for a homogeneous group like a string quartet. Create an automation and map it to the intensity parameter of many instruments, which also fluctuate individually.

Composing with Accelerating Rhythms

Nathan Ho — Thu, 21 Sep 2023 00:00:00 +0000

Thanks to all who checked out my album Haywire Frontier_. Yesterday, I gave a remote talk for the NOTAM SuperCollider meetup on the project. The talk wasn’t recorded, but I decided to rework it into prose. This is partially for the benefit of people that missed the event, but mostly because I’m too lazy to research and write a new post wholly from scratch this month.

It’s not necessary to listen to the album to understand this post, but of course I would appreciate it.

Acceleration from notes to an entire piece

One of the earliest decisions I had to make while planning out Haywire Frontier was how to approach rhythm. I’m a huge fan of breakcore and old school ragga jungle (Venetian Snares’ work convinced me to dedicate my life to electronic music), and partially as a result of that, unpitched percussion and complex rhythms are central to a lot of my output.

However, I resolved pretty early on that I didn’t want the rhythmic material of the project to fall into the grids and time signatures of dance music. My reasons for this are nebulous and difficult to articulate, but I think a big part is that I wanted to challenge myself. When I make beat-based music, which I do frequently, I tend to think relative to established genres like drum-‘n’-bass or techno or house, and I mimic the tropes of what I want to imitate. Removing those guardrails, while still trying to make music conducive to active listening, puts me out of my comfort zone. I like to put myself in creative situations where I feel a little awkward or uncomfortable, because if there’s anything I personally fear in my creative output, it’s complacency.[1]

So beats are out. An alternative, which I have used a lot in the past, is a type of randomized rhythm I call the “i.i.d. rhythm,” or “Pwhite-into-\dur rhythm:”

SuperCollider code:

// NB: Full aggregated code from example, plus SynthDefs, are at the end of the post.
Routine({
    loop {
        s.bind { Synth(\kick) };
        rrand(0.03, 0.6).wait;
    };
}).play;

In these rhythms, the inter-onset intervals (IOIs), or time between successive hits, are chosen with a single random distribution. In statistics terms, the IOIs are i.i.d., or independently and identically distributed. The distribution is uniform in this example, but you can use log-uniform, or any distribution over the positive real numbers.

Every SuperCollider user has written one of these rhythms at some point. They’re perfectly serviceable for some applications. However, for rhythmic material that drives an entire percussion section, I have to admit that I find these tiresome and uninspiring. In one word, what these rhythms lack is phrasing.

If you were to grab a non-musician, give them a snare drum, and ask them to hit it “randomly,” their result would be nothing like this. They might produce a cluster of rapid hits, then silence, then a nearly steady rhythm, and modulate between all those approaches. That’s to say nothing of a free jazz drummer who’s spent years training to produce complex, compelling rhythms that may not fall on a grid. It’s well known to psychologists that humans are very bad at producing data that passes randomness tests; I view it as Geiger-counter-type rhythms failing to pass humanity tests.

What is there between the extremes of i.i.d. rhythms and grid-based rhythms? There’s a lot, and at some point I arrived at accelerating rhythms, which ended up being a major strand clearly audible throughout the album.

Routine({
    loop {
        (0.9 ** (0..10)).do { |duration|
            s.bind { Synth(\kick) };
            duration.wait;
        };
    };
}).play;

This is a sequence of 10 durations that follow a decaying geometric series, which then restarts periodically. (This is not a Risset rhythm; the acceleration suddenly restarts, and there’s no auditory illusion here.) This kind of rhythm begins to tap into the notions of motion and phrasing, and is immediately far more inspiring for me than total randomness. I don’t remember exactly how I arrived at it, but Conlon Nancarrow’s Canon X may have planted the seed. Speaking of influences, I’m not claiming that any ideas I present here are novel; this is just my personal journey with rhythm.

As a minor point of housekeeping, I find the expression 0.9 * (0..10) to be a bit unmusical, as I can’t compute the total duration of a phrase in my head. To fix this, I normalize the sum of those durations to 1 and then explicitly multiply it by the number of seconds a single accelerating phrase lasts:

Routine({
    loop {
        ((0.9 ** (0..10)).normalizeSum * 3.5).do { |duration|
            s.bind { Synth(\kick) };
            duration.wait;
        };
    };
}).play;

This is minor, but for me code is the “score” of my music, and little cosmetic choices like this have a butterfly effect on how I compose.

The next step, which was a key moment in the inception of the album, was to recurse this process. Now the phrase durations themselves can accelerate according to a geometric series:

Routine({
    ((0.8 ** (0..8)).normalizeSum * 15.0).do { |phraseDuration|
        ((0.75 ** (0..10)).normalizeSum * phraseDuration).do { |duration|
            s.bind { Synth(\kick) };
            duration.wait;
        };
    };
}).play;

There is obviously perceptible motion both at the scale of a phrase and at the scale of an entire section. Also note that the 15.0 allows me to enter the duration in seconds of this entire section. While mathematically and computationally simple, having the section duration explicitly encoded as a number has a big psychological impact on how I work. Instead of thinking of this rhythm as an endless generative process that gets cut off at some point, this makes me think of plotting out the next 15 seconds of music and then subdividing it. When I make algorithmic music based on infinitely unfolding processes, I often have trouble with formal structure, and thinking in terms of directions and goals helped a lot with ossifying a piece with a beginning, middle, and end.

You can imagine where this is going, but some quick digressions first. The first is some minor sound design seasoning. I find the drum to be a little too mechanical-sounding, so I like to pass the duration into the synth and have its timbre react accordingly:

Routine({
    ((0.8 ** (0..8)).normalizeSum * 15.0).do { |phraseDuration|
        ((0.75 ** (0..10)).normalizeSum * phraseDuration).do { |duration|
            s.bind { Synth(\kick, [duration: duration]) };
            duration.wait;
        };
    };
}).play;

I’ve written about this previously as the Duration Trick. I haven’t shown the actual kick patch, but it is quieter and softer for shorter durations, which mimicks how a musician tends to play notes with softer velocities for rapid passages.

Another, more dramatic change is that we can start interrupting this recursive rhythm with… well, anything, really. In this case I’ve added a 50% chance of a clanky snare drum at the end of each phrase (I love clanky, SOPHIE-style snare drums):

Routine({
    ((0.8 ** (0..8)).normalizeSum * 15.0).do { |phraseDuration|
        ((0.75 ** (0..10)).normalizeSum * phraseDuration).do { |duration|
            s.bind { Synth(\kick, [duration: duration]) };
            duration.wait;
        };
        if(0.5.coin) {
            s.bind { Synth(\snare) };
            0.28.wait;
        };
    };
}).play;

And the next step is, you guessed it, three levels of acceleration.

Routine({
    ((0.8 ** (0..8)).normalizeSum * 50.0).do { |sectionDuration|
        ((0.8 ** (0..8)).normalizeSum * sectionDuration).do { |phraseDuration|
            ((0.75 ** (0..10)).normalizeSum * phraseDuration).do { |duration|
                s.bind { Synth(\kick, [duration: duration]) };
                duration.wait;
            };
            if(0.5.coin) {
                s.bind { Synth(\snare) };
                0.28.wait;
            };
        };
    };
}).play;

Soon after I started experimenting with this approach to recursively accelerating rhythms, I realized there was a connection to both mathematics and a long-lost childhood hobby of mine. These rhythms are a loose sonification of the ordinal number system, which are a hierarchy of “infinities” developed by Georg Cantor.

Ordinal numbers are rather special to me because (life story incoming) as a kid, I was really fascinated by the study of large numbers, so much that in 2008 I created a wiki about them called “Googology Wiki.” I was 11 at the time. The ordinal number system is closely linked to certain systems for producing extremely fast-growing functions, and as such the site has many articles on various landmarks in the ordinal number system. I grew disinterested in the wiki over the years, but over time it ended up getting a lot of traction. Although I can’t take credit for the majority of its content, it’s easily the most famous thing I’ve created, having amassed tens of thousands of articles and even some citations in academic papers. Fair warning if you visit the wiki — it was made on then-Wikia, now Fandom, and it is now sadly plastered with video ads for the MCU.

I’m not the first person to connect ordinals to rhythms, in fact. In a blog post titled “Ordinal music,” Adam P. Goucher noticed that there’s a vaguely ordinal-like structure in the buildups in PSY’s “Gangnam Style.” (Stay tuned for my analysis of chaotic dynamical systems in “Call Me Maybe.”) He produced an audio track that sonifies four levels of recursive acceleration.

One of the tracks on my album is like that, comprising a pure, skeletal sonification of recursive acceleration, although only three levels. It’s meant as an interlude and a nod to the process music of composers like Tom Johnson. For the rest of the album, ordinal rhythms gave me inspiration for structuring form, but I frequently interrupted and altered them to keep the pieces exciting. For these pieces I didn’t use a geometric series for the third level. Instead, I wrapped up the section as a function, and manually typed in the section lengths:

var playSectionA;

playSectionA = { |sectionDuration|
    ((0.8 ** (0..8)).normalizeSum * sectionDuration).do { |phraseDuration|
        ((0.75 ** (0..10)).normalizeSum * phraseDuration).do { |duration|
            s.bind { Synth(\kick, [duration: duration]) };
            duration.wait;
        };
        if(0.5.coin) {
            s.bind { Synth(\snare) };
            0.28.wait;
        };
    };
};

Routine({
    playSectionA.(20.0);
    playSectionA.(7.0);
    playSectionA.(5.3);
}).play;

Structuring the piece this way lets me view the ordinal-rhythm sections as a sort of “refrain” or “chorus” that accelerates with every iteration, and I can interweave them with other sections:

Routine({
    playSectionA.(20.0);
    playSectionB.();
    playSectionA.(7.0);
    playSectionC.();
    playSectionA.(5.3);
    playOutro.();
}).play;

I’ve experimented in the past with algorithmic approaches to form, but I always end up with dialing in high-level formal structure manually. However, on a conceptual level, I find it really nice how form has emerged very naturally from the low-level materials.

Other variations

Branching off from my investigations into rhythm, there are a few other variants and related concepts that made their way into the album. My general philosophy for finding these rhythms is that they need to 1. sound good and 2. be elementary to represent in code. Concise, elegant code is a constraint I take seriously in creative coding; I try to avoid engineering complex software to compose music.

Mixed acceleration and deceleration

You can of course reverse the entire rhythm to produce recursive deceleration. More exciting to me is the idea of reversing only some levels in the hierarchy. Here’s what happens when the phrases and the entire piece are accelerating, but the section is decelerating:

Routine({
    ((0.8 ** (0..8)).normalizeSum * 50.0).do { |sectionDuration|
        ((0.8 ** (0..8)).normalizeSum.reverse * sectionDuration).do { |phraseDuration|
            ((0.75 ** (0..10)).normalizeSum * phraseDuration).do { |duration|
                s.bind { Synth(\kick, [duration: duration]) };
                duration.wait;
            };
        };
    };
}).play;

There are eight different possibilities with three levels of acceleration/deceleration, and they all have different “vibes” that are hard to predict on paper. The possibilities multiply if the phrases and sections are themselves heterogeneous; for example one can pick randomly between acceleration and deceleration for each phrase or section.

Sorted random rhythms

Another way to generate accelerating and decelerating rhythms is to generate a bunch of durations with a random distribution and then sort them. It’s not radically different in sound from a deterministic approach, but it sounds a little less predictable and I find the controls easier to tweak.

Routine({
    (({ exprand(0.01, 0.5) } ! 20).sort.normalizeSum * 3.5).do { |duration|
        s.bind { Synth(\kick, [duration: duration]) };
        duration.wait;
    };
}).play;

Recursive subdivision

Here, the notion of acceleration and deceleration is generalized with an arbitrary “row” of durations that are recursively subdivided in a self-similar or fractal manner:

Routine({
    var row;
    row = [0.2, 1, 1.4, 0.8, 1].normalizeSum;
    (row * 20.0).do { |sectionDuration|
        (row * sectionDuration).do { |phraseDuration|
            (row * phraseDuration).do { |duration|
                s.bind { Synth(\kick, [duration: duration]) };
                duration.wait;
            };
        };
    };
}).play;

This is an old concept, and not far off from the tradition of total serialism. (As someone who often uses mathematical concepts, my philosophy sometimes overlaps with 20th-century modernist music.) John Cage structured his prepared piano studies this way, and John Coolidge Adams’ piano piece Phrygian Gates has “marks” placed with a scheme like this. I didn’t use this much on my album, except for the final track.

Sinusoidal rhythms

This is just a steady beat with tempo fluctuating according to a sine wave. This is one of the more prosaic ideas here as far as mathematics goes, but it still sounds really great to my ears.

Routine({
    30.do { |i|
        var duration;
        duration = cos(i * 2pi / 20).linlin(-1, 1, 0.05, 0.2);
        s.bind { Synth(\kick, [duration: duration]) };
        duration.wait;
    };
}).play;

I can imagine some variants of this — you can substitute the cosine function for any mathematical function you want. Maybe a sum of sinusoids to produce “additive synthesis rhythms?” What other audio signal processing algorithms can generate compelling tempo signals?

Gridless discrete rhythms

These are actually a type of i.i.d. rhythm, but instead of using a continuous distribution a discrete distribution is used. The durations are randomly selected from a finite set of IOIs, which are not related to each other by an audible grid.

Routine({
    30.do { |i|
        var duration;
        duration = [0.02, 0.1, 0.432].choose;
        s.bind { Synth(\kick, [duration: duration]) };
        duration.wait;
    };
}).play;

If an IOI repeats, the listener may perceive a brief grid; the randomness starts to form recognizable patterns.

My track “You Poor Thing” elaborates on this concept using a first-order Markov chain of discrete IOIs that also determines the type of drum hit (kick, snare, etc). The B-section that occurs at the midway point uses gridless discrete rhythms, but manually sequenced.

Conclusion

As an album with shamelessly maximalist aesthetics, there are many, many strands in Haywire Frontier. This is only one, but it was the most important for me developmentally. Working out the rhythmic material was a breakthrough for figuring out how I wanted the project to sound, and turned out to be a huge boon for getting past the classic problem of the blank page.

I should mention that these approaches all technically fall under the umbrella of xenrhythm, a variant of xenharmony, and a term so obscure that it makes basically no appearance outside a Facebook group and Fandom (ugh) wiki with 15 pages. One blog post that’s been sitting in my drafts is a survey of xenrhythms, structured much like my survey of non-standard oscillators.

Footnotes

Appendix: full source code

Here’s the full SuperCollider code I used to generate the examples. I don’t share SynthDefs publicly that much, but I’m dropping this as a thanks to all those who supported the album.

(
SynthDef(\kick, {
    var snd, duration, velocity;
    duration = \duration.kr(1.0);
    velocity = duration.linlin(1, 0, 1, 0);
    snd = SinOsc.ar(
        60
        * (1 + (8 * Env.perc(0, 0.001).ar * velocity))
        * (1 + (8 * Env.perc(0, 0.03).ar * velocity))
        * (1 + (0.5 * Env.perc(0, 0.3).ar * velocity))
        * ([1, -1] * 0.1).midiratio
    );
    snd = snd * (1 + (Env.perc(0, 0.03).ar * velocity));
    snd = snd + (BPF.ar(Hasher.ar(Sweep.ar), 8321, 0.3) * Env.perc(0.001, 0.003).ar * 1.dbamp * velocity);
    snd = snd.tanh;
    snd = snd + (BPF.ar(Hasher.ar(Sweep.ar), 3321, 0.3) * Env.perc(0.03, 0.05).ar * -10.dbamp * velocity);
    snd = snd * velocity.sqrt;
    snd = snd + GVerb.ar(snd.sum * -30.dbamp, 30, 1);
    snd = snd * Env.perc(0.001, duration.min(0.6)).ar(Done.freeSelf);
    snd = snd * -3.dbamp;
    Out.ar(\out.kr(0), snd);
}).add;

SynthDef(\snare, {
    var snd;
    snd = SinOsc.ar(
        260
        * (1 + (3 * Env.perc(0.001, 0.04, curve: -6).ar))
        * [1, 4.3, 8.4]
    );
    snd = snd * [0, -8, -12].dbamp;
    snd = snd * Env.perc(0.001, [0.3, 0.1, 0.03]).ar;
    snd = snd.sum;
    snd = snd + (BPF.ar(WhiteNoise.ar, 2310, 0.25) * Env.perc(0.03, 0.3).ar * 12.dbamp);
    snd = snd + (BPF.ar(WhiteNoise.ar, 7310, 0.3) * Env.perc(0.003, 0.04).ar * 8.dbamp);
    snd = snd.tanh;
    snd = snd + PitchShift.ar(snd, 0.06, 2.4);
    snd = snd + PitchShift.ar(snd * -5.dbamp, 0.08, 1.3);
    snd = snd * Env.linen(0.001, 0.23, 0.01).ar(Done.freeSelf);
    snd = snd * -7.dbamp;
    snd = snd ! 2;
    Out.ar(\out.kr(0), snd);
}).add;
)

(
Routine({
    20.do {
        s.bind { Synth(\kick) };
        rrand(0.03, 0.6).wait;
    };
}).play;
)

(
Routine({
    3.do {
        (0.75 ** (0..10)).do { |duration|
            s.bind { Synth(\kick) };
            duration.wait;
        };
    };
}).play;
)

(
Routine({
    ((0.8 ** (0..8)).normalizeSum * 15.0).do { |phraseDuration|
        ((0.75 ** (0..10)).normalizeSum * phraseDuration).do { |duration|
            s.bind { Synth(\kick) };
            duration.wait;
        };
    };
}).play;
)

(
Routine({
    ((0.8 ** (0..8)).normalizeSum * 15.0).do { |phraseDuration|
        ((0.75 ** (0..10)).normalizeSum * phraseDuration).do { |duration|
            s.bind { Synth(\kick, [duration: duration]) };
            duration.wait;
        };
    };
}).play;
)


(
Routine({
    ((0.8 ** (0..8)).normalizeSum * 15.0).do { |phraseDuration|
        ((0.75 ** (0..10)).normalizeSum * phraseDuration).do { |duration|
            s.bind { Synth(\kick, [duration: duration]) };
            duration.wait;
        };
        if(0.5.coin) {
            s.bind { Synth(\snare) };
            0.28.wait;
        };
    };
}).play;
)

(
Routine({
    ((0.8 ** (0..8)).normalizeSum * 50.0).do { |sectionDuration|
        ((0.8 ** (0..8)).normalizeSum * sectionDuration).do { |phraseDuration|
            ((0.75 ** (0..10)).normalizeSum * phraseDuration).do { |duration|
                s.bind { Synth(\kick, [duration: duration]) };
                duration.wait;
            };
            if(0.5.coin) {
                s.bind { Synth(\snare) };
                0.28.wait;
            };
        };
    };
}).play;
)

(
Routine({
    ((0.8 ** (0..8)).normalizeSum * 50.0).do { |sectionDuration|
        ((0.8 ** (0..8)).normalizeSum.reverse * sectionDuration).do { |phraseDuration|
            ((0.75 ** (0..10)).normalizeSum * phraseDuration).do { |duration|
                s.bind { Synth(\kick, [duration: duration]) };
                duration.wait;
            };
        };
    };
}).play;
)

(
Routine({
    (({ exprand(0.01, 0.5) } ! 20).sort.normalizeSum * 3.5).do { |duration|
        s.bind { Synth(\kick, [duration: duration]) };
        duration.wait;
    };
}).play;
)

(
Routine({
    var row;
    row = [0.2, 1, 1.4, 0.8, 1].normalizeSum;
    (row * 20.0).do { |sectionDuration|
        (row * sectionDuration).do { |phraseDuration|
            (row * phraseDuration).do { |duration|
                s.bind { Synth(\kick, [duration: duration]) };
                duration.wait;
            };
        };
    };
}).play;
)

(
Routine({
    30.do { |i|
        var duration;
        duration = cos(i * 2pi / 20).linlin(-1, 1, 0.05, 0.2);
        s.bind { Synth(\kick, [duration: duration]) };
        duration.wait;
    };
}).play;
)

(
Routine({
    30.do { |i|
        var duration;
        duration = [0.02, 0.1, 0.432].choose;
        s.bind { Synth(\kick, [duration: duration]) };
        duration.wait;
    };
}).play;
)

This is just my competitive personality showing, not a prescription for how everyone else should work.

Nathan Ho - Haywire Frontier

Nathan Ho — Sat, 02 Sep 2023 00:00:00 +0000

My first full-length solo album, Haywire Frontier, is releasing on Saturday, September 9th on the Japanese label Tokinogake. It is available for preorder now, and you can listen to the opening track “Trickster Deity.”

Here are the liner notes:

In 2008, at the age of 11, I created Googology Wiki on my parents’ computer. “Googology” is word for the study of large numbers and fast-growing functions, deriving from the 9-year-old Milton Sirotta’s coinage of the term “googol.” The website was never meant to go beyond my personal use, and I gradually drifted away from it. Fifteen years later, it has grown to tens of thousands of articles and a community of hundreds of active users.

Haywire Frontier is a 40-minute musical tribute to a strange corner of amateur mathematics whose growth I somewhat-inadvertently catalyzed, with rhythmic and formal material deriving from Georg Cantor’s “ordinal number” system, integral to the study of large numbers.

The album was sequenced and synthesized entirely in SuperCollider with no samples, external hardware, or third-party plugins.

Credits:

John Tejada, mastering
Isa Hanssen (Instagram), cover art
Special thanks to a0n0, Charlie Burgin (Sahy Uhns), William Fields, RM Francis, Joonas Siren (Forces), Ben Tillotson, Nathan Turczan.

I expect to write about this project in the near future. Thank you for listening, and for all your support.

An Intro to Wavelets for Computer Musicians

Nathan Ho — Wed, 30 Aug 2023 00:00:00 +0000

I wasn’t able to get this post fully complete in time for my self-imposed monthly deadline. I have decided to put it up in an incomplete state and clean it up in early September. I hope it is informative even in its current condition, which gets increasingly sketchy towards the end. Open during construction.

Among DSP types, those unfamiliar with wavelets often view them as a mysterious dark art, vaguely rumored to be “superior” to FFT in some way but for reasons not well understood. Computer musicians with a penchant for unusual and bizarre DSP (for instance, people who read niche blogs devoted to the topic) tend to get particularly excited about wavelets purely for their novelty. Is the phase vocoder too passé for you? Are you on some kind of Baudelairean hedonic treadmill where even the most eldritch Composers Desktop Project commands bore you?

Well, here it is: my introduction to wavelets, specifically written for those with a background in audio signal processing. I’ve been writing this post on and off for most of 2023, and while I am in no way a wavelet expert, I finally feel ready to explain them. I’ve found that a lot of wavelet resources are far too detailed, containing information mainly useful to people wishing to invent new wavelets rather than people who just want to implement and use them. After you peel back those layers, wavelets are surprisingly not so scary! Maybe not easy, but I do think it’s possible to explain wavelets in an accessible and pragmatic way. The goal here is not to turn you into a wavelet guru, but to impart basic working knowledge (with some theory to act as a springboard to more comprehensive resources).

Before we go further, I have to emphasize an important fact: while wavelets have found many practical uses in image processing and especially biomedical signal processing, wavelets are not that common in audio. I’m not aware of any widely adopted and publicly documented audio compression codec that makes use of wavelets. For both audio analysis alone and analysis-resynthesis, the short-time Fourier transform and the phase vocoder are the gold standard. The tradeoffs between time and frequency resolution are generally addressable with multiresolution variants of the STFT.

There is no one “wavelet transform” but a huge family of methods. New ones are developed all the time. To limit the scope of this post, I will introduce the two “classical” wavelet transforms: the Continuous Wavelet Transform (CWT) and Multiresolution Analysis (MRA). I’ll also go over popular choices of individual wavelets and summarize their properties. There are other wavelet transforms, some of more musically fertile than CWT or MRA, but you can’t skip the fundamentals before moving on to those. My hope is that demystifying wavelet basics will empower more DSP-savvy artists to learn about these curious creatures.

Table of Contents

Preliminaries

I should first address a question some of my readers have: what are wavelet transforms? The best answer I can give is that a wavelet transform is any transform related to the original Continuous Wavelet Transform (CWT). Virtually all wavelet transforms capture information about the signal that is localized in both time and frequency. The wavelet family of transforms are therefore examples of “time-frequency transforms.” In the case of image processing, “time” is not actually time but rather “space,” and the input signals are two-dimensional.

The short-time Fourier transform (STFT), which underlies audio spectrograms and the phase vocoder technique, also captures localized time and frequency information. However, the CWT has an important property that the STFT doesn’t, which is scale-invariance. In image processing, scale-invariance implies that a 100-pixel square and a 200-pixel square have similar fingerprints in the wavelet domain, which is useful in image recognition. In audio processing, slowing down an audio file by 50% also translates to a simple wavelet-domain transformation. However, this doesn’t have the same parity with perception: playback speed can drastically change how we perceive audio, such as turning Dolly Parton into George Michael. Scale invariance is a big deal in CWT, but do we perceive audio waveforms in a scale-invariant manner? I would argue we don’t, so this advantage of wavelet transforms doesn’t apply so well for audio.

Again, wavelet transforms are not inherently “better” than Fourier transform variants, and they have their own advantages and disadvantages. The STFT still reigns king for audio frequency transforms. When wavelet transforms are used in audio, they are much more often used for analysis than for the implementation of wavelet-domain effects. Of course, commercial interests play a role here — there’s a lot more money in audio data mining than music production — but fundamentally the Continuous Wavelet Transform is inherently difficult to use in effect design, much harder and more computationally expensive than the phase vocoder.

Still, wavelets are fun and interesting and cool and the math is beautiful, otherwise I wouldn’t be writing 8,000 words on them. They also have a reputation for being very difficult, a reputation which is partially earned. I wouldn’t jump into them if you’re a DSP beginner and are easily intimidated by equations. You will need the following:

Single-variable integral calculus, complex numbers, dot products of vectors
Convolution theorem, and the equivalence of linear time-invariant filters and convolution
Shannon–Nyquist sampling theorem, including its implications when signals are upsampled or downsampled by integer factors
Digital FIR filters at a basic level (knowledge of classical filter design methods such as Parks–McClellan, etc. is not required)

I have papered over some finer mathematical details that are not necessary for working knowledge of wavelets. For example, the mathematician’s conception of (L^2(\mathbb{R})) requires that functions that differ by finitely many points are considered equivalent. However, this distinction is unimportant for scientific and engineering applications. If you’re pedantic enough to care about that sort of thing, you’d be better off reading a proper wavelet textbook. Kaiser’s A Friendly Guide to Wavelets [Kaiser1994] is a good introduction (although “friendly” only to those with a math background), while Mallat’s A Wavelet Tour of Signal Processing [Mallat1999] is the definitive 800-page tome, a perfect holiday gift for the true wavelet lover in your life.

Any mistakes are my own. Get in touch if you find something wrong. Due to the size of this post, errors are inevitable.

Crash course on Hilbert spaces

A lot of resources on wavelets dive into operator theory and Hilbert spaces. We’ll take a small bite out of that field, only what we need for one-dimensional wavelets. If you want to dig deeper, the first chapter of [Kaiser1994] is pretty comprehensive.

The dot product of two (n)-dimensional vectors in (\mathbb{R}^n) is given as:

[x \cdot y = \sum_{i=1}^{n} x[i] y[i]]

The intuition for the dot product is that it produces a scalar that factors in both the similarity and lengths of the vectors. The dot product is related to the Euclidean norm:

[||x|| = \sqrt{x \cdot x} = \sqrt{\sum_{i=1}^n x[i]^2}]

The foundations of operator theory generalize these two concepts. The first step is to make the entries of the vectors generally complex-valued. An immediate hitch is that if we use the standard dot product, then in general (x \cdot x) is no longer a nonnegative real number, and (||x||) can’t be real either. This is undesirable because we no longer have things like the triangle inequality; complex numbers do not have a natural ordering like the real numbers do. To repair this, we modify the dot product into the inner product:[1]

[\langle x, y \rangle = \sum_{i=1}^{n} x[i] y[i]^*]

where (^*) denotes the complex conjugate. The inner product is identical to the dot product for real vectors. For complex vectors, the inner product is in general a complex value; its absolute value is a “similarity with length” measure much like the dot product. However, the norm is always real and nonnegative:

[||x|| = \sqrt{\langle x, x \rangle} = \sqrt{\sum_{i=1}^n x[i] x[i]^*} = \sqrt{\sum_{i=1}^n |x[i]|^2}]

In the context of audio, the interpretation of (\mathbb{R}^n) is not geometric anymore. Instead, it comprises (n) consecutive samples of a time-domain signal. With this interpretation, the norm (||x||) is thought of as “signal amplitude,” and it is proportional to the root-mean-square amplitude of (x). The inner product of two audio signals (\langle x, y \rangle) still functions as a “similarity with length” measure.

(\mathbb{R}^n) has a limitation: it’s unnecessarily restrictive to limit an audio signal to a specific, arbitrary duration. To address this, we bring in our second generalization, which is the space of discrete-time signals. These are vectors (x[i]) with infinitely many entries, indexed by integers (i \in \mathbb{Z}). The norm of an infinite discrete signal is:

[||x|| = \sqrt{\sum_{i=-\infty}^\infty |x[i]|^2}]

However, this infinite summation is not guaranteed to converge, for example a signal where all values are 1. To avoid this problem, we just say those signals are invalid, and restrict the discourse to square-integrable signals where the summation (\sum_{i=-\infty}^{\infty} |x[i]|^2) is defined. A necessary condition for square-integrable signals is that they asymptotically taper off to 0 in both directions; this is a “time locality” property.

The inner product on infinite signals is a straightforward extension of the inner product of finite signals:

[\langle x, y \rangle = \sum_{i=-\infty}^{\infty} x[i] y[i]^*]

You can confirm that this summation always converges if the two vectors are square-integrable, and also that (||x|| = \sqrt{\langle x, x \rangle}) still holds.

Our final generalization concerns continuous-time signals, which are time-series, complex-valued functions (f: \mathbb{R} \rightarrow \mathbb{C}). In a continuous-time signal, the infinite summation turns into an integral:

[||f|| = \sqrt{\int_{-\infty}^{\infty} |f(u)|^2 du}]

which must be well-defined, in which case we call (f) a square-integrable function. Like a square-integrable signal, (f) must asymptotically tend to 0 in both directions. (\sin x) is not square-integrable, but (e^{-x^2} \cos x) is.

The inner product also becomes an integral in the obvious way:

[\langle f, g \rangle = \int_{-\infty}^{\infty} f(u) g(u)^* du]

Again, this integral is guaranteed to converge if (f) and (g) are square-integrable, and (||x|| = \sqrt{\langle x, x \rangle}) still holds.

We’ve talked about four different structures:

real finite-dimensional vectors, notated (\mathbb{R}^n)
complex finite-dimensional vectors, notated (\mathbb{C}^n)
square-integrable discrete-time signals, notated (\ell^2(\mathbb{Z}))
square-integrable continuous-time signals, notated (L^2(\mathbb{R}))

Each is equipped with a norm and an inner product, and all are examples of Hilbert spaces. We won’t concern ourselves with a precise definition of this term, but intuitively, it’s a vector space with a norm and inner product that behave reasonably like those of the above spaces. We’ll only talk about (L^2(\mathbb{R})) and (\ell^2(\mathbb{Z})) in this post, so the general definition doesn’t matter much.

If you start studying wavelet transforms of other spaces such as periodic signals, 2D signals (image processing) and 3D signals (medical imaging), or even signals on spheres (audio spatialization), then the Hilbert space criteria are a common way to help ensure that wavelet transforms are sufficiently well-behaved. The term “Hilbert space” comes up all the time in wavelet papers, but in reality you don’t need a super deep understanding of that to decipher a lot of wavelet literature. Mentally substitute “vector space + norm + inner product that behave like (\mathbb{C}^n)” and that’ll work in most cases.

Continuous Wavelet Transform

With the inner product under our belt, the Continuous Wavelet Transform is actually not that hard to define. Here is the important piece of intuition for the CWT, and what I hope will be an “aha” moment if you’ve been struggling to understand it:

The Continuous Wavelet Transform is just an infinitely dense bank of bandpass-shaped filters tuned to all possible frequencies.

When you think “wavelet,” think “impulse response of a bandpass filter.” Furthermore, all the bandpass filters in the CWT are closely related to each other: they are the result of taking a single bandpass and scaling it in frequency. The prototype bandpass is more properly known as the…

Mother wavelet

…which is a square-integrable function (\psi \in L^2(\mathbb{R})). The mother wavelet may be real-valued (\psi: \mathbb{R} \rightarrow \mathbb{R}) or generally complex-valued (\psi: \mathbb{R} \rightarrow \mathbb{C}). Mother wavelets are often, but not always, subject to the following rules:

It must have a norm of 1: (||\psi|| = 1).
It must have an integral of 0, i.e. no dc bias:

[\int_{-\infty}^{\infty} \psi(u) du = 0]

We’ll break both these rules, but I introduce them to assist with building intuition. These connect to the bandpass filter, because a reasonable definition of a bandpass filter is that 1. its dc bias is zero and 2. its magnitude response tapers off to zero for infinite frequencies. The first requirement is the above integral rule, and the second requirement is automatically satisfied for us, because functions in (L^2(\mathbb{R})) always have tapered magnitude spectra. Thus square-integrable functions not only have time locality, but also frequency locality.

A common way to produce a wavelet is to start with a sinusoid and envelope it with a tapered function, such as (\psi(t) = e^{-t^2} \sin(3 t)). I’ve left out the normalization factor in the formula that would ensure (||\psi|| = 1), but other than that this fits all the criteria.

The wavelet actually doesn’t need to be a bandpass, but rather somewhat bandpass-shaped. If it has a peak at a nonzero frequency and dips at least a little as you approach dc, it’s a useful wavelet. The wavelet (\psi(t) = e^{-t^2} \cos(3 t)) is the real Morlet wavelet. As you can see from the value of its magnitude spectrum at 0 frequency, its dc bias is not exactly zero, but still looks reasonably like a bandpass. This is a very common wavelet, and the one we’ll be using for many examples.

Wavelets can also be complex-valued, such as in the complex Morlet wavelet: (\psi(t) = \exp(3jt - t^2)).

[TODO: Vertical and horizontal scales may be wrong on the frequency-domain plots.]

As far as I can tell real wavelets are slightly more common than complex wavelets, although both are used. We’ll stick to the real Morlet wavelet and talk about other established wavelets in a later section.

Time-shifting the mother wavelet

Given a square-integrable input signal (f), consider the inner product of (f) with (\psi):

[\langle f, \psi \rangle = \int_{-\infty}^{\infty} f(u) \psi(u)^* du]

To gain some intuition over what this inner product is doing, it “analyzes” (f) with (\psi) and produces a single complex number that gives us information about the similarity of (f) and (\psi). If (f) has a lot of energy exactly where (\psi) has energy, then (\langle f, \psi \rangle) is likely to be higher in magnitude. If (f) and (\psi) don’t really line up much then the inner product will be close to zero.

However, because of (\psi)’s time-locality, this analysis only gives us information about a certain window of time. To gain more information about (f), let’s shift the mother wavelet around in time (t \in \mathbb{R}):

[\psi_t(u) = \psi(u - t)]

We can then take the inner product (\langle f, \psi_t \rangle) to analyze (f) at any time. If we expand the inner product

[\langle f, \psi_t \rangle = \int_{-\infty}^{\infty} f(u) \psi(u - t)^* du]

then sharp readers will notice that, as a function of (t), (\langle f, \psi_t \rangle) is just convolution of (f) with the complex conjugate of the mother wavelet (\psi). Since we are convolving (f) with a wavelet, this is just a bandpass filter — nothing exciting yet.

Scaling the mother wavelet

The mother wavelet is also frequency-local, so evaluating (\langle f, \psi_t \rangle) for every (t) only really gives us information about (f) in one frequency band. Here is where the wavelet magic comes in. We scale the mother wavelet in time to change its frequency range:

[\psi_{s,t}(u) = \psi\left( \frac{u - t}{s} \right)]

where (s \neq 0) is the scaling factor and (t) is the time shift applied after scaling. (s > 1) expands or dilates the wavelet, (0 < s < 1) compresses it, (s < 0) performs time-reversal and scaling. In practice, negative scales (s < 0) are often ignored. The full gamut of functions (\psi_{s,t}) gives us the full set of daughter wavelets.

The scale multiplies the center frequency of the bandpass-shaped filter by a factor (1/s). So when you hear high scale, think lower frequency, and vice versa.

It is generally desirable to have all daughter wavelets retain unit norm. From the properties of the norm, we multiply the daughter wavelet by the following corrective factor.

Definition

Given a mother wavelet (\psi), its corresponding set of daughter wavelets are given by (\psi_{s,t} : \mathbb{R} \rightarrow \mathbb{C}) where (s, t \in \mathbb{R}) and (s \neq 0):

[\psi_{s,t}(u) = \frac{1}{\sqrt{|s|}} \psi\left( \frac{u - t}{s} \right)]

(Note that sometimes this factor is written as (1/|s|) or omitted entirely, depending on the application.)

A property of the Fourier transform states that time-scaling a function around 0 also scales its Fourier transform. When a wavelet is scaled by (s), its frequency response is scaled by (1 / s). (Its amplitude also changes.) Thought of as a filter, scaling moves around its center frequency.

Its bandwidth is also scaled by the same amount, so its Q remains constant. For this reason the CWT overlaps with some versions of the Constant-Q Transform, but alas CQT means different things depending on who you talk to.

In audio, it’s common to view filter frequency responses on a log-log scale, like in typical parametric equalizers. Here are a few frequency responses of selected Morlet wavelets with logarithmic frequency and decibel units:

As you can see, they are all formed by a rigid “diagonal” translation of the mother wavelet. The horizontal component of this translation is the result of frequency scaling, while the vertical component is just a change in gain.

Definition of the CWT

We now have all the information we need to define the CWT. With the inner product and the daughter wavelets defined, the formula for expressing the transform is particularly terse.

Definition

Continuous Wavelet Transform. Given a square-integrable signal (f \in L^2(\mathbb{R})) and a mother wavelet (\psi), the Continuous Wavelet Transform (\tilde{f} : (0, \infty) \times \mathbb{R} \rightarrow \mathbb{C}) is defined as:

[\tilde{f}(s, t) = \langle f, \psi_{s,t} \rangle]

While the Fourier transform turns a one-dimensional function of time into a one-dimensional function of frequency, the CWT produces a two-dimensional function, with the parameters of scale and time shift. There is no discretization here: (\tilde{f}(s, t)) is defined for every (s \neq 0) and every real (t).

Being complex, (\tilde{f}(s, t)) has both phase and magnitude. (If the signal and wavelet are real-valued, (\tilde{f}(s, t)) is of course real, but it can be negative.) If we plot the magnitude (|\tilde{f}(s, t)|) as a two-dimensional image with time on the horizontal axis and scale on the vertical axis, we produce something called the scalogram. The visual of the scalogram is extremely useful for understanding the CWT and its variants.

If we take this rectangular-windowed sine wave sweep:

and the real Morlet wavelet, we may view our first scalogram:

[TODO: There might be windowing artifacts in this since I had to truncate the wavelet for practical computation.]

I’ve followed two common conventions in scalograms: a log scale for the vertical axis, and to place smaller scales (and therefore higher frequencies) towards the top.

You can see a diagonal line here that looks pretty close to straight, indicating that the frequency of the sine wave sweep I put in is exponential. We could have also learned that from a spectrogram. However, a typical spectrogram would show a continuous line; here the scalogram breaks up the sweep into a discrete set of “hot spots.” This scalogram’s time resolution divides the sine wave into individual pulses, while still being aware of its changing frequency. However, the time-frequency resolution situation is dependent on the choice of wavelet.

Keep in mind that scalograms, like spectrograms, typically depict only magnitude and not phase, so they don’t contain all the information that the CWT provides.

Inverse Continuous Wavelet Transform

With the CWT established, musicians familiar with FFT processing via the phase vocoder are likely itching to start modifying and inverting the CWT to create scalogram-based effects. Not so fast, soldier.

My own introduction to time-frequency transforms was through FFT in computer music, and I imagine that’s likely the case for much of my audience. It was a rude awakening for me to learn that the Fourier transform is rather special, and in general not all time-frequency transforms are actually invertible in theory and practice. Many wavelet transform variants fail to be invertible at all.

The CWT, the primordial organism that every wavelet-like transform descends from, is in fact invertible. Sort of, kind of, sometimes. If I give you the CWT (\tilde{f}) of a signal, the Inverse Continuous Wavelet Transform (ICWT) allows us to recover (f). However, there are some important caveats:

It is easy to come up with (\tilde{f}) that do not have any inverse CWT. If I modify (\tilde{f}) even a tiny amount, there may not be any (f) that corresponds to it, because the CWT is highly redundant. This is in stark contrast to, say, the discrete Fourier transform, where any modification done in the Fourier domain will always have a valid inverse.
Actually computing the ICWT is difficult in the general case. Some wavelets make it easier. [TODO: explain which ones]
A constant bugbear in the design of FFT pitch shifters, etc. is the problem of phase coherence. In the case of the CWT, phases are even more unintuitive and difficult to work with than they are in FFT.
If (\psi) has a dc offset of 0, then the CWT can’t possibly capture any information about the dc offset of (f), and the ICWT cannot recover it. This isn’t a huge problem for audio signals where dc offsets are generally considered a nuisance, but it’s worth noting to avoid confusion.

Overall, my impression from the literature is that the ICWT is not very common. The CWT is used most frequently for analysis and visualization, and not so much for processing.

Due to all the thorny problems involved, I won’t cover the ICWT in depth. However, we will later discuss an easily invertible variant of the CWT called Multiresolution Analysis.

Discrete-Time Continuous Wavelet Transform

The Continuous Wavelet Transform, as described, is not actually implementable on a computer since takes an analog signal and produces an infinitely dense 2D plane of complex numbers. The CWT can be modified to operate on discrete-time signals, resulting in a discretized version of the CWT. Often this is just called the CWT in practical applications, other times it’s called the “Discrete-Time Continuous Wavelet Transform” or DT-CWT. I don’t need to point out how irritating the naming situation is, but unfortunately, the term “Discrete Wavelet Transform” typically refers to something different.

In the DT-CWT, the mother wavelet (\psi) is still continuous-time and square-integrable with unit norm. The daughter wavelets are again produced by time-shifting and scaling, but after that they are sampled into discrete signals in (\ell^2(\mathbb{Z})):

[\psi_{s,t}[u] = \frac{1}{\sqrt{s}} \psi\left( \frac{u - t}{s} \right)]

This is of course the same formula as before, but I’ve used square brackets to indicate that (\psi_{s,t}) now takes only integer values of (u). Note that although the time parameter (u) is discrete, (s) and (t) are still continuous.

Many (although not all) wavelets are non-bandlimited, so by picking values of the daughter wavelets at integer time points, we may introduce aliasing. However, we know that (\psi) is bandpass-shaped, so there is some frequency that (\psi) has negligible energy above. This lets us place a lower bound on how small (s) can be. In particular, if (\psi) has negligible energy above frequency (F) then we only need (s > F / \text{Nyquist}). There is no need for a specialized antialiasing mechanism: the wavelet itself already provides our lowpass!

The DT-CWT is pretty much identical to the CWT:

Discrete-Time Continuous Wavelet Transform

Given a square-integrable discrete-time signal (f \in L^2(\mathbb{Z})) and a mother wavelet (\psi), the Discrete Continuous Wavelet Transform (\tilde{f} : (0, \infty) \times \mathbb{R} \rightarrow \mathbb{C}) is defined as:

[\tilde{f}(s, t) = \langle f, \psi_{s,t} \rangle]

But (\tilde{f}(s, t)) is still defined for all real (s) and (t), so this is still not fully “discrete.” We have to find some way to quantize (s) and (t).

Quantizing frequency

Generally there’s not much use in really fine-grained shifts of (t) that are less than a sample, so it’s okay to assume that (t) is an integer.

In the case of (s), a common approach is to use exponentially spaced scaling factors, such as (s = 2^{k / d}) where (k) is an integer index and (d) is a number of divisions we care about per octave. For analysis of Western music, setting (d) to a multiple of 12, such as 24 or 48, is a possible choice.

There is a natural lower limit to (s), because at some point the daughter wavelets are so squashed that they have very little energy below Nyquist. As mentioned above, this bound is (s > F / \text{Nyquist}) where the mother wavelet has little to no energy above (F). There is nothing stopping you from using smaller (s) values, but the discretized daughter wavelets will just be aliased garbage and won’t tell you anything that higher (s) values don’t do.

In theory, (s) has no upper limit. As it approaches infinity the daughter wavelets analyze lower and lower frequencies and approach dc. However, in all practical applications you have a lower bound on useful frequencies. In audio, this is usually the 20 Hz lower limit on human hearing.

Although it is common in applications to use exponentially spaced (s) values, be aware that it isn’t a requirement. If you want, you can use linear frequencies, or mel frequencies, or Bark frequencies, or a small set of hand-picked frequencies (such as DTMF tones).

SciPy’s scipy.signal.cwt implements the DT-CWT where (t) is set to all integer sample values, and the values of (s) are specified as an arbitrary array. (See docs.) Notably, SciPy does not provide an icwt that takes us back to the time domain.

Downsampling DT-CWT bands

To recap, we started by conceptualizing the CWT as an infinitely dense bank of bandpass filters on a continuous-time signal. With the DT-CWT we discretized the input signal, then we quantized the frequency bands, producing a finite bank of bandpass filters. At this point we have an algorithm with a straightforward computer implementation.

If you think about what happens for lower frequencies and larger values of (s), the signal formed by convolving with the corresponding daughter wavelets has little high-frequency energy. If the mother wavelet has little to no energy above a frequency (F), then the signal produced by bandpass-filtering the input at scale (s) is effectively bandlimited at (F / s). If (F / s < \text{Nyquist} / 2) then we don’t lose much information by downsampling the signal by a factor of two. So, for (s \geq 2 F / \text{Nyquist}) we only care about every other sample. As the signal is already quasi-bandlimited, the downsampling can be done by naively dropping every other sample; no additional filtering is required. Put another way, for suitably large (s) our set of daughter wavelets can be reduced by only considering (\psi_{s,t}) for even (t).

In general, for scale (s), the corresponding bandpassed signal can be downsampled by up to a factor of (k), where (k = \left\lfloor \frac{sF}{\text{Nyquist}} \right\rfloor). Since we compute the CWT through a stack of convolutions, we can reduce computational cost by just not computing the dropped samples in the first place.

Let’s consider a DT-CWT with 12 bands per octave where each (s) value has been downsampled by the maximum allowable integer factor. If we place a dot at every value of (s) and (t) corresponding to a daughter wavelet (\psi_{s,t}), we’ll get something lilke this:

Thus, the result of the DT-CWT is a single complex value at each of these dots. While somewhat pretty-looking (reminiscent of driving by orchards in NorCal), this downsampling scheme is not done in practice because the data structure needed to store these points is tough to work with. The pattern of dots is ultimately periodic along the time axis, but the period is very long.

I wrote that the downsampling can be done up to a factor of (k), but it need not be exactly (k). A common approach is to round the downsampling factors down to integer powers of 2.

In exchange for slightly suboptimal downsampling, we have something much easier to work with. The pattern of dots has a short period, which is simply the time distance between samples in the bottom octave. (We still get all these benefits even if the spacing of (s) is not exponential.)

This variable-rate downsampling is more than just a shortcut to save space and kiloflops. It’s a fundamental component of our next topic:

Multiresolution Analysis (Fast Wavelet Transform)

If you thought the DT-CWT sounds like a fairly intensive algorithm, you’d be right — as described, it’s a heavy operation. You may wonder if there is an equivalent of the Fast Fourier Transform for the DCWT.

There is in fact a Fast Wavelet Transform, but unlike the FFT, the Fast Wavelet Transform is not a fast implementation of the CWT, but something different. It’s closely related to the CWT, and it is lends itself to performance optimization much easier than the DT-CWT, but it’s different from the DCWT in crucial ways.

In particular, the FWT implements a constrained form of CWT called Multiresolution Analysis (MRA). MRA differs from CWT in several crucial aspects:

Invertibility: MRA has no redundancy and is easy to invert. You can alter the MRA coefficients and perform an inverse FWT, and there will be no problems synthesizing a modified signal. The implementations of the forward and inverse FWT are both simple to implement and computationally efficient.
Restrictions on wavelets: MRA only has these nice properties for a special subset of wavelets known as orthogonal wavelets. The Morlet wavelet is not orthogonal, but the Haar and Daubechies wavelets are, and we’ll see them in a bit.
Coarse frequency resolution: In MRA, the scale parameter is constrained to nonnegative integer powers of 2: (s = 1, 2, 4, 8, \ldots).[2] In musical terms, the frequency resolution is one octave per band.

The first means that MRA is immediately useful for the creation of novel musical effects. The last, however, is a serious bummer for music processing, since you just don’t get much frequency resolution with the Fast Wavelet Transform. This doesn’t meant that MRA is musically useless, but I want to temper any hopes that it’s some kind of magic bullet for effect design.

Note

The terms “Fast Wavelet Transform,” “Multiresolution Analysis,” and “Discrete Wavelet Transform” differ in the following ways: …uh, I don’t know. My understanding is that Multiresolution Analysis is the mathematical concept, while the Fast Wavelet Transform is the algorithm that implements it. I avoid the term Discrete Wavelet Transform because it is too easily confused with the DT-CWT, but unfortunately it is in common use. As irksome as the naming situation is, the names are less important than the concepts, and fortunately most wavelet papers open by defining the transforms they use.

This section introduces the MRA. Bear with me, as it’s going to look pretty different from everything we’ve talked about. It will connect back with the CWT, I promise.

Analysis with MRA

I’m going to assume real wavelets here. Modifications are necessary for complex wavelets, albeit only minor ones.

To implement MRA, we are given a reconstruction lowpass filter with impulse response (h[t]) where (t \in \mathbb{Z}) Practical implementations require (h) to be FIR; IIR lowpass filters are also of use since they can be truncated.

We can’t just pick any (h); it has to satisfy certain special properties which will be shortly described. One is that the cutoff of (h) should be half Nyquist, or one quarter of the sample rate. [TODO: note amplitude normalization.] There is no requirement that (h[t]) is causal, meaning it may be nonzero for (t < 0). In e.g. image processing there is no notion of “causality,” but in real-time audio this adds a complication, one that we’ll talk about later.

From the reconstruction lowpass filter we produce three other filters:

[\begin{aligned} \begin{align*} h^[t] &= h[-t] , \text{(analysis lowpass filter)} \ g[t] &= -(-1)^t h[-t] , \text{(reconstruction highpass filter)} \ g^[t] &= g[-t] , \text{(analysis highpass filter filter)} \ \end{align*} \end{aligned}]

Here’s what they look like:

[TOOD: fix crashing text]

(h^) is the time-reversal of the filter about (t = 0). Filter nerds can verify that the convolution (h * h^) produces a zero phase lowpass filter whose magnitude spectrum is the square of the magnitude spectrum of (h); this is the exact same principle as that of forward-backward filtering. (The superscript (^) is not quite the complex conjugate but rather the adjoint of the filter, which I don’t have the space to describe. Don’t confuse it with the centered (), which is plain old convolution.)

(g) time-reverses (h) and flips the sign of every other coefficient, and (g^) in turn time-reverses (g). These are highpass filters: multiplying (modulating) a signal with a sine wave at Nyquist produces a spectral inversion. Again (g * g^) is zero phase, and its magnitude spectrum is the square of (g)’s magnitude spectrum.

The filters I’ve plotted above in this example are rather special ones, with (h) being derived from the 8th-order Daubechies wavelet. These are FIR filters and they vanish to zero outside of the range I’ve plotted. Also, noting the ranges on the X-axes, we have (h[t] = 0) for (t < 0), so (h) and (g^*) are causal; this is not necessarily true of all filters. We can plot their magnitude responses on a linear-linear scale to confirm that these are indeed low and highpass filters whose cutoffs are half Nyquist:

[TODO: add zero padding prior to FFT for smoother response]

(g) and (g^) have the same magnitude responses and so do (h) and (h^); between each adjoint pair the phase responses (not shown) are sign flips of each other.

We define the linear (but not time-invariant!) operators (S_\Downarrow) and (S_\Uparrow). (S_\Downarrow) downsamples a signal by 2x by discarding every other sample: ((S_\Downarrow f)[t] = f[2t]). (S_\Uparrow) upsamples a signal 2x by inserting zeros: ((S_\Uparrow f)[2t] = f[t]) and ((S_\Uparrow f)[2t + 1] = 0). In the frequency domain, (S_\Downarrow) folds the top half of the digital signal down to the top half, and (S_\Uparrow) mirrors the entire spectrum around half Nyquist.

We are now ready to define the Fast Wavelet Transform:

Definition

The Fast Wavelet Transform of a discrete real signal (x) is defined recursively with the following algorithm:

Define (b_0 = x).
- Repeat for (m = 1,2,\cdots,J):
  1. Lowpass with (h^) and downsample: (b_{m + 1} = S_\Downarrow(b_m * h^)). This passes on to the next repetition.
  2. In parallel with 2a, highpass with (g^) and downsample: (a_{m + 1} = S_\Downarrow(b_m * g^)). Save this.
Output the signals (a_1, a_2, \cdots, a_J) and the signal (b_J).

A signal with subscript (m) has a sample rate of (2^{-m}f_s) where (f_s) is the sample rate of the original signal.

I know you’re looking for a block diagram, so here it is, topping out at (J = 3):

The output signals are outlined with gray boxes. Note that the (a_\cdot) and (b_\cdot) steps are just variable names, not actual filters or processing steps.

The downsampling operator naturally introduces aliasing, but since (h^) is a lowpass, it acts as an antialiasing filter by attenuating the high frequencies. Complementarily, (g^) filters out the low frequencies, and the downsampling operator mirrors these high frequencies around half Nyquist.

If we only want to use MRA to analyze the signal, then we can just take our downsampled signals and be done with it. But a magical property of the MRA states that, under certain conditions, it is possible to use these analysis signals to recover (u^0).

Definition

Given an MRA (a_1, a_2, \cdots, a_J, b_J), the original signal may be reconstructed with Inverse Fast Wavelet Transform:

Repeat for (m = J - 1, \ldots, 0):

[b_m = (S_\Uparrow b_{m + 1}) * h + (S_\Uparrow a_{m + 1}) * g]

Output (x = b_0).

Block diagram block diagram block diagram:

In plain English, we take the lowpass/highpass pair, upsample them, and convolve them with the reconstruction lowpass and highpass filters. Then we add them together to try to reconstruct the highpass signal.

I wrote that we try to reconstruct (b_0). In reality, (h[t]) must be designed in the first place so that perfect reconstruction is possible. In practice, someone implementing an MRA will know a priori that (h[t]) is suited for this purpose. The takeaway is that you can’t plug random lowpass filters into MRA and get perfect reconstruction.

Implementation details

This algorithm is almost implementable right out of the block diagram, but we first need to discuss a minor optimization and a major caveat.

[TODO: polyphase filtering, latency compensation in non-causal filters]

Connecting MRA to the CWT, part 1: the intuition

It is entirely understandable if you are having trouble understanding the relationship to wavelets here. That was the case for me when I first looked at the FWT algorithm. However, as promised, there is a deep connection between these two transforms. The MRA-CWT linkage gets a little technical, and we’ll see a more precise treatment in the next section, but I want to set it up with a visual explanation first.

The recursive downsampling scheme results in multiple streams at different sampling rates, all of which are power-of-two divisions of the original signal’s sample rate. If we place a dot at the time of every sample, and write the different streams as rows, we get a familiar diagram:

[TODO: MRA dots diagram]

If you guessed that the MRA values are in fact inner products produced by the DT-CWT at power-of-two scales, you win a point:

Theorem

MRA’s (a_m) signals evaluate certain points of the DT-CWT. [TODO: exactly specify the relationship]

This is quite beautiful, but why this is the case isn’t obvious. To gain some intuition, we’ll examine what the MRA is doing in a filter theory sense. As a jumping-off point, we’ll start with the signal (a_2) produced by the FWT. (a_2) is the result of the following operations on the input signal:

Lowpass filter with cutoff (f_N/2) and downsample by a factor of 2.
Highpass filter with cutoff (f_N/4) and downsample again by a factor of 2. (Here (f_N) is the Nyquist frequency of the original signal prior to downsampling. Relative to the current signal’s sample rate, the cutoffs of these filters are always half Nyquist; a common cuss word for these is “half-band filter.”)

If we ignore the downsampling operations for a moment, we find that this is a highpass filter followed by a lowpass filter, which is a bandpass that roughly isolates the frequency content in the range ([f_N / 4, f_s / 2]). What about (a_3)?

Lowpass filter with cutoff (f_N/2) and downsample by a factor of 2.
Lowpass filter with cutoff (f_N/4) and downsample by a factor of 2.
Highpass filter with cutoff (f_N/8) and downsample by a factor of 2.

Again, if the downsampling operations are ignored, this is a bandpass filter that isolates the range ([f_N / 8, f_s / 4]). In general, we may conclude that in the absence of downsampling, the (a_m) signals for (m > 1) are the result of a bandpass filter with a rough passband of ([2^{-m - 1} f_s, 2^{-m} f_s]).

The connection to the CWT should start to feel closer now that bandpass filters have shown up, but we aren’t quite home free yet; we still have those pesky downsampling operations, which being non-time-invariant are not guaranteed to commute with other operations. It turns out that they do commute approximately, and we can conceptually move all the downsampling to the end:[3]

Bandpass filter to the range ([2^{-m - 1} f_N, 2^{-m} f_N]).
Downsample by a factor of (2^{-m}). As the signal only has significant energy up to (2^{-m} f_N), there’s no aliasing.
Downsample by a factor of 2. This folds the upper half of the spectrum and spectrally inverts it to the range ([0, 2^{-m - 1} f_N]).

So the MRA really does have bandpass filter-like functionality, and in step 2 it downsamples out redundant information as we discussed in the DT-CWT. Step 3 isn’t something we’ve seen before — it further compresses the DCWT band with an additional downsampling stage. As this just amounts to leaving out half the dots on the DCWT, we are still technically evaluating the DCWT, just at fewer locations. The fact that we are able to omit half the points without information loss is specific to the shape of the bandpass filter; if its lower band edge was any lower than (2^{-m - 1} f_N) then the signal crashes into itself. (This may be the case if the wavelet is not orthogonal, such as the Morlet wavelet.)

The impulse response of the bandpass filter is of course a daughter wavelet. With the information we have, we can compute the daughter wavelet itself: it is a cascade of (m) analysis lowpass filters at increasing power-of-two scales, followed by one last highpass filter at scale (m). Since all daughter wavelets are related by scaling and time-shifting, we can compute the mother wavelet to arbitrary time precision by increasing (m) as high as we want.

I didn’t mention the final (b_J) signal at the bottom of the FWT. All you need to know for intuition is that this signal’s chain is just lowpasses, and therefore it isn’t actually part of the DT-CWT. To properly describe it we need yet another new concept (sorry), that of the scaling function — and we’ll leave that for the nerds to peruse in the next section.

Connecting MRA to the CWT, part 2: the math

This section is optional. [TODO: This section also probably has some mistakes.]

Welcome, nerds. A scaling function is denoted (\phi(t)), a square-integrable function (\mathbb{R} \rightarrow \mathbb{C}). Like a mother wavelet, it has both time- and frequency-locality. Unlike a wavelet, (\phi(t)) has a dc value (integral) of 1 instead of 0, so the frequency locality is around dc. The scaling function is a lowpass rather than a bandpass; there is no specific requirement of what its cutoff is.

For convenience, we define a dilation operator (D), which stretches out a function by factor of 2 and retains its norm:

[Df(u) = \frac{1}{\sqrt{2}} D(u / 2)]

and a time-shift operator, which displaces the function in time by an integer shift:

[T^nf(u) = f(u - n)]

Scaling functions do not exist for general wavelets as far as I understand; they are only used in conjunction with orthogonal wavelets in the context of MRA. (Many wavelet resources fail to make this clear, and it tripped me up a lot.) An orthogonal wavelet has exactly one associated scaling function, and a scaling function has exactly one asssociated orthogonal wavelet if it satisfies a property called the two-scale relation:

[\phi(u) = \sum_{t = -\infty}^{\infty} h[t] \phi(u - t) = h * D\phi]

where (h[t]) is an integer-indexed sequence of real numbers. In English, the dilation of (\phi) is a linear combination of copies of the original (\phi) translated by integer time shifts. The coefficients (h[t]) are precisely the reconstruction lowpass filter used in the FWT. Note that (h * \phi_{2,0}) is a mild abuse of notation because (h) is a discrete signal and (\phi_{}) is a continuous signal; to make this formally valid imagine that (h) is realized as train of Dirac delta functions at integer time values (and therefore its Fourier transform is periodic).

Some wavelet schemes start with (h[t]) and use that to define the scaling function. Others go in the opposite direction, starting with (\phi(t)) and deriving (h[t]):

[h[n] = \int_{-\infty}^\infty D\phi(t)^* \phi(t - n) = \langle T^n\phi, D\phi \rangle]

In either case, the final piece of the puzzle, which you have likely been anticipating if you made it this far, is to connect (\phi(t)) to the actual mother wavelet (\psi(t)) ([Mallat1999], Theorem 7.3):

[D\psi = g * \phi]

[TODO: Above formula may be sligthly incorrect, check normalization factor.]

Thus the mother wavelet is the combination of the lowpass filter (\phi) and the highpass filter (g), squashed by a factor of two. That a lowpass and highpass compose to form a bandpass should square with our intuition about wavelets.

With these three equations, we can convert among MRA filter coefficients, wavelets, and scaling functions. Converting between the three is not guaranteed to be simple — sometimes (h) can be derived but (\phi) and (\psi) lack a closed form, and other times we’re given (\psi) or (\phi) and we need numerical methods to find (h). In all cases, remember that these conversions only make sense if the following equivalent statements are true: the wavelet is orthogonal, the scaling function satisfies the two-scale relation, and the filter admits perfect reconstruction.

Complicated? Sure. Fortunately, these aren’t tools you’ll need much for typical use and implementation of FWT. In many cases, you’re just given (h). I’ve done this excursion into the math because I wanted an explanation that personally makes sense to me.

Specific wavelets

So far, we’ve only seen two actual wavelets: the real and complex Morlet wavelets for the CWT, and one of the Daubechies wavelets for the FWT. Let’s go over a few other common one-dimensional wavelets. Normalization factors are usually not shown for simplicity.

Morlet wavelet (Gabor wavelet)

[Morlet wavelet diagram]

Real or complex, infinite support, not orthogonal.

The Morlet wavelet is likely the most commonly used wavelet for the CWT. It has both real and complex versions. The real version is formed by enveloping a cosine wave with a Gaussian:

[\psi(t) = e^{-t^2} \cos(\omega t)]

where (\omega) determines the Q factor of the filter, with higher (\omega) resulting in higher Q. The complex version replaces the cosine wave with a complex sinusoid:

[\psi(t) = e^{-t^2} e^{j \omega t}]

Haar wavelet

[Haar wavelet & scaling function diagram]

Real, finite support, orthogonal.

The Haar wavelet predates the wavelet transform by decades, when it formed the basis of the Haar transform. It is a rectangular-shaped “wiggle:”

[\begin{aligned} \phi(t) = \begin{cases} 1 & 0 \leq t \leq \frac{1}{2} \ -1 & \frac{1}{2} < t \leq 1 \ 0 & \text{otherwise} \ \end{cases} \end{aligned}]

Its scaling function is even simpler, a rectangular bump:

[\begin{aligned} \phi(t) = \begin{cases} 1 & 0 \leq t \leq 1 \ 0 & \text{otherwise} \ \end{cases} \end{aligned}]

It can be confirmed visually that (\phi(2t) = (\phi(t) + \phi(t - 1)) / 2), so the filter coefficients are (h = \frac{1}{\sqrt{2}}[1,,1]).

Shannon wavelet

[Shannon wavelet & scaling function diagram]

Real, infinite support, orthogonal.

The Shannon wavelet is the wavelet answer to the ideal filter, or sinc filter. It’s only really useful as a theoretical ideal, not as a practical wavelet. It is easiest to define with its scaling function:

[\phi(t) = \frac{\sin(\pi t)}{\pi t} = \text{sinc}(t)]

Daubechies wavelets

[Daubechies wavelets & scaling functions diagram]

Real, finite support, orthogonal.

The Daubechies wavelets are common choices for MRA. The (n) th-order Daubechies wavelet’s filter has length (2n); the case (n = 1) is the Haar wavelet.

The Daubechies wavelets are most easily specified as filter coefficients, but even they requires a fairly intricate numerical algorithm. Fortunately, you likely don’t need to implement that – SciPy has scipy.signal.daub, which gives you the FIR filter coefficients. (\phi(t)) and (\psi(t)) have no closed form, and generally have a complex fractal structure. They look pretty trippy, especially for naturally occuring mathematical functions.

Mexican hat wavelet (Ricker wavelet)

[Mexican hat wavelet diagram]

Real or complex, infinite support, not orthogonal.

The Mexican hat wavelet (more boringly, the Ricker wavelet) is real:

[\psi(t) = (1 - t^2) e^{-t^2}]

Others

Other orthogonal wavelets include the Meyer wavelet, symlets, and coiflets. If you aren’t constrained by orthogonality, then practically any function with reasonable time and frequency locality can be used as a wavelet. For example, for audio some have used the gammatone or gammachirp functions as wavelets.

Conclusion

Phew! This is my longest work of writing ever at 8,000 words, and I had to switch the entire site from MathJax to KaTeX because the 200+ math formulas became incredibly slow to load.

If you made it through all that or even skimmed through it, then congratulations. If it adds any clarity to the world of wavelets for you, then my mission is successful. Here are some bullet points on the most important takeaways about the CWT and the MRA.

Main Takeaways

The CWT takes a 1D complex-valued function of time and converts it into a 2D complex-valued function of time and scale by piping the input signal into an infinite bank of bandpass-like filters.
The impulse responses of those filters are the daughter wavelets, and are all the result of frequency-scaling and time-shifting a mother wavelet.
The CWT is generally difficult to invert, making it of limited use in audio effects.
The DT-CWT discretizes the CWT by working on discrete-time signals and selecting a finite number of scales, which are conventionally spaced evenly on a log scale, and optionally downsamples signals at higher scales.
MRA completely eliminates the redundancy and results in an easily invertible version of the CWT, but only has a frequency resolution of one band per octave, and to achieve perfect reconstruction the wavelet must be orthogonal.
The FWT is implemented by a simple recursive ladder of half-band filtering and downsampling, and its inverse is equally simple.

To be frank, when I started seriously investigating wavelets for audio applications, I was a little disappointed — the CWT and MRA aren’t as musical as I hoped. The theory of wavelets is beautiful, but in the audio space STFT almost always gets the job done.

However, when I dug deeper into the numerous wavelet transform variants developed over the years, I started finding more unusual transforms that seem to have more musical potential. I won’t divulge what exactly they are yet, as I’m currently experimenting with them for future projects. Stay tuned!

Acknowledgements

Thanks to Julius Smith and R. Tyler McLaughlin for feedback and comments.

Footnotes

References

Kaiser1994
Kaiser, Gerald. 1994. A Friendly Guide to Wavelets.
Mallat1999
Mallat, Stéphane. 1999. A Wavelet Tour of Signal Processing.

Frustratingly, there is a notational disagreement over whether the inner product (\langle x, y \rangle) involves taking the complex conjugate of (x) or (y). Mathematicians and DSPists prefer taking the conjugate of (y) (“linear in first argument”), while in physics the conjugate of (x) is more common (“linear in second argument”). I personally prefer the physics notation, but to avoid confusion with anyone consulting DSP textbooks I’ve gone with the math notation here.
You can also use powers of 3, or any higher integer base, but that only makes the resolution coarser.
I admittedly struggle to explain why we can do this in words, but trust me, it works — but remember that it is an approximation.

The Duration Trick

Nathan Ho — Tue, 04 Jul 2023 00:00:00 +0000

The Duration Trick is something I was egotistical enough to believe I discovered, but after a recent conversation with William Fields (a musical hero of mine) I have learned that I’m in no way the first to come across it. Both Fields and the Max/MSP Shop Boys have been using something like this for a while, I’m told. Learning about this case of convergent evolution spurred me to bump up this already-planned post in the queue.

Simply put, the Duration Trick is when a synthesizer patch with discrete note on/off events is given advance notice of the duration of each note. Thus, if the sequencer is modeled as sending messages to a synthesizer:

Every “note on” message is accompanied with an anticipated duration.

It’s possible for such a patch to exactly anticipate the ending of the note, so note offs don’t even need to be transmitted, although I like to give the option to cut off the note prematurely. Additionally, duration can function as a velocity-like parameter that impacts other synthesis parameters such as amplitude or brightness, so short and long notes differ in more ways than just timing.

Imagine a monophonic subtractive synthesis patch with a lowpass filter that gradually opens for each note on. Traditionally, the lowpass filter’s trajectory is independent of the note duration, and may run its course or be cut short:

With duration information, it’s possible to guarantee that the lowpass filter reaches a precise goal at the end of each note:

I find the second example slightly more exciting in this simple demonstration. For a more complex example, jump about an hour into a video session I uploaded in May. The Duration Trick may sound like a small change at first, but it had a pretty radical impact on my music and sound design when I started using it. It shines especially for transitional sweeps that need to arrive right on time. Arguably, anyone who drags a reverse cymbal sample leading up to a drop is in a sense using the Duration Trick.

Note off events in MIDI can arrive at any time, so the Duration Trick isn’t achievable with standard traditional synthesizer hardware in the absence of some CC-based hack. (This is one of many reasons that pigeonholing everything into MIDI events has had long-term negative effects on music tech, but I digress.) The Duration Trick is therefore easiest to implement in one of the “nerd” music software environments like Csound, SuperCollider, etc., particularly anything that permits scripting. The trick is possible in a real-time context, but the sequencer must of course be able to look ahead far enough to know the durations at all, so it’s more semi-real-time than fully real-time. Durations are always available for music sequenced offline, and are generally available in algorithmic composition as well.

Musicians who play or sing melodies generally don’t think in individual note ons and offs, but rather phrases and gestures if not something higher-level. Even the most reactive and on-the-fly improvisations often require calculating at a few notes ahead, and this will impact subtleties of playing style. The Duration Trick alone doesn’t capture the complexities of musicians playing acoustic instruments, but it still appears to be a valuable stepping stone to breathing some more life into a synth patch.

Correlated Granular Synthesis

Nathan Ho — Sun, 04 Jun 2023 00:00:00 +0000

Decades after Curtis Roads’ Microsound, granular synthesis is making appearances here and there in the commercial plugin market. While it’s nice to see a wider audience for left-field sound design, I have my quibbles with some of the products out there. From what I’ve heard, so many of these products’ demos are covered in reverb in obvious compensation for something, showing that the plugins seem most suited for background textures and transitional moments. In place of sound, the developers seem to prioritize graphics — does watching 3D particles fly around in a physics simulation inspire the process of music production, or distract from it?

Finally, and most importantly, so many granular “synths” are in fact samplers based on buffer playback. The resulting sound is highly dependent on the sampled source, almost more so than the granular transformations. Sample-based granular (including sampling live input such as in Ableton Live’s Grain Delay) is fun and I’ve done it, but in many ways it’s become the default approach to granular. This leaves you and me, the sound design obsessives, with an opportunity to explore an underutilized alternative to sampled grains: synthesized grains.

This post introduces a possibly novel approach to granular synthesis that I call Correlated Granular Synthesis. The intent is specifically to design an approach to granular that can produce musical results with synthesized grains. Sample-based granular can also serve as a backend, but the idea is to work with the inherent “unflattering” quality of pure synthesis instead of piggybacking off the timbres baked into the average sample.

Correlated Granular Synthesis is well suited for randomization in algorithmic music context. Here’s a random sequence of grain clouds generated with this method:

Grainlet synthesis

In Microsound, Roads describes a granular synthesis technique he calls grainlet synthesis. (Dude sure likes neologisms.) Inspired by how wavelets couple frequency with duration, grainlet synthesis refers to any kind of granular synthesis where “parameter linkage” is present. He illustrates with a directed graph of parameters, with arrows showing the dependencies between parameters:

The description is light on details, but as far as I can fathom, the parameters are mostly randomized while parameters with dependencies are compute deterministically from other parameters. In the above diagram, we have parameter 2 controlling parameter 7, so after randomizing parameter 2 for a given grain, parameter 7’s value is found by computing an increasing function of parameter 2’s value. The arrow from 8 to 4 has a gray circle on it indicating inverse linkage – as parameter 8 increases, parameter 4 decreases.

Roads’ grainlet diagram popped into my head recently, and I realized that the trichotomy of linkage vs. inverse linkage vs. no linkage is really a special case of the general concept of correlation in statistics. Thus, Correlated Granular Synthesis was born: granular synthesis where grain parameters are generated by sampling a multivariate probability distribution.

Overall, where many extant granular synthesis methods are opinionated about DSP algorithms, Correlated Granular Synthesis is opinionated about how parameters are set. Thus it really serves as a frontend to granular synthesis methods, allowing us to imagine Correlated Glisson Synthesis, Correlated Trainlet Synthesis, Correlated Pulsar Synthesis, Correlated Vosim Granular Synthesis, etc.

Absolute time as a grain parameter

“Density” (e.g. average grains per second) is a critical parameter for any granular synthesizer. While experimenting with Correlated Granular Synthesis, I realized there’s an alternative characterization where instead of density, we directly think of the absolute timestamp of the parameter as a parameter just like pitch, amplitude, etc.

A traditional granular synthesis approach is to consider all parameters modulatable over time, and these modulations are left up to the composer or performer. In Correlated Granular Synthesis, time is just another grain parameter, and if a non-time parameter “modulates” it’s really a special case of statistical correlation. An important implication is that Correlated Granular Synthesis is a more or less offline sequencer where all grains have to be planned out in advance, although the synthesis itself can be in real time.

Selecting probability distributions

Before we start correlating anything, let’s first pick out the marginal probability distributions. In a multivariate probability distribution, a marginal PDF is what you get when you ignore all variables but one. Here’s a 2D probability distribution with the marginal PDFs displayed:

In this case, the sampling process is independent for each variable. The Pearson and Spearman rank correlations are both very close to 0, and there isn’t any interaction between these features. We’ll address that soon, but first, let’s talk about the marginal distributions themselves.

To simulate a univariate probability distribution, a simple and general method is to use inverse transform sampling. If (X(x)) is the probability distribution function, then we define the cumulative distribution function (F(x) = \int_{-\infty}^x X(t) dt). Most importantly, the inverse CDF (F^{-1}(x)) (a.k.a quantile function) takes a uniformly distributed random variable in the range ([0, 1]) and transforms it to a variable with PDF (X).

Not all distributions have a guaranteed closed-form inverse CDF. In the worst case, we can numerically integrate (X) and construct a table of values to approximate the inverse CDF. We can even let the user draw their own PDFs, although that strikes me as excessive. Personally, I value simplicity of implementation, so my preferred approach is to select probability distributions that already have a nice closed-form inverse CDF and supply some parameters that let us mess with the shape.

I like bounded distributions because it’s easier to control wacky values. Out of established bounded distributions with at least one shape parameter, the Kumaraswamy distribution

[X_K(x) = a b x^{a - 1} (1 - x^a)^{b - 1}]

has two shape parameters (a,b > 0) and a closed form inverse CDF:

[F_K^{-1}(x) = (1 - (1 - x)^{1 / b})^{1 / a}]

As for what the shape parameters actually do, it’s a bit hard to describe, but here’s a plot of some PDFs:

Another bounded distribution with a nice inverse CDF is the continuous Bernoulli distribution:

[X_C(x) = \lambda^x (1 - \lambda)^{1 - x} N(\lambda)]

with inverse CDF

[F_C^{-1}(x) = X_C^{-1}(-2 x \tanh^{-1}(1 - 2 \lambda) - (\lambda - 1) N(\lambda))]

where (X_C^{-1} = -\frac{\log [-x / ((\lambda - 1) N(\lambda)) ]}{\log (1 - \lambda) - \log \lambda}) and (N(\lambda) = \frac{2\tanh^{-1}(1 - 2\lambda)}{1 - 2 \lambda}), if WolframAlpha has not failed me. (0 < \lambda < 1) is the shape parameter. The shape parameter is easier to describe here, it just tilts the PDF:

Both these distributions are defined for (x \in (0, 1)) and must be scaled to the appropriate range.

It’s tempting to use a single highly general distribution with many shape parameters, but I’d recommend having multiple types of distributions that do one thing well with a small number of parameters. This gives us the benefits of the switching principle.

Introducing correlation with linear transformations

The simplest way to add correlation is to use a linear transformation: sample an independent random variable (\mathbf{x}), then put it through a matrix (\mathbf{x}’ = \mathbf{A}\mathbf{x}).

That’s it, we have correlation. If you imagine the x-axis as time and the y-axis as frequency, this is an upward sweep. I’ve gotten good results just randomizing the matrix (\mathbf{A}) with a uniform distribution, and using tanh to bias it a bit away from zero.

(In the special case where all variables have independent standard normal distributions, the correlation matrix is given as (\mathbf{K} = \mathbf{A}\mathbf{A}^\top), and you can actually recover (\mathbf{A}) from (\mathbf{K}) using the Cholesky decomposition. While the math of correlation in multivariate distributions is quite deep, in practice we don’t need any of that to make interesting granular synths.)

If you don’t introduce any correlation between variables but do select interesting probability distributions, the result doesn’t necessarily sound bad either:

Here’s the audio from the top of the post again, for comparison:

Both samples use the same DSP (windowed hard-synced sine waves), the only difference is the selection of multivariate distributions. Chalk it up to confirmation bias, but to me the second sample is livelier and has a stronger sense of motion and gesture.

Mixture distributions

A fun thing to do is to use marginal PDFs that have multiple local minima and maxima. Here’s an independent 2D distribution:

There aren’t really any established parametric distributions that are capable of even these simple shapes, but it is possible to combine any number of distributions using a mixture. Mixture distributions are defined by a finite number of component distributions (D_1, D_2, \ldots, D_n) and a number of mixing weights (w_1, w_2, \ldots, w_n > 0) that all sum to one. To draw a single sample from a mixture distribution, pick a number (1 \leq k \leq n) with probability (w_k), and sample from distribution (D_k).

Kernel smoothing

Mixture models combine multiple probability distributions by randomly choosing between them for each sample. An alternative approach is to instead add together two or more independent random variables, possibly with weights. A well-known result of probability theory is that the addition of random variables results in convolution of their PDFs. If one of the PDFs is normal or roughly normal-shaped, it acts as a lowpass filter and diffuses the PDF. This is known as kernel smoothing.

Nonlinear warping

It’s also possible to warp the data nonlinearly with whatever crazy functions you want. Here’s what I got throwing some random functions together including a Möbius transformation and a Cartesian-to-polar conversion for some reason:

As fun as this looks, this kind of visual is of limited use for higher-dimensional PDFs, and the sound is more important anyway. Given that the number of possible formulas is basically unlimited, algorithmic methods are advisable for navigating this space.

Conclusion

This is another synthesis post where I’ve put in the minimal work to get to sounds that I find interesting, but made very little effort to design an intuitive interface. (That’s generally how things go for this blog: research first, development eventually.) As I said, this method is well suited to randomization — the sort of thing where you keep changing the seed until you get a result you like. How to produce an interface is a tough question, because there are so many parameters and it’s hard to predict what settings are musical.

During the creation of this post, I researched standard methods for simulating multivariate normal distributions and landed on something called a Gaussian copula. An earlier and much longer version of this post involved a detailed explainer on how Gaussian copulae work, but when I actually listened to their effects, it became clear that there wasn’t any particular reason to bring them into the picture. (Interestingly, Gaussian copulae, or misuse of them, are said to have played a role in the 2008 financial crisis.) Ad hoc approaches to designing multivarite PDFs not only require less effort, but sound better. Lesson learned: before spending multiple days writing about something, it’s best to implement and listen to it first.

A Preliminary Theory of Sound Design

Nathan Ho — Wed, 24 May 2023 00:00:00 +0000

This post is my attempt at explaining my own philosophy of sound design. It’s not in final form, and subject to amendment in the future.

The type of sound design I refer to is specific to my own practice: the creation of sounds for electronic music, especially experimental music, and especially music produced using pure synthesis as opposed to recorded or sampled sound. These ideas presented might have broader applications, but I have no delusions that they’re in any way universal.

The theory I expound on isn’t in the form of constraints or value judgements, but rather a set of traits. Some are general, some specific, and my hope is that considering how a sound or a piece relates to these traits will take me (and possibly you) in some directions not otherwise considered. Some of the traits contain assertions like “if your sound does X, your audience will feel Y,” which in a composition may be employed directly, carefully avoided, or deconstructed. You’ll also note that the theory is concerned mostly with the final product and its impact on the listener, not so much the compositional or technical process. (Friends and collaborators are well aware that I use highly idiosyncratic and constrained processes, but those are less about creating music and more about “creating creating music.”)

No theory of sound design will replace actually working on sound design. Sound design isn’t a spectator sport, nor a cerebral exercise, and it has to be practiced regularly like a musical instrument. Reading this post alone is unlikely to make you a better sound designer, but if it’s a useful supplement to time spent in the studio, I’d consider this work of writing a success.

I will deliberately avoid talking about the topics of melody, harmony, counterpoint, consonance vs. dissonance, and tuning systems, and I’ll only talk about rhythm abstractly. There are many existing resources dedicated to these topics in a wide variety of musical cultures.

Transients

It is hard to overstate the perceptual importance of the initial transient of a sound object. A tiny click or burst of noise added to an attack, or a fade in, can make a huge difference.

Some questions to ask about a sound include: how sharp is the attack? Are there multiple discrete transients? Do parameters such as timbre, pitch, etc. change at the transient? Which direction? How fast?

Contrast

Contrast refers to not just differences in musical parameters such as pitch, register, brightness, transient, modulation speed and depth, etc. but also how quickly these things change.

Lack of contrast will put the audience into a more passive state of listening. Changes that are very frequent and/or similar to each other will be perceived as a texture. In between these two extremes, constrast will catch the listener’s ear.

Regularity

An obvious axis of regularity is steady tempo vs. random pulses. In modular and modular-adjacent environments (especially text-based languages) it’s tempting to think of regularity as a dichotomy between grid-based and completely random rhythms, but there is a whole world in between – slight humanization, acceleration and deceleration, steady rhythms with interruptions, etc.

Regularity also happens at the microsound level; silencing every other pitch period of an oscillator evokes a radically different effect from randomly silencing each period at a 50% probability.

Directionality

While contrast refers to all forms of change, directionality is about reaching a goal. Reverse cymbals and risers leading up to a downbeat are classic examples.

Directionality can even happen in the absence of change. In music with a steady pulse, our ears are conditioned to anticipate meter and hypermeter. For many listeners, a 4/4 beat with 4-bar groupings will suggest a new section at the end of 16 bars, even if there isn’t particularly strong change.

An offshoot of directionality is axiality – motion in opposite directions, either simultaneously or sequentially. (I borrowed this high-falutin’ term from Brian Ferneyhough.) Note that sequential axiality need not be continuous or shaped like a triangle wave; there can be a sudden jump along with the reversal of direction (e.g. 4 3 2 1, then 4 5 6 7).

Failing to properly command directionality is a perennial issue in my own music, especially when I use algorithmic techniques. If you’ve ever written a patch and wondered, “but how do I turn it into a piece with a beginning, middle, and end?” you may be having directionality issues. However, this isn’t advice, and I recommend speaking to a licensed clinical musicologist.

Space

This one’s fairly obvious, but I will point out that there are more ways to evoke space than just reverb in post. There’s also long decay times, frequency coloration, wide multichannel image, amplitude modulation of individual partials, complex resonances that mimic those of physical instruments, and chorus-like detuning. Reverbs themselves have tramendous variety outside of the standard modulated Lexicon/Alesis sound, with many interesting “bad” reverbs, both acoustic and simulated.

Vocals

The human brain is absurdly good at speech processing, and even vague and crude imitations of speech can instantly grab the ear. Vocal formants automatically stand out even when not specifically emphasized, and if they fail to sound natural or human, they can stand out even more.

Fidelity

Fidelity is more culturally bound than most of the traits here, but so many forms of reduced audio quality are instantly recognizable. Examples of reduced fidelity include waveshaping distortion, sharp filters, noise, vinyl crackle, wow and flutter, aliasing, bitcrushing, MP3 chirps, and dynamic range compression. Not all low-fidelity effects are downstream – techniques such as chiptune arpeggios, timbres considered cheesy or cliche, etc. can recall older eras of music and technology as well.

Verticality

This belongs to a cluster of terms like “arrangement,” “layering,” etc. but I ended up with “verticality” because it’s more specific and concrete than those. Verticality is a measure of change that happens simultaneously across all layers as opposed to individual parts.

If a piece mostly comprises changes to individual layers, a sudden global change will attract the listener’s attention. Steve Reich’s mid-era pieces, starting with “Music for Mallet Instruments, Voices, and Organ,” often feature evolving repetition with a small number of sudden, global changes.

Disorganized conclusion

An influence here is Disney’s 12 Principles of Animation. The 12 Principles are a great read for any electronic musician, as many of them apply beautifully to the design of modulation signals, a topic I plan on writing about more eventually.

For synthesis, getting past the stage of merely good patches and progressing to thinking about mid- and high-level traits like these is a sort of “coming of age” for sound designers. I feel that only recently (past year or so) something clicked for me and I went from designing patches to composing music. The technical process of synthesis has gradually felt more automatic and I can concentrate more on what I’m trying to say with the sounds. Writing this little manifesto is a fun exercise to clarify my thinking – if writing is your thing I encourage you to try writing your own theory of electronic music. Send it to me, also.

Audio Texture Resynthesis

Nathan Ho — Tue, 25 Apr 2023 00:00:00 +0000

Left: spectrogram of a child singing. Right: spectrogram of resynthesized audio.

Background

I was alerted to audio texture resynthesis methods by a student of mine who was interested in the collaborative work of researcher Vincent Lostanlen, musician Florian Hecker, and several others [Lostanlen2019] [Lostanlen2021] [Andén2019] [Muradeli2022]. Their efforts are built on an analysis method called “Joint Time-Frequency Scattering” (JTFS) based on the Continuous Wavelet Transform. In an attempt to understand the work better, I binged a wavelet transform textbook,[1] implemented a simplified version of JTFS-based resynthesis, and and briefly exchanged emails with Lostanlen. His helpful answers gave me the impression is that while JTFS is a powerful analysis technique, resynthesis was more of a side project and there are ways to accomplish similar effects that are more efficient and easier to code without compromising too much on musicality.

Audio texture resynthesis has some history in computer music literature [Schwartz2010], and some researchers have used resynthesis to help understand how the human brain processes audio [McDermott2011].

After some experimentation with these methods, I found that it’s not too hard to build a simple audio texture resynthesizer that exhibits clear musical potential. In this blog post, I’ll walk through a basic technique for making such a system yourself. There won’t be any novel research here, just a demonstration of a minimum viable resynthesizer and my ideas on how to expand on it.

Algorithm

The above-mentioned papers have used fancy techniques including the wavelet transform and auditory filter banks modeled after the human ear. However, I was able to get decent results with a standard STFT spectrogram, then using phase reconstruction to get time-domain audio samples. The full process looks like this:

Compute a magnitude spectrogram (S) of the time-domain input signal (x). A fairly high overlap is advised.
Compute any number of feature vectors (F_1(S),, F_2(S),, \ldots,, F_n(S)) and define their concatenation as (F(S)).
Initialize a randomized magnitude spectrogram (\hat{S}).
Use gradient descent on (\hat{S}) to minimize the error (E(\hat{S}) = ||F(S) - F(\hat{S})||) (using any norm such as the squared error).
Use phase reconstruction such as the Griffin-Lim algorithm on (\hat{S}) to produce a resynthesized signal (\hat{x}).

The cornerstone of making this algorithm work well is that we choose an (F(S)) that’s differentiable (or reasonably close). This means that the gradient (\nabla E) can be computed with automatic differentiation (classical backpropagation). As such, this algorithm is best implemented in a differentiable computing environment like PyTorch or Tensorflow.

The features (F(S)), as well as their relative weights, greatly affect the sound. If (F(S)) is highly time-dependent then the resynthesized signal will mimic the original in evolution. On the other hand, if (F(S)) does a lot of pooling across the time axis then the resynthesized signal will mostly ignore the large-scale structure of the input signal. I’m mostly interested in the latter case, where (F(S)) significantly “remixes” the input signal and disregards the overall structure of the original.

We will represent (S) as a 2D tensor where the first dimension is frequency and the second is time. As a matrix, each row is an FFT bin, and each column a frame.

If using a fancy alternative to the magnitude spectrogram such CWT or cochlear filter banks, you may have to do gradient descent all the way back to the time-domain samples (x). These analysis methods break down to linear frequency transforms that produce complex numbers followed by computing the absolute value of each bin, so differentiability is maintained.

Bin-specific statistics

A simple feature is to take the mean magnitude over time for each frequency bin. Using only this feature and no others, resynthesis produces noise colored according to the time-averaged spectrum of the original signal.

If we pool each frequency bin over time into a probability distribution, we can also use the variance, skewness, and kurtosis of each bin as features. It’s not hard to see that all such features are differentiable for all practical purposes.

Covariances

To capture relationships between bins, we can compile the covariances of each pair of rows in (S). This feature is really critical for helping the resynthesizer understand how partials correlate with each other in complex instrument timbres.

Spectral statistics

Another approach is to compute spectral features for each individual frame, and then average those features across frames. These features include spectral centroid (mean), variance, skewness, kurtosis, spectral flux, and spectral flatness[2]. They can be evaluated for the entire spectrum of each frame, for portions of the spectrum, or both. The time series produced by the spectral features can themselves be subject to more advanced statistical features than the mean, including covariances.

Modulation spectrogram

All features described up until now are invariant to any permutation of frames in (S). If we resynthesize audio using only such features, the results will sound like fluttery colored noise and lack any of the time evolution properties of the original signal.

In both Lostanlen’s JTFS and the older research of McDermott and Simoncelli, a common approach to capturing change over time is to use the “spectrogram of the spectrogram.” We can borrow from this concept by taking each row of (S), treat it as a signal, and compute its magnitude spectrogram. Compiling all these spectrograms produces a modulation spectrogram, which is a 3D tensor whose dimensions are audio frequency, modulation frequency, and time. The above statistical features that pool over time can then be applied to the modulation spectrogram.

The window size of the modulation spectrogram affects the largest scale of time evolution that the resynthesizer cares about.

JTFS has the additional property of capturing not just amplitude modulation but also frequency modulation, using a two-dimensional wavelet transform. Some kind of covariance feature on the modulation spectrogram might be able to emulate this, but due to the size of the modulation spectrogram it’s likely best to take a limited number of covariances (perhaps only on neighboring auditory frequency bins), but I have not tried this.

Weighting

Features will vary considerably in absolute values and bias the squared error function by arbitrary amounts. A simple way to normalize them is to divide each (F_k(\hat{S})) by (||F_k(S)||). Then, each (F_k) can be multiplied by a relative weight to emphasize or de-emphasize certain features.

Within multidimensional features, there is also the question of how to weight across their different values. In the case of the auditory spectrogram, weighting using an inverted equal-loudness curve makes sense. I found that weighting is particularly critical across modulation frequency bins in the modulation spectrograms, as without weighting fast amplitude modulations seem to be over-represented. Weights that are inversely proportional to modulation frequency seem to sound good, inspired by the somewhat tenuous hypothesis that natural and “human” phenomena broadly mimic (1/f) noise (pink noise).

Speeding up optimization

From Lostanlen’s dissertation [Lostanlen2017] we can glean a useful trick that greatly speeds up the gradient descent process. This is the bold driver heuristic, where the learning rate of the gradient descent optimizer is multiplied by 1.1 if the loss has decreased, and multiplied by 0.5 if the loss has increased. Using torch.optim.Adam, I found that this adaption of the learning rate was extremely helpful for improving speed of convergence.

Sound examples

Each sound is 5 seconds long and mono. The resynthesis was computed with 100 iterations of PyTorch’s Adam optimizer and each took about two minutes each to process on a standard laptop using CPU.

Singing, input:

Singing, resynthesized:

Drums, input:

Drums, resynthesized:

Some weird metallic artifacts are clearly audible in the result. It seems that these artifacts are caused by spectrogram regions that are nearly silent in the original audio. I have not yet been successful in finding a way to mitigate them.

Further directions

Tuning: Selecting the weights in a way that sounds right is a tricky question, and my approach to it so far has been far from systematic. As a start, I’d suggest a stochastic hill climb-like approach mediated by listening tests: present a reference resynthesis, then three variants with some weights randomly altered. Have an expert listener pick their favorite, then repeat using that as a reference. In a productized version of this resynthesizer, I imagine that some of these weights should be exposed to the user, which makes this partially an interface design problem.

More features: Off the top of my head, here are a few ideas: inharmonicity, chroma vector (pitch class profile), autocorrelation function of time-series features to detect periodicity.

Attack/release distinction: All features discussed so far are invariant if the entire spectrogram is reversed. In contrast, the human perception of sound is far from agnostic to time reversal – the transient is arguably the most important part of any sound object. Analysis features of time-series data could be added that distinguish attack and release and make the resynthesized audio less slushy.

Other transforms: Mel-spectrograms, MFCCs, wavelets, and auditory filter banks serve as alternatives to the standard magnitude spectrogram. A multiresolution STFT might not be a bad idea either, as it borrows the advantages of wavelets but is easier to code up. (One DSP expert told me that he considers wavelets to be overkill for audio analysis.) However, depending on the transform there is the tricky question of how to recover a time-domain signal. The brute force solution is to perform gradient descent on the time-domain samples.

Multichannel: Thinking about how to generalize this to stereo or multichannel frightens me. On top of features for each channel, you’ll want correlations between features on different channels, but the most serious issue is that phase coherence between channels is not guaranteed by standard Griffin-Lim. If you just want a stereo texture then I’m sure separate synthesis for mid and sides will sound fine, but if you want to accurately mimic the locations of sound sources in the original audio, good luck.

Morphing: It’s possible to interpolate between the features of two analyzed spectrograms, but I suspect this wouldn’t sound too interesting in the absence of a Dynamic Frequency Warping-like interpolator on frequency-sensitive features.

Conclusions

Code is on GitHub. It was written in a single day, so don’t roast me.

More tuning and research is necessary before I’d be willing to package this as a serious product for other people. Additionally, the “singing” sample sounds surprisingly recognizable, but it doesn’t sound like much of an improvement over simple granular synthesis. Still, given that I was able to achieve these results with a fairly straightforward feature set, I have confidence this general paradigm for audio resynthesis is quite powerful.

Thanks to dietcv, Vincent Lostanlen, and Julius Smith for helping with this project.

References

Footnotes

Andén2019
Andén, Joakim; Lostanlen, Vincent; and Mallat, Stéphane. 2019. “Joint Time-Frequency Scattering.”
Lostanlen2017
Lostanlen, Vincent. 2017. Convolutional Operators in the Time-frequency Domain.
Lostanlen2019
Lostanlen, Vincent and Hecker, Florian. 2019. “The Shape of RemiXXXes to Come: Audio texture resynthesis with time-frequency scattering.”
Lostanlen2021
Lostanlen, Vincent and Hecker, Florian. 2021. “Synopsis Seriation: A computer music piece made with time-frequency scattering and information geometry.”
McDermott2011
McDermott, Josh H. and Simoncelli, Eero P. 2011. “Sound Texture Perception via Statistics of the Auditory Periphery.”
Muradeli2022
Muradeli, John et al. 2022. “Differentiable Time-Frequency Scattering on the GPU.”
Schwartz2010
Schwartz, Diemo and Schnell, Norbert. “Descriptor-based Sound Texture Sampling.”

Wavelets have a reputation among DSP people as arcane and shrouded in mystery. It’s often rumored that they offer “better time-frequency resolution” than the windowed Fourier transform, but the advantages are not unilateral, and the best option depends on context. I hope to write an explainer on them on this blog at some point to help demystify wavelets for musicians, but that might be a while.
The standard formula for spectral flatness causes serious instability in a gradient descent context, so I modified it to (\text{flatness}(x) = \frac{\exp\left(\frac{1}{N} \sum_i \ln(1 + |x_i|) \right)}{1 + \frac{1}{N}\sum_i |x_i|}). Adding 1 prior to taking the logarithm prevents extremely large negative values, and adding 1 to the denominator prevents division by very small amounts. Also note the absolute values – real magnitude spectra are nonnegative-valued, but in the optimization process negative magnitudes can happen. The nondifferentiability of the absolute value doesn’t cause a problem for gradient descent in practice.

Opinionated Advice for SuperCollider Beginners

Nathan Ho — Thu, 16 Mar 2023 00:00:00 +0000

I’m often asked where to find the best resources for learning SuperCollider. My informed and professional answer, which comes from over a decade of experience with this software as a user, developer, and educator, is as follows:

I’m not really sure?

The landscape of SC learning materials has changed a lot since I started in 2012/13. I’ve heard good things about Eli Fieldsteel’s video tutorials and Bruno Ruviaro’s “A Gentle Introduction” ebook, so I guess check those out. I haven’t perused their tutorials at length, but Eli and Bruno are both seasoned professionals at SC education, so you can’t really go wrong there. Also, I have my own SuperCollider Tips blog post (which I have just updated today), which is not a structured tutorial but addresses common beginner problems. But ultimately, the most efficient learning strategy really depends on what you intend to make, because the applications for SC are so diverse, as are the backgrounds of users coming to SC.

That’s probably not too enlightening. However, I do have a lot of advice for beginning SC users, drawing from my own tortuous path through learning to make electronic music.

What follows are a bunch of guidelines that I recommend beginning SC users follow, or at least consider. A good number of them are specific to my idiosyncratic approach to SC, which I’ve expounded on in my YouTube presence: heavy focus on sound design through synthesis, and only sporadic use of hardware and real-time interaction. Also, although some of my advice may help live coders, I’m pretty uninterested in live coding myself, so I can’t give specific advice there (creative coding I’m all for, but when I share my screen it’s for education, not seamless performance art).

Patterns and Ndef are optional

I haven’t used Patterns in years. If they don’t appeal to you for whatever reason then you can always use Routine and Synth. Watch out for OSC pre-emption, which is done automatically in Patterns but not in Synths. You must wrap certain Synth-related calls in Server.default.bind { ... } to ensure accurate timing in a sequenced piece. I wrote a mini-article about this.

I’m often asked why I don’t use Patterns. I expect to eventually write a blog post that answers that at length, although I have to figure out how not to go into a rage-filled diatribe about the default Event type.

Ndef, ProxySpace, etc. are collectively known as JITLib. As they are primarily built as a live coding environment, for non-live coders their advantages are not particularly strong over just using Synth.

Don’t install tons of quarks or third-party plugins

I wrote a post about this called “Uninstall your quarks.” I don’t even have sc3-plugins installed anymore these days. The constraint of doing everything in vanilla SC is weirdly liberating. Your mileage may vary.

Find non-SC production tutorials and translate them to SC

This is the single biggest thing that helped me reach a point where I was actually happy with my own SC music. The dance and hip hop production communities on YouTube have a wonderful collection of free learning resources. Even if you’re more interested in more experimental genres, the fundamentals of making drums, pads, keys, bells, etc. are critical to build up to more left-field sound design. Channels I recommend are: SeamlessR, Au5, Bunting, Mr. Bill, Ned Rush.

Gordon Reid’s “Synth Secrets” articles are often recommended. There’s useful stuff in there, but being magazine articles they aren’t accompanied by sound examples, which is problematic for learning sound design. I highly recommend sticking to learning resources where you can actually hear the instructor’s results.

Look up commercial audio products and try to clone them in SC

I keep up with cdm.link and a few “synthfluencer” YouTube channels (Sarah Belle Reid, Benn Jordan, Richard Devine, the VAPE) to learn about the latest hardware synths, Eurorack modules, VST plugins, etc. Cynically speaking, a lot of them are about showing off overpriced toys, but as a converted disciple of the ’Collider, you are permanently cured of Gear Acquisition Syndrome! If a particular piece of gear sounds interesting, then download the manual, watch a demo video, and see if you can remake parts of it in SC code. Don’t make a GUI, just replicate the sound. Even if you can’t get a perfect clone, it will certainly give you ideas for patches.

Practice coding from scratch

Because it’s so easy to copy-paste text, beginners often treasure and recycle snippets of SC code, written by themselves or others. In the particular case of sound design, I recommend against this because the physical process of writing and tuning patches is incredibly important for building fluency in SC. Become a SynthDef-making machine. Write so many SynthDefs that they become ephemeral and disposable, like patches on a hardware modular synth. If you constantly hold on to SynthDefs rather than writing new ones, you’re discouraging yourself from exploring, which I’d argue ultimately leads to stagnation as a sound designer.

If you are an experienced programmer, this may require a change in thinking. SC is not like a standard software engineering environment where code is written once to fulfill a utilitarian purpose (modulo maintenance and refactoring). Instead, SC is an instrument that has to be practiced, and the process is as important as the result.

“Practice” is the key word in this. Every few days since 2016, I do an SC practice session saved to a directory in Dropbox, named like 2023-03-16-electro-swing.scd. I don’t habitually revisit old patches unless I’m bored and want to wince a bit at how my sound design was five years ago.

If you’re new to programming, get honest feedback on your code from SC users

In the software industry, an ideal code review not only catches problems before production but also helps educate everyone involved. Self-taught programmers in particular can get into weird habits. Having an experienced programmer look over your code can permanently improve your abilities.

If you’re new to music production, get honest feedback from producers

The biggest reason I recomend this is mixing. The best way to learn to mix is to get personalized feedback from a professional audio engineer who appreciates the genre you’re working in. I’m not an accomplished mixing engineer myself, but I am blessed to have friends who will tell me that I need to cut 12 kHz and above by exactly 4.5 dB, and also that hi hat is garbage.

Also, if you intend to use conventional Western harmony and melody and you’re self-taught in that domain, I do recommend lessons with a composition or jazz piano teacher if you can afford them. Once I attended a concert by someone who played a Bach prelude on the piano with all the accidentals confidently wrong. Self-learning is great, but there are dangers too.

Don’t build ambitious software projects in SC

sclang and its class library are pretty janky, and my experiences building and maintaining larger projects in them have been less than ideal. For those reasons I have mostly stopped building major sclang projects for my own practice. I don’t always mind working on software engineering in place of music, but sclang as an ecosystem has major issues with error handling, error message readability, and debugging tools, and it can result in a lot of frustration.

Programmers coming to SC often get stuck in this mindset that they can only employ SC in a musical context if they start building a wrapper around SC so they can “properly” use it. This is like building a pianola so you can play the piano – it’s fine to do that, but it’s not actually making music. With some creativity, the raw SC interface is already perfectly ripe for artistic works, if you think of it as an instrument rather than an API. If you need examples of what such a process looks like, I use SC in exactly this way in my YouTube channel.

If you decide to break this guideline, proceed with caution. Keep your software rudimentary and only architect complex features if you’re 100% sure that they will pay off as better art. Your audience hears cool music, not cool software, so be conscious of diminishing returns. It’s also critical to psychologically separate your music from your software: maybe spend a week on maintenance, then freeze the code for a week just to make music, and repeat. You could also hire someone else to build or maintain the system for you – I’ve freelanced for artists who have done exactly that.

Also, don’t make the one big patch that does everything. That’s setting yourself up for failure. SuperCollider is the one big patch that does everything.

Conclusion

Good luck on your SC journey.

Negative Compression

Nathan Ho — Thu, 23 Feb 2023 00:00:00 +0000

One blog post I’ve been meaning to write for a while is a comprehensive review of the design of dynamic range compressors and limiters, both digital and analog. Textbook compressor designs can be easily found, but like reverbs there are lots of weird little tricks from both hardware and software designs that supposedly define the distinctive musical character of different compressors. It may be a while before I finish that post because, while I’ve read a lot about the DSP of compressors, I don’t feel yet qualified to write on design. I haven’t yet designed a compressor plugin that I’m happy with, nor done a lot of compressor wine tasting, and the musical and psychoacoustic aspects of compressors are to me at least as important as the signal math.

Nevertheless, there’s a weird corner of compressor design that I feel inspired to talk about, and it’s called negative compression. It’s a feature of a few commercial compressors; I’m not sure which was the first, but I first learned about the concept from Klanghelm DC1A. Negative comp is the source of considerable confusion – just watch the Gearspace pundits go at it.

The brief description is that a standard compressor, upon receiving a signal with increasing amplitude, will reach a point where the output amplitude will increase at a slower rate. If the compressor is a perfect limiter, the output amplitude will hit a hard limit and refuse to increase. A negative compressor takes it further – the output signal will eventually get quieter over time as the amplitude increases. If you feed a percussive signal into a negative compressor and drive it hard enough, it will punch a hole in the signal’s amplitude, and can split a transient in two. It can be a pretty bizarre effect, and seems underutilized.

This explanation should be enough for most, but you know this blog. We do the math here. In this post, I will explain the basic mathematics of compressors to demystify negative compression, propose variants of negative compressors, and demonstrate how to do negative compression in SuperCollider.

Compressor basics

Let’s first briefly define standard feedforward compression. A compressor requires two audio signals – an input signal to be attenuated, and a sidechain to control the attenuation. The sidechain is usually identical to the input signal, or is a filtered version of it to control response to different frequency ranges; if they are entirely different signals then it’s sidechain compression. The compressor operates by feeding the sidechain through an envelope follower to track its amplitude, and a function called the gain curve maps the amplitude to a gain control signal. The gain control signal is simply multiplied by the input signal to produce the output of the compressor. There are many variations, but pretty much all feedforward compressors are based on this foundation.

The gain curve is critical to the sound, and for proper compressor action the gain curve should decrease as the sidechain gets louder. Let (x \geq 0) be the amplitude detected by the envelope follower (peak, RMS, whatever – as long as it increases roughly linearly with the amplitude of the sidechain), and let (g(x)) be the gain curve.

If the compressor isn’t in sidechain mode, then it’s meaningful to talk about a closely related function that doesn’t have a standard name but I call the “response curve.” The response curve is defined as (f(x) = x g(x)), and approximately represents the output amplitude of an input signal with stable amplitude such as a sine wave. (In practice, the envelope follower usually makes the output signal an imperfect sine wave, making “amplitude” ambiguous and frequency-dependent, but this is a rough model.) The response curve is usually what is depicted in pictures of compressors, but importantly, the response curve only applies if we aren’t sidechaining.

A common way to describe the response curve is that (f(x) = x) if (x) is below a threshold amplitude (T) (in linear amplitude units), and above that threshold (f(x)) increases by 1 dB for every (R\ \text{dB}) that (x) increases. (R > 0) is the compression ratio, usually described as a ratio such as 4:1. Mathematically:

[\begin{aligned} f(x) = \begin{cases} x & \text{if } x < T \ T (x / T)^{1 / R} & \text{if } x \geq T \ \end{cases} \end{aligned}]

(f(x)) is often drawn using a log-log graph, and such graphs appear as piecewise linear because (\log_{10} f(10^x)) is. Although I introduced the ratio in terms of dB, there’s no specific dependency on the decibel scale in this formula.

As (R \rightarrow \infty) the response curve approaches (f(x) = \text{min}(x, T)), which is a limiter with a ratio of ∞:1. Higher ratios correspond to heavier compression.

The gain curve is computed as (g(x) = f(x) / x), which gives (g(x) = 1) for (x < T) and (g(x) = (x / T)^{1 / R - 1}) otherwise. While the response curve conventionally increases, the gain curve decreases past the threshold.

This textbook “log-log piecewise linear” model produces a perfectly acceptable compressor, but it’s not universal. It’s merely a visual simplification of how compressors work. Practical compressors – especially analog designs – can have arbitrary curves for (f(x)), and while they might offer controls labeled “threshold” and “ratio,” these are more romantically defined and based on fitting the piecewise model to the curve.

Negative compression

Negative compression happens when (f(x)) is nonmonotonic, such as decreasing above the threshold. In the textbook model, this is the natural extension of flattening the compressor’s curve so it bends back on itself. If we define the unitless “log-log slope” above the threshold as (S = 1 / R), normal compressor operation is (0 < S < 1), (S = 0) is a limiter, and (S < 0) is negative compression.

In terms of ratio, normal compressors are (R > 1), (R \rightarrow \infty) approaches a limiter, and for negative compression, (R) wraps around back to negative values, (R < 0). Counterintuitively, negative ratios are “beyond” a positive infinite ratio, and -4:1 is “more compressed” than ∞:1; this is a consequence of ratio being the reciprocal of the slope. (The region (0 < R < 1), corresponding to compression ratios like 0.5:1, is not negative compression but rather expansion or noise gating.) This trips people up when discussing negative compression because they’re used to thinking in ratios, when slope more naturally extends to negative numbers.

Like I said, a negative compressor can result in an output decrease in amplitude for an input increase in amplitude, creating a “vacuum” effect on transients. But this is only true if we aren’t doing sidechain compression. Remember when I said that the response curve only makes sense if the sidechain signal is the same as the input signal? For such cases, it’s more useful to look at (g) than (f). Recalling that (g(x) = (x / T)^{1 / R - 1}) above the threshold, we can see that regardless of whether (R < 0) or (R > 1), the gain curve is always nonincreasing because the exponent (1 / R - 1) is negative. It turns out that sidechain compression with a negative ratio is just extra-dramatic sidechain compression, and isn’t all that special compared to positive ratios.

Variants

Soft knee: Negative compression can be done with either a hard knee or a soft knee. The soft knee becomes less and less audible as other settings become more dramatic. Aside from softening the response curve, the gain control signal can also be lowpass filtered prior to multiplication with the input signal. This is a non-textbook trick found in some commercial designs (I first saw it in an analog schematic).

Zigzag compression: One natural extension is to have a zigzagging response curve with multiple critical points. Ordinary compression is to zigzag compression as saturation is to wavefolding. Reducing the attack and decay times of a compressor makes it behave more and more like a waveshaper, so zigzag compression is in a sense a generalization of wavefolding.

Multiband negative compression: self-explanatory.

Implementation in SuperCollider

I will take a moment to rant: don’t use Compander. It downsamples the gain control signal to control rate to save CPU usage, and although it does linear interpolation, the aliasing artifacts are pretty audible to me at a standard block size of 64, and very audible if I try to set fast attack and release.

Luckily, building a feedforward compressor is easy with Amplitude, as long as we remember to use Amplitude.ar and not Amplitude.kr. Here’s my implementation:

// Public Domain, Creative Commons Zero

(
var compGain;
compGain = { |in, threshold, slope, attack = 0.05, release = 0.3|
    var inArray, amplitude;
    inArray = if(in.isArray) { in } { [in] };
    amplitude = (Amplitude.ar(inArray, attack, release).sum / inArray.size.sqrt).max(-100.dbamp).ampdb;
    ((amplitude - threshold).max(0) * (slope - 1)).lag(attack).dbamp;
};

{
    var snd;
    snd = SinOsc.ar(440) * Env.perc(0.01, 1.0).ar;
    snd = snd * compGain.(snd, threshold: -10, slope: -2, attack: 0.1, release: 0.3);
    (snd * -6.dbamp) ! 2;
}.play(fadeTime: 0);
)

threshold is in dBFS, and slope is (S). Again, slope is the reciprocal of ratio, so the setting slope: 1 / 4 corresponds to a 4:1 compressor, slope: 0 is a limiter, and slope: -1 / 5 is a -5:1 ratio.

You may specify either a single channel or an array of signals, in which case individual envelope followers are applied to each channel and the amplitudes are combined under the assumption of equal power panning. This ensures that the different channels are ducked by equal amounts, preserving the spatial image. This is often called “stereo link” in commercial units. If you’d rather compress each channel individually (as in “dual mono”), invoke compGain individually for each channel: snd.collect { |channel| channel * compGain.(channel, ...) }.

The .lag(attack) method call does post-smoothing of the gain control as mentioned before. I like it because it maintains a more organic sound even for more dramatic negative comp settings. Feel free to remove it.

Conclusion

Thank you for reading my conclusion.

Interstate Hydra - Nocturnes

Nathan Ho — Mon, 16 Jan 2023 00:00:00 +0000

Exactly three years ago I made an ambient EP under the name “Interstate Hydra” and put it up on Bandcamp. Up until now I have kept this alias a secret and only shown these tunes to a few people, but I figured it’s been long enough.

Nocturnes_ was made entirely in Audacity using the “sound dumplings” method described in an earlier post. This was a total 180 from my usual workflow, which is to synthesize everything with SuperCollider code.

“For Ellie” is a single released months after Nocturnes, and is formally the simplest Interstate Hydra track, comprising a row of sound dumplings in arch form.

I don’t plan on returning to the Interstate Hydra alias for quite some time, but I hope you enjoy these tracks. Thank you for listening.

A Closer Look at Clarence Barlow's ISIS

Nathan Ho — Sun, 15 Jan 2023 00:00:00 +0000

In 2005, Clarence Barlow published a paper on Intra-Samplar Interpolating Sinusoids (ISIS), an audio analysis-synthesis algorithm. It did not make much of a splash, with the paper having only 7 citations in Google Scholar. Nevertheless, it produces some interesting sounds, so let’s dive into it.

First, some context. The precursor to ISIS is a technique Barlow calls spectastics. In this method, the short-time Fourier transform is computed of an audio signal, and at each frame, the magnitude spectrum is resampled logarithmically to 12EDO and used as a probability distribution to select a pitch. The pitch sequence forms an extremely rapid melody, which can be synthesized or played on a robotic instrument. Barlow describes the spectasized melody as “remarkably like the original sound recording.”

In ISIS, this concept is taken to an extreme by making the “melody” a constant amplitude sine wave whose frequency is changed every sample. Given a digital signal that doesn’t exceed ±1, we can interpolate between any two successive samples with a partial cycle of a sine wave. An image helps here; the dots show the sampled digital signal.

For example, if the samples are 0 and 1, as seen in the first two samples in the image, we can interpolate with a quarter sine wave with a period of 4 samples and a frequency of 1/4th the sample rate. There are actually infinitely many ways to do this interpolation. For example, from 0 to 1 we can also have a sine wave that completes 5/4ths of a cycle. We can even assume that the initial phase of the sine wave is (\pi) and the final phase (5\pi/2), or have the sine wave going backwards with phase ramping from 0 to (-3\pi/2).

To resolve the ambiguity, ISIS restricts the frequency to be always nonnegative so the phase never goes backward, and always assumes the phase at every sample point is in the range ([-\pi/2, \pi/2]) modulo (2\pi). There are other approaches, such as picking the minimum possible nonnegative frequency or the frequency of minimum absolute value, which may produce interesting alternative sounds. I won’t get into these (this is a relatively low-effort post), but feel free to try them out.

Definition

Barlow’s formulas actually have a mistake, taking the logarithm of a signal that can range into negative values. The signal first needs to have a modulo 1 operation applied to it. Second, converting the frequency signal to logarithmic isn’t even necessary in the first place. To correct for these issues, the presentation of the algorithm is modified here for clarity, but the sounds should be faithful to the original. (Also, what’s up with the weird symbol he uses for sample rate? Looks like a combination of the letters “S” and “R”? What’s wrong with (f_s)?)

ISIS takes a signal (|x[t]| \leq 1). The analysis process is

[y[t] = \text{frac}\left(\frac{\arcsin(x[t]) - \arcsin(x[t - 1])}{2\pi}\right)]

and resynthesis is

[\begin{aligned} z[t] &= \text{frac}(z[t - 1] + y[t]) \ x[t] &= \sin(2\pi z[t]) \ \end{aligned}]

where (\text{frac}(x) = \text{mod}(x,,1) = x - \lfloor x \rfloor) where (\lfloor x \rfloor) rounds to negative infinity.

The frac operation in the second equation is not theoretically necessary due to the periodicity of the sine function, but (z[t]) will lose floating-point precision if allowed to approach infinity, and degradation will be plainly audible after a few seconds. Which, I mean, hey, if you’re into that, sure?

I was first learning to program SuperCollider UGens in C++ when I found out about ISIS, and promptly made it into two UGens. I now realize that this algorithm can be written with existing UGens in just three lines:

(
{
    var sig;
    sig = /* Define input signal here */;
    sig = sig.asin;
    sig = ((sig - Delay1.ar(sig)) / 2pi) % 1;
    /* Process analyzed signal here */
    sig = (Phasor.ar(DC.ar(0), sig, 0.0, 1.0) * 2pi).sin;
    sig ! 2;
}.play(fadeTime: 0);
)

The above code snippet is in the Public Domain, Creative Commons Zero, Guaranteed Lawyer-Free. Note the use of Phasor instead of Integrator; Phasor without an input trigger is like an Integrator with an inbuilt modulo. If we were to use Integrator followed by a modulo, we would run into the precision problem described above.

ISIS effects

As long as the input signal does not peak outside ±1, ISIS is lossless; it achieves perfect reconstruction for all signals. This can be verified easily from the above equations. Of course, the fun part is messing with the analyzed signal prior to resynthesis.

Pitch shifting

(y[t]) is thought of as a frequency signal, so our first thought might be to multiply it by a constant (k), as an abstract sort of “pitch shifter.” Let’s try pitch shifting a 100 Hz sine wave and increasing (k) from 1 to 2:

Curiously, if (k) is an odd integer, the ISIS pitch shifter actually produces a sine wave whose frequency is multiplied by (k). For all other (k), the effect sounds nothing like a standard pitch shifter, and more like a really weird distortion unit.

In general, if the input signal is periodic, the output signal of a pitch shifter will roughly retain that periodicity. It may be interesting as a guitar pedal.

Values of (k) extremely close to 1 such as 1.001 seem to produce rapid amplitude modulation effects. I tried applying ISIS pitch shifting to complex polyphonic signals, but pitch shifting turned them into a wall of noise!

Frequency shifting

Frequency shifting is accomplished by adding an offset to (y[t]). Note that adding 1 does nothing due to the frac function. Here’s a 100 Hz sine wave with an offset gradually increasing from 1 to 2:

Filtering

The effect of lowpass filtering (y) is hard to describe. Here’s a 100 Hz sine wave with a lowpass on the frequency signal opening from 100 Hz to 16000 Hz:

The lowpass seems to have the opposite effect one might expect. Lower cutoffs produce more high-end fizz, and higher cutoffs are more muted. Go figure.

The frequency signal has a range of 0 to 1, so when highpass filtering (y), it makes the most sense to first subtract 0.5 from the signal, perform the highpass, then add 0.5 back in. Here’s a 100 Hz sine wave with a highpass sweeping from 1 Hz to 16000 Hz:

FM

There are two ways to do FM here. Linear FM adds an oscillator to the frequency signal, while exponential FM multiplies by one. Below are 100 Hz sine waves with randomly modulated linear and exponential FM, respectively:

ISIS as a filter

Although presented as an analysis/resynthesis technique, ISIS is really a chain of stateless nonlinearities and standard linear filters. (As they say, everything is a filter.) In particular, the chain is:

arcsine nonlinearity → differentiator → wrap nonlinearity → [arbitrary processing] → integrator → sine nonlinearity

The arcsine and sine nonlinearities cancel each other out, as do the differentiator and integrator. The wrap nonlinearity is the odd one out, as it doesn’t have a corresponding inverse operation. If the analyzed signal is passed through unharmed, the wrap does nothing due to the periodicity of the sine function, but if the analyzed signal is modified, the wrap has a pretty big impact on the sound. It is possible to run ISIS without the wrap nonlinearity, but the effects above are a lot less interesting by comparison.

We can try to gain a better understanding of ISIS by imagining what happens with small signals close to 0. The arcsine becomes linear, the differentiator cuts dc and low frequencies with a +6 dB/octave slope, and the wrap nonlinearity effectively inverts, amplifies, and clips the signal into a 1-bit pulse wave. The integrator boosts the lows again with a -6 dB/octave slope, turning the pulse wave into a triangle wave. Finally, the sine nonlinearity acts as a wavefolder.

With this view in terms of traditional filters in mind, some of the transformations described above begin to make more intuitive sense. Multiplication of the frequency signal corresponds to driving the wavefolder harder. Addition by a constant turns into a ramp when integrated, so it’s identical to adding a saw wave to the signal prior to wavefolding. Linear FM is simply adding a signal in prior to the nonlinearity, and exponential FM is effectively amplitude modulation. I don’t have a good analysis for the effects of EQ filters, however.

Conclusion

Please don’t put me on a list.

An Approach to Sound Synthesis with L-Systems

Nathan Ho — Sun, 01 Jan 2023 00:00:00 +0000

An early obsession of mine when I was first learning about music tech was a paper by Stelios Manousakis titled “Non-Standard Sound Synthesis with L-Systems,” published 2009 in Leonardo Music Journal. At the time I was a snob about making strictly “academic” electronic music (think Stockhausen or Xenakis) and viewed things like FM and subtractive as too plebian, so synthesis methods that branded themselves as “non-standard” were very alluring. If I could give my younger self some advice, I’d tell him that you have to build foundations as a sound designer first before you can get to the crazy experimental stuff. How are you going to make a cool alien soundscape from another dimension if you can’t make a nice snare from scratch? Every good post-tonal composer understands tonality, if only to know how to avoid it. You have to know what rules you’re breaking.

With more artistic maturity under my belt, I revisited Manousakis’ paper recently and still found it interesting, so I decided to do some riffing on it. In this post, I’ll quickly explain what L-systems are (there are lots of better explanations online) and walk through a complete, “minimum viable” system for generating sound with them. Sound examples are embedded.

L-systems and turtle graphics

Briefly, L-systems (short for “Lindenmayer systems”) generate a sequence of words (strings) using rewriting rules applied repeatedly to an initial word called the axiom. A classic L-system is given by the rules a → ab and b → a with the axiom a. At each iteration, each symbol in the word is replaced by the right-hand side of its corresponding rule. The first few iterations of this L-system are:

a
ab
aba
abaab
abaababa
abaababaabaab
abaababaabaababaababa

Most L-systems that are objects of study undergo approximately exponential growth in this way. L-systems can produce simple patterns like abababab as well as aperiodic results like the above, generating apparent randomness while being completely deterministic.

A popular use of L-systems is to apply them to turtle graphics. To do so, each symbol in the L-system corresponds to an instruction to a turtle in 2D space, such as “move forward one unit” or “turn left 30 degrees.” Following the instructions sequentially from one string and plotting the turtle’s trajectory in space produces a drawing. Organic-looking fractals may result.

In “Non-Standard Sound Synthesis with L-Systems” as well as his more detailed 2006 master’s thesis Musical L-Systems, Manousakis proposes a lot of different possibilites to explore, but the basic idea is that he’s sonifying turtle graphics produced by L-systems and mapping the turtle’s state over time (position, orientation, etc.) to synthesis parameters. These L-systems operate at different time scales, from microsound to the form of an entire piece.

L-systems and microsound

At first impression, this all may sound rather boring, as one can take any synthesis algorithm and use turtle graphics to modulate its parameters over time. As there is a separation of concerns between the L-system and the synthesis method, calling this “nonstandard synthesis” may be an overstatement of its novelty. You could argue that the L-system with turtle graphics are effectively a random LFO generator.

The key to averting this criticism is to use an L-system deeply integrated into the synthesis algorithm using microsound techniques. Take for example an L-system with symbols interpreted as follows:

p: output one cycle of a sine wave
.: output one cycle of silence
F: increase sine wave frequency
f: decrease sine wave frequency
+: increase amplitude
-: decrease amplitude

Depending on the L-system’s rules and axiom, a variety of sounds may result. A long uninterrupted sequence of p’s will generate a static tone, patterns like pFpFpF and pfpfpf will produce sweeps and chirps, and patterns like p.p.p. can even output distinct timbres by periodically alternating between pulses of audio and silence.

There are a few issues with this toy example that make it unmusical. First, the synthesis method is too simple, and it mostly produces wandering sine wave tones. Second, the symbols that decrease and increase frequency and amplitude can cause those parameters to rocket off to positive or negative infinity.

A functional L-system synthesizer

To upgrade the synthesis technique, let’s use windowed oscillator hard sync. In this synthesis method, there are two frequencies: the sync frequency and intrinsic frequency. A sine wave runs at the intrinsic frequency, and every 1 / sync frequency seconds, it has its phase reset to 0. The resulting waveform is multiplied by a signal that starts at 1, decreases to 0, and resets to 1 every 1 / sync frequency seconds. There are now three synthesis parameters to modulate: the two frequencies, and the amplitude. The new set of symbols is as follows:

p: output one cycle of a sine wave
.: output one cycle of silence
S: increase sync frequency
s: decrease sync frequency
I: increase intrinsic frequency
i: decrease intrinsic frequency
+: increase amplitude
-: decrease amplitude

To solve the out-of-range problem, one approach is to clamp the frequencies and amplitude to minimum and maximum values. However, this causes another problem: words that contain a lot of S’s and not many s’s will immediately max out the sync frequency, and this strikes me as unmusical. The idea I had is to fold over the frequencies and amplitudes in a sine wave shape, so if the word keeps outputting S’s repeatedly, the sync frequency will eventually start decreasing instead. In other words, the six symbols SsIi+- have the effect of stepping the phases of three random LFO’s forward or backward. LFO shapes other than sine waves are possible too.

We now have a solid system for interpreting the output of the L-system. The remaining work to make this a complete sound-generation system is to design the L-system itself. This is pretty easy, as it turns out that randomly generating rules works surprisingly well. To do this, we assign a word of 2-3 random symbols as the right-hand side of each rule, and voila, we have an L-system. As a special provision, the rule corresponding to the symbol . must generate at least one p to reduce the chance of long stretches of silence.

As a loose end, there are a few ways to approach generating the word iteratively. We can start with an axiom, iterate the L-system until the word is long enough, and synthesize the final word. Alternatively, we can iterate the L-system and synthesize the result of every generation, concatenating all the synthesized waveforms. The latter is somewhat more attractive to me as there’s a chance that the listener might hear the iterative process, although I’m not sure how much it actually matters. I also cut off each word to a maximum of 1500 symbols; it will become clear why I did this later.

We have completely described an approach to sound generation with L-systems and windowed oscillator sync. It has the following parameters:

minimum and maximum sync frequency
amount sync frequency is increased or decreased with the S and s symbols
minimum and maximum intrinsic frequency
amount intrinsic frequency is increased or decreased with the I and i symbols
amount amplitude is increased or decreased with the + and - symbols

So, how’s it sound? Here’s the result of randomizing the above parameters and generating some uncurated sounds:

The windowed oscillator sync algorithm and the sine wave LFOs very much imprint on the resulting sounds, possibly more so than the L-system. Still, we get a mix of obvious patterns and aperiodic noisy motions that clearly sonify the systematic way the symbols are produced. To me, the sounds are pretty interesting and many seem like they’d be fairly difficult to produce without L-systems, so I’d declare this experiment a modest success.

Mutating the rules

One problem is that the results get pretty repetitive and boring after around 10 seconds. For sounds that display long-term evolution, we can try changing the L-system’s rules over time. Every generation, we take each rule and randomly remove and add symbols. I’ve found that the evolution of the sound is highly sensitive to rule mutations (which is a good thing – it means the thing we’re sonifying is really impacting the sound), so it suffices to mutate each rule with only a 10% chance and add and delete only one symbol.

As the L-system process exponentially amplifies the number of symbols, it’s necessary to cut the words from each generation short to ensure that mutations happen in a timely manner. Otherwise the mutations will become sparser and sparser. The maximum word size becomes a parameter that determines how quickly the L-system evolves. I’ve set this to 1500.

Here are some results, again uncurated:

Assessment and further directions

I went with the simplest possible synthesis method because I wanted to get as close to “hearing the L-system” as possible while still being sort of musical. As such, the sounds I got are raw, digital, and shamelessly granular. I wouldn’t use them as a final piece, but they could serve as raw materials for sampling and adding post-effects.

To make this more musical, the first thing I’d try is a more sophisticated synthesis algorithm. Pulsar synthesis would be my top pick, as it is very general. I’m envisioning using a buffer of recorded sound as the source for the pulsarets, modulating the buffer position with turtle graphics, and blurring the line between sampling recognizable chunks (0.1 seconds) to extracting timbre as a wavetable (< 0.02 seconds). If a pure synthesis approach is preferred, having multiple symbols corresponding to multiple types of synthesized pulsarets could help with the diversity of output.

The sine wave LFOs are very distinctive and wear themselves out rather quickly. To reduce fatigue, it may be wise to expose the minimum and maximum values of those LFOs as user parameters (or targets for random modulation). Maybe there are better approaches to using symbols to modulate parameters. It is common in graphical L-systems to have symbols that execute small bits of conditional logic, which could be exploited to produce a wider variety of modulations that avoid the “rocketing” problem. However, as much as possible I prefer complexity to come from the L-system rather than its interpretation.

Rule mutation is a big deal. It upgrades this system from a static texture synthesizer to an algorithmic music generator that allows for a balance between coherence and change. If you wait long enough (or set mutation to happen fast enough), the system can arrive at a completely different place to where it was before. I imagine running this in real time and allowing the user to save and recall the L-system to create repetition and formal structure rather than just letting the thing do a random walk.

As-is, rules are generated and mutated in a mostly nondiscerning way. The rule generation system could be biased to produce more musical results. Alternatively, instead of using random generation, the rules could be exposed to the user for direct editing as a mode of expression.

There are many variations to L-systems, including stochastic L-systems and context-sensitive L-systems. One significant one is “branched” L-systems where, for instance, the symbol [ stores and pushes the current state of the turtle to a stack and the symbol ] pops and restores the state. These are common in graphical L-systems when creating fractals. Manousakis proposes an interpretation of branched L-systems as producing polyphony, with appropriate limits on the number of simultaneous voices.

I’m not yet clear on how to use L-systems to generate macro-level structure in a way that’s musically compelling and transparent with respect to the L-system’s behavior. To throw out an idea, the mutation process could itself be commandeered by an L-system.

Conclusions

Admittedly, the sound examples in this post demonstrate more potential than musicality. I’m fine with that. I don’t know if I’ll end up incorporating L-systems into a more serious artistic pursuits, but it’s good to know that they show promise.

TopMod, Blender, and Curved Handles

Nathan Ho — Tue, 06 Dec 2022 00:00:00 +0000

I’ve been working on a big project using Blender. More on that in a future post, but here’s a preview. The project has gotten me into a 3D art headspace, and I took a detour recently from this major project to work on a small one.

Blender isn’t my first exposure to 3D. When I was a kid, I played around a lot with POV-Ray – an outdated program even back when I was using it, but still a fun piece of history. I also found out about something called TopMod, which is an obscure research 3D modeling program. I was interested in it primarily because of Bathsheba Grossman, who used it in a series of lovely steel sculptures based on regular polyhedra.

The core idea of TopMod is that, unlike other 3D modeling programs, meshes are always valid 2-manifolds, lacking open edges and other anomalies like doubled faces. This is ensured with a data structure called the Doubly Linked Face List or DLFL. In practice, TopMod is really a quirky collection of miscellaneous modeling algorithms developed by Ergun Akleman’s grad students. These features give a distinctive look to the many sculptures and artworks made with TopMod. I identify the following as the most important:

Subdivision surfaces. TopMod implements the well-known Catmull-Clark subdivision surface algorithm, which rounds off the edges of a mesh. However, it also has a lesser known subsurf algorithm called Doo-Sabin. To my eyes, Doo-Sabin has a “mathematical” look compared to the more organic Catmull-Clark.

Rind modeling. This feature makes the mesh into a thin crust, and punches holes in that crust according to a selected set of faces.

Curved handles. In this tool, the user selects two faces of the mesh, and TopMod interpolates between those two polygons while creating a loop-like trajectory that connects them. The user also picks a representative vertex from each of the two polygons. Selecting different vertices allows adding a “twist” to the handle.

The combination of these three features, as pointed out by Akleman et al., allows creating a family of cool-looking sculptures in just a few steps:

Start with a base polyhedron, often a Platonic solid.
Add various handles.
Perform one iteration of Doo-Sabin.
Apply rind modeling, removing contiguous loops of quadrilateral faces.
Perform one or more iterations of Catmull-Clark or Doo-Sabin.

(I would like to highlight step 3 to point out that while Doo-Sabin and Catmull-Clark look similar to each other in a final, smooth mesh, they produce very different results if you start manipulating the individual polygons they produce, and the choice of Doo-Sabin is critical for the “TopMod look.”)

TopMod has other features, but this workflow and variants thereof are pretty much the reason people use TopMod. The program also has the benefit of being easy to learn and use.

The catch to all this is that, unfortunately, TopMod doesn’t seem to have much of a future. The GitHub has gone dormant and new features haven’t been added in a long time. Plus, it only seems to support Windows, and experienced users know it crashes a lot. It would be a shame if the artistic processes that TopMod pioneered were to die with the software, so I looked into ways of emulating the TopMod workflow in Blender. Let’s go feature by feature.

First we have subdivision surfaces. Blender’s Subdivision Surface modifier only supports Catmull-Clark (and a “Simple” mode that subdivides faces without actually smoothing out the mesh). However, a Doo-Sabin implementation is out there, and I can confirm that it works in Blender 3.3. An issue is that this Doo-Sabin implementation seems to produce duplicated vertices, so you have to go to edit mode and hit Mesh -> Merge -> By Distance, or you’ll get wacky results doing operations downstream. This may be fixable in the Doo-Sabin code if someone wants to take a stab at it. Also worth noting is that this implementation of Doo-Sabin is an operator, not a modifier, so it is destructive.

EDIT: Turns out, Doo-Sabin can be done without an addon, using Geometry Nodes! This StackExchange answer shows how, using the setup in the image below: a Subdivide Mesh (not a Subdivision Surface) followed by a Dual Mesh. The Geometry Nodes modifier can then be applied to start manipulating the individual polygons in the mesh.

Rind modeling can be accomplished by adding a Solidify modifier, entering face select mode, and simply removing the faces where you want holes punched. An advantage over TopMod is that modifiers are nondestructive, so you can create holes in a piecemeal fashion and see the effects interactively. To actually select the faces for rind modeling, TopMod has a tool to speed up the process called “Select Face Loop;” the equivalent in Blender’s edit mode is entering face select mode and holding down Alt while clicking an edge.

Curved handles have no equivalent in Blender. There is an unresolved question on the Blender StackExchange about it. To compensate for this, I spent the past few days making a new Blender addon called blender-handle. It’s a direct port of code in TopMod and is therefore under the same license as TopMod, GPL.

My tool is a little awkward to use – it requires you to select two vertices and then two faces in that order. TopMod, by comparison, requires only two well-placed clicks to create a handle. I’m open to suggestions from more experienced Blender users on how to improve the workflow. That said, this tool also has an advantage over TopMod’s equivalent: parameters can be adjusted with real-time feedback in the 3D view, instead of having to set all the parameters prior to handle creation as TopMod requires. The more immediate the feedback, the more expressive an artistic tool is.

Installation and usage instructions are available at the README. blender-handle is 100% robust software entirely devoid of bugs of any kind, and does not have a seemingly intermittent problem where sometimes face normals are inverted.

The rest of this post will be a nerdy dive into the math of the handle algorithm, so I’ll put that after the break. Enjoy this thing I made in Blender with the above methods (dodecahedron base, six handles with 72-degree twists, Doo-Sabin, rind modeling, then Catmull-Clark):

Description of the handle algorithm

The paper introducing the curved handle algorithm is “Interactive Construction of Multi-Segment Curved Handles” by V. Srinivasan, E. Akleman, and J. Chen, published 2002. The paper is worth a read, but while I was porting the TopMod code I found it was missing some details. As a result, I’m using this space to describe the full algorithm unambiguously. I’ve tried to make my addon code readable and well-commented, but it doesn’t hurt to describe it in prose also.

As an overview, the algorithm produces cross sections of the handle by varying a parameter (0 \leq t \leq 1), where (t = 0) produces face 1 and (t = 1) produces face 2. A cubic Hermite spline is used to interpolate between the centroids of the two faces, with the derivatives of the spline at the endpoints determined by the normals. The polygonal faces are assumed approximately planar and projected into 2D, converted to polar coordinates, and linearly interpolated.

Let (\mathbf{c}_1) and (\mathbf{c}_2) be the centroids of the two faces, and let (\mathbf{n}_1) and (-\mathbf{n}_2) be the normals. Then the centroid of the handle’s cross section at time (t) is:

[\mathbf{c}(t) = \mathbf{c}_1 H_1(t) + \mathbf{c}_2 H_1(1 - t) + w_1 \mathbf{n}_1 H_2(t) - w_2 \mathbf{n}_2 H_2(1 - t)]

where (w_1) and (w_2) are scalar “weight” parameters, and (H_1(t) = 2t^3 - 3t^2 + 1) and (H_2(t) = t^3 - 2t^2 + t). From the values and derivatives of (H_1) and (H_2) each at 0 and 1, it can be seen that (\mathbf{c}(t)) starts at (\mathbf{c}_1) and ends at (\mathbf{c}_2), and (\mathbf{c}‘(0) = w_1 \mathbf{n_1}) and (\mathbf{c}’(1) = w_2 \mathbf{n_2}). (w_1) and (w_2) control how much the handle sticks out according to the normals. If they are zero the handle is perfectly straight, if they are both negative a tunnel is produced instead of a handle, and if they are opposite signs then you can make a nonorientable “handle-tunnel” like in a Klein bottle.

(\mathbf{c}(t)) gives us the centroid of the intermediate polygons that form the cross sections of the handle. (\mathbf{c}’(t)) gives us a normal vector of the plane in which the intermediate polygon at (t) falls. With these defined, what remains is to show how to interpolate between the two polygons.

A face is defined as 1. a circular list of 3D vertex coordinates, and 2. a normal vector. So that the user can control the amount of twist to the handle, each face’s vertex list has one designated as a “reference vertex.” We will assume temporarily that the faces have the same number of sides, but we’ll fix that later.

The first steps involve some normalization. First, we reverse the vertex list of face 2 and flip its normal vector (that’s why I wrote (-\mathbf{n}_2) above). We cyclically permute the vertex lists so that each list’s first entry is its reference vertex. Each face is then translated so that its centroid is at the origin.

The next step is to rotate the two faces so their normals are identical, and if the faces are approximately planar then they will be in a shared plane called the rotation plane. The vector (\mathbf{r} = \text{normalize}(\mathbf{c}_2 - \mathbf{c}_1)) is orthogonal to the rotation plane; I’m not totally sure why they chose this, and not clear on whether it actually matters which plane you choose. Anyway, to rotate face 1 so that its normal (\mathbf{n}_1) now becomes (\mathbf{r}), the axis of rotation is (\mathbf{a} = \mathbf{n}_1 \times \mathbf{r}). The angle of rotation is the angle (\phi) between (\mathbf{n}_1) and (\mathbf{r}). (\arcsin ||\mathbf{a}||) works, but only if (0 \leq \phi \leq \pi / 2). A more general formula is (\phi = \text{atan2}(||\mathbf{a}||, \mathbf{n}_1 \cdot \mathbf{r})). Face 2 is rotated by the same process.

Next, we set up a 2D Cartesian coordinate system in the rotation plane such that every point (a\mathbf{x} + b\mathbf{y}) on the plane is given by a pair ((a, b)). Any two unit vectors orthogonal to each other and to (\mathbf{r}) will do, so we can use one of the edges of face 1 as (\mathbf{x}) and normalize (\mathbf{r} \times \mathbf{x}) to produce (\mathbf{y}). We then project each 3D polygon onto the rotation plane; the 2D coordinates of point (\mathbf{p}) can be computed as (a = \mathbf{p} \cdot \mathbf{x}) and (b = \mathbf{p} \cdot \mathbf{y}). The pair ((a, b)) is then converted into polar coordinates ((r, \theta)).

Now each vertex list is a sequence of polar coordinates. As the faces are assumed to have the same number of sides, linear interpolation can be performed individually on the radii and angles. The angles, however, require preprocessing to avoid artifacts caused by the wraparound at (2\pi). For face (i \in {1,2}), we have a sequence of angles (\theta_1^{(i)}, \theta_2^{(i)}, \ldots, \theta_n^{(i)}). We want this sequence to be nondecreasing, so for each pair (\theta_k, \theta_{k + 1}) we add (2\pi) to the latter until (\theta_{k + 1} \geq \theta_k). Furthermore, (\theta_1^{(2)} - \theta_1^{(1)}) should not exceed (\pi), or the handle will appear excessively twisty – if this is the case, subtract (2\pi) from all (\theta_k^{(2)}). Finally, if the user does want extra 360-degree twists, an integer multiple of (2\pi) may be added to face 2’s angles. With these transformations, the two sequences of angles can be linearly interpolated. The result is a new list of polar coordinates for each (t).

As the last step, we reconstruct the 3D polygon from these polar coordinates by going through the transformation steps backwards and inverse. We convert these polar coordinates into Cartesian coordinates ((a, b)) and use the linear combination (a\mathbf{x} + b\mathbf{y}) to get 3D points. We then use an axis-angle rotation as before to rotate (\mathbf{r}) to the intermediate normal (\text{normalize}(\mathbf{c}’(t))). Add the resulting points to the centroid (\mathbf{c}(t)), and you have a cross section. Connect all the cross sections together with quadrilaterals, and that’s a handle!

Sorry, one more thing. If the two polygons have a different number of sides, you’ll have to do an additional step somewhere to make them match. TopMod seems to just chop off the extra vertices from the larger polygon; I prefer to take one vertex from the smaller polygon and repeat it until the polygons match. Smarter algorithms that are aware of the polar coordinates of the vertices are possible. Also note that some triangles will be necessary in place of quadrilaterals when connecting the vertices.

Variations can be easily conceived:

TopMod has a “pinch” feature where each cross section of the handle is uniformly scaled about its centroid by a function of (t) that starts and ends at 1. This allows making the handle thinner or thicker towards its midpoint. Weirdly enough, although this feature works in the Windows build of TopMod, the code path that implements it seems to be bypassed (the arguments that control the pinch are not passed in).
The handle can self-intersect if the weights are too small compared to the size of the faces. The pinch feature was likely designed to allow compensation for this, but could self-intersection be detected and countered automatically?
If the weight values are fixed, the handle construction is not scale invariant, which might be a nuisance for users working on objects with different levels of detail. The weights could be scaled by a factor depending on the sizes of the faces and the distance between their centroids.
Could the lengths of the segments be made more even?
What alternatives to a cubic Hermite spline look good?

Sound Dumplings: a Sound Collage Workflow

Nathan Ho — Wed, 26 Oct 2022 00:00:00 +0000

Most of the music I produce doesn’t use any samples. I’m not prejudiced against sample-based workflows in any way; I just like the challenge of doing everything in SuperCollider, which naturally encourages synthesis-only workflows (as sample auditioning is far more awkward than it is in a DAW). But sometimes, it’s fun and rewarding to try a process that’s the diametric opposite of whatever you do normally.

Two things happened in 2019 that led me to the path of samples. The first was that I started making some mixes, solely for my own listening enjoyment, that mostly consisted of ambient and classical music, pitch shifted in Ardour so they were all in key and required no special transition work. I only did a few of these mixes, but with the later ones I experimented a bit with mashing up multiple tracks. The second was that I caught up to the rest of the electronic music fanbase and discovered Burial’s LPs. I was wowed by his use of Sound Forge, a relatively primitive audio editor.

Not long into my Burial phase, I decided I’d try making sample-based ambient music entirely in Audacity. I grabbed tracks from my MP3 collection (plus a few pirated via youtube-dl), used “Change Speed” to repitch them so they were all in key, threw a few other effects on there, and arranged them together into a piece. I liked the result, and I started making more tracks and refining my process.

Soon I hit on a workflow that I liked a lot. I call this workflow sound dumplings. A sound dumpling is created by the following process:

Grab a number of samples, each at least a few seconds long, that each fit in a diatonic scale and don’t have any strong rhythmic pulse.
Add a fade in and fade out to each sample.
Use Audacity’s “Change Speed” to get them all in a desired key. Use variable speed playback, not pitch shifting or time stretching.
Arrange the samples into a single gesture that increases in density, reaches a peak, then decreases in density. It’s dense in the middle – hence, dumpling.
Bounce the sound dumpling to a single track and normalize it.

The step that is most difficult is repitching. A semitone up is a ratio of 1.059, and a semitone down is 0.944. Memorize those and keep doing them until the sample sounds in key, and use 1.5 (a fifth up) and 0.667 (a fifth down) for larger jumps. It’s better to repitch down than up if you can – particularly with samples containing vocals, “chipmunk” effects can sound grating. Technically repeated applications of “Change Speed” will degrade quality compared to a single run, but embrace your imperfections. Speaking of imperfections, don’t fuss too much about the fades sounding unnatural. You can just hide it by piling on more samples.

Most sound dumplings are at least 30 seconds long. Once you have a few sound dumplings, it is straightforward to arrange them into a piece. Since all your sound dumplings are in the same key, they can overlap arbitrarily. I like to use a formal structure that repeats but with slight reordering. For example, if I have five sound dumplings numbered 1 through 5, I could start with a 123132 A section, then a 454 B section, then 321 for the recap. The formal process is modular since everything sounds good with everything.

Sound dumplings are essentially a process for creating diatonic sound collages, and they allow working quickly and intuitively. I think of them as an approach to manufacturing music as opposed to building everything up from scratch, although plenty of creativity is involved in sample curation. Aside from the obvious choices of ambient and classical music, searching for the right terms on YouTube (like “a cappella” and “violin solo”) and sorting by most recent uploads can get you far. In a few cases, I sampled my previous sound dumpling work to create extra-dense dumplings.

The tracks I’ve produced with this process are some of my favorite music I’ve made. However, like my mixes they were created for personal listening and I only share them with friends, so I won’t be posting them here. (EDIT: I changed my mind, see my music as Interstate Hydra.) If you do want to hear examples of sound dumpling music, I recommend checking out my friend Nathan Turczan’s work under the name Equuipment. He took my sound dumpling idea and expanded on it by introducing key changes and automated collaging of samples in SuperCollider, and the results are very colorful and interesting.

If you make a sound dumpling track, feel free to send it my way. I’d be interested to hear it.

Matrix Modular Synthesis

Nathan Ho — Tue, 13 Sep 2022 00:00:00 +0000

Today’s blog post is about a feedback-based approach to experimental sound synthesis that arises from the union of two unrelated inspirations.

Inspiration 1: Buchla Music Easel

The Buchla Music Easel is a modular synthesizer I’ve always admired for its visual appearance, almost more so than its sound. I mean, look at it! Candy-colored sliders, knobs, and banana jacks! It has many modules that are well explored in this video by Under the Big Tree, and its standout features in my view are two oscillators (one a “complex oscillator” with a wavefolder integrated), two of the famous vactrol-based lowpass gates, and a five-step sequencer. The video I linked says that “the whole is greater than the sum of the parts” with the Easel – I’ll take his word for it given the price tag.

The Music Easel is built with live performance in mind, which encompasses live knob twiddling, live patching, and playing the capacitive touch keyboard. Artists such as Kaitlyn Aurelia Smith have used this synth to create ambient tonal music, which appears tricky due to the delicate nature of pitch on the instrument. Others have created more out-there and noisy sounds on the Easel, which offers choices between built-in routing and flexible patching between modules and enables a variety of feedback configurations for your bleep-bloop-fizz-fuzz needs.

Inspiration 2: CFI Model

While idly browsing the proceedings of a conference, I found a paper called “A Model for Collective Free Improvisation” by Canonne and Garnier. Written in 2011, it presents a mathematical model of a genre of experimental music deriving from free jazz. In this model (which I will call the “CFI Model”), a group of musicians listen to each other and react, with each player’s output modeled by a signal that roughly indicates the complexity of their improvisation over time. At the end of this post I’ll dig into a technical review of their research, complete with math formulas, but for now the important thing to know is that the listening process is modeled by a matrix.

Not all my readers are mathematically inclined, and I sometimes get polite comments that my posts are too technical, so I’ll explain roughly how a matrix works. Player A listens to player B and responds to the complexity of B’s playing with one of three options: joining player B, ignoring player B, and doing the opposite of player B. We construct a table listing all posssible combinations of player A and player B, and we place a 1, 0, or -1 depending on whether player A follows, ignores, or contrasts with player B, respectively. Note that player A and player B might be the same person, which happens when player A listens to themselves, and this is given by entries along the upper left to lower right diagonal (a.k.a. the main diagonal). The resulting table of entries is our matrix. Here is an example.

		Player B
		1	2	3	4
Player A	1	1	0	-1	0
	2	0	0	0	-1
	3	1	0	0	0
	4	0	1	1	0

Each player’s actual decision depends on the sum of the contributions from all the players they are listening to. So for each player A, we look along their row of entries and multiply each matrix entry by each player B’s signal. Sum up all the results, and you have an output signal for each row. This is essentially matrix-vector multiplication. Crucially, the output of the matrix affects the input, producing complex feedback behavior over time. The exact way the output affects the input requires getting into differential equations and will be saved for a later section.

In the model, this matrix is populated randomly with 1’s, 0’s, and -1’s, which creates a wide variety of behaviors ranging from convergence to static values (perhaps drones or ostinati) to oscillations (perhaps call and response). The paper also models the concept of boredom over time, and when the players collectively become too bored, they choose to listen to different people. This is represented by randomizing the matrix again, changing the behavior of the differential equations as they evolve. The result is long-term modulation between static, oscillatory, and sometimes chaotic behavior, which is best shown with a picture. Depicted is a simulation of the CFI Model with four musicians, with moments of matrix re-randomization indicated with vertical dashed lines.

Canonne and Garnier point out that the signal for each player could be made multidimensional with different factors for pitch, rhythm, timbre, etc. This could make the CFI Model suitable for algorithmic music generation (as it is I feel it is a little too simple for that, but read on).

Matrix Modular Synthesis

Having verbosely introduced the Buchla Music Easel and the CFI Model, I can now finally get to the topic of this post. How do these concepts come together? Imagine a set of modules like the Easel (or any other modular synthesizer, really), and then feed every module back into the inputs of every other module in a matrix configuration, with a random matrix that changes over time. I call this Matrix Modular Synthesis, and it is related to the Feedback Integrator Networks concept I discussed a few months ago.

Implementation in SuperCollider is not too hard. The only tricky part is that SuperCollider has no built-in matrix support, so I instead took advantage of its multichannel expansion feature. (If using an alternate language to sclang without multichannel expansion, matrix-vector multiplication can be done with a loop.) Also note LFNoise2.kr(0.5).round as a hacky way of getting random 1’s, 0’s, and -1’s. All the “modules” are based on components of the Buchla Music Easel. The comments should explain the rest.

// Run before booting server.
Server.default.options.blockSize = 1;

(
SynthDef(\easel, {
    var snd, numInputs, numOutputs;
    numInputs = 12;
    numOutputs = 7;
    snd = LocalIn.ar(numOutputs);
    // Matrix-vector multiplication using multichannel expansion.
    snd = snd * ({ { LFNoise2.kr(0.5).round } ! numInputs } ! numOutputs);
    snd = snd.sum;
    // If additional modules are added, remember to update numInputs and numOutputs.
    // numInputs should equal the maximum i in snd[i], plus one.
    // numOutputs should equal the number of entries in the snd array.
    snd = [
        // Oscillator that reaches LFO range
        SinOsc.ar(snd[0].linexp(-1, 1, 0.1, 8000)),
        // Oscillator in audio range only
        SinOsc.ar(snd[1].linexp(-1, 1, 100, 8000)),
        // Wavefolder with controllable gain
        (snd[2] * snd[3].linexp(-1, 1, 1, 10)).fold2,
        // Two filters with controllable cutoff and resonance
        MoogFF.ar(snd[4], snd[5].linexp(-1, 1, 10, 8000), snd[6].linlin(-1, 1, 0, 4)) * 3.dbamp,
        MoogFF.ar(snd[7], snd[8].linexp(-1, 1, 10, 8000), snd[9].linlin(-1, 1, 0, 4)) * 3.dbamp,
        // Metallic reverb (spring reverb in the Easel)
        FreeVerb.ar(snd[10], mix: 1),
        // Five-step sequencer
        Demand.ar(Impulse.ar(snd[11].linexp(-1, 1, 0.1, 1000)), 0, Dseq([-0.1, 0.5, -0.9, 0.3, -0.4], inf))
    ];
    if(snd.size != numOutputs) {
        Error("Please ensure numOutputs matches number of modules").throw;
    };
    snd = Sanitize.ar(snd);
    LocalOut.ar(snd);
    // Grab output from one of the filters. Try changing the index here, or even using snd.sum.
    snd = Limiter.ar(LeakDC.ar(snd[3]));
    snd = Pan2.ar(snd, 0);
    Out.ar(\out.kr(0), snd);
}).add;
)

Synth(\easel);

Here’s how it sounds – beeps, fizzes, and crackles abound:

The selection of modules heavily influences the aesthetics of the patch. The choice of output signal is also important; options include a single output, a random switch between outputs, or all of the module outputs summed together.

When I was building up this patch, I was surprised at how varied and complex it sounded even with just two oscillators FMing each other.

Although I set SC to use single-sample feedback, the patch also produces good sounds with a standard block size like 64. Try both.

Not all matrices produce acceptable results. Some result in silence. One could analyze the audio output and re-randomize the matrix if an extended period of silence happens. An even more sophisticated patch could use machine listening features on the output and incorporate the boredom model from the paper. Would that sound better than simple randomness? I don’t know, try it.

Matrix Modular Synthesis generates harsh and noisy tones, but with the selection of the right modules it could produce more accessible results. One option worth exploring is quantizing the oscillator frequencies to a scale.

Discussion of the CFI Model

The CFI Model, or rather a simplified version of it, is a system of (N \geq 1) ordinary differential equations. (\mathbf{x}(t)) is a size-(N) vector of signals that vary over time (t). Signal (x_k) roughly means the complexity of what musician (k) plays, with values larger in absolute value meaning more complex musical content. The signals can also be negative, for reasons I don’t fully understand. Let (\mathbf{\Omega}) be an (N \times N) matrix that indicates which musicians are listening to which, as discussed in a previous section. Then the CFI Model is

[\tau_c \frac{d\mathbf{x}}{dt} = \mathbf{\Omega}\mathbf{x} - \mathbf{x}^3]

where (\tau_c) is a constant with time units and (\mathbf{x}^3) is computed pointwise. The (\mathbf{x}^3) term is ad hoc, and it introduces a nonlinearity that “fights” (\mathbf{x}) and prevents solutions from exploding to infinity. Note that (\mathbf{x}(0)) must not be all zeros or (\mathbf{x}) will be zero forever, so (\mathbf{x}) should be kicked off with, say, random values.

The solutions to this system of equations are hard to analyze due to the nonlinearity, but can be tested experimentally with numerical integration. In the case (N = 1), if (\Omega_{11} \geq 0) then (x_1) appears to converge to (\pm\sqrt{\Omega_{11}}) provided that (x_1(0) \neq 0). If (N = 2), a matrix like (\mathbf{\Omega} = \begin{bmatrix} 0 & 1 \ -1 & 0 \end{bmatrix}) results in oscillations as player 1 follows player 2 and player 2 contrasts with player 1. In general, populating (\mathbf{\Omega}) with random values of different signs seems to produce convergence, oscillation, or sometimes chaos.

The boredom model (called “boreness” in the paper) is as follows. Define “information” (I) as the following scalar signal:

[I = \frac{1}{2} \sum_k \left[ x_k^2 + \tau_n^2 \left(\frac{dx_k}{dt}\right)^2 \right]]

where (\tau_n) is another constant with time units. (I) is roughly the overall busyness of the music at any moment in time; both high absolute values of (x_k) and high rate of change of (x_k) contribute toward information, which is always nonnegative.

Define (b) or “boredom” as a signal with (b(0) = 0). Usually (b) grows as (db/dt = I), but when (b) reaches a constant (b_\text{max}), (b) is reset to 0 and (\mathbf{\Omega}) is re-randomized with entries selected from ({-1, 0, 1}). These discontinuities technically can make derivatives invalid, but that doesn’t matter much during numerical integration. By the way, I simply used the Euler method to simulate the CFI Model, which is perfectly sufficient for getting the desired behavior.

I am omitting some details where musical events are treated as discrete “clusters” and the signal (\mathbf{x}) becomes piecewise constant, as they are not all that important to the model.

Conclusions

Feedback has been a mainstay of sound synthesis for decades, so the true novelty of Matrix Modular Synthesis could be reasonably challenged. However, I think the idea of simulating random live patching with a dynamic matrix adds a new level of control as opposed to manual and fixed feedback setups. It certainly helps that it sounds good, too.

Pulsar Synthesis

Nathan Ho — Sun, 31 Jul 2022 00:00:00 +0000

Curtis Roads’ book Microsound is a must-read for any nerdy music technologist. Despite its age, it contains many exciting techniques under the granular synthesis umbrella that still sound fresh and experimental. I recently started exploring one of these methods, called pulsar synthesis, and thought it’d be fun to write my own review of it and demonstrate how to accomplish it in SuperCollider or any similar environment.

Pulsar synthesis produces periodic oscillations by alternating between a short, arbitrary waveform called the pulsaret (after “wavelet”) and a span of silence. The reciprocal of the pulsaret’s length is the formant frequency, which is manipulated independently of the fundamental frequency by changing the speed of the pulsaret’s playback and adjusting the silence duration to maintain the fundamental. Roads calls this “pulsaret-width modulation” or PulWM. If the formant frequency dips below the fundamental frequency, the silence disappears and the pulsarets can either be truncated (regular PulWM) or overlapped (overlapped PulWM or OPulWM). Roads describes OPulWM as “a more subtle effect” due to phase cancellation.

The pulsaret signal can be anything. The book suggests the following, among others: a single sine wave cycle, multiple sine wave periods concatenated, and a bandlimited impulse (sinc function). Another option is to take the waveform from live input, transferring the timbre of input over to a quasiperiodic signal and behaving slightly more like an effect than a synth.

Actual pulsars are neutron stars that emit beams of light out of their magnetic poles while spinning rapidly. As it spins, a pulsar with just the right orientation will shine a beam on Earth repeatedly and look like a periodic flashing, which can happen in a wide frequency range with periods often ranging from seconds to milliseconds. The signal they produce is not on-off. As the beam swings by us, the perceived light increases in intensity, reaches a peak, and decreases to effectively zero.

Pulsarets are usually oscillatory signals rather than the more bell-shaped signals that real pulsars produce. We can make pulsarets more accurately reflect pulsar optics using pointwise (sample by sample) multiplication of the pulsaret with a window function whose length is identical to that of the pulsaret. Suggested window functions include rectangular, Gaussian, linearly decreasing or increasing, and exponentially decreasing or increasing. The window function can even itself be oscillatory, resulting in frequency-domain convolution with the pulsaret.

Roads emphasizes an important aspect of pulsar synthesis: the fundamental frequency can venture below 20 Hz to create a train of individually audible pulses, blurring the line between rhythm and pitch.

A single pulsar synthesizer reads from the following parameters, which may vary over time: fundamental frequency, formant frequency, pulsaret waveform, window, output amplitude, and output panning.

Advanced pulsar synthesis

Roads and Alberto de Campo’s graphical program PulsarGenerator, originally developed in 1991 using Synth-O-Matic (an early SuperCollider predecessor), is described in detail in Microsound. In the development of this software, several new concepts were added to pulsar synthesis.

One way to command more exciting timbres is to use multiple pulsar synthesizers added together, with the sole constraint that they all share fundamental frequency so they fuse into one tone. PulsarGenerator used three such pulsar synthesizers. Roads notes the possibility of spatializing individual formants separately from each other.

Another trick for rattly, malfunctioning sounds is to make a pulsar synthesizer output pulsarets intermittently, in a scheme known as pulsaret masking. This can be a regular process where e.g. three pulsarets are emitted followed by two periods of silence (burst masking), or a random process where e.g. each pulsaret has a 10% chance of being replaced with silence (stochastic masking). Finally, channel masking randomly decides which channel each pulsaret is sent to according to a probability table, and these channels are spatialized.

Roads suggests convolving pulsar synthesis output with short recorded audio samples. Using bird calls and other animals as source material, he employed this technique in the second movement, “Organic,” of his work Clang-Tint_ (1991-94), as well as several other works. Roads additionally recommends using percussion samples for convolution.

Building a pulsar synthesis patch in SuperCollider

In February 2022, Marcin Pietruszewski published nuPG (New Pulsar Generator), which is compatible with recent versions of SuperCollider 3. If you want an out-of-the-box pulsar synthesis platform wrapped up in a nice GUI, then you know where to click.

It’s also not too hard to roll your own pulsar synthesis in SuperCollider with some math. The first step is to introduce a signal pulsaretPhase which is 0 at the beginning of the pulsaret, 1 at the end, and continues increasing during silence until the period resets. This can be created using an LFSaw that ramps from 0 to formantFreq / freq at the frequency freq:

(
{
    var snd, freq, formantFreq, pulsaretPhase;
    freq = 100;
    formantFreq = 400;
    pulsaretPhase = LFSaw.ar(freq, iphase: 1).linlin(-1, 1, 0, 1) * formantFreq / freq;
    pulsaretPhase;
}.plot(0.02);
)

pulsaretPhase is then used in two ways. First, if it’s greater than or equal to 1, the pulsaret has stopped playing and silence is produced. Second, the pulsaretPhase indexes into the pulsaret signal. Here’s a sine wave pulsaret with proper gating:

(
{
    var snd, freq, formantFreq, pulsaretPhase;
    freq = 100;
    formantFreq = 400;
    pulsaretPhase = LFSaw.ar(freq, iphase: 1).linlin(-1, 1, 0, 1) * formantFreq / freq;
    snd = sin(pulsaretPhase * 2pi) * (pulsaretPhase < 1);
    snd;
}.plot(0.02);
)

We can generalize to multiple concatenated sine waves by multiplying the term inside the sin(...) by a value we’ll call sineCycles:

(
{
    var snd, freq, formantFreq, sineCycles, pulsaretType, pulsaretPhase;
    freq = 100;
    formantFreq = 400;
    sineCycles = 4;
    pulsaretPhase = LFSaw.ar(freq, iphase: 1).linlin(-1, 1, 0, 1) * formantFreq / freq;
    snd = sin(pulsaretPhase * 2pi * sineCycles.floor) * (pulsaretPhase < 1);
    snd;
}.plot(0.02);
)

We can apply an approximately exponentially decaying window to the pulsaret, as opposed to the rectangular one we’ve been using so far:

(
{
    var snd, freq, formantFreq, sineCycles, pulsaretPhase, window;
    freq = 100;
    formantFreq = 200;
    sineCycles = 4;
    pulsaretPhase = LFSaw.ar(freq, iphase: 1).linlin(-1, 1, 0, 1) * formantFreq / freq;
    window = pulsaretPhase.lincurve(0, 1, 1, 0, -4);
    snd = sin(pulsaretPhase * 2pi * sineCycles.floor) * window * (pulsaretPhase < 1);
    snd;
}.plot(0.02);
)

That’s enough plotting. Let’s hear it with random modulation of all the parameters we discussed, including amplitude and pan position:

(
{
    var snd, freq, formantFreq, sineCycles, pulsaretPhase, window, randomLFO;
    randomLFO = { LFNoise2.kr(5) };
    freq = randomLFO.().linexp(-1, 1, 1, 1000);
    formantFreq = randomLFOs.().linexp(-1, 1, 2, 8000);

    sineCycles = randomLFO.().linlin(-1, 1, 1, 4);
    pulsaretPhase = LFSaw.ar(freq, iphase: 1).linlin(-1, 1, 0, 1) * formantFreq / freq;
    window = pulsaretPhase.lincurve(0, 1, 1, 0, -4);
    snd = sin(pulsaretPhase * 2pi * sineCycles.floor) * window * (pulsaretPhase < 1);
    snd = snd * randomLFO.().linlin(-1, 1, 0, 1);
    snd = Pan2.ar(snd, randomLFO.() * 0.4);
    snd = snd * -5.dbamp;
    snd ! 2;
}.play(fadeTime: 0);
)

As discussed in Microsound, let’s use three separate formants and pan them around the stereo field separately:

(
{
    var snd, freq, formantCount, formantFreq, sineCycles, pulsaretPhase, window, randomLFO, randomLFOs;
    formantCount = 3;
    randomLFO = { LFNoise2.kr(5) };
    randomLFOs = { { randomLFO.() } ! formantCount };
    freq = randomLFO.().linexp(-1, 1, 1, 1000);
    formantFreq = randomLFOs.().linexp(-1, 1, 2, 8000);
    sineCycles = randomLFOs.().linlin(-1, 1, 1, 4);
    pulsaretPhase = LFSaw.ar(freq, iphase: 1).linlin(-1, 1, 0, 1) * formantFreq / freq;
    window = pulsaretPhase.lincurve(0, 1, 1, 0, -4);
    snd = sin(pulsaretPhase * 2pi * sineCycles.floor) * window * (pulsaretPhase < 1);
    snd = snd * randomLFOs.().linlin(-1, 1, 0, 1);
    snd = Pan2.ar(snd, randomLFOs.() * 0.4);
    snd = snd.flop.sum;
    snd = snd * -2.dbamp;
    snd;
}.play(fadeTime: 0);
)

This gets old fast, and the shapes of the random LFOs are the culprit. They are all constantly moving smoothly and there’s a lack of space to make movement sound special, leading to monotony. Recalling the switching method that I talked about in my last blog post, let’s have each random LFO produce a random static value every now and then. Also, let’s modulate the rate of all LFOs with another global slow LFO that determines their overall rate:

(
{
    var snd, freq, formantCount, formantFreq, sineCycles, pulsaretPhase, window, randomLFO, randomLFOs, lfoRate;
    formantCount = 3;
    lfoRate = LFDNoise1.kr(0.3).linexp(-1, 1, 0.1, 16);
    randomLFO = {
        var trigger;
        trigger = Dust.kr(lfoRate);
        Select.kr(ToggleFF.kr(trigger), [
            LFNoise2.kr(lfoRate),
            TRand.kr(-1, 1, trigger)
        ]);
    };
    randomLFOs = { { randomLFO.() } ! formantCount };
    freq = randomLFO.().linexp(-1, 1, 1, 1000);
    formantFreq = randomLFOs.().linexp(-1, 1, 2, 8000);
    sineCycles = randomLFOs.().linlin(-1, 1, 1, 4);
    pulsaretPhase = LFSaw.ar(freq, iphase: 1).linlin(-1, 1, 0, 1) * formantFreq / freq;
    window = pulsaretPhase.lincurve(0, 1, 1, 0, -4);
    snd = sin(pulsaretPhase * 2pi * sineCycles.floor) * window * (pulsaretPhase < 1);
    snd = snd * randomLFOs.().linlin(-1, 1, 0, 1);
    snd = Pan2.ar(snd, randomLFOs.() * 0.4);
    snd = snd.flop.sum;
    snd = snd * -2.dbamp;
    snd;
}.play(fadeTime: 0);
)

As always, it’s not just the synthesis recipe, it’s also how you modulate it.

To bring this patch into the 21st century, we can stack on a few effects. Here are three pitch shifters (not just any pitch shifter, but the weird and metallic PitchShift) and three frequency shifters, all modulated randomly:

(
{
    var snd, freq, formantCount, formantFreq, sineCycles, pulsaretPhase, window, randomLFO, randomLFOs, lfoRate;
    formantCount = 3;
    lfoRate = LFDNoise1.kr(0.3).linexp(-1, 1, 0.1, 16);
    randomLFO = {
        var trigger;
        trigger = Dust.kr(lfoRate);
        Select.kr(ToggleFF.kr(trigger), [
            LFNoise2.kr(lfoRate),
            TRand.kr(-1, 1, trigger)
        ]);
    };
    randomLFOs = { { randomLFO.() } ! formantCount };
    freq = randomLFO.().linexp(-1, 1, 1, 1000);
    formantFreq = randomLFOs.().linexp(-1, 1, 2, 8000);
    sineCycles = randomLFOs.().linlin(-1, 1, 1, 4);
    pulsaretPhase = LFSaw.ar(freq, iphase: 1).linlin(-1, 1, 0, 1) * formantFreq / freq;
    window = pulsaretPhase.lincurve(0, 1, 1, 0, -4);
    snd = sin(pulsaretPhase * 2pi * sineCycles.floor) * window * (pulsaretPhase < 1);
    snd = snd * randomLFOs.().linlin(-1, 1, 0, 1);
    snd = Pan2.ar(snd, randomLFOs.() * 0.4);
    snd = snd.flop.sum;
    [0.2, 0.1, 0.05].do { |windowSize|
        snd = PitchShift.ar(snd, windowSize, randomLFO.().linexp(-1, 1, 0.5, 2)) * 6.dbamp;
        snd = FreqShift.ar(snd, randomLFO.() * 100);
    };
    snd = Limiter.ar(snd);
    snd = snd * -2.dbamp;
    snd;
}.play(fadeTime: 0);
)

The result certainly sounds varied and unpredictable to me:

Here’s another snippet I made also using pulsar synthesis, various effects, and heavy random modulation of everything:

This time I used a downward sine wave chirp for the pulsaret, synthesizing something akin to the “moisture bass” concept we discussed a while back, and when the pulsar synthesizer reaches low frequencies the individual pulsarets sound like 909-style kick drums. The chirp qualities are audible even through the many effects, which should allay any concerns that pulsar synthesis is unnecessary and interchangeable with a “simpler” synthesis method like subtractive.

There are also two identical and parallel copies of the pulsar synthesizer and effects chain, both of which are wired to a vocoder with one signal as the carrier and the other as a modulator. This was inspired by Roads’ use of convolution (ignore the fact that vocoding isn’t really convolution), and by applying this wobbly dynamic EQ to the carrier signal a rougher and more unstable sound is achieved, not unlike a low-bitrate MP3 at times.

Acknowledgements

Thanks to dietcv on the SuperCollider forums, who reminded me of pulsar synthesis and inspired this post.

"Switching" in Procedural Generation

Nathan Ho — Sat, 23 Jul 2022 00:00:00 +0000

A friend recently told me of an idea he had: what if a video game employing procedural generation could legitimately surprise its own developer? This applies to music as well – I wish I had such moments more often in my algorithmic composition work.

Pondering this question made me think back to a recent video by producer Ned Rush: Melodic Phrase Generator in Ableton Live 11. In it, the artist starts with four C notes in the piano roll and uses a variety of MIDI filters to transform them into a shimmering pandiatonic texture with lots of variation. It’s worth a watch, but one moment that especially made me think is where he pulls out an arpeggiator and maps a random control signal to its “Style” setting (timestamp). Ableton Live allows mapping control signals to discrete popup menus as well as continuous knobs, so this causes the arpeggiator to switch randomly between 18 arpeggiation styles, including Up, Down, UpDown, Converge, Diverge, Pinky Up, Chord Trigger, Random, and more.

This was the most important part of Ned Rush’s patch, as these styles individually sound good and different from each other, so switching between them randomly sounds both good and diverse. From this, we can gather a valuable technique for procedural generation: switching or collaging among many individual algorithms, each of which has considerable care put into it. Imagine a “synthesizer” (or level generator, or whatever) with many different switched modes, coated in layers of “effects,” each with their own switched modes. Even with simple, haphazard randomness controlling the modes, the system will explore numerous combinatorial varieties. If this wears itself out quickly, then the modes can be grouped into categories, with switching within a category happening on a smaller time scale (or space scale) and switching between categories happening on a larger time scale.

Switching is a handy way to aid in achieving those elusive moments where a generative system does something truly unexpected. I recommend experimenting with it next time you play with algorithmic creativity.

Integer Ring Modulation

Nathan Ho — Thu, 31 Mar 2022 00:00:00 +0000

When I think of ring modulation – or multiplication of two bipolar audio signals – I usually think of a complex, polyphonic signal being ring modulated by an unrelated sine wave, producing an inharmonic effect. Indeed, this is what “ring modulator” means in many synthesizers’ effect racks. I associate it with early electronic music and frankly find it a little cheesy, so I don’t use it often.

But if both signals are periodic and their frequencies are small integer multiples of a common fundamental, the resulting sound is harmonic. Mathematically this is no surprise, but the timbres you can get out of this are pretty compelling.

I tend to get the best results from pulse waves, in which case ring modulation is identical to an XOR gate (plus an additional inversion). Here’s a 100 Hz square wave multiplied by a second square wave that steps from 100 Hz, 200 Hz, etc. to 2000 Hz and back.

As usual, here is SuperCollider code:

(
{
    var freq, snd;
    freq = 100;
    snd = Pulse.ar(freq) * Pulse.ar(freq * LFTri.ar(0.3, 3).linlin(-1, 1, 1, 20).round);
    snd ! 2;
}.play(fadeTime: 0);
)

Try pulse-width modulation, slightly detuning oscillators for a beating effect, multiplying three or more oscillators, and filtering the oscillators prior to multiplication. There are applications here to synthesizing 1-bit music.

Credit goes to Sahy Uhns for showing me this one some years ago.

EDIT 2023-01-12: I have learned that Dave Rossum used this technique in Trident, calling it “zing modulation.” See this YouTube video.

Feedback Integrator Networks

Nathan Ho — Thu, 24 Mar 2022 00:00:00 +0000

Giorgio Sancristoforo’s recent software noise synth Bentō is a marvel of both creative DSP and graphic design. Its “Generators” particularly caught my eye. The manual states:

Bentō has two identical sound generators, these are not traditional oscillators, but rather these generators are models of analog computing patches that solve a differential equation. The Generators are designed to be very unstable and therefore, very alive!

There’s a little block diagram embedded in the synth’s graphics that gives us a good hint as to what’s going on.

We’re looking at a variant of a state variable filter (SVF), which consists of two integrators in series in a feedback loop with gain stages in between. Sancristoforo adds an additional feedback loop around the second integrator, and what appears to be an additional parallel path (not entirely sure what it’s doing). It’s not possible for chaotic behavior to happen in an SVF without a nonlinearity, so presumably there’s a clipper (or something else?) in the feedback loops.

While thinking about possible generalized forms of this structure, I realized that the signal flow around integrators can be viewed as 2 x 2 feedback matrix. A form of cross-modulation across multiple oscillators can be accomplished with a feedback matrix of greater size. I thought, why not make it like a feedback delay network in artificial reverberation, with integrators in place of delays? And so a new type of synthesis is born: the “feedback integrator network.”

An N x N feedback integrator network consists of N parallel signals that are passed through leaky integrators, then an N x N mixing matrix. First-order highpass filters are added to block dc, preventing the network from blowing up and getting stuck at -1 or +1. The highpass filters are followed by clippers. Finally, the clipper outputs are added back to the inputs with single-sample delays in the feedback path. I experimented with a few different orders in the signal chain, and the order presented here is the product of some trial and error. Here’s a block diagram of the 3 x 3 case:

A rich variety of sounds ranging from traditional oscillations to noisy, chaotic behavior results. Here are a few sound snippets of a 8 x 8 feedback integrator network, along with SuperCollider code. I’m using a completely randomized mixing matrix with values ranging from -1000 to 1000. Due to the nonlinearities, the scaling of the matrix is important to the sound. I’m driving the entire network with a single impulse on initialization, and I’ve panned the 8 parallel outputs across the stereo field.

// Run before booting server.
Server.default.options.blockSize = 1;

(
{
    var snd, n;
    n = 8;
    snd = Impulse.ar(0);
    snd = snd + LocalIn.ar(n);
    snd = Integrator.ar(snd, 0.99);
    snd = snd * ({ { Rand(-1, 1) * 1000 } ! n } ! n);
    snd = snd.sum;
    snd = LeakDC.ar(snd);
    snd = snd.clip2;
    LocalOut.ar(snd);
    Splay.ar(snd) * 0.3;
}.play(fadeTime: 0);
)

The four snippets above were not curated – they were the first four timbres I got out of randomization.

There’s great fun to be had by modulating the feedback matrix. Here’s the result of using smooth random LFOs.

I’ve experimented with adding other elements in the feedback loop. Resonant filters sound quite interesting. I’ve found that if I add anything containing delays, the nice squealing high-frequency oscillations go away and the outcome sounds like distorted sludge. I’ve also tried different nonlinearities, but only clipping-like waveshapers sound any good to my ears.

It seems that removing the integrators entirely also generates interesting sounds! This could be called a “feedback filter network,” since we still retain the highpass filters. Even removing the highpass filters, resulting in effectively a single-sample nonlinear feedback delay network, generates some oscillations, although less interesting than those with filters embedded.

While using no input sounds interesting enough on its own, creating a digital relative of no-input mixing, you can also drive the network with an impulse train to create tonal sounds. Due to the nonlinearities involved, the feedback integrator network is sensitive to how hard the input signal is driven, and creates pleasant interactions with its intrinsic oscillations. Here’s the same 8 x 8 network driven by a 100 Hz impulse train, again with matrix modulation:

Further directions in this area could include designing a friendly interface. One could use a separate knob for each of the N^2 matrix coefficients, but that’s unwieldy. I have found that using a fixed random mixing matrix and modulated, independent gain controls for each of the N channels produces results just as diverse as modulating the entire mixing matrix. A interface could be made by supplying N unlabeled knobs (likely less) and letting the user twist them for unpredictable and fun results.

OddVoices Dev Log 3: Pitch Contours

Nathan Ho — Mon, 07 Feb 2022 00:00:00 +0000

This is part of an ongoing series of posts about OddVoices, a singing synthesizer I’ve been building. OddVoices has a Web version, which you can now access at the newly registered domain oddvoices.org.

Unless we’re talking about pitch correction settings, the pitch of a human voice is generally not piecewise constant. A big part of any vocal style is pitch inflections, and I’m happy to say that these have been greatly improved in OddVoices based on studies of real pitch data. But first, we need…

Pitch detection

A robust and high-precision monophonic pitch detector is vital to OddVoices for two reasons: first, the input vocal database needs to be normalized in pitch during the PSOLA analysis process, and second, the experiments we conduct later in this blog post require such a pitch detector.

There’s probably tons of Python code out there for pitch detection, but I felt like writing my own implementation to learn a bit about the process. My requirements are that the pitch detector should work on speech signals, have high accuracy, be as immune to octave errors as possible, and not require an expensive GPU or a massive dataset to train. I don’t need real time capabilities (although reasonable speed is desirable), high background noise tolerance, or polyphonic operation.

I shopped around a few different papers and spent long hours implementing different algorithms. I coded up the following:

Cepstral analysis [Noll1966]
Autocorrelation function (ACF) with prefiltering [Rabiner1977]
Harmonic Product Spectrum (HPS)
A simplified variant of Spectral Peak Analysis (SPA) [Dziubinski2004]
Special Normalized Autocorrelation (SNAC) [McLeod2008]
Fourier Approximation Method (FAM) [Kumaraswamy2015]

There are tons more algorithms out there, but these were the ones that caught my eye for some reason or another. All methods have their own upsides and downsides, and all of them are clever in their own ways. Some algorithms have parameters that can be tweaked, and I did my best to experiment with those parameters to try to maximize results for the test dataset.

I created a test dataset of 10000 random single-frame synthetic waveforms with fundamentals ranging from 60 Hz to 1000 Hz. Each one has harmonics ranging up to the Nyquist frequency, and the amplitudes of the harmonics are randomized and multiplied by (1 / n) where (n) is the harmonic number. Whether this is really representative of speech is not an easy question, but I figured it would be a good start.

I scored each algorithm by how many times it produced a pitch within a semitone of the actual fundamental frequency. We’ll address accuracy issues in a moment. The scores are:

Algorithm	Score
Cepstrum	9961/10000
SNAC	9941/10000
FAM	9919/10000
Simplified SPA	9789/10000
ACF	9739/10000
HPS	7743/10000

All the algorithms performed quite acceptably with the exception of the Harmonic Product Spectrum, which leads me to conclude that HPS is not really appropriate for pitch detection, although it does have other applications such as computing the chroma [Lee2006].

What surprised me most is that one of the simplest algorithms, cepstral analysis, also appears to be the best! Confusingly, a subjective study of seven pitch detection algorithms by McGonegal et al. [McGonegal1977] ranked the cepstrum as the 2nd worst. Go figure.

I hope this comparison was an interesting one in spite of how small and unscientific the study is. Be reminded that it is always possible that I implemented one or more of the algorithms wrong, didn’t tweak it in the right way, or didn’t look much into strategies for improving it.

The final algorithm

I arrived at the following algorithm by crossbreeding my favorite approaches:

Compute the “modified cepstrum” as the absolute value of the IFFT of (\log(1 + |X|)), where (X) is the FFT of a 2048-sample input frame (x) at a sample rate of 48000 Hz. The input frame is not windowed – for whatever reason that worked better!
Find the highest peak in the modified cepstrum whose quefrency is above a threshold derived from the maximum frequency we want to detect.
Find all peaks that exceed 0.5 times the value of the highest peak.
Find the peak closest to the last detected pitch, or if there is no last detected pitch, use the highest peak.
Convert quefrency into frequency to get the initial estimate of pitch.
Recompute the magnitude spectrum of (x), this time with a Hann window.
Find the values of the three bins around the FFT peak at the estimated pitch.
Use an artificial neural network (ANN) on the bin values to interpolate the exact frequency.

The idea of the modified cepstrum, i.e. adding 1 before taking the logarithm of the magnitude spectrum, is borrowed from Philip McLeod’s dissertation on SNAC, and prevents taking the logarithm of values too close to zero. The peak picking method is also taken from the same resource.

The use of an artificial neural network to refine the estimate is from the SPA paper [Dziubinski2004]. The ANN in question is a classic feedforward perceptron, and takes as input the magnitudes of three FFT bins around a peak, normalized so the center bin has an amplitude of 1.0. This means that the center bin’s amplitude is not needed and only two input neurons are necessary. Next, there is a hidden layer with four neurons and a tanh activation function, and finally an output layer with a single neuron and a linear activation function. The output format of the ANN ranges from -1 to +1 and indicates the offset of the sinusoidal frequency from the center bin, measured in bins.

The ANN is trained on a set of synthetic data similar to the test data described above. I used the MLPRegressor in scikit-learn, set to the default “adam” optimizer. The ANN works astonishingly well, yielding average errors less than 1 cent against my synthetic test set.

In spite of the efforts to find a nearly error-free pitch detector, the above algorithm still sometimes produces errors. Errors are identified as pitch data points that exceed a manually specified range. Errors are corrected by linearly interpolating the surrounding good data points.

Source code for the pitch detector is in need of some cleanup and is not yet publicly available as of this writing, but should be soon.

Vocal pitch contour phenomena

I’m sure the above was a bit dry for most readers, but now that we’re armed with an accurate pitch detector, we can study the following phenomena:

Drift: low frequency noise from 0 to 6 Hz [Cook1996].
Jitter: high frequency noise from 6 to 12 Hz.
Vibrato: deliberate sinusoidal pitch variation.
Portamento: lagging effect when changing notes.
Overshoot: when moving from one pitch to another, the singer may extend beyond the target pitch and slide back into it [Lai2009].
Preparation: when moving from one pitch to another, the singer may first move away from the target pitch before approaching it.

There is useful literature on most of these six phenomena, but I also wanted to gather my own data and do a little replication work. I had a gracious volunteer sing a number of melodies consisting of one or two notes, with and without vibrato, and I ran them through my pitch detector to determine the pitch contours.

Drift and jitter: In his study, Cook reported drift of roughly -50 dB and jitter at about -60 to -70 dB. Drift has a roughly flat spectrum and jitter has a sloping spectrum of around -8.5 dB per octave. My data is broadly consistent with these figures, as can be seen in the below spectra.

Drift and jitter are modeled as (f \cdot (1 + x)) where (f) is the static base frequency and (x) is the deviation signal. The ratio (x / f) is treated as an amplitude and converted to decibels, and this is what is meant by drift and jitter having a decibel value.

Cook also notes that drift and jitter also exhibit a small peak around the natural vibrato frequency, here around 3.5 Hz. Curiously, I don’t see any such peak in my data.

Synthesis can be done with interpolated value noise for drift and “clipped brown noise” for jitter, added together. Interpolated value noise is downsampled white noise with sine wave segment interpolation. Clipped brown noise is defined as a random walk that can’t exceed the range [-1, +1].

Vibrato is, not surprisingly, a sine wave LFO. However, a perfect sine wave sounds pretty unrealistic. Based on visual inspection of vibrato data, I multiplied the sine wave by random amplitude modulation with interpolated value noise. The frequency of the interpolated value noise is the same as the vibrato frequency.

Also note that vibrato takes a moment to kick in, which is simple enough to emulate with a little envelope at the beginning of each note.

Portamento, overshoot, and preparation I couldn’t find much research on, so I sought to collect a good amount of data on them. I asked the singer to perform two-note melodies consisting of ascending and descending m2, m3, P4, P5, and P8, each four times, with instructions to use “natural portamento.” I then ran all the results through the pitch tracker and visually measured rough averages of preparation time, preparation amount, portamento time, overshoot time, and overshoot amount. Here’s the table of my results.

Interval	Prep. time	Prep. amount	Port. time	Over. time	Over. amount
m3 ascending	0.1	0.7	0.15	0.2	0.5
m3 descending	no preparati	on	0.1	0.3	1
P4 ascending	no preparati	on	0.1	0.3	0.5
P4 descending	no preparati	on	0.2	0.2	1
P5 ascending	0.1	0.5	0.2	no overshoot
P5 descending	no preparati	on	0.2	0.1	1
P8 ascending	0.1	1	0.25	no overshoot
P8 descending	no preparati	on	0.15	0.1	1.5

As one might expect, portamento time gently increases as the interval gets larger. There is no preparation for downward intervals, and spotty overshoot for upward intervals, both of which make some sense physiologically – you’re much more likely to involuntarily relax in pitch rather than tense up. Overshoot and preparation amounts have a slight upward trend with interval size. The overshoot time seems to have a downward trend, but overshoot measurement is pretty unreliable.

Worth noting is the actual shape of overshoot and preparation.

In OddVoices, I model these three pitch phenomena by using quarter-sine-wave segments, and assuming no overshoot when ascending and no preparation when descending.

Further updates

Pitch detection and pitch contours consumed most of my time and energy recently, but there are a few other updates too.

As mentioned earlier, I registered the domain oddvoices.org, which currently hosts a copy of the OddVoices Web interface. The Web interface itself looks a little bland – I’d even say unprofessional – so I have plans to overhaul it especially as new parameters are on the way.

The README has been heavily updated, taking inspiration from the article “Art of README”. I tried to keep it concise and prioritize information that a casual reader would want to know.

References

Cook1996
Cook, P. R. 1996. “Identification of Control Parameters in an Articulatory Vocal Tract Model with Applications to the Synthesis of Singing.”
Dziubinski2004
Diubinski, M. and Kostek, B. 2004. “High Accuracy and Octave Error Immune Pitch Detection Algorithms.”
Kumaraswamy2015
Kumaraswamy, B. and Poonacha, P. G. 2015. “Improved Pitch Detection Using Fourier Approximation Method.”
Lai2009
Lai, Wen-Hsing. 2009. “An F0 Contour Fitting Model for Singing Synthesis.”
Lee2006
Lee, Kyogu. 2006. “Automatic Chord Recognition from Audio Using Enhanced Pitch Class Profile.”
McGonegal1977
McGonegal, Carol A. et al. 1977. “A Subjective Evaluation of Pitch Detection Methods Using LPC Synthesized Speech.”
McLeod2008
McLeod, Philip. 2008. “Fast, Accurate Pitch Detection Tools for Music Analysis.”
Noll1966
Noll, A. Michael. 1966. “Cepstrum Pitch Determination.
Rabiner1977
Rabiner, L. 1977. “On the Use of Autocorrelation Analysis for Pitch Detection.”

OddVoices Dev Log 2: Phase and Volume

Nathan Ho — Sun, 23 Jan 2022 00:00:00 +0000

This is the second in an ongoing series of dev updates about OddVoices, a singing synthesizer I’ve been developing over the past year. Since we last checked in, I’ve released version 0.0.1. Here are some of the major changes.

New voice!

Exciting news: OddVoices now has a third voice. To recap, we’ve had Quake Chesnokov, a powerful and dark basso profondo, and Cicada Lumen, a bright and almost synth-like baritone. The newest voice joining us is Air Navier (nav-YEH), a soft, breathy alto. Air Navier makes a lovely contrast to the two more classical voices, and I’m imagining it will fit in great in a pop or indie rock track.

Goodbye GitHub

OddVoices makes copious use of Git LFS to store original recordings for voices, and this caused some problems for me this past week. GitHub’s free tier caps the amount of Git LFS storage and the monthly download bandwidth to 1 gigabyte. It is possible to pay $5 to add 50 GB to both storage and bandwidth limits. These purchases are “data packs” and are orthogonal to GitHub Pro.

What’s unfortunate is that all downloads by anyone (including those on forks) contribute to the monthly download bandwidth, and even worse, downloads from GitHub Actions do also. I am easily running CI dozens of times per week, and multiplied by the gigabyte or so of audio data, the plan is easily maxed out.

A free GitLab account has a much more workable storage limit of 10 GB, and claims unlimited bandwidth for now. GitLab it is. Consider this a word of warning for anyone making serious use of Git LFS together with GitHub, and especially GitHub Actions.

Goodbye MBR-PSOLA

OddVoices, taking after speech synthesizers of the 90’s, is based on concatenation of recorded segments. These segments are processed using PSOLA, which turns them into a sequence of frames (grains), each for one pitch period. PSOLA then allows manipulation of the segment in pitch, time, and formants, and sounds pretty clean. The synthesis component is also computationally efficient.

One challenge with a concatenative synthesizer is making the segments blend together nicely. We are using a crossfade, but a problem arises – if the phases of the overlapping frames don’t approximately match, then unnatural croaks and “doubling” artifacts happen.

There is a way to solve this: manually. If one lines up the locations of the frames so they are centered on the exact times when the vocal folds close (the so-called “glottal closure instant” or GCI), the phases will match. Since it’s difficult to find the GCI from a microphone signal, an electroglottograph (EGG) setup is typically used. I don’t have an EGG on hand, and I’m working remotely with singers, so this solution has to be ruled out.

A less daunting solution is to use FFT processing to make all phases zero, or set every frame to minimum phase. These solve the phase mismatch problem but sound overtly robotic and buzzy. (Forrest Mozer’s TSI S14001A speech synthesis IC, memorialized in chipspeech’s Otto Mozer, uses the zero phase method – see US4214125A.) MBR-PSOLA softens the blows of these methods by using a random set of phases that are fixed throughout the voice database. Dutoit recommends only randomizing the lower end of the spectrum while leaving the highs untouched. It sounds pretty good, but there is still an unnatural hollow and phasey quality to it.

I decided to search around the literature and see if there’s any way OddVoices can improve on MBR-PSOLA. I found [Stylianou2001], which seems to fit the bill. It recommends computing the “center” of a grain, then offsetting the frame so it is centered on that point. The center is not the exact same as the GCI, but it acts as a useful stand-in. When all grains are aligned on their centers, their phases should be roughly matched too – and all this happens without modifying the timbre of the voice, since all we’re doing is a time offset.

I tried this on the Cicada voice, and it worked! I didn’t conduct any formal listening experiment, but it definitely sounded clearer and lacking the weird hollowness of the MBROLA voice. Then I tried it on the Quake voice, and it sounded extremely creaky and hoarse. This is the result of instabilities in the algorithm, producing random timing offsets for each grain.

Frame adjustment

Let (x[t]) be a sampled quasiperiodic voice signal with period (T), with a sample rate of (f_s). We round (T) to an integer, which works well enough for our application. Let (w[t]) be a window function (I use a Hann window) of length (2T). Brackets are zero-indexed, because we are sensible people here.

The PSOLA algorithm divides (x) into a number of frames of length (2T), where the (n)-th frame is given by (s_n[t] = w[t] x[t + nT]).

Stylianou proposes the “differentiated phase spectrum” center, or DPS center, which is computed like so:

[\eta = \frac{T}{2\pi} \arg \sum_{i = -T}^{T - 1} s_n^2[t] e^{2 \pi j t / T}]

(\eta) is here expressed in samples. The DPS center is not the GCI. It’s… something else, and it’s admitted in the paper that it isn’t well defined. However, it is claimed that it will be close enough to the GCI, hopefully by a near-constant offset. To normalize a frame on its DPS center, we recalculate the frame with an offset of (\eta): (s’_n[t] = w[t] x[t + nT + \text{round}(\eta)]).

The paper also discusses the center of gravity of a signal as a center close to the GCI. However, the center of gravity is less robust than the DPS center, as it can be shown that the center can be computed from just a single bin in the discrete Fourier transform, whereas the DPS center involves the entire spectrum.

Here’s where we go beyond the paper. As discussed above, for certain signals (\eta) can be noisy, and using this algorithm as-is can result in audible jitter in the result. The goal, then, is to find a way to remove noise from (\eta).

After many hours of experimenting with different solutions, I ended up doing a lowpass filter on (\eta) to remove high-frequency noise. A caveat is that (\eta) is a circular value that wraps around with period (T), and performing a standard lowpass filter will smooth out discontinuities produced by wrapping, which is not what we want. The trick is to use an encoding common in circular statistics, and especially in machine learning: convert it to sine and cosine, perform filtering on both signals, and convert it back with atan2. A rectangular FIR filter worked perfectly well for my application.

Overall the result sounds pretty good. There are still some minor issues with it, but I hope to iron those out in future versions.

Volume normalization

I encountered two separate but related issues regarding the volume of the voices. The first is that the voices are inconsistent in volume – Cicada was much louder than the other two. The second, and the more serious of the two, is that segments can have different volumes when they are joined, and this results in a “choppy” sound with discontinuities.

I fixed global volume inconsistency by taking the RMS amplitude of the entire segment database and normalizing it to -20 dBFS. For voices with higher dynamic range, this caused some of the louder consonants to clip, so I added a safety limiter that ensures the peak amplitude of each frame is no greater than -6 dBFS.

Segment-level volume inconsistency can be addressed by examining diphones that join together and adjusting their amplitudes accordingly. Take the phoneme /k/, and gather a list of all diphones of the form k* and *k. Now inspect the amplitudes at the beginning of k* diphones, and the amplitudes at the end of *k diphones. Take the RMS of all these amplitudes together to form the “phoneme amplitude.” Repeat for all other phonemes. Then, for each diphone, apply a linear amplitude envelope so that the beginning frames match the first phoneme’s amplitude and the ending frames match the second phoneme’s amplitude. The result is that all joined diphones will have a matched amplitude.

Conclusion

The volume normalization problem in particular taught me that developing a practical speech or singing synthesizer requires a lot more work than papers and textbooks might make you think. Rather, the descriptions in the literature are only baselines for a real system.

More is on the way for OddVoices. I haven’t yet planned out the 0.0.2 release, but my hope is to work on refining the existing voices for intelligibility and naturalness instead of adding new ones.

References

Stylianou2001
Stylianou, Yannis. 2001. “Removing Linear Phase Mismatches in Concatenative Speech Synthesis.”

OddVoices Dev Log 1: Hello World!

Nathan Ho — Tue, 11 Jan 2022 00:00:00 +0000

The free and open source singing synthesizer landscape has a few projects worth checking out, such as Sinsy, eCantorix, meSing, and MAGE. While each one has its own unique voice and there’s no such thing as a bad speech or singing synthesizer, I looked into all of them and more and couldn’t find a satisfactory one for my musical needs.

So, I’m happy to announce OddVoices, my own free and open source singing synthesizer based on diphone concatenation. It comes with two English voices, with more on the way. If you’re not some kind of nerd who uses the command line, check out OddVoices Web, a Web interface I built for it with WebAssembly. Just upload a MIDI file and write some lyrics and you’ll have a WAV file in your browser.

OddVoices is based on MBR-PSOLA, which stands for Multi-Band Resynthesis Pitch Synchronous Overlap Add. PSOLA is a granular synthesis-based algorithm for playback of monophonic sounds such that the time, formant, and pitch axes can be manipulated independently. The MBR part is a slight modification to PSOLA that prevents unwanted phase cancellation when crossfading between concatenated samples, and solves other problems too. For more detail, check out papers from the MBROLA project. The MBROLA codebase itself has some tech and licensing issues I won’t get into, but the algorithm is perfect for what I want in a singing synth. Note that OddVoices doesn’t interface with MBROLA.

I’ll use this post to discuss some of the more interesting challenges I had to work on in the course of the project so far. This is the first in a series of posts I will be making about the technical side of OddVoices.

Vowel mergers

OddVoices currently only supports General American English (GA), or more specifically the varieties of English that I and the singers speak. I hope in the future that I can correct this bias by including other languages and other dialects of English.

When assembling the list of phonemes, the cot-caught merger immediately came up. I decided to merge them, and make /O/ and /A/ aliases except for /Or/ and /Ar/ (here using X-SAMPA). To reduce the number of phonemes and therefore phoneme combinations, I represent /Or/ internally as /oUr/.

A more interesting merger concerns the problem of the schwa. In English, the schwa is used to represent an unstressed syllable, but the actual phonetics of that syllable can vary wildly. In singing, a syllable that would be unstressed in spoken English can be drawn out for multiple seconds and become stressed. The schwa isn’t actually sung in these cases, and is replaced with another phoneme. As one of the singers put it, “the schwa is a big lie.”

This matters when working with the CMU Pronouncing Dictionary, which I’m using for pronouncing text. Take a word like “imitate” – the second syllable is unstressed, and the CMUDict transcribes it as a schwa. But when sung, it’s more like /I/. This is simply a limitation of the CMUDict that I don’t have a good solution for. In the end I merge /@/ with /V/, since the two are closely related in GA. Similarly, /3`/ and /@`/ are merged, and the CMUDict doesn’t even distinguish those.

Real-time vs. semi-real-time operation

A special advantage of OddVoices over alternative offerings is that it’s built from scratch to work in real time. That means that it can become a UGen for platforms like SuperCollider and Pure Data, or even a VST plugin in the far future. I have a SuperCollider UGen in the works, but there’s some tricky engineering work involving communication between RT and NRT threads that I haven’t tackled yet. Stay tuned.

There is a huge caveat to real time operation: singers don’t operate in perfect real time! To see why, imagine the lyrics “rice cake,” sung with two half notes. The final /s/ in “rice” has to happen before the initial /k/ in “cake,” and the latter happens right on the third beat, so the singer has to anticipate the third beat with the consonant /s/. But in MIDI and real-time keyboard playing, there is no way to predict when the note off will happen until the third beat has already arrived.

VOCALOID handles this by being its own DAW with a built-in sequencer, so it can look ahead as much as it needs. chipspeech and Alter/Ego work in real time. In their user guides, they ask the user to shorten every MIDI note to around 50%-75% of its length to accommodate final consonant clusters. If this is not done, a phenomenon I call “lyric drift” happens and the lyrics misalign from the notes.

OddVoices supports two possible modes: true real-time mode and semi-real-time mode. In true real-time mode, we don’t know the durations of notes, so we trigger the final consonant cluster on a note off. Like chipspeech and Alter/Ego, this requires manual shortening of notes to prevent lyric drift. Alternatively, OddVoices supports a semi-real-time mode where every note on is accompanied by the duration of the note. This way OddVoices can predict the timing of the final consonant cluster, but still operate in otherwise real-time.

Semi-real-time mode is used in OddVoices’ MIDI frontend, and can also be used in powerful sequencing environments like SC and Pd by sending a “note length” signal along with the note on trigger. I think it’s a nice compromise between the constraints of real-time and the omniscience of non-real-time.

Syllable compression

After I implemented semi-real-time mode, another problem remained that reared its head in fast singing passages. This happens when, say, the lyric “rice cake” is sung very quickly, and the diphones _r raI aIs (here using X-SAMPA notation), when concatenated, will be longer than the note length. The result is more lyric drift – the notes and the lyrics diverge.

The fix for this was to peek ahead in the diphone queue and find the end of the final consonant cluster, then add up all the segment lengths from the beginning to that point. This is how long the entire syllable would last. This is then compared to the note length, and if it is longer, the playback speed is increased for that syllable to compensate. In short, consonants have to be spoken quickly in order to fit in quickly sung passages.

The result is still not entirely satisfactory to my ears, and I plan to improve it in future versions of the software. Syllable compression is of course only available in semi-real-time mode.

Syllable compression is evidence that fast singing is phonetically quite different from slow singing, and perhaps more comparable to speech.

Stray thoughts

This is my second time using Emscripten and WebAssembly in a project, and I find it an overall pleasant technology to work with (especially with embind for C++ bindings). I did run into an obstacle, however, which was that I couldn’t figure out how to compile libsndfile to WASM. The only feature I needed was writing a 16-bit mono WAV file, so I dropped libsndfile and wrote my own code for that.

I was surprised by the compactness of this project so far. The real-time C++ code adds up to 1,400 lines, and the Python offline analysis code only 600.

Hearing Graphs

Nathan Ho — Mon, 20 Sep 2021 00:00:00 +0000

My latest project is titled Hearing Graphs, and you can access it by clicking on the image below.

Hearing Graphs is a sonification of graphs using the graph spectrum – the eigenvalues of the graph’s adjacency matrix. Both negative and positive eigenvalues are represented by piano samples, and zero eigenvalues are interpreted with a bass drum. The multiplicity of each eigenvalue is represented by hitting notes multiple times.

Most sonifications establish some audible relationship between the source material and the resulting audio. This one remains mostly incomprehensible, especially to a general audience, so I consider it a failed experiment in that regard. Still, it was fun to make.

Moisture Bass

Nathan Ho — Fri, 17 Sep 2021 00:00:00 +0000

If you haven’t heard of the YouTube channel Bunting, it gets my strong recommendation. Bunting creates excellent style imitations of experimental bass music artists and breaks them down with succinct explanations. Notable is his minimal tooling: he uses mostly Ableton Live stock plugins and the free and open source wavetable synth Vital.

His latest tutorial, mimicking the style of the artist Resonant Language, contains several bass sounds with a property he calls “moisture” (timestamp). These bass sounds are created by starting with a low saw wave, boosting the highs, and running the result through Ableton Live’s vocoder set on “Modulator” mode. According to the manual, this enables self-vocoding, where the same signal is the modulator and carrier. An abstract view of a vocoder would suggest that this does little or nothing to the saw wave other than change its spectral tilt, but the reality is much more interesting. Hear for yourself an EQ’d saw wave before and after self-vocoding:

A closer inspection of the latter waveform shows why the self-vocoded saw sounds the way it does. Here’s a single pitch period:

The discontinuity in the saw signal is decorated with a chirp, or a sine wave that rapidly descends in frequency. This little 909 kick drum every pitch period is responsible for the “moisture” sound. Certainly there have been no studies on the psychoacoustics of moisture bass (for lack of a better term), but I suspect that it mimics dispersive behavior, lending a vaguely acoustic sound.

The chirp originates from the bandpass filters in the vocoder. The frequencies of the vocoder are exponentially spaced, so the bandpass filters have to increase in bandwidth for higher frequencies to cover the gaps. Larger bandwidth means lower Q, and lower Q reduces the ring time in the filter’s impulse response. The result is that low frequencies ring longer when the vocoder is pinged and high frequencies ring shorter. Mix them all together, and you have an impulse response resembling a chirp.

Self-vocoding with exponentially spaced bands is clever, but it isn’t the only way to create this effect. One option is to eliminate the vocoding part and use only the exponentially spaced bandpass filters, like an old-school filter bank. This sounds just like self-vocoding but requires fewer bandpass filters to work. In my experiments, I found that putting the resulting signal through nonlinear distortion is necessary to bring out the moisture property.

A more direct approach is to use wavefolding on a curved signal. The slope of the input signal controls the rate that it scrubs through wavefolding function, and thus controls the frequency of the resulting triangle wave. By modulating the slope from high in absolute value down to zero, a triangle wave descending in frequency is created. This is best explained visually:

And here’s how it sounds:

Announcing Canvas

Nathan Ho — Tue, 14 Sep 2021 00:00:00 +0000

Canvas (working title) is a visual additive synthesizer for Windows, macOS, and Linux where you can create sound by drawing an image. This is accomplished with a bank of 239 oscillators spaced at quarter tones, with stereo amplitudes mapped to the red and blue channels of the image. You can import images and sonify them, you can import sounds and convert their spectrograms to images, and you can apply several image filters like reverb, tremolo, and chorus. Check out the demo:

Canvas is directly inspired by the Image Synth from U&I Software’s MetaSynth. When I first heard of MetaSynth, I wrote it off as a gimmick that could only produce sweeps and whooshes. It wasn’t until I heard Benn Jordan’s recent demo and learned of its powerful set of image filters that I was immediately sold on the approach. I decided to build an alternative that’s just featureful enough to yield musical results, a sort of MS Paint to MetaSynth’s Photoshop.

Canvas has a lot of rough edges, and currently requires building from scratch if using Linux or macOS. Nevertheless, I hope it is of some interest to the music tech community.

A Survey of Nonstandard Oscillators

Nathan Ho — Sun, 15 Aug 2021 00:00:00 +0000

This is a collection of synthesis algorithms that meet the following criteria:

They must synthesize pitched tones — periodic or quasiperiodic waveforms.
I can include or exclude anything at my arbitrary discretion. I’m biased towards algorithms that are parametric rather than data- or sample-driven.

Algorithms that meet these criteria and fall outside of the typical cuisines of subtractive, FM/PM, wavetable, and additive synthesis are what I call nonstandard oscillators, hence the title. An earlier version of this post left it at just nonstandard oscillators, but I’ve also added in the more vanilla ones for comparison.

Many of these descriptions are intentionally brief, and I add detail only if I feel I have something useful to add or if the topic is particularly obscure. Many algorithms overlap, some are generalizations of others.

As a word of caution, inexperienced and information-hungry computer musicians find nonstandard synthesis methods particularly attractive because FM and subtractive have gotten too “basic” for them. (This is autobiographical here.) If you’re tired of standard synthesis methods, you haven’t spent enough time with them — FM and subtractive are capable some incredibly far-out and alien sounds, many yet to be discovered. I do encourage you to explore these methods if they inspire you, but they are neither sufficient nor necessary for interesting sound design, which is a product of lots of practice.

Last updated: 2025-02-09.

Additive synthesis

Added 2023-12-17.

Additive synthesis traditionally uses a bank of sine waves tuned to integer partials of a fundamental. The possibilities for algorithms implemented in the additive domain are tremendous: EQ filters, phaser, chorus, reverb, multiband dynamics, etc. all have their own bizarro-world additive versions. Harmor and Razor are good sources of inspiration for such algorithms. Additive synthesis also lends itself very well to interesting use of spatialization, even with only two channels.

Allpass phase distortion synthesis

Added 2024-01-02.

See [Pekonen2008]. Take a “carrier” sine wave oscillator and run it through a one-pole allpass filter whose coefficient is rapidly modulated using another oscillating signal of the same frequency. (Because allpass filters rotate phase, there is a relationship here to phase distortion synthesis, although the implementation is very different.)

Bitwise logical modulation

Added 2023-01-13.

Two oscillators – sine waves are used in examples – are quantized in amplitude and combined with bitwise operations such as AND, OR, or XOR [Kleimola2008]. One oscillator may be zero frequency, effectively acting as a discontinous waveshaper on the other oscillator.

Dynamic Stochastic Synthesis / Breakpoint Synthesis / Gendy

Attributed to Iannis Xenakis, Dynamic Stochastic Synthesis uses a piecewise linear function whose breakpoints (critical points) are randomly jostled around every period [Serra1993]. It is possible for the pitch period to change, but you can also clamp it to a fixed pitch if desired. See [Radna2023] for an extension hybridizing this method with wavetable synthesis.

Bytebeat

Added 2023-02-22.

Bytebeat refers to 8-bit music produced with one-liners in C, extensively using bitwise operations, and is associated with demoscene culture. [Heikkilä2011] A full work of bytebeat is assumed to have rhythm, melody, and form, but a small slice of the concept allows us to make standalone oscillators too.

In the “TOY*” mode in Mutable Instruments’ Braids Eurorack module [Gillet2015], the formula (((t ^ (x << 1)) & ~x) + (x >> 1)) % 256 is used to synthesize an oscillator where t is an unsigned 8-bit phase value and x is an unsigned 8-bit timbre parameter. The exact details of the bitwise operators are not particularly important. For additional tones mimicking circuit bending, the oscillator’s output is fed through a sample rate reducer.

Chaotic oscillators with sync

Added 2024-11-27.

Real chaotic oscillators are not included in this list; when directly used as waveforms they generally do not produce pitched sounds. However, any chaotic oscillator can be hard-synced by resetting its state variables periodically, as see in Newfangled Audio’s plugins Pendulate and Recirculate.

Discrete summation formulas (DSF)

This is a nearly 50-year-old algorithm for generating summed sine waves evenly spaced in linear frequency [Moorer1975]. You can control the spectral tilt, but partial amplitudes are monotonic. Inharmonicity is also possible.

DSF can generate a finite range of partials, and therefore produce perfectly bandlimited waveforms. DSF is the underlying algorithm to synthesize bandlimited impulse trains in Csound and SuperCollider.

The real musicality of DSFs is found in combining more than one of them. Given a fundamental, you can separately synthesize harmonics 1-5, harmonics 6-10, harmonics 11-31 but only the odd-numbered ones, etc., apply different spectral tilts to each one, and mix them together to produce an oscillator with a complex spectrum and lots of control.

Fake resonance

Made famous by the Casio CZ series, fake resonance is produced by taking a hard synced sine wave oscillator and windowing each pitch period with a decaying envelope to simulate a highly resonant filter being pinged by an oscillator. Fake lowpass, bandpass, and highpass waveforms are possible with all integer harmonics (saw wave-style) or odd harmonics only (pulse wave-style) [Gillet2015], although the highpass configuration produces discontinuities therefore strong aliasing (unless an aliasing suppression method such as PolyBLEP is used).

FOF (formant-wave-function) synthesis

Added 2024-01-02.

FOF is a granular synthesis method for creating formants [Rodet1984]. Each grain is a sine wave tuned to the formant frequency and then enveloped with an attack and decay; Rodet uses an exponential decay multiplied by a cosine-shaped fade in. To produce a single formant, a new grain is launched every pitch period, and allowed to overlap with future grains. Parameters are the formant frequency, attack and decay times, and initial phase. Multiple FOF synthesizers may be added together to synthesize instruments and vocals as done in the CHANT synthesizer.

FM/PM synthesis

Added 2023-12-17.

Made famous by Yamaha, FM’s basic unit is a standard wavetable oscillator. Timbres are explored by allowing oscillators to modulate each others’ frequencies.

Phase modulation synthesis is conceptually similar, and in fact two-op oscillator FM is equivalent to two-op PM but with an integrator added as a post-processing step after the modulating oscillator (as phase is the integral of frequency).

The Casio VZ series’ iPD (“interactive phase distortion”) has nothing to do with actual phase distortion, and is actually mostly a PM synthesizer with some zing modulation support.

Hypergrowl

Added 2023-01-01.

Hypergrowl is a technique for creating a variable wavetable, demonstrated by electronic musician Au5 in a popular YouTube tutorial using Serum. Many slightly detuned square waves are synthesized at say 100 Hz, and the resulting signal is sliced into 100 Hz pitch periods and chopped up into a variable wavetable. Then those variable wavetables are again synthesized at 100 Hz with detuning and modulation of wavetable position, and the output signal is chopped up and resampled into wavetables again. The process is repeated several times until an extremely complex variable wavetable is produced. As a final optional touch, the even harmonics are removed from the wavetables by altering the Fourier series.

IXA synthesis

See my last blog post. IXA is a variant of PM synthesis, but with a distorted phase on the carrier oscillator so a sine wave is produced when the index is zero.

Karplus-Strong synthesis

Added 2023-05-06.

An exciter is fed through a short feedback delay line so an audible pitch is produced. Attenuation and a one-pole lowpass filter are usually placed in the loop. The phase response of any filter(s) in the loop can introduce varying degrees of inharmonicity.

Short bursts of noise are customary as exciter signals, producing fairly convincing plucked string sounds. Other signals can be used as well – I really like the sound of rapid sine wave chirps as exciters.

Karplus-Strong synthesis was a precursor to Digital Waveguide Synthesis (DWG). I haven’t included DWG here as I consider it too general to really qualify as an oscillator.

Linear Automata Synthesis (LASy)

Added 2024-01-02.

In [Chareyron1990], a simple but clever approach to combining cellular automata with wavetable synthesis was presented:

Start with a discrete-valued signal whose samples range from 0 to n - 1 inclusive.
Add an r-sample delay. The initial state of the delay line memory is quite important, and it can be filled with any waveform of choice.
Take the last k samples to produce a tuple of k integers.
Run the result into a lookup table that maps these tuples to a single value from 0 to n - 1 inclusive.
Feed back into the input.

The r-sample delay line memory acts as the tape of a 1D cellular automaton, and the lookup table is its rule set. As the space of possible lookup tables quickly grows unwieldy, Chareyron suggests as a starting point using k = 3 and a lookup table that depends only on the sum of the three last samples of input. This cross section alone turns out to be quite powerful in its ability to self-generate modulating timbres.

The Newtonator

Added 2024-02-06.

The Newtonator is an old open-source LV2 plugin where waveforms are synthesized with a simple physics simulation of a single particle moving in one dimension, with its position over time directly sonified as sample values. In its most basic settings, the Newtonator simulates a particle in constant acceleration if its position is above the “floor,” and if below the floor it has the same acceleration but with a sign flip. This produces a sine-wave-like oscillator with parabolic segments.

By adding more physical phenomena, a wider variety of sounds is possible. The Newtonator provides many options for this: the particle can undergo collisions of varying elasticity with one or more obstacles, the positions of the floor and obstacles can modulate, there may be a “dead zone” near the floor where acceleration is 0 and the particle has constant velocity, and many others. To help ensure the generation of pitched tones, periodically the velocity is “reset” (by a mechanism I confess don’t quite understand).

OkiComputer

Added 2023-01-12.

NI Reaktor’s OkiComputer is a wavetable synthesizer with a row of 16 configurable “cells,” each of which has separate controls for wavetable, amplitude, integer frequency ratio, phase offset, phase distortion, and a combined “sample rate reducer” and bitcrusher. The “sample rate reducer” isn’t one in the traditional sense, and perhaps better described as time quantization of phase so that the sample rate scales with the frequency of the oscillator; this is the main thing that I find novel about OkiComputer. The output is produced by crossfading between consecutive cells, making it akin to Vector Synthesis and Wave Sequencing Synthesis.

Although not part of the oscillator, a key part of OkiComputer’s sound is the downstream chorus, followed by another sample rate reducer and bitcrusher. The synth excels at aggressively digital bass sounds, particularly with the sample rate reducer and bitcrusher.

Oscillator sync

Oscillator hard sync consists of an oscillator and an impulse train that resets the phase of said oscillator. The impulse train determines the fundamental frequency, while the oscillator’s intrinsic frequency creates a screaming formant at that frequency. Traditionally the oscillator is a saw or pulse, but any wavetable can be used.

There are other variants such as soft sync and reverse sync. Soft sync resets the oscillator only if the phase is below a certain threshold, but generally does not produce periodic waveforms. Reverse sync multiplies the oscillator’s frequency by -1 when a trigger is received.

Phase distortion synthesis (PD)

Introduced in the Casio CZ series, phase distortion synthesis allows transformation of a sine wave by altering the phase ramp with a nonlinearity [Casio].

Polygonal waveform synthesis

Polygonal waveform synthesis sonifies a regular polygon (especially a star polygon) by sweeping circularly and interpreting the X and/or Y coordinates as a waveform [Hohnerlein2016].

Pulsar synthesis

Added 2022-07-31.

Pulsar synthesis [Roads2001], developed by Curtis Roads and Alberto de Campo, consists of a short, arbitrary waveform followed by a length of silence, and the two are then alternated and looped to form a periodic oscillation at the fundamental frequency. The short waveform may be played back at any speed. The reciprocal of the length of the pulsaret in seconds is usually heard as a formant.

See my post titled “Pulsar Synthesis.”

Raster scanning synthesis

Added 2022-02-27.

This is a form of time-domain image sonification where an image’s color values are translated directly into digital sample values and unraveled in reading order [Yeo2006]. The strong correlations between successive rows of a typical image result in an audible pitch period with evolving timbre. Images can be sourced from photos, drawings, live footage, etc. or generated algorithmically. This method is a cousin of scanned synthesis.

Raster scanning can also be used for converting periodic sounds to images, applying image filters to sounds, or applying audio effects to images.

Scanned synthesis

Scanned synthesis is wavetable synthesis where the wavetable is taken from a physical model that evolves slowly (not the audio-rate physical models of instruments) [Verplank2000]. The wavetable can also come from a physical interface or even video. In a way, scanned synthesis is sonification of any evolving entity as a wavetable. For an impressive application of two-dimensional scanned synthesis in a multitouch context, see Robert Tubb’s Wablet [Tubb2011].

Spatial matrix synthesis (SM synthesis)

Added 2025-02-09.

SM synthesis produces an ambisonic signal, which is produced by panning a signal around a 3D environment extremely rapidly using audio-rate panning signals. [Schmele2024] If the input signal and panning signals are all audio-rate oscillators at simple pitch ratios, we get pitched tones similar to the FM/PM/ring modulation family. Unlike those methods, SM synthesis produces tones that are “spatial by design;” i.e., the ambisonic sound field is inherent to the synthesis method.

Subtractive synthesis

Added 2023-12-17.

“Subtractive synthesis” can refer to any use of a filter to shape timbre, but here I’m talking in the narrower sense about subtractive synthesis found in analog synths (and synths that are analog at heart). The sound sources here are those for which VCOs have practical implementations: sawtooth, pulse with PWM, triangle, sine. The filter is traditionally a resonant lowpass filter, but many synths offer bandpass and highpass options.

Tropical additive synthesis

Added 2022-12-14.

Tropical additive synthesis starts with a set of sine wave oscillators of amplitude 1, adds user-controllable dc offsets to each, and takes the minimum of all the signals at every point in time [Bocci].

If the partials are tuned a harmonic series or a subset of it, the waveform is periodic just like ordinary additive synthesis. Unlike additive synthesis, the perceived timbre is highly sensitive to the relative phases of the partials. Tropical additive synthesis tends to produce rich, buzzy spectra resembling a fuzz pedal on an electric guitar.

Vector phase shaping (VPS)

This is a take on phase distortion that adds a second dimension of timbral control [Kleimola2011]. It can generate formants.

Vector synthesis

Added 2023-02-15.

Vector synthesis is a variant of wavetable synthesis first developed in the Prophet VS. It employs four static wavetable oscillators running in parallel, allowing the user to pick from many built-in waveforms. The four oscillators are associated with the four corners of a square, and a two-dimensional control mixes between them. This control can be modulated manually with a joystick or automatically with separate LFO and envelope controls for the X and Y coordinates. The oscillators may be individually detuned and/or modulated in pitch by oscillators.

The mixed oscillator is typically processed with a filter.

Vocal FM synthesis

This patented method modifies FM synthesis to be used as a formant synthesizer [Chafe]. Technically, FM can already be used to synthesize formants, by using both sine waves for the carrier and modulator and setting the modulator frequency to be a large multiple of the carrier frequency, but Chafe’s research fixes issues when the fundamental or formant is modulated over time.

VOSIM

Smooth pulses of varying lengths and amplitudes, formed by the function sin(x)^2, are concatenated to produce formants [Kaegi1972]. This makes VOSIM useful as a rudimentary vowel synthesizer.

Walsh function oscillators

Added 2024-01-30.

The Walsh-Hadamard Transform (WHT) is bit like the Discrete Fourier Transform, but sine waves are replaced with Walsh functions, which are a certain set of digital signals with only sample values of -1 and 1. Some, all though not all, of the Walsh functions are square waves at octaves of a fundamental. Any digital signal with a power-of-two number of samples can be represented as a linear combination of Walsh functions. Thus a Walsh function additive synthesizer naturally results from this, and many standard additive synthesis formulas can be ported over to Walsh functions. See [Tillman2020].

A sawtooth wave has a particularly simple WHT, allowing it to be expressed in terms of square waves. Some hardware synthesizers using DCOs (digitally controlled oscillators) synthesized their saw waves using such a sum of square waves, but did so imperfectly, and the flaws in the resulting saw wave are quite unique to this synthesis method [Gabrielli2019].

Wave Sequencing Synthesis

Added 2023-01-01, updated 2024-01-02.

The Cheetah MS800 is an obscure and not well-liked synthesizer released in 1989, marketing its synthesis method as “Digital Wave Sequencing Synthesis.” The device is a wavetable synthesizer with 20 built-in single-cycle waveforms. Evolving timbres are created with a sequence of commands of the form “crossfade to wavetable #X over Y seconds.” There is also a GOTO command which allows creating a loop point in the command sequence, and a specially designated command that is jumped to when a MIDI note is released.

Unlike other wavetable synthesizers, the MS800 has no filter at all, so timbral variation can only be generated by crossfading between wavetables. While the Cheetah presents the purest form of this, the core idea of crossfading among wavetables is also found in the Keytek CTS-2000, and some Ensoniq synths including the SD-1 and TS-10 (thanks miunau for the tip).

Wavetable synthesis

Added 2023-12-17.

The original meaning of “wavetable synthesis” meant a waveform produced by playing back a single buffer with repetition to produce a perfectly periodic signal. With the rise in popularity of Massive and Serum, the popular meaning of “wavetable synthesis” generalized to a two-dimensional buffer, allowing crossfading between a continuous bank of one-dimensional wavetables and therefore adding a dimension of timbre control. I call them “static wavetable synthesis” and “variable wavetable synthesis” to disambiguate them.

This is a very general way to synthesize an oscillator. A common choice is to use wavetables sourced by sampling audio. Serum offers the option to provide an audio sample and chop it up into a 2D wavetable. Wavetables can also be created from scratch with various editing methods, one of the most interesting ones being 2D wavetables based on mathematical formulas.

Wavefolding

A waveform, usually a sine wave, is waveshaped with a continuous but nonmonotonic nonlinear transfer function. Timbre is evolved by changing how hard the nonlinearity is driven.

Zing modulation

Zing modulation, as named in Rossum Electro-Music’s Trident/Triton, is just multiplying two oscillators together in the time domain, with the stipulation that both input signals are oscillators tuned to a simple ratio. It is therefore a special case of ring modulation.

Non-qualifiers

These techniques are worth a mention, but don’t really make the list for me:

Sophisticated physical modelling methods like modal synthesis, waveguide synthesis, and finite difference methods are highly general methods and can synthesize both pitched and unpitched tones. I guess that’s also true of many methods that are on the list, but see Rule 2.
A number of synthesizers use a sampled transient together with a standard oscillator for instrument body. There are some marketing terms for this out there like “linear additive synthesis” and “vowel-consonant synthesis.” Combining sampled onsets with oscillators a smart idea, but it’s not really a novel kind of oscillator in its own right. My bias against samples may be speaking here though.

References

Bocci
Bocci, C. and Sancristoforo, G. Date unknown. “Tropical Additive Synthesis.”
Casio
--. Date unknown. Casio CZ-1 Operation Manual.
Chafe
Chafe, Chris. Date unknown. “Glitch Free FM Vocal Synthesis.”
Chareyron1990
Chareyron, Jacques. 1990. “Digital Synthesis of Self-modifying Waveforms by Means of Linear Automata.”
Gabrielli2019
Gabrielli, Leonardo et al. 2019. “Analysis and Emulation of Early Digitally-Controlled Oscillators Based on the Walsh-Hadamard Transform.”
Gillet2015
Gillet, Emilie. Source code to Mutable Instruments Braids.
Heikkilä2011
Heikkilä, Ville-Matias. 2011. “Discovering novel computer music techniques by exploring the space of short computer programs.”
Hohnerlein2016
Hohnerlein, Christoph; Rest, Maximilian; and Smith, Julius O. 2016. “Continuous Order Polygon Waveform Synthesis.” Proc. Int. Computer Music Conf.
Kaegi1972
Kaegi, Werner and Tempelaars, Stan. 1978. “VOSIM – A New Sound Synthesis System.” Journal of the Audio Engineering Society, Vol. 26, No. 6.
Kleimola2008
Kleimola, Jari. 2008. “Audio Synthesis by Bitwise Logical Modulation.”
Kleimola2011
Kleimola, Jari et al. 2011. “Vector Phaseshaping Synthesis.” Proc. 14th Int. Conf. on Digital Audio Effects (DAFx-11).
Moorer1975
Moorer, James A. 1975. “The Synthesis of Complex Audio Spectra by Means of Discrete Summation Formulae.”
Pekonen2008
Pekonen, Jussi. 2008. “Coefficient-Modulated First-Order Allpass Filter as Distortion Effect.”
Radna2023
Radna, Raphael. 2023. “Dynamic Stochastic Wavetable Synthesis.”
Roads2001
Roads, Curtis. 2001. Microsound.
Rodet1984
Rodet, Xavier. 1984. “Time-Domain Formant-Wave-Function Synthesis.”
Schmele2024
Schmele, Timothy and Garriga, Adan. 2024. “Spatial Matrix Synthesis.”
Serra1993
Serra, Marie-Hélène. 1993. “Stochastic Composition and Stochastic Timbre: GENDY3 by Iannis Xenakis.” Perspectives of New Music, Vol. 13, No. 1 (Winter, 1993), pp. 236-357.
Tillman2020
Tillman, J. Donald. 2020. “Exploring Walsh Functions.”
Tubb2011
Tubb, Robert. 2011. “The Wablet: An Investigation into Scanned Synthesis and Multi-Touch.” Queen Mary University of London MSc project report.
Verplank2000
Verplank, Bill; Matthews, Max; and Shaw, Robert. Circa 2000. “Scanned Synthesis.”
Yeo2006
Yeo, Woon Seung and Berger, Jonathan. 2006. “Raster Scanning: A New Approach to Image Sonification, Sound Visualization, Sound Analysis And Synthesis.”

IXA Synthesis

Nathan Ho — Fri, 28 May 2021 00:00:00 +0000

Vintage synth hacker and generative electronic music producer acreil, a.k.a. Scott Nordlund, posted a video breaking down a PD patch he made. Go watch it now. Of the delightful DSP tricks to be found within, the most delightful to me is his review of the Casio CTK-1000, and the unearthing of its synthesis method.

The Casio CTK-1000 is a 1993 sythesizer and rompler whose manual boasts a new type of synthesis called “IXA Sound Source,” distinguished from phase distortion synthesis from the CZ series and “interactive phase distortion” or iPD from the Casio VZ series. These are largely ways of getting around Yamaha’s all-dominating FM synthesis patents.

Nordlund dug up Casio’s patent US5164530, and while we have no way of telling what’s really inside the CTK-1000, the document gives us useful clues. (There are three other patents, but none of them seem to contain much interesting information beyond this one.) The patent describes a phase modulation synthesis method where the carrier is a triangle wave and the modulator is a sine wave. However, the intrinsic phase of the triangle wave is distorted with a weird waveform made of sine wave segments such that a sine wave is produced if the modulation index is zero. Here’s a block diagram:

and a closer look at the weird waveform:

For readers who prefer a more formal description, define a pulse wave (\Pi_x(t)) as a function with period (x) such that (\Pi_x(t) = 1) if (0 \leq t < x / 2) and (\Pi_x(t) = 0) if (x / 2 \leq t < x). Define a triangle wave (\Delta(t)) as a function with period (4) such that

[\begin{aligned} \Delta(t)

\begin{cases} t & 0 \leq t < 1 \ 2 - t & 1 \leq t < 3 \ t - 4 & 3 \leq t < 4 \ \end{cases} \end{aligned}]

Then the weird waveform (W(t)) with period (1) is defined as:

[W(t)

(2 \Pi_{1/2}(t) - 1) \sin\left[2\pi,\text{mod}\left(t, \tfrac{1}{2}\right)\right]

2 \Pi_{1/2}\left(t + \tfrac{1}{4}\right)
2 \Pi_1\left(t + \tfrac{1}{2}\right)]

An IXA oscillator with period (1), index (n), and ratio (r \in \mathbb{N}) is defined as:

[\text{IXA}(n, r, t)

\Delta\left( W(t) + n \sin(2 \pi r t) \right)]

It can be seen that (\text{IXA}(0, r, t) = \sin(2 \pi t)), i.e. turning off the modulator results in a sine wave.

IXA synthesis, as it could be called, produces unique buzzy waveforms with a deep and growly sound even with low index:

It sounds a bit like wavefolding. Classic two-operator sine wave FM, by comparison, sounds more muted when the index is low and squelchy when the index is high:

Here’s a spectrogram of partials for IXA with ratio set to 1:

Here is two-operator FM synthesis with sine waves for comparison. Note the change of scale in the X-axis – FM requires a higher modulation index to achieve sounds comparable in brightness to IXA.

The amplitude scale is linear in both graphs. Notice how the FM synthesis, whose harmonics follow one of the Bessel functions, has a sort of “resonance” at increasing even harmonics and quickly drops off to near-zero above that. IXA synthesis produces smoother spectra with less obvious resonance, and if you look closely there is some high-frequency fuzz above the dropoff.

IXA also works with non-integer ratios, but to me these sound harsh, like a bell FM patch run through nonlinear distortion:

Slight detuning of the modulation frequency, however, produces a pleasant beating effect.

The patent also touches on some variations. One is to pick a different (W(t)) so it no longer produces a sine wave when the modulation index is zero. I have experimented with this a little, but haven’t gotten anything good out of it. Another, more interesting, option is to open up the modulator to be the output of another IXA synthesis oscillator, or the output of the current oscillator in a feedback configuration. Chaining two or three IXA oscillators seems to automatically produce brostep bass waveforms, but I’m not sure if it’s useful for much else:

The way I see it, three or more operators are useful in FM to access richer sounds, but a single IXA operator is already complex enough and additional operators are limited in use.

Generally, anywhere FM or PM is used, it’s worth trying to substitute IXA to see how it sounds. For example, IXA produces formant-like results when setting a high modulation ratio, so FM vocal synthesis may apply to IXA as well.

SuperCollider code

Here’s a function you can copy-paste into your own SC patches:

// Public Domain

(
var ixa;

ixa = { |freq, in, index|
    var phase, waveform, snd;
    phase = Phasor.ar(Impulse.ar(0), freq / SampleRate.ir, 0, 1);
    waveform = (phase % 0.5 * 2pi).sin * (2 * (phase % 0.5 < 0.25) - 1);
    waveform = waveform + (2 * (phase % 0.5 >= 0.25));
    waveform = waveform + (2 * (phase >= 0.5));
    snd = (waveform + (in * index)).fold2;
    snd;
};

{
    var freq = 100;
    ixa.(freq, SinOsc.ar(freq), Line.kr(0, 10, 3)) * 0.5 ! 2;
}.play(fadeTime: 0);
)

Circular Membrane Modes

Nathan Ho — Fri, 01 Jan 2021 00:00:00 +0000

Just as the modal frequencies of an ideal string produce the overtone series, other ideal acoustic objects such as circular membranes produce their own sets of frequency ratios that we can think of as “non-Pythagorean overtone series” from an alternate universe. Like the classical overtone series, they arise naturally from physical laws and produce an infinitely ascending microtonal chord. These modal frequencies are found in physics textbooks, but I haven’t seen musicians talk about them much, nor anyone represent them in music notation like we do the standard overtone series.

Modal frequencies of a circular membrane are labeled by two mode numbers, overall forming a two-dimensional grid of frequency ratios rather than a linear series. For example, the lowest mode of a circular membrane is traditionally notated “0,1”. Just as the first and lowest classical overtones are the important ones, it’s mostly the low mode numbers that receive much attention from mechanical engineers. In the below charts, these are the tones in the upper left region.

I’ve notated the frequencies here on three levels. Read the note head position and sharp/flat/natural for the nearest 12ET pitch, and the up/down arrows indicate 50-cent alterations to a finer 24EDO approximation. The fearless can ignore all arrows and read the signed numbers for the cent deviations from 12ET (not 24EDO!). Modal synthesis decouples fundamental frequency from frequency ratios, so the choice of C2 as fundamental is arbitrary, and all charts are transposable without affecting their faithfulness to the physical model. Try them out on your instrument of choice – even the crude 12ET approximations generate some lovely chords that remind me a little of Morton Feldman.

Click the screenshot below to download a PDF version.

Is there anything perceptually important about this series? I don’t think so. You could replace these chords with random pitches, and the results would likely sound similar to an average listener. But as a demonstration of complex microtonal harmonies arising from first principles of acoustics, I think it’s conceptually compelling and a fruitful source of harmonic inspiration.

Venn 7

Nathan Ho — Thu, 31 Dec 2020 00:00:00 +0000

I have released Venn 7, a little artistic Web app that turns symmetric 7-fold Venn diagrams into a musical interface. Click the screenshot below to visit the app, which also has an attached explanatory article.

There are some bugs and minor issues with it which I hope to hammer out as I receive feedback. Check the repository for bug reports and some technical documentation.

Chip Fuzzing Synthesis

Nathan Ho — Sun, 22 Nov 2020 00:00:00 +0000

I’m unsure whether I read about this or dreamt it (this year has been a blur) but I recall someone fuzzing a retro sound chip, most likely the Yamaha OPL3, by sending it random bits for its synthesis parameters and recording the output. Drawing from this, we can explore “chip fuzzing synthesis,” the art of feeding total digital randomness into a synthesis algorithm and seeing what comes out.

There is no specific need for a real retro sound chip or even an accurate emulation of one, but it helps to understand how some old sound chips operate to look for inspiration. As an example, we can look at the Commodore 64’s SID. This chip is an analog subtractive synthesizer, providing three oscillators with frequency inputs, waveform selection (saw, pulse, triangle, noise), and ADSR envelope generators, all mixed into a filter with controllable cutoff and famously nonfunctioning resonance.

The parameters of the SID are controlled by an internal set of 32 8-bit registers, which are written to using a 5-bit parallel address bus and an 8-bit parallel data bus. In C-like pseudocode, communication with the SID can be emulated like so:[1]

char sidRegisters[32];

// Parallel ports used to communicate with SID.
char addressBus = 0;
char dataBus = 0;

void writeSid(char address, char value)
{
    addressBus = address & 31;
    dataBus = value;
    sidRegisters[addressBus] = dataBus;
}

The SID interprets the sidRegisters array and maps various bits and bytes to analog synth parameters. For example, registers 0 and 1, taken as a 16-bit integer, control the frequency of an oscillator, and individual bits in register 4 select the waveform and enable ring modulation and hard sync.

Fuzzing the address and data buses is the equivalent of calling writeSid repeatedly with randomized address and value, writing random data to random registers. The exact rate at which random data is written is up to you. I find that slow randomization produces the most coherent results and has the least chance of turning the output into white noise. A few hundred times a second is a good start.

It also suffices to take a simpler route and feed high-frequency random noise (sample-and-hold, maybe) into every parameter of a synth. Again, we don’t need a vintage emulation at all – a minimal subtractive monosynth with waveform selection, ADSR envelope, and a few switchable filter types is adequate to get glitchy sounds. So here’s a little patch:

This is sonically uncompromising (and hey, maybe that’s your thing), but still makes a useful raw source for more polished sound design. Here’s the same patch as above with some minor modifications and a lot of post-effects like granulators, distortion, and reverb:

The outcome of chip fuzzing synthesis is highly dependent on the choice of synthesis algorithm, the set of parameters, and the ranges for said parameters. I can imagine fuzzing FM, subtractive, additive, physical modelling, and parameters of an effects chain. The more inputs to fuzz, the better – especially inputs that switch features on and off, exhibit complex interactions with other inputs, and/or unearth bugs and artifacts.

The above code may seem on the vacuous side, but writing it out this way allows for transformations that fall under the umbrella of bit bending. For example, performing a bitwise AND on the addressBus or dataBus is the equivalent of disconnecting pins of the chip. Similar bitwise operations permit shorting pins to 1, rewiring pins to other pins, downsampling individual pins, etc. Bit bending operations can be used in conjunction with chip fuzzing, or for glitching out “normal” musical input.

Low Battery Audio Effects

Nathan Ho — Mon, 25 May 2020 00:00:00 +0000

Searching YouTube for videos of low battery toys and keyboards brings up results like “Demon Possessed Singing Trout.” Please watch the video before proceeding.

I have a limited understanding of electronics, but a compulsory need to explain this phenomenon due to tech blogger ego syndrome. A low battery has an abnormally high internal resistance, causing its voltage to sag in response to the loads it’s supporting. If it’s powering multiple things, they will interact in strange ways. The distorted audio from the singing fish sounds like the clock rate is dropping in reaction to the load of the speaker. (The servo motors might also be causing voltage sag, although it isn’t entirely clear from the video.)

The speaker/clock interaction is interesting since it works in a feedback loop: the clock controls the playback rate, and the amplitude of the output audio draws current that affects the clock. This inspires a general method for turning an audio algorithm into a “low battery” version:

Run a DSP algorithm such as sample playback, synthesizer, effect, etc. that can be operated at a variable clock rate.
Apply filters like a full-wave rectifier, envelope follower, or simple lowpass to simulate speaker load. Optional.
Apply a highpass filter to block dc. (This helps prevent the algorithm from getting stuck.)
Use this signal to control the clock rate of the DSP algorithm, so that a signal of higher amplitude lowers the clock rate.

The casual experiments I’ve done with this are promising. At subtle settings, this creates wandering, droopy pitch bends. Pushed to the extreme, it produces squelchy signal-dependent distortion. I especially like its effect on percussive signals, where louder transients are stretched out and any rhythmic pulse becomes irregular. I’m imagining software plugins that emulate digital hardware could be augmented with a “battery” knob that lets the user control how much the clock rate sags in response to the output signal.

SuperCollider Tips

Nathan Ho — Mon, 23 Mar 2020 00:00:00 +0000

Here are some coding tips that may be of use for people using the SuperCollider language. Mostly a “stuff I wish I knew years earlier” post.

For some softer advice concerning broader technical and artistic use of SC, I have a new blog post called “Opinionated Advice for SuperCollider Beginners.”

Updated 2023-03-16.

Levels management and volume safety

Added 2023-03-16.

I’m putting this one first because it’s very, very important!

SC is notorious for producing sounds that can damage your hearing. Luckily, accidents are preventable, and it’s 100% possible to practice safe SuperColliding by getting into good habits. I have written the article “Levels and volume safety” on scsynth.org as an extensive guide to managing levels in SC.

A quick summary:

If you are on macOS, upgrade to at least SC 3.12 right dang now and add Server.default.options.safetyClipThreshold = 1 to your startup file so the audio output clips.
Work at low levels in your system’s volume control, but high levels in SC. If your audio is at a comfortable level and peaks in SC at -6 dBFS, the loudest sounds can only peak at 6 dB louder than that, so a synthesis accident can be startling, but unlikely to be dangerous.
As a corollary: if SC produces a quiet signal, do not turn up the volume using your computer’s volume control! Instead, turn it up in SC.
Use a Limiter on the master bus.
Type in gain amounts as e.g. * -60.dbamp instead of * 0.001.

EnvGen, Env, and done actions

Here is a standard way to write an envelope generator:

env = EnvGen.kr(Env.adsr(0.01, 0.03, 0.7, 0.1), gate, doneAction: 2);
snd = snd * env;

There is nothing wrong with this, but there are nicer ways to write it. In 3.9 (?), SuperCollider added a bunch of constants like Done.freeSelf to improve readability:

env = EnvGen.kr(Env.adsr(0.01, 0.03, 0.7, 0.1), gate, doneAction: Done.freeSelf);
snd = snd * env;

Of course Done.freeSelf is just a constant 2, but I much prefer spelling it out to make the intentions more clear. In practice, I only use Done.freeSelf and Done.none. Check the Done helpfile for the full list.

In 3.7, a particularly nice shortcut was added, the instance method Env:-kr:

env = Env.adsr(0.01, 0.03, 0.7, 0.1).kr(Done.freeSelf, gate)
snd = snd * env;

Note the order of arguments – done action first, then gate. Internally, this creates an EnvGen and does the exact thing as above, but saves you typing and looks nicer. After I started writing envelope generators this way, I’ve never looked back.

One final bit of advice I have for envelope generators: always default to writing them in audio rate (ar) and not control rate (kr). Many real-life envelopes, such as percussive signals or Ryoji Ikeda, can get fast enough that the downsampling audibly loses sharpness. I write this:

env = Env.adsr(0.01, 0.03, 0.7, 0.1).ar(Done.freeSelf, gate);
snd = snd * env;

kr technically saves you some CPU, but unless you’re actually running into performance issues with your SynthDefs, this isn’t something I’d worry about. Get into the habit of using ar, only use kr when you need it.

Alternate symbol syntax in arrays

The code [a: 3, b: 4] does the same thing as [\a, 3, \b, 4]. This is particularly nice when writing arguments Synth:

synth = Synth(\foo, [\freq, 440, \amp, 0.1]);
synth = Synth(\foo, [freq: 440, amp: 0.1]);

You can run sclang from the command line

SC can be run without the IDE like a “normal” programming language. Just run sclang script.scd at the command line.

script.scd must be formatted in a special way. Rather than the usual floating chunks of code delimited by parentheses, the script must be a single code block executed all at once. This is equivalent in the IDE to booting the interpreter and immediately running Language → Evaluate File.

Let’s look at an example. If you have IDE code like this (with apologies for the terrible “music”):

// Run this first:
Server.default.boot;

// Run this second:
(
SynthDef(\sine, {
    var snd;
    snd = SinOsc.ar(\freq.kr(440));
    snd = snd * Env.perc(0.01, 0.3).ar(Done.freeSelf);
    Out.ar(0, snd);
}).add;
)

// Run this third:
(
Routine({
    10.do {
        Synth(\sine, [freq: exprand(100, 8000), amp: 0.1]);
        rrand(0.1, 0.5).yield;
    };
}).play;
)

To run this from the command line, condense all code blocks into a single block that is the equivalent of running all in sequence:

Server.default.waitForBoot {
    SynthDef(\sine, {
        var snd;
        snd = SinOsc.ar(\freq.kr(440));
        snd = snd * Env.perc(0.01, 0.3).ar(Done.freeSelf);
        Out.ar(0, snd);
    }).add;
    Server.default.sync;
    Routine({
        10.do {
            Synth(\sine, [freq: exprand(100, 8000), amp: 0.1]);
            rrand(0.1, 0.5).yield;
        };
        0.exit;
    }).play;
};

The method Server:-sync, in a Routine, is necessary since SynthDef:-add works asynchronously. If your code consists of SynthDef definitions and then Synth instantiations, you should put a Server:-sync in between them to avoid race conditions. Note that Server:-sync must take place in a Routine (in this case, waitForBoot already wraps our code in a Routine).

You have to write 0.exit manually at the completion of your piece. sclang doesn’t do the node.js thing where it exits automatically when there are no more outstanding asynchronous tasks.

This refactoring process might seem like an enormous pain in the neck, and a complete upheaval of the standard workflow in SuperCollider. I’d make a case that the first code example is “wrong” from a software engineering standpoint. Splitting a linear program across multiple, interactively executed code blocks and guessing the timing of asynchronous tasks simply isn’t a good programming practice, and proper asynchronous programming is needed if you want to run sclang as a command-line application.

That said, instantaneous feedback on your code (in particular, without having to restart sclang and scsynth) is tremendously helpful in a music programming environment, and a big reason why the IDE’s nonlinear execution style is so dominant. Using the IDE trades off proper coding structure for the creative benefits of interactivity. I tend to write in the nonlinear style when making music, and the more structured linear style when designing something more robust and application-like.

NamedControl

I wrote a post on the scsynth.org forums about NamedControls, why they are superior in multiple ways to the typical way to write SynthDefs. Instead of this:

// Old way
SynthDef(\sine, {
    |freq = 440, amp = 0.1, out = 0|
    Out.ar(out, SinOsc.ar(freq) * amp);
}).add;

Write this:

// New way
SynthDef(\sine, {
    Out.ar(\out.kr(0), SinOsc.ar(\freq.kr(440)) * \amp.kr(0.1));
}).add;

This seems like a purely surface-level change, and maybe even a step backwards since you no longer automatically have a nice summary of the arguments at the top of the SynthDef. However, NamedControls have tremendous advantages in readability over argument-style SynthDefs when you start using rate specifiers:

// Old way 1
// WARNING: names of actual arguments are "freq", "amp", and "out"!
// Also, don't accidentally name any argument starting with "a_", "k_", "i_", or "t_" or
// you'll experience difficult-to-diagnose bugs!
SynthDef(\sine, {
    |a_freq = 440, k_amp = 0.1, i_out = 0|
    Out.ar(i_out, SinOsc.ar(a_freq) * k_amp);
}).add;

// Old way 2
// WARNING: remember to update the "rates" array if you add, remove, or reorder arguments!
SynthDef(\sine, {
    |freq = 440, amp = 0.1, out = 0|
    Out.ar(out, SinOsc.ar(freq) * amp);
}, rates: [\ar, nil, \ir]).add;

// New way
SynthDef(\sine, {
    Out.ar(\out.kr(0), SinOsc.ar(\freq.kr(440)) * \amp.kr(0.1));
}).add;

Or multichannel arguments:

// Old way
SynthDef(\sine, {
    |freqs = #[440, 440, 440, 440, 440, 440, 440, 440]|
    ...
}).add;
// WARNING: attempting to abbreviate as "freqs = (440.dup(8))" will cause silent failure!

// New way
SynthDef(\sine, {
    var freqs = \freqs.kr(440.dup(8));
    ...
}).add;

As you can see with the warnings I’ve peppered around here, the old style has some nasty surprises lying in wait. I go into more detail on these arguments in the forum post, so go check it out.

mul/add

The mul and add arguments on UGens exist entirely for historical reasons, back when the compile-time optimization of SynthDefs was less smart. Nowadays, you can simply use the * and + binary operators, at no CPU penalty:

SinOsc.ar(440) * mul + add

This seems much more readable to me than

SinOsc.ar(440, mul, add)

since you don’t need to memorize the number of arguments that SinOsc expects. Ah hell, I forgot the phase argument. I meant SinOsc.ar(440, 0, mul, add).

mul and add also have a hidden surprise. Their behavior in multichannel expansion is inconsistent with regular UGen arguments:

{ LFNoise0.ar(freq: [100, 100], mul: 1) }.plot(0.2); // two different noise signals
{ LFNoise0.ar(freq: 100, mul: [1, 1]) }.plot(0.2); // two of the same noise signal

Internally, they are expanding differently:

{ LFNoise0.multiNew(\audio, [100, 100]) * 1 }.plot(0.2);
{ LFNoise0.multiNew(\audio, 100) * [1, 1] }.plot(0.2);

To avoid this trap, and improve the readability of your code, I recommend always using the binary operators.

Amplitude

Ever tried to use the Amplitude UGen for analysis, vocoding, compressor design, or whatever, and gotten weird or unexpected results?

env = Amplitude.kr(sig);

Amplitude has poor default settings, with both attack and release coefficients at 0.01. In most practical applications of envelope followers, you will want release significantly longer than attack. A good start would be:

env = Amplitude.ar(sig, 0.01, 0.1);

You will have to adjust these coefficients to fit your specific application. For analysis, you may want to crank the attack all the way down to 0 to maximize sensitivity and responsiveness. For vocoding, you may want slow, long-term attack/release like 0.1/1.0. Adjust until it sounds good.

Also, .ar vs .kr. There is generally no need to write Amplitude.kr and throw away sharp transient information. Use Amplitude.ar unless the optimization is specifically needed.

Always wrap `Synth.new` and `Synth:set` in `Server.default.bind`

Added 2023-03-16.

This tip was originally a thread on scsynth.org. This is an abridged version. For more discussion, see the thread.

If you venture outside the confines of the Patterns system, there is an important gotcha rarely covered in SC tutorials. Run these examples:

(
var s;
s = Server.default;
Routine({
    SynthDef(\ping, { Out.ar(\out.kr(0), (SinOsc.ar(440) * -5.dbamp * Env.perc(0.001, 0.1).ar(Done.freeSelf)) ! 2) }).add;
    s.sync;
    loop {
        Synth(\ping);
        0.05.wait;
    };
}).play;
)

(
var s;
s = Server.default;
Routine({
    SynthDef(\ping, { Out.ar(\out.kr(0), (SinOsc.ar(440) * -5.dbamp * Env.perc(0.001, 0.1).ar(Done.freeSelf)) ! 2) }).add;
    s.sync;
    loop {
        s.bind { Synth(\ping); };
        0.05.wait;
    };
}).play;
)

The first sounds jittery and uneven, but the second one sounds fine. The same is true of Synth:set:

(
var s;
s = Server.default;
Routine({
    var synth;
    SynthDef(\ping2, { Out.ar(\out.kr(0), (SinOsc.ar(440) * -5.dbamp * Env.perc(0.001, 0.1).ar(Done.none, \trigger.tr)) ! 2) }).add;
    s.sync;
    synth = Synth(\ping2);
    loop {
        synth.set(\trigger, 1);
        0.05.wait;
    };
}).play;
)

(
var s;
s = Server.default;
Routine({
    var synth;
    SynthDef(\ping2, { Out.ar(\out.kr(0), (SinOsc.ar(440) * -5.dbamp * Env.perc(0.001, 0.1).ar(Done.none, \trigger.tr)) ! 2) }).add;
    s.sync;
    s.bind { synth = Synth(\ping2); };
    loop {
        s.bind { synth.set(\trigger, 1); };
        0.05.wait;
    };
}).play;
)

The use of Server.default.bind is especially critical if you mix Pattern and non-Pattern code:

(
var s;
s = Server.default;
Routine({
    SynthDef(\ping, { Out.ar(\out.kr(0), (SinOsc.ar(\freq.kr(440)) * -5.dbamp * Env.perc(0.001, 0.1).ar(Done.freeSelf)) ! 2) }).add;
    s.sync;
    Synth(\ping);
    Pbind(\instrument, \ping, \freq, Pseq([660], 1)).play;
    (instrument: \ping, freq: 880).play;
}).play;
)

(
var s;
s = Server.default;
Routine({
    SynthDef(\ping, { Out.ar(\out.kr(0), (SinOsc.ar(\freq.kr(440)) * -5.dbamp * Env.perc(0.001, 0.1).ar(Done.freeSelf)) ! 2) }).add;
    s.sync;
    s.bind { Synth(\ping); };
    Pbind(\instrument, \ping, \freq, Pseq([660], 1)).play;
    (instrument: \ping, freq: 880).play;
}).play;
)

The first example attempts to play a Synth, a Pattern, and an Event at the same time. The Synth arrives early in the first example, while all are on time in the second example.

The client and server communicate by OSC. OSC messages, when in bundles, can be optionally adorned with a “time tag” that indicates the exact time when the message should be executed. If no time tag is specified, or the message is not in a bundle, the receiver must execute the OSC message as soon as it is received. A common use for time tags is to pre-empt OSC messages so their timing can be accurate instead of at the mercy of any inherent latency in OSC communication.

An unadorned Synth.new sends an /s_new message with no time tag, and so the server executes the OSC message whenever it’s received.

s.bind { ... } is shorthand for s.makeBundle(s.latency, { ... }). .makeBundle causes the Server object to temporarily change the behavior of sendMsg so that attempts to send new OSC messages instead add those OSC messages to a bundle. The function is immediately executed, and after it is completed, the OSC messages are scheduled s.latency seconds ahead. You can change s.latency if you want; the default of 0.2 is rather high. (s.latency is commonly misunderstood to be related to audio latency. It isn’t. In fact, the only place it is used is in OSC pre-emption, and scsynth isn’t even aware of it. Maybe it should have been called s.oscLatency?)

It is important to note that s.bind { ... }, despite having a callback function, is not asynchronous. The function is run immediately, and s.bind does not return until after the callback has completed execution.

The Patterns system – or more accurately, the default Event type – automatically runs .makeBundle. You can override this with the \latency key in the default Event type. Try setting it to nil in a pattern, which removes the time tag.

There is one case where you shouldn’t use s.bind { ... }: real-time interaction, such as from a MIDI controller, sensor, or another application. In such cases, it’s preferable to sacrifice timing accuracy for the sake of minimizing latency.

Preventing SC from automatically starting JACK on Linux

Added 2023-03-16.

On Linux using JACK, SC automatically tries to start a JACK server if none seem to be running, and that JACK server will shut down when SC does. This can be annoying if you’d rather start JACK yourself.

This isn’t SC’s fault since this behavior is the default when creating a client in the JACK API. Fortunately, JACK allows overriding this behavior by defining the environment variable JACK_NO_START_SERVER. If you want this to be the case for all JACK applications, you can add export JACK_NO_START_SERVER=1 to your ~/.profile. To make this setting SC-specific, "JACK_NO_START_SERVER".setenv("1"); in your startup file should do the trick.

Getting help

The longest-standing SC communities are the sc-users and sc-dev mailing lists. More recent, and less well known, are the web forum at scsynth.org and the Slack chat. If you are interested in participating in the community, I recommend checking them out.

An Analog-Style Frequency Shifter

Nathan Ho — Tue, 03 Mar 2020 00:00:00 +0000

In this post, I will give an overview of a well-known method for designing frequency shifter effects in analog electronics, and some notes on implementing a digital version.

Frequency shifting is a special case of single sideband (SSB) modulation, a device often used in radio electronics. SSB in telecommunications involves shifting audio frequencies by e.g. 10 MHz so they can be transmitted as radio waves, then shifting down 10 MHz at the receiver. For this reason, SSB is well-studied, and there are a few different ways to implement it. This post describes a design that has found use in electronic music applications, where the shift is small (up to ±20 kHz, and often in tiny shifts like 1 Hz). Wikipedia calls it a “Hartley modulator,” but I can’t find any source to corroborate that terminology.

A fantastic reference to look at for the design of an analog frequency shifter is [Haible1996], which provides a full set of schematics of a high-quality real-world analog unit. I’ll only provide details up to an abstract block diagram that can be easily digitized.

This post starts off pretty theoretical, with an attempt to explain how the frequency shifter works. Admittedly, this is an explanation more for myself than it is for others, but I hope it helps. If not, feel free to skip the first two sections and check out the following links which might explain it better: [Smith2007] [Boschen2016] [Boschen2020]

Ring modulation of real signals

A real audio signal (f(t)) has a Fourier transform satisfying (F(-\omega) = F(\omega)^*), i.e. all negative frequency content is the complex conjugate of the positive frequency content. An individual sine wave (\cos \theta t) produces Dirac delta-shaped spikes in the frequency spectrum at (\theta) and at (-\theta). To illustrate, here is a depiction of the absolute value of such a spectrum vs. angular frequency:

The Fourier transform has a convolution theorem, where convolving two signals in the time domain is identical to multiplying them in the frequency domain. Critical to understanding the frequency-domain behavior of ring modulation is the counterpart:

Multiplying two signals in the time domain is identical to convolving them in the frequency domain.

A ring modulator mutliplies a signal (f(t)) by a sinusoid (\cos \theta t).[1] By performing convolution of these two signals in the frequency domain, we get two sidebands: (f(t)) frequency shifted up by (\theta), and (f(t)) frequency shifted down by (\theta). This is because convolution with a shifted Dirac delta function is a horizontal shift.

Here’s what it looks like visually. The left diagram is the magnitude spectrum of some arbitrary real signal, and the right shows the magnitude spectra of the two shifted copies:

(Sorry about the goofy-looking diagrams in this post. I wanted to make them more professional, but that took too much work.)

The ring modulator is already almost a frequency shifter, but it produces two shifted copies. What if there was a way for a ring modulator to produce only one sideband? Hmmm…

Frequency shifting of analytic signals

Most audio signal processing concerns real signals, so it’s easy to forget that they are a subset of the larger domain of complex signals. A complex signal (A + jB) can be represented in analog or digital environments using two real signals, one for the real part (A) and the other for the imaginary part (B). In the frequency domain, complex signals are not beholden to the constraint that real signals have on negative frequencies. Their positive spectrum can be completely independent of the negative spectrum.

In frequency shifting, we are interested in a subset of complex signals called the analytic signals. A complex signal (f) is analytic if (F(\omega) = 0) for all (\omega < 0). [Smith2007] That is, the Fourier transform of (f) has no negative frequency content.

Why is this useful for frequency shifting? While real signals decompose into real sinusoids of the form (A \cos(\omega t + \phi)), the building block of an analytic signal is the complex sinusoid, which has the general form (A e^{j (\omega t + \phi)} = A \cos(\omega t + \phi) + A j \sin(\omega t + \phi)), where (A \geq 0), (\omega \geq 0), and (0 \leq \phi < 2\pi). Instead of the dual spikes of the real sinusoid, a complex sinusoid’s spectrum is a single spike at a positive frequency.

Since multiplication in the time domain is convolution in the frequency domain, multiplying by a complex sinusoid (\cos \omega t + j \sin \omega t) convolves with a single Dirac delta function at (\omega). The lack of negative frequency content means that only the upper sideband is generated. That’s a frequency shift by (\omega).

This even works if the complex sinusoid has negative frequency. Although the shifting process crashes into DC and generally produces a non-analytic signal, this isn’t an issue.

A ring modulator multiples a real signal by a real sinusoid, while a frequency shifter (or at least our specific design of it) multiplies an analytic signal by a complex sinusoid.

There is one fundamental property of analytic signals that we need to know before we proceed. The imaginary and real parts of an analytic signal have a special relationship: the imaginary part is the Hilbert transform of the real part of the signal. The Hilbert transform of a signal is a “constant -90-degree phase shift,” or more accurately, a -90-degree phase shift of all positive frequency content and a +90-degree phase shift of all negative frequency content (the latter is to ensure the signal stays real).[2] The Hilbert transform property of analytic signals can be easily proven from the definition of the Fourier transform.

Borrowing terms from telecommunications, analytic signals are often known as “quadrature signals” or “(I/Q) signals.” The real part of the signal is denoted (I) (for “in-phase”) and the imaginary part (Q) (“quadrature”).

Converting between analytic and real signals

So, nice. Analytic signals allow for easy implementation of a frequency shifter. So how do we convert from real to analytic signals and vice versa?

Converting from complex signals back to real signals is trivial. We just drop the imaginary part:

To convert from real to analytic, we can try using a Hilbert transform to construct the imaginary part:

Unfortunately, the Hilbert transform is noncausal, with an impulse response that tails infinitely in both directions. In digital settings, FIR filters are a common approach for approximating the Hilbert transform (with some latency that has to be compensated with a delay line in the (I) signal). The literature on this is extensive thanks to Software-Defined Radio applications: just search for “Hilbert FIR.”

In the analog domain, an IIR filter method exists for converting a real signal to an analytic one: a “90-degree phase-difference network.” Rather than directly implementing the Hilbert transform as a filter, a phase-difference network takes an input signal X and puts it through two parallel allpass filters, producing two signals A and B such that B is roughly the Hilbert transform of A. “Roughly” means that the phase shift of B relative to A is close to -90 degrees over a useful bandwidth, say 20 Hz to 20 kHz.

The allpass filters are formed from a cascade of one-pole one-zero allpass filters, each having the following transfer function:

[H(s) = \frac{s + p}{s - p}]

where (p) is real and negative, representing the location of the pole. How do we select the locations of the poles? The mathematics are beyond me, but there is a published algorithm for optimal design of such filters described in [Weaver1953] and elaborated on in a well-known document from ElectroNotes [Hutchins]. The algorithm receives minimum and maximum frequencies to define the operating bandwidth, and you can either specify the number of allpasses (“design a network with (n) poles”) or the error in the phase difference (“design a network with less than (E) degrees of deviation from 90 degrees”). I wrote up some Python/NumPy code for this, available at the end of the post.

If we specify the operating range as 20 Hz to 20 kHz, a phase difference network with 12 poles, six in each path, can achieve less than 2 degrees of deviation from 90 degrees in the audio frequency range. Here are the locations of the poles that I get (units in radians/second):

Poles, path A:
    -45.363318968136156
    -344.47613274064236
    -1402.0629595305616
    -5624.48591607002
    -22572.489209831336
    -100338.8419291972
Poles, path B:
    -157.380399635128
    -699.584653466801
    -2807.6107358762515
    -11262.951449077736
    -45841.68695841799
    -348108.72310368466

I skipped an important detail. Maybe A and B are almost 90 degrees apart, but how do they relate to the input signal? The relationship isn’t straightforward, so a graph will help. Both branches of the network are allpass filters, meaning that their gain is unity at all frequencies, so only their phase responses are worth looking at. Let’s plot the phase response of Allpass A and Allpass B:

Here’s the phase difference between Allpass A and Allpass B to show how close we are to -90 degrees:

Good enough for our needs. (Open question for readers: how low does this error need to be for it to be inaudible in the resulting effect?)

The Complete Frequency Shifter

We’ve now defined conversion processes between analytic and real signals, and we have all the tools needed to complete our frequency shifter:

Convert rom a real signal to an analytic signal (I + jQ).
Multiply by a complex sinusoid.
Convert the complex signal back to a real signal.

Note that the modulation step, the multiplication must be complex multiplication. If the input signal is (I + jQ), complex multiplication expands as follows:

[(I + jQ) (\cos \omega t + j \sin \omega t) = I \cos \omega t - Q \sin \omega t + j(I \sin \omega t + Q \cos \omega t)]

But since we’re immediately converting back to a real signal, the imaginary term doesn’t matter, and we just need to compute (I \cos \omega t - Q \sin \omega t). That completes the block diagram of our analog frequency shifter:

This is only a block diagram. Fleshing this out into actual analog schematics is an entirely different story, and I won’t claim any expertise in that. Again, just check out [Haible1996] for yourself.

Digitization

Analog one-pole filters are simple to digitize using the bilinear transform.

[H(z) = \frac{k + z^{-1}}{1 + k z^{-1}}]

where (k = \frac{p + 2f_s}{p - 2f_s}), where (f_s) is the sample rate and (p) is the location of the analog pole. Even without pre-warping, this achieves excellent agreement to the analog phase response, and it’s not worth fretting over inaccuracies in my opinion.

Antialiasing is a valid concern when working with digital frequency shifters. With positive frequency shifts, frequency content may be shifted above Nyquist. This can be countered by pre-application of a lowpass filter at (f_s / 2 - f) if frequency shifting upward by frequency (f).

Variations

We have a nice-sounding frequency shifter based on an established algorithm in signal processing. What twists can we add to give it some unique character?

In an analog design, the one-pole allpass filter is typically an active design containing an op-amp. Op-amps can be a source of nonlinearity, so this suggests a “dirty” frequency shifter with waveshaping after each allpass stage.[3] The sine wave modulator signal can also be made non-ideal with harmonic distortion or temperature drift.

I don’t think it’s worthwhile to do exact modeling of every nonlinearity and imperfection in an analog frequency shifter. All I am suggesting here is that effect design doesn’t need to stop at an idealized mathematical model, and that there is a potentially interesting space to explore in deliberately imperfect frequency shifters, especially using imperfections directly inspired by analog circuits.

Conclusion

Python/NumPy code for designing the phase difference networks and making the plots is available on GitHub. Admittedly I haven’t done a lot of work to verify its correctness, so please let me know if you discover errors.

Edit 2020-03-06: I mistakenly claimed that the phase responses of the allpass filters are nearly linear, completely forgetting that the X-axis of the plot is logarithmic. Oops. I’ll put up a graph of the group delay if I have time.

References

Actually

Boschen2016
Boschen, Dan. 2016. Answer on DSP StackExchange. https://dsp.stackexchange.com/a/31360
Boschen2020
Boschen, Dan. 2020. Answer on DSP StackExchange. https://dsp.stackexchange.com/a/63812
Haible1996
Haible, Jürgen. 1996. “JH FS-1 Frequency Shifter.” http://www.jhaible.info/tonline_stuff/hj_fs.html
Hutchins
Hutchins, Bernie. “The Design of Wideband Analog 90° Phase Differencing Networks Without A Large Spread of Capacitor Values.” http://electronotes.netfirms.com/EN168-90degreePDN.PDF
Smith2007
Smith, Julius O. 2007. “Analytic Signals and Hilbert Transform Filters.” https://ccrma.stanford.edu/~jos/r320/Analytic_Signals_Hilbert_Transform.html
Weaver1953
Weaver, Donald K. 1953. “Design of an RC Wide-Band 90-Degree Phase-Difference Network.”

This is of course idealized ring modulation. An analog diode ring modulator is a nonlinear device that doesn’t implement multiplication perfectly. A more technically accurate term might be “heterodyning,” but the term “ring modulation” feels more natural for me to use as a synthesizer enthusiast.
I almost wrote that the Hilbert transform is a type of allpass filter, but while researching this post I learned that this is not true! The Hilbert transform has a gain of 1 for all positive and negative frequencies, but a gain of 0 at DC.
Most op-amps in analog filters are in a negative feedback configuration, which usually means the saturation characteristic is close to hard clipping. Using soft clipping to simulate an op-amp may be a little romanticized, but if it sounds good, it is good.

Exploring Modal Synthesis

Nathan Ho — Mon, 16 Sep 2019 00:00:00 +0000

Modal synthesis is simple in theory: a method of synthesizing sound where an excitation signal is fed into a parallel bank of resonators [Bilbao2006]. Each resonator, known as a “mode,” has the impulse response of an exponentially decaying sine wave and has three parameters: frequency, amplitude, and decay time. It’s a powerful method that excels in particular for pitched percussion like bells and mallet instruments.

Despite its conceptual simplicity, modal synthesis has a daunting number of parameters to tune – three times the number of modes, to be exact – and setting them by hand isn’t practical or musically expressive. How are we supposed to get great sound design when we have so many degrees of freedom? In interface design parlance, what we want is a divergent mapping [Rovan1997] that takes a small number of controls, maybe four knobs, and maps them to the several dozen frequencies, amplitudes, and decay times that control the modal synthesizer.

While it is possible to reverse engineer modes from a recorded sample (see [Ren2012] for a particularly impressive application), I’m most interested in “parametric” modal synthesis where each model derives from mathematical formulas that allow the user to navigate around a multidimensional space of timbres. I found a number of such algorithms while shopping around the literature from both computer music and mechanical engineering, so I decided to summarize them in a concise reference for sound designers and composers.

Bear in mind that I’m absolutely not an expert in physics or mechanics, and any mistakes are my own. The interested reader can also check out the citations to explore how these formulas were derived, and maybe even delve into the physics to invent your own parametric modal synthesis formulas.

As with most of my blog posts, I expect to be continuously updating this article as I learn more about modal synthesis (and also to fix my mistakes, of which there are certainly many).

The essentials

A modal synthesis model is a collection of (N) modes, each characterized by a triple ((f_k, A_k, R_k)) consisting respectively of a frequency, an amplitude, and a loss factor. (k = 1,,2,,3,,\ldots) is known as the mode number.

The loss factor (R_k) is worthy of some explanation. In synthesizers, we usually think in terms of the decay time (\tau_k), which is defined as the amount of time the mode’s exponential envelope takes to decay from (1) to (1/e). In physics and electrical engineeering, (\tau_k) is known as the “time constant.” In modal synthesis models, I’ve found that it’s easier to think in terms of the reciprocal (R_k = 1 / \tau_k). It simplifies many of the formulas we’ll see here.

Modal synthesis is agnostic to pitch, since you can easily repitch any modal synthesis model by multiplying all the (f_k) by the same frequency ratio. In a synthesizer, pitch is typically a user input, so we only need to store the frequency ratios of each mode to some fundamental, (f_0), which may be arbitrarily defined as (f_0 = f_1). To represent the decoupling of frequencies and ratios, I will write equations like this:

[f_k \propto k]

where (\propto) means “proportional to.” This means that the right-hand side should be multiplied by a constant so that the desired pitch is synthesized. In this case, (f_k = k f_0).

Most computer music packages provide a resonator filter, but explaining where it actually comes from may be of interest. The conventional resonant filter in modal synthesis is designed by taking a two-pole complex resonator and digitizing it with the Impulse-Invariant Transform. The Bilinear Transform is avoided since it creates unnecessary frequency warping and renders our (f_k) inaccurately, and the resulting biquad filter also responds poorly to modulation of (f_k).

Stiff string

A string that is clamped at both ends can be described in ideal with the wave equation, which has modal frequencies (f_k) that are integer harmonics of a fundamental (f_0). That is, (f_k \propto k). This is pretty boring.

To make it less boring, we can stretch out the modes with an inharmonicity factor (B):

[f_k \propto k [1 + (k - 1) B]]

This formula I found in Mutable Instruments’ “Elements” Eurorack module [Gillet2015], which exposes (B) in a parameter called “stretch factor” ranging from -0.06 to 2. (B = 0) tunes the modes to the overtone series. Small, positive values of (B) on the order of (10^{-3}) will add some pleasant metallic qualities, and larger values will lose the sense of pitch. (B < 0) compresses the modes rather than stretching them, which sounds quite strange.

The above can be viewed as a fast approximation of the “stiff string” model in piano modeling literature [Bank2010]:

[f_k \propto k \sqrt{1 + B k^2}]

Loss Factors

You can try setting all your modes to the same decay time, but this rarely sounds good. Frequency-dependent damping is important for getting realistic sounds, since real objects have high-frequency losses as vibrations propagate through them. Most resources I have found employ a second-order decay model for loss factors [Chaigne1993A] [Stoelinga2011] [Chaigne1998].

[R_k = b_1 + b_3 f_k^2]

where (b_1) (units in Hz) and (b_3) (units in seconds) are constants. (b_1) controls the global decay time, while (b_3) controls how much high frequencies are damped. To get an idea of reasonable parameters, [Chaigne1993B] uses (b_1 = 0.5,\text{s}^{-1}) and (b_3 = 1.58 \cdot 10^{-10},\text{s}) for the middle C strings on a piano (note that the cited source uses angular frequency in the above expression, so I’ve divided (b_3) by (4\pi^2)). If you’re curious about the names for these loss factors, they derive directly from coefficients of the stiff string’s partial differential equations.

A side effect of this model in the context of a synthesizer is that low notes will have dramatically longer decay times than high notes. A good way around this would be to make (b_3) be a frequency-dependent loss factor for each mode, and introduce a new coefficient (b_0) controlling the loss factor by pitch:

[R_k = b_1 + b_3 f_k^2 / f_0^2 + b_0 f_0^2]

This way you can still get high-frequency damping, and different decay times for each note, and the two are decoupled.

Amplitudes

When a resonating body undergoes vibrations, it also has to radiate those vibrations into its environment for our ears or microphones to hear. Different objects have frequency-dependent radiation efficiencies. In [Stoelinga2011], a simple high-pass filter is used to emulate the effect where low frequencies are more difficult to radiate:

[A_k^\text{blue} \propto f_k]

“Blue” here indicates that we’ve imposed a 3 dB per octave spectral tilt identical to that of blue noise. In the end, the overall spectral shape of the modal synthesis model is in the hands of your excitation signal and whatever EQ decisions you use during mixing, so this isn’t that important of a term.

In the particular case of the stiff string model, there is a trick we can use that greatly enhances the realism, inspired by [Bank2010]. An excitation to a string will affect different modes differently depending on where it is done – for example, striking at a node of a standing wave won’t activate it, while striking it at an antinode will strongly activate it. This is why the position of plucking or bowing a string is critical for getting a pleasant sound out of an instrument.

Using (x) to indicate the excitation position from 0 to 1, a good approximation of its effects is:

[A_k^\text{pos} = \sin (\pi k x)]

The result is a more complex spectral envelope, which is good news. Even better, to simulate a human plucking or hitting a string, (x) can be randomized to create realistic timbral variation – even small random variations will sound like clearly different timbres. In the case of a piano or harpsichord where (x) is constant, it can at least be tweaked for each individual note for the best sound.

Note that, if we change the (A_k) to respond to changing (x), the multiplication by amplitudes should happen before the resonant filter. That is, the signal path should be excitation → multiply by (A_k) → resonant filter, not excitation → resonant filter → multiply by (A_k). In the latter case, shifting the exciter position will cause the amplitudes to suddenly change, which isn’t physically sensible.

The final amplitudes can be produced by multiplying these components: (A_k = A_k^\text{blue} A_k^\text{pos}).

Excitation

There are many ways to design excitation signals for a modal synthesizer, and I feel I haven’t personally explored all of them. Short, impulsive DC impulses with triangular-ish shapes seem useful enough for many practical purposes.

In real life, the interaction between the exciter and resonator is not one-way. It’s well-known in piano modeling that the string pushes back on the hammer and they undergo complex interactions. [Bank2010] does extensive modeling of the hysteretic effects. What is more important is the outcome – little oscillations in the excitation signal.

Rapidly modulated sine waves, or “chirps,” can make for interesting modal synthesis exciters.

Uniform beams

In the above stiff string model, “stiffness” is derived from a model of rigidity known as the Euler-Bernoulli beam theory. While the stiff string is a taut string with small amounts of beam-like rigidity, there are also rigid resonators like xylophone and marimba keys that have no string-like properties and can be handled with beam theory alone.

Beam theory will give you different harmonics depending on the boundary conditions. The most musically relevant case is the “free-free beam” with no clamping, which gives us harmonics of the form

[f_k \propto B^\text{free}(k)]

Here (B^\text{free}(k)) is defined as the (k)-th positive solution to the equation

[\cos(\pi x) = \frac{1}{\cosh(\pi x)}.]

If one end is clamped, a situation found in mbiras and toy pianos, we have a “cantilever beam”. This is the same as the above, but (B^\text{cant}(k)) is defined as the (k)-th positive solution to the equation

[\cos(\pi x) = -\frac{1}{\cosh(\pi x)}.]

(I don’t have a peer-reviewed source for these two equations on hand, but they’re well-known and can be found in undergraduate mechanical engineering courses.)

These two implicit equations don’t have analytical solutions. However, graphing (\cos(\pi x)) along with (\pm 1 / \cosh(\pi x)) shows us that the solutions quickly converge to the form (n + 0.5) for positive integer (n), and this convergence is very fast considering that (\cosh) exhibits approximately exponential growth.

It suffices to simply have the first few modes precomputed, and the rest approximated with integers + 0.5:

[\begin{aligned} B^\text{free}(1) &= 1.50561873 \ B^\text{free}(2) &= 2.49975267 \ B^\text{free}(k) &= k + 0.5 \quad (k \geq 3) \ B^\text{cant}(1) &= 0.59686416 \ B^\text{cant}(2) &= 1.49417561 \ B^\text{cant}(3) &= 2.50024695 \ B^\text{cant}(k) &= k - 0.5 \quad (k \geq 4) \ \end{aligned}]

These approximations beyond these five precomputed values are accurate to well within 0.1 cents, so any deviations from the true solutions should be inaudible. Sonically, both beam models conform to odd harmonics but with the first few modes out of tune.

To account for the different effects of the position of excitation, you can use the mildly complicated formulas for the shape of the free-vibrating beams. This seems overkill to me though, and I think the simple sine wave-based formula of the clamped string should be good enough for a realistic sound.

This model is for a beam with uniform thickness, which is not the case with the arched keys of xylophones and vibraphones.

Rectangular and circular membranes

Membranes are found mainly in drum heads, and are modeled using a two-dimensional wave equation, making them a generalization of strings. With a two-dimensional resonator, we drop the mode number (k) and replace it two dimensions, (m) and (n), both positive integers.

A rectangular membrane clamped at the border has modal frequencies

[f_{mn} \propto \sqrt{m^2 + (\alpha n)^2}]

where (\alpha \geq 1) is the aspect ratio of the rectangle, the ratio of the (m)-axis to the (n)-axis. Again, (\propto) indicates proportionality, and indicates that the right-hand side should be multiplied by the appropriate factor so that (f_{1,1} = f_0).

The most famous membrane equation is the circular membrane clamped at the edge, which has modal frequencies

[f_{mn} = J_{(m - 1)n}]

where (J_{(m - 1)n}) is the (n)-th positive root of the Bessel function (J_{m - 1}). Bessel function zeros can be found with, for example, SciPy’s scipy.special.jn_zeros.

Amplitudes that take into account the strike point are also two-dimensional now. In the case of the rectangular membrane, the effect on membrane modes at the strike point ((x,y)) is given by

[A_{mn}^\text{pos} = \sin(\pi m x) \sin(\pi n y).]

The excitation position effects for the circular membrane are somewhat more complicated. For music purposes, I don’t see any issues with simply reusing the rectangular formula – the sonic effect is basically the same.

Rectangular plates and tubular bells

Just as membranes use the two-dimensional wave equation, rigid plates use a two-dimensional analog of Euler-Bernoulli beam theory.

The vibrations of rectangular plates have been well-studied, e.g. in [Leissa1973]. It’s well known that a clamped rectangular plate has resonant frequencies

[f_{mn} \propto m^2 + (\alpha n)^2.]

Unfortunately, other boundary conditions like free plates result in partial differential equations with no known analytical solutions. However, if we simplify and treat each point of a plate as a superposition of two Euler-Bernoulli beams at right angles, we can work around this, sacrificing some accuracy. This was done in the tubular bell model of [Rabenstein2010], which arrives at

[f_{mn} \propto \sqrt{m^4 + (\alpha B^\text{free}(n))^4}]

Modifications to Rabenstein’s model show how to derive the modal frequencies for a tube clamped at one end:

[f_{mn} \propto \sqrt{m^4 + (\alpha B^\text{cant}(n))^4}]

Or a plate that’s completely free:

[f_{mn} \propto \sqrt{B^\text{free}(m)^4 + (\alpha B^\text{free}(n))^4}]

Be warned: the above two are my own work, so I can’t necessarily guarantee that they’re correct!

Efficiency considerations

Modal synthesis is somewhat computationally expensive. Moore’s Law has increased the usefulness of modal synthesis over the years, but it’s still worth discussing how it can eat up CPU cycles and how we can mitigate that.

The models that we’ve discussed generate an infinite sequence of modes, but only a finite number of them will be in the audible range. Theoretically, we should be able to remove any modes if (f_k \geq 20,\text{kHz}) (or the Nyquist limit if your sample rate is below 40 kHz), but this has two undesirable side effects:

If you continuously modulate pitch, some modes will cross the 20 kHz threshold and will zap in and out of existence. This will create discontinuities in the output signal.
Generally, lower pitches will require more modes, and you will get a different real-time CPU load depending on which note you play.

The first issue can be prevented by making a fake lowpass filter that smoothly crossfades modes to 0 as they approach 20 kHz. However, the second is a well-known issue in modal synthesis, and the best attack strategy depends on your design goals with regards to audio quality vs. CPU efficiency.

If deterministic CPU usage is a concern for you, it is probably wise to lose some bandwidth and simulate a fixed set of modes, say the lowest eight modes. It’s up to you to find out how much you can get away with before it starts degrading the output quality.

Standard strategies for reducing CPU cycles also apply. Modal synthesis doesn’t hurt particularly from downsampling. It’s also very ripe for vectorization and/or parallel computation on multiple threads.

Miscellany

Other physical models: If the resonances of an object can’t be easily solved with partial differential equations, you will have to use more powerful analytical tools like the Finite Element Method, which decomposes a body into a network of masses connected by springs, reducing mode identification to an eigenvalue problem. FEM is complicated in practice and I’m still trying to figure out how to implement it. MODALYS is one FEM tool designed for musicians, but it’s closed source, so I’m not really about that.

The Finite Element Method is not to be confused with the Finite Difference Method, where the model is discretized into a grid and simulated directly in the time domain.

909-style frequency sweep: Modal synthesis of drums benefits from a completely non-physical trick where the frequency is swept downward on the initial transient. This is an idea lifted from the Nord Drum, and reminds me of the kick drum in the Roland TR-909. My preferred envelope is (f(t) = f_0 [1 + e(t)]) where (e(t)) is an exponentially decaying signal.

Use as an audio effect: Modal synthesis isn’t just for impact percussion – a modal synthesis model can accept arbitrary audio input and take on the role of an effect, like a very metallic mini-reverb. In such cases, it’s usually a good idea to use significantly reduced decay times for better transparency, and/or introduce a dry-wet mix control.

Series and parallel: Used in Ableton Live’s Collision, multiple modal synthesis models can be combined. Layering is a given in sound design, and cascading multiple modal synthesizers can sound neat as well. You can do things such as simulating plucked strings attached to a wooden sound board.

Detuned copies: As seen in piano strings, Balinese gamelan, and EDM super saws, detuned copies of the same sound will thicken it and add evolution to the timbre. Modal synthesis in particular permits a computational shortcut as suggested in [Bank2010] – only the loudest modes require detuned copies to get the desired effect.

Feedback: This one’s easy – feed back the output of the summed modes into the excitation with a single-sample delay. Distortion or compression may be used in the feedback path.

Sum and difference tones: Modal synthesis is based entirely on treating resonances as linear filters, and doesn’t take into account nonlinear resonances that can happen in real objects. These are fiendishly hard to simulate, but there is a cool idea from [Bank2010] that allows us to simulate sum and difference tones that tend to arise from nonlinearities. For pairs of modes (f_j,,f_k), create a new mode with frequency (f_j \pm f_k), loss factor (R_j + R_k), and amplitude (A_j A_k). (Be aware that multiplying the amplitudes of the modes now makes this model sensitive to scaling factors, since that is how nonlinearities work.) To avoid adding hundreds of new modes, it’s best to be selective and only do this for a small number of modes, such as the loudest three.

Other modifications: There are limitless variations possible here. There’s plenty of ad hoc modifications to experiment with, like detuning a single mode or removing it entirely. I’ll dump some open questions I thought of while working on this article. Can the beam, plate, and membrane models be augmented with additional controls to bring them more degrees of freedom? Are there ways to hybridize modal synthesis with other physical modeling techniques like digital waveguide synthesis? Is it possible to interpolate between different modal synthesis models, e.g. a drum that slowly morphs into a tube?

Conclusions

I have a confession to make: I haven’t actually tried realizing and listening every idea in this article. I wrote it only so that I can gather all this knowledge in one place. In practice, turning a mathematical formula into music requires skill in sound design. Creating interesting sounds with modal synthesis and employing them in a quality work of music is left as an exercise to the reader.

Thanks to Josh Mitchell for inspiring me to finish this article, which languished as a draft for a long time. I’d also like to thank the authors of the various citations for graciously allowing me to use their formulas without their permission.

References

Bank2010
Bank, Balázs et al. 2010. “A Modal-Based Real-Time Piano Synthesizer.” IEEE Trans. on Audio, Speech, and Language Processing.
Bilbao2006
Bilbao, Stefan. 2006. “Modal Synthesis.” https://ccrma.stanford.edu/~bilbao/booktop/node14.html
Chaigne1993A
Chaigne, Antoine and Askenfelt, Anders. 1993. “Numerical simulations of piano strings. I. A physical model for a struck string using finite difference methods.”
Chaigne1993B
Chaigne, Antoine and Askenfelt, Anders. 1993. “Numerical simulations of piano strings. II. Comparisons with measurements and systematic exploration of some hammer-string parameters.”
Chaigne1998
Chaigne, Antoine and Doutaut, Vincent. 1998. “Numerical simulations of xylophones. I. Time-domain modeling of the vibrating bars.” https://asa.scitation.org/doi/abs/10.1121/1.418117
Gillet2015
Gillet, Émilie. 2015. Source code of Mutable instruments’ “Elements.” https://github.com/pichenettes/eurorack/blob/2a468734bdd53bea0f25165bf34d92a4c571db13/elements/dsp/resonator.cc#L64
Leissa1973
Leissa, A. W. 1973. “The free vibration of rectangular plates.” Journal of Sound and Vibration. https://www.sciencedirect.com/science/article/pii/S0022460X73803712
Rabenstein2010
Rabenstein, Rudolf et al. 2010. “Tubular Bells: A Physical and Algorithmic Model.” IEEE Transactions on Audio, Speech, and Language Processing.
Ren2012
Ren, Zhimin et al. 2012. “Example-Guided Physically Based Modal Synthesis.”
Rovan1997
Rovan, Joseph Butch et al. “Instrumental Gestural Mapping Strategies as Expressivity Determinants in Computer Music Performance.”
Stoelinga2011
Stoelinga, Christophe N. J. and Lutfi, Robert A. “Modeling manner of contact in the synthesis of impact sounds for perceptual research.” Journal of the Acoustical Society of America.

First Post

Nathan Ho — Thu, 03 Aug 2017 00:00:00 +0000

Welcome to my blog. This blog will have quality content, and that’s a guarantee.