graeme. hello

Spectrogram Phases

Sun, 28 Sep 2025 19:44:32 +0100

There was a tweet recently pointing out the apparent structurelessness of the phase data that is usually discarded when you compute audio spectrograms. People familiar with audio will know phase is important so this is pretty weird. So here’s quick post about what makes the phases so bad and some tricks to show there is structure to be found in them, but no promises any of it will be useful.

I’m going to use this track for all the plots in this post. It has a bunch of big long chirps and even a big block of pink noise at the end. Don’t ask me. I think you can see someone turning the knob on a filter sweep as well.

The phases basically just looks like noise. By the way, it’s all grey, even though there is no grey in the color map, because I’m telling matplotlib to do the downsampling in color space. Averaging phases does the wrong thing. Phases are periodic and I’m too lazy to figure out a better way to downsample this data for display. A color off the color map is way less deceiving than a color on the color map, I think.

1

If you don’t window at all (actually a rectangular window) a bunch of structure appears in the phases. But we introduced a discontinuity at the boundaries so there’s loads of bogus high frequencies. Presumably some of this new phase structure is also bogus and just whatever it has to be to place a jump discontinuity between the first and last sample.

Here’s the big brain response to this, which I got from a Miller Puckette paper. This is for Hann (raised cosine) windows. You can observe, empirically, two things. First is that when you take the DFT of a pure tone then the phase of the whole thing jumps by $\pi$ at the bin the tone is in. Second is that when you window this tone the phase now jumps twice, back and forth. So you say okay, if a bin is actually responding to a tone, then probably its adjacent bins have totally opposite phase. So, as complex numbers, they’re pointing in the opposite direction, and you can get a better estimate of the phase by flipping them and averaging. This is just

X = np.fft.rfft(x * window)
X[1:-1] -= X[:-2] + X[2:]

because we don’t care about their magnitudes. In the context of phase vocoders this really works which is kind of fucked up, because if we actually now look at the phases it looks like nothing has changed.

But if we diff these two images we get this. So there’s definitely some legible structure in there.

Another thing you can do is just taper the edges of the rectangular window, here I’m just doing 10 cosine samples at either end of a 8192 window. This kills off the high frequencies, which you can see by just eyeballing the spectrogram, but presumably still has all the problems rectangular windows have in terms of sidelobe fall off and whatever else.

It’s removed all those big streaks, so probably they really were bogus. Apparently this is called a Tukey window.

2

Phases are relative. The way to understand this is that if you slide a signal around in time all the phases update (at different rates, even), but audio sounds the same no matter when you hit play.

I’m going to suggest defining phases relative to the phase in same bin in the last window, as phase deltas. I think you should be a little suspicious about this because we do actually care about the vertical phase structure and this will give every bin its own zero phase reference angle, making phases within a single window independent of each other. I don’t think this matters too much as its easy to recover a shared-reference phase in context, and in cases where the phase is not just noise I think the vertical structure is probably pretty stable from step to step. So let’s just do it. I’m using the Tukey window here.

That’s a lot of structure. To get this I unwrapped the phases and computed deltas, both along the time dimension:

# Xs : shape (window, bin)
phases = np.angle(Xs)
phase_deltas = np.diff(np.unwrap(phases, axis=0), prepend=0, axis=0)

The horizontal banding is pretty weird. This happens because higher bins' phases advance faster than lower bins, so their deltas are just bigger. We can get a normalized representation by dividing out the rate at which the sine under that bin advances:

n = len(window)
dt = n // 4 # spectrogram step size
normalized_frequencies = np.linspace(0, 1/2, n//2 + 1)
phase_normalizer = np.angle(np.exp(2j * np.pi * normalized_frequencies * dt))
# ...
normalized_phase_deltas = phase_deltas / phase_normalizer

And that looks like this. Please ignore the incorrect units.

You can see chirps in the phase plot now. This is pretty closely related to instantaneous frequency but I don’t think I’ve ever seen it normalized this way. Someone must have.

End

I’m pretty dubious about this representation being of any use for machine learning. Maybe it would work if you never atan2 out the phase and kept it as a complex number (getting the phase delta is just a complex division). You definitely don’t want to be taking the derivate through anything that wraps the way an Euler angle does.

I had a quick look at recent papers to see what ML people were doing here before yoloing this post out there. It doesn’t seem like anybody is using STFT phases. EnCodec, at least, uses the real and imaginary parts of STFTs at multiple window sizes. That makes sense.

Twelve tones are inescapable

Mon, 31 Mar 2025 12:00:00 +0000

Speculations. Nothing new just held up at a different angle than what you’ve seen before.

I want to write about something interesting you can do with a single pentatonic scale. I’m just going to begin with what I noticed that got me thinking about this.

Take $\text C$ major pentatonic,

$\text{C D E G A}$ And transpose it onto $\text{G}$ . $\text{G A } \underline{\text B} \text{ D E}$ A new note popped out. The structure of the scale didn’t change but we needed a new note to represent it. If we didn’t know about $\text B$ , now we do.

Now watch this. I’ve added the number of semitones between each note and color coded notes by the transposition they are introduced in. Start with $\text C$ major pentatonic: $\color{#0073e6}\text{C D E G A} \\ \color{#000}\small\text{2 2 3 2 3}$ Transpose onto $\text E$ : $\color{#0073e6}\text{E }\color{#89ce00}\text{F♯ }\color{#89ce00}\text{G♯ }\color{#0073e6}\text{B }\color{#89ce00}\text{C♯} \\ \color{#000}\small\text{2 2 3 2 3}$

Rotate to $\text G♯$ : $\color{#89ce00}\text{G♯ } \color{#0073e6}\text{B } \color{#89ce00}\text{C♯ } \color{#0073e6}\text{E } \color{#89ce00}\text{F♯ }\\ \color{#000}\small\text{3 2 3 2 2}$ Transpose back to $\text C$ : $\color{#0073e6}\text{C } \color{#e6308a}\text{D♯ } \color{#e6308a}\text{F } \color{#89ce00}\text{G♯ } \color{#e6308a}\text{A♯} \\ \color{#000}\small\text{ 3 2 3 2 2}$

And we’ve got all of the notes in 2 transpositions: $\color{#0073e6}\text{A } \color{#e6308a}\text{A♯ } \color{#89ce00}\text{B } \color{#0073e6}\text{C } \color{#89ce00}\text{C♯ } \color{#0073e6}\text{D } \color{#e6308a}\text{D♯ } \color{#0073e6}\text{E } \color{#e6308a}\text{F } \color{#89ce00}\text{F♯ } \color{#0073e6}\text{G } \color{#89ce00}\text{G♯ } \\ \color{#000}\small\text{1 1 1 1 1 1 1 1 1 1 1 1}$

So that’s cool. Now what I would like to do is get an understanding of what is going on here that doesn’t rely on existing twelve-tone theory. Being able to lump pitch-classes into one of twelve ordered symbols that wrap around is doing a lot of work. So the rest of this post is about how much structure you need before pentatonic scales start implying the chromatic scale. Then we get to speculatin'. If this doesn’t sound interesting to you you can stop reading now.

String theory

First I need to explain why the $\text G♯$ onto $\text C$ transposition is a totally normal thing to do. By rotating we get a mode, but why care about modes?

So suppose you have a pair of strings. If you have a string instrument handy, you can try playing around with this. For pitches I will use the first two scales above, so one tuned to $\text C$ and one tuned to $\text E$ . We get the $\text C$ and $\text E$ scales by placing the 2-2-3-2 pattern onto them.

F♯

G♯

C♯

Now we want to play one of the scales one the opposite string. Looking for the $\text E$ scale on the $\text C$ string we find the same notes here (up to octave equivalence, I went back and forth on this but I don’t think it matters):

C♯

F♯

G♯

Now we move the pattern we just found directly over to the $\text E$ string, and there we go, all the notes, no weird tricks, as promised.

G♯

A♯

D♯

This is pretty much immediate, right. You want to play the same scale on strings tuned to different pitches, and out it drops.

Now, that got rid of symbolic manipulation and modes but I’m still relying on equal temperament. So I say it’s immediate but justifying 12-TET takes a lot of theory. There’s a gap here between experimenting with pentatonics on strings and reaching 12 tones that theory has already filled in.

That’s easy enough to fix. We’ve got to redo the above with non 12-TET patterns. Pattern because even scales are too much theory baggage for this part. To get away from any lingering chromaticism I’m going to dump down the fifth at $1/3$ –I’ll get to why even people who don’t care about $3:2$ ratios or, hell, harmony, will find it notable later–and then fill in the empty space by placing nodes equally spaced between the gaps. This turns out to give a seven-limit tuning (exercise for the reader):

1/9

2/9

1/3

5/12

1/9

2/9

1/3

5/12

These two patterns are like our $\text C$ and $\text E$ before, except now the green string is tuned to the $2/9$ node of the blue. Find those green tones on the blue string, by ear, so I’m not going to pretend we have integer ratios for these:

0.09

0.22

0.31

0.40

0.48

0.09

0.22

0.31

0.40

0.48

Placing the pattern on the green string without considering how it sounds made the red $\text G♯$ analogue. Now find it on the blue string, again by ear, so these produce the same pitches as on the green string,

0.06

0.19

0.29

0.40

0.46

Looking at the full gamut of notes we’ve generated gives…

0.09

0.22

0.31

0.40

0.48

0.06

0.19

0.29

0.40

0.46

1/9

2/9

1/3

5/12

Yeah. Compare to 12-TET:

0.06

0.21

0.29

0.37

0.47

0.00

0.16

0.25

0.37

0.44

0.00

0.11

0.21

0.33

0.41

One thing to watch out for is we didn’t start with the major pentatonic scale so we shouldn’t expect the nodes to correspond to nodes we found using the 12-TET scale. That is, they are not in the wrong position, or shifted over from where they ought to be. They just are where they are. But notice that even though we’ve lost the clear 12 tone structure, we’re still landing near 12-TET nodes. It’s the fifth that’s causing that.

It’s not hard to see why this clustering happens in principle. Once you have a fifth then you also have a fourth by the interval between it and the octave. A whole tone is then the interval between them. Stacking whole tones on top of the fifth will eventually produce a semitone relative to the fundamental.

C

F

G

A

B

C♯

Once you’ve identified this thing as a semitone and try to unify it with other near-semitones–by ear or mathematically–you’re entering chromatic territory. So all the pieces are there once you have the fifth and the octave.

Mathematically, this is equivalent to Pythagorean tuning. I’ve just motivated it differently. For whatever reason nowadays the way it’s presented is by stacking fifths and pointing out the comma you get after a comical number of fifths–try to do it on a real string and you will see what I mean–but slightly older textbooks like John Backus' The Acoustical Foundations of Music (1969, pretty interesting) have the actual way it was done which was to alternate fifths upwards and fourths downwards. A fourth down followed by a fifth up directly gives you the whole tone.

To finish this bit off I want to look at applying this procedure to $\text{C D E G A}$ again but in Pythagorean tuning and with just intonation. Here’s 12-TET all on one string first, to compare with.

0.06

0.21

0.29

0.37

0.47

0.00

0.16

0.25

0.37

0.44

0.00

0.11

0.21

0.33

0.41

Pythagorean:

0.06

0.21

0.30

0.38

0.47

0.01

0.17

0.26

0.38

0.45

1/9

0.21

1/3

11/27

Just intonation:

0.04

1/5

0.29

0.36

0.47

0.15

0.23

0.36

0.43

1/9

1/5

1/3

2/5

0.49

Pythagorean tuning is doing a very good job at preserving structure we happen to already know is there. But if you didn’t know, then what it is giving you is a way of moving patterns from one string to another that mostly just works. Without it, scale correspondences between strings jump all over the place, without clear structure and with annoying discrepancies everywhere, which you can see in just intonation and my arbitrary pentatonic scale.

To harp on just intonation a bit, I will quote acoustician John Backus,

Because of these difficulties, the just scale has never been of any practical use. Its theoretical attraction to individuals with numerological inclinations is extremely strong, however, so much so that it has even been called the “natural” scale, as though it had some fundamental basis in nature not possessed by other scales. It appears in practically every book dealing with the acoustics of music, where it has been given an emphasis it does not deserve.

John Backus, The Acoustical Foundations of Music, 1969

It turns out people tried really hard to make just intonation work during the Renaissance. Back then, Vincenzo Galilei in Dialogue on Ancient And Modern Music¹ did a similar thing to what I did above and systematically (and much more thoroughly than me) looked at how the intervals combine into near-misses. To what extent he is responsible for them abandoning just intonation, I don’t know, but he wanted everyone to move on to meantone and equal temperaments (sometimes both at once), and they did. Another thing: it was a matter of debate among themselves whether they sang in just intonation or not, and they didn’t. This is pretty amusing to me as people still say the same thing. Galilei might have been the first person to actually check. If we’ve had 500 years of this, how many other things that I take for granted are just wrong? Do barbershop quartets really sing in just intonation? Wikipedia…

However, “In practice, it seems that most leads rely on an approximation of an equal-tempered scale for the melody, to which the other voices adjust vertically in just intonation."[5]

Oh my god dude it keeps happening

This post part two: what’s the deal with fifths anyway?

For a long time now I’ve been pretty sure the usual story you hear that justifies the chromatic scale by going just intonation consonances $\rightarrow$ Pythagorean approximation $\rightarrow$ meantone etc. $\rightarrow$ equal temperament is bogus. Not just historically unsupported but, as a story we tell ourselves, it seems off, to me. On the other hand, throwing up your hands and declaring it all arbitrary and culturally contingent is denying a lot of the underlying structure that pops up everywhere.

One reason I think pentatonic scales are important in motivating the 12 tone system is purely anthropological: they come along for the ride once you have anything resembling a scale and so are everywhere. But it’s not the case that what you end up with are pentatonic scales with a fifth. Gamelan music is probably the best example where they dodged this and developed an entirely different approach to harmonic structure that uses a lot of inharmonic timbres. So transposing pentatonic scales alone isn’t enough to motivate twelve tones, even if you find my trick above convincing.

Along similar lines, there’s an interesting 2024 preprint by McBride et al., Melody predominates over harmony in the evolution of musical scales across 96 countries, that points out folk melodies survive by ease of use: the whole tone to third interval range is in a goldilocks zone for singing. They focus on song but the same argument works for instruments too. It’s just hard to be mechanically precise about pitch until you invent a monochord or something like it. Their point on top of that is that, once you gather them all up and analyse them, the scales you get out of this in practice are not particularly harmonic (consonant) in general.

So what’s not bogus. Well, if you’re sufficiently serious about music for a long enough time you will eventually learn something about tone production.

Salviati: A string which has been struck begins to vibrate and continues the motion as long as one hears the sound; these vibrations cause the immediately surrounding air to vibrate and quiver; then these ripples in the air expand far into space and strike not only all the strings of the same instrument but even those of neighboring instruments. Since that string which is tuned to unison with the one plucked is capable of vibrating with the same frequency, it acquires, at the first impulse, a slight oscillation; after receiving two, three, twenty, or more impulses, delivered at proper intervals, it finally accumulates a vibratory motion equal to that of the plucked string, as is clearly shown by equality of amplitude in their vibrations. This undulation expands through the air and sets into vibration not only strings, but also any other body which happens to have the same period as that of the plucked string. Accordingly if we attach to the side of an instrument small pieces of bristle or other flexible bodies, we shall observe that, when a spinet is sounded, only those pieces respond that have the same period as the string which has been struck; the remaining pieces do not vibrate in response to this string, nor do the former pieces respond to any other tone.

Galileo Galilei, Two New Sciences, 1638, trans. Henry Crew and Alfonso De Salvio

Yeah I’ve been rooting around old texts from before 12-TET got really locked in to see what they find salient about music. The oldest I’ve got is the Record of Music which is dated to Warring States period China, so 200-400 BCE or so. They had proper Pythagorean twelve-tone tuning back then, too. I’m just going to quote from it, in full, starting from the very beginning, until we hit something relevant.

The Roots of Music

In all cases, the arising of music (yin) is born in the hearts of men. The movement of men’s hearts is made so by [external] things. They are touched off by things and move, thus they take shape in [human] sound (sheng). Sounds respond to each other, and thus give birth to change.

Scott Cook, Yue ji - Record of Music. Introduction, translation, notes, and commentary. Asian Music, vol. 26 no. 2, 1995

Yup. And like all sufficiently old texts it comes with commentary from ~100 CE that shows they understood this to be talking about sympathetic resonance, like Galileo up there. (By the way a gong here is pretty much the root note of a pentatonic scale.)

With musical instruments, when one plays a gong, then many gong’s respond. But this is not sufficient to be found enjoyable as music. For this reason, they are given to change and caused to be mixed together. The Yi Jing says: ‘Alike sounds respond to each other, alike qi seek each other.’

Zheng Xuan

Now, I need to point out that what follows in the translation I’ve got is 2 pages of modern-day translator’s commentary on just these tiny fragments alone. And it did not occur to them, at all, that this was about resonance. They focus entirely on the metaphysics, movement of men’s hearts or whatever. So my interpretation here is not necessarily self-evident, but, come on. The commentary is saying it directly: with musical instruments, when you play a $\text C$ , $\text C$ s respond!

This is what is so special about fifths: the $\text F$ below and the $\text G$ above also respond. For higher harmonics the effect of this falls off very rapidly. And, it does sound good, but it doesn’t resonate at its own pitch. You hear a harmonic, $\text C$ an octave up. Timbre-wise it’s more like a reverb. So, if you have a string instrument, whenever your open strings form unisons, octaves, and fifths against whatever note you’re playing, you get a richer sound that you can’t get without them. But as Zheng Xuan says, you need more than unisons to make music. Musicians really want this.

At this point, if all you have are 3 strings tuned a fifth apart, say $\text{C}_3$ , $\text{ G}_3$ and $\text{ D}_4$ , then finding $\text G_3$ and $\text D_3$ on the $\text C_3$ string is exactly the first two steps of a Pythagorean tuning. Sympathetic resonance leads you right to it, even if your musical sense is focused exclusively on melodic structure and chords don’t interest you.

Finally, while I do think this is extremely relevant to the historical development of instruments, you also shouldn’t ignore that Chinese text I quoted, which says, paraphrasing: when you hum the right way at things that resonate, they hum back. The significance of this to them will have been slightly different than its significance to us, right.

By the way, the music stuff out of ancient China is incredible. We know they used the chromatic scale because we keep digging up bronze bells that come in sets tuned to it. And you can hit them different places to produce one of two tones, a third apart, and we have no idea how they found the shape that does that, and cannot yet reproduce it ourselves without copying. At this point, the oldest bells are a thousand years older than any written record of them using Pythagorean tuning to tune the things in the first place. This has led one professor of Chinese art ‘n’ archaeology to say

But the instruments that mattered in ancient China were bells, not strings, and the theorists of a bell culture do not think in such terms. I am proposing instead that, in a culture focused on bells, and at a time when scales were a strictly aural phenomenon, the chromatic scale was discovered through transposition of the pentatonic scale. If in China the chromatic scale was not a difficult idea, surely the reason is that the theorists who discovered it were not doing string-length calculations that made transposition awkward. The chromatic scale as they conceived it would have been an equal-semitone scale because there was no reason for it to be anything else.

Robert Bagley, The Prehistory of Chinese Music Theory, Proceedings of the British Academy, Volume 131, 2004

But as I say above, I think you can get away from arithmetic on strings, too. (Also can you even build a bell forge without knowing a lot about ratios of lengths? Serious question. Surely there will be a lot of hoisting involved. It sounds positively Archimedean to me.)

But in any case, the Pythagorean numerology only comes later, when you and the priests are trying to justify what you are doing. Which is totally fair, because if you can’t show that your music unites the harmony of the stars with the harmony of the state, what are we even doing here,

End

This post ended up with its own folder in zotero. Man.

But, okay. Here’s what I think. Forget all that stuff about the harmonic series and its consonances leading to the chromatic scale. Forget about the diatonic scales too, even. Say that stuff comes later, not before.

Say instead sympathetic resonance between strings leads you to the procedure of Pythagorean tuning, and it turns out you want to use it because it gives you nice way to organise all the pentatonic scales you play alongside the singers and guys with flutes. As a bonus, that procedural knowledge is something you can teach to just about anybody. You will still adjust your string tuning and intonation while playing a little bit to fit with what the other musicians are doing so you are not yet playing the chromatic scale. But over time this provides a framework that eventually gets set in stone, due to the development of instruments that can only produce fixed pitches.

As a nice just-so story I think this one is a lot more plausible. And I think it gets closer to what is actually important in music. But that’s just me

Vincenzo Galilei. Dialogue on Ancient and Modern Music. Translated by Claude V. Palisca. Music Theory Translation Series. New Haven: Yale University Press, 2003.

The introduction to this book by the translator is worth reading if you’re interested in what is the deal, historically, with all these scales. After showing mathematically the mess commas get you into, Galilei then does scale construction by ear, by the way. ↩︎

Generating Unique Random Numbers

Fri, 27 Sep 2024 20:08:28 +0000

This post has two things:

Two somewhat obscure non-linear invertible operations that work on only part of a register, and

A fast stateless random permutation generator that uses them, and passes some statistical tests for RNGs you may have seen before

The ideas are pretty simple so the hope is you can also get an intuition for how to think about the problem, which is the best thing you can get out of this post.

I got my intuition from these:

Bob Jenkins' talk on writing hash functions,

Marc B Reynolds' post on integer bijections (everything in this post is an integer bijection),

Brent Burley’s Practical Hash-based Owen Scrambling, highly recommend this one if you like path tracing,

Andrew Helmer’s blog post about Andrew Kensler’s permute in this correlated multi-jitter paper. Helmer’s post is where I first saw this idea, I think.

Random numbers that don’t repeat. Let me explain the basic idea first before I get into the weeds. We decide to generate unique random numbers in $[0, N)$ , here 8, by shuffling this list of numbers: $(0,1,2,3,4,5,6,7)$ Adding the same number mod $N$ to every element gives us back a new list with the same elements in a different order, $(0,1,2,3,4,5,6,7) + 3 \mod 8 = (3,4,5,6,7,0,1,2)$ And so does multiplying by an odd number, mod $N$ , $(3,4,5,6,7,0,1,2) * 5 \mod 8 = (7,4,1,6,3,0,5,2)$ And this now looks pretty shuffled. So, take in the original numbers one at a time, apply these and other bijections the same way on each number, and out the other end we get a pseudorandom permutation without using any memory. We never need to construct the initial list.

The operations we can use are any that can be undone (any shuffle can be re-sorted), so you hear people call them bijections and invertible functions interchangeably.

I got interested in this after looking at the low discrepancy sequence stuff, but I brushed off the connection in my post about it. With the tools there, its easy to generate numbers that don’t repeat but hard to make them random. That is, you could feed truly random bits in and you’d randomly select from a small set of random-looking sequences, which is not what we want. At the time, I didn’t know much about this problem, just that what I was doing didn’t work. This EA Seed post of Alan Wolfe and William Donnelly that uses Feistel networks got me interested again. The thing that really got me is that quality/speed trade-off they mention: what does it take to max out the quality? How can you tell you’ve reached it? Rather than answer these questions I made my own that always maxes the quality out.

To do that you need some organising principle. Mine is this: use all your bits.

What do I mean by this?

In the above example we have 8 numbers we can do an add with, from which we get 3 bits worth of permutations. The multiply is 2 bits, we lose a bit due to the odd constraint. We won’t miss it for larger $N$ . To choose these randomly we need 5 random bits, for 32 possible multiply-adds. That’s 32 different permutations we can get after one multiply-add, and, yes, any sequence of multiply-adds can be replaced by just one multiply-add, so out of these two operations we can’t get more than 32 permutations. No matter how many random bits we have with which to choose different madds, we’ll always end up with a sequence that can be reduced to a choice made with just 5 bits. Any more than that bits than that are wasted. We haven’t used them.

There are $8! = 42320$ permutations of length 8.

Feistel networks, entropy, and so on

What Feistel networks do is give you non-linear bijections, and you can turn the crank on them to get as many permutations as you want. How many turns of the crank you need can be found in this 2022 Mitchell et al. paper that adapts philox to generate permutations. Philox is (mostly) a Feistel network. To get good statistics, turns out it’s quite a lot: they want 24. This is the kind of round count you see in cryptography. What gives?

In a Feistel network, you split each number into two halves, one half that keys a round function and another half that gets permuted with a xor. This means to know how one number is scrambled by the round function, you only need to know half the bits in each step, and this means you get effectively half the entropy out of each round you could be getting (see this footnote if you are suspicious¹). But this structure also makes it super easy to invert, which is very useful for encryption, right. For permutations of length 5, the function in the Mitchell paper will use 72 bits of entropy (and asks for some 700 bits that it just won’t use). You need 7 bits to index all 120 permutations!!

This half-the-bits thing is important and is a good example of the kind of thinking you want to being doing when making these things, so, here. This is to illustrate something about bijections and the Feistel part isn’t super important so I’m not going to go in detail on that. Look:

This is a single round with the 8-bit input using a xor by some arbitrary function f as the round function. One Feistel round permutes only the blue box to produce the purple box; the orange box remains unchanged in the output and the low bits reappear, unchanged, as the new high bits. That’s what I’m talking about in the previous paragraph. But if you stare at this a while you may realise there are only 16 ways that purple box can end up, and the only thing f and seed can do is make the choice more or less predictable. This is the important part: a better choice of f can’t give you more permutations than that. If you only do this one thing, then each round you get at most 16x new permutations, and they’re the same new permutations regardless of f. A bad choice of f might fall short of all 16x, but ruling those out, the thing a better choice of f is giving you is a more unpredictable order with respect to seed.

And that might sound good but for most $N$ there are so many permutations we don’t care about the order. If the order you see permutations in matters, that’s bias. There are just too many permutations for it to be otherwise. I’ll get back to this point later.

I say a Feistel network will require too many bits, but it turns out to be really hard to use the minimal number of bits. Since I’ve brought this up, check this out. How many bits will the Fisher-Yates shuffle use? It’s the standard shuffle, and there used to be a little ecosystem of blog posts on how careless implementations could give bad statistics. My goal with this post is to match Fisher-Yates statistics-wise. (Code from Wikipedia, why not):
-- To shuffle an array a of n elements (indices 0..n-1):
for i from 0 to n−2 do
j ← random integer such that i ≤ j < n
exchange a[i] and a[j]
On the first iteration, we generate a number between $0$ and $N$ , so $\log_2 N$ bits are needed to describe the number of possible outcomes of the first iteration. The second is between $1$ and $N,$ that’s $\log_2(N - 1)$ bits, and it’s smaller because we have one less possible values to choose from, right. Continue like this, then by the end we’ve used

$\begin{aligned} \sum_{i=0}^{N - 2} \log_2(N - i) &= \sum_{i=2}^{N} \log_2 i \\ &= \log_2\prod_{i=2}^N i \\ &= \log_2 N! \end{aligned}$

bits of entropy. Exactly enough bits for the $N!$ possible permutations: if you interpret the bits it draws as a number, then that number directly indexes into the big Lehmer-list in the sky of permutations of size $N$ . So, assuming we could draw a fractional number of bits from an RNG, this shuffle is optimal in the number of bits it will use. Something that took me a while to realise is that the exact same reasoning applies to figuring out the minimum number of bits it would take to encode the next value in a permutation, given those you’ve already seen. :o

We won’t get anywhere close to this lower bound, in both senses: for small $N$ I’ll use too many bits, and for large $N$ , well, nobody ever will get close to using enough.

If all this talk of bits and entropy is lost on you: read David MacKay’s Inference book.

Anyway, all that’s why I went looking for other ways to map bits to bijections. My thinking is this: using cheaper non-linear bijections that operate on all the bits at once can produce more random lookin' permutations per unit time. The space of permutations that can be sampled at all can expand up to twice as quickly by having no half-width bottlenecks.

Two invertible operations

1

Everyone into this junk knows odd multiplies are invertible and even multiplies aren’t. That’s wrong: even multiplies are perfectly invertible. It’s only after truncating the bits that don’t fit in the register that invertibility is lost. They don’t have invertibility $\text{mod } 2^b$ , but here, $b$ is probably small, and there is still plenty of register left over. So, don’t truncate: shift.

a : value to permute
k : random bits, or constant, whatever
m = (1 << bitwidth(a)) - 1 (mask for getting a back into range)
k_low_bits = (k & -k) - 1 (turn trailing zeros into a mask)
k += ((k & m) == 0) (fix up if k is 0)
ak = a * k (multiply)
a = ak + ((ak >> bits) & k_low_bits) (rotate)

Spelled out: even multiplies can be split into two operations, an odd multiply and a left shift; the shift amount is the number of trailing zeros. To restore invertibility we could shift back to undo that shift, but this lands it back at an odd multiply so we’re still down a bit. Instead, do a rotate, by shifting the bits otherwise masked out into the low bits that were just zeroed out by the multiply.

This is fine as is for small bit-widths, but if bitwidth(a) is greater than the half the register width then k can have so many trailing zeros that the multiply runs out of register. For bitwidth(a) up at like 30 then it can only handle at most 2 trailing zeros. We don’t care about the missing bit that much in that case, so just detect when there are too many trailing zeros and turn it into a multiply by one:

km = m & (~0u >> bits) (note, km == m when bits < regsiter width / 2)
k += ((k & km) == 0) (new fix up)

This is getting a bit complicated. The up shot is we get an almost full-bit multiply and a non-linearity out of it. ‘Almost’ because we still can’t multiply by zero. To underscore the point a bit, the shift amount is constant with respect to $N$ but the effective rotation amount depends on k.

If you choose k randomly then the random rotates we get follow a geometric distribution: $1/2$ chance the first bit is zero, $1/4$ the first and second bit are both zero, … So it’s usually 0, 1 or 2. That works well here because it means it just works for all $N$ –for small $N$ , if we only have, say, 3 bits then we can only do rotates of 1 or 2 bits, so if we have a lot of larger rotates we aren’t getting the non-linearities we want. Likewise for large $N$ , where we need to avoid shifting away bits at the opposite end, we don’t have to worry about that happening often either. But for other $N$ we’d really like to get bits moving in different ways. So there’s room for improvement, maybe.

2

The second invertible operation I’d like to point out is actually a repeat from last time, where it appeared in the Laine-Karras scramble (and I got it from the Brent Burley paper originally). But this time I got to it by thinking: the above is using the low zero bits of k as the shift amount. Can we swap our operands around and use the low zero bits of a as the shift amount? Then we’d get a varying rotate per a. But I couldn’t figure out a way to use that idea that was actually invertible. However in a moment of zen I realized the following is invertible:

a ^= (a * k) << 1

Which is exactly equivalent to the multiplication by even constants in the nested uniform scramble.

This is pretty interesting in that, if you look at it like a Feistel function, then rather than splitting up the permutation into equal sized classes indexed by half the bits, it splits it into classes indexed by the number of trailing zeros in a. Not only are the classes shuffled independently, so values in one class stay in that class, each class is a different size. Within each, the permute is driven by the multiply. (Sorry if all this is a bit abstract all of a sudden).

But in light of what I said before about Feistel networks I will use it like this, because the bits in k are valuable:

a ^= ((a * k) << 1) ^ k

So that’s two bijections. One is a little complicated code-wise but is really simple conceptually; an odd multiply followed by a rotate. It’s easy to see why it’s invertible. One is simple code-wise and is complicated conceptually; I don’t know what to call it. It’s a bit more subtle I think.

permute

To use these, I need to define the problem. We want to randomly sample permutations, uniformly and independently. For small $N$ , this means sometimes sampling 0, 1, 2, 3, 4… as a random permutation. So, that’s a stateless function like this, where seed enumerates permutations in a pseudorandom order.

uint32_t permute(uint32_t i, uint32_t N, uint32_t seed);
// Usage:
for (uint32_t i = 0; i < N; i++) {
uint32_t index = permute(i, N, seed);
// ... use index
}

Secondary goal is that it should just work. No need to seed it well. Cycle walking for non powers of two².

The idea is that generating pseudorandom bits is expensive, so do it once at the start, and shift out the generated bits from a register as a source of entropy. Then use each bit of entropy at least once. I found the two bijections above weren’t enough on their own for good avalanche so I gray code between each use of entropy bits:

static uint32_t hash32(uint32_t x) {
x ^= x >> 16;
x *= 0x21f0aaad;
x ^= x >> 15;
x *= 0xd35a2d97;
x ^= x >> 15;
return x;
}
static uint32_t log2_floor(uint32_t a) {
return 31u - _lzcnt_u32(a);
}
static uint32_t log2_ceil(uint32_t a) {
return log2_floor(a - 1u) + 1u;
}
uint32_t permute(uint32_t i, uint32_t n, uint32_t seed) {
uint32_t bits = log2_ceil(n);
uint32_t mask = (1u << bits) - 1u;
uint32_t multiplier_mask = mask & (~0u >> bits);
uint32_t index_seed = (i >> bits) ^ n;
uint32_t state0 = seed + index_seed;
uint32_t state1 = hash32(index_seed - seed);
do {
uint32_t state = state0;
uint32_t rounds = 2;
while (rounds--) {
uint32_t p = state;
do {
uint32_t q = p;
p >>= bits;
uint32_t r = p ^ state;
p >>= bits;
uint32_t s = p ^ state;
p >>= bits;
q &= ~1u;
q += ((q & multiplier_mask) == 0u) << 1;
uint32_t q_lb = (q & (0u - q)) - 1u;
i ^= ((i * p) << 1) ^ p;
i ^= (i & mask) >> 1;
uint32_t iqr = (i * q) + r;
i = iqr + ((i ^ (iqr >> bits)) & q_lb);
i ^= (i & mask) >> 3;
i ^= ((i * s) << 1) ^ s;
i ^= (i & mask) >> 7;
} while (p);
state = state1;
}
i &= mask;
} while (i >= n);
return i;
}

I kept it all within 32-bits to make it easier to analyze. It doesn’t work for $N > 2^{31}$ . If you want that, go to 64-bits. Replace the hash and add

i ^= (i & mask) >> 15;

somewhere. One benefit of having no constants is it’s clear how to extend this thing to higher bit counts.

Fiddly detail

The high bits in the index would otherwise go to waste so I use them as more seed bits, so each seed really defines a sequence of permutations per $N$ , rather than just one permutation per $N$ .

The log2_ceil is a little exotic in a non-systems language context but it is pretty much free here with lzcnt, you probably want to pass it in precomputed if it or an equivalent instruction is not available, the alternatives really dent the speed of this thing. It’s from Travis Downs.

The hash function is one of those prospecting for hash functions hashes. The way I use it, you can see you could also pull it out precomputed once for the whole permutation, if you really wanted to.

As well as the add, I sneak in a xor of the original bits in middle of the multiply-rotate. Philox-y. And there’s always at least 1 bit of rotation.

I don’t like the while (p), I’d rather use a fixed number of iterations derived from n. As it is, larger n shift bits out faster and stop earlier. But it will also stop early if the state becomes 0. It works better statistics-for-time this way, which I don’t really understand.

Justifications

For $N$ larger than 12, 32 bits is not enough seeds to index every permutation, so it’s fine to fall short of hitting every permutation, so long as the sampling is³ unbiased.

To be unbiased we need this: it doesn’t matter how you seed this thing. Consecutive seeds must produce permutations that are uncorrelated. It doesn’t take long before the space of $N!$ permutations it’s supposed to be sampling from is so vast that all $2^{32}$ permutations are a vanishingly small subset. So, if nearby seeds are correlated, we already know we are sampling from a biased sample of all permutations. That’s why I use the seed directly in the first round: we could hash it, and this will probably give us better (state0, state1) pairs to use (there are $2^{64}$ such pairs and we’ll use $2^{32}$ of them), but there is no simple way to avoid having many mostly-zero states in one side of the pair; all we can control is what order we get them in.

For that reason, I don’t think we can get away with using only 32 bits of state, even though we’ll only ever map to 32 bits worth of permutations. If we hash, we can use an invertible hash to guarantee each seed maps to unique hash bits. But this means we just shuffled the low-entropy seeds around. Or we use a non-invertible hash, and we know for sure some seeds map to the same permutation–can’t be having it. Even if we pack this thing to the gills with intimidating constants, a low entropy seed will mean we land on a permutation near (in some sense) to the default permutation selected by the constants.

Expanding the seed an extra 32 bits solves all this.

To get an idea for why we really cannot have any correlations period, think about lexicographically sorting all the permutations and looking for correlations between nearby seeds. If $N = 500$ then for the first 10 values we have ${500 \choose 10} > 2^{67}$ possible permutation-heads, so by the 11th value we expect the sorted permutations to be completely uncorrelated again.

The restriction to 32-bit integers is somewhat artificial but it also enabled reasoning like the above that I’m not sure I’d have got to without it.

Testing these things

A quick test that will fail most ideas you come up with is to generate permutations of size 8, each value is 3 bits so you only need 24 bits to represent an entire permutation. Generate a bunch, pack each into an int and compare the number of unique ints you get to unique numbers from a PRNG in $[0, 8!)$ . Or math it out, but we’re all friends here. This fails permute when using a single round.

To get an idea for the behaviour with larger $N$ you can make a matrix diagram that visualises how often a each input index gets mapped to each output index. This works but I found you could look good here while still being biased (this is true of avalanche as well). Instead, try this: look for correlations between adjacent indices. So, instead of counting (i, permute(i)) pairs, count (permute(i), permute(i + 1)) pairs. Somewhat interestingly, since these are always different, the pairs this gives you form a particular kind of permutation called a derangement. With that said, here’s permute with only 1 round, $N=1024$ and $16N$ consecutive seeds near 0:

Scaled down by $2\cdot16N\cdot\frac{1}{(N-1)}$ which puts the expected average at $0.5$ . And fed through the BrGB colour map from matplotlib which is white in the middle. It’s much more clear what’s going on compared to grayscale.

The zeros along the diagonal are expected, as it’s a derangement. But it’s not quite there yet, note that near the diagonal it is slightly darker and undersampled. Here’s two rounds:

👍. These are all the seeds around zero–all ones, all zeros, 31 ones, 31 zeros, that kind of junk.

The last test I like is for small $N$ , counting repeat permutations. Similar to the first test but more involved to actually implement. This is an interesting test because when we randomly sample permutations of length $13$ , there are more than $2^{32}$ permutations to sample but the number of samples before we expect to see a repeat is far less than $2^{32}$ . Knuth in TAOCP somewhere has an approximate formula for this $Q(N!) \sim 1 + \sqrt{\frac{\pi N!}{2}} +\space\dots$ which is also lurking on the Wikipedia article on the birthday problem. From this you get that only after $N > 21$ or $22$ or so is it reasonable to map each seed to a unique permutation, as $Q(21!) \approx 2^{33}$ but $Q(20!)\approx 2^{31}$

Between 13 and 20, in order to look statistically right, we need to be producing repeats–even though there are more permutations than seeds!!

This test is the hardest I’ve found to pass: an older function I had was a real slow thing with some Feistel rounds that looked otherwise good, and this test failed it at $N = 19$ . Unfailing it is how I got to the multiply-rotate. At the time I was looking at gaps, but M.E. O’Neill points out you can also count repeats and tells you how many samples to take and how to compute p-values so let’s just use that. I’m going to dump a table on you now. Each row looks at consecutive seeds starting at $0$ :

N	samples	dupes	expected	unique_dupes	p
3	16	10	10.32	4	0.54
4	31	14	13.42	8	0.63
5	70	19	16.80	16	0.75
6	170	21	18.49	18	0.76
7	449	18	19.38	18	0.44
8	1270	16	19.78	16	0.24
9	3810	13	19.93	13	0.07
10	12048	14	19.98	14	0.11
11	39959	19	19.99	19	0.47
12	138420	19	20.00	19	0.47
13	499080	20	20.00	20	0.56
14	1867387	16	20.00	16	0.22
15	7232357	19	20.00	19	0.47
16	28929425	19	20.00	19	0.47
17	119279073	12	20.00	12	0.04
18	506058246	20	20.00	20	0.56
19	2205856754	26	20.00	26	0.92
20	4294967295	5	3.79	5	0.82
21	4294967295	1	0.18	1	0.99
22	4294967295	0	0.01	0	0.99

Yeh: looks random. The p-value breaks down a bit towards the end there because we are taking so few (!!) samples, but whatever. The unique_dupes column is just a sanity check that we aren’t getting the same repeated permutation.

Counting repeats from all $2^{32}$ permutations turns out to be a bit non-trivial. Code is here if you want.

I’ve been less thorough with testing really large $N$ . I dunno. Avalanche isn’t really a great test here, necessary but not sufficient type beat.

End

I’ve been sitting on this draft unpublished for a while now. I picked up some entropy intuition sometime around writing my low discrepancy noise post, I was trying to figure out why the permutations you can get out of xorshift and LCGs are so bad.

It’s invertible, but I have no idea what the inverse function will look like.

I was aiming for the performance to be around the same as Andrew Kensler’s permute, the correlated multi-jitter one. By the way, if you understood what I was talking about with the Feistel bits thing you can just eyeball it to get an idea of the permutations you’ll get out of it. My permute is about 2x slower, or 1.4x if you let the optimizer precompute the seed hash for you. Given the statistical quality I’m after I think this is alright, I can just throw this thing at people and not worry about them seeding it wrong or whatever. I think you can do better speed-wise but I don’t know how much better.

And the small $N$ thing. Why the focus on that? Couldn’t you just shuffle an array every time for those? Like, sure, maybe, but really my issue is: if you can’t nail the small $N$ case where it’s easy to verify the statistics are good, why would you think large $N$ is any better? Okay that’s it. ThaNKS for reading

This sidesteps the fact that over the permutation as a whole you get the full bit-width worth of bits of entropy. It’s only the path through state space taken by neighbouring values that are bit-deficient. For 8 bits, you’d have a 4 bits worth of round functions that each map 4 bits to 4 bits. 16 round functions times 16 possible outputs each is 8 bits total to describe the full permutation. Each round function is totally independent of what the other round functions are doing, and it’s only the number of bits required to encode the path a given value takes that is half that required to encode the entire permutation. Really weird.

This alone isn’t enough to explain the high round counts you see in practice. Interestingly, if you ask instead about the average number of bits that will be flipped then the required round counts come out close to what you see empirically… but is that just a coincidence?? ↩︎
Rejection sampling is faster, and even fast on the GPU as that Michell paper points out. I think it’s better statistics-wise too; it just can’t be dropped in and used anywhere. ↩︎
indistinguishable from an unbiased sample until we exhaust the seed space. Do you care about this distinction? Did I need to add it? ↩︎

WTF Are Modular Forms

Mon, 25 Mar 2024 00:34:54 +0000

This post is part of the HMN Learning Jam. One weekend to fill your head with stuff and one weekend to dump it back out. I’ve spent a few more days here and there (that convinced me I could learn the rest of what I needed to know in a weekend) but otherwise that’s about how much time I’ve got in knowing about modular forms. This is a bit out of step from the rest of the submissions which are more programming focused but look: someone had to.

As always happens I probably learned the most while I was writing this post and not when I was reading stuff.

I’m gonna take a very non-standard approach that I’ve seen fragments of here and there in the Literature, so experts on this stuff know it all, but the path it takes is so irrelevant to what working with modular forms is actually like that a mathematician would never be so cruel as to present it this way.

However, I’m not a mathematician and am completely new to all this. From what I can tell, this path I’ll take here is closer to what was going on when modular forms first appeared in mathematics 200 years ago. It’s pretty far removed from modern day modular forms. But this means we get to see what modular forms are in a certain naive sense that hopefully doesn’t require a degree in mathematics to understand.

To get the most out of this, you want to know a a little linear algebra and be comfortable with complex numbers. You should also probably read the first half with a graphing calculator open. On the other hand, I’ve deliberately avoided some convenient mathematics notations/definitions that make maths inaccessible to many people, which makes life harder for me and probably people that know that stuff.

The two books I found most useful for this are:

Development of Mathematics in the 19th Century, Felix Klein,1928

Complex Analysis, Elias M. Stein and Rami Shakarchi, 2003

The first is by the Klein bottle guy. He was pretty much there when modular forms became modular forms and is where I got this general approach from. He’s more brisk about it. Cool book, could not have understood any of this without it.

Complex Analysis is a textbook for undergraduates. Goes slow in the right places. Really well written if you know how to read maths textbooks.

Another book I’d like to recommend that you’ll probably have trouble discovering on your own is

Elliptic Curves, Henry McKean & Victor Moll, 1999

This is in my preferred style for maths textbooks, which I have heard called “breezy” before. Some people figure out they’ve got a tiny audience and write conversationally directly to them, it’s great. No sweating on proofs, just big picture ideas from people fully in the post-rigorous phase of life, and they trust you have dryer references to fall back on if you need them. But it is squarely aimed at real deal mathematicians so beyond me for the most part.

A cool resource is Richard E Borcherds lecture series on youtube if you want to see what it’s really about to a mathematician. He’s got a Fields medal in this stuff.

Another cool video is this MathKiwi video which in some sense starts where I leave off and has many good visualisations.

To round this list out here’s a bunch of mathematicians saying what they find interesting about modular forms. Check out how many different angles on them there are.

I can’t motivate why you’d want to know about modular forms. If you want to know this stuff, you want to know, and if you don’t, you don’t. Instead, I want to start at a basic number theory question and make a bee line for modular forms. Modular forms come from a few related ideas hanging together, so we’ll have to gather up those ideas first.

Here’s our number theory question: given an integer $n$ , how many solutions does $n^2 = a^2 + b^2$ have? I start here because it makes the link to questions like these and Fermat’s Last Theorem clear, but I am going to say well clear of any weird factorisations and algebra that usually immediately follow this question, don’t worry.

What a lot of number theorists like to do when encountering something new is to compute large tables of numbers, and here since you never have to look at $a$ and $b$ above $n$ you don’t have that many $(a,b)$ pairs to check. Well, for a few $n$ , anyway. You can get pretty far along crunching numbers. Then you stare at the numbers and hope for insight. This works surprisingly well.

But instead the way to modular forms is to draw the sucker. This turns it into the following question: on how many places does the circle $n^2 = x^2 + y^2$ lie on integer coordinates? Now instead of crunching numbers, you just draw a circle on a grid. Solutions are where the circle lies on a point where the grid lines intersect. That set of points of intersection over the entire $xy$ plane is called a lattice.

The year is 1799 and we would like to solve this problem with a straightedge and compass. We dutifully draw the grid. We try it for one $n$ , fill in the row in our table, try a second $n$ , fill in the next row… Number theorists love their tables. Eventually, $n$ is too big for our compass. What do? We shrink the lattice. Instead of ever larger circles we ask about ever denser lattices. And instead of leaving this as just a convenience out in the world, we can make it real in the mathematics, and change the problem to looking for rational numbers that satisfy

$\frac{x^2}{n^2} + \frac{y^2}{n^2} = 1.$

Play with this a bit in a graphing calculator and you’ll realise that once you’ve found a solution you can extend a line through it from the origin, and lattice points on that line are solutions for some other $n$ . So something about the lattice is encoding information about the sums of two squares.

With lattice we encountered the first term that comes up all the time when people talk about modular forms, and maybe see how it’s relevant to number theory, so we are making progress.

So, that’s a circle. For this exposition I would like to encourage you to act like a mathematician and be willing to entertain idle questions. What about where an ellipse goes through a lattice point? We apply a similar trick as before, and get an axis-aligned ellipse by stretching $x$ and $y$ , with $n^2 = ax^2 + by^2.$ Instead of drawing the ellipse, we stretch the lattice in $x$ and $y$ , and we are back to a circle. So we think of an ellipse as a transformed circle, and apply the inverse transform to the lattice instead, and this let’s us answer more questions with our compass. Very good.

I quickly made these in tldraw cause you really gotta be thinking in pictures to follow this and it’s just good manners to show the pictures. However I haven’t actually checked these are valid transforms. Sometimes even in a graphing calculator you’ll see what looks like an intersection and plug in the numbers and find it’s a near miss. So. No rigour here. Also I think the axis $a$ and $b$ here should be sqrtd. Too late now.

What about an ellipse that isn’t aligned to the axis? Rotating the circle does nothing and rotating the lattice is the same as rotating the circle the other way. Instead, we have to shear the ellipse. What Gauss observed is that if we have $n^2 = ax^2 + bxy + cy^2$ then we can interpret this as tilting our lattice so it now tiles the plane with parallelograms, rather than squares. The way to get the lattice sheared just right is to set the sides of each parallelogram to $\sqrt a$ and $\sqrt c$ and the slope of the line to $\frac{b}{\sqrt{4ac}}$ . (I got this from Klein and verified it in a graphing calculator. Beats me. Visually it makes sense but I’m out of time to spell it out).

And once again: we can read out integer solutions to this equation by drawing a circle of radius $n$ around the origin, and counting the lattice points it goes through.

The right hand side of that equation is a binary quadratic form and number theorists talk about it as a function in its own right, $q(x,y) = ax^2 + bxy + cy^2.$ And now we’ve landed on a form, so modular forms are appearing over the horizon. What’s a form? What makes thing form? It’s something polynomial-like that obeys $P(tx) = t^kP(x)$ for some $k$ , and, in 2 variables, and you can figure more variables out yourself, $P(tx, ty) = t^kP(x, y).$ You see that? Linear scaling in the input becomes power scaling in the output. You can see this immediately for $P(x)=x^2$ , where $P(tx) = (tx)^2 = t^2x^2 = t^2P(x)$ . So $x^2$ is a form. When we have two variables as in $q(x,y)$ , then the exponents in each term need to sum to the same number, which gives you the $k$ . In $q(x,y)$ it’s $2$ . That makes it quadratic. And it’s binary because we stop at $y$ and don’t have a $z$ .

You can imagine why this property might be interesting to people. Distance travelled at constant acceleration or whatever.

Anyway, whatever modular forms are, we now know they are a kind of function, and we expect to see that relationship between its inputs and outputs pop up.

To get to modularity we now want to look more at the lattice and other things we can do to it. The key observation is this: if we have a transformation that sends every lattice point to another lattice point on the original lattice, then with it we can generate more solutions to questions like those above, analogous to extending out the line through the origin we saw with the circle. Since we’re interested in integer solutions we’ll want transformations we can express with integers. The easiest to find are rotating the lattice by right angles, which give transformations like $\begin{bmatrix}0 & -1 \\1 & 0\end{bmatrix}\begin{bmatrix}x \\ y\end{bmatrix}=\begin{bmatrix}-y \\ x\end{bmatrix}.$ Another one is to shift everything over one lattice point; the translation $\begin{bmatrix}1 & 1 \\ 0 & 1\end{bmatrix}.$ If you apply that one to $q$ you’ll get $q'(x,y) = a(x+y)^2+b(x+y)y+cy^2$ and if you plot that sucker $= n^2$ you’ll find it gives a new ellipse that intersects $q$ where $q$ goes through a lattice point, i.e. is an integer solution to $q'(x,y) = n^2$ . And if you expand out the terms you’ll find yourself back on a binary quadratic form.

You can apply these over and over with each other to get more elaborate transformations that stretch out the parallelograms. The thing they all have in common is they have determinant 1. We recall from 3blue1brown that the determinant is how a matrix will scale the area of the parallelogram, so what these matrices are doing is preserving the area of each cell in the lattice. So we have two ways of seeing why these matrices and their compositions will let lattice points line up 1-to-1 before and after: firstly because we’ve built them out of simple individual steps that always do this (and it turns out any such integer $\det 1$ matrix can be built this way), and secondly because we know the area never changes, so the density of the lattice is always the same.

Mathematicians call these–2x2 integer matrices with determinant 1–unimodular. I have no idea why. Since you can multiply them together with abandon and always get back a unimodular matrix, $\det AB = \det A \cdot \det B$ , that makes these things collectively a group, the modular group.

We’ve got modular. We’ve got forms. We can see what they each have to do with lattices and what lattices have to do with number theory. But still, they’re somewhat disconnected.

To tie these things together we need complex numbers. We’ve been thinking about lattices on the plane this whole time, and now we think of each lattice point as $z = x + iy$ . And as usual with complex numbers we can now think of the action of multiplying by $z$ as rotating and scaling the complex plane, so $z$ sends the lattice point $(1,0$ ) to $(x, y$ ).

To line things up the same way mathematicians do, here’s another way to define a lattice. Take two complex numbers, $\omega_1$ and $\omega_2$ (this is the notation everyone uses), and so long as they do not lie on the same line, we can generate a lattice with $m\omega_1 + n\omega_2$ , for integer $m, n$ , stepping out in steps of $\omega_1$ as $m$ increases and in steps of $\omega_2$ as $n$ increases.

If you think about it a bit you can see a purely real $\omega_1$ and a purely imaginary $\omega_2$ will give you a square lattice. Non-orthogonal $\omega$ will give parallelograms.

Now, in this context we’re interested in lattices with integer parts in $\omega$ , and there are a lot of symmetries in the plane we want to get rid of, so applying a trick similar to shrinking the lattice to turn questions about integers into questions about rational numbers, we divide out the scale and imaginary part of $\omega_1$ , and substitute $\omega_1, \omega_2$ for $1, \tau = 1, \omega_2 / \omega_1$ . This is more like the lattices we’ve been talking about already to shear ellipses, where stepping out in $m$ will step along the $x$ axis. However, unlike before, we always step out in steps of $1$ , rather than $\sqrt a$ . Compare that to the rational number variant of the sums of squares question we started on.

Mathematicians loathe and fear the bottom half of the complex plane and so swap $\omega_1$ and $\omega_2$ freely to ensure $\tau$ lies on the positive half-plane.

So, now we want to think about a normalized lattice that has points in $x$ axis in steps of $1$ and into the upper half of the complex plane in steps of $\tau$ . We get $\tau$ from our choice of $\omega_1$ and $\omega_2$ .

This next step I can’t justify all that well and found I didn’t understand properly when I came to write it. We have a point on the lattice, $z$ . We would like to multiply it by a unimodular matrix and get back another point on the lattice. But we can’t multiply a 2x2 matrix by a single $z$ . We need a second term. It turns out we need homogeneous coordinates–you know, like from video games–and tack a one next to it. Then we get

$\begin{bmatrix}a & b \\ c & d\end{bmatrix}\begin{bmatrix}z \\ 1\end{bmatrix}= z\begin{bmatrix}a \\ c\end{bmatrix} + \begin{bmatrix}b \\ d\end{bmatrix}=\begin{bmatrix}az + b \\ az +d \end{bmatrix}$ which gives us two new $\omega_1$ and $\omega_2$ , and we scale that sucker back down to normalise back to $\tau$ , and so we the action of these matrices is understood the same as projection matrices. Written directly, that’s $\frac{az + b}{cz + d}.$ This seems truthy to me, in the sense of collapsing together structure as in our line-through-the-origin case, but I can’t explain it fully on it if you press me on it. Remember that here $a, b, c, d$ are all integers.

Now we come to modular forms proper. Before, forms were (implicitly) purely real, but now we see we’re knee deep in complex numbers. We had $P(tx) = t^kP(x).$ These are forms with respect to linear scaling. What we want are forms with respect to the modular group. This means we’re going to have some function on the complex plane, and it’s inputs are going to be related to its outputs by some power scaling law, after those inputs have been acted on by the modular group.

Let’s step back a bit. In this exposition, what we’re really interested in is the lattice. We want modular forms to tell us about the lattice; interest in modular forms in their own right comes later. The tack I’ve seen taken is to say what we’re really interested in are modular functions, rather than forms, $f(\frac{az + b}{cz + d})= f(z),\ \ \det([a\ b; c\ d])= 1$ which act more or less exactly like our lattice in that it doesn’t change after being acted on by unimodular matrices, but now for all $z$ in the complex plane. Upper half-plane. Whatever. The hope is that such functions can encode number-theoretic information we care about, the same way $q(x,y)=n^2$ reveals an ellipse.

They then relax this requirement, saying such functions are too hard to find or maybe uninteresting, and require instead $f(\frac{az+b}{cz+d})=(cz+d)^k f(z).$ That’s our modular form. So… why this? Where’d $az+b$ go? And how to interpret $k$ ? The way I interpret this is projective geometry-wise, so if you know homogenous coordinates you’ll be used to being able to pile up a bunch of transforms then divide out $w$ at the end to land back on a plane. That $(cz + d)$ is saying under a unimodular transformation, $z$ needs to be sent somewhere on the lattice? (I think!) of points scaling by $cz+d$ is able to identify, projectively, as the same point. But, this is pretty handwavey and I think I can only get this idea across to people who already know projection matrices and homogeneous coordinates and all that, and I don’t really have an good intuition for how its working here as opposed to in Euclidian 3D.

Naturally, mathematicians are even more restrictive about what kind of function $f$ can be, which I won’t bother with.

Here’s the punchline. We don’t have any such $f$ yet and this post is not really going to go into them. My goal has been to get at why if you find such an $f$ , number theory tends to drop out of it, without definition overload. I’ll go over two kinds quickly.

The first is the historical first, the lemniscate elliptic functions. They arose when Euler was investigating the physics of… elastic ribbons. They were particularly interested in this figure 8 curve, called a lemniscate (Latin for ribbon, naturally), which Gauss figured out a $\sin$ and $\cos$ analogue to, and these turn out to be ‘doubly periodic’ when extended to complex numbers in the same way as these lattices.

The second is a much more direct construction that I think was come up with to nail this case, the Weierstrass elliptic functions. Conceptually, you just drop a divide by zero on every lattice point. This makes the lattice ‘puncture’ the complex plane which I find weirdly satisfying. You’d like to be able to just do

$f(z) = \sum_{m,n} \frac{1}{(z - (m + n\tau))^2}$ and make it so that as $z$ nears a lattice point the $z-(m + n\tau)$ term for that point goes to zero, but this turns out not to converge. Fixing it is a nice trick and then showing it’s a modular form is also simple if you are comfortable whippin' sums. For an explanation of this I really recommend the Stein and Sharkachi book, it’s good and will occasionally repeat information that needs repeating, a rare skill among mathematicians.

Putting these to work in real theorems and such is also another matter entirely. So this post hasn’t captured the character and feel of how modular forms are used at all. If you do have any skill in that, and you try to define modular forms, you start in weird places, and start talking about q-series (not the $q$ above), and rational points on elliptic curves, and Fourier expansions of partition functions, and

Some low discrepancy noise functions

Wed, 10 Aug 2022 00:00:00 +0000

This post is about an attempt to generate blue noise at a point: no state, no offline training, just arithmetic on an index. Honestly I don’t have a good reason for wanting this but it’s probably been at the back of my mind for like 2 years–it just seemed like it ought to be possible. It turns out to be doable in 1D and with a little lookup table we can push it to 2D.

The kind of blue noise we’re going to get is the classic “3dB per octave"¹ blue noise, which might turn out to be not all that useful, who knows. But it will also be low discrepancy, and the tricks in this post to keep it that way all the way to 2D are pretty cool I think. The plan is to take a sequence we already know has good discrepancy, whiten its frequency spectrum, and then turn that into blue noise.

Motivating example: dither

So: low discrepancy blue noise. Breaking that down:

Low discrepancy: values the noise takes on are evenly spread out. Read on a bit if this is unclear.
Blue: no or little low frequencies in the noise’s frequency spectrum.
Noise: unpredictable.

A problem I had writing this post is that it’s not obvious why you’d even want noise with these properties, and the technical answer–reduce variance–is a bit mystifying. This was a good excuse for me to look at dithering a bit more closely. Bart Wronski has a nice series of posts on dither if it’s new to you, but hopefully you can follow this either way.

Here, we want to represent a 64x64 gray square, with a gray value of $0.5$ , using only black and white pixels. If we know ahead of time that it is solid gray, we could use a checkerboard pattern as a way to approximate “gray”. But suppose we don’t, and we decide to choose whether to make a pixel black or white by drawing a random value in $[0,1)$ and comparing it to the source pixel; if the random value is less, black, if it’s greater, white. Since we’re using random values, we can do this multiple times and get a different dither pattern every time. Using some functions I’ll get into below, we get this.

PRNG

Low discrepancy

Blue

The PRNG is quite bad. By bad I mean like, to my eye, the third square seems to have too many white pixels. So, yes, it will give 1:1 white and black pixels on average, but that’s averaged over all the dithering you could ever do. That’s the problem: we don’t care about the quality of unrealized dither.

The middle column, maybe you can convince yourself it looks different, I dunno. It’s subtle. If you take it upon yourself to count number of white pixels you’ll find that very close to half of them are white, in all four squares. That’s low discrepancy. The noise values we got covered $[0,1)$ much more evenly than random noise. This gives us frequencies of black and white pixels that are always close to the frequencies of the dither we know would be ideal for solid gray.

However, that’s not true over the entire square; there are clumps of black pixels and clumps of white pixels. That’s what blue noise addresses. Which you can see just looking at it, right. This gives spatial frequencies of black and white pixels that better match the ideal dither.

That’s two different kinds of frequency–one in the histogram sense and another in the spectrum sense. I don’t quite know how independent they are. On the one hand, once you have some noise values, you can just rearrange them to get different frequency spectrums with the same discrepancy. On the other, if you want the noise to be low discrepancy ‘everywhere’, without any rearrangement, that rules out long lasting low frequency content. What you’ll find is the more samples you are willing to wait around for until a given run of draws becomes low discrepancy, the more freedom in shaping the spectrum you’ll have. Noise that is low discrepancy everywhere and at all scales has to be blue, though, I think.

Back to the 64x64 squares. Generating a thousand each, the standard deviation of the number of white pixels in a random PRNG square is 32 (!). It’s less than 1 for both the low discrepancy versions. Counting just the top left quarter, at 32x32, it becomes 16, 12, and 2, for the PRNG, white and blue noise respectively. That’s what reducing variance means.

Background

The API that I want is a counter hash, so a function that takes in an index, hashes it, and spits out a noise value at that index. No state carries forward from index to index, every value is computed independently. Like this:

// integer index to 0.32 fixed point
uint32_t noise(uint32_t index);
// usage
for (uint32_t i = 0; i < 512; i++) {
// convert the noise to a float in [0, 1)
 float x = (float)noise(i) * 1.0p-32f;
// ... do something with x
}

We’re going to follow Practical Hash-based Owen Scrambling by taking a sequence we already know is low discrepancy and shuffling it. We’ll need the nested uniform scramble from that paper, too. As for the sequence, the golden ratio sequence is very easy to compute in the form I want, so we’ll use that.

import np
i32 = np.int32
u32 = np.uint32
def golden_ratio_sequence(i: u32) -> u32:
return u32(i) * u32(2654435769) # 0.618... in 0.32 fixed point
def reverse_bits32(x: u32) -> u32:
x = u32(x)
x = ((x >> u32(1)) & u32(0x55555555)) | ((x & u32(0x55555555)) << u32(1))
x = ((x >> u32(2)) & u32(0x33333333)) | ((x & u32(0x33333333)) << u32(2))
x = ((x >> u32(4)) & u32(0x0F0F0F0F)) | ((x & u32(0x0F0F0F0F)) << u32(4))
x = ((x >> u32(8)) & u32(0x00FF00FF)) | ((x & u32(0x00FF00FF)) << u32(8))
x = ( x >> u32(16) ) | ( x << u32(16))
return x
def nested_uniform_scramble(x: u32) -> u32:
x = reverse_bits32(x)
x ^= x * u32(0x6c50b47c)
x ^= x * u32(0xb82f1e52)
x ^= x * u32(0xc7afe638)
x ^= x * u32(0x8d22f6e6)
return reverse_bits32(x)
def okay_blue_noise(i: u32) -> u32:
return golden_ratio_sequence(nested_uniform_scramble(i))

You can get a reordering of any sequence you like by hashing the index with nested uniforms scramble.

It’s a bit intimidating, but the important thing about it is each hex constant there is an even number. In binary, multiplying by an even number sends bits to the left, so each xor permutes bits in a low-to-high direction. The authors wanted to permute bits high-to-low, and so reverse the bit order before and after. So, it’s a hash, where the action of each bit on the hash is constrained to only permute bits below itself. And the xors are invertible, meaning each input maps to a unique output.

A consequence of this that can give you some intuition for what it does is that the high bit of the input is always preserved. So, as a counter hash, the first $2^n$ inputs map to u32s below $2^n$ . And since you know that holds for $2^{n+1}$ as well, you also get that $[2^n, 2^{n+1}]$ maps to $[2^n, 2^{n+1}]$ . In these intervals, the shuffled sequence produces the same values as the underlying sequence, just in a different order. This doesn’t fully explain its behaviour as a hash but it’s enough for us to get going with.

By the way, you can also think of the (fixed point) golden ratio sequence as a shuffle of the integers in u32. They all appear exactly once.

This has all been background, but before we get to the good stuff, let’s look at what we have:

import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (9,2)
def spectrum(seq):
S = np.fft.rfft(2 * seq - 1.0)
return np.abs(S) / len(seq)
def plots(name: str, seq_u32):
seq = seq_u32 * np.ldexp(1, -32)
figure, (histogram_axis, spectrum_axis) = plt.subplots(1, 2)
histogram_axis.hist(seq, 128)
histogram_axis.set_xlabel(f"{name} histogram: {len(seq)} points")
dft = spectrum(seq)
spectrum_axis.plot(dft)
spectrum_axis.set_xlabel(f"{name} spectrum: {len(seq)} points")
n = 3333

Bad histogram…

from numpy.random import default_rng
rng = default_rng()
plots("PRNG", rng.integers(0, 0x1_0000_0000, size=n))

Bad spectrum…

i = np.arange(n)
plots("golden ratio sequence", golden_ratio_sequence(i))

Noise with a flat histogram?

plots("nested uniform scramble-shuffled grs", okay_blue_noise(i))

This already has some low frequency attenuation. Well, if you drop the last bit reversal step and the golden ratio sequence and instead use it directly, not as a shuffle, it comes out even better. But my plan here is to clear the way by getting to white noise, and to use that as a base for more noise colours, so forget I said anything.²

Xorshift

We need more unpredictability. To that end, let’s look at using xorshift as a counter hash. Specifically, lets look at the low bits.

def xorshift(x: u32) -> u32:
x = u32(x)
x ^= x << u32(13)
x ^= x >> u32(17)
x ^= x << u32(5)
return x
i = np.arange(16)
print(i)
print(xorshift(i) & 0b1111)

[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15]
[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15]

Uh, hold on,

xorshift(i + 64 + 16) & 0b1111

[ 5 4 7 6 1 0 3 2 13 12 15 14 9 8 11 10]

The lower 4 bits are a (not very random) permutation of the original 4-bit sequence!! The Xorshift* variants preserve this property and is at least superficially random-looking so let’s use that instead from now on (constant from Melissa O’Neill):

def xorshift_star(x: u32) -> u32:
x = u32(x)
x ^= x << u32(13)
x ^= x >> u32(17)
x ^= x << u32(5)
x *= u32(0x9e02ad0d)
return x
xorshift_star(i + 64 + 16) & 0b1111

[ 1 4 11 14 13 0 7 10 9 12 3 6 5 8 15 2]

This works for powers of two up to $2^{16}$ . This isn’t a good way of generating permutations–it can’t make all that many–but we don’t need it to be good. LCGs work this way, too.

Now, because of this we can bestow xorshift with the high-bit-preserving property³ of nested uniform scramble by fixing the high bit and masking off any new bits placed above it. And we know that when we do this, every input still maps to a unique output, so long as we only scramble the lower 2 bytes. Using all 16 bits is way too random, so we can cap the number of bits we allow to be scrambled this way, and use the rest as a seed that selects a permutation. Looks like this:

def all_ones_below_high_bit(x: u32) -> u32:
x = u32(x)
x |= (x >> u32(16))
x |= (x >> u32(8))
x |= (x >> u32(4))
x |= (x >> u32(2))
x |= (x >> u32(1))
# this last shift: the high bit is not included
return x >> u32(1)
def unfolded_masked_xorshift(x: u32, cap_mask: u32) -> u32:
mask = all_ones_below_high_bit(x & cap_mask)
upper = x & ~mask
lower = xorshift_star(x) & mask
result = upper + lower
return result

Every bit we include in this scramble increases discrepancy–bits can permute bits to their left, here. We can claw back a bit by treating the high bit of cap_mask as a kind of sign bit and flipping all this logic to work backwards when its set.

def masked_xorshift(x: u32, bits: u32 = 8) -> u32:
# all ones if (x & 0x100) == 0x100, all zeros otherwise
sign_mask = i32(x << u32(31 - bits)) >> i32(31)
sign_mask = u32(sign_mask)
cap_mask = u32((1 << bits) - 1)
return unfolded_masked_xorshift(x ^ sign_mask, cap_mask) ^ sign_mask

This is all pretty abstract. You can see what it does by plotting the difference with its input:

i = np.arange(1024)
plt.plot(i - masked_xorshift(i))

As a shuffle, this shows you where elements are being moved to, relative to their original position. With bits = 8, that distance is never more than 128. So, while this scramble doesn’t have all the nice properties of nested uniform scramble, we do get a version of the $[2^n, 2^{n+1}]$ bijection property that repeats inside every chunk of size 256. So discrepancy-wise it’s kind of bad, but we can bound the bad with bits.

Now we’ve got low discrepancy white noise:

def white_shuffle(i: u32) -> u32:
s = i
s = masked_xorshift(s)
s = nested_uniform_scramble(s)
return s
def white(i: u32) -> u32:
return golden_ratio_sequence(white_shuffle(i))
i = np.arange(3333)
plots("white", white(i))

Look closely at the lowest frequencies of the spectrum. There is a slight gradient away from zero. This is what bits controls: the slope of that gradient. I found that at 11 bits it’s qualitatively pure white noise with no obvious correlations (… aside from an usually flat histogram). But it’s just too much random. For example, the variance in the dither example really suffers. And variance is what we care about.

Next: gotta eyeball it in 2D. You gotta.

plt.rcParams['image.cmap'] = 'gray'
plt.rcParams['image.interpolation'] = 'none'
px = 1/plt.rcParams['figure.dpi']
n = 512
i = np.arange(n*n)
f, ax = plt.subplots(figsize=(n*px, n*px)); ax.axis('off')
ax.imshow(white(i).reshape((n,n)) * np.ldexp(1, -32))

Ah. The edges. It’s bad. Okay, that’s our choice of bits. It’s too small. But we don’t have to increase bits, we can get away with doing another nested uniform scramble. This is kind of painful on x64, where bit reversal spews instructions everywhere, but, eh. There’s a trade-off between performance and discrepancy here and I completely lack any insight on it.

def white_shuffle(i: u32) -> u32:
s = i
s = nested_uniform_scramble(s)
s = masked_xorshift(s)
s = nested_uniform_scramble(s)
return s
f, ax = plt.subplots(figsize=(n*px, n*px)); ax.axis('off')
ax.imshow(white(i).reshape((n,n)) * np.ldexp(1, -32))

👍

Blue noise

Now, to transmute this into blue noise. Filtering the noise directly destroys low discrepancy, so no perfectly tuned cut off frequency and ripple here. But I think what I’m about to suggest might seem to come out of nowhere for some people. With that in mind, here’s what I was thinking about when I realised this: the unshuffled golden ratio sequence has the property that

golden_ratio_sequence(i * -1) == 1 - golden_ratio_sequence(i)

In other words, you can go backwards from zero, and it’s the same, just inverted around $0.5$ . So what if we interleaved the forwards sequence with the backwards sequence? The two sequences would meet in the middle. Seems natural, right?

This turns out to do something very specific in the frequency domain: it fills in the new frequency bins–twice as many samples, twice as many bins–with a copy of the frequency spectrum then high pass filters the whole thing. This is entirely due to the right hand side of the above equation, so when we use that form, it also works on the shuffled sequence.

But why!? If you’re comfortable with DSP this can be hand-waved as being the composition of three steps: zero-stuffing to repeat the frequency spectrum⁴, applying the simple low-pass filter $[1, 1]$ , and then reversing the frequency spectrum by inverting every second sample. This turns the low-pass filter we just applied into a high-pass filter.

Even so, that last bit might need more explaining if you haven’t seen it before. This is directed at DSP people (don’t worry about it). First, in this context, $1 - a_i=-1\cdot a_i,$ because in 0.32 fixed point, we have 0x1_0000_0000 - a[i] = 0 - a[i]. From this we get that inverting every second sample is equivalent to point-wise multiplication by the signal

$x[n] = ...-1,1,-1,1... = e^{i\pi n}$

which is your frequency shift in the time domain⁵ operator set to $\pi$ , a half turn. This places all the negative frequencies into the positive part of spectrum, and because of the conjugate symmetry of real signal DFTs, they are the mirror image of the positive frequencies.

DSP talk’s over. Intuitively, something like this should happen–low frequencies are slow movement, high frequencies are rapid movement. Forcing the sequence to repeatedly jump from one half to the other pushes all the frequencies higher.

For ‘noise’ this has the weird property that every second value is just the previous value flipped around $0.5$ . A gray code round can get rid of that–rightward invertible action of the bits like before by the way, this is a scramble. Shift here is by 6 because reserving the high bits preserves the frequency spectrum. That’s where all the energy in the signal is.

def kronecker_sequence(i: u32, a: u32) -> u32:
return u32(i) * u32(a)
def blue(i: u32) -> u32:
s = white_shuffle(i >> 1)
b = kronecker_sequence(s, 2654435770) # 0.31 fixed point golden ratio
odd = u32(i & 1)
b ^= (odd ^ u32(1)) - u32(1); # negate on odd indices
b += odd
b ^= b >> u32(6) # gray code round
return b
i = np.arange(3333)
plots("blue", blue(i))

Observations without narrative flow

We can get red noise by not doing the inversion step. This duplicates every element, which is bad, but we can scramble every odd element. (Technically, scrambling is a thing you do to the values of a low discrepancy sequence, and shuffling a thing a thing you do to their indexes. In fixed point they are kind of duals to one another, at least when you have a proper scramble, like nested uniform scramble is.) This raises the spectre of a scrambled odd value colliding with an unscrambled even value. None of this is ideal from a discrepancy point of view, but whatever.

There is another purely real rotation we can do: a quarter turn of the frequency spectrum, using $e^{i\pi n/2} = \ldots -1,0,1,0,-1,0,1 \ldots$ You can fill in the zeros with another sequence generated with another shuffled Kronecker sequence. Martin Robert’s R2 works well. A quarter turn of red noise and a quarter turn of blue noise gets you green? and.. violet? I guess? noise? With the middle frequencies band-passed or cut. Unfortunately, it doesn’t extend to 2D using the technique I’m about to describe so I’m dropping it here. That it doesn’t work is interesting in its own right, I suppose.

blue doesn’t have equidistribution. That is, some inputs map to the same output. 0 appears twice and who knows about 0x8000_0000. But it’s also lost an entire bit in the interleaving of two sequences. These are both length $2^{32}$ and they interleave to a sequence of length $2^{33}$ . We only have 32 bits to index it with. Rounding the constants to have a zero bit in the last place (and a one bit in the second last place) gives us a pair of sequences of length $2^{31}$ . Now every value before the gray code round is even, but we know exactly what values we’re missing, the odd values, and what values we are double counting, the even values. I’m not bothered about that low bit, because we’re going to round it away anyway on conversion to float, and if you want integers, the high bits are better.

2D blue noise

Dumping it out in 2D doesn’t work, but the way it doesn’t work is instructive so let’s look at it.

def spectrum_2d(img):
dft = np.abs(np.fft.fftshift(np.fft.fft2(img)))
dft /= dft.shape[0]
return np.clip(dft, 0, 1)
def lookit(image):
f, ax = plt.subplots(figsize=(2*image.shape[0]*px, image.shape[1]*px))
ax.axis('off')
ax.imshow(np.hstack((image, spectrum_2d(image))))
n = 384
i = np.arange(n*n).reshape((n,n))
noise = blue(i) * np.ldexp(1, -32)
lookit(noise)

It’s blue horizontally, but in no other direction. Okay. At this point, we can generate another noise image, transpose it so it’s blue vertically, and drop it on top, like

noise[x,y] + more_noise[y,x]

and yeah it works but the spectrum isn’t that good and the triangle distribution you get out of the sum isn’t all that low-discrepancy either. If you don’t care about discrepancy, you could use some cheap PRNG here instead of low discrepancy noise and filter this right in a shader. Since you could design the filter to smooth out some of the anisotropy this method has, I think it could be pretty good.

But to preserve low discrepancy we can decide that the issue is the path the noise takes over the image, and so let’s find a better path. I tried to find a way to do this with just arithmetic, and I got nowhere. But I did find a path that works that’s easy to compute offline, and it turns out we can tile the path. So this might be an okay compromise.

No imaginative thinking here, horizontal path bad circular path good. This bins pixels into concentric rings, then pixels within each ring are sorted by their angle.

def spiral(n: u32, lo: float, hi: float):
x, y = np.meshgrid(np.linspace(lo, hi, n), np.linspace(lo, hi, n))
# two sqrts: one for the distance, two to adjust for spirals closer to
# the origin being tighter (think random sampling in a circle)
xy = np.round(np.sqrt(np.sqrt(x*x + y*y)) * np.sqrt(n*n + n*n))
# rescaled to a reasonable range that makes debugging possible without
# a third eye
angles = (np.arctan2(y, x) + np.pi) / (2 * np.pi)
# sort by magnitudes then angles (I don't know why lexsort is
# little-endian), then invert the sort
spiral = np.lexsort((angles.flatten(), xy.flatten())).argsort()
return spiral.reshape((n,n))
s = spiral(n, -1.0, 1.0)
noise = blue(s) * np.ldexp(1, -32)
lookit(noise)

… looks weird. Something to do with the origin. Don’t wanna think about it.

s = spiral(n, 2.0, 4.0)
noise = blue(s) * np.ldexp(1, -32)
lookit(noise)

Nice. One reason this path is hard to replicate/approximate using only arithmetic on x and y is that it wraps at the edges in a nice symmetry breaking way that I don’t really understand:

spiral(8, 2.0, 4.0)

[[ 0 2 1 6 10 20 19 32]
[ 4 3 7 12 11 21 34 33]
[ 5 8 14 13 23 22 35 47]
[ 9 16 15 25 24 37 36 48]
[18 17 27 26 39 38 49 56]
[30 29 28 41 40 51 50 57]
[31 44 43 42 53 52 59 58]
[46 45 55 54 62 61 60 63]]

And here it is with a 64x64 path, tiled using the z-order curve:

def left_shift_2(x: u32) -> u32:
x = (x ^ (x << 16)) & 0x0000ffff
x = (x ^ (x << 8)) & 0x00ff00ff
x = (x ^ (x << 4)) & 0x0f0f0f0f
x = (x ^ (x << 2)) & 0x33333333
x = (x ^ (x << 1)) & 0x55555555
return u32(x)
def z_order(x: u32, y: u32) -> u32:
return left_shift_2(x) + (left_shift_2(y) << u32(1))
tile_bits = 6
tile_n = 1 << tile_bits
tile_mask = tile_n - 1
tile_path = spiral(tile_n, 2.0, 4.0)
def blue_2d(x: u32, y: u32) -> u32:
x_lo = x & tile_mask
y_lo = y & tile_mask
x_hi = x >> tile_bits
y_hi = y >> tile_bits
tile = z_order(x_hi, y_hi)
i = (tile << u32(2*tile_bits)) + tile_path[y_lo, x_lo]
return blue(i)
x, y = np.meshgrid(np.arange(n), np.arange(n))
noise = blue_2d(x, y) * np.ldexp(1, -32)
lookit(noise)

Histogram’s still good:

plt.hist(noise.flatten(), 384)

So that’s a 8KB lookup table that’s good for as much noise as you can index with an integer, and you can always get more constants to replace the golden ratio with if you need more. (I’m not super satisfied with this, I wanted 0KB). Also some tiling artifacts where there is a too consistent lack of correlation between adjacent pixels at tile boundaries, which is weird to think about. If you didn’t see it forget I said anything!!

Part of the reason this works at all is that the quality of the noise that we’re threading through the image is not, spectrum-wise, all that good. With better noise, not only do you see the tiling clearly, the spectrum is really no better–it’s determined by the spiral function in a way I don’t understand. So I’m not sure if this could be extended to 3D with just a path. But even if it does, for higher dimensions I doubt it would work all that well without huge lookup tables, because as the number of dimensions goes up the number of edge voxels increase faster than the number of interior voxels.

Dither revisited

Since we started with dither, here’s some real dither. Usually for dither you want triangular noise rather than uniform. Triangular refers to the shape of the histogram. Alan Wolfe has a nice way of getting a triangle distribution out of uniform noise that perfectly preserves discrepancy, so I’m going to use that. Hastily converted to python:

def uniform_to_triangle_dist(x):
# From demofox @ https://www.shadertoy.com/view/4t2SDh
x = (x + 0.5) % 1
orig = x * 2.0 - 1.0
nz = orig != 0
x[~nz] = -1
x[nz] = orig[nz] / np.sqrt(np.abs(orig[nz]))
x = x - np.sign(orig) + 0.5
x = (x - 0.5) * 0.5 + 0.5
return x

An alternative would be to use R2 to get another noise value to add to the first, you only need an extra multiply after all the shuffling to get it. Yet another alternative would be use more noise offset by half a tile to obscure the tiling artifacts.

Code’s here. Quantized to 4 bits with 2 bits of dither. The top row is a gradient we’re dithering and what you get quantizing without dithering. Then it’s left: uniform noise dither, right: triangle noise dither. And top to bottom: random, this post’s white noise, this post’s blue noise, and last I’ve put known-good void and cluster blue noise.

Yup, it dithers 👍 Not as good as a good texture, but that’s okay.

The white noise… I don’t know what I expected.

Bonus

The low bits of the spiral function have a blue spectrum:

s = (spiral(384, -0.08, 0.08) & 7) / 8
lookit(s)

Which suggests maybe we can reverse the bits and get something interesting out?

s = u32(spiral(384, -0.08, 0.08))
s = masked_xorshift(s, 2)
s ^= s >> 1
s = reverse_bits32(s)
s = nested_uniform_scramble(s)
lookit(s / s.max())

Huh. Conceptually, this is just a reordering of pixels–you could replace that divide by a shift to turn it into a true permutation. It’s not noise, though. I tried some things, like rotating some tiles, placing the origin of the spiral at the corners, adding random jitter to the magnitude and angles in spiral, but nothing worked very well at getting rid of the obvious structure while maintaining the spectrum. This is interesting, though.

Bonus 2

I have no idea what is going on here.

def red(i: u32) -> u32:
s = white_shuffle(i >> 1)
r = kronecker_sequence(s, 2654435770) # 0.31 fixed point golden ratio
r[(i & 1) == 1] ^= r[(i & 1) == 1] >> u32(6)
return r
def red_2d(x: u32, y: u32) -> u32:
# note the right shift--this isn't z-order
i = left_shift_2(x) + (left_shift_2(y) >> u32(1))
return red(i)
noise = red_2d(x, y) * np.ldexp(1, -32)
lookit(noise)

Histogram’s okay, too, but it’s pretty bad within the bins.

plt.hist(noise.flatten(), 384)

Bottom text

Here’s the code for this post without all my words around it.

Also known as “linear”. Who comes up with this stuff? ↩︎
An interesting reason to use the golden ratio sequence is it turns out it’s easy to come up with more constants–that is, aside from the golden ratio–that let you generate multiple streams of correlated noise cheaply. R2 is one way, and there are others. But it’s a whole thing. ↩︎
If it seems weird that I’m so singularly focused on this property of nested uniform scramble it’s because I figured out masked_xorshift before I found out about that paper, and I’m not sure I would have figured it out if I thought nested uniform scramble was the model to follow.⁶ ↩︎
This fact is pretty elementary DSP-wise but Bart Wronski’s post was the only thing I could find that discusses it at a high level without assuming you are the sort of person who thinks that negative frequencies are a reasonable idea. ↩︎
I told you not to worry about it!! But I can’t find any explanation of this that isn’t laden down with DSP nonsense so while I’m at it: the DFT gives you a representation of a signal as a sum of scaled-and-shifted sine waves, right? And you shift a signal around in time and all the sine waves change phase, in lockstep with each other. In the DFT sine waves are encoded as complex numbers–a magnitude for the sine wave’s scale, and an angle, for its phase–and it just so happens that multiplying a complex number by unit complex numbers only modifies the angle. So shifting signals in time can be done in the frequency domain by multiplying the complex coefficient of each sine wave by exp(1j * carefully_chosen_shift_amount_per_sine_wave * 2 * pi). The shorter the wavelength, the larger the shift, because shorter waves complete more of their cycle per unit time.

That’s shifting in time. If you do the exact same thing in the time domain–multiply the time domain samples by those unit complex numbers–you shift the frequency components around. That’s shifting in frequency. Okay? Okay. ↩︎
And the reason I keep italicizing nested uniform scramble is because nested uniform scramble is a nested uniform scramble. There are others, you see. ↩︎

Difference Decay

Wed, 29 Dec 2021 00:00:00 +0000

Here’s a variation on the damper I keep coming up with uses for. I find my code that uses it a bit subtle and annoying to figure out, hence this post.

That theorangeduck post is about springs, which could be an interesting extension to this, but this post is setting our sights lower.

My first use for this was to clean up a noisy/unreliable clock by using a high resolution clock, taking the noisy clock’s drift as authoritative. But I think of it more generally, as combining two signals s and n to produce a third with the short-term character of s but the long-term average of n:

float update(float *accumulator, float dt, float s, float n)
{
float acc = *accumulator;
float err = s - n;
acc += err;
acc *= expf(-dt);
float x = n + acc;
acc -= err;
*accumulator = acc;
return x;
}

accumulator is initialized to $0$ and we expect dt to be small and positive, so you can mentally substitute expf(-dt) with 1 - dt.

The easiest way to explain it is to work backwards. The +=, -= pair removes a delay term (err_prev in the following), and is equivalent to:

err = s - n;
acc = (acc + (err - err_prev)) * expf(-dt);
err_prev = err;
x = n + acc;

To see what this is doing, suppose we don’t apply the expf(-dt) decay factor. Then, over all s and n, we get something like cumsum([0, diff(s - n)]), which is a no-op. Written out,

$\begin{aligned} \text{acc}_i &= \text{acc}_{i-1} + (\text{err}_i - \text{err}_{i-1}) \\ &= \sum_{k=0}^i (\text{err}_k - \text{err}_{k-1}) \\ &= \sum_{k=0}^i \text{err}_k - \sum_{k=1}^i \text{err}_{k-1} \\ &= \text{err}_i \\ &= s_i - n_i \end{aligned}$

so our value for x would always equal the most recent s. Likewise, if we zeroed out acc completely, the damper would return n every time.

Applying the decay to acc lets us decay old updates to the difference between $s$ and $n$ . If $s$ and $n$ grow at the different rates, this shrinks the gap between them and stops them drifting apart. But if $n$ updates less frequently or with jitter, this fills in the missing/incorrect detail with $s$ .

If you care about tuning the decay factor or having something more momentum than exponential decay, check out the theorangeduck post. It’s good!! One thing to note is this damper doesn’t have any stability problems for large dt that I’m aware of, you just lose more history.

While retreading the maths for this post and trying to see if I could get it to look more intuitively damper-y, I noticed that if you scale the error difference term by the inverse of the decay to cancel out the first decay you get

$\begin{aligned} \text{acc}_i & = [\text{acc}_{i-1}+(\text{err}_i - \text{err}_{i-1}) \exp(dt)] \exp(-dt) \\ &= \text{acc}_{i-1} \exp(-dt) +(\text{err}_i - \text{err}_{i-1}) \end{aligned}$

which looks similar to the simple damper:

x = lerp(x, g, 1.0f - expf(-dt))
= g + (x - g)*expf(-dt)

Instead of repeatedly folding g into x, we’ve isolated x - g as acc, and we repeatedly fold updates to g into acc, which we decay. This idea of isolating the difference is what I was thinking about when I came up with this; it’s really another phase inversion trick. So maybe applying this scaling factor is more correct?

Anyway, I came up with this to fix a decades old visual stuttering issue in the rhythm game Etterna, a fork of Stepmania. They position objects on the screen by repeatedly querying the system audio API for it’s playback position. This turns out to work exactly as you’d hope on some hardware and APIs, and really not work at all on others. The game is visually stripped down enough that this looks like it has bad frame pacing issues, but the issue was entirely due to the reported audio position. There are variations here (figured out with Tracy):

WaveOut, interestingly, appears to directly report whatever the hardware driver tells it. This means it works great for some devices and terrible on others. I first hit this issue when a driver update for my USB sound card changed the behaviour here, introducing jitter you could see. On the other hand, with my motherboard’s audio you instead see nice steady drift away from QueryPerformanceCounter.
DirectSound appears to correct for drift and jitter, and this turns out to be bad. When you sample playback positions it looks to be in sync with wall time, but audio hardware does drift, and DirectSound eventually corrects for this by jumping. Which means a visible jump in the game. Furthermore, it only reports a new sample position every 10ms, and at the point you are querying you have no idea how long ago the update to the position occurred. You basically end up with a few milliseconds of jitter, unless you’re running at very high frame rates.
ALSA and PulseAudio appear to have no way to query this, so you only know the time you submitted the last buffer. This is probably fine, and possibly none of this would have been a problem if the game never queried the other APIs in the first place. You can get a continuous estimate by extrapolating from the submission time, which I suspect is what you get in the good WaveOut case. Due to the architecture of the game, this ended up with the same unknown update time issue as DirectSound.
And possibly more, but I stopped looking.

So I wanted something that would eliminate jitter for all of these, without any parameter tuning, and not degrade the ideal ‘good driver under WaveOut’ case in any way. This worked well.

One thing here worth mentioning, where s is high-quality time from QueryPerformanceCounter or equivalent and n is low-quality time from somewhere else, is that we only have samples $s(t_0)$ and $n(t_1)$ . That is, we do not actually know the values of $t$ , and we just assume $t_0 \approx t_1$ . For this problem in particular, you might want to take a third sample $s(t_2)$ and check that $s(t_2) - s(t_0)$ is sufficiently small before using $n(t_1)$ , as your time slice can run out between sampling $s$ and $n$ .

I wish I could say something about control theory here, since this kind of thing seems to be right in its wheelhouse. But I don’t really know any.

stb_ds: string interning

Thu, 27 Aug 2020 00:00:00 +0000

stb_ds is a generic container library for C. It was probably from seeing the same technique in bitwise, but I’ve had it at the back of my mind for a while now that one of the good use cases for stb_ds was easy string interning. But it’s not mentioned anywhere, not even in the gentle introduction! Well, the pieces are there, but how to put them together is a little subtle.

By the way, string interning is when you keep at most one copy of a given string in memory, and use immutable references to such copies as your string value type. In return, you get $O(1)$ string equality and you use less memory.

You can do string interning with a single stb_ds hash table, and that’s it. I’m pretty sure stb_ds is designed to ensure that this is possible–you can’t do it with most hash tables. Several features fitting together just right makes it work.

First, table entries are stored contiguously in memory, and their order is stable, even when the hash table is resized. This is in constrast to other hash table implementations that store their data in sparse arrays. This isn’t just an implementation detail but a documented part of the API. For more on why you might want a hash table that does this, check out this blog post from PyPy.

Second, stb_ds can hash strings for use as key types. Once you have string interning, you only need primitive type keys for your hash tables. But if you don’t have proper string keys before you have interning, you need to figure out what you’re doing. We don’t have to worry about it.

Third, string hash tables can be configured to store their keys in a memory arena managed by the hash table. This means we won’t have to manage the intern pool memory at all. The arena gives us stable pointers and minimises the number of calls to the underlying allocator.

And lastly, although stb_ds’s hash table maps keys to values, it’s possible to use it without specifying values at all. (And you don’t waste any space doing this.)

All of these are useful in their own right! But check this out:

typedef struct { char *key; } Intern;
static Intern *interns = NULL;
ptrdiff_t intern(char *str)
{
if (str == NULL) {
return -1;
}
if (interns == NULL) {
sh_new_arena(interns);
}
ptrdiff_t result = shgeti(interns, str);
if (result == -1) {
shputs(interns, (Intern) { .key = str });
result = shlen(interns) - 1;
}
return result;
}
char *from_intern(ptrdiff_t handle)
{
char *result = NULL;
if (handle >= 0 && handle < shlen(interns)) {
result = interns[handle].key;
}
return result;
}

Table entries go into the hash table as just a key, without a corresponding value. To get a value to use as a handle, we use their index in the table’s key storage. When we want to turn a handle back into char *, we index off the interns pointer.

Since pointers to the interned strings are stable, you might want to hand out those pointers directly, instead of handles. On the other hand, pointer-sized handles are somewhat big on 64-bit systems, so there you might want to use a smaller handle type. On 64-bit, the above code is a bit worst-of-both-worlds, but I wanted to show the idea as cleanly as possible.¹

If you’re using a version of stb_ds older than v0.65 the above code won’t work, as using shputs will store the pointer in key directly instead of a pointer to the copy it will allocate from the arena. Prior to v0.65, you had to manage the intern pool’s memory directly. That’s okay, because stb_ds exposes its internal memory arena functionality as part of the API. The idea is the same, just slightly less convenient.

Also, I first call shgeti there, but you really ought to be able to get away with just calling shputs and calling it a day. If you use pointers instead of handles this works, because shputs returns the inserted key. But I’m not convinced you can rely on that for future versions.

And… I think that’s it.

If they’re smaller than pointer-sized, handles can overflow. I figure if I post code it should be working code that can be copy-pasted somewhere without blowing up some indeterminate time in the future. Although, I’m not keen on the behaviour it has with null strings and invalid handles. I feel like both of those indicate that bad things are happening, and returning null pointers is just adding fuel to the fire. ↩︎

deep sky object

Wed, 20 May 2020 00:00:00 +0000

A couple months ago I set @tiny_dso running. It’s a twitter art bot, which I guess is less of an exciting thing nowadays, but it’s an idea I’ve wanted to do for at least a few years now: turn @tiny_star_field's tweets into computer generated imitations of astrophotography.

tiny_star_field is getting a bit intermittent nowadays, so I set the bot to work it’s way backwards through tiny_star_field's old tweets. I wanted to make it something you could follow for a long time and still be surprised by, rather than something you’d scroll down once and see everything it’s capable of. We’ll see if I succeeded there; I’m not sure. It’s hard to tweak the parameters for that kind of thing, especially since when I’m working on it I generate hundreds of images, which skews my perception of what’s happening too much or not enough.

It’s been running for a while, so I can let it speak for itself:

I don’t have it in me to write more about it with a coherent structure so I’m just going to dump some notes on the implementation here. I wrote most of the code a couple years ago, returned to it a couple times, and really just decided to get it running on twitter recently. There’s probably some stuff I intended to do that I’ve completely forgotten about, so this is mostly technical details I can read out of the code.

The bot part that talks to twitter is just javascript that lives on glitch. The image generator is an executable that spits pngs out of standard out.

It’s written in ion, the language Per Vognsen created for bitwise. It’s basically C99, except you can omit type names sometimes, you use . instead of ->, and there is some notion of modules. Oh, and it has out of order declarations. I think when I started writing this the language was fully 2 weeks old. It looked pretty cool and I didn’t want to use any libraries in this project¹, so it fit my needs fine. As new and unfinished compilers go, it was pretty reliable. But it didn’t receive a whole lot of development past that. Coming back to the code to get the images onto twitter I found it had some bugs to work around, mostly to do with getting it to generate code that would compile on linux².

I found tweaking the image generation intolerable if it took more than half a second or so, so I spent some time optimising it. That means multi-threading, vector instructions, and a lot of profiling. This wasn’t a problem with ion, because it can generate C code, and will take your word for it if you tell it some function or type exists. Also, it uses the C preprocessor to get line information to debuggers (and profilers). All in all, very nice. Good language.

I learned early on that the easiest way to render a passable looking star was to place a single white pixel on an otherwise blank texture and just use mipmaps to filter it for rendering. Rotating a shrunken, very bright, pixel gets you endless variation on how the stars look.

Pixel values fall in the range $[0, 1]$ and, if you think on it, stars really ought to be much brighter than that. So this method works best if you just let your values overflow then adaptively crunch the image back down with post-processing. Most astrophotography has gone through reams of processing, anyway.

All the “pixel shader”-like work is done using a pair of functions, pixel_iter_begin and pixel_iter_next. Rather than explaining what they do, here’s the code for drawing a texture:

func draw_tex(dest: Image*, target: Rect, tex: Tex*) {
lod := compute_lod_level(dest.size, target.size, tex.size);
for (it := pixel_iter_begin(dest, target); pixel_iter_next(&it)) {
rgba := tex_lookup_lod(tex, it.npos, lod);
*it.pixel = color_blend(*it.pixel, rgba);
}
}

This would work in C, but it’d look like for (PixelIter it = ...; pixel_iter_next(&it);) {} . Note that trailing semicolon–next gets called before, not after, every iteration. Now, this way of iterating over pixels is very general and so pretty slow. It just makes doing pixel-by-pixel stuff extremely low friction to write code for. I think most programmers would reach for function pointers or generics to separate the pixel shading code from the iteration, but it’s not necessary, and a pain to actually use. It turns out that the iterator code was not performance critical at all, so the overhead didn’t matter much, and the implementation is also very naive.

The pixel iterator also takes care of multi-threading. Here’s the definition of the Image struct:

struct Image {
pixels: Color*;
size: int2;
wr: WritableRegion;
stride: int;
offset: int;
}

The WritableRegion allows the pixel iterator to clip the pixels being iterated over to the block the thread is responsible for rendering. The code that actually uses the pixel iterator doesn’t have to think about blocks or threads at all. It’s nice!

The drawback is that if you want to draw into a side buffer before compositing into a destination buffer, then you end up allocating the full size and only use a block in the middle somewhere. Avoiding that is what stride and offset are for. Usually, you’d sample an individual pixel like this:

sample := img.pixels[pos.x + pos.y*img.x];

Instead, I do this:

sample := img.pixels[pos.x + pos.y*img.stride - img.offset];

This way, you can sample an image by talking about coordinates in $[0, 1]^2$ while the image only has storage allocated for $[\frac{3}{8}, \frac{4}{8}]^2$ (for example), and all it takes is an extra subtraction.³

All the star colours are chosen by linearly interpolating between RGB values obtained by colour-picking from wikipedia. They’re pretty close so I never had to do anything fancy involving splines or colour space transformations.

Most of the look comes from blurring the entire image and layering the blurred and unblurred parts together in arbitrary ways. Think blend modes in image editors. It turns out it’s pretty easy to write a fast blur, and you don’t even have to think about how to vectorise it because everything is in 4 independent colour components already.

The diffraction spikes are blurs, too. I think a lot of people jump to the fourier transform to do diffraction spikes but I couldn’t be bothered with that–just take a box filter with a hole cut out of the middle and repeat it a few times. Same principle as using iterated box filters to approximate a gaussian blur, as in the links above, but non-separable this time, so you need to do vertical and horizontal passes both on the original image (i.e., not in series).

If you look at actual diffraction spikes you can see different colours get diffracted more or less. I think this is probably the same principle as ground waves, where lower frequency radio waves diffract around the surface of the Earth more than higher frequency waves. So, I use different sizes of filter on each colour component; larger for larger wavelengths, I think, but you can only do so much in RGB.

Honestly, though, the diffraction code is kind of terrible. Every time I think about it I think of a better way to implement it. For example, I haven’t vectorised it, because the memory accesses are different for each colour component. But to get diffractions at an angle I do a song and dance where I rotate a copy of the entire image to another buffer–this would be a good time to rearrange the data into colour planes to make it vectorisable: multiple rows of a plane at once. I could even reuse the other blur code at that point, and implement the convolution (A - B)x as Ax - Bx. But I don’t! And I never will, now.

The nebulas are a hot mess of noise functions, mostly cellular noise with a healthy dose of domain warping. It all happens on a 2D plane; I didn’t want to think about 3D volumetric anything for this project.

I do the cellular noise in fixed-point arithmetic. This was out of curiosity more anything else; the implementation comes pretty naturally from wanting to use a certain SSE2 instruction. I’d like to write more about it, but not in this post.

By the way, glitch is pretty cool. It spins up an instance of who-knows-what for you and clang is just sitting there, waiting to compile whatever you want. A very old version of clang. That doesn’t have the intrinsics I use. Well, it has gcc, too.

If you’ve scrolled all the way down here and only now have decided you want the bot’s twitter, here you go.

Except stb_image_write.h for the pngs. On Windows, I don’t write pngs, but render to a buffer managed by SDL. So they both have one dependency, but it’s a different dependency, I guess. ↩︎
I think most, if not all, of those bugs have been fixed in this fork. ↩︎
You could also shunt the pixels pointer off into no-mans-land and pray you get all the new weird boundary conditions right. ↩︎

Server-side KaTeX With Hugo: Part 2

Sun, 19 Jan 2020 00:00:00 +0000

So, despite saying I didn’t want to do this in my last post on this, I went and forked Hugo. Now rendering with KaTeX is way faster, and basically free when using hugo server.

This was more driven by curiosity than a desire to make things fast. At some point a couple weeks ago I was reminded of QuickJS, and from there it was a short series of small steps to my downfall. QuickJS really enabled this. It was easy to replace the javascript from my previous post with an executable that ran the qjs interpreter on KateX bytecode, and easy to then link it to a Haskell program that drove Pandoc as a library rather than using the command line. Having done that, getting Goldmark, Hugo’s markdown processor, to render TeX using KaTeX was also a small amount of work.

Rather than this post just being, “hey, I’m doing this now instead,” I guess I’ll talk a bit more about it.

Goldmark is fairly extensible so shoe-horning in TeX-awareness just means telling Goldmark’s parser to call our code when it hits a $: I put maths between $ and $$, so $x$ gets rendered as $x$ . But taking responsibility for parsing any markdown yourself means you have to think about weird edge cases¹.

Take links. Links in markdown have the following syntax: [foo](bar.tld). So, how should [$foo](bar.tld$) be parsed? In my opinion, there is a correct answer here: that’s a link. I’ve found people citing some old RFC that prohibits $ in urls, but they’re valid. Anyone who writes [$](en.wikipedia.org/wiki/$) wants that to be a link, and I can’t think of any exceptions.

By the way, that $ in the previous paragraph also needs to be parsed as a $, and not the opening dollar of maths.

So, those are links. What about [$[0, 1]$](url)? There’s something satisfying about being able to link maths: $[0,1]$ . So, that first example we want to not parse as TeX, and the second we do, and in order to support both we have to know if we are inside a link or not. Parsing!

Thankfully, we have a working TeX-aware markdown processor already: Pandoc. After fixing up some minor differences between how Goldmark and Pandoc renders the HTML, Pandoc, with the old filter from last time, can generate a bunch of test cases. I don’t follow Pandoc’s example in one place; Pandoc’s rule for allowing $[]$ inside links seems to be that the brackets must be balanced. I opted for requiring [ to appear before ] (which would otherwise close the link).

Next, some threading stuff. Hugo uses goroutines, and the QuickJS runtime can only be used single-threaded. We don’t want to make each goroutine queue to access QuickJS, and we also want to keep our changes to Hugo to a minimum. From a C perspective, there’s a really obvious solution: give each thread its own QuickJS runtime using thread-local storage. But goroutines aren’t threads, and the Go scheduler wants to schedule goroutines to any thread it likes. This means that between two calls into QuickJS the goroutine can move thread, and whatever way we have of communicating with the QuickJS runtime from Go needs to be okay with this happening.

What I decided to do was to just never allocate anything that would be passed back to Go, and have the Go code pass in memory instead. This makes the C code pure as far as Go is concerned, so we can use thread-local and not worry about the Go scheduler. It also means we don’t have to call free, which is always good.

Last time, I reported that a single page with TeX took half a second to render. Now, my entire site takes 240ms. Without KaTeX, it’s around 150ms. I still find this a bit slower than it really ought to be, but I suppose for how little work it was, it’s pretty good.

The Commonmark spec is a great resource for all the markdown parsing gotchas you’ve never thought about before. ↩︎

Calculating LOD

Tue, 31 Dec 2019 13:29:00 +0000

Angelo Pesce (@kenpex) recently tweeted:

This is a calibration poll. Please don't spoil the answer, RT appreciated - Do you know how a GPU knows which mips to fetch when a regular tex2D(tex,uv) is called with trilinear filtering?
— c0de517e/AngeloPesce (@kenpex) December 3, 2019

I would have counted myself as ‘not sure’, but I had a pretty good idea what it would involve and I was also pretty sure I could figure it out. Instead of just being pretty sure about it I decided to test my intuitions against a GPU and made this shadertoy.

Most of the code is machinery to get a triangle up that we can rotate around in 3D space. I find it vaguely amusing to write a triangle rasteriser in a fragment shader, so that’s what I did. This post is just about LOD, so if you’re interested in how the rasteriser works I recommend starting with Fabien Giesen’s The Barycentric Conspiracy.

So, mipmaps: we have a texture with a series of successively smaller mip levels, each filtered properly prior to rendering to avoid aliasing. In this post I’m just going to assume textures are always square, and always a power of two in size. These restrictions can be lifted but I don’t really see why you would, except maybe to save space when using very large non-square textures.

2D

I’d already done this in 2D for a small project of mine, so this coloured my intuition a bit.

I was just drawing textures in squares of different orientations and sizes–it turns out an easy way to draw stars is to put a single very bright pixel in a big texture, and let mipmaps take care of the rest. So, here, we just have to do log2(texture_size / max(target_width, target_height)), that is, the $\log_2$ of the inverse of the amount of scaling applied to the texture. To see why, each mip has relative sizes 1/1, 1/2, 1/4, ..., 1/n. We can get the relative size of the target rect by looking at its width and height, and we want to turn that relative size into an index into this list of sizes for which we have mips. So, we need the function $2^{-n} \mapsto -n$ . That’s $\log_2$ .

Using $\log_2$ also takes care of the fractional part of mip selection, so we can do trilinear filtering. If you think about it this is somewhat interesting, because we’re blending in a larger mip onto a target we already know is too small to not alias (in general). But having a jump discontuinity between mips looks way worse than low levels of aliasing.

3D

The main thing about 3D is that the LOD level can change across the surface of a triangle.

We need to think a bit more carefully about what’s happening. As we step along pixels in screen space, we also step along texels in texture space. While the former is constant, the latter steps vary in size due to perspective. Reasoning this way works in 2D, too, we just never had to: the constant, screen space step is our 1 in the series of relative mip sizes, and the relative, texture space step size is the denominator up to n. The question is how to get the step size at a given pixel, given that we shade each pixel independently of its neighbours. Now, I guess if you know the graphics pipeline quite well the answer to this jumps out at you, but I want to be able to justify why you would use the functions you do.

So, in 2D, the texture space step size is constant, and we’re able to compute it by assuming we’re going to draw the entire texture and comparing its original size with its new width and height in screen space. Now, to get the step size at a point in 3D, we can approximate it by asking about a smaller part of the texture, say a 2x2 patch rather than the entire thing, at that point. To drive the point home here, in 2D we had

$\frac{ T(\textbf{0} + \textbf{x} ) - T(\textbf{0}) }{ \left\| \textbf{x} \right\| }$

with $T$ being some transform and $\textbf{x}$ being one side of the texture, and we’re replacing it with

$\frac{ T(\textbf{p} + h\textbf{x} ) - T(\textbf{p}) }{ h\left\| \textbf{x} \right\| }$

and making $h$ really small. So, we need two of these for the derivatives in $x$ and $y$ , and you can get an approximation of them–basically this same one here–in OpenGL with dFdx() and dFdy().

In the shadertoy, this is all implemented like this.

// uv comes in as an interpolated input
vec2 size = vec2(textureSize(iChannel0, 0));
vec2 dx = dFdx(uv * size);
vec2 dy = dFdy(uv * size);
float d = max(length(dx), length(dy));
float lod = max(0., log2(d));
vec3 col = textureLod(tex, p, lod).xyz;

Eyeballing the results this seems to work, but if we take the difference between textureLod() with our lod value versus using texture() and scale it up a bit to magnify errors, we get this.

It should be black. The problem is obvious in retrospect, but rather than thinking too hard about it I had a look through the DX11 Functional Specification, which tells you to do this to the derivative vectors:

float A = dx.y * dx.y + dy.y * dy.y;
float B = -2.0 * (dx.x * dx.y + dy.x * dy.y);
float C = dx.x * dx.x + dy.x * dy.x;
float F = dx.x * dy.y - dy.x * dx.y;
F = F*F;
float p = A - C;
float q = A + C;
float t = sqrt(p*p + B*B);
dx.x = sqrt(F * (t+p) / (t * (q+t)));
dx.y = sqrt(F * (t-p) / (t * (q+t))) * sign(B);
dy.x = sqrt(F * (t-p) / (t * (q-t))) * -sign(B);
dy.y = sqrt(F * (t+p) / (t * (q-t)));

Ellipses

The source it references only as [Heckbert 89] turns out to be a master’s thesis, Fundamentals of Texture Mapping and Image Warping, which includes this in its appendices. The DX11 spec says it gives you a “proper orthogonal Jacobian matrix”, where proper is not a technical term. So, this is looking at the projection as a 2D transform from a point $(x, y)$ in screen space to a point $(u, v)$ in texture space. Now, it’s not linear, so you can’t represent it with a 2x2 matrix, but you can look at its derivative to get a linear approximation at a point. That’s the Jacobian, and we can construct it by just putting our derivative vectors in the columns of a matrix. But this Jacobian won’t be proper.

Basically, the issue is that up until now I’ve been neglecting to consider off-axis scaling. To solve this, we want to look at how the unit circle is transformed, not just the axes. That is, we look at the ellipse formed by applying the Jacobian to the unit circle. Weirdly, a pretty good explanation of this idea can be found in the Wikipedia article on singular value decomposition. Anyway, here’s a shadertoy showing how the above code modifies the axes:

The right ellipse’s axes have been obtained by orthogonalising those on the left. The important thing is that both pairs of axes describe the same ellipse. The reason I went through the effort of creating that shadertoy, though, is because I wanted to check that putting those axes into the columns of two matrices would actually give you matrices that created the same ellipse. Because it’s a completely different transform, right, one of them isn’t even orthogonal! Of course, it does give you the same ellipse, and I guess the reason why that’s possible is obvious: each point on the plane is sent to a different place in either matrix, but the unit circle traces out the same ellipse for both. To that end, I’ve coloured each point by its original position inside the unit circle.

What this gives you is the maximum scaling in all directions, not just the directions the texture’s $x$ and $y$ axes have been mapped to in screen space.

Now, applying that correction, we’re much closer:

Still not perfect, though. I’m not sure what’s missing. DX11 allows for approximations here, and OpenGL’s specification doesn’t really say anything at all about it, so maybe that’s it. I’m really not sure.

Blocks

Finally, when I first wrote the rasteriser I did it the natural way, something like

if (inside_triangle) {
color = calculate_color()
write_pixel(color)
}

But if you look at the shadertoy now, you’ll see it’s written as

color = calculate_color()
if (inside_triangle) {
write_pixel(color)
}

which looks like it wastes a bunch of effort colouring pixels it never writes. The reason for this is that the dFdx and dFdy functions aren’t magic: they really do take an approximation of the derivative by subtracting neighbouring pixel values from each other in 2x2 blocks. Since we’re running the shader in parallel for each pixel in the block, different pixels in the block can branch in different directions. At that point, the GPU is missing neighbouring values for dFdx/dFdy to compute derivatives with. The derivative computation breaks down and we can’t sample mipmaps at all. On my machine, this gave a LOD value of 0 which resulted in a seam of aliasing down the edges of triangles. At first I thought this was a bug in the inclusion test, but no, it’s just old fashioned non-uniform flow control.

End

I’m kinda disappointed I wasn’t able to get the LOD computation exact even after going through DX11 and OpenGL specifications. And it is just the LOD that’s wrong–with fixed LOD, the rasteriser does trilinear filtering down to texelFetch() just fine. Also, I’ve not talked about anisotropic filtering at all, which I’ve now discovered I know even less about than I thought I did. I’d like to know more about how it’s implemented on GPUs but it appears to be like, a trade secret, or something.

I think writing this post took longer than writing the shadertoy.

Server-side KaTeX With Hugo

Sun, 15 Dec 2019 18:01:49 +0000

Update (2020-01-19): I lasted some 3 weeks before gettng rid of this. Also, I’ve changed the title from ‘static’ to ‘server-side’ which is more accurate.

Hugo is a static website generator written in Go. Katex is a math typesetting library written in Javascript. This site uses them both.

I keep javascript turned off on websites unless white-listed. I notice when websites fail to load without javascript and I notice when websites don’t require it at all. I’d like mine to be one of the latter, but also, I want to have proper maths typesetting. Now, usually the way you get maths onto the internet is you dump some LaTeX surrounded by $$ into a document, embed some random javascript file, and everything happens automatically. I don’t like that.

What I want instead is for Hugo to typeset my maths. But I still want hugo server to render everything properly, and I don’t want a complicated multi-stage build process, and I really don’t want to maintain my own build of Hugo. Basically, I want to be able to dump the Hugo executable somewhere and have everything work when I type

cd blog
hugo

Unfortunately, Hugo doesn’t really support Katex per se. It uses Markdown parsers that know to leave text surrounded by $$ alone, and Katex searches for them at page-load. These parsers are compiled into Hugo, and at no point can you intercept them to do further processing of your own.

But Hugo supports Pandoc, and Pandoc is happy to pass you a copy of the AST to fiddle with before it actually renders anything. What’s more, Pandoc knows that $ means inline maths and $$ means display maths, so we can just read that information off of the AST and have Katex format it as inline or display appropriately.

So, the outlines of a solution are starting to appear. We need to write a Pandoc filter that replaces LaTeX maths with Katex’s HTML typesetting.

But there is a problem. There are two problems.

Hugo’s support of Pandoc is fairly limited. The command-line parameters it uses are hard-coded into Hugo, and they seem to just be whatever the guy who committed the PR adding Pandoc support needed (thanks to that guy, by the way. We couldn’t get even this far without him). We need to pass our own arguments to get Pandoc to filter anything.
Pandoc doesn’t support passing arguments to filters. We can tell Pandoc to execute node, but we can’t tell Pandoc to execute node katex.js. This seems to be a weird artifact of how Pandoc shells out to filters, at least on Windows.

There is a simple, elegant, and generally terrible solution to both of these problems.

The reason Hugo can call out to Pandoc, and the reason Pandoc can call out to Node, at all, is because they’re both in PATH. So, we just have to arrange PATH to our liking before running Hugo. I always run Hugo from VSCode, so I can do this on a per-project basis, and not contaminate the rest of my system with this, ah, workaround.

I didn’t investigate the best way to do this; I suspect on Linux it would be a bit easier. But, on Windows, I didn’t want to think too hard about it, so I just compiled a couple C programs. Now I have the following folder structure:

blog/
├── content/
├── ...
└── pandoc/
├── node_modules/
├── katex.exe
├── katex.js
└── pandoc.exe

The first directory in PATH whenever I run Hugo is path/to/blog/pandoc. node_modules contains only Katex, required by katex.js, the pandoc filter. katex.exe opens a pipe to node pandoc/katex.js and pandoc.exe opens a pipe to pandoc --katex --filter=katex. Full paths to these are hard-coded into the executables, because I never learn.

Then, in the front matter for a post that needs Katex, we use the Pandoc renderer:

---
title: "Static Katex With Hugo"
date: 2019-12-16T02:01:49Z
markup: pandoc
---

The code for the executables and Pandoc filter are pretty rote. Here’s a gist, if you’re interested.

There’s one big problem with this. It’s slow. For every post with maths to typeset, we have to shell out to pandoc and then it has to shell out to Node. This takes around 500ms per page for me, as opposed to 40ms for other pages. Pandoc without a filter, by the way, takes about 80ms. I can live with this for single posts as I’m writing, but I suspect before long I won’t be able to stand how long a full build takes.

That 500ms is mostly spent starting up Node and initialising Katex. So, one solution is to keep Node running and pass all the posts to a single instance. That doesn’t sound too hard to do myself, but my hope is Hugo will have made some progress on that themselves before I feel the need to–there are now so many preprocessors written in javascript they’re already thinking about it, as far as I know. But what you really want is a LaTeX to HTML renderer written in Go.

$e^{i\pi} + 1 = 0$

The Discrete Fourier Transform, But With Triangles

Sat, 14 Dec 2019 18:34:03 +0100

Here’s something I’ve been wondering about lately: what happens if you replace all the sine waves in the fourier transform with triangle waves? If you are like me and would like to play around with such a thing, this post has a method to get a representation of a signal as scaled-and-shifted band-limited triangle waves in $O(n\log n)$ time. This is fast enough that we can compute first and ask questions about what it’s good for later (I’ve yet to find anything). Actually, the stuff in this post will work for any periodic function, not just triangle waves, but this was the idea that motivated me to figure this out.

To get more precise in what I’m trying to do here, here’s what I want. The DFT takes a signal made up of sine waves with amplitudes $a_k$ , phases $\theta_k$ , and integer frequencies $k$ , and produces a signal that has the value $a_ke^{i\theta_k}$ at $k$ . I want a transform that does the same for triangle waves.

I had a few bad starts on this–for one thing, triangle waves are not orthogonal to each other at all–but it all came together when I asked the following question: What is a cosine wave given as a sum of triangle waves?

First, we need a definition of a triangle wave. With $K$ being the number of harmonics and $m = 2k + 1$ ,

$\text{tri}(x) = \sum_{k=0}^{K-1} \frac{1}{m^2} \cos(2\pi mx).$

I’ve adapted¹ this from the first of many definitions of a triangle wave on Wikipedia, where it’s presented without justification or citation. Personally, I don’t understand where it comes from, but it is pretty much the frequency domain represention of a triangle wave and that’s all we need. For the rest of this post, I’m going to leave $K$ implicitly defined as the largest value it can take without introducing aliasing in its current context.

For now, we just want to think about finding the $X_k$ that gives us

$\cos(2\pi \textbf{x}) = \sum_{k=0}^{N-1} X_k\text{tri}(k\textbf{x})$

where $\textbf{x}[n] = n/N$ . Restricting ourselves to band-limited triangle waves gives us the fact that the lowest frequency of $\text{tri}(k\textbf{x})$ is $k$ . Then $\text{tri}(\textbf{x})$ with $k = 1$ is the only member of our series that is not orthogonal to $\cos(2\pi \textbf{x})$ , so we know $X_1 = 1$ . But this introduces odd harmonics all the way up, and so the coefficient of the next odd triangle wave at $k = 3$ must use its lowest frequency to subtract out the first odd harmonic of the triangle wave at $k = 1$ , so $X_3 = -1/9$ . This adds yet more harmonics for later triangle waves to clean up. We keep going like this until we’ve accounted for all $N$ triangle waves.

If that was too quick, here it is in pictures. The frequency domain representation is overlaid on top.

It’s not hard to see that this logic mostly works for arbitrary functions, too. We lose the orthogonality to kick off the process, but the whole point of the DFT is that sines and cosines form an orthogonal basis, so we only need orthogonality to the lowest frequency. As we go, we successively remove frequencies, so there is always a new lowest frequency to remove².

Before we can implement this we also need to handle the phase of each frequency component. When we see $a_ke^{i\theta_k}$ at $k$ , we need to align the phase of the lowest frequency of $\text{tri}(kx)$ with $\theta_k$ . We’re working in the frequency domain, so we need this thing from the shift theorem

$X_n e^{-i2\pi n\ell/N}$

to apply a shift of $\ell$ . In our case, $X$ contains the $1/m^2$ factors of the current triangle wave. We just need to know $\ell$ . Now, the lowest frequency of $\text{tri}(kx)$ is $X_k$ , so we want to choose $\ell$ to satisfy

$X_n e^{-i2\pi n\ell/N} = X_k e^{i\theta_k} \quad \text{if } n = k.$

$\ell_k = -\frac{N}{2\pi}\frac{1}{k} \theta_k.$

This part seemed obvious when I was just turning ideas in my head into code but now that I write it out like this I’m not so sure.

This is all we need to find the $X_k$ in

$f(\textbf{x}) = \sum_{k=0}^{N-1} |X_k|\text{tri}(k\textbf{x} + \frac{1}{2\pi}\arg(X_k)).$

Iterative algorithms spelled out in maths notation always feel a bit clunky to me, so rather than doing that, here’s some Python code to do it:

import np
def cis(t):
return np.cos(t) + 1j*np.sin(t)
def inverse_tri_sum_bandlimited(x):
assert(np.isrealobj(x)) # real signals only. i'm lazy
n = len(x)
nyquist = n // 2
y = np.fft.fft(x)
m = np.arange(3, nyquist, 2)
coefs = 1 / (m*m)
for k in range(1, nyquist//2):
# indices of the non-zero components of a triangle wave
# with fundamental frequency k, not including the fundamental
nz = np.arange(k*3, nyquist, k*2)
# construct the frequency domain of the triangle wave and subtract it off
y[nz] -= np.abs(y[k]) * coefs[:len(nz)] * cis(nz * (np.angle(y[k]) / k))
# fix up conjugate symmetry
y[nyquist+1:] = np.conj(y[nyquist-1:0:-1])
return y

This makes use of the observation that once a frequency in y has been zeroed out we never look at it again, so we can do the algorithm in-place. Also, band-limited triangle waves above half the nyquist frequency are just sine waves, so we don’t need to consider them at all.

Some Graphs To Think About

The nice thing about this algorithm is the result is in the same representation as given by the DFT. So, if the DFT is $\mathcal{F}$ and our triangle transform is $\mathcal{T}$ , then we can replace all the triangle waves in a function with sine waves by applying $\mathcal{F}^{-1}\mathcal{T}$ . Right? So, this does what you might expect:

But look at this:

What gives? Well, this is a problem the DFT has, too. We’re getting a representation of the signal as a sum of triangle waves but $\text{tri}(1.5\textbf{x})$ isn’t one of them. In the analogous case for the DFT this isn’t so bad because it ends up being represented by nearby sine waves. You get a nice exponential drop-off away from the true frequency. For triangle waves, you don’t have that at all. Now, this is without windowing, so there is also a big jump between the last and first sample that we probably ought to do something about. Maybe if we apply a triangular window…

I don’t know what I expected.

Anyway, the reason I did all this was to see if anything interesting happened when you messed around with it. For example, we can swap out $\mathcal{F}$ for $\mathcal{T}$ in the convolution theorem. Here is a gaussian filter applied in the “triangle domain”, with $\sigma=2$ and with $\sigma=100$ . Image from Wikimedia.

Interesting! I don’t really have an intuition for what’s happening on for $\sigma = 2$ . Larger values of $\sigma$ look a bit more like what I expected.

I also tried filtering audio this way. Removing high frequency triangle waves–maybe I should give those a cute name like triquencies, because strictly speaking each one is multiple frequencies so I find it weird to refer to them that way–introduces high harmonics of low frequency content in the signal. This sounds really nice if you have something with lots of harmonic material–you get a nice wash of soft sawtooth buzz–and pretty bad on something with sparse hamonics, like a plucked string. It would be nice to be able to use this for some audio effect, buuut $\mathcal{T}$ isn’t linear, so you can’t use the usual methods to convolve small filters with long signals. Oh, well.

Odds and Ends

$O(n\log n)$ . I didn’t spell out this claim, however the naive algorithm is $O(n^2)$ . But if we skip triangle wave frequencies that are zero then we only need to do $n(\frac{1}{2} + \frac{1}{4} + \frac{1}{6} + \frac{1}{8} + ... + \frac{1}{n})$ operations. Apparently, that series grows logarithmically.

What about non-band-limited triangle waves? Well, it turns out the band-limited version is close enough you can just stick it in a non-linear least-squares solver and it’ll find you something near machine precision, despite the discontinuous Jacobian. But it’s just too slow for large problems to be all that interesting. Another thought is that if the problem is just that aliasing causes us to lose orthogonality, then can we not just upsample until the first $N$ band-limited triangle waves are effectively identical to non-band-limited? But when I tried this it would produce better results up to around 64x upsampling, and then stop, not even close to the least-squares method. I don’t really understand this at all! It could just be the problem is ill-defined but it seems unlikely to me.

At the top I said this would work for any periodic function: the DFT of any signal can be used in place of the fourier series of the triangle wave. But I’m not so interested in trying out other exotic functions like square waves as they would all have the problems laid out here, just because of how the series of functions was constructed. By band-limiting each basis function, we get a nice understanding of how the signal spans the fourier basis once shifted and sampled. But by that same token, only the complex exponential results in a series of orthogonal functions when constructed this way.

Now I can stop thinking about this.

Notably, I’ve swapped out the $\sin$ for $\cos$ , which also makes $(-1)^k$ term you might be familiar with disappear. I do this because in the DFT, cosines with zero phase are represented by purely real frequency components. If we align our triangle wave with cosine, then our triangle wave is also made of purely real frequency components. This makes the discussion a little easier. ↩︎
This is basically matching pursuit with a specific dictionary and a method to cheaply compute the next atom to remove. ↩︎

Dumb Tricks With Phase Inversion

Sun, 02 Jun 2019 12:34:03 +0100

Phase inversion takes a signal and flips it upside down . Most people are introduced to it as a button you push in a DAW while troubleshooting phase problems, but with some thinking it’s capable of a lot more than that. Some digging in the audio DSP literature will uncover a stray sentence or two that acknowledges this, but I get the impression that it strikes the people who already know it as too obvious to be worth mentioning. The really nice thing about it is that it all works in a DAW–you don’t need to break out Max/MSP or Reaktor to experiment with it.

I’m pretty sure I originally got the basic idea for all this from vladg/sound.

My intuition for this is fairly algebraic so I’ll be leaning on that a little, but all you need to have is an intuition for what is happening when you route audio around in a DAW. That said, I will be using notation from digital signal processing that’s too useful to give up: x[n] reads as the nth sample of a signal x. Then c[n] = a[n] + b[n] describes c by telling you how to compute the nth sample of c in terms of signals a and b. We could drop the [n], but this makes clear that we don’t need to know anything about a or b other than their nth samples to compute the nth sample of c. For anything more complicated than this, I’ll use graphics.

Diffing

Phase inversion negates a signal, y[n] = -x[n], so it turns addition (mixing) into subtraction.

+ = = -

The first thing to notice about this is if we add a signal to itself, phase inverted, we get silence.

+ =

There is a vocals isolation trick that relies on this. That’s not what this post is about.

What we’re interested in is signals before and after applying an audio effect. Some audio plugins have this built-in; Ableton’s EQ8 allows you to listen to each equaliser band in isolation. You could always do this yourself by flipping the phase of the equalised signal and mixing it back in with the original. Doing so in a DAW can be cumbersome, so when plugins do it for you, it can be very nice if this is all you want. But there is nothing stopping you taking the difference of a whole chain of effects, and you have to do it yourself for plugins that don’t have this feature.

But there’s a consequence to this that is not obvious. We can add the difference signal to the dry signal to recreate the wet signal. Algebraically, b[n] = a[n] + (b[n] - a[n]). This is what this post is about. Whenever we have an effect in a signal chain, we can interpret the effect as mixing in a difference signal to change the dry audio to the wet audio, and we are free to do any processing we want on the difference signal. Now, this is an interpretation, and it’s a wrong one. But we can have some fun with it.

By the way, I’m trying to keep this post DAW-agnostic but at the end I describe how I do this in Ableton Live in a self-contained way that doesn’t involve six hundred sends. So, check that out if you’re confused how to actually do this in a DAW.

Dry/Wet Mixing

The simplest thing we can do with the difference signal idea is to implement dry/wet mixing. This is less useful for single effects, which have this built in, than it is for effect chains, which don’t.

This is easy: turn the gain down on the difference signal. This will fade between from the wet to the dry signal. But if you’re willing to deal with a little algebra I think it’s worth exploring to what extent this is really dry/wet mixing, or something else. In dry/wet mixing, we mix two signals in such a way that the net gain sums to one,

c[n] = g*a[n] + (1 - g)*b[n],

where g fades between a when g is 0 (-∞ dB) and b when g is 1 (0 dB). By multiplying out and regathering we get

c[n] = b[n] + g*(a[n] - b[n])

so a dry/wet mix knob is directly equivalent to mixing in the difference signal. Note that a and b have swapped around; the phase of the difference signal has been inverted. In practice, you get to choose which way you want, which I’ll touch on later.

This is no big deal. Putting an effect chain on a return track and balancing it with the source track will be easiser most of the time.

Filters

This is well known among DSP people. A highpass filter can be made from a lowpass filter by subtracting the filtered signal from the original: hp[n] = x[n] - lp[n]. It works the other way around, too, and with bandpass filters. If you think about this for a moment this is kind of obvious but may be surprising. If a lowpass filter removes high frequencies, then whatever is left are low frequencies, and subtracting those from the original signal will get rid of those low frequencies. But our differencing operation operates on single samples from each signal–independent of any samples before or after–and a single sample is just a number without any frequency information. So, given a lowpass filter, there is some highpass filter for which we can know the value of hp[n] knowing only the two samples x[n] and lp[n]. Showing why this works mathematically isn’t hard¹ but something cool is happening here and I think it would be a mistake to dismiss it as obvious.

With a little tinkering we can start making bandpass and bandstop filters the same way, and go on to shelving and bell filters. But all we’ve done so far is recreate things we can already do, and I want to get to the good stuff.

Multiband Compression

The reason why you’d want to use the filters described above is that we can recover the original signal exactly by mixing them together again: x[n] = lp[n] + hp[n]. We don’t have to think about how we parameterise the filters or anything because we’ve used our phase inversion trick to make one filter from the other. Now all that is left is to compress just one of those filtered signals, do the sum, and we have a multiband compressor. Building this out in a DAW can be tricky but, to be clear here, it lets you combine any set of filter plugins and any set of compressor plugins into a multiband compressor. That includes parametric equalisers, and doesn’t have the weird overlapping spectrums that throwing bandpass filters around everywhere does.²

To do this, you need to remove the frequencies you want to compress, compress the difference signal, then add it back to the filtered signal. Before, we were adding the difference signal to the dry signal. Now, we’re adding the processed difference signal to the wet signal. We could have used the dry signal; we have a choice here between a[n] + compress(b - a)[n] and b[n] + compress(a - b)[n]. For compression, I find the latter much easier to reason about. The opposite way still works, but you’re adding in a phase-inverted copy of compressed frequencies to the dry signal. I have no intuition for that.

Compressors

The above has a degree of conceptual elegance, which we’ll now dispense with. Let’s take the signal

and use a compressor to flatten its peaks.

This sounds dreadful, but we can subtract it from the original signal to produce

which gives the peaks, and we can do whatever we want with them. But the isolation isn’t perfect. The decaying tone causes the compressor to dampen the signal more when the tone is loud, and some of the tone ends up in the peaks signal. What we care about is that they sum exactly to the original signal, and we can now process them independently.

Saturators

Applying a saturator to the difference signal of a saturator reduces the amount of distortion. The simplest saturator is the hard-clip , and the difference signal of this is also a saturator, . If we hard-clip this curve, we get , and adding this to the original signal produces the saturator curve . This saturator doesn’t clip very quiet or very loud signal values, only those somewhere inbetween.

One of the nice things about this is that good saturators have good anti-aliasing, and so will any curves you make this way.

Less esoterically, think of saturation as adding harmonic content. Taking the difference isolates the added harmonics, and you can apply EQ to them.

Another option is to EQ before saturating, take the difference between the EQ’d and saturated signals, then add that back to the original signal. Now you have a saturator that responds differently to certain frequencies. Mixing linear and non-linear effects this way will mess with any intuition you have for them–don’t think you’re being scientific when you do this.

Bit-depth Reduction

You can think of bit-depth reduction as an extreme saturation curve, . This particular curve reduces a signal to 3 bits of dynamic range: 000, 001, 010, 011, 100, 101, 110, and 111. The entire dynamic range is mapped to these 8 values, so there are 8 steps. Its difference signal is .

There are two ways to interpret it. One is that it’s wraparound distortion (see the pong~ object in Max/MSP). Another is that it’s giving you back the bits the other curve throws away. Either way, I like experimenting with this. I’m one of those unfortunate people who likes the sound of bit crushing, and you can use this to get a lot more variation (and dynamic range) out of it.

Doing It

The best way of actually doing this took me a while to figure out. I use Ableton Live, which has audio effect racks. They are almost enough to do everything we want to do, but if you’re using a DAW that instead relies on complex routing, it might be too unwieldly to use in practice. I’ve settled on using two audio effect racks, one for the a[n] + (b[n] - a[n]) case, and another for the b[n] + (a[n] - b[n]) case, which I call positive and negative.

The positive case is easiest, because a[n] is the original dry signal, and we can duplicate it as needed by adding more chains to an audio effect rack.

The outer A chain is empty. The inner -A chain contains a utility effect with both channels inverted. Then, you place any effect you want the difference signal of in the B chain, and can process the difference signal after the inner audio effect rack. The utility effects gain knob acts as a dry/wet mix; the effect is 100% wet at 0 dB.

The negative case is more tricky. Implementing b[n] + (a[n] - b[n]) requires routing that effect racks can’t do. You could produce b twice, but having to update the every parameter twice (or being limited to the effect rack’s macro controls) is a huge pain. Instead, I use the side chain from some of Ableton’s effects, which can capture audio before or after any effect rack. See here for an image, where I’ve used the compressor’s side chain, set to audition, to tap audio from the B chain before inverting it. To use this, we put an effect in the B chain, and everything after the inner effect rack is the difference signal. Collapse the side chain rack and never think about it. This effect’s final utility gain is 100% dry at 0 dB (if the difference signal is left unmodified).

One thing is missing from these: it’s really nice to be able to apply effects to the original signal after we’ve used it to produce a difference signal but before we’ve summed them back together. To do that, put another empty effect rack in the A (positive) or B (negative) chain, and use the side chain audition to tap that rack pre-fx. Now, anything before the emtpy rack gets differenced, and anything after only gets sent to the output.

I think you should figure out recreating this for yourself, or even find your own solution, because keeping track of what signal you’re producing and what needs to be inverted at what point in the signal chain is informative. My early attempts weren’t pretty. But if you’d rather just use mine, here’s Difference (Positive) and Difference (Negative) (Live 10. I don’t know if these things are backwards compatible, but I don’t use any features that are 10 only).

The process for finding the frequency content of a signal, the fourier transform, is linear. ↩︎
One thing I’m taking for granted here is that plugins correctly report the delay they introduce to the signal chain. It’s hard to write a compressor people want to use without knowing exactly how much delay you’re inserting into the signal chain, but equalisers and filters can introduce frequency-specific delays. I couldn’t say how common or uncommon it is for plugins to handle this properly. All this still works, but the difference signal contains information you don’t want it to contain. Then again, if a filter is messing with the phase of your audio, it’s already screwing up your audio! You don’t care about theoretical correctness if you’re still using that filter. ↩︎