The idea of this project is to detect clap patterns with the mic.
Let's start.
Here is the little code that records a stream of mic input signal (mac m1, 48K sample rate)
import numpy as np
import sounddevice as sd
import time
from datetime import datetime, timedelta
from bokeh.plotting import figure, show
def main():
device_info = sd.query_devices(kind="input")
assert isinstance(device_info, dict)
samplerate = device_info["default_samplerate"]
frames = np.array([], dtype=float)
def callback(indata: np.ndarray, *_) -> None:
nonlocal frames
frames = np.concatenate([frames, indata.reshape(-1)])
stream = sd.InputStream(
device=device_info["index"],
channels=1,
samplerate=samplerate,
callback=callback,
)
start = datetime.now()
try:
with stream:
while True:
time.sleep(1)
frames_now = start + timedelta(seconds=frames.size / samplerate)
print(datetime.now() - frames_now)
finally:
plot(frames, samplerate)
def plot(frames: np.ndarray, samplerate: int):
f = figure(width=1000, height=500)
f.line(np.arange(frames.size) / samplerate, frames)
show(f)
np.save('frames.npy', frames)
if __name__ == "__main__":
main()
Double snap.
Clap.
Keyboard.
Let's have a look at the double snap because I don't want to bother my neighbors. The clap is way easier, high amplitude.
Here we zoomed in the first snap.
It's not just a peak, if we look closely, identifying a peak would not work to infer a snap.
Let's record another, forgot to save the signal, and then we try to make this stuff more continuous to identify a peak by flipping the negative values.
Here is the new snap, a little less amplitude as it's getting late and I don't want to bother the neighbors.
The double one.
The first one of the double snap.
Let's flip the negative values.
Cool, now we have a peak, more or less. We can already see it's not a peak, many tiny peaks.
Let's try to denoise it then. This is what we currently have:
Moving average? Centered? 5 left + 5 rights / 10?
No, let's not do that, it's going to flatten the whole signal and we need to reason with another one. Why don't we try to downsample it by the length of peak / 3, so we see a few data points and for each window, we take the max.
The width is about 5 ms so, let's resample at 2 ms.
window = int(np.ceil(samplerate * new_freq))
abs_frames = downsample(abs_frames, window, lambda x: np.max(x, axis=-1))
x = downsample(x, window, lambda x: x[:, 0])
def downsample(x, window, agg):
assert window > 1, 'cannot downsample'
# x must be a multiple of window
to_pad = np.ceil(x.size / window) * window - x.size
x = np.concat([x, np.repeat(x[-1], to_pad)])
return agg(x.reshape(-1, window))
Alright, interesting, looks good on this one. Now find peaks will give us the peak.
peaks, _ = find_peaks(abs_frames, height=0.08)
Perfect.
Let's try it on the whole signal.
Perfect, we identify our snaps, let me try it on a signal where I type on the keyboard and also clap. Hopefully, the keyboard does not generate FP, should be lower. Let me close the window for my neighbors and try again.
Damn, didn't see that coming, the claps generated a lot of peaks, more or less in the same range as the snaps. That should be okay with my algo idea to identify the snap pattern based on an amount of time between them.
Food for thought, intuitively, it looks like the downsampling should be proportional to the amplitude of the signal.
The good news is the amplitude of peaks of my keyboard are way lower, which is an external one, more noisy than the one on my mac.
Actually, let's try to increase the downsampling window. The window should not exceed the distance between two peaks or they'll get merged. The width between the peaks of the snaps is around > 60ms. Let's try.
Ok, problem, we have a single peak and we need two. Why is that? Well, we need to make sure we sample a point in the valley in between.
So, the sampling should be distance / 2 (exclusive). 60 ms is actually shorter than the actual distance, we'll be safe. Let's try with 30 ms.
Ok, much better, we identified our peaks all good.
To avoid identifying the small peaks on the claps, let's add a little prominence to the find_peaks algo. Prominence is basically the height relative to shoulders.
find_peaks(Y, height=0.03, prominence=0.02)
One more thing I'm not happy with:
X = downsample(X, window, lambda x: x[:, 0])
Y = downsample(Y, window, lambda x: np.max(x, axis=-1))
I want the time (X) to be as precise as possible on the apex. Currently, it just takes the first time of the window which might not be the same as the apex. Let's fix that.
window = int(np.ceil(samplerate * new_freq))
window_arg_max = downsample(Y, window, lambda x: np.argmax(x, axis=-1))
downsample_idxs = np.arange(window_arg_max.size) * window + window_arg_max
X = X[downsample_idxs]
Y = Y[downsample_idxs]
If we hadn't done it, it could have been at most 30ms off.
Quick sanity check, looks good, apexes are aligned.
Before:
Alright, let's build this algo.
So, we get a list of peaks in a window. We want to apply some pattern to describe a double snap, for example, if we get 2 close peaks within this amplitude (around 0.8) within a specific distance, that's probably it. We also need to make sure there are no peaks left and right in a specific window of time or we might pick a sub pattern.
Then, we can build different patterns, for more complex snaps or claps series. But we'll do that later.
Let's think about how to represent this pattern.
We need two things:
- distance between peaks
- amplitude
Each of those is within a range, not going to be the exact same every time. We're not even talking about context (mic, environment, different hands, …), but let's try to make something work in a single context.
Basically, for each item, we want a range: [start, end]
And what about 2 lists? one for the amplitudes and another one for the distances. The distances will have one more item as we want to make sure we pad the pattern on each side.
Alright, so here is the algo. Let's not look too much into it, this is not the final one.
from dataclasses import dataclass
_range = tuple[float, float] # inclusive end
@dataclass
class Pattern:
name: str
distances: list[_range]
amplitudes: list[_range]
@dataclass
class Span:
pattern: Pattern
peak_start_idx: int
peak_end_idx: int # inclusive
def identify_patterns(
time: list[float],
amplitudes: list[float],
window_duration: float,
patterns: list[Pattern],
) -> list[Span]:
# we pad the peak lists to avoid conditions in the loop
time = [0] + list(time) + [window_duration]
amplitudes = [0] + list(amplitudes) + [0]
assert len(time) == len(amplitudes)
N = len(time)
spans = []
for pattern in patterns:
M = len(pattern.amplitudes)
i = 1
while i < N - M:
assert len(pattern.distances) == M + 1
if all(
dist[0] < time[i + j] - time[i + j - 1] <= dist[1]
for j, dist in enumerate(pattern.distances)
) and all(
ampl[0] < amplitudes[i + j] <= ampl[1]
for j, ampl in enumerate(pattern.amplitudes)
):
spans.append(
Span(
pattern=pattern,
peak_start_idx=i - 1,
peak_end_idx=i + M - 1 - 1,
)
)
i += M # we don't overlap matching patterns
else:
i += 1
# TODO: me might still overlap between patterns
return spans
This is what we get, our double clap was correctly identified.
Let's try to run it on a longer window and see if it works.
And oops, they are not detected. It seems peaks are being merged. Let's have a closer look.
The last one has ~15 ms in between peaks, so we would need to lower the threshold from 30 ms to 7ms. Which will generate FP on claps.
Let's try it.
Ok, that works pretty well. 3 have not been identified:
- 1: below height threshold
- *: snapped too fast or missed one. That's ok.
Let's now try to add a clap. We can expect lots of FP now that we have reduced the sampling threshold.
I will do double snap, clap and double snap.
That's not good :-) Too many FP on the clap as expected.
Let's rethink the peak identification algo.
The first 10 seconds are double snaps, then claps and finally keyboard.
I'm thinking of rescoping the project to just claps to avoid FP on key presses. Height can separate them pretty cleanly.
Before I do anything, let's get another sample with some fast food ambient background. A video I'll play on my phone.
Interesting, just a double snap and it got picked up.
Well, that might still be different in real conditions, ok, but it gives me some confidence.
Thinking that even if we get some FP key presses, they will not match a pattern as they are quite sparse. Or I could remove them by increasing the peak height threshold. Or probably use some context to identify them. Let's stick to the plan. Patterns over snaps and claps. Let's just try to get rid of the FPs on claps.
Alright, so new recording:
- 3 double snaps
- 3 claps
- 3 double claps
The boxes correspond to the left and right bases of the peaks. Generated because I used prominence. I was not expecting that.
https://en.wikipedia.org/wiki/Topographic_prominence
The prominence of a peak is the least drop in height necessary in order to get from the summit to any higher terrain.
So, here, 0.18, that's correct.
Let me plot the prominences of each peak.
Hehe, now, easy to filter out the peaks we don't want to. We can compute the base of the peak. base = apex - prominence.
base < 0.04
Alright, good, no FP.
Let me try to add some key presses.
Yes, their base is very low as higher peaks are far off, therefore minimum valley is baseline basically.
Do we still need to downsample the signal?
Let's try.
Yes 😂
I'm tempted to try to fix the keyboard part, but let's keep it for later.
Alright, time to build the pattern builder.
The idea is to record a set of sounds for a specific pattern. The double snap for example, and then build a pattern from the set. We will expect each item in the set to have the same number of peaks. For each peak, we'll aggregate some statistics about the amplitude and the distance between peaks. That's going to be our pattern.
Done. Here we have the double snaps, 9 recordings. 2 didn't get 2 peaks.
The others (aligned on first peak)
This is the pattern we've identified. Took min/max for now. We could register more of them and take a different percentile to avoid outsiders.
{
"name": "double_snap",
"distances": [
[
0.016666666666666607,
0.08289583333333339
]
],
"amplitudes": [
[
0.0913705825805664,
0.24785305559635162
],
[
0.0534886009991169,
0.18498311936855316
]
]
}
Let's try it.
Does not work too well, let's try to record some more double snaps to make the pattern more accurate.
Ok, new pattern:
{
"name": "double_snap",
"distances": [
[
0.016666666666666607,
0.0925625000000001
]
],
"amplitudes": [
[
0.0913705825805664,
0.543587863445282
],
[
0.0534886009991169,
0.18498311936855316
]
]
}
We can't do much better as the rest of the double snaps have a single peak.
Let me try another recording of the double snap.
Well, it does not work too well. Btw, I had to change in identifier, the threshold to detect peaks, the resampling frequency and peak height:
min_peak_height=0.05, new_freq=0.007
The main issue is that the double snap is too quick, not much time to distinguish the 2 peaks sometimes.
Let's record a double clap. There will be more distance between peaks.
{
"name": "double_clap",
"distances": [
[
0.25325,
0.2848125
]
],
"amplitudes": [
[
0.05565951392054558,
2.0927278995513916
],
[
0.6075712442398071,
1.819767951965332
]
]
}
Funny actually, the distance is quite consistent.
Let's try it.
I'm sure we could improve it by adding more samples to the set, but let's try to mix clap and double snap.
Cool, it worked.
Let's try a more complex one: tatatataaatatata
Let's try.
Damn it, let's try again.
I had to add some more data to the pattern. It does not work too well. This one was lucky. Let's try some more to get a good sense of it.
This one is more representative.
Alright, pausing this project. That was interesting. The next step is to be able to understand why the pattern did not work on a sequence. There is also some issue with peak detection.
Code is here: github.com/ewoij/clap
Continued in part 2.