Continuing from Identifying clap patterns.

Alright, let's keep going. Time to understand why the patterns don't match.

This one probably failed because we have a FP peak.

Let's reduce the base threshold to 0.03.

We got it, but we now miss 2 others that we used to have because their peaks are missing now.

Got it working by setting the threshold to 0.036. I'm obviously not very happy with it, it feels a little like bricolage.

Alright, now, let's have a look at why we are missing some double snaps.

The amplitude of the second snap is just below the expected amplitude.

Another one is the distance that was too fast.

Let's have a look at some double claps.

For double claps, the timing is off.

Overall, we also have this issue with FP peaks.

I think most issues can be fixed by adding examples in the dataset. It's not too bad. I'm not happy with the peak detection, but I will leave it as is for now and build the stream processing and output in the console the pattern that was detected.

Stream processing

I think there are different ways to go about it. For example, we could detect individual peaks and then process those with a window, but I'm going to go for something simpler even if less performant. It will also be easier to debug, we could store the signal and replay the algo. Arguably, we could also do that with peaks, but probably a bit messier.

I'm going to use a moving window over the raw signal, and do what we did so far, detect peaks and patterns.

We need to think of a few things:

window size
processing frequency
deduplication
overlapping patterns

First, we know the window size has to be larger than the widest pattern + some padding on each side to avoid detecting subpatterns. We don't want a peak before, nor after below a second.

Let's call this window Wp for pattern window.

Now, let's imagine we have a pattern in time:

In orange, we have our pattern window (Wp). Since we're going to process windows at a specific frequency, for example, every 0.1 s or 1 s, … we can't guarantee the Wp window will contain the pattern. It could start a little bit before, truncating the end and so on.

This is exactly what we don't want and can't control, because it would require us to know where the pattern is.

We want the full window (orange) over the pattern to be processed to detect it. We know the algo runs every F seconds. If we are lucky, it will run exactly at S and then we're fine, but what if it starts earlier? Then Wp won't be large enough to capture the whole pattern. We are guaranteed the processing will run between S-F and S because it runs every F. Therefore, if we extend the window to Wp + F, we are guaranteed to have every pattern covered.

Let's name it: W = Wp + F

Those windows will overlap, which means we'll have to process the same things multiple times. Let's try to get a sense of how much that will be.

First of all, an interesting observation is that wherever we place our pattern, a window will fully process it. Exactly what we wanted.

Regarding the reprocessing, we can see that it is twice in our scenario. We reprocess the same span at most twice. But it depends on the ratio between Wp / F, the larger, the more duplicated processing we will do.

In this scenario where Wp = 3F, we'll reprocess the same span 4 times.

Let's try to come up with a formula to compute it. We basically need to know in a window W, how many times we start a processing. That should give us the number of overlapping ones.

R (number of reprocessings) = W / F = (F + Wp) / F = 1 + Wp / F = ceil(Wp / F) + 1

So, the minimum is 2 and the smaller F becomes, the more overlapping windows we'll get.

Let's take some real numbers to see if that would make sense.

Wp = 3 s
F = 0.1 s because I don't want to wait too long before detecting a pattern

R = 3 / 0.1 + 1 = 31 duplicates

Few things here:

It's quite a lot. Processing the stream 31 times basically. Sounds quite inefficient.
We need to test whether the processing takes less than 0.1 s (F) to process 3.1 s spans. The raw data is 48K hertz.

I would like to have this program running as a background process and therefore consume as little CPU as needed. We could try this approach and measure the CPU usage. Or we could increase F to 0.5 s which would give us an R of 7.

Let's try that actually because it should be quite easy to do and then we could compare CPU usage with a different approach. No deduplication, overlapping patterns for now.

Quick code:

def processing_loop(
    Y_raw_shared: list[np.ndarray],
    Y_raw_event: Event,
    processing_frequency: float,
    window: float,
    samplerate: int,
    patterns: list[Pattern],
):
    start_idx = 0
    Y_raw_start_idx = 0
    window_size = int(window * samplerate)

    while True:
        # wait for more data
        Y_raw_event.wait()
        Y_raw_event.clear()

        Y: np.ndarray = Y_raw_shared[0]
        start = start_idx - Y_raw_start_idx
        end = start + window_size

        if end > Y.size:
            # not enough data
            continue

        Y = Y[start:end]

        X, Y, peaks = identify_peaks(
            Y, samplerate, min_peak_height=0.05, new_freq=0.007
        )

        spans = identify_patterns(
            time=X[peaks],
            amplitudes=Y[peaks],
            window_duration=X.max(),
            patterns=patterns,
        )

        # relative to stream start
        X = X + start_idx / samplerate

        for span in spans:
            print(f"{span.pattern.name} {X[peaks[span.peak_start_idx]]} {X[peaks[span.peak_end_idx]]}")

        start_idx += int(processing_frequency * samplerate)

TBC

The other approach is to only identify peaks on the stream. Which would shorten considerably the window and therefore R. Let's see how large a peak is.

It's about 0.5 seconds, this is a massive one. I think we could still detect it with 100 ms. However, I'm not sure of how the computation of the prominence will be affected. What happens if there is no higher point?

Let's try to see what happens.

TBC

Code is here: github.com/ewoij/clap