## Signal arrangement plugin types

mandolini_
Posts: 9
Joined: Fri Feb 14, 2020 1:50 am

### Signal arrangement plugin types

Hi -

I'm having difficulties understanding how mhachain and overlapadd work, and what their differences are. From my understanding after reading the plugins document (http://www.openmha.org/docs/openMHA_plugins.pdf), mhachain is useful for creating a chain of plugins to create signal flow (such as in example 11, short time Fourier transform: wave2spec -> identity -> spec2wave), and overlapadd takes care of the signal transformation of that example, for example.

Why, then, do I see overlapadd being used as a subparser/config parameter of mhachain (in config files such as example 14: example_dc_simple_live.cfg), and vice versa? I see examples of peculiarities like this scattered throughout the examples and it is difficult to decipher. What is the difference between these?

Code: Select all

# fshift_live.cfg
mha.plugin_name = mhachain
mha.mhachain.algos = [fshift_hilbert gain]

# example_dc_simple_live.cfg
mhalib = transducers

# example-wave2spec-spec2wave.cfg
mhalib = mhachain
mha.algos = [wave2spec identity spec2wave]
It seems they are used nearly interchangeably, if not arbitrarily. When do I know which one to use? I see in the documentation that overlapadd is not realtime-safe, yet it is used in the dc_simple_live example using Jack as the realtime audio engine.

On a related note, I am getting the following error when using a sequence of plugins with mhachain.

Code: Select all

>> mha_set(openmha, 'cmd', 'start')
Error using mhactl_wrapper>mhactl_wrapper_2 (line 55)
An error occured while sending to MHA server:
[1: 'cmd=start'] (mhafw_lib) The processing library returned invalid fragment size.

Error in mhactl_wrapper (line 33)
r = interface( handle, query );

Error in mha_set (line 15)
mhactl_wrapper(handle,assignments);
I can paste my configuration file here if it is useful for resolving my issue.

Thanks!

mandolini_
Posts: 9
Joined: Fri Feb 14, 2020 1:50 am

### Re: Signal arrangement plugin types

As an edit to the above:
I'm currently using mhachain to perform a simple passthrough as a sanity check with the following sequence of plugins and parameters.

Code: Select all

srate = 48000
fragsize = 256
iolib = MHAIOJack
mhalib = mhachain
mha.algos = [downsample wave2spec identity spec2wave upsample] # eventually want identity running at 16kHz
mha.downsample.ratio = 1 # dummy value for now
Curiously, the chain of plugins works just fine without the resampling plugins. But of course, this would mean that the remaining inner 3 plugins are operating at 48kHz, instead of 16kHz like I am hoping to achieve.

tobiasherzke
Posts: 21
Joined: Mon Jun 24, 2019 12:51 pm

### plugin overlapadd vs plugin combination wave2spec + spec2wave

You are correct. Plugin overlapadd behaves, in most cases, exactly as the combination of plugins wave2spec and spec2wave, with the difference that overlapadd needs to actively load another plugin which processes the STFT signal, while with the wave2spec + spec2wave combination you can place the STFT processing plugins between these two transformation plugins in the same chain. When using overlapadd, you need to load the plugin that processes the STFT signal inside the overlapadd plugin and then refer to it as overlapadd.some_name in the configuration.

Initially we thought that we can replace wave2spec and spec2wave with just 1 plugin, which is why overlapadd was introduced, and why it behaves the same as the combination. Later we found some use cases where wave2spec and spec2wave are still useful: Mostly, when you only need one of the transformations and do not want to pay for the inverse. Additionally, this fact about overlapadd is mostly what you want, but maybe not always: From the caption of the second figure in the overlapadd section of the plugin manual (at the time of this writing, Figure 10 in the document):
In this setup, $$W^α$$ is applied before FFT and $$W^{1−α}$$ is used for post-windowing.
If you do not want this coupling for non-zeropadding WOLA setups, then you need to use wave2spec+spec2wave.

These differences will not be relevant for most use cases. The most visible difference is what you have noticed, the additional hierarchy level in the configuration files that is required when using overlapadd. Apart from this, think of them of being the same unless you see some comments explaining why one is used and not the other. I do not think that we have examples of such setups currently in the published examples.

tobiasherzke
Posts: 21
Joined: Mon Jun 24, 2019 12:51 pm

### (mhafw_lib) The processing library returned invalid fragment size.

(mhafw_lib) The processing library returned invalid fragment size.

This error occurs when your signal processing setup produces as output signal a different number of samples per audio channel than what it received. You can produce such a situation e.g. if you upsample the signal, but then do not downsample again, or downsample with a different factor.

tobiasherzke
Posts: 21
Joined: Mon Jun 24, 2019 12:51 pm

overlapadd is real-time safe. If you have found documentation saying different, please point us to that documentation so that we can investigate.

mandolini_
Posts: 9
Joined: Fri Feb 14, 2020 1:50 am

### Re: Signal arrangement plugin types

I was mistakenly looking at the resampling plugin - not overlapadd - when mentioning that it was not real-time safe. On that note, however, it looks like example 9 (localizer steering beamformer) uses resampling in real-time, using Jack. Is this enough to assume that it can be used in real-time applications safely? I tried replacing the downsample/upsample combo in my algos chain with resampling, and I no longer got the 'invalid fragment size' error and can hear the audio in real-time just fine.

tobiasherzke
Posts: 21
Joined: Mon Jun 24, 2019 12:51 pm

### Re: Signal arrangement plugin types

Right, the documentation for plugin resampling contains this warning:
A synchronous resampling ringbuffer such as this causes varying computational loads in the outer processing buffer. It is therefore not real-time safe.

An example to illustrate this point:

Code: Select all

srate=44100
fragsize=64
iolib=MHAIOJack
io.con_in=[system:capture_1]
io.con_out=[system:playback_1]
mhalib=resampling
mha.srate=16000
mha.fragsize=23
mha.plugin_name=gain
mha.gain.gains=-6
cmd=start

This configuration needs a Jack server running at 44.1kHz with block size 64 samples per channel. One block of audio from the sound card has a duration approximately 1.45 milliseconds. The resampling plugin resamples the audio to sampling rate 16kHz and provides a blocks of 23 samples per channel to the loaded processing plugin, here gain. One block of resampled audio as seen by the gain plugin has therefore a duration of approximately 1.44 milliseconds.

Most of the time, whenever resampling processes 1 block of audio at 44.1 kHz from the sound card, it will re-sample this to 1 block of audio at 16kHz and call gain to process that one block. However, because the absolute duration of the audio blocks at 16kHz is slightly shorter than at 44.1kHz in this example, once in approximately every 100 times, it will need to have two audio blocks at 16kHz processed to catch up, and it will call the signal processing method of the plugin gain twice.

2. "It is therefore not real-time safe"

One best practice when doing real-time audio processing is to design your signal processing algorithms in a way that the computational effort required to process one block of audio data is constant. E.g. the computational effort should not depend on the signal's content or on the current time of day. This best practice is violated by the resampling plugin as described above.

Nevertheless, we still tend to use the resampling plugin in our real-time configurations. You ask

3. Preparing openMHA configurations for real-time processing usage
Is this enough to assume that it can be used in real-time applications safely?
A real-time safe implementation of a plugin does not mean that you can always use it in real-time applications. It could still need more CPU for signal processing than your system can provide reliably. Similarly, usage of a plugin that violates one or more best practices of real-time signal processing algorithm design can still be possible in a real-time setup when you have enough CPU headroom.

The key to trusting your configuration to run in real-time lies in testing. Run your signal processing setup for extended periods of time in realistic conditions and watch for dropouts. qjackctl will display the dropouts that it detects, but whether you use Jack or not, you should also listen to your setup and watch out for dropouts subjectively.

Having all your signal processing algorithms obey real-time best practices will make this test easier and more trustworthy, not eliminate it. Behavior of resampling is still relatively easy to test: The variations in computational load caused by resampling jump back and forth between two possible conditions. This is better than e.g. writing to a file or printing an analysis to the terminal every 1000th block, in these conditions the computational load would jump between constant and unpredictable. Having resampling in your real-time configuration will increase the ammount of testing that you will need to do.