This is a very interesting project.
My expertise is in FPGA development.
I am interested in FPGA's being used in hearing aids.
Are you able to do everything that you need to do on a CPU?
Is there any computationally intensive task which would benefit from adding in an FPGA?
I wonder if bio mimicry on an FPGA might help in this process.
Warm Regards
Christopher Lozinski
FPGA Coprocessor
Re: FPGA Coprocessor
Dear Christopher,
one thing in advance: I am not an OpenMHA (co)developer, but lead a development in a related direction, see https://www.youtube.com/watch?v=J8qHL-vFvSM.
We have already considered using a coprocessor to outsource particularly computationally intensive tasks. Such a component would ideally be very small (4-9 mm2), consume very little power (<1 mA), and have few connectors.
I wonder how the data (e.g. [uC] -> audio samples -> [CoProc] -> complex spectrum -> [uC]) could be communicated efficiently and with low latency if no direct access to memory is possible, as is the case with many SoCs today. Would e.g. a fast and energy-efficient transmission via SPI be feasible?
May there be related products already available on the market that I am not aware of?
I would be happy to hear you thoughts on this.
Best regards,
Tamas
one thing in advance: I am not an OpenMHA (co)developer, but lead a development in a related direction, see https://www.youtube.com/watch?v=J8qHL-vFvSM.
We have already considered using a coprocessor to outsource particularly computationally intensive tasks. Such a component would ideally be very small (4-9 mm2), consume very little power (<1 mA), and have few connectors.
I wonder how the data (e.g. [uC] -> audio samples -> [CoProc] -> complex spectrum -> [uC]) could be communicated efficiently and with low latency if no direct access to memory is possible, as is the case with many SoCs today. Would e.g. a fast and energy-efficient transmission via SPI be feasible?
May there be related products already available on the market that I am not aware of?
I would be happy to hear you thoughts on this.
Best regards,
Tamas
-
PythonLinks
- Posts: 5
- Joined: Tue Aug 08, 2023 7:38 pm
Re: FPGA Coprocessor
What a great project.
SPI should be fine.
From Wikipedia:
"It is possible to find SPI adapters on the market today that support up to 100 MHz serial interfaces, with virtually unlimited access length. " And if you keep your circuit traces short, you should definitely be fine.
In any case audio is usually sampled at 44Khz, and these FPGAs circuits run in the 30-80 Mhz range, so you have a 1000 clock cycles to do something.
Many of the SOC's have an Axi interface running to the FPGA. That is usually 32 bits wide. Some FPGAs will have ADC and DAC, saving space.
So solution #1 is an SOC with a built in Axi interface.
But they are often quite expensive.
Solution #2 is a soft core microprocessor on the FPGA.
Here is the smallest soft core I know of which runs C on an FPGA.
https://en.wikipedia.org/wiki/ZPU_(processor)
Great idea, I will add it into the work I am doing!
I suspect that the SOC's get quite expensive, so an FPGA with a soft cpu core is cheaper, and thus really good for encouraging lots of users to hack on it.
> Such a component would ideally be very small (4-9 mm2), consume very little power (<1 mA), and have few connectors.
I run on the Lattice ICE40UltraPlus. Here is the packaging.
> 30-ball WLCSP (2.15 x 2.55 mm)
You can leave most balls disconnected.
And the data sheet.
https://www.latticesemi.com/en/Products ... 3118BDC07F
The other thing you should probably do is input the raw data directly into the FPGA. Have the FPGA do the band pass filtering, run an FFT on it, and report the data, or maybe even do the frequency shifting, and inverse FFT. Then pass that to the microprocessor, or to the soft core.
I plan on doing FFT on my many core FPGA.
You can read more about my plans.
https://forth.pythonlinks.info/
The FFT software already exists for a single core, I just have to parallelize it to be faster. Does not look that hard, and is most needed.
We actually have a whole discussion group about the FFT.
here is a discord invite.
https://discord.gg/ZQrbySTX
By the way, my plan was to just run 4 cables from a circuit board in a backpack. Two lavalier microphones, and two earbuds! I would much prefer to use your device.
I am happy to voice chat.
Christopher Lozinski
lozinski@PythonLinks.info
@PythonLinks@Mastodon.Social
SPI should be fine.
From Wikipedia:
"It is possible to find SPI adapters on the market today that support up to 100 MHz serial interfaces, with virtually unlimited access length. " And if you keep your circuit traces short, you should definitely be fine.
In any case audio is usually sampled at 44Khz, and these FPGAs circuits run in the 30-80 Mhz range, so you have a 1000 clock cycles to do something.
Many of the SOC's have an Axi interface running to the FPGA. That is usually 32 bits wide. Some FPGAs will have ADC and DAC, saving space.
So solution #1 is an SOC with a built in Axi interface.
But they are often quite expensive.
Solution #2 is a soft core microprocessor on the FPGA.
Here is the smallest soft core I know of which runs C on an FPGA.
https://en.wikipedia.org/wiki/ZPU_(processor)
Great idea, I will add it into the work I am doing!
I suspect that the SOC's get quite expensive, so an FPGA with a soft cpu core is cheaper, and thus really good for encouraging lots of users to hack on it.
> Such a component would ideally be very small (4-9 mm2), consume very little power (<1 mA), and have few connectors.
I run on the Lattice ICE40UltraPlus. Here is the packaging.
> 30-ball WLCSP (2.15 x 2.55 mm)
You can leave most balls disconnected.
And the data sheet.
https://www.latticesemi.com/en/Products ... 3118BDC07F
The other thing you should probably do is input the raw data directly into the FPGA. Have the FPGA do the band pass filtering, run an FFT on it, and report the data, or maybe even do the frequency shifting, and inverse FFT. Then pass that to the microprocessor, or to the soft core.
I plan on doing FFT on my many core FPGA.
You can read more about my plans.
https://forth.pythonlinks.info/
The FFT software already exists for a single core, I just have to parallelize it to be faster. Does not look that hard, and is most needed.
We actually have a whole discussion group about the FFT.
here is a discord invite.
https://discord.gg/ZQrbySTX
By the way, my plan was to just run 4 cables from a circuit board in a backpack. Two lavalier microphones, and two earbuds! I would much prefer to use your device.
I am happy to voice chat.
Christopher Lozinski
lozinski@PythonLinks.info
@PythonLinks@Mastodon.Social
-
PythonLinks
- Posts: 5
- Joined: Tue Aug 08, 2023 7:38 pm
Re: FPGA Coprocessor
Here is the chip family you want.
https://www.quicklogic.com/products/soc/
It has both an FPGA and an ARM processor on the same chip.
Better yet, they communicate through shared memory.
The only thing is do not use the Pulse density Microphone. I cannot do an FFT on the data. Use a Pulse Code modulation Microphone. Probably over I2S,
And there may be an issue with memory access latency across the bus.
In any case there will be more such chips. Before you choose a particular chip, please ask me what I think of it. My expertise in this area is growing daily.
And in my talks, I now show your images, and mention your group.
And when you get to it, I would love to know what algorithms you want to run on the FPGA. My focus is on FFT, Frequency shifting, Inverse FFT, and cross correlation for sound localization and removal. What else is needed?
I hope hat helps.
Christopher Lozinski
https://www.quicklogic.com/products/soc/
It has both an FPGA and an ARM processor on the same chip.
Better yet, they communicate through shared memory.
The only thing is do not use the Pulse density Microphone. I cannot do an FFT on the data. Use a Pulse Code modulation Microphone. Probably over I2S,
And there may be an issue with memory access latency across the bus.
In any case there will be more such chips. Before you choose a particular chip, please ask me what I think of it. My expertise in this area is growing daily.
And in my talks, I now show your images, and mention your group.
And when you get to it, I would love to know what algorithms you want to run on the FPGA. My focus is on FFT, Frequency shifting, Inverse FFT, and cross correlation for sound localization and removal. What else is needed?
I hope hat helps.
Christopher Lozinski