Table of content
Overview
The Maestro sync example demonstrates the use of synchronous pipelines (Tx and Rx in this case) processing in a single streamer task on the ARM cortex core utilizing the Maestro Audio Framework library.
The application is controlled by commands from a shell interface using serial console.
The feature is useful for testing the latency of the pipeline or implementing algorithms requiring reference signals such as echo cancellation. The VoiceSeeker library available in this example is not featuring AEC (Acoustic Echo Cancellation), but NXP is offering it in the premium version of the library. More information about the premium version can be found at VoiceSeeker page. The demo uses two pipelines running synchronously in a single streamer task:
- Playback (Tx) pipeline:
- Playback of audio data in PCM format stored in flash memory to the audio Line-Out connector (speaker).
- Recording (Rx) pipline:
- Record audio data using a microphone.
- VoiceSeeker processing.
- Wake words + voice commands recognition.
- Save the VoiceSeeker output to the voiceseeker_output.pcm file on the SD card.
As shown in the table below, the application is supported on several development boards, and each development board may have certain limitations, some development boards may also require hardware modifications or allow to use of an audio expansion board. Therefore, please check the supported features and Hardware modifications or Example configuration sections before running the demo.
Mode | Standard |
Feature | Audio inputs [num of channels] | Audio outputs [num of channels] | Additional libraries |
On board codec | DMICs | aud-exp-42448 | On board codec | aud-exp-42448 | VIT | VoiceSeeker |
EVKC-MIMXRT1060 | 1 | X | X | 1 | X | OK | OK |
EVKB-MIMXRT1170 | X | 1-2 | X | 1-2 | X | OK | OK |
LPCXpresso55s69 | X | X | X | X | X | X | X |
EVK-MCXN5XX | X | X | X | X | X | X | X |
RW612BGA | X | X | X | X | X | X | X |
RW612QFN | X | X | X | X | X | X | X |
Limitations:
- Addition labraries
- VIT:
- The VIT is supported only in the MCUXpresso IDE.
- VoiceSeeker:
- The VoiceSeeker is supported only in the MCUXpresso IDE.
Known issues:
More information about supported features can be found on the Supported features page.
Hardware requirements
- Desired development board
- Micro USB cable
- Speaker with 3.5 mm stereo jack
- Personal computer
- Optional:
Hardware modifications
Some development boards need some hardware modifications to run the application. If the development board is not listed here, its default setting is required.
- EVKC-MIMXRT1060:
- Please make sure resistors below are removed to be able to use SD-Card.
- R368, R347, R349, R365, R363
- Please Make sure J99 is installed.
Preparation
- Connect a micro USB cable between the PC host and the debug USB port on the development board
- Open a serial terminal with the following settings:
- 115200 baud rate
- 8 data bits
- No parity
- One stop bit
- No flow control
- Download the program to the target board.
- Insert the speaker into the Line-Out connector (headphone jack) on the development board.
- Optional: Insert an SD card into the SD card slot to record to the VoiceSeeker output.
- Either press the reset button on your development board or launch the debugger in your IDE to begin running the demo.
Running the demo
When the example runs successfully, you should see similar output on the serial terminal as below:
*****************************
Maestro audio sync demo start
*****************************
Copyright 2022 NXP
[APP_SDCARD_Task] start
[APP_Shell_Task] start
>> [APP_SDCARD_Task] SD card drive mounted
Type help
to see the command list. Similar description will be displayed on serial console:
>> help
"help": List all the registered commands
"exit": Exit program
"version": Display component versions
"start [nosdcard]": Starts a streamer task.
- Initializes the streamer with the Memory->Speaker pipeline and with
the Microphone->VoiceSeeker->VIT->SDcard pipeline.
- Runs repeatedly until stop command.
nosdcard - Doesn't use SD card to store data.
"stop": Stops a running streamer:
"debug [on|off]": Starts / stops debugging.
- Starts / stops saving VoiceSeeker input data (reference and microphone data)
to SDRAM.
- After the stop command, this data is transferred to the SD card.
Details of commands can be found here.
Example configuration
The example can be configured by user. Before configuration, please check the table to see if the feature is supported on the development board.
- Enable the premium version of VoiceSeeker:
- The premium version of the VoiceSeeker library with AEC is API compatible with this example.
- To get the premium version, please visit VoiceSeeker page.
- The following steps are required to run this example with the VoiceSeeker&AEC library.
- Link the
voiceseeker.a
library instead of voiceseeker_no_aec.a
.
- Set the
RDSP_ENABLE_AEC
definition to 1U
in the voiceseeker.h
file
- VIT model generation:
- For custom VIT model generation (defining own wake words and voice commands) please use https://vit.nxp.com/
Functionality
The start <nosdcard>
command calls the STREAMER_Create
function from the app_streamer.c
file that creates pipelines with the following elements:
- Playback pipeline:
- ELEMENT_MEM_SRC_INDEX
- ELEMENT_SPEAKER_INDEX
- Record pipeline:
- ELEMENT_MICROPHONE_INDEX
- ELEMENT_VOICESEEKER_INDEX
- ELEMENT_VIT_PROC_INDEX
- ELEMENT_FILE_SINK_INDEX (If the
nosdcard
argument is not used)
Processing itself can be started with the STREAMER_Start
function.
Each of the elements has several properties that can be accessed using the streamer_get_property
or streamer_set_property
function. These properties allow a user to change the values of the appropriate elements. The list of properties can be found in streamer_element_properties.h
. See the example of setting property value in the following piece of code from the app_streamer.c
file:
buf.
location = (int8_t *)TESTAUDIO_DATA;
buf.
size = TESTAUDIO_LEN;
prop.
prop = PROP_MEMSRC_SET_BUFF;
prop.
val = (uintptr_t)&buf;
{
return kStatus_Fail;
}
prop.
prop = PROP_MEMSRC_SET_MEM_TYPE;
{
return kStatus_Fail;
}
prop.
prop = PROP_MEMSRC_SET_SAMPLE_RATE;
prop.
val = DEMO_SAMPLE_RATE;
{
return kStatus_Fail;
}
@ STREAM_OK
General success.
Definition streamer_api.h:150
#define AUDIO_DATA
Audio data.
Definition streamer_api.h:127
int32_t streamer_set_property(STREAMER_T *streamer, int8_t pipeline_id, ELEMENT_PROPERTY_T prop, bool block)
Set element property.
Definition streamer.c:785
Element Property structure.
Definition streamer_api.h:570
uint16_t prop
property type
Definition streamer_api.h:571
uintptr_t val
property value
Definition streamer_api.h:572
Definition streamer_element_properties.h:182
uint32_t size
Buffer size.
Definition streamer_element_properties.h:184
int8_t * location
Buffer pointer.
Definition streamer_element_properties.h:183
Some of the predefined values can be found in the streamer_api.h
.
States
The application can be in 2 different states:
Commands in detail
Legend for diagrams:
flowchart TD
classDef function fill:#c6d22c
classDef condition fill:#7cb2de
classDef state fill:#fcb415
classDef error fill:#FF999C
A((State)):::state
B{Condition}:::condition
C[Error message]:::error
D[Process function]:::function
help, version
flowchart TD
classDef function fill:#c6d22c
classDef condition fill:#7cb2de
classDef state fill:#fcb415
classDef error fill:#FF999C
A((Idle)):::state --> C[Write help or version]:::function
B((Running)):::state --> C
C --> E((No state
change)):::state
start [nosdcard]
flowchart TD
classDef function fill:#c6d22c
classDef condition fill:#7cb2de
classDef state fill:#fcb415
classDef error fill:#FF999C
A((Idle)):::state --> B{nosdcard
parameter?}:::condition
B -- Yes --> CH[Playing to Line-out and
recording]:::function
CH --> L((Running)):::state
B -- No --> C{Is SD card
inserted?}:::condition
C -- Yes --> E[Playing to Line-out and
recording to SD card]:::function
E --> F((Running)):::state
F --> G{Debugging
is enabled?}:::condition
G -- No --> F
G -- Yes --> H[Save reference and
microphone data to SDRAM]:::function
H --> F
C -- No --> D[Error: Insert SD
card first]:::error
D --> A
J((Running)):::state --> K[Error: The streamer task is
already running]:::error
K --> J
stop
flowchart TD
classDef function fill:#c6d22c
classDef condition fill:#7cb2de
classDef state fill:#fcb415
classDef error fill:#FF999C
A((Idle)):::state --> A
B((Running)):::state --> C{Is debugging
enabled?}:::condition
C --Yes -->E[Copy reference and
microphone data to
the SD card]:::function
E --> G((Idle)):::state
C -- No --> G
debug [on|off]
flowchart TD
classDef function fill:#c6d22c
classDef condition fill:#7cb2de
classDef state fill:#fcb415
classDef error fill:#FF999C
A((Idle)):::state --> B[Error: First, start
the streamer task]:::error
C((Running)):::state --> D{Any
parameter?}:::condition
D -- Yes --> F{Started with
nosdcard
parameter?}:::condition
F -- No --> H[Set debugging]:::function
H --> C
F --Yes --> G[Error: Debugging cannot be used]:::error
G --> C
D -- No --> E[Error: Use the parameter
either on or off]:::error
E --> C