Processing Time

Table of content

The individual time measurements were conducted using a logic analyzer by monitoring changes in the GPIO port levels on the EVKC-MIMXRT1060 development board. These measurements were executed for each individual pipeline run, capturing the timing at each corresponding element, and, when relevant, the interconnections between these elements.

Maestro playback example

For the Maestro playback example the following reference audio file was used: test_48khz_16bit_2ch.wav. In this example, the pipeline depicted in the diagram was considered. Media codecs WAV and MP3 were taken into account. To compare the times spent on the SSRC block, sampling rates for both codecs were selected: 44.1 kHz and 48 kHz.

The measurement of streamer pipeline run started at the beginning of streamer_process_pipelines(): streamer.c and ended in the function streamer_pcm_write(): streamer_pcm.c just before the output buffer.

In the scenario involving the WAV codec, the audio file was accessed in every iteration of the streamer pipeline. Meaning, during each run, the file was read directly from the SD card. However, in the case of the MP3 codec, where data processing necessitates complete MP3 frames, the file wasn’t read during every run. Rather, it was accessed periodically, triggered when the codec buffer lacked a complete MP3 frame of data. The total time spent on codec processing varies significantly depending on the type and implementation of the codec. For certain types of codecs, like FLAC, there may be multiple file accesses during a single pipeline run. The provided values are specific to the reference implementation. For details about the codecs please see see audio-voice-components documentation middleware\audio_voice\components\.

The duration of the streamer pipeline illustrates that with a sampling frequency of 48 kHz, there is no resampling occurring at the SSRC element. Consequently, the overall pipeline time is lower than in the case of 44.1 kHz audio, where resampling takes place.

To enhance comprehension of the system’s behavior, histograms of the pipeline run times and its elements are included. The greater time variance with the MP3 codec is precisely due to the absence of file reads in every run. In clusters with shorter times, there are no file accesses, while in clusters with longer times, file reads occur. This indicates that the majority of runs do not involve file access.

WAV 48 kHz

WAV 44 kHz

MP3 48 kHz file read

MP3 48 kHz w/o file read

MP3 44 kHz file read

MP3 44 kHz w/o file read

mean

1.11 ms

1.76 ms

2.87 ms

0.51 ms

3.22 ms

0.89 ms

min

1.03 ms

1.60 ms

2.74 ms

0.41 ms

2.33 ms

0.74 ms

max

1.29 ms

2.23 ms

3.24 ms

1.83 ms

3.73 ms

1.12 ms


Time on each element

In the tables and histograms below, the timings for individual elements and their connections are provided. Given that the file reading function was invoked during the codec’s operation, the tables for individual elements display the total time on the codec element, the time on the codec element before the file read, and the time on the codec element after the file read. The individual blocks in the tables are as follows:

  • streamer - total time of one pipeline run without time on output buffers

  • codec start - time on decoder before file read

  • codec end - time on decoder after file read

  • codec total - codec_start+codec_end

  • file_src - file reading time

  • SSRC_proc - time on SSRC element

  • audio_sink - time on audio sink without ouput buffers

  • pcm_write - time on output buffers

  • link - time on element links

The start times of the time intervals for individual blocks and their respective links were measured by altering the GPIO pin level in the following functions:

  • streamer - streamer_process_pipelines():streamer.c

  • codec - decoder_sink_pad_process_handler():decoder_pads.c

  • file_src - filesrc_read():file_src_rtos.c

  • SSRC_proc - SSRC_Proc_Execute():ssrc_proc.c

  • audio_sink - audiosink_sink_pad_chain_handler():audio_sink.c

  • pcm_write - streamer_pcm_write():streamer_pcm.c

  • link - pad_push():pad.c


WAV 48kHz

streamer

codec total

codec start

file_src

codec end

link codec-SSRC

SSRC_proc

link SSRC-audio_sink

audio_sink

pcm_write

mean

1.119 ms

152 μs

31 μs

0.843 ms

120 μs

5 μs

64 μs

2 μs

40 μs

20.228 ms

min

1.026 ms

125 μs

21 μs

0.773 ms

104 μs

<1 μs

47 μs

<1 μs

30 μs

19.805 ms

max

1.290 ms

193 μs

49 μs

1.311 ms

144 μs

23 μs

93 μs

14 μs

91 μs

20.324 ms

WAV 44kHz

streamer

codec total

codec start

file_src

codec end

link codec-SSRC

SSRC_proc

link SSRC-audio_sink

audio_sink

pcm_write

mean

1.765 ms

178 μs

44 μs

0.853 ms

134 μs

5 μs

671 μs

3 μs

42 μs

21.472 ms

min

1.604 ms

145 μs

33 μs

0.770 ms

112 μs

<1 μs

574 μs

<1 μs

33 μs

18.163 ms

max

2.233 ms

218 μs

57 μs

1.335 ms

161 μs

18 μs

715 μs

5 μs

89 μs

21.746 ms


MP3 48 kHz w/ file read

streamer

codec total

codec start

file_src

codec end

link codec-SSRC

SSRC_proc

link SSRC-audio_sink

audio_sink

pcm_write

mean

2.871 ms

441 μs

279 μs

2.271 ms

162 μs

6 μs

56 μs

3 μs

50 μs

11.019 ms

min

2.739 ms

353 μs

74 μs

1.353 ms

26 μs

<1 μs

40 μs

<1 μs

34 μs

10.091 ms

max

3.244 ms

570 μs

409 μs

2.728 ms

467 μs

18 μs

80 μs

14 μs

62 μs

12.910 ms

MP3 48 kHz w/o file read

streamer

codec total

codec start

file_src

codec end

link codec-SSRC

SSRC_proc

link SSRC-audio_sink

audio_sink

pcm_write

mean

0.508 ms

403 μs

x

x

x

8 μs

39 μs

3 μs

36 μs

11.326 ms

min

0.407 ms

208 μs

x

x

x

<1 μs

25 μs

<1 μs

21 μs

7.715 ms

max

1.834 ms

563 μs

x

x

x

41 μs

69 μs

16 μs

104 μs

12.941 ms


MP3 44 kHz w/ file read

streamer

codec total

codec start

file_src

codec end

link codec-SSRC

SSRC_proc

link SSRC-audio_sink

audio_sink

pcm_write

mean

3.217 ms

436 μs

367 μs

2.300 ms

66 μs

7 μs

403 μs

3 μs

51 μs

12.188 ms

min

2.329 ms

383 μs

73 μs

1.411 ms

26 μs

2 μs

318 μs

<1 μs

35 μs

9.119 ms

max

3.726 ms

547 μs

464 μs

2.801 ms

441 μs

27 μs

454 μs

12 μs

65 μs

12.529 ms

MP3 44 kHz w/o file read

streamer

codec total

codec start

file_src

codec end

link codec-SSRC

SSRC_proc

link SSRC-audio_sink

audio_sink

pcm_write

mean

0.891 ms

437 μs

x

x

x

9 μs

388 μs

3 μs

38 μs

11.934 ms

min

0.738 ms

268 μs

x

x

x

<1 μs

290 μs

<1 μs

22 μs

8.964 ms

max

1.115 ms

620 μs

x

x

x

45 μs

438 μs

17 μs

92 μs

12.624 ms

Maestro record example

Typical execution times of the streamer pipeline and its individual elements for the EVKC-MIMXRT1060 development board are detailed in the following tables. The duration spent on output buffers and reading from the microphone is excluded from traversal measurements. Three measured pipelines are depicted in the figure below. The first involves a loopback from microphone to speaker, supporting both mono and stereo configurations. The second pipeline is a mono voice control setup, comprising microphone and VIT blocks. The final pipeline is a stereo voice control setup, integrating microphone, voice seeker, and VIT blocks. The measurement of streamer pipeline run started at the beginning of streamer_process_pipelines():streamer.c and ended in the function streamer_pcm_write(): streamer_pcm.c just before the output buffer.


The individual blocks in the tables are as follows:

  • streamer - total time of one pipeline run without time on output buffers and without time reading from the microphone

  • audio_src_start - time on audio src before reading from the microphone

  • audio_src_end - time on audio src after reading from the microphone

  • pcm_read - reading from the microphone

  • voiceseeker - time on voice seeker element

  • vit - time on VIT element

  • audio_sink - time on audio sink without ouput buffers

  • pcm_write - time on output buffers

  • link - time on element links

The start times of the time intervals for individual blocks and their respective links were measured by altering the GPIO pin level in the following functions:

  • streamer - streamer_process_pipelines():streamer.c

  • audio_src - audiosrc_src_process():audio_src.c

  • pcm_read - streamer_pcm_read():streamer_pcm.c

  • voiceseeker - audio_proc_sink_pad_chain_handler():audio_proc.c

  • vit - vitsink_sink_pad_chain_handler():vit_sink.c

  • audio_sink - audiosink_sink_pad_chain_handler():audio_sink.c

  • pcm_write - streamer_pcm_write():streamer_pcm.c

  • link - pad_push():pad.c

Pipeline Microphone -> Speaker

microphone -> speaker mono

streamer

audio_src_start

pcm_read

audio_src_end

link audio_src-audio_sink

audio_sink

pcm_write

mean

43 μs

3 μs

29.938 ms

29 μs

<1 μs

10 μs

18 μs

min

26 μs

<1 μs

29.350 ms

19 μs

<1 μs

5 μs

12 μs

max

72 μs

12 μs

29.957 ms

44 μs

1 μs

15 μs

25 μs

microphone -> speaker stereo

streamer

audio_src_start

pcm_read

audio_src_end

link audio_src-audio_sink

audio_sink

pcm_write

mean

115 μs

5 μs

29.861 ms

54 μs

2 μs

55 μs

23 μs

min

94 μs

<1 μs

29.768 ms

43 μs

<1 μs

50 μs

12 μs

max

154 μs

14 μs

29.880 ms

67 μs

8 μs

65 μs

49 μs

Pipeline Microphone -> VIT

microphone -> VIT

streamer

audio_src_start

pcm_read

audio_src_end

link audio_src-vit

vit

mean

7.380 ms

30 μs

22.624 ms

78 μs

2 μs

7.261 ms

min

2.641 ms

10 μs

2.2265 ms

58 μs

<1 μs

2.559 ms

max

7.780 ms

42 μs

2.7341 ms

94 μs

5 μs

7.624 ms

Pipeline Microphone -> Voice seeker -> VIT

microphone -> voice seeker -> VIT

streamer

audio_src_start

pcm_read

audio_src_end

link audio_src-voiceseeker

voiceseeker

link voiceseeker-vit

vit

mean

9.916 ms

22 μs

20.084 ms

84 μs

4 μs

2.386 ms

13 μs

7.407 ms

min

4.983 ms

19 μs

19.738 ms

72 μs

<1 μs

2.228 ms

2 μs

2.662 ms

max

10.423 ms

34 μs

24.777 ms

100 μs

7 μs

2.522 ms

31 μs

7.729 ms