[bootlin/training-materials updates] master: preemptrt stuff (f450f749)
Maxime Chevallier
maxime.chevallier at bootlin.com
Thu Jan 5 11:17:14 CET 2023
Repository : https://github.com/bootlin/training-materials
On branch : master
Link : https://github.com/bootlin/training-materials/commit/f450f749dc271627fb4c9bc0c7ec22298575e150
>---------------------------------------------------------------
commit f450f749dc271627fb4c9bc0c7ec22298575e150
Author: Maxime Chevallier <maxime.chevallier at bootlin.com>
Date: Thu Aug 18 10:26:53 2022 +0200
preemptrt stuff
Signed-off-by: Maxime Chevallier <maxime.chevallier at bootlin.com>
>---------------------------------------------------------------
f450f749dc271627fb4c9bc0c7ec22298575e150
.../preempt-rt-application-development.tex | 42 ++------
.../preempt-rt-benchmarking.tex | 114 +++++++++++++++++++++
.../preempt-rt-patch-download.tex | 93 +++++++++++++++--
3 files changed, 211 insertions(+), 38 deletions(-)
diff --git a/labs/preempt-rt-application-development/preempt-rt-application-development.tex b/labs/preempt-rt-application-development/preempt-rt-application-development.tex
index e818c077..dc5d2d0b 100644
--- a/labs/preempt-rt-application-development/preempt-rt-application-development.tex
+++ b/labs/preempt-rt-application-development/preempt-rt-application-development.tex
@@ -3,41 +3,21 @@
During this lab, you will:
\begin{itemize}
\item Analyse the source code of an application
- \item Find ways to improve the maximum latency
- \item Trace the program's execution using perf
+ \item Better understand the page-faulting mechanism
+ \item Use ftrace to display the program's behaviour
\end{itemize}
\section{Compile and run the program}
-In the \code{bootlin-labs-data} folder, you'll find a C program called \code{test_counter.c}
-This program's goal is to count rising edges on a GPIO input pin.
+Compile the pgflt program : \code{make}
-The program stores the timestamp for each edge, ans stores it into a buffer.
+Test the various options, and analyse how the Stack and Heap are allocated and
+pre-faulted.
-Your goal is to implement the logic to compute the maximum latency during the
-program's execution, and to try to improve it by following the best practises.
-
-Using a signal generator, send a 10KHz square signal on pin 28 on connector 9.
-
-Alternatively, apply the patch 0001-enable-spidev.patch file on your devicetree
-to have the /dev/spidev0.0 device available, and use it to send pulses onto the
-pin 28.
-
-\subsection{Program configuration}
-
-Pay attention to the initialization sequence for the program, and make sure you
-use the right memory-management practises.
-
-The program will run by default in \code{SCHED_OTHER}. Make sure that the scheduling
-algorithm used is the right one.
-
-Make sure that the timestamps are gathered using the right clock.
-
-\subsection{Benchmarking}
-
-Using \code{perf}, try to analyse the program. What can you say about the number
-of pagefaults and their occurence ?
-
-The GPIO controller on the board uses edge-triggered interrupts for event
-reporting. Make sure these interrupts have the right priority.
+Use ftrace to visualise the page-fault occurings :
+\begin{bashinput}
+ trace-cmd record -e *page_fault* ./pgflt 0x1f
+\end{bashinput}
+This is a good example where kernelshark can be useful, try to export the trace
+to your host machine for cold analysis.
diff --git a/labs/preempt-rt-benchmarking/preempt-rt-benchmarking.tex b/labs/preempt-rt-benchmarking/preempt-rt-benchmarking.tex
index d0dcd563..f5c44268 100644
--- a/labs/preempt-rt-benchmarking/preempt-rt-benchmarking.tex
+++ b/labs/preempt-rt-benchmarking/preempt-rt-benchmarking.tex
@@ -195,3 +195,117 @@ Add the \code{isolcpus=1} to the \code{append} line to isolate CPU 1.
Reboot your target, and run cyclictest on CPU1. What can you notice ?
+\subsection{Using the Tracing subsystem}
+
+In order to use Ftrace, we need to make sure it is enabled in the kernel
+configuration. Using \code{make linux-menuconfig}, go in the "Kernel hacking" section
+and enable the following options :
+
+\begin{itemize}
+ \item Generic Kernel Debugging Instruments -> Debug Filesystem
+ \item Tracers -> Kernel Function Tracer
+ \item Tracers -> Kernel Function Profiler
+ \item Tracers -> Interrupts-off Latency Tracer
+ \item Tracers -> Preemption-off Latency Tracer
+ \item Tracers -> Scheduling Latency Tracer
+ \item Tracers -> Tracer to detect hardware latencies
+\end{itemize}
+
+You can now recompile your kernel and re-generate your full image :
+
+\begin{bashinput}
+make linux-rebuild
+make
+\end{bashinput}
+
+Once you've rebooted your board with ftrace enabled, you'll need to mount the
+Tracing Filesystem, that gets automatically mounted in debugfs :
+
+\code{mount -t debugfs debugfs /sys/kernel/debug}
+\code{mount -t tracefs nodev /sys/kernel/tracing}
+
+You can now move into the tracing subsystem's main directory :
+
+\code{cd /sys/kernel/debug/tracing}
+
+You can now take a little tour of the files here :
+
+\begin{itemize}
+ \item \code{available_tracers} : Lists all tracers you can use. Each tracer
+ corresponds to what and how do you want the tracing to occur.
+ \item \code{current_tracer} : Write the tracer you want to use in this file
+ \item \code{tracing_on} : Write 1 to start tracing, 0 to stop tracing
+ \item \code{trace} : The content of the trace buffer after tracing is finished
+\end{itemize}
+
+First, try the \code{preemptirqsoff} tracer and start tracing :
+
+\begin{bashinput}
+echo preemptirqsoff > current_tracer
+echo 1 > tracing_on
+sleep 10
+echo 0 > tracing_on
+cat trace
+\end{bashinput}
+
+Give a try to other traces : \code{wakeup_rt}, \code{function}, \code{function_graph},
+using various of the stressing methods proposed above. Can you get meaningful information ?
+
+\section{Using trace-cmd and kernelshark}
+
+Trace-cmd is a wrapper over the raw silesystem-based interface of ftrace. There
+are various ways of using it, either standalone, or by recording the behaviour
+of the system during a command.
+
+\subsection{Recording}
+
+To record traces during a command, run trace-cmd with the \code{record} keyword :
+
+\begin{bashinput}
+# record a full function_graph trace for the duration of the cyclictest run
+trace-cmd record -p function_graph cyclictest -p 80 -D 10
+
+# Display the output
+trace-cmd report
+\end{bashinput}
+
+Running a recording session will automatically export the generated trace in a
+"trace.dat" file, that can be displayed with \code{trace-cmd report} or \code{kernelshark}.
+
+\subsection{Manual tracing}
+
+Another approach is to simply start an ftrace session, without relying on a process's
+execution. This is useful in combination with the \code{wakeup_rt}, \code{wakeup}, \code{preemptirqsoff} tracers.
+
+\begin{bashinput}
+ # Start a tracing session
+ trace-cmd start -p wakeup_rt
+
+ # Stop a tracing session
+ trace-cmd stop
+
+ # Display the generated trace
+ trace-cmd show
+
+ # Export and clear the trace buffer into trace.dat
+ trace-cmd extract
+
+ # Display the extraced trace
+ trace-cmd report
+\end{bashinput}
+
+This approach also allows you to use \code{cyclictest} in conjunction with \code{ftrace},
+since you can start at tracing session with \code{trace-cmd start} and have it
+stopped automatically when cyclictest detects a high latency :
+
+\begin{bashinput}
+ # Start a tracing session
+ trace-cmd start -p function_graph
+
+ # Launch cyclictest with the breaktrace option
+ cyclictest -p 80 --breaktrace=2000 --tracemark
+
+ # When the high latency is hit, you can display the trace
+ trace-cmd show
+\end{bashinput}
+
diff --git a/labs/preempt-rt-patch-download/preempt-rt-patch-download.tex b/labs/preempt-rt-patch-download/preempt-rt-patch-download.tex
index 2dca8b43..8ddcd1f8 100644
--- a/labs/preempt-rt-patch-download/preempt-rt-patch-download.tex
+++ b/labs/preempt-rt-patch-download/preempt-rt-patch-download.tex
@@ -24,7 +24,7 @@ Since we're going to do Buildroot development, let's clone the
Buildroot source code from its Git repository:
\begin{bashinput}
-git clone https://git.buildroot.net/git/buildroot.git
+git clone git://git.busybox.net/buildroot
\end{bashinput}
In case this is blocked on your network, you can download the Buildroot
@@ -39,10 +39,10 @@ We're going to start a branch from the {\em 2021.11} Buildroot
release, with which this training has been tested.
\begin{bashinput}
-git checkout -b felabs 2021.11
+git checkout -b felabs 2022.02.1
\end{bashinput}
-\section{Configuring Buildroot}
+\section{Configuring Buildroot - STM32MP157}
If you look under \code{configs/}, you will see that there is a file
named \code{stm32mp157a_dk1_defconfig}, which is a ready-to-use Buildroot
@@ -54,14 +54,36 @@ We'll use that configuration as a basis for our setup :
make stm32mp157a_dk1_defconfig
\end{bashinput}
+\section{Configuring Buildroot - Beaglebone Black}
+
+If you look under \code{configs/}, you will see that there is a file
+named \code{beaglebone_defconfig}, which is a ready-to-use Buildroot
+configuration file to build a system for the STM32MP157A platform.
+
+We'll use that configuration as a basis for our setup :
+
+\begin{bashinput}
+make beaglebone_defconfig
+\end{bashinput}
+
+The default configuration for the BeagleBone includes some vendor-specific patches
+that won't apply anymore onto the mainline linux-rt kernel. Simply remove the
+patch file :
+
+\begin{bashinput}
+rm board/beaglebone/patches/linux/*
+\end{bashinput}
+
+\section{Selecting a -RT kernel}
+
We'll configure Buildroot to download the Linux Kernel with the RT Patch applied.
From the \code{make menuconfig} interface, change the Kernel sources's custom
-tarball location to the following :
+git repository location to the following :
\url{git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-stable-rt.git}
-We'll use the latest stable RT version, which is \code{v5.15.14-rt27}. Set that
+We'll use the latest stable RT version, which is \code{v5.15.36-rt41}. Set that
value in the \code{Custom repository version}
Make sure you enable a toolchain with wchar and C++ support, it will be required
@@ -94,12 +116,12 @@ While the build is ongoing, don't hesitate to take a look at the latest
version of the patchset :
\begin{bashinput}
-wget https://cdn.kernel.org/pub/linux/kernel/projects/rt/5.14/patches-5.15.14-rt27.tar.gz
+wget https://cdn.kernel.org/pub/linux/kernel/projects/rt/5.14/patches-5.15.36-rt41.tar.gz
\end{bashinput}
Look at the \code{series} file for more information about each individual patch.
-\section{Setting up serial communication with the board}
+\section{Setting up serial communication with the board - STM32MP157}
The STM32MP1 devkit's serial port can be accessed through the micro-USB connector.
@@ -137,6 +159,63 @@ you wish to exit \code{picocom}, press \code{[Ctrl][a]} followed by
There should be nothing on the serial line so far, as the board is not
powered up yet.
+\section{Setting up serial communication with the board - Beaglebone}
+
+The Beaglebone serial connector is exported on the 6 pins close to one
+of the 48 pins headers. Using your special USB to Serial adapter provided
+by your instructor, connect the ground wire (blue) to the pin closest
+to the power supply connector (let's call it pin 1), and the \code{TX} (red)
+and \code{RX} (green) wires to the pins 4 (board \code{RX}) and
+5 (board \code{TX})\footnote{See
+\url{https://www.olimex.com/Products/Components/Cables/USB-Serial-Cable/USB-Serial-Cable-F/}
+for details about the USB to Serial adapter that we are using.}.
+
+You always should make sure that you connect the \code{TX} pin of the cable
+to the \code{RX} pin of the board, and vice-versa, whatever the board and
+cables that you use.
+
+\begin{center}
+\includegraphics[width=8cm]{common/beaglebone-black-serial-connection.jpg}
+\end{center}
+
+Once the USB to Serial connector is plugged in, a new serial port
+should appear: \code{/dev/ttyUSB0}. You can also see this device
+appear by looking at the output of \code{dmesg}.
+
+To communicate with the board through the serial port, install a
+serial communication program, such as \code{picocom}:
+
+\begin{bashinput}
+sudo apt install picocom
+\end{bashinput}
+
+If you run \code{ls -l /dev/ttyUSB0}, you can also see that only
+\code{root} and users belonging to the \code{dialout} group have
+read and write access to this file. Therefore, you need to add your user
+to the \code{dialout} group:
+
+\begin{bashinput}
+sudo adduser $USER dialout
+\end{bashinput}
+
+{\bf Important}: for the group change to be effective, in Ubuntu 18.04, you have to
+{\em completely reboot} the system \footnote{As explained on
+\url{https://askubuntu.com/questions/1045993/after-adding-a-group-logoutlogin-is-not-enough-in-18-04/}.}.
+A workaround is to run \code{newgrp dialout}, but it is not global.
+You have to run it in each terminal.
+
+Now, you can run \code{picocom -b 115200 /dev/ttyUSB0}, to start serial
+communication on \code{/dev/ttyUSB0}, with a baudrate of \code{115200}. If
+you wish to exit \code{picocom}, press \code{[Ctrl][a]} followed by
+\code{[Ctrl][x]}.
+
+Insert the SD card in the dedicated slot on the BeagleBone Black. Press the S2
+push button (located just above the previous slot), plug in the USB cable and
+release the push button. You should see boot messages on the console.
+
+Wait until the login prompt, then enter \code{root} as user.
+Congratulations! The board has booted and you now have a shell.
+
\section{Checking that we run a Patched kernel}
To check that we are indeed running a kernel with the preempt-RT patch applied and
More information about the training-materials-updates
mailing list