[bootlin/training-materials updates] master: preempt-rt: update labs (e61a5037)
Maxime Chevallier
maxime.chevallier at bootlin.com
Wed Jan 18 18:52:05 CET 2023
Repository : https://github.com/bootlin/training-materials
On branch : master
Link : https://github.com/bootlin/training-materials/commit/e61a50374cc714e553791418a94e131eca48ddc9
>---------------------------------------------------------------
commit e61a50374cc714e553791418a94e131eca48ddc9
Author: Maxime Chevallier <maxime.chevallier at bootlin.com>
Date: Wed Jan 18 11:30:16 2023 +0100
preempt-rt: update labs
Signed-off-by: Maxime Chevallier <maxime.chevallier at bootlin.com>
>---------------------------------------------------------------
e61a50374cc714e553791418a94e131eca48ddc9
.../preempt-rt-application-development.tex | 9 +-
.../preempt-rt-benchmarking.tex | 27 ++++--
.../preempt-rt-patch-download.tex | 100 ++++-----------------
3 files changed, 42 insertions(+), 94 deletions(-)
diff --git a/labs/preempt-rt-application-development/preempt-rt-application-development.tex b/labs/preempt-rt-application-development/preempt-rt-application-development.tex
index dc5d2d0b..63c3f92f 100644
--- a/labs/preempt-rt-application-development/preempt-rt-application-development.tex
+++ b/labs/preempt-rt-application-development/preempt-rt-application-development.tex
@@ -2,22 +2,19 @@
During this lab, you will:
\begin{itemize}
- \item Analyse the source code of an application
\item Better understand the page-faulting mechanism
- \item Use ftrace to display the program's behaviour
+ \item Debug interferences with a time-sensitive application
\end{itemize}
\section{Compile and run the program}
-Compile the pgflt program : \code{make}
+Compile the crc\_test program : \code{make}
Test the various options, and analyse how the Stack and Heap are allocated and
pre-faulted.
Use ftrace to visualise the page-fault occurings :
\begin{bashinput}
- trace-cmd record -e *page_fault* ./pgflt 0x1f
+ trace-cmd record -e *page_fault* ./crc_test
\end{bashinput}
-This is a good example where kernelshark can be useful, try to export the trace
-to your host machine for cold analysis.
diff --git a/labs/preempt-rt-benchmarking/preempt-rt-benchmarking.tex b/labs/preempt-rt-benchmarking/preempt-rt-benchmarking.tex
index f5c44268..8be3380b 100644
--- a/labs/preempt-rt-benchmarking/preempt-rt-benchmarking.tex
+++ b/labs/preempt-rt-benchmarking/preempt-rt-benchmarking.tex
@@ -23,6 +23,7 @@ In the Buildroot \code{make menuconfig} interface, enable the following packages
\item scheduling utilites from the util-linux package
\item python3
\item screen
+ \item dropbear
\end{itemize}
You'll also need to enable a few features in the kernel. Run \code{make linux-menuconfig} and select
@@ -33,7 +34,7 @@ the following options, located in "Kernel Hacking" :
\item Tracers -> Function tracer, Interrupts-off tracer, Preemption off tracer, scheduling latency tracer...
\end{itemize}
-Re-build the image, and boot it on the board : \code {make linux-rebuild && make}
+Re-build the image, and boot it on the board : \code {make linux-rebuild all}
Let's first start by establishing the baseline latency by simply running \code{cyclictest} :
@@ -79,13 +80,13 @@ Some kernel options can also be useful :
To get a first idea of the wakeup latencies you can expect on your system, launch
\code{cyclictest} on the target, and take a look at the Max latency. The lower, the
-better. You shouldn't get bit latency spikes.
+better. You shouldn't get big latency spikes.
By running \code{cyclictest} as is, you will run the benchmark with the default
scheduling policy (\code{SCHED_OTHER}), and without any CPU pinning.
You may not notice huge latencies right away, since the system at that point isn't
-doint much else. You can try to load the system and see how that affects the latencies
+doing much. You can try to load the system and see how that affects the latencies
To run cyclictest with a real-time scheduling policy, use the \code{-p <prio>} option.
Cyclictest doesn't play well with the \code{chrt} command, since it will itself re-set
@@ -102,14 +103,17 @@ First, setup the board's network interface :
\code{ip link set eth0 up}
-\code{udhcpc -i eth0}
+\code{ip address add 192.168.0.2/24 dev eth0}
-Then run the \code{iperf3} server in the background :
+Make sure that your computer has its network interface on the same subnet as your
+target.
+
+On the board,run the \code{iperf3} server in the background :
\code{iperf3 -s -D}
Re-run your cyclictest benchmark, and start sending traffic to your target. From
-you host computer, run \code{iperf3 -c <addr_of_your_board>}. You should start
+you host computer, run \code{iperf3 -c 192.168.0.2}. You should start
seeing the latency rising up.
Try comparing the latencies you get between \code{cyclictest} and \code{cyclictest -p 40}. Do you
@@ -157,7 +161,7 @@ also to change the interrupt CPU affinity.
For cyclictest, you can either run cyclictest with the \code{-a <cpu_num>} option,
or use \code{taskset -c <cpu_num> cyclictest ...}.
-Try running hackbench and cyclictest on the same CPU, and then on differenc CPU and
+Try running hackbench and cyclictest on the same CPU, and then on different CPU and
compare the induced latencies.
\subsection{Interrupt Pinning}
@@ -309,3 +313,12 @@ stopped automatically when cyclictest detects a high latency :
trace-cmd show
\end{bashinput}
+\section{Using the osnoise tracer}
+
+The following command is used to generate an histogram of os noises with the
+\code{osnoise} tracer :
+
+\code{osnoise hist -c 1 -P f:49 -d 1m}
+
+The output shoud show a maximum latency around 50 000 microseconds. Why is that ?
+
diff --git a/labs/preempt-rt-patch-download/preempt-rt-patch-download.tex b/labs/preempt-rt-patch-download/preempt-rt-patch-download.tex
index 8ddcd1f8..b7634dff 100644
--- a/labs/preempt-rt-patch-download/preempt-rt-patch-download.tex
+++ b/labs/preempt-rt-patch-download/preempt-rt-patch-download.tex
@@ -28,50 +28,36 @@ git clone git://git.busybox.net/buildroot
\end{bashinput}
In case this is blocked on your network, you can download the Buildroot
-tarball \code{buildroot-2021.11.tar.bz2} from
+tarball \code{buildroot-2022.11.tar.bz2} from
\code{https://buildroot.org/downloads/} and extract it. However in this
case, you won't be able to use {\em Git} to visualize your changes and
keep track of them.
Go into the newly created \code{buildroot} directory.
-We're going to start a branch from the {\em 2021.11} Buildroot
+We're going to start a branch from the {\em 2022.11} Buildroot
release, with which this training has been tested.
\begin{bashinput}
-git checkout -b felabs 2022.02.1
+git checkout -b felabs 2022.11
\end{bashinput}
-\section{Configuring Buildroot - STM32MP157}
+\section{Configuring Buildroot}
-If you look under \code{configs/}, you will see that there is a file
-named \code{stm32mp157a_dk1_defconfig}, which is a ready-to-use Buildroot
-configuration file to build a system for the STM32MP157A platform.
-
-We'll use that configuration as a basis for our setup :
+First of all, apply the patches provided in the \code{buildroot-patches} folder :
\begin{bashinput}
-make stm32mp157a_dk1_defconfig
+ git am ~/preempt-rt/preempt-rt-lab-data/buildroot-patches/*
\end{bashinput}
-\section{Configuring Buildroot - Beaglebone Black}
-
If you look under \code{configs/}, you will see that there is a file
-named \code{beaglebone_defconfig}, which is a ready-to-use Buildroot
+named \code{stm32mp157a_dk1_defconfig}, which is a ready-to-use Buildroot
configuration file to build a system for the STM32MP157A platform.
We'll use that configuration as a basis for our setup :
\begin{bashinput}
-make beaglebone_defconfig
-\end{bashinput}
-
-The default configuration for the BeagleBone includes some vendor-specific patches
-that won't apply anymore onto the mainline linux-rt kernel. Simply remove the
-patch file :
-
-\begin{bashinput}
-rm board/beaglebone/patches/linux/*
+make stm32mp157a_dk1_defconfig
\end{bashinput}
\section{Selecting a -RT kernel}
@@ -81,9 +67,9 @@ We'll configure Buildroot to download the Linux Kernel with the RT Patch applied
From the \code{make menuconfig} interface, change the Kernel sources's custom
git repository location to the following :
-\url{git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-stable-rt.git}
+\url{git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git}
-We'll use the latest stable RT version, which is \code{v5.15.36-rt41}. Set that
+We'll use the latest development RT version, which is \code{v6.2-rc3-rt1}. Set that
value in the \code{Custom repository version}
Make sure you enable a toolchain with wchar and C++ support, it will be required
@@ -116,7 +102,7 @@ While the build is ongoing, don't hesitate to take a look at the latest
version of the patchset :
\begin{bashinput}
-wget https://cdn.kernel.org/pub/linux/kernel/projects/rt/5.14/patches-5.15.36-rt41.tar.gz
+wget https://cdn.kernel.org/pub/linux/kernel/projects/rt/6.2/patches-6.2-rc3-rt1.tar.gz
\end{bashinput}
Look at the \code{series} file for more information about each individual patch.
@@ -159,62 +145,7 @@ you wish to exit \code{picocom}, press \code{[Ctrl][a]} followed by
There should be nothing on the serial line so far, as the board is not
powered up yet.
-\section{Setting up serial communication with the board - Beaglebone}
-
-The Beaglebone serial connector is exported on the 6 pins close to one
-of the 48 pins headers. Using your special USB to Serial adapter provided
-by your instructor, connect the ground wire (blue) to the pin closest
-to the power supply connector (let's call it pin 1), and the \code{TX} (red)
-and \code{RX} (green) wires to the pins 4 (board \code{RX}) and
-5 (board \code{TX})\footnote{See
-\url{https://www.olimex.com/Products/Components/Cables/USB-Serial-Cable/USB-Serial-Cable-F/}
-for details about the USB to Serial adapter that we are using.}.
-
-You always should make sure that you connect the \code{TX} pin of the cable
-to the \code{RX} pin of the board, and vice-versa, whatever the board and
-cables that you use.
-
-\begin{center}
-\includegraphics[width=8cm]{common/beaglebone-black-serial-connection.jpg}
-\end{center}
-
-Once the USB to Serial connector is plugged in, a new serial port
-should appear: \code{/dev/ttyUSB0}. You can also see this device
-appear by looking at the output of \code{dmesg}.
-
-To communicate with the board through the serial port, install a
-serial communication program, such as \code{picocom}:
-
-\begin{bashinput}
-sudo apt install picocom
-\end{bashinput}
-
-If you run \code{ls -l /dev/ttyUSB0}, you can also see that only
-\code{root} and users belonging to the \code{dialout} group have
-read and write access to this file. Therefore, you need to add your user
-to the \code{dialout} group:
-
-\begin{bashinput}
-sudo adduser $USER dialout
-\end{bashinput}
-
-{\bf Important}: for the group change to be effective, in Ubuntu 18.04, you have to
-{\em completely reboot} the system \footnote{As explained on
-\url{https://askubuntu.com/questions/1045993/after-adding-a-group-logoutlogin-is-not-enough-in-18-04/}.}.
-A workaround is to run \code{newgrp dialout}, but it is not global.
-You have to run it in each terminal.
-
-Now, you can run \code{picocom -b 115200 /dev/ttyUSB0}, to start serial
-communication on \code{/dev/ttyUSB0}, with a baudrate of \code{115200}. If
-you wish to exit \code{picocom}, press \code{[Ctrl][a]} followed by
-\code{[Ctrl][x]}.
-
-Insert the SD card in the dedicated slot on the BeagleBone Black. Press the S2
-push button (located just above the previous slot), plug in the USB cable and
-release the push button. You should see boot messages on the console.
-
-Wait until the login prompt, then enter \code{root} as user.
-Congratulations! The board has booted and you now have a shell.
+\section{Flash the image to the SDCard}
\section{Checking that we run a Patched kernel}
@@ -225,3 +156,10 @@ First, use the \code{uname -a} command. Running an RT kernel should show the \co
The other way to check is to look at the file \code{/sys/kernel/realtime}. It's content is always '1', but the file only exists for RT kernels.
+Take a look at the boot logs, can you see something that's not going right ? Why ?
+
+\section{Fixing our first kernel bug}
+
+You should see quite a lot of \code{BUG: scheduling while atomic} messages. The
+actual issue is non-trivial to fix, but it's a nice example to analyse.
+
More information about the training-materials-updates
mailing list