[bootlin/training-materials updates] master: kernel: Add DMA lab (7c1a33c5)
Miquel Raynal
miquel.raynal at bootlin.com
Fri Mar 31 19:51:30 CEST 2023
Repository : https://github.com/bootlin/training-materials
On branch : master
Link : https://github.com/bootlin/training-materials/commit/7c1a33c508453eb0f4685b215d26222d6f343514
>---------------------------------------------------------------
commit 7c1a33c508453eb0f4685b215d26222d6f343514
Author: Miquel Raynal <miquel.raynal at bootlin.com>
Date: Fri Mar 31 17:10:29 2023 +0200
kernel: Add DMA lab
Signed-off-by: Miquel Raynal <miquel.raynal at bootlin.com>
>---------------------------------------------------------------
7c1a33c508453eb0f4685b215d26222d6f343514
labs/kernel-serial-dma/kernel-serial-dma.tex | 247 +++++++++++++++++++++
mk/linux-kernel.mk | 2 +
.../kernel-driver-development-lab-dma.tex | 10 +
3 files changed, 259 insertions(+)
diff --git a/labs/kernel-serial-dma/kernel-serial-dma.tex b/labs/kernel-serial-dma/kernel-serial-dma.tex
new file mode 100644
index 00000000..27ed034c
--- /dev/null
+++ b/labs/kernel-serial-dma/kernel-serial-dma.tex
@@ -0,0 +1,247 @@
+\subchapter{DMA}{Objective: learn how to use the \code{dma} API to
+ handle DMA buffers and coherency, as well as the \code{dmaengine}
+ API to deal with DMA controllers through a generic abstraction}
+
+During this lab, you will:
+
+\begin{itemize}
+\item Setup streaming mappings with the \code{dma} API
+\item Configure a DMA controller with the \code{dmaengine} API
+\item Configure the hardware to trigger DMA transfers
+\item Wait for DMA completion
+\end{itemize}
+
+\section{Setup}
+
+This lab is a continuation of all the previous {\em serial} labs. Use
+the same kernel, environment and paths!
+
+\section{Preparing the driver}
+
+We will use DMA in the write path. As we will receive data from
+userspace, we will need a bounce buffer, we can eg. rename \code{buf}
+into \code{rx_buf} and create another buffer of the same size, named
+\code{tx_buf} in our \code{serial_dev} structure.
+
+As we will also need the \code{resource} structure with the MMIO physical
+addresses from outside of the \code{->probe()}, it might be relevant to save
+the \code{resource} pointer used to derive the \code{miscdev} name into the
+\code{serial_dev} structure as well.
+
+Before going further, re-compile and test your driver.
+
+The \code{serial_write} callback can now be renamed \code{serial_write_pio},
+while we will implement a new callback named
+\code{serial_write_dma}. Remember to also update the \code{.write}
+member of \code{serial_fops} to run the DMA variant. If you want to
+avoid warnings because \code{serial_write_pio} would now be unused, you
+can define it with the \code{__maybe_unused} keyword.
+
+Let's now create two helpers supposed to initialize and cleanup our DMA
+setup. We will call \code{serial_init_dma()} right before registering
+the \code{misc} device. In the \code{->probe()} error path and in the
+remove callback, we will call \code{serial_cleanup_dma()}.
+
+\section{Prepare the DMA controller}
+
+The OMAP UART controller can make use of an external DMA controller. On
+the AM3359, it is actually wired to a DMA controller named EDMA. So we
+will have to deal with the \code{dmaengine} API in order to prepare DMA
+transfers on the controller side. The idea of this API is to fully
+abstract the characteristics of the DMA controller.
+
+In a complete driver we should probably use the helpers checking
+capabilities. Let's just skip this part and assume the two IPs are
+compatible and the addressing masks properly set to 32-bit.
+
+We should at the very least request the DMA channel to use. Open the SoC
+device tree and find the uart2 and uart4 nodes. Look for \code{dma}
+channel properties and their names. Yes, while uart2 seem to be
+connected to the DMA controller through two different channels (one for
+each direction), uart4 is not. Hence, when requesting the channels with
+\kfunc{dma_request_chan()}, we must take care of not error-out upon the
+absence of channel. Mind the return value which is a \kstruct{dma_chan}
+pointer, it must be checked with the \code{IS_ERR()} macro. You may display the
+corresponding error string with {\tt \%pe}!
+
+This channel will be used by all the \code{dmaengine} helpers, so better
+save it in our \code{serial_dev} structure.
+\begin{verbatim}
+struct serial_dev {
+ ...
+ struct dma_chan *txchan;
+};
+\end{verbatim}
+
+We can now configure the DMA controller with details about the upcoming
+transfers:
+\begin{itemize}
+\item memory to device transfers
+\item the source will be memory, we will map buffers when they come,
+ there is no particular constraint on this side
+\item the destination is the UART Tx FIFO, we will ask the DMA to
+ transfer the bytes one after the other (hardware signaling already
+ handles the internal ``flow'')
+\item we shall not use the UART Tx FIFO directly, to be generic we shall use
+ \kfunc{dma_map_resource} first (and save it in \code{serial_dev} to be able
+ to unmap it later)
+\end{itemize}
+
+\begin{verbatim}
+struct dma_slave_config txconf = {};
+dma_addr_t fifo_dma_addr;
+
+serial->fifo_dma_addr = dma_map_resource(dev, serial->res->start + UART_TX * 4,
+ 4, DMA_TO_DEVICE, 0);
+if (dma_mapping_error(dev, serial->fifo_dma_addr)) ...
+
+txconf.direction = DMA_MEM_TO_DEV;
+txconf.dst_addr_width = DMA_SLAVE_BUSWIDTH_1_BYTE;
+txconf.dst_addr = serial->fifo_dma_addr;
+ret = dmaengine_slave_config(serial->txchan, &txconf);
+if (ret) ...
+\end{verbatim}
+
+The cleanup helper should on its side call
+\kfunc{dmaengine_terminate_sync} just to be sure no transfer is
+ongoing, right before un-mapping the FIFO with \kfunc{dma_unmap_resource} and
+releasing the DMA channel with \kfunc{dma_release_channel}.
+
+It is time to recompile your driver and see if any header is missing...
+
+\section{Prepare the UART controller}
+
+On its side, the UART controller must assert some signals to drive the DMA
+flow. We must enable the controlling logic on the Tx DMA channel, by enabling
+\code{DMACTL} in mode 3. We also configure the UART to transmit all the bytes
+as soon as they get in.
+
+\begin{verbatim}
+#define OMAP_UART_SCR_DMAMODE_CTL3 0x7
+#define OMAP_UART_SCR_TX_TRIG_GRANU1 BIT(6)
+
+/* Enable DMA */
+reg_write(serial, OMAP_UART_SCR_DMAMODE_CTL3 | OMAP_UART_SCR_TX_TRIG_GRANU1,
+ UART_OMAP_SCR);
+\end{verbatim}
+
+\section{Process user write requests}
+
+It is now time to deal with user buffers again.
+
+Before doing anything in the \code{write} hook, we shall fill-in the
+\code{serial_dev} structure with:
+\begin{itemize}
+\item a \code{bool txongoing} flag to prevent concurrent uses of the same
+ Tx DMA channel (would be possible by queuing new requests, but let's keep this
+ implementation simple) while not holding any lock for the full duration of
+ the operation.
+\item a \code{struct completion txcomplete} object to asynchronously inform the
+ write thread that the DMA transaction is over (very much like we did with the
+ \code{waitqueue} in the interrupt lab). This object shall be initialized with
+ \code{init_completion(&serial->txcomplete)}.
+\end{itemize}
+
+\begin{verbatim}
+struct serial_dev {
+ ...
+ struct dma_chan *txchan;
+ bool txongoing;
+ struct completion txcomplete;
+};
+\end{verbatim}
+
+As we do not want to deal with concurrent operations anymore to simplify DMA
+handling, we can start and end the write hook with something like:
+
+\begin{verbatim}
+/* Prevent concurrent Tx */
+spin_lock_irqsave(&serial->lock, flags);
+if (serial->txongoing) {
+ spin_unlock_irqrestore(&serial->lock, flags);
+ return -EBUSY;
+}
+serial->txongoing = true;
+spin_unlock_irqrestore(&serial->lock, flags);
+
+...
+
+spin_lock_irqsave(&serial->lock, flags);
+serial->txongoing = false;
+spin_unlock_irqrestore(&serial->lock, flags);
+\end{verbatim}
+
+Also do not forget to check the absence of DMA channel, return
+\code{-EOPNOTSUPP} in this case.
+
+The first step in this \code{->write()} hook is to use \code{serial->tx_buf} as
+bounce buffer by copying the user data using \kfunc{copy_from_user}. Let's
+handle up to \code{SERIAL_BUFSIZE} bytes at a time. One can use \kfunc{min_t}
+to derive the right amount of bytes to deal with.
+
+Before mapping the buffer for DMA purposes, we need to handle one last
+thing. The OMAP 8250 UART controller has a limitation, its internal circuitry
+to trigger DMA transfers is a bit broken, and the first character needs to be
+fed manually, so let's just save that first byte away:
+
+\begin{verbatim}
+char first;
+
+/* OMAP 8250 UART quirk: need to write the first byte manually */
+first = serial->tx_buf[0];
+\end{verbatim}
+
+Now we can remap the buffer. We have a single buffer so we can use
+\kfunc{dma_map_single}. The output value is a \ksym{dma_addr_t}. Save this
+value as we will reuse it. Also do not forget to check its validity with
+\kfunc{dma_mapping_error}.
+
+We now have all the missing information compared to the \code{serial_init_dma}
+step, like the \ksym{dma_addr_t} of the buffer and its length. Let's create a
+descriptor filled with all the default information known by the DMA controller
+plus the additional details we can now provide:
+
+\begin{verbatim}
+struct dma_async_tx_descriptor *desc;
+
+desc = dmaengine_prep_slave_single(dma->txchan, dma->dma_addr + 1,
+ len - 1, DMA_MEM_TO_DEV,
+ DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
+if (!desc) ...
+\end{verbatim}
+
+Mind the \code{+ 1} and \code{- 1} operations in the snippet above, they are
+here to skip the first character which we will send manually.
+
+We can now use the returned descriptor to register a callback. This callback
+will just call \kfunc{complete} over the completion object. Which also means
+this completion object could be re-initialized while we register the callback,
+just in case.
+
+The DMA transfer contained in the descriptor can now be queued into the DMA
+controller queue:
+
+\begin{verbatim}
+dma_cookie_t cookie;
+
+cookie = dmaengine_submit(desc);
+ret = dma_submit_error(cookie);
+if (ret) ...
+\end{verbatim}
+
+The transfer can be triggered. This is usually an operation that is only
+required on the DMA controller side, but remember here we also need to trigger
+it on the UART controller side:
+
+\begin{verbatim}
+dma_async_issue_pending(dma->txchan);
+reg_write(serial, first, UART_TX);
+\end{verbatim}
+
+The transfer being asynchronous, it is finally required to wait for completion
+with one of the \kfunc{wait_for_completion} variants, and to call
+\kfunc{dma_unmap_single} right after it.
+
+You can now test your driver. Writing a single character is of course not
+relevant as our driver would just use the previous method to send it. Try with
+a string instead!
diff --git a/mk/linux-kernel.mk b/mk/linux-kernel.mk
index ef1a835c..01518f10 100644
--- a/mk/linux-kernel.mk
+++ b/mk/linux-kernel.mk
@@ -37,6 +37,7 @@ LINUX_KERNEL_SLIDES = \
kernel-driver-development-concurrency \
kernel-driver-development-lab-locking \
kernel-driver-development-dma \
+ kernel-driver-development-lab-dma \
kernel-driver-development-debugging \
kernel-driver-development-lab-debugging \
kernel-porting-title \
@@ -67,4 +68,5 @@ LINUX_KERNEL_LABS = setup \
kernel-serial-output \
kernel-serial-interrupt \
kernel-locking \
+ kernel-serial-dma \
kernel-debugging \
diff --git a/slides/kernel-driver-development-lab-dma/kernel-driver-development-lab-dma.tex b/slides/kernel-driver-development-lab-dma/kernel-driver-development-lab-dma.tex
new file mode 100644
index 00000000..4f61a5cf
--- /dev/null
+++ b/slides/kernel-driver-development-lab-dma/kernel-driver-development-lab-dma.tex
@@ -0,0 +1,10 @@
+\setuplabframe
+{DMA}
+{
+ \begin{itemize}
+ \item Setup streaming mappings with the \code{dma} API
+ \item Configure a DMA controller with the \code{dmaengine} API
+ \item Configure the hardware to trigger DMA transfers
+ \item Wait for DMA completion
+ \end{itemize}
+}
More information about the training-materials-updates
mailing list