cachepc-linux

Fork of AMDESE/linux with modifications for CachePC side-channel attack
git clone https://git.sinitax.com/sinitax/cachepc-linux
Log | Files | Refs | README | LICENSE | sfeed.txt

dev-subdev.rst (18973B)


      1.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
      2
      3.. _subdev:
      4
      5********************
      6Sub-device Interface
      7********************
      8
      9The complex nature of V4L2 devices, where hardware is often made of
     10several integrated circuits that need to interact with each other in a
     11controlled way, leads to complex V4L2 drivers. The drivers usually
     12reflect the hardware model in software, and model the different hardware
     13components as software blocks called sub-devices.
     14
     15V4L2 sub-devices are usually kernel-only objects. If the V4L2 driver
     16implements the media device API, they will automatically inherit from
     17media entities. Applications will be able to enumerate the sub-devices
     18and discover the hardware topology using the media entities, pads and
     19links enumeration API.
     20
     21In addition to make sub-devices discoverable, drivers can also choose to
     22make them directly configurable by applications. When both the
     23sub-device driver and the V4L2 device driver support this, sub-devices
     24will feature a character device node on which ioctls can be called to
     25
     26-  query, read and write sub-devices controls
     27
     28-  subscribe and unsubscribe to events and retrieve them
     29
     30-  negotiate image formats on individual pads
     31
     32Sub-device character device nodes, conventionally named
     33``/dev/v4l-subdev*``, use major number 81.
     34
     35Drivers may opt to limit the sub-device character devices to only expose
     36operations that do not modify the device state. In such a case the sub-devices
     37are referred to as ``read-only`` in the rest of this documentation, and the
     38related restrictions are documented in individual ioctls.
     39
     40
     41Controls
     42========
     43
     44Most V4L2 controls are implemented by sub-device hardware. Drivers
     45usually merge all controls and expose them through video device nodes.
     46Applications can control all sub-devices through a single interface.
     47
     48Complex devices sometimes implement the same control in different pieces
     49of hardware. This situation is common in embedded platforms, where both
     50sensors and image processing hardware implement identical functions,
     51such as contrast adjustment, white balance or faulty pixels correction.
     52As the V4L2 controls API doesn't support several identical controls in a
     53single device, all but one of the identical controls are hidden.
     54
     55Applications can access those hidden controls through the sub-device
     56node with the V4L2 control API described in :ref:`control`. The ioctls
     57behave identically as when issued on V4L2 device nodes, with the
     58exception that they deal only with controls implemented in the
     59sub-device.
     60
     61Depending on the driver, those controls might also be exposed through
     62one (or several) V4L2 device nodes.
     63
     64
     65Events
     66======
     67
     68V4L2 sub-devices can notify applications of events as described in
     69:ref:`event`. The API behaves identically as when used on V4L2 device
     70nodes, with the exception that it only deals with events generated by
     71the sub-device. Depending on the driver, those events might also be
     72reported on one (or several) V4L2 device nodes.
     73
     74
     75.. _pad-level-formats:
     76
     77Pad-level Formats
     78=================
     79
     80.. warning::
     81
     82    Pad-level formats are only applicable to very complex devices that
     83    need to expose low-level format configuration to user space. Generic
     84    V4L2 applications do *not* need to use the API described in this
     85    section.
     86
     87.. note::
     88
     89    For the purpose of this section, the term *format* means the
     90    combination of media bus data format, frame width and frame height.
     91
     92Image formats are typically negotiated on video capture and output
     93devices using the format and
     94:ref:`selection <VIDIOC_SUBDEV_G_SELECTION>` ioctls. The driver is
     95responsible for configuring every block in the video pipeline according
     96to the requested format at the pipeline input and/or output.
     97
     98For complex devices, such as often found in embedded systems, identical
     99image sizes at the output of a pipeline can be achieved using different
    100hardware configurations. One such example is shown on
    101:ref:`pipeline-scaling`, where image scaling can be performed on both
    102the video sensor and the host image processing hardware.
    103
    104
    105.. _pipeline-scaling:
    106
    107.. kernel-figure:: pipeline.dot
    108    :alt:   pipeline.dot
    109    :align: center
    110
    111    Image Format Negotiation on Pipelines
    112
    113    High quality and high speed pipeline configuration
    114
    115
    116
    117The sensor scaler is usually of less quality than the host scaler, but
    118scaling on the sensor is required to achieve higher frame rates.
    119Depending on the use case (quality vs. speed), the pipeline must be
    120configured differently. Applications need to configure the formats at
    121every point in the pipeline explicitly.
    122
    123Drivers that implement the :ref:`media API <media-controller-intro>`
    124can expose pad-level image format configuration to applications. When
    125they do, applications can use the
    126:ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` and
    127:ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls. to
    128negotiate formats on a per-pad basis.
    129
    130Applications are responsible for configuring coherent parameters on the
    131whole pipeline and making sure that connected pads have compatible
    132formats. The pipeline is checked for formats mismatch at
    133:ref:`VIDIOC_STREAMON <VIDIOC_STREAMON>` time, and an ``EPIPE`` error
    134code is then returned if the configuration is invalid.
    135
    136Pad-level image format configuration support can be tested by calling
    137the :ref:`VIDIOC_SUBDEV_G_FMT` ioctl on pad
    1380. If the driver returns an ``EINVAL`` error code pad-level format
    139configuration is not supported by the sub-device.
    140
    141
    142Format Negotiation
    143------------------
    144
    145Acceptable formats on pads can (and usually do) depend on a number of
    146external parameters, such as formats on other pads, active links, or
    147even controls. Finding a combination of formats on all pads in a video
    148pipeline, acceptable to both application and driver, can't rely on
    149formats enumeration only. A format negotiation mechanism is required.
    150
    151Central to the format negotiation mechanism are the get/set format
    152operations. When called with the ``which`` argument set to
    153:ref:`V4L2_SUBDEV_FORMAT_TRY <VIDIOC_SUBDEV_G_FMT>`, the
    154:ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` and
    155:ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls operate on
    156a set of formats parameters that are not connected to the hardware
    157configuration. Modifying those 'try' formats leaves the device state
    158untouched (this applies to both the software state stored in the driver
    159and the hardware state stored in the device itself).
    160
    161While not kept as part of the device state, try formats are stored in
    162the sub-device file handles. A
    163:ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` call will return
    164the last try format set *on the same sub-device file handle*. Several
    165applications querying the same sub-device at the same time will thus not
    166interact with each other.
    167
    168To find out whether a particular format is supported by the device,
    169applications use the
    170:ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctl. Drivers
    171verify and, if needed, change the requested ``format`` based on device
    172requirements and return the possibly modified value. Applications can
    173then choose to try a different format or accept the returned value and
    174continue.
    175
    176Formats returned by the driver during a negotiation iteration are
    177guaranteed to be supported by the device. In particular, drivers
    178guarantee that a returned format will not be further changed if passed
    179to an :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` call as-is
    180(as long as external parameters, such as formats on other pads or links'
    181configuration are not changed).
    182
    183Drivers automatically propagate formats inside sub-devices. When a try
    184or active format is set on a pad, corresponding formats on other pads of
    185the same sub-device can be modified by the driver. Drivers are free to
    186modify formats as required by the device. However, they should comply
    187with the following rules when possible:
    188
    189-  Formats should be propagated from sink pads to source pads. Modifying
    190   a format on a source pad should not modify the format on any sink
    191   pad.
    192
    193-  Sub-devices that scale frames using variable scaling factors should
    194   reset the scale factors to default values when sink pads formats are
    195   modified. If the 1:1 scaling ratio is supported, this means that
    196   source pads formats should be reset to the sink pads formats.
    197
    198Formats are not propagated across links, as that would involve
    199propagating them from one sub-device file handle to another.
    200Applications must then take care to configure both ends of every link
    201explicitly with compatible formats. Identical formats on the two ends of
    202a link are guaranteed to be compatible. Drivers are free to accept
    203different formats matching device requirements as being compatible.
    204
    205:ref:`sample-pipeline-config` shows a sample configuration sequence
    206for the pipeline described in :ref:`pipeline-scaling` (table columns
    207list entity names and pad numbers).
    208
    209
    210.. raw:: latex
    211
    212    \begingroup
    213    \scriptsize
    214    \setlength{\tabcolsep}{2pt}
    215
    216.. tabularcolumns:: |p{2.0cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}|
    217
    218.. _sample-pipeline-config:
    219
    220.. flat-table:: Sample Pipeline Configuration
    221    :header-rows:  1
    222    :stub-columns: 0
    223    :widths: 5 5 5 5 5 5 5
    224
    225    * -
    226      - Sensor/0
    227
    228        format
    229      - Frontend/0
    230
    231        format
    232      - Frontend/1
    233
    234        format
    235      - Scaler/0
    236
    237        format
    238      - Scaler/0
    239
    240        compose selection rectangle
    241      - Scaler/1
    242
    243        format
    244    * - Initial state
    245      - 2048x1536
    246
    247        SGRBG8_1X8
    248      - (default)
    249      - (default)
    250      - (default)
    251      - (default)
    252      - (default)
    253    * - Configure frontend sink format
    254      - 2048x1536
    255
    256        SGRBG8_1X8
    257      - *2048x1536*
    258
    259        *SGRBG8_1X8*
    260      - *2046x1534*
    261
    262        *SGRBG8_1X8*
    263      - (default)
    264      - (default)
    265      - (default)
    266    * - Configure scaler sink format
    267      - 2048x1536
    268
    269        SGRBG8_1X8
    270      - 2048x1536
    271
    272        SGRBG8_1X8
    273      - 2046x1534
    274
    275        SGRBG8_1X8
    276      - *2046x1534*
    277
    278        *SGRBG8_1X8*
    279      - *0,0/2046x1534*
    280      - *2046x1534*
    281
    282        *SGRBG8_1X8*
    283    * - Configure scaler sink compose selection
    284      - 2048x1536
    285
    286        SGRBG8_1X8
    287      - 2048x1536
    288
    289        SGRBG8_1X8
    290      - 2046x1534
    291
    292        SGRBG8_1X8
    293      - 2046x1534
    294
    295        SGRBG8_1X8
    296      - *0,0/1280x960*
    297      - *1280x960*
    298
    299        *SGRBG8_1X8*
    300
    301.. raw:: latex
    302
    303    \endgroup
    304
    3051. Initial state. The sensor source pad format is set to its native 3MP
    306   size and V4L2_MBUS_FMT_SGRBG8_1X8 media bus code. Formats on the
    307   host frontend and scaler sink and source pads have the default
    308   values, as well as the compose rectangle on the scaler's sink pad.
    309
    3102. The application configures the frontend sink pad format's size to
    311   2048x1536 and its media bus code to V4L2_MBUS_FMT_SGRBG_1X8. The
    312   driver propagates the format to the frontend source pad.
    313
    3143. The application configures the scaler sink pad format's size to
    315   2046x1534 and the media bus code to V4L2_MBUS_FMT_SGRBG_1X8 to
    316   match the frontend source size and media bus code. The media bus code
    317   on the sink pad is set to V4L2_MBUS_FMT_SGRBG_1X8. The driver
    318   propagates the size to the compose selection rectangle on the
    319   scaler's sink pad, and the format to the scaler source pad.
    320
    3214. The application configures the size of the compose selection
    322   rectangle of the scaler's sink pad 1280x960. The driver propagates
    323   the size to the scaler's source pad format.
    324
    325When satisfied with the try results, applications can set the active
    326formats by setting the ``which`` argument to
    327``V4L2_SUBDEV_FORMAT_ACTIVE``. Active formats are changed exactly as try
    328formats by drivers. To avoid modifying the hardware state during format
    329negotiation, applications should negotiate try formats first and then
    330modify the active settings using the try formats returned during the
    331last negotiation iteration. This guarantees that the active format will
    332be applied as-is by the driver without being modified.
    333
    334
    335.. _v4l2-subdev-selections:
    336
    337Selections: cropping, scaling and composition
    338---------------------------------------------
    339
    340Many sub-devices support cropping frames on their input or output pads
    341(or possible even on both). Cropping is used to select the area of
    342interest in an image, typically on an image sensor or a video decoder.
    343It can also be used as part of digital zoom implementations to select
    344the area of the image that will be scaled up.
    345
    346Crop settings are defined by a crop rectangle and represented in a
    347struct :c:type:`v4l2_rect` by the coordinates of the top
    348left corner and the rectangle size. Both the coordinates and sizes are
    349expressed in pixels.
    350
    351As for pad formats, drivers store try and active rectangles for the
    352selection targets :ref:`v4l2-selections-common`.
    353
    354On sink pads, cropping is applied relative to the current pad format.
    355The pad format represents the image size as received by the sub-device
    356from the previous block in the pipeline, and the crop rectangle
    357represents the sub-image that will be transmitted further inside the
    358sub-device for processing.
    359
    360The scaling operation changes the size of the image by scaling it to new
    361dimensions. The scaling ratio isn't specified explicitly, but is implied
    362from the original and scaled image sizes. Both sizes are represented by
    363struct :c:type:`v4l2_rect`.
    364
    365Scaling support is optional. When supported by a subdev, the crop
    366rectangle on the subdev's sink pad is scaled to the size configured
    367using the
    368:ref:`VIDIOC_SUBDEV_S_SELECTION <VIDIOC_SUBDEV_G_SELECTION>` IOCTL
    369using ``V4L2_SEL_TGT_COMPOSE`` selection target on the same pad. If the
    370subdev supports scaling but not composing, the top and left values are
    371not used and must always be set to zero.
    372
    373On source pads, cropping is similar to sink pads, with the exception
    374that the source size from which the cropping is performed, is the
    375COMPOSE rectangle on the sink pad. In both sink and source pads, the
    376crop rectangle must be entirely contained inside the source image size
    377for the crop operation.
    378
    379The drivers should always use the closest possible rectangle the user
    380requests on all selection targets, unless specifically told otherwise.
    381``V4L2_SEL_FLAG_GE`` and ``V4L2_SEL_FLAG_LE`` flags may be used to round
    382the image size either up or down. :ref:`v4l2-selection-flags`
    383
    384
    385Types of selection targets
    386--------------------------
    387
    388
    389Actual targets
    390^^^^^^^^^^^^^^
    391
    392Actual targets (without a postfix) reflect the actual hardware
    393configuration at any point of time. There is a BOUNDS target
    394corresponding to every actual target.
    395
    396
    397BOUNDS targets
    398^^^^^^^^^^^^^^
    399
    400BOUNDS targets is the smallest rectangle that contains all valid actual
    401rectangles. It may not be possible to set the actual rectangle as large
    402as the BOUNDS rectangle, however. This may be because e.g. a sensor's
    403pixel array is not rectangular but cross-shaped or round. The maximum
    404size may also be smaller than the BOUNDS rectangle.
    405
    406
    407Order of configuration and format propagation
    408---------------------------------------------
    409
    410Inside subdevs, the order of image processing steps will always be from
    411the sink pad towards the source pad. This is also reflected in the order
    412in which the configuration must be performed by the user: the changes
    413made will be propagated to any subsequent stages. If this behaviour is
    414not desired, the user must set ``V4L2_SEL_FLAG_KEEP_CONFIG`` flag. This
    415flag causes no propagation of the changes are allowed in any
    416circumstances. This may also cause the accessed rectangle to be adjusted
    417by the driver, depending on the properties of the underlying hardware.
    418
    419The coordinates to a step always refer to the actual size of the
    420previous step. The exception to this rule is the sink compose
    421rectangle, which refers to the sink compose bounds rectangle --- if it
    422is supported by the hardware.
    423
    4241. Sink pad format. The user configures the sink pad format. This format
    425   defines the parameters of the image the entity receives through the
    426   pad for further processing.
    427
    4282. Sink pad actual crop selection. The sink pad crop defines the crop
    429   performed to the sink pad format.
    430
    4313. Sink pad actual compose selection. The size of the sink pad compose
    432   rectangle defines the scaling ratio compared to the size of the sink
    433   pad crop rectangle. The location of the compose rectangle specifies
    434   the location of the actual sink compose rectangle in the sink compose
    435   bounds rectangle.
    436
    4374. Source pad actual crop selection. Crop on the source pad defines crop
    438   performed to the image in the sink compose bounds rectangle.
    439
    4405. Source pad format. The source pad format defines the output pixel
    441   format of the subdev, as well as the other parameters with the
    442   exception of the image width and height. Width and height are defined
    443   by the size of the source pad actual crop selection.
    444
    445Accessing any of the above rectangles not supported by the subdev will
    446return ``EINVAL``. Any rectangle referring to a previous unsupported
    447rectangle coordinates will instead refer to the previous supported
    448rectangle. For example, if sink crop is not supported, the compose
    449selection will refer to the sink pad format dimensions instead.
    450
    451
    452.. _subdev-image-processing-crop:
    453
    454.. kernel-figure:: subdev-image-processing-crop.svg
    455    :alt:   subdev-image-processing-crop.svg
    456    :align: center
    457
    458    **Figure 4.5. Image processing in subdevs: simple crop example**
    459
    460In the above example, the subdev supports cropping on its sink pad. To
    461configure it, the user sets the media bus format on the subdev's sink
    462pad. Now the actual crop rectangle can be set on the sink pad --- the
    463location and size of this rectangle reflect the location and size of a
    464rectangle to be cropped from the sink format. The size of the sink crop
    465rectangle will also be the size of the format of the subdev's source
    466pad.
    467
    468
    469.. _subdev-image-processing-scaling-multi-source:
    470
    471.. kernel-figure:: subdev-image-processing-scaling-multi-source.svg
    472    :alt:   subdev-image-processing-scaling-multi-source.svg
    473    :align: center
    474
    475    **Figure 4.6. Image processing in subdevs: scaling with multiple sources**
    476
    477In this example, the subdev is capable of first cropping, then scaling
    478and finally cropping for two source pads individually from the resulting
    479scaled image. The location of the scaled image in the cropped image is
    480ignored in sink compose target. Both of the locations of the source crop
    481rectangles refer to the sink scaling rectangle, independently cropping
    482an area at location specified by the source crop rectangle from it.
    483
    484
    485.. _subdev-image-processing-full:
    486
    487.. kernel-figure:: subdev-image-processing-full.svg
    488    :alt:    subdev-image-processing-full.svg
    489    :align:  center
    490
    491    **Figure 4.7. Image processing in subdevs: scaling and composition with multiple sinks and sources**
    492
    493The subdev driver supports two sink pads and two source pads. The images
    494from both of the sink pads are individually cropped, then scaled and
    495further composed on the composition bounds rectangle. From that, two
    496independent streams are cropped and sent out of the subdev from the
    497source pads.
    498
    499
    500.. toctree::
    501    :maxdepth: 1
    502
    503    subdev-formats