-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

=============================================================================
FreeBSD-EN-20:11.ena                                            Errata Notice
                                                          The FreeBSD Project

Topic:          Stability issues in ena(4) driver

Category:       core
Module:         ena
Announced:      2020-06-09
Credits:        Colin Percival
Affects:        FreeBSD 12.1
Corrected:      2020-05-12 18:44:41 UTC (stable/12, 12.1-STABLE)
                2020-06-09 16:10:57 UTC (releng/12.1, 12.1-RELEASE-p6)

For general information regarding FreeBSD Errata Notices and Security
Advisories, including descriptions of the fields above, security
branches, and the following sections, please visit
<URL:https://security.FreeBSD.org/>.

I.   Background

The ena(4) driver is used to access the Elastic Network Adapter network
interface on recent Amazon Elastic Compute Cloud (EC2) instances.  It is
designed to make full use of the EC2 network, operating at up to 100 Gbps.

II.  Problem Description

The ena(4) driver attempts to allocate a large number of 16 kB mbuf clusters
to use as buffers for received packets.  Under low memory conditions, the
process of attempting to allocate these buffers may take a long time.

On FreeBSD 12.x, these allocations are attempted from an interrupt thread
running at high priority, and can subsequently starve other parts of the
system of CPU time.

It appears that systems using the ZFS filesystem may be more likely to
experience this issue, possibly due to the memory pressure created by the
allocation of memory to the ZFS ARC.

III. Impact

The system may become unresponsive, appearing to "lock up" for periods of
several seconds.  The ena(4) driver may log errors such as

  The number of lost tx completion is above the threshold (129 > 128).

and reset the device; the nvme(4) driver may also experience timeouts
resulting in a range of errors being reported and the device being reset,
and in some cases causing kernel panics.

IV.  Workaround

No workaround is available.  Systems which are not part of the Amazon
Elastic Compute Cloud (EC2) are not affected; nor are older EC2 instance
types, which lack the Elastic Network Adapter hardware.

V.   Solution

Upgrade your system to a supported FreeBSD stable or release / security
branch (releng) dated after the correction date, and reboot.

Perform one of the following:

1) To update your system via a binary patch:

Systems running a RELEASE version of FreeBSD on the i386 or amd64
platforms can be updated via the freebsd-update(8) utility:

# freebsd-update fetch
# freebsd-update install
# shutdown -r +10min "Rebooting for errata update"

2) To update your system via a source code patch:

The following patches have been verified to apply to the applicable
FreeBSD release branches.

a) Download the relevant patch from the location below, and verify the
detached PGP signature using your PGP utility.

[FreeBSD 12.1]
# fetch https://security.FreeBSD.org/patches/EN-20:11/ena.patch
# fetch https://security.FreeBSD.org/patches/EN-20:11/ena.patch.asc
# gpg --verify ena.patch.asc

b) Apply the patch.  Execute the following commands as root:

# cd /usr/src
# patch < /path/to/patch

c) Recompile your kernel as described in
<URL:https://www.FreeBSD.org/handbook/kernelconfig.html> and reboot the
system.

NOTE: This patch can have a small (~2%) impact on the throughput of the
ena(4) device.  The original performance can be restored -- at the expense
of latency spikes and system instability under memory pressure -- by adding
the following to /boot/loader.conf:

        hw.ena.enable_9k_mbufs="1"

VI.  Correction details

The following list contains the correction revision numbers for each
affected branch.

Branch/path                                                      Revision
- -------------------------------------------------------------------------
stable/12/                                                        r360985
releng/12.1/                                                      r361970
- -------------------------------------------------------------------------

To see which files were modified by a particular revision, run the
following command, replacing NNNNNN with the revision number, on a
machine with Subversion installed:

# svn diff -cNNNNNN --summarize svn://svn.freebsd.org/base

Or visit the following URL, replacing NNNNNN with the revision number:

<URL:https://svnweb.freebsd.org/base?view=revision&revision=NNNNNN>

VII. References

The latest revision of this advisory is available at
<URL:https://security.FreeBSD.org/advisories/FreeBSD-EN-20:11.ena.asc>
-----BEGIN PGP SIGNATURE-----

iQKTBAEBCgB9FiEE/A6HiuWv54gCjWNV05eS9J6n5cIFAl7fuERfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEZD
MEU4NzhBRTVBRkU3ODgwMjhENjM1NUQzOTc5MkY0OUVBN0U1QzIACgkQ05eS9J6n
5cITew/+MQpJIKHrlWjqcR9fvAeqR4rX3aBVNMkambeTa2nKY4/PH8XzVLnKvyuE
O7ZtzaU3bz5xYkyYnlYSGstTRWk/BRCRgpZvzlPWVsvsFINgCHO5QsKHVyC0+/M2
Do3qRwxYDYPh0+Bz8QGuQGmpfMNpsuA//rfa12Ywp7jqu3IP8+0bBuWM71J7Bkyf
WhEJIehFiU9iGUFS2LF7000CzIwmcTitCtYJFxxwaYYHGHT5nMtljcrsKK9I1N+Q
f3mQ7dK8ryQ+H+CupfHodAYUeZyJbj/xVggdnzY45R1B9Tdr15VMPuurcqXyXIft
Uj4UzdDUNgbUw4WQqfvqn8CDAT0V54NRhDs8AkYWn+GqaDDSnvcsKnI/V2UJgqTn
Wp8ZClq7bF0mcBp7g1t97peseZsP+papJKsaVLKrTT+dw1x/aWz1cvK51x5UPWoW
qaiFVMf5l9R0ZDcQxsyN/7gyHnZRR8X1wl1gVPVcJ8EEWs6Gh4sRTzXSDMDqsA/P
89hA6vPVeOmBqKM3BlR8MrDPIALjzxNvzu65NMLMcebyPudaY9a2GEd8acw6mwxh
hQ015y9+MoMV8xD3gfjNLdQS7899eK1MwXu1UZPcUEOlYErAhjoGo51N4D6mfU62
7sDoy2Yt380Nmp57nUoanHkADzXPn8OV8fvlsTN6a0J4ZLu+KNM=
=FDu4
-----END PGP SIGNATURE-----