./Mellanox-rshim-user-space-0fd5d3f/0000775000175000017500000000000014564775540016637 5ustar tai271828tai271828./Mellanox-rshim-user-space-0fd5d3f/Makefile.am0000664000175000017500000000102314563673752020670 0ustar tai271828tai271828# SPDX-License-Identifier: GPL-2.0-only # Copyright (C) 2019 Mellanox Technologies. All Rights Reserved. # AUTOMAKE_OPTIONS = foreign SUBDIRS = src man dist_doc_DATA = README.md confdir = $(sysconfdir) dist_conf_DATA = etc/rshim.conf dist_sbin_SCRIPTS = scripts/bfb-install AM_DISTCHECK_CONFIGURE_FLAGS = \ --with-systemdsystemunitdir=$$dc_install_base/$(systemdsystemunitdir) if HAVE_SYSTEMD systemdsystemunit_DATA = rshim.service else if OS_FREEBSD init_ddir = $(sysconfdir)/rc.d/ init_d_SCRIPTS = freebsd/rshim endif endif ./Mellanox-rshim-user-space-0fd5d3f/etc/0000775000175000017500000000000014563673752017413 5ustar tai271828tai271828./Mellanox-rshim-user-space-0fd5d3f/etc/rshim.conf0000664000175000017500000000245414563673752021411 0ustar tai271828tai271828# # This is the rshim driver configuration file. # # # Display level for the 'misc' file # #DISPLAY_LEVEL 0 # # Timeout in seconds when pushing BFB while target side is not reading the # boot stream # #BOOT_TIMEOUT 150 # # Once set to 1, the driver will ignore all rshim writes and returns 0 for # rshim read. It could be used in certain cases, such as during FW_RESET or # bypassing the rshim PF to VM # #DROP_MODE 0 # # Delay in seconds for rshim over USB, which is added after chip reset and # before pushing the boot stream # #USB_RESET_DELAY 1 # # Delay in seconds for rshim over PCIe, which is added after chip reset and # before pushing the boot stream # #PCIE_RESET_DELAY 5 # # Interrupt polling interval in seconds when running rshim over direct memory # mapping # #PCIE_INTR_POLL_INTERVAL 10 # # Setting is to 0 will disallow rshim PCIe BAR mapping via VFIO # #PCIE_HAS_VFIO 1 # # Setting is to 0 will disallow rshim PCIe BAR mapping via UIO # #PCIE_HAS_UIO 1 # # Static mapping of rshim name and device. # Uncomment the 'rshim' line to configure the mapping. # # rshim-name device-name #rshim0 usb-2-1.7 #rshim1 pcie-0000:04:00.2 # # Ignored rshim devices. # Uncomment the 'none' line to configure the ignored devices. # #none usb-1-1.4 #none pcie-lf-0000:84:00.0 ./Mellanox-rshim-user-space-0fd5d3f/README.md0000664000175000017500000000603014563673752020116 0ustar tai271828tai271828 BlueField Rshim Host Driver The rshim driver provides a way to access the rshim resources on the BlueField target from external host machine. The current version implements device files for boot image push and virtual console access. It also creates virtual network interface to connect to the BlueField target and provides a way to access the internal rshim registers. *) Build Linux: Make sure autoconf/automake/pkg-config tools are available. Run bootstrap.sh for the first time to generate the configure file. Then run the ./configure script followed by make & make install to build and install it. FreeBSD: Require FreeBSD 12.0+ with packages autoconf, automake, gmake, libepoll-shim, libpciaccess, libpci, pkgconf. Follow the same steps above to build it. Use 'gmake install' to install it. *) Usage rshim -h syntax: rshim [--help|-h] [--backend|-b usb|pcie|pcie_lf] [--device|-d device-name] [--foreground|-f] [--debug-level|-l <0~4>] *) Device Files Each rshim target will create a directory /dev/rshim\/ with the following device files. \ is the device id, which could be 0, 1, etc. - /dev/rshim\/boot Boot device file used to push boot stream to the ARM side, for example, cat install.bfb > /dev/rshim/boot - /dev/rshim\/console Console device, which can be used by console apps to connect to the ARM side, such as screen /dev/rshim/console - /dev/rshim\/rshim Device file used to access the rshim registers. The read/write offset is encoded as "((rshim_channel << 16) | register_offset)". - /dev/rshim\/misc Key/Value pairs used to read/write misc information. For example, Display the content: cat /dev/rshim/misc DISPLAY_LEVEL 0 (0:basic, 1:advanced, 2:log) BOOT_MODE 1 (0:rshim, 1:emmc, 2:emmc-boot-swap) BOOT_TIMEOUT 100 (seconds) SW_RESET 0 (1: reset) DEV_NAME usb-3.3 Display more infomation: echo "DISPLAY_LEVEL 1" > cat /dev/rshim/misc cat /dev/rshim/misc ... PEER_MAC 00:1a:ca:ff:ff:01 # Target-side MAC address PXE_ID 0x01020304 # PXE DHCP-client-identifier The 'PEER_MAC' attribute can be used to display and set the target-side MAC address of the rshim network interface. It works when the target-side is in UEFI BootManager or in Linux where the tmfifo has been loaded. The new MAC address will take effect in next boot. Initiate a SW reset: echo "SW_RESET 1" > /dev/rshim/misc *) Multiple Boards Support Multiple boards could connect to the same host machine. Each of them has its own device directory /dev/rshim/. Network subnet needs to be set properly just like any other standard NIC. *) How to change the MAC address of the ARM side interface Update the 'PEER_MAC' attribute in the misc file like below. Display the value to confirm it's set. Reboot the device to take effect. echo "PEER_MAC 00:1a:ca:ff:ff:10" > /dev/rshim\/misc ./Mellanox-rshim-user-space-0fd5d3f/man/0000775000175000017500000000000014563673752017413 5ustar tai271828tai271828./Mellanox-rshim-user-space-0fd5d3f/man/Makefile.am0000664000175000017500000000021614563673752021446 0ustar tai271828tai271828# SPDX-License-Identifier: GPL-2.0-only # Copyright (C) 2019 Mellanox Technologies. All Rights Reserved. # man8_MANS = rshim.8 bfb-install.8 ./Mellanox-rshim-user-space-0fd5d3f/man/bfb-install.80000664000175000017500000000131014563673752021674 0ustar tai271828tai271828.\" Manpage for bfb-install. .TH man 8 "19 Nov 2020" "2.0" "bfb-install man page" .SH NAME bfb-install \- BFB installing script for BlueField SoC over rshim driver .SH SYNOPSIS rshim [options] .SH DESCRIPTION bfb-install is a utility script to install BFB images on BlueField SoC over the rshim driver. .SH OPTIONS -b, --bfb .in +4n This is the BFB image to use, which is pushed as the boot stream. .in -c, --config .in +4n This is an optional configuration file to use, usually called bf.cfg. .in -f, --rootfs .in +4n This is the optional rootfs tar.xz file which is uaually used when installing Yocto. .in -r, --rshim .in +4n This is the rshim device to use, which can be 'rshim' or '/dev/rshim'. .in ./Mellanox-rshim-user-space-0fd5d3f/man/rshim.80000664000175000017500000000736214563673752020636 0ustar tai271828tai271828.\" Manpage for rshim. .TH man 8 "18 Dec 2019" "2.0" "rshim man page" .SH NAME rshim \- user-space rshim driver for BlueField SoC .SH SYNOPSIS rshim [options] .SH DESCRIPTION rshim is the user-space rshim driver for BlueField SoC. It provides ways to access the rshim resources on the BlueField target via USB or PCIe from external host machine. The current version implements virtual console, virtual network interface, boot stream push, register access and some utility commands. This driver will probe the rshim backend and create directory /dev/rshim for each of them with the following device files. .SS /dev/rshim/boot Boot device file used to push boot stream to the target, for example, .in +4n .nf cat install.bfb > /dev/rshim/boot .SS /dev/rshim/console Console device file, which can be used by console tools to connect to the target, such as .in +4n .nf screen /dev/rshim/console .SS /dev/rshim/rshim Device file used to access rshim register space. When reading/writing to this file, the offset is encoded as "((rshim_channel << 16) | register_offset)". This file can be used by tools like openocd to do CoreSight debugging. .SS /dev/rshim/misc Key/Value pairs used to read/write misc information. For example Display the output .in +4n .nf cat /dev/rshim0/misc DISPLAY_LEVEL 0 (0:basic, 1:advanced, 2:log) BOOT_MODE 1 (0:rshim, 1:emmc, 2:emmc-boot-swap) BOOT_TIMEOUT 100 (seconds) SW_RESET 0 (1: reset) DEV_NAME pcie-04:00.2 DEV_INFO BlueField-1(Rev 0) .fi .in Initiate SW reset .in +4n .nf echo "SW_RESET 1" > /dev/rshim/misc .fi .in Enable the advanced options .in +4n .nf echo "DISPLAY_LEVEL 1" > /dev/rshim/misc cat /dev/rshim0/misc DISPLAY_LEVEL 1 (0:basic, 1:advanced, 2:log) BOOT_MODE 1 (0:rshim, 1:emmc, 2:emmc-boot-swap) BOOT_TIMEOUT 100 (seconds) SW_RESET 0 (1: reset) DEV_NAME pcie-04:00.2 DEV_INFO BlueField-1(Rev 0) PEER_MAC 00:1a:ca:ff:ff:01 (rw) PXE_ID 0x00000000 (rw) VLAN_ID 0 0 (rw) .fi .in .SH OPTIONS -b, --backend .in +4n Specify the backend to attach, which can be one of usb, pcie or pcie_lf. If not specified, the driver will scan all rshim backends unless the '-d' option is given with a device name specified. .in -d, --device .in +4n Specify the device name to attach with the format below. The backend driver can be deduced from the device name, thus the '-b' option is not needed. PCIe backend: pcie-:.. Example: pcie-04:00.2 Devices can be found with command 'lspci -n'. PCIe backend in livefish mode: pcie-lf-:.. Example: pcie-04:00.2 Devices can be found with command 'lspci -n'. USB backend: usb--xx.xx. Example: usb-2-1.7 Devices can be found under /sys/bus/usb/devices/. .in -f, --foreground .in +4n Run in forground. .in -i, --index .in +4n Specify the index to create device path /dev/rshim. It's also used to create network interface name tmfifo_net. This option is needed when multiple rshim instances are running. .in -l, --log-level .in +4n Log level (0:none, 1:error, 2:warning, 3:notice, 4:debug) Log messages will be printed to standard output when running in foreground, or in syslog when running as a daemon. .in -v, --version .in +4n Display version .in .SH CONFIGURATION FILE Rshim configuration file (/etc/rshim.conf) can be used to specify the static mapping between rshim devices and rshim names. It can also be used to ignore some rshim devices. Example: .in +4n # Map usb-2-1.7 to rshim0 .br rshim0 usb-2-1.7 # Map pcie-0000:04:00.2 to rshim1 .br rshim1 pcie-0000:04:00.2 # Ignore usb-1-1.4 .br none usb-1-1.4 .in ./Mellanox-rshim-user-space-0fd5d3f/rshim.service0000664000175000017500000000052014563673752021341 0ustar tai271828tai271828# SPDX-License-Identifier: GPL-2.0-only # Copyright 2019 Mellanox Technologies. All Rights Reserved. [Unit] Description=rshim driver for BlueField SoC Documentation=man:rshim(8) After=network.target [Service] Restart=always Type=forking ExecStart=-/usr/sbin/rshim $OPTIONS KillMode=control-group [Install] WantedBy=multi-user.target ./Mellanox-rshim-user-space-0fd5d3f/bootstrap.sh0000775000175000017500000000024614563673752021216 0ustar tai271828tai271828#!/bin/sh # SPDX-License-Identifier: GPL-2.0-only # Copyright (C) 2019 Mellanox Technologies. All Rights Reserved. # autoreconf -ivf || exit 1 rm -rf autom4te.cache ./Mellanox-rshim-user-space-0fd5d3f/configure.ac0000664000175000017500000000707714563673752021141 0ustar tai271828tai271828# SPDX-License-Identifier: GPL-2.0-only # Copyright (C) 2019 Mellanox Technologies. All Rights Reserved. # AC_INIT([rshim], [2.0.20]) AC_CONFIG_AUX_DIR(config) AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AC_LANG(C) AC_PROG_CC AM_PROG_CC_C_O AC_CONFIG_HEADERS([config.h]) AC_CONFIG_FILES([ Makefile rshim.spec src/Makefile man/Makefile rhel/rshim.spec freebsd/rshim ]) AC_CANONICAL_HOST PKG_PROG_PKG_CONFIG AC_ARG_WITH([systemdsystemunitdir], [AS_HELP_STRING([--with-systemdsystemunitdir=DIR], [Directory for systemd service files])],, [with_systemdsystemunitdir=auto]) AS_IF([test "x$with_systemdsystemunitdir" = "xyes" -o "x$with_systemdsystemunitdir" = "xauto"], [ def_systemdsystemunitdir=$($PKG_CONFIG --variable=systemdsystemunitdir systemd) AS_IF([test "x$def_systemdsystemunitdir" = "x"], [AS_IF([test "x$with_systemdsystemunitdir" = "xyes"], [AC_MSG_ERROR([systemd support requested but pkg-config unable to query systemd package])]) with_systemdsystemunitdir=no], [with_systemdsystemunitdir="$def_systemdsystemunitdir"])]) AS_IF([test "x$with_systemdsystemunitdir" != "xno"], [AC_SUBST([systemdsystemunitdir], [$with_systemdsystemunitdir])]) AM_CONDITIONAL([HAVE_SYSTEMD], [test "x$with_systemdsystemunitdir" != "xno"]) AC_CHECK_HEADERS_ONCE([syslog.h]) AC_ARG_ENABLE([usb], AS_HELP_STRING([--enable-usb], [Enable rshim over USB (default is yes) ]), [build_usb=$enableval], [build_usb=yes]) AM_CONDITIONAL([BUILD_RSHIM_USB], [test "x$build_usb" = "xyes"]) AC_ARG_ENABLE([pcie], AS_HELP_STRING([--enable-pcie], [Enable rshim over PCIe (default is yes) ]), [build_pcie=$enableval], [build_pcie=yes]) AM_CONDITIONAL([BUILD_RSHIM_PCIE], [test "x$build_pcie" = "xyes"]) AC_ARG_ENABLE([fuse], AS_HELP_STRING([--enable-fuse], [Enable fuse / cuse (default is yes) ]), [build_fuse=$enableval], [build_fuse=yes]) AM_CONDITIONAL([BUILD_RSHIM_FUSE], [test "x$build_fuse" = "xyes"]) case $host in *-linux*) AC_MSG_RESULT([Linux]) backend=linux ;; *-freebsd*) AC_MSG_RESULT([FreeBSD]) backend=freebsd ;; *) AC_MSG_ERROR([unsupported operating system $host]) esac AS_IF([test "x$build_pcie" = "xyes"], [ PKG_CHECK_MODULES(libpci, libpci, [], [AC_MSG_ERROR([Can't find libpci])]) ]) AS_IF([test "x$build_usb" = "xyes"], [ PKG_CHECK_MODULES(libusb, libusb-1.0 >= 1.0, [], [AC_MSG_ERROR([Can't find libusb-1.0])]) if test $backend = freebsd; then AC_CHECK_LIB(usb, libusb_init, [], [AC_MSG_ERROR([Missing libusb_init in libusb])]) else AC_CHECK_LIB(usb-1.0, libusb_init, [], [AC_MSG_ERROR([Missing libusb_init in libusb-1.0])]) fi AC_CHECK_FUNCS([libusb_get_port_numbers libusb_get_device_address]) ]) AS_IF([test "x$build_fuse" = "xyes"], [ if test $backend = freebsd; then AC_CHECK_LIB(cuse, cuse_dev_create) else PKG_CHECK_MODULES(fuse, fuse, [], PKG_CHECK_MODULES(fuse, fuse3, [use_fuse3=yes], [AC_MSG_ERROR([Can't find fuse])])) if test $use_fuse3 = yes; then AC_SUBST(CPPFLAGS, "$CPPFLAGS -DFUSE_USE_VERSION=30") else AC_SUBST(CPPFLAGS, "$CPPFLAGS -DFUSE_USE_VERSION=29") fi fi ]) if test $backend = freebsd; then AC_SUBST(CPPFLAGS, "$CPPFLAGS -I${prefix}/include/libepoll-shim") AC_SUBST(LDFLAGS, "$LDFLAGS -L${prefix}/lib") AC_CHECK_HEADERS([sys/epoll.h],[],[AC_MSG_ERROR([Missing libepoll-shim])]) AC_CHECK_LIB(epoll-shim, epoll_create1) AC_SUBST(CPPFLAGS, "$CPPFLAGS -DDEFAULT_RSHIM_CONFIG_FILE='\"${prefix}/etc/rshim.conf\"'") else AC_SUBST(CPPFLAGS, "$CPPFLAGS -DDEFAULT_RSHIM_CONFIG_FILE='\"/etc/rshim.conf\"'") fi AM_CONDITIONAL([OS_FREEBSD], [test "x$backend" = "xfreebsd"]) AC_CHECK_LIB(pthread, pthread_create) AC_OUTPUT ./Mellanox-rshim-user-space-0fd5d3f/LICENSE0000664000175000017500000000107514563673752017650 0ustar tai271828tai271828This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License version 2 as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/old-licenses/gpl-2.0.txt ./Mellanox-rshim-user-space-0fd5d3f/rhel/0000775000175000017500000000000014563673752017572 5ustar tai271828tai271828./Mellanox-rshim-user-space-0fd5d3f/rhel/rshim.spec.in0000664000175000017500000002162214563673752022200 0ustar tai271828tai271828# SPDX-License-Identifier: GPL-2.0-only # Copyright (C) 2019 Mellanox Technologies. All Rights Reserved. # Name: rshim Version: @VERSION@ Release: 0%{?dist} Summary: User-space driver for Mellanox BlueField SoC License: GPLv2 URL: https://github.com/mellanox/rshim-user-space Source0: https://github.com/Mellanox/rshim-user-space/releases/download/%{name}-%{version}/%{name}-%{version}.tar.gz BuildRequires: gcc, autoconf, automake, make BuildRequires: pkgconfig(libpci), pkgconfig(libusb-1.0), pkgconfig(fuse) BuildRequires: systemd BuildRequires: systemd-rpm-macros Requires: kmod(cuse.ko) Suggests: kernel-modules-extra %description This is the user-space driver to access the BlueField SoC via the rshim interface. It provides ways to push boot stream, debug the target or login via the virtual console or network interface. %prep %setup -q -n %{name}-%{version} %build ./bootstrap.sh %configure %make_build %install %make_install %post %systemd_post rshim.service %preun %systemd_preun rshim.service %postun %systemd_postun_with_restart rshim.service %files %license LICENSE %doc README.md %config(noreplace) %{_sysconfdir}/rshim.conf %{_sbindir}/rshim %{_unitdir}/rshim.service %{_mandir}/man8/rshim.8.gz %changelog * Fri Feb 16 2024 Liming Sun - 2.0.20 - rshim_pci: adjust delay time for nic_fw reset - bfb-install: Exit on "Linux up" * Wed Jan 10 2024 Liming Sun - 2.0.19 - Fix incorrect console message drop - Allow runtime debug code for DK cards * Thu Dec 14 2023 Liming Sun - 2.0.18 - Clear scratchpad1 register when setting drop_mode * Wed Nov 22 2023 Liming Sun - 2.0.17 - bfb-install: Fix duplicate output * Thu Nov 16 2023 Liming Sun - 2.0.16 - Remove fuse build dependency * Tue Nov 14 2023 Liming Sun - 2.0.15 - Add BFB completion condition for enhanced NIC mode * Fri Nov 10 2023 Liming Sun - 2.0.14 - Fix 9f19cfb4a75687ae * Wed Nov 08 2023 Liming Sun - 2.0.13 - Several robust fixes - Add fuse3 support * Mon Oct 23 2023 Liming Sun - 2.0.12 - BF3: Add UPTIME display in seconds * Tue Sep 26 2023 Liming Sun - 2.0.11 - Remove version 0 support for NIC FW_RESET - bfb-install: Return failure code * Mon Sep 18 2023 Liming Sun - 2.0.10 - Fix interrupt handling for NIC FW_RESET * Sat Jun 17 2023 Liming Sun - 2.0.9 - rshim/usb/bf3: fix timeout logic * Tue May 16 2023 Liming Sun - 2.0.8 - Fix the fall-back logic of direct-mapping * Thu Mar 30 2023 Liming Sun - 2.0.7 - Avoid opening /dev/uio multiple times - Update common files to dual-license - Adjust rshim reset delay * Sun Nov 20 2022 Liming Sun - 2.0.6-19 - BF3: Support 4B access for PCIe * Tue Oct 25 2022 Liming Sun - 2.0.6-18 - pcie: fix initialization issue when setting DROP_MODE in rshim.conf * Thu Oct 20 2022 Liming Sun - 2.0.6-17 - pcie: Avoid using cached pci_dev - rshim_fuse: display misc file even when rshim is not accessible * Thu Oct 06 2022 Liming Sun - 2.0.6-16 - pcie: Support mixed vfio and direct mapping mode * Thu Sep 29 2022 Liming Sun - 2.0.6-15 - Add dependency of libfuse2 for .deb - rshim-pcie: add a new bad-access code - Fix a potential NULL pointer access during USB disconnect - Adjust default boot timeout to 150s * Tue Aug 16 2022 Liming Sun - 2.0.6-14 - Avoid potential race when stopping the rshim process - Add configuration option to enable/disable PCIe VFIO/UIO - Fix warnings for compiling on 32-bit BMC - Mustang rshim usb supports for 4B and 8B transactions * Sun Jul 17 2022 Liming Sun - 2.0.6-13 - BF3: Support 32-bit CR-space access via USB - Avoid kernel-modules-extra dependency on ctyunos * Thu Jun 16 2022 Liming Sun - 2.0.6-12 - Optimize the rshim_work_fd - Detect new USB/rshim hot plugin * Mon May 16 2022 Liming Sun - 2.0.6-11 - Avoid kernel crash when unbind rshim from uio * Mon May 02 2022 Liming Sun - 2.0.6-10 - Fix several compiling issues for FreeBSD * Thu Apr 28 2022 Liming Sun - 2.0.6-9 - Use per-device memory-map mode * Mon Apr 18 2022 Liming Sun - 2.0.6-8 - Add interrupt polling for direct mmap() mode - Fix several coverity warnings * Thu Apr 07 2022 Liming Sun - 2.0.6-7 - Keep intr_fd during rshim_pcie disable/enable - Mustang: Add support for rshim over pcie and pcie_lf * Wed Mar 30 2022 Liming Sun - 2.0.6-6 - Clear scratchpad1 to 0 before PCI resources are unmapped - Fallback to UIO if VFIO failed * Fri Mar 18 2022 Liming Sun - 2.0.6-5 - PCIe: Add UIO and IRQ support - PCIe: Remove 32-bit support * Mon Feb 28 2022 Liming Sun - 2.0.6-4 - VFIO support - Fix potential race in rshim_work_signal * Mon Nov 29 2021 Liming Sun - 2.0.6-3 - Adjust the defaul value of usb_reset_delay to 5 - Add a delay after USB probe - Make the reset delay configurable * Wed Nov 03 2021 Liming Sun - 2.0.6-2 - bfb-install: Handle new indications for installation completion - Clean up some un-needed register definition - Fix MTU of the tmfifo_net0 interface on FreeBSD - Several fixes to prevent hypervisor crash - Refine some BF-2 Rev0 workaround condition * Wed May 12 2021 Liming Sun - 2.0.6-1 - Disable the background timer if no rshim devices - Setting default path for rshim config file * Wed Mar 10 2021 Liming Sun - 2.0.5-10 - PCIe hotplug support - Reduce CPU utilization when there is no rshim device * Wed Jan 27 2021 Liming Sun - 2.0.5-9 - Fix potential tmfifo data loss - Add workaround checking for Bluefield-2 REV-0 - Fix network traffic stop issue when Tx buffer full * Fri Dec 11 2020 Liming Sun - 2.0.5-8 - Don't allow any register access when DROP_MODE is set - Avoid potential race in rshim_fifo_read * Wed Dec 09 2020 Liming Sun - 2.0.5-7 - Fix potential dead-lock when calling rshim_access_check - Ignore rshim access checking when global drop mode is enabled - Fix some secure boot related issue * Wed Dec 02 2020 Liming Sun - 2.0.5-6 - Add some default configuration in rshim.conf - Change the debug level of Rshim byte access widget timeout - Add bfb-install script * Thu Oct 29 2020 Liming Sun - 2.0.5-5 - Check rshim accessibility when re-enabling it - Enable console output during boot stream pushing - Add some delay for the pcie_lf probe - Auto-start rshim service after installation * Fri Sep 25 2020 Liming Sun - 2.0.5-4 - Some robust fixes for USB rshim - Fix a typo in pcie mmap * Mon Aug 17 2020 Liming Sun - 2.0.5-3 - Fix several coverity warnings - Add workaround to boot Viper rev A0 in LiveFish mode - Display/configure OPN string for BlueField-2 * Fri Jul 24 2020 Liming Sun - 2.0.5-2 - Add configuration file support - misc: Display device version / revision ID - Add service file for FreeBSD * Tue Jun 16 2020 Liming Sun - 2.0.5-1 - Improve response time to ctrl+c for boot stream - Fix a rpmbuild issue when make_build is not defined - Add DROP_MODE configuration in misc file - Avoid reading the fifo if still booting - Fix configure issue for FreeBSD 12.1-RELEASE - Add domain id to the DEV_NAME in the misc file - Fix the debian copyright format - Enhance rshim_pcie_enable function * Tue Apr 21 2020 Liming Sun - 2.0.4-1 - Update .spec file according to review comments - Fix the 'KillMode' in rshim.service - Support process termination by SIGTERM - Fix some compiling warnings and configure issue for FreeBSD - Fix a read()/write() issue in rshim_pcie.c caused by optimization * Tue Apr 14 2020 Liming Sun - 2.0.3-1 - Enable pci device during probing - Map the pci resource0 file instead of /dev/mem - Add copyright header in bootstrap.sh - Add 'Requires' tag check in the rpm .spec for kernel-modules-extra - Fix the 'rshim --version' output * Thu Apr 09 2020 Liming Sun - 2.0.2-1 - Remove unnecessary dependency in .spec and use make_build - Add package build for debian/ubuntu - Fix some format in the man page - Add check for syslog headers * Mon Mar 23 2020 Liming Sun - 2.0.1-1 - Rename bfrshim to rshim - Remove rshim.spec since it's auto-generated from rshim.spec.in - Fix warnings reported by coverity - Add rhel/rshim.spec.in for fedora - Move rshim to sbin and move man page to man8 * Fri Mar 13 2020 Liming Sun - 2.0-1 - Update the spec file according to fedora packaging-guidelines * Mon Dec 16 2019 Liming Sun - Initial packaging ./Mellanox-rshim-user-space-0fd5d3f/scripts/0000775000175000017500000000000014563673752020327 5ustar tai271828tai271828./Mellanox-rshim-user-space-0fd5d3f/scripts/bfb-install0000775000175000017500000001050414563673752022452 0ustar tai271828tai271828#!/bin/sh # Copyright (c) 2020, NVIDIA Corporation # All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are met: # # 1. Redistributions of source code must retain the above copyright notice, this # list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright notice, # this list of conditions and the following disclaimer in the documentation # and/or other materials provided with the distribution. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED # WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR # ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES # (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; # LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND # ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # # The views and conclusions contained in the software and documentation are those # of the authors and should not be interpreted as representing official policies, # either expressed or implied, of the FreeBSD Project. usage () { echo "syntax: bfb-install --bfb|-b [--config|-c ] \\" echo " [--rootfs|-f ] --rshim|-r [--help|-h]" } bfb= cfg= rootfs= rshim= options=`getopt -n bfb-install -o b:c:f:r:h \ -l help,bfb:,config:,rootfs:,rshim: -- "$@"` eval set -- $options while [ "$1" != -- ]; do case $1 in --help|-h) usage; exit 0 ;; --bfb|-b) shift; bfb=$1 ;; --config|-c) shift; cfg=$1 ;; --rootfs|-f) shift; rootfs=$1 ;; --rshim|-r) shift; rshim=$1 ;; esac shift done shift if [ $# -ne 0 ]; then usage >&2 exit 1 fi if [ -z "${bfb}" -o -z "${rshim}" ]; then echo "Error: Need to provide both bfb file and rshim device name." usage >&2 exit 1 fi if [ ! -e "${bfb}" ]; then echo "Error: ${bfb} not found." exit 1 fi if [ ."$(echo "${rshim}" | cut -c1-1)" != ."/" ]; then rshim="/dev/${rshim}" fi if [ ! -e "${rshim}/boot" ]; then echo "Error: ${rshim}/boot not found." exit 1 fi if [ -n "${rootfs}" -a ! -e "${rootfs}" ]; then echo "Error: ${rootfs} not found." exit 1 fi if [ -n "${cfg}" -a ! -e "${cfg}" ]; then echo "Error: ${cfg} not found." exit 1 fi if [ $(id -u) -ne 0 ]; then echo "Error: Need root permission to push BFB on local host." exit 1 fi pv=$(which pv 2>/dev/null) if [ -z "${pv}" ]; then echo "Warn: 'pv' command not found. Continue without showing BFB progress." fi # Push the boot stream. echo "Pushing bfb${cfg:+ + cfg}${rootfs:+ + rootfs}" sh -c "cat ${bfb} ${cfg:+$cfg} ${rootfs:+${rootfs}} ${pv:+| ${pv} | cat -} > ${rshim}/boot" RETVAL=$? if [ $RETVAL -ne 0 ]; then echo "Failed to push BFB" exit $RETVAL fi # Print the rshim log. echo "Collecting BlueField booting status. Press Ctrl+C to stop…" last="" finished=0 while [ $finished -eq 0 ]; do last_len=${#last} cur=$(echo 'DISPLAY_LEVEL 2' > ${rshim}/misc && cat ${rshim}/misc | sed -n '/^ INFO/,$p') RETVAL=$? if [ $RETVAL -ne 0 ]; then echo "Failed to read ${rshim}/misc" exit $RETVAL fi cur_len=${#cur} sleep 1 if echo ${cur} | grep -Ei "Linux up|Reboot|finished|DPU is ready|In Enhanced NIC mode" >/dev/null; then finished=1 fi # Overwrite if current length smaller than previous length. if [ ${last_len} -eq 0 -o ${last_len} -gt ${cur_len} ]; then echo "${cur}" | sed '/^[[:space:]]*$/d' last="${cur}" continue fi # Overwrite if first portion doesn't match. sub_cur=$(echo "${cur}" | dd bs=1 count=${last_len} 2>/dev/null) if [ "${sub_cur}" != "${last}" ]; then echo "${cur}" | sed '/^[[:space:]]*$/d' last="${cur}" continue fi # Nothing if no update. if [ ${last_len} -eq ${cur_len} ]; then [ $finished -eq 0 ] && continue; fi # Print the diff. echo "${cur}" | dd bs=1 skip=${last_len} 2>/dev/null | sed '/^[[:space:]]*$/d' last="${cur}" done exit 0 ./Mellanox-rshim-user-space-0fd5d3f/src/0000775000175000017500000000000014563673752017427 5ustar tai271828tai271828./Mellanox-rshim-user-space-0fd5d3f/src/Makefile.am0000664000175000017500000000125714563673752021470 0ustar tai271828tai271828# SPDX-License-Identifier: GPL-2.0-only # Copyright (C) 2019 Mellanox Technologies. All Rights Reserved. # sbin_PROGRAMS = rshim rshim_SOURCES = rshim.c rshim_log.c rshim_net.c rshim_regs.c rshim_CPPFLAGS = -Wall -DHAVE_RSHIM_NET # USB (library is already added by AC_CHECK_LIB) if BUILD_RSHIM_USB rshim_SOURCES += rshim_usb.c rshim_CPPFLAGS += $(libusb_CFLAGS) -DHAVE_RSHIM_USB endif # PCIe if BUILD_RSHIM_PCIE rshim_SOURCES += rshim_pcie.c rshim_pcie_lf.c rshim_CPPFLAGS += $(libpci_CFLAGS) -DHAVE_RSHIM_PCIE LIBS += $(libpci_LIBS) endif # FUSE / CUSE if BUILD_RSHIM_FUSE rshim_SOURCES += rshim_fuse.c rshim_CPPFLAGS += $(fuse_CFLAGS) -DHAVE_RSHIM_FUSE LIBS += $(fuse_LIBS) endif ./Mellanox-rshim-user-space-0fd5d3f/src/rshim_log.c0000664000175000017500000002646614563673752021574 0ustar tai271828tai271828// SPDX-License-Identifier: GPL-2.0 or BSD-3-Clause /* * Copyright (C) 2019-2023 Mellanox Technologies. All Rights Reserved. * */ #include #include #include #include "rshim.h" /* Log module */ const char * const rshim_log_mod[] = { "MISC", "BL1", "BL2", "BL2R", "BL31", "UEFI", "PSC" }; /* Log level */ const char * const rshim_log_levels[] = { "INFO", "WARN", "ERR", "ASSERT" }; /* Log type. */ #define BF_RSH_LOG_TYPE_UNKNOWN 0x00ULL #define BF_RSH_LOG_TYPE_PANIC 0x01ULL #define BF_RSH_LOG_TYPE_EXCEPTION 0x02ULL #define BF_RSH_LOG_TYPE_UNUSED 0x03ULL #define BF_RSH_LOG_TYPE_MSG 0x04ULL /* Utility macro. */ #define BF_RSH_LOG_MOD_MASK 0x0FULL #define BF_RSH_LOG_MOD_SHIFT 60 #define BF_RSH_LOG_TYPE_MASK 0x0FULL #define BF_RSH_LOG_TYPE_SHIFT 56 #define BF_RSH_LOG_LEN_MASK 0x7FULL #define BF_RSH_LOG_LEN_SHIFT 48 #define BF_RSH_LOG_ARG_MASK 0xFFFFFFFFULL #define BF_RSH_LOG_ARG_SHIFT 16 #define BF_RSH_LOG_HAS_ARG_MASK 0xFFULL #define BF_RSH_LOG_HAS_ARG_SHIFT 8 #define BF_RSH_LOG_LEVEL_MASK 0xFFULL #define BF_RSH_LOG_LEVEL_SHIFT 0 #define BF_RSH_LOG_PC_MASK 0xFFFFFFFFULL #define BF_RSH_LOG_PC_SHIFT 0 #define BF_RSH_LOG_SYNDROME_MASK 0xFFFFFFFFULL #define BF_RSH_LOG_SYNDROME_SHIFT 0 #define BF_RSH_LOG_HEADER_GET(f, h) \ (((h) >> BF_RSH_LOG_##f##_SHIFT) & BF_RSH_LOG_##f##_MASK) #define AARCH64_MRS_REG_SHIFT 5 #define AARCH64_MRS_REG_MASK 0xffff typedef struct { char *name; uint32_t opcode; } rshim_log_reg_t; static rshim_log_reg_t rshim_log_regs[] = { {"actlr_el1", 0b1100000010000001}, {"actlr_el2", 0b1110000010000001}, {"actlr_el3", 0b1111000010000001}, {"afsr0_el1", 0b1100001010001000}, {"afsr0_el2", 0b1110001010001000}, {"afsr0_el3", 0b1111001010001000}, {"afsr1_el1", 0b1100001010001001}, {"afsr1_el2", 0b1110001010001001}, {"afsr1_el3", 0b1111001010001001}, {"amair_el1", 0b1100010100011000}, {"amair_el2", 0b1110010100011000}, {"amair_el3", 0b1111010100011000}, {"ccsidr_el1", 0b1100100000000000}, {"clidr_el1", 0b1100100000000001}, {"cntkctl_el1", 0b1100011100001000}, {"cntp_ctl_el0", 0b1101111100010001}, {"cntp_cval_el0", 0b1101111100010010}, {"cntv_ctl_el0", 0b1101111100011001}, {"cntv_cval_el0", 0b1101111100011010}, {"contextidr_el1", 0b1100011010000001}, {"cpacr_el1", 0b1100000010000010}, {"cptr_el2", 0b1110000010001010}, {"cptr_el3", 0b1111000010001010}, {"vtcr_el2", 0b1110000100001010}, {"ctr_el0", 0b1101100000000001}, {"currentel", 0b1100001000010010}, {"dacr32_el2", 0b1110000110000000}, {"daif", 0b1101101000010001}, {"dczid_el0", 0b1101100000000111}, {"dlr_el0", 0b1101101000101001}, {"dspsr_el0", 0b1101101000101000}, {"elr_el1", 0b1100001000000001}, {"elr_el2", 0b1110001000000001}, {"elr_el3", 0b1111001000000001}, {"esr_el1", 0b1100001010010000}, {"esr_el2", 0b1110001010010000}, {"esr_el3", 0b1111001010010000}, {"esselr_el1", 0b1101000000000000}, {"far_el1", 0b1100001100000000}, {"far_el2", 0b1110001100000000}, {"far_el3", 0b1111001100000000}, {"fpcr", 0b1101101000100000}, {"fpexc32_el2", 0b1110001010011000}, {"fpsr", 0b1101101000100001}, {"hacr_el2", 0b1110000010001111}, {"har_el2", 0b1110000010001000}, {"hpfar_el2", 0b1110001100000100}, {"hstr_el2", 0b1110000010001011}, {"far_el1", 0b1100001100000000}, {"far_el2", 0b1110001100000000}, {"far_el3", 0b1111001100000000}, {"hcr_el2", 0b1110000010001000}, {"hpfar_el2", 0b1110001100000100}, {"id_aa64afr0_el1", 0b1100000000101100}, {"id_aa64afr1_el1", 0b1100000000101101}, {"id_aa64dfr0_el1", 0b1100000000101100}, {"id_aa64isar0_el1", 0b1100000000110000}, {"id_aa64isar1_el1", 0b1100000000110001}, {"id_aa64mmfr0_el1", 0b1100000000111000}, {"id_aa64mmfr1_el1", 0b1100000000111001}, {"id_aa64pfr0_el1", 0b1100000000100000}, {"id_aa64pfr1_el1", 0b1100000000100001}, {"ifsr32_el2", 0b1110001010000001}, {"isr_el1", 0b1100011000001000}, {"mair_el1", 0b1100010100010000}, {"mair_el2", 0b1110010100010000}, {"mair_el3", 0b1111010100010000}, {"midr_el1", 0b1100000000000000}, {"mpidr_el1", 0b1100000000000101}, {"nzcv", 0b1101101000010000}, {"revidr_el1", 0b1100000000000110}, {"rmr_el3", 0b1111011000000010}, {"par_el1", 0b1100001110100000}, {"rvbar_el3", 0b1111011000000001}, {"scr_el3", 0b1111000010001000}, {"sctlr_el1", 0b1100000010000000}, {"sctlr_el2", 0b1110000010000000}, {"sctlr_el3", 0b1111000010000000}, {"sp_el0", 0b1100001000001000}, {"sp_el1", 0b1110001000001000}, {"spsel", 0b1100001000010000}, {"spsr_abt", 0b1110001000011001}, {"spsr_el1", 0b1100001000000000}, {"spsr_el2", 0b1110001000000000}, {"spsr_el3", 0b1111001000000000}, {"spsr_fiq", 0b1110001000011011}, {"spsr_irq", 0b1110001000011000}, {"spsr_und", 0b1110001000011010}, {"tcr_el1", 0b1100000100000010}, {"tcr_el2", 0b1110000100000010}, {"tcr_el3", 0b1111000100000010}, {"tpidr_el0", 0b1101111010000010}, {"tpidr_el1", 0b1100011010000100}, {"tpidr_el2", 0b1110011010000010}, {"tpidr_el3", 0b1111011010000010}, {"tpidpro_el0", 0b1101111010000011}, {"vbar_el1", 0b1100011000000000}, {"vbar_el2", 0b1110011000000000}, {"vbar_el3", 0b1111011000000000}, {"vmpidr_el2", 0b1110000000000101}, {"vpidr_el2", 0b1110000000000000}, {"ttbr0_el1", 0b1100000100000000}, {"ttbr0_el2", 0b1110000100000000}, {"ttbr0_el3", 0b1111000100000000}, {"ttbr1_el1", 0b1100000100000001}, {"vtcr_el2", 0b1110000100001010}, {"vttbr_el2", 0b1110000100001000}, {NULL, 0b0000000000000000}, }; static char *rshim_log_get_reg_name(uint64_t opcode) { rshim_log_reg_t *reg = rshim_log_regs; while (reg->name) { if (reg->opcode == opcode) return reg->name; reg++; } return "unknown"; } static int rshim_log_show_crash(rshim_backend_t *bd, uint64_t hdr, char *buf, int size) { int rc = 0, i, module, type, len, n = 0; uint64_t opcode, data; char *p = buf; uint32_t pc, syndrome, ec; module = BF_RSH_LOG_HEADER_GET(MOD, hdr); if (module >= sizeof(rshim_log_mod) / sizeof(rshim_log_mod[0])) module = 0; type = BF_RSH_LOG_HEADER_GET(TYPE, hdr); len = BF_RSH_LOG_HEADER_GET(LEN, hdr); if (type == BF_RSH_LOG_TYPE_EXCEPTION) { syndrome = BF_RSH_LOG_HEADER_GET(SYNDROME, hdr); ec = syndrome >> 26; n = snprintf(p, size, " Exception(%s): syndrome = 0x%x%s\n", rshim_log_mod[module], syndrome, (ec == 0x24 || ec == 0x25) ? "(Data Abort)" : (ec == 0x2f) ? "(SError)" : ""); } else if (type == BF_RSH_LOG_TYPE_PANIC) { pc = BF_RSH_LOG_HEADER_GET(PC, hdr); n = snprintf(p, size, " PANIC(%s): PC = 0x%x\n", rshim_log_mod[module], pc); } p += n; size -= n; for (i = 0; i < len/2; i++) { rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->scratch_buf_dat, &opcode, RSHIM_REG_SIZE_8B); if (rc) break; rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->scratch_buf_dat, &data, RSHIM_REG_SIZE_8B); if (rc) break; opcode = (le64toh(opcode) >> AARCH64_MRS_REG_SHIFT) & AARCH64_MRS_REG_MASK; n = snprintf(p, size, " %-16s0x%llx\n", rshim_log_get_reg_name(opcode), (unsigned long long)data); p += n; size -= n; } return p - buf; } static int rshim_log_format_msg(char *buf, int len, const char* msg, ...) { va_list args; va_start(args, msg); len = vsnprintf(buf, len, msg, args); va_end(args); return len; } static int rshim_log_show_msg(rshim_backend_t *bd, uint64_t hdr, char *buf, int size) { int rc; int module = BF_RSH_LOG_HEADER_GET(MOD, hdr); int len = BF_RSH_LOG_HEADER_GET(LEN, hdr); int level = BF_RSH_LOG_HEADER_GET(LEVEL, hdr); int has_arg = BF_RSH_LOG_HEADER_GET(HAS_ARG, hdr); uint32_t arg = BF_RSH_LOG_HEADER_GET(ARG, hdr); uint64_t data; char *msg, *p; if (len <= 0) return -EINVAL; if (module >= sizeof(rshim_log_mod) / sizeof(rshim_log_mod[0])) module = 0; if (level >= sizeof(rshim_log_levels) / sizeof(rshim_log_levels[0])) level = 0; msg = malloc(len * sizeof(uint64_t) + 1); if (!msg) return 0; p = msg; while (len--) { rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->scratch_buf_dat, &data, RSHIM_REG_SIZE_8B); if (rc) { free(msg); return 0; } memcpy(p, &data, sizeof(data)); p += sizeof(data); } *p = '\0'; if (!has_arg) { len = snprintf(buf, size, " %s[%s]: %s\n", rshim_log_levels[level], rshim_log_mod[module], msg); } else { len = snprintf(buf, size, " %s[%s]: ", rshim_log_levels[level], rshim_log_mod[module]); // coverity[ +tainted_string_sanitize_content : arg-2 ] len += rshim_log_format_msg(buf + len, size - len, msg, arg); len += snprintf(buf + len, size - len, "\n"); } free(msg); return len; } int rshim_log_show(rshim_backend_t *bd, char *buf, int size) { uint64_t data, idx, hdr; time_t t0, t1; int i, n, rc, type, len; char *p = buf; n = snprintf(p, size, "---------------------------------------\n"); p += n; size -= n; n = snprintf(p, size, " Log Messages\n"); p += n; size -= n; n = snprintf(p, size, "---------------------------------------\n"); p += n; size -= n; /* Take the semaphore. */ time(&t0); while (true) { rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->semaphore0, &data, RSHIM_REG_SIZE_8B); if (rc) { RSHIM_ERR("couldn't read RSH_SEMAPHORE0\n"); return p - buf; } if (!data) break; /* Add a timeout in case the semaphore is stuck. */ time(&t1); if (difftime(t1, t0) > 1) break; } /* Read the current index. */ rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->scratch_buf_ctl, &idx, RSHIM_REG_SIZE_8B); if (rc) { RSHIM_ERR("couldn't read RSH_SCRATCH_BUF_CTL\n"); goto done; } idx = (idx >> RSH_SCRATCH_BUF_CTL__IDX_SHIFT) & RSH_SCRATCH_BUF_CTL__IDX_MASK; if (idx <= 1) goto done; /* Reset the index to 0. */ rc = bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->scratch_buf_ctl, 0, RSHIM_REG_SIZE_8B); if (rc) { RSHIM_ERR("couldn't write RSH_SCRATCH_BUF_CTL\n"); goto done; } i = 0; while (i < idx) { rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->scratch_buf_dat, &hdr, RSHIM_REG_SIZE_8B); if (rc) { RSHIM_ERR("couldn't read RSH_SCRATCH_BUF_DAT\n"); goto done; } hdr = le64toh(hdr); type = BF_RSH_LOG_HEADER_GET(TYPE, hdr); len = BF_RSH_LOG_HEADER_GET(LEN, hdr); i += 1 + len; /* Ignore if wraparounded. */ if (i > idx) break; switch (type) { case BF_RSH_LOG_TYPE_PANIC: case BF_RSH_LOG_TYPE_EXCEPTION: n = rshim_log_show_crash(bd, hdr, p, size); p += n; size -= n; break; case BF_RSH_LOG_TYPE_MSG: n = rshim_log_show_msg(bd, hdr, p, size); p += n; size -= n; break; default: /* Drain this message. */ while (len--) bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->scratch_buf_dat, &data, RSHIM_REG_SIZE_8B); break; } } /* Restore the idx value. */ bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->scratch_buf_ctl, idx, RSHIM_REG_SIZE_8B); done: /* Release the semaphore. */ bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->semaphore0, 0, RSHIM_REG_SIZE_8B); return p - buf; } ./Mellanox-rshim-user-space-0fd5d3f/src/rshim_fuse.c0000664000175000017500000010460514563673752021745 0ustar tai271828tai271828// SPDX-License-Identifier: GPL-2.0-only /* * Copyright (C) 2019 Mellanox Technologies. All Rights Reserved. * */ #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef __linux__ #if FUSE_USE_VERSION >= 30 #include #include #else #include #include #endif #include #elif defined(__FreeBSD__) #include #include #include #include #else #error "Unsupport OS for fuse" #endif #include "rshim.h" /* Name of the sub-device types. */ char *rshim_dev_minor_names[RSH_DEV_TYPES] = { [RSH_DEV_TYPE_RSHIM] = "rshim", [RSH_DEV_TYPE_BOOT] = "boot", [RSH_DEV_TYPE_TMFIFO] = "console", [RSH_DEV_TYPE_MISC] = "misc", }; /* BF3 ref clock. */ #define BF3_REF_CLK_IN_HZ 200000000 #ifdef __linux__ static void rshim_fuse_boot_open(fuse_req_t req, struct fuse_file_info *fi) { rshim_backend_t *bd = fuse_req_userdata(req); int rc = -ENODEV; if (bd) rc = rshim_boot_open(bd); if (rc) fuse_reply_err(req, -rc); else fuse_reply_open(req, fi); } #elif defined(__FreeBSD__) static int rshim_fuse_boot_open(struct cuse_dev *cdev, int fflags) { rshim_backend_t *bd = cuse_dev_get_priv0(cdev); int rc; rc = rshim_boot_open(bd); switch (rc) { case 0: return CUSE_ERR_NONE; case -EBUSY: return CUSE_ERR_BUSY; case -ENODEV: return CUSE_ERR_INVALID; default: return CUSE_ERR_OTHER; } } #endif #ifdef __linux__ static int rshim_fuse_copy_in(void *dest, const void *src, int count) { memcpy(dest, src, count); return 0; } static void rshim_fuse_boot_write(fuse_req_t req, const char *user_buffer, size_t count, off_t off, struct fuse_file_info *fi) { rshim_backend_t *bd = fuse_req_userdata(req); int rc = -ENODEV; if (bd) rc = rshim_boot_write(bd, user_buffer, count, rshim_fuse_copy_in); if (rc >= 0) fuse_reply_write(req, rc); else fuse_reply_err(req, -rc); } #elif defined(__FreeBSD__) static int rshim_fuse_copy_in(void *dest, const void *src, int count) { return cuse_copy_in(src, dest, count); } static int rshim_fuse_boot_write(struct cuse_dev *cdev, int fflags, const void *user_buffer, int count) { rshim_backend_t *bd = cuse_dev_get_priv0(cdev); int rc; rc = rshim_boot_write(bd, user_buffer, count, rshim_fuse_copy_in); switch (rc) { case 0: return CUSE_ERR_NONE; case -EBUSY: return CUSE_ERR_BUSY; case -ENODEV: return CUSE_ERR_INVALID; default: return CUSE_ERR_OTHER; } } #endif #ifdef __linux__ static void rshim_fuse_boot_release(fuse_req_t req, struct fuse_file_info *fi) { rshim_backend_t *bd = fuse_req_userdata(req); if (bd) rshim_boot_release(bd); fuse_reply_err(req, 0); } #elif defined(__FreeBSD__) static int rshim_fuse_boot_release(struct cuse_dev *cdev, int fflags) { rshim_backend_t *bd = cuse_dev_get_priv0(cdev); rshim_boot_release(bd); return CUSE_ERR_NONE; } #endif #ifdef __linux__ static const struct cuse_lowlevel_ops rshim_boot_fops = { .open = rshim_fuse_boot_open, .write = rshim_fuse_boot_write, .release = rshim_fuse_boot_release, }; #elif defined(__FreeBSD__) static const struct cuse_methods rshim_boot_fops = { .cm_open = rshim_fuse_boot_open, .cm_write = rshim_fuse_boot_write, .cm_close = rshim_fuse_boot_release, }; #endif void rshim_fuse_input_notify(rshim_backend_t *bd) { int chan = bd->rx_chan; RSHIM_DBG("rshim_fifo_input: woke up readable chan %d\n", chan); #ifdef __linux__ if (bd->fuse_poll_handle[chan]) fuse_lowlevel_notify_poll(bd->fuse_poll_handle[chan]); #elif defined(__FreeBSD__) cuse_poll_wakeup(); #endif } /* Console operations */ #ifdef __linux__ static void rshim_fuse_console_open(fuse_req_t req, struct fuse_file_info *fi) { rshim_backend_t *bd = fuse_req_userdata(req); int rc = -ENODEV; if (bd) rc = rshim_console_open(bd); if (!rc) fuse_reply_open(req, fi); else fuse_reply_err(req, -rc); } #elif defined(__FreeBSD__) static int rshim_fuse_console_open(struct cuse_dev *cdev, int fflags) { rshim_backend_t *bd = cuse_dev_get_priv0(cdev); int rc; rc = rshim_console_open(bd); switch (rc) { case CUSE_ERR_NONE: return CUSE_ERR_NONE; case -EBUSY: return CUSE_ERR_BUSY; default: return CUSE_ERR_OTHER; } } #endif #ifdef __linux__ static void rshim_fuse_console_read(fuse_req_t req, size_t size, off_t off, struct fuse_file_info *fi) { rshim_backend_t *bd = fuse_req_userdata(req); char buf[512]; int rc; if (!bd) { fuse_reply_err(req, ENODEV); return; } if (off) { fuse_reply_err(req, EINVAL); return; } if (size > sizeof(buf)) size = sizeof(buf); rc = rshim_fifo_read(bd, buf, size, TMFIFO_CONS_CHAN, fi->flags & O_NONBLOCK); if (rc < 0) fuse_reply_err(req, -rc); else fuse_reply_buf(req, buf, rc); } #elif defined(__FreeBSD__) static int rshim_fuse_console_read(struct cuse_dev *cdev, int fflags, void *peer_ptr, int size) { rshim_backend_t *bd = cuse_dev_get_priv0(cdev); bool nonblock = fflags & CUSE_FFLAG_NONBLOCK; char buf[4096]; int len = 0; int delta; int err; int rc; while (size > 0) { delta = sizeof(buf); if (delta > size) delta = size; err = rshim_fifo_read(bd, buf, delta, TMFIFO_CONS_CHAN, nonblock); if (err < 0) { if (err == -EAGAIN) { if (len != 0) return len; else return CUSE_ERR_WOULDBLOCK; } else if (err == -EINTR) { if (len != 0) return len; else return CUSE_ERR_SIGNAL; } else { return CUSE_ERR_OTHER; } } rc = cuse_copy_out(buf, peer_ptr, err); if (rc != CUSE_ERR_NONE) return rc; size -= err; peer_ptr = (char *)peer_ptr + err; len += err; /* return on short read */ if (err != delta) break; } return len; } #endif #ifdef __linux__ static void rshim_fuse_console_write(fuse_req_t req, const char *buf, size_t size, off_t off, struct fuse_file_info *fi) { rshim_backend_t *bd = fuse_req_userdata(req); int rc; if (!bd) { fuse_reply_err(req, ENODEV); return; } if (off) { fuse_reply_err(req, EINVAL); return; } rc = rshim_fifo_write(bd, buf, size, TMFIFO_CONS_CHAN, fi->flags & O_NONBLOCK); if (rc >= 0) fuse_reply_write(req, rc); else fuse_reply_err(req, -rc); } #elif defined(__FreeBSD__) static int rshim_fuse_console_write(struct cuse_dev *cdev, int fflags, const void *peer_ptr, int size) { rshim_backend_t *bd = cuse_dev_get_priv0(cdev); char buf[4096]; bool nonblock = fflags & CUSE_FFLAG_NONBLOCK; int len = 0; int rc; int err; while (size > 0) { rc = sizeof(buf); if (rc > size) rc = size; err = cuse_copy_in(peer_ptr, buf, rc); if (err != CUSE_ERR_NONE) return err; rc = rshim_fifo_write(bd, buf, rc, TMFIFO_CONS_CHAN, nonblock); if (rc < 0) { if (rc == -EAGAIN) { if (len != 0) return len; else return CUSE_ERR_WOULDBLOCK; } else if (rc == -EINTR) { if (len != 0) return len; else return CUSE_ERR_SIGNAL; } else { return CUSE_ERR_OTHER; } } size -= rc; peer_ptr = (char *)peer_ptr + rc; len += rc; } return len; } #endif #ifdef __linux__ static void rshim_fuse_console_fsync(fuse_req_t req, int datasync, struct fuse_file_info *fi) { rshim_backend_t *bd = fuse_req_userdata(req); int rc = -ENODEV; if (bd) rc = rshim_fifo_fsync(bd, TMFIFO_CONS_CHAN); fuse_reply_err(req, -rc); } static void rshim_fuse_console_ioctl(fuse_req_t req, int cmd, void *arg, struct fuse_file_info *fi, unsigned int flags, const void *in_buf, size_t in_bufsz, size_t out_bufsz) { rshim_backend_t *bd = fuse_req_userdata(req); if (!bd) { fuse_reply_err(req, ENODEV); return; } pthread_mutex_lock(&bd->mutex); switch (cmd) { case TCGETS: if (!out_bufsz) { struct iovec iov = { arg, sizeof(struct termio) }; fuse_reply_ioctl_retry(req, NULL, 0, &iov, 1); } else { fuse_reply_ioctl(req, 0, &bd->cons_termios, sizeof(struct termio)); } break; case TCSETS: case TCSETSW: case TCSETSF: if (!in_bufsz) { struct iovec iov = {arg, sizeof(bd->cons_termios)}; fuse_reply_ioctl_retry(req, &iov, 1, NULL, 0); } else { memcpy(&bd->cons_termios, in_buf, sizeof(bd->cons_termios)); fuse_reply_ioctl(req, 0, NULL, 0); } break; default: fuse_reply_err(req, ENOSYS); break; } pthread_mutex_unlock(&bd->mutex); } #elif defined(__FreeBSD__) static int rshim_fuse_console_ioctl(struct cuse_dev *cdev, int fflags, unsigned long cmd, void *peer_data) { rshim_backend_t *bd = cuse_dev_get_priv0(cdev); int rc = CUSE_ERR_INVALID; int value; pthread_mutex_lock(&bd->mutex); switch (cmd) { case TIOCGETA: rc = cuse_copy_out(&bd->cons_termios, peer_data, sizeof(struct termios)); break; case TIOCSETA: case TIOCSETAW: case TIOCSETAF: rc = cuse_copy_in(peer_data, &bd->cons_termios, sizeof(struct termios)); break; case TIOCEXCL: case TIOCNXCL: case FIONBIO: case FIOASYNC: rc = 0; break; case FIONREAD: pthread_mutex_lock(&bd->ringlock); value = rshim_fifo_size(bd, TMFIFO_CONS_CHAN, true) ? 1 : 0; pthread_mutex_unlock(&bd->ringlock); rc = cuse_copy_out(&value, peer_data, sizeof(value)); break; default: break; } pthread_mutex_unlock(&bd->mutex); return rc; } #endif #ifdef __linux__ static void rshim_fuse_console_poll(fuse_req_t req, struct fuse_file_info *fi, struct fuse_pollhandle *ph) { rshim_backend_t *bd = fuse_req_userdata(req); unsigned int revents = 0; bool poll_rx = false, poll_tx = false, poll_err = false; if (!bd) { fuse_reply_err(req, ENODEV); return; } rshim_fifo_check_poll(bd, TMFIFO_CONS_CHAN, &poll_rx, &poll_tx, &poll_err); if (poll_rx) revents |= POLLIN | POLLRDNORM; if (poll_tx) revents |= POLLOUT | POLLWRNORM; if (poll_err) revents |= POLLERR; if (ph) { if (!bd->fuse_poll_handle[TMFIFO_CONS_CHAN]) bd->fuse_poll_handle[TMFIFO_CONS_CHAN] = ph; else if (ph != bd->fuse_poll_handle[TMFIFO_CONS_CHAN]) fuse_pollhandle_destroy(ph); } fuse_reply_poll(req, revents); } #elif defined(__FreeBSD__) static int rshim_fuse_console_poll(struct cuse_dev *cdev, int fflags, int events) { rshim_backend_t *bd = cuse_dev_get_priv0(cdev); unsigned int revents = 0; bool poll_rx = false, poll_tx = false, poll_err = false; rshim_fifo_check_poll(bd, TMFIFO_CONS_CHAN, &poll_rx, &poll_tx, &poll_err); if (poll_rx) revents |= CUSE_POLL_READ; if (poll_tx) revents |= CUSE_POLL_WRITE; if (poll_err) revents |= CUSE_POLL_ERROR; return revents; } #endif #ifdef __linux__ static void rshim_fuse_poll_handle_destroy(rshim_backend_t *bd, int chan) { if (bd->fuse_poll_handle[chan]) { fuse_pollhandle_destroy(bd->fuse_poll_handle[chan]); bd->fuse_poll_handle[chan] = NULL; } } static void rshim_fuse_console_release(fuse_req_t req, struct fuse_file_info *fi) { rshim_backend_t *bd = fuse_req_userdata(req); if (bd) rshim_console_release(bd, rshim_fuse_poll_handle_destroy); fuse_reply_err(req, 0); } #elif defined(__FreeBSD__) static int rshim_fuse_console_release(struct cuse_dev *cdev, int fflags) { rshim_backend_t *bd = cuse_dev_get_priv0(cdev); rshim_console_release(bd, NULL); return CUSE_ERR_NONE; } #endif #ifdef __linux__ static const struct cuse_lowlevel_ops rshim_console_fops = { .open = rshim_fuse_console_open, .read = rshim_fuse_console_read, .write = rshim_fuse_console_write, .fsync = rshim_fuse_console_fsync, .ioctl = rshim_fuse_console_ioctl, .poll = rshim_fuse_console_poll, .release = rshim_fuse_console_release, }; #elif defined(__FreeBSD__) static const struct cuse_methods rshim_console_fops = { .cm_open = rshim_fuse_console_open, .cm_read = rshim_fuse_console_read, .cm_write = rshim_fuse_console_write, .cm_ioctl = rshim_fuse_console_ioctl, .cm_poll = rshim_fuse_console_poll, .cm_close = rshim_fuse_console_release, }; #endif /* Misc file operations routines */ struct rshim_misc { char buffer[4069]; off_t len; off_t offset; int ready; }; #ifdef __linux__ static void rshim_fuse_misc_open(fuse_req_t req, struct fuse_file_info *fi) { rshim_backend_t *bd = fuse_req_userdata(req); if (bd) { struct rshim_misc *ptr = calloc(1, sizeof(*ptr)); fi->fh = (uintptr_t)ptr; fuse_reply_open(req, fi); rshim_ref(bd); } else { fuse_reply_err(req, ENODEV); } } #elif defined(__FreeBSD__) static int rshim_fuse_misc_open(struct cuse_dev *cdev, int fflags) { rshim_backend_t *bd = cuse_dev_get_priv0(cdev); struct rshim_misc *ptr = calloc(1, sizeof(*ptr)); if (ptr == NULL) return CUSE_ERR_NO_MEMORY; cuse_dev_set_per_file_handle(cdev, ptr); rshim_ref(bd); return CUSE_ERR_NONE; } #endif #ifdef __linux__ static void rshim_fuse_misc_read(fuse_req_t req, size_t size, off_t off, struct fuse_file_info *fi) #elif defined(__FreeBSD__) static int rshim_fuse_misc_read(struct cuse_dev *cdev, int fflags, void *peer_ptr, int size) #endif { #ifdef __linux__ rshim_backend_t *bd = fuse_req_userdata(req); struct rshim_misc *rm = (void *)(uintptr_t)fi->fh; #elif defined(__FreeBSD__) rshim_backend_t *bd = cuse_dev_get_priv0(cdev); struct rshim_misc *rm = cuse_dev_get_per_file_handle(cdev); off_t off; #endif char opn[RSHIM_YU_BOOT_RECORD_OPN_SIZE + 1] = ""; uint8_t *mac = bd->peer_mac; int rc, len = sizeof(rm->buffer), n; struct timespec ts; struct timeval tp; uint64_t value; char *p; if (rm->ready) { #ifdef __linux__ fuse_reply_buf(req, NULL, 0); return; #elif defined(__FreeBSD__) goto ready; #endif } rm->ready = 1; if (!bd) { #ifdef __linux__ fuse_reply_err(req, ENODEV); return; #elif defined(__FreeBSD__) return CUSE_ERR_INVALID; #endif } pthread_mutex_lock(&bd->mutex); /* Boot mode. */ rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->boot_control, &value, RSHIM_REG_SIZE_8B); if (rc) { RSHIM_ERR("couldn't read BOOT_CONTROL register\n"); value = 0; } p = rm->buffer; n = snprintf(p, len, "%-16s%d (0:basic, 1:advanced, 2:log)\n", "DISPLAY_LEVEL", bd->display_level); p += n; len -= n; n = snprintf(p, len, "%-16s%lld (0:rshim, 1:emmc, 2:emmc-boot-swap)\n", "BOOT_MODE", (unsigned long long)value & RSH_BOOT_CONTROL__BOOT_MODE_MASK); p += n; len -= n; n = snprintf(p, len, "%-16s%d (seconds)\n", "BOOT_TIMEOUT", bd->boot_timeout); p += n; len -= n; n = snprintf(p, len, "%-16s%d (0:normal, 1:drop)\n", "DROP_MODE", bd->drop_mode); p += n; len -= n; /* SW reset flag is always 0. */ n = snprintf(p, len, "%-16s%d (1: reset)\n", "SW_RESET", 0); p += n; len -= n; /* Display the driver name. */ n = snprintf(p, len, "%-16s%s\n", "DEV_NAME", bd->dev_name); p += n; len -= n; /* Display device info. */ n = snprintf(p, len, "%-16sBlueField-%d(Rev %d)\n", "DEV_INFO", bd->ver_id, bd->rev_id); p += n; len -= n; /* Display OPN info (for BlueField-2 and above). */ if (bd->ver_id >= RSHIM_BLUEFIELD_2) { rshim_get_opn(bd, opn, RSHIM_YU_BOOT_RECORD_OPN_SIZE); if (!strlen(opn)) strcpy(opn, "N/A"); n = snprintf(p, len, "%-16s%s\n", "OPN_STR", opn); p += n; len -= n; } if (bd->ver_id == RSHIM_BLUEFIELD_3) { rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->uptime, &value, RSHIM_REG_SIZE_8B); if (!rc) { n = snprintf(p, len, "%-16s%ld(s)\n", "UP_TIME", value/BF3_REF_CLK_IN_HZ); p += n; len -= n; } } if (bd->display_level == 1) { gettimeofday(&tp, NULL); /* Skip SW_RESET while pushing boot stream. */ n = snprintf(p, len, "%-16s%d (1: skip)\n", "BOOT_RESET_SKIP", bd->skip_boot_reset); p += n; len -= n; /* * Display the target-side information. Send a request and wait for * some time for the response. */ bd->peer_ctrl_req = 1; bd->peer_ctrl_resp = 0; memset(mac, 0, 6); bd->has_cons_work = 1; rshim_work_signal(bd); ts.tv_sec = tp.tv_sec + 1; ts.tv_nsec = tp.tv_usec * 1000; rc = pthread_cond_timedwait(&bd->ctrl_wait_cond, &bd->mutex, &ts); if (rc) RSHIM_DBG("Timeout in getting peer response.\n"); n = snprintf(p, len, "%-16s%02x:%02x:%02x:%02x:%02x:%02x (rw)\n", "PEER_MAC", mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]); p += n; len -= n; n = snprintf(p, len, "%-16s0x%08x (rw)\n", "PXE_ID", htonl(bd->pxe_client_id)); p += n; len -= n; n = snprintf(p, len, "%-16s%d %d (rw)\n", "VLAN_ID", bd->vlan[0], bd->vlan[1]); p += n; } else if (bd->display_level == 2) { n = rshim_log_show(bd, p, len); p += n; } rm->len = p - rm->buffer; #ifdef __linux__ if (size > (int)(rm->len - off)) size = rm->len - off; pthread_mutex_unlock(&bd->mutex); fuse_reply_buf(req, rm->buffer + off, size); return; #elif defined(__FreeBSD__) ready: if (size > (int)(rm->len - rm->offset)) size = rm->len - rm->offset; off = rm->offset; rm->offset += size; pthread_mutex_unlock(&bd->mutex); rc = cuse_copy_out(rm->buffer + off, peer_ptr, size); if (rc != CUSE_ERR_NONE) return rc; else return size; #endif } #ifdef __linux__ static void rshim_fuse_misc_write(fuse_req_t req, const char *user_buffer, size_t size, off_t off, struct fuse_file_info *fi) #elif defined(__FreeBSD__) static int rshim_fuse_misc_write(struct cuse_dev *cdev, int fflags, const void *user_buffer, int size) #endif { char buf[4096]; #ifdef __linux__ rshim_backend_t *bd = fuse_req_userdata(req); const char *p = buf; #elif defined(__FreeBSD__) rshim_backend_t *bd = cuse_dev_get_priv0(cdev); const char *p = buf; #endif int i, rc = 0, value = 0, mac[6], vlan[2] = {0}, old_value; char opn[RSHIM_YU_BOOT_RECORD_OPN_SIZE + 1] = ""; uint64_t val64 = 0; char key[32]; if (!bd) { #ifdef __linux__ fuse_reply_err(req, ENODEV); return; #elif defined(__FreeBSD__) return CUSE_ERR_INVALID; #endif } if (size >= sizeof(buf)) size = sizeof(buf) - 1; #ifdef __linux__ if (off) goto invalid; memcpy(buf, user_buffer, size); #elif defined(__FreeBSD__) rc = cuse_copy_in(user_buffer, buf, size); if (rc != CUSE_ERR_NONE) return rc; #endif buf[size] = 0; if (sscanf(buf, "%s", key) != 1) goto invalid; p += strlen(key); if (strcmp(key, "DISPLAY_LEVEL") == 0) { if (sscanf(p, "%d", &value) != 1) goto invalid; bd->display_level = value; } else if (strcmp(key, "BOOT_TIMEOUT") == 0) { if (sscanf(p, "%d", &value) != 1) goto invalid; bd->boot_timeout = value; } else if (strcmp(key, "DROP_MODE") == 0) { if (sscanf(p, "%d", &value) != 1) goto invalid; pthread_mutex_lock(&bd->mutex); old_value = (int)bd->drop_mode; value = !!value; if (value == old_value) { pthread_mutex_unlock(&bd->mutex); goto done; } bd->drop_mode = 0; if (bd->enable_device && bd->enable_device(bd, value ? false : true)) bd->drop_mode = 1; else bd->drop_mode = value; if (bd->drop_mode) bd->drop_pkt = 1; pthread_mutex_unlock(&bd->mutex); /* * Check if another endpoint driver has already attached to the * same rshim device before enabling it. */ if (!bd->drop_mode) { rshim_lock(); pthread_mutex_lock(&bd->mutex); if (rshim_access_check(bd)) { RSHIM_WARN("rshim%d is not accessible\n", bd->index); bd->drop_mode = 1; } pthread_mutex_unlock(&bd->mutex); rshim_unlock(); } } else if (strcmp(key, "BOOT_MODE") == 0) { if (sscanf(p, "%x", &value) != 1) goto invalid; pthread_mutex_lock(&bd->mutex); rc = bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->boot_control, value & RSH_BOOT_CONTROL__BOOT_MODE_MASK, RSHIM_REG_SIZE_8B); pthread_mutex_unlock(&bd->mutex); } else if (strcmp(key, "SW_RESET") == 0) { if (sscanf(p, "%x", &value) != 1) goto invalid; if (value) { if (!bd->has_reprobe) { /* Detach, which shouldn't hold bd->mutex. */ rshim_notify(bd, RSH_EVENT_DETACH, 0); pthread_mutex_lock(&bd->mutex); /* Reset the TmFifo. */ rshim_fifo_reset(bd); bd->is_booting = 1; pthread_mutex_unlock(&bd->mutex); } /* SW reset. */ pthread_mutex_lock(&bd->mutex); rc = rshim_reset_control(bd); pthread_mutex_unlock(&bd->mutex); if (!bd->has_reprobe) { /* Attach. */ sleep(bd->reset_delay); pthread_mutex_lock(&bd->mutex); bd->is_booting = 0; rshim_notify(bd, RSH_EVENT_ATTACH, 0); pthread_mutex_unlock(&bd->mutex); } } } else if (strcmp(key, "BOOT_RESET_SKIP") == 0) { if (sscanf(p, "%x", &value) != 1) goto invalid; bd->skip_boot_reset = !!value; } else if (strcmp(key, "PEER_MAC") == 0) { if (sscanf(p, "%x:%x:%x:%x:%x:%x", &mac[0], &mac[1], &mac[2], &mac[3], &mac[4], &mac[5]) != 6) goto invalid; pthread_mutex_lock(&bd->mutex); for (i = 0; i < 6; i++) bd->peer_mac[i] = mac[i]; bd->peer_mac_set = 1; bd->has_cons_work = 1; rshim_work_signal(bd); pthread_mutex_unlock(&bd->mutex); } else if (strcmp(key, "PXE_ID") == 0) { if (sscanf(p, "%x", &value) != 1) goto invalid; pthread_mutex_lock(&bd->mutex); bd->pxe_client_id = ntohl(value); bd->peer_pxe_id_set = 1; bd->has_cons_work = 1; rshim_work_signal(bd); pthread_mutex_unlock(&bd->mutex); } else if (strcmp(key, "VLAN_ID") == 0) { if (sscanf(p, "%d %d", &vlan[0], &vlan[1]) == EOF) goto invalid; pthread_mutex_lock(&bd->mutex); bd->vlan[0] = vlan[0]; bd->vlan[1] = vlan[1]; bd->peer_vlan_set = 1; bd->has_cons_work = 1; rshim_work_signal(bd); pthread_mutex_unlock(&bd->mutex); } else if (!strcmp(key, "OPN_STR")) { if (sscanf(p, "%16s", opn) != 1) goto invalid; rshim_set_opn(bd, opn, RSHIM_YU_BOOT_RECORD_OPN_SIZE); } else if (!strcmp(key, "DEBUG_CODE")) { if (sscanf(p, " 0x%lx", &val64) != 1) goto invalid; pthread_mutex_lock(&bd->mutex); rc = bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->scratchpad1, val64, RSHIM_REG_SIZE_8B); if (!rc) bd->debug_code = val64; pthread_mutex_unlock(&bd->mutex); } else { invalid: #ifdef __linux__ fuse_reply_err(req, EINVAL); return; #elif defined(__FreeBSD__) return CUSE_ERR_INVALID; #endif } done: #ifdef __linux__ if (!rc) fuse_reply_write(req, size); else fuse_reply_err(req, -rc); #elif defined(__FreeBSD__) return size; #endif } #ifdef __linux__ static void rshim_fuse_misc_release(fuse_req_t req, struct fuse_file_info *fi) { rshim_backend_t *bd = fuse_req_userdata(req); free((void *)(uintptr_t)fi->fh); fuse_reply_err(req, 0); if (bd) rshim_deref(bd); } #elif defined(__FreeBSD__) static int rshim_fuse_misc_release(struct cuse_dev *cdev, int fflags) { rshim_backend_t *bd = cuse_dev_get_priv0(cdev); struct rshim_misc *rm = cuse_dev_get_per_file_handle(cdev); free(rm); if (bd) rshim_deref(bd); return CUSE_ERR_NONE; } #endif #ifdef __linux__ static const struct cuse_lowlevel_ops rshim_misc_fops = { .open = rshim_fuse_misc_open, .read = rshim_fuse_misc_read, .write = rshim_fuse_misc_write, .release = rshim_fuse_misc_release, }; #elif defined(__FreeBSD__) static const struct cuse_methods rshim_misc_fops = { .cm_open = rshim_fuse_misc_open, .cm_read = rshim_fuse_misc_read, .cm_write = rshim_fuse_misc_write, .cm_close = rshim_fuse_misc_release, }; #endif /* Rshim file operations routines */ /* ioctl message header. */ typedef struct { uint32_t addr; uint64_t data; } __attribute__((packed)) rshim_ioctl_msg; /* ioctl message header for Mustang. Unlike BF1 and BF2, Mustang * HW enables different USB transfer sizes: 1B, 2B, 4B and 8B. */ typedef struct { uint32_t addr; uint64_t data; uint8_t data_size; } __attribute__((packed)) rshim_ioctl_msg2; enum { RSHIM_IOC_READ = _IOWR('R', 0, rshim_ioctl_msg), RSHIM_IOC_WRITE = _IOWR('R', 1, rshim_ioctl_msg), RSHIM_IOC_READ2 = _IOWR('R', 0, rshim_ioctl_msg2), RSHIM_IOC_WRITE2 = _IOWR('R', 1, rshim_ioctl_msg2), }; #ifdef __linux__ static void rshim_fuse_rshim_ioctl(fuse_req_t req, int cmd, void *arg, struct fuse_file_info *fi, unsigned int flags, const void *in_buf, size_t in_bufsz, size_t out_bufsz) { rshim_backend_t *bd = fuse_req_userdata(req); rshim_ioctl_msg msg; rshim_ioctl_msg2 msg2; struct iovec iov; uint64_t data = 0; uint16_t chan, offset; int rc = 0; if (!bd) { fuse_reply_err(req, ENODEV); return; } switch (cmd) { case RSHIM_IOC_READ: case RSHIM_IOC_WRITE: iov.iov_base = arg; iov.iov_len = sizeof(msg); if (!in_bufsz) { fuse_reply_ioctl_retry(req, &iov, 1, NULL, 0); return; } if (in_bufsz != sizeof(msg)) { fuse_reply_err(req, EINVAL); return; } if (!out_bufsz) { fuse_reply_ioctl_retry(req, &iov, 1, &iov, 1); return; } memcpy(&msg, in_buf, sizeof(msg)); /* * Get channel and offset from the 32-bit address. * For BlueField-3 USB, it also supports passing the linear CR-space * address where upper 16-bit is saved in 'chan' and lower 16-bit is * saved in 'offset'. */ chan = msg.addr >> 16; offset = msg.addr & 0xFFFF; if (bd->ver_id <= RSHIM_BLUEFIELD_2 || strncmp(bd->dev_name, "usb", 3)) { chan &= 0xF; } if (cmd == RSHIM_IOC_WRITE) { pthread_mutex_lock(&bd->mutex); rc = bd->write_rshim(bd, chan, offset, msg.data, RSHIM_REG_SIZE_8B); pthread_mutex_unlock(&bd->mutex); } else { pthread_mutex_lock(&bd->mutex); rc = bd->read_rshim(bd, chan, offset, &data, RSHIM_REG_SIZE_8B); msg.data = data; pthread_mutex_unlock(&bd->mutex); } if (!rc) fuse_reply_ioctl(req, 0, &msg, sizeof(msg)); else fuse_reply_err(req, -rc); break; case RSHIM_IOC_READ2: case RSHIM_IOC_WRITE2: iov.iov_base = arg; iov.iov_len = sizeof(msg2); if (!in_bufsz) { fuse_reply_ioctl_retry(req, &iov, 1, NULL, 0); return; } if (in_bufsz != sizeof(msg2)) { fuse_reply_err(req, EINVAL); return; } if (!out_bufsz) { fuse_reply_ioctl_retry(req, &iov, 1, &iov, 1); return; } memcpy(&msg2, in_buf, sizeof(msg2)); /* * Get channel and offset from the 32-bit address. * For BlueField-3 USB, it also supports passing the linear CR-space * address where upper 16-bit is saved in 'chan' and lower 16-bit is * saved in 'offset'. */ chan = msg2.addr >> 16; offset = msg2.addr & 0xFFFF; if (bd->ver_id <= RSHIM_BLUEFIELD_2) chan &= 0xF; if (cmd == RSHIM_IOC_WRITE2) { pthread_mutex_lock(&bd->mutex); rc = bd->write_rshim(bd, chan, offset, msg2.data, msg2.data_size); pthread_mutex_unlock(&bd->mutex); } else { pthread_mutex_lock(&bd->mutex); rc = bd->read_rshim(bd, chan, offset, &data, msg2.data_size); msg2.data = data; pthread_mutex_unlock(&bd->mutex); } if (!rc) fuse_reply_ioctl(req, 0, &msg2, sizeof(msg2)); else fuse_reply_err(req, -rc); break; default: fuse_reply_err(req, ENOSYS); break; } } #elif defined(__FreeBSD__) static int rshim_fuse_rshim_ioctl(struct cuse_dev *cdev, int fflags, unsigned long cmd, void *peer_data) { rshim_backend_t *bd = cuse_dev_get_priv0(cdev); int rc = CUSE_ERR_INVALID; rshim_ioctl_msg msg; rshim_ioctl_msg2 msg2; uint64_t data; pthread_mutex_lock(&bd->mutex); switch (cmd) { case RSHIM_IOC_READ: rc = cuse_copy_in(peer_data, &msg, sizeof(msg)); if (rc == CUSE_ERR_NONE) { data = msg.data; rc = bd->read_rshim(bd, (msg.addr >> 16) & 0xF, /* channel # */ msg.addr & 0xFFFF, /* addr */ &data, RSHIM_REG_SIZE_8B); if (!rc) rc = cuse_copy_out(&msg, peer_data, sizeof(msg)); else rc = CUSE_ERR_INVALID; } break; case RSHIM_IOC_WRITE: rc = cuse_copy_in(peer_data, &msg, sizeof(msg)); rc = bd->write_rshim(bd, (msg.addr >> 16) & 0xF, /* channel # */ msg.addr & 0xFFFF, /* addr */ msg.data, RSHIM_REG_SIZE_8B); if (rc) rc = CUSE_ERR_INVALID; break; case RSHIM_IOC_READ2: rc = cuse_copy_in(peer_data, &msg2, sizeof(msg2)); if (rc == CUSE_ERR_NONE) { data = msg2.data; rc = bd->read_rshim(bd, msg2.addr >> 16, /* channel # */ msg2.addr & 0xFFFF, /* addr */ &data, msg2.data_size); if (!rc) rc = cuse_copy_out(&msg2, peer_data, sizeof(msg2)); else rc = CUSE_ERR_INVALID; } break; case RSHIM_IOC_WRITE2: rc = cuse_copy_in(peer_data, &msg2, sizeof(msg2)); rc = bd->write_rshim(bd, msg2.addr >> 16, /* channel # */ msg2.addr & 0xFFFF, /* addr */ msg2.data, msg2.data_size); if (rc) rc = CUSE_ERR_INVALID; break; default: break; } pthread_mutex_unlock(&bd->mutex); return rc; } #endif int rshim_fuse_got_peer_signal(void) { #if defined(__FreeBSD__) return cuse_got_peer_signal(); #else return -1; #endif } #ifdef __linux__ static const struct cuse_lowlevel_ops rshim_rshim_fops = { .open = rshim_fuse_misc_open, .ioctl = rshim_fuse_rshim_ioctl, .release = rshim_fuse_misc_release, }; #elif defined(__FreeBSD__) static const struct cuse_methods rshim_rshim_fops = { .cm_open = rshim_fuse_misc_open, .cm_ioctl = rshim_fuse_rshim_ioctl, .cm_close = rshim_fuse_misc_release, }; #endif static void *cuse_worker(void *arg) { #ifdef __linux__ struct fuse_session *se = arg; int rc; rc = fuse_session_loop(se); fuse_session_destroy(se); return (void *)(unsigned long)rc; #elif defined(__FreeBSD__) signal(SIGHUP, &rshim_sig_hup); while (cuse_wait_and_process() == CUSE_ERR_NONE) ; return NULL; #endif } int rshim_fuse_init(rshim_backend_t *bd) { char buf[128], *name; #ifdef __linux__ time_t t0, t1; const char *bufp[] = {buf}; struct cuse_info ci = {.dev_info_argc = 1, .dev_info_argv = bufp, .flags = CUSE_UNRESTRICTED_IOCTL}; static const struct cuse_lowlevel_ops *ops[RSH_DEV_TYPES] = #elif defined(__FreeBSD__) static const struct cuse_methods *ops[RSH_DEV_TYPES] = #endif { [RSH_DEV_TYPE_BOOT] = &rshim_boot_fops, [RSH_DEV_TYPE_TMFIFO] = &rshim_console_fops, [RSH_DEV_TYPE_RSHIM] = &rshim_rshim_fops, [RSH_DEV_TYPE_MISC] = &rshim_misc_fops, }; int i, rc; #if defined(__FreeBSD__) if (cuse_init() != CUSE_ERR_NONE) return -1; #endif for (i = 0; i < RSH_DEV_TYPES; i++) { #ifdef __linux__ static const char * const argv[] = {"./rshim", "-f"}; int multithreaded = 0; name = rshim_dev_minor_names[i]; /* * Check whether path already exists. Adding a loop in case the * device was re-ceated during SW_RESET. */ snprintf(buf, sizeof(buf), "/dev/rshim%d/%s", bd->index, name); time(&t0); while (!access(buf, F_OK)) { time(&t1); if (difftime(t1, t0) > 5) { RSHIM_ERR("%s already exists\n", buf); return -1; } } snprintf(buf, sizeof(buf), "DEVNAME=rshim%d/%s", bd->index, name); if (!ops[i]) continue; bd->fuse_session[i] = cuse_lowlevel_setup(sizeof(argv)/sizeof(char *), (char **)argv, &ci, ops[i], &multithreaded, bd); if (!bd->fuse_session[i]) { RSHIM_ERR("Failed to setup CUSE %s\n", name); return -1; } fuse_remove_signal_handlers(bd->fuse_session[i]); rc = pthread_create(&bd->fuse_thread[i], NULL, cuse_worker, bd->fuse_session[i]); if (rc) { RSHIM_ERR("Failed to create cuse thread %m\n"); return rc; } #elif defined(__FreeBSD__) name = rshim_dev_minor_names[i]; snprintf(buf, sizeof(buf), "rshim%d/%s", bd->index, name); if (!ops[i]) continue; bd->fuse_session[i] = cuse_dev_create(ops[i], bd, NULL, 0 /* UID_ROOT */, 0 /* GID_WHEEL */, 0600, "rshim%d/%s", bd->index, name); if (!bd->fuse_session[i]) { RSHIM_ERR("Failed to setup CUSE %s\n", name); return -1; } rc = pthread_create(&bd->fuse_thread[i], NULL, cuse_worker, bd->fuse_session[i]); if (rc) { RSHIM_ERR("Failed to create cuse thread %m"); return rc; } #endif } return 0; } int rshim_fuse_del(rshim_backend_t *bd) { int i; for (i = 0; i < RSH_DEV_TYPES; i++) { if (bd->fuse_session[i]) { #ifdef __linux__ fuse_session_exit(bd->fuse_session[i]); #elif defined(__FreeBSD__) cuse_dev_destroy(bd->fuse_session[i]); #endif bd->fuse_session[i] = NULL; } } for (i = 0; i < RSH_DEV_TYPES; i++) { if (bd->fuse_thread[i]) { pthread_kill(bd->fuse_thread[i], SIGINT); pthread_join(bd->fuse_thread[i], NULL); bd->fuse_thread[i] = 0; } } return 0; } ./Mellanox-rshim-user-space-0fd5d3f/src/rshim_net.c0000664000175000017500000002114414563673752021565 0ustar tai271828tai271828// SPDX-License-Identifier: GPL-2.0 or BSD-3-Clause /* * Copyright (C) 2019-2023 Mellanox Technologies. All Rights Reserved. * */ #include #ifdef __linux__ #include #include #include #endif #ifdef __FreeBSD__ #include #include #include #endif #include #include #include #include "rshim.h" static uint8_t rshim_net_default_mac[6] = {0x00, 0x1A, 0xCA, 0xFF, 0xFF, 0x02}; /* Set non-blocking. */ static int rshim_if_set_non_blocking(int fd) { int flags, err; flags = fcntl(fd, F_GETFL, 0); if (flags == -1) { RSHIM_ERR("fcntl %m\n"); return -1; } flags |= O_NONBLOCK; err = fcntl(fd, F_SETFL, flags); if (err == -1) { RSHIM_ERR("fcntl %m\n"); return -1; } return 0; } #ifdef __linux__ /* Open tun/tap interface. */ static int rshim_if_open(char *ifname, int index) { char cmd[128]; struct ifreq ifr; int s, fd, rc; fd = open("/dev/net/tun", O_RDWR); if (fd < 0) { rc = system("modprobe tun"); if (rc == -1) RSHIM_DBG("Failed to load the tun module %m\n"); fd = open("/dev/net/tun", O_RDWR); if (fd < 0) { RSHIM_ERR("Can't open %s: %m\n", ifname); return -1; } } memset(&ifr, 0, sizeof(ifr)); ifr.ifr_flags = IFF_TAP | IFF_NO_PI; snprintf(ifr.ifr_name, sizeof(ifr.ifr_name), "%s", ifname); rc = ioctl(fd, TUNSETIFF, (void *) &ifr); if (rc < 0) { RSHIM_ERR("ioctl failed: errno=%d\n", errno); close(fd); return -1; } memcpy(ifr.ifr_hwaddr.sa_data, rshim_net_default_mac, 6); ifr.ifr_hwaddr.sa_family = ARPHRD_ETHER; ifr.ifr_hwaddr.sa_data[5] += index * 2; if (ioctl(fd, SIOCSIFHWADDR, &ifr)) { RSHIM_ERR("ioctl SIOCSIFHWADDR failed"); close(fd); return -1; } s = socket(AF_INET, SOCK_DGRAM, 0); if (s < 0) { RSHIM_ERR("socket failed: %m\n"); close(fd); return -1; } if (ioctl(s, SIOCGIFFLAGS, &ifr) >= 0) { ifr.ifr_flags |= IFF_UP; ioctl(s, SIOCSIFFLAGS, &ifr); } close(s); rshim_if_set_non_blocking(fd); sprintf(cmd, "ifup %s 2>/dev/null&", ifname); if (system(cmd) == -1) RSHIM_DBG("Failed to call ifup\n"); return fd; } #elif defined(__FreeBSD__) /* Open tun/tap interface. */ static int rshim_if_open(char *ifname, int index) { struct ifreq ifr; int s, fd; s = socket(AF_INET, SOCK_DGRAM, 0); if (s < 0) { RSHIM_ERR("socket failed: %m\n"); return -1; } fd = open("/dev/tap", O_RDWR); if (fd < 0) { system("kldload -qn if_tap"); fd = open("/dev/tap", O_RDWR); if (fd < 0) { RSHIM_ERR("Can't open %s: %m\n", ifname); close(s); return -1; } } memset(&ifr, 0, sizeof(ifr)); if (ioctl(fd, TAPGIFNAME, &ifr) < 0) { RSHIM_ERR("ioctl TAPGIFNAME failed"); close(fd); close(s); return -1; } ifr.ifr_data = ifname; if (ioctl(s, SIOCSIFNAME, &ifr) < 0) { char temp[sizeof(ifr.ifr_name)]; memcpy(temp, ifr.ifr_name, sizeof(temp)); snprintf(ifr.ifr_name, sizeof(ifr.ifr_name), "%s", ifname); /* cleanup old device */ if (ioctl(s, SIOCIFDESTROY, &ifr) < 0) { RSHIM_ERR("SIOCIFDESTROY failed: %m\n"); close(s); close(fd); return -1; } snprintf(ifr.ifr_name, sizeof(ifr.ifr_name), "%s", temp); /* try to rename device again */ if (ioctl(s, SIOCSIFNAME, &ifr) < 0) { RSHIM_ERR("SIOCIFNAME failed: %m\n"); close(s); close(fd); return -1; } } snprintf(ifr.ifr_name, sizeof(ifr.ifr_name), "%s", ifname); ifr.ifr_mtu = ETHERMTU; if (ioctl(s, SIOCSIFMTU, &ifr) < 0) { RSHIM_ERR("ioctl SIOCSIMTU failed"); close(s); close(fd); return -1; } memcpy(ifr.ifr_addr.sa_data, rshim_net_default_mac, 6); ifr.ifr_addr.sa_family = AF_LINK; ifr.ifr_addr.sa_len = 6; ifr.ifr_addr.sa_data[5] += index * 2; if (ioctl(s, SIOCSIFLLADDR, &ifr) < 0) { RSHIM_ERR("ioctl SIOCSIFLLADDR failed"); close(s); close(fd); return -1; } if (ioctl(s, SIOCGIFFLAGS, &ifr) >= 0) { ifr.ifr_flags |= IFF_UP; ioctl(s, SIOCSIFFLAGS, &ifr); } close(s); rshim_if_set_non_blocking(fd); return fd; } #else #error "Unsupported platform" #endif static int rshim_if_read(int fd, char *buf, size_t len) { return read(fd, buf, len); } static int rshim_if_write(int fd, const char *buf, size_t len) { return write(fd, buf, len); } #ifdef __linux__ static void rshim_if_close(int fd) { struct ifreq ifr; char cmd[128]; int rc; memset(&ifr, 0, sizeof(ifr)); rc = ioctl(fd, TUNGETIFF, (void *) &ifr); if (!rc && ifr.ifr_name[0]) { sprintf(cmd, "ifdown %s 2>/dev/null&", ifr.ifr_name); if (system(cmd) == -1) RSHIM_DBG("Failed to call ifdown\n"); } ioctl(fd, TUNSETPERSIST, 0); close(fd); } #elif defined(__FreeBSD__) static void rshim_if_close(int fd) { struct ifreq ifr; int s; memset(&ifr, 0, sizeof(ifr)); if (ioctl(fd, TAPGIFNAME, &ifr) < 0) { close(fd); return; } close(fd); s = socket(AF_INET, SOCK_DGRAM, 0); if (s < 0) { RSHIM_ERR("socket failed: %m\n"); return; } if (ioctl(s, SIOCIFDESTROY, &ifr) < 0) { RSHIM_ERR("SIOCIFDESTROY failed: %m\n"); close(s); return; } close(s); } #else #error "Platform not supported" #endif int rshim_net_init(rshim_backend_t *bd) { struct epoll_event event; char ifname[IFNAMSIZ]; int rc, fd[2]; snprintf(ifname, sizeof(ifname), "tmfifo_net%d", bd->index); bd->net_fd = rshim_if_open(ifname, bd->index); if (bd->net_fd < 0) return bd->net_fd; memset(&event, 0, sizeof(event)); event.data.fd = bd->net_fd; event.events = EPOLLIN; rc = epoll_ctl(rshim_epoll_fd, EPOLL_CTL_ADD, bd->net_fd, &event); if (rc == -1) { RSHIM_ERR("epoll_ctl failed: %d %d\n", rshim_epoll_fd, bd->net_fd); goto fail; } rc = pipe(fd); if (rc == -1) { RSHIM_ERR("Failed to create net pipe"); goto fail; } event.data.fd = fd[0]; event.events = EPOLLIN; rc = epoll_ctl(rshim_epoll_fd, EPOLL_CTL_ADD, fd[0], &event); if (rc == -1) { RSHIM_ERR("epoll_ctl failed: %d %d\n", rshim_epoll_fd, fd[0]); goto fail; } bd->net_notify_fd[0] = fd[0]; bd->net_notify_fd[1] = fd[1]; return 0; fail: rshim_if_close(bd->net_fd); bd->net_fd = -1; return rc; } int rshim_net_del(rshim_backend_t *bd) { struct epoll_event event; if (bd->net_notify_fd[0] >= 0) { memset(&event, 0, sizeof(event)); event.data.fd = bd->net_notify_fd[0]; epoll_ctl(rshim_epoll_fd, EPOLL_CTL_DEL, bd->net_notify_fd[0], &event); close(bd->net_notify_fd[0]); close(bd->net_notify_fd[1]); bd->net_notify_fd[0] = -1; bd->net_notify_fd[1] = -1; } if (bd->net_fd >= 0) { memset(&event, 0, sizeof(event)); event.data.fd = bd->net_fd; epoll_ctl(rshim_epoll_fd, EPOLL_CTL_DEL, bd->net_fd, &event); rshim_if_close(bd->net_fd); bd->net_fd = -1; } return 0; } void rshim_net_rx(rshim_backend_t *bd) { rshim_net_pkt_t *pkt = &bd->net_rx_pkt; int len, total_len; bd->net_rx_pending = false; for (;;) { total_len = sizeof(pkt->hdr); while (bd->net_rx_len < total_len) { len = rshim_fifo_read(bd, (char *)pkt + bd->net_rx_len, total_len - bd->net_rx_len, TMFIFO_NET_CHAN, true); if (len <= 0) return; bd->net_rx_len += len; } total_len = ntohs(pkt->hdr.len) + sizeof(pkt->hdr); /* Drop invalid data. */ if (total_len > sizeof(*pkt)) { bd->net_rx_len = 0; continue; } while (bd->net_rx_len < total_len) { len = rshim_fifo_read(bd, (char *)pkt + bd->net_rx_len, total_len - bd->net_rx_len, TMFIFO_NET_CHAN, true); if (len <= 0) return; bd->net_rx_len += len; } if (pkt->hdr.len) { rshim_if_write(bd->net_fd, pkt->buf, ntohs(pkt->hdr.len)); pkt->hdr.len = 0; } bd->net_rx_len = 0; } } void rshim_net_tx(rshim_backend_t *bd) { rshim_net_pkt_t *pkt = &bd->net_tx_pkt; int len, written; do { if (!pkt->hdr.len || bd->net_tx_len >= sizeof(pkt->hdr) + ntohs(pkt->hdr.len)) { bd->net_tx_len = 0; pkt->hdr.len = 0; len = rshim_if_read(bd->net_fd, pkt->buf, sizeof(pkt->buf)); if (len <= 0) return; pkt->hdr.data = 0; pkt->hdr.type = VIRTIO_ID_NET; pkt->hdr.len = htons(len); } len = ntohs(pkt->hdr.len) + sizeof(pkt->hdr) - bd->net_tx_len; written = rshim_fifo_write(bd, (char *)pkt + bd->net_tx_len, len, TMFIFO_NET_CHAN, true); if (written > 0) bd->net_tx_len += written; } while (written == len); } ./Mellanox-rshim-user-space-0fd5d3f/src/rshim_regs.h0000664000175000017500000001562614563673752021754 0ustar tai271828tai271828/* SPDX-License-Identifier: GPL-2.0 or BSD-3-Clause */ /* * Copyright (C) 2019-2023 Mellanox Technologies. All Rights Reserved. * */ #ifndef __RSHIM_REGS_H__ #define __RSHIM_REGS_H__ #ifdef __ASSEMBLER__ #define _64bit(x) x #else /* __ASSEMBLER__ */ #ifdef __tile__ #define _64bit(x) x ## UL #else /* __tile__ */ #define _64bit(x) x ## ULL #endif /* __tile__ */ #endif /* __ASSEMBLER */ #ifdef __KERNEL__ #include #else #include #endif #ifndef __DOXYGEN__ #define RSH_BOOT_FIFO_DATA 0x408 #define RSH_BOOT_FIFO_COUNT 0x488 #define RSH_BOOT_FIFO_COUNT__LENGTH 0x0001 #define RSH_BOOT_FIFO_COUNT__BOOT_FIFO_COUNT_SHIFT 0 #define RSH_BOOT_FIFO_COUNT__BOOT_FIFO_COUNT_RESET_VAL 0 #define RSH_BOOT_FIFO_COUNT__BOOT_FIFO_COUNT_MASK 0x3ff #define RSH_BOOT_CONTROL 0x528 #define RSH_BOOT_CONTROL__LENGTH 0x0001 #define RSH_BOOT_CONTROL__BOOT_MODE_SHIFT 0 #define RSH_BOOT_CONTROL__BOOT_MODE_WIDTH 2 #define RSH_BOOT_CONTROL__BOOT_MODE_RESET_VAL 0 #define RSH_BOOT_CONTROL__BOOT_MODE_RMASK 0x3 #define RSH_BOOT_CONTROL__BOOT_MODE_MASK 0x3 #define RSH_BOOT_CONTROL__BOOT_MODE_VAL_NONE 0x0 #define RSH_BOOT_CONTROL__BOOT_MODE_VAL_EMMC 0x1 #define RSH_BOOT_CONTROL__BOOT_MODE_VAL_EMMC_LEGACY 0x3 #define RSH_RESET_CONTROL 0x500 #define RSH_RESET_CONTROL__LENGTH 0x0001 #define RSH_RESET_CONTROL__RESET_CHIP_SHIFT 0 #define RSH_RESET_CONTROL__RESET_CHIP_WIDTH 32 #define RSH_RESET_CONTROL__RESET_CHIP_RESET_VAL 0 #define RSH_RESET_CONTROL__RESET_CHIP_RMASK 0xffffffff #define RSH_RESET_CONTROL__RESET_CHIP_MASK 0xffffffff #define RSH_RESET_CONTROL__RESET_CHIP_VAL_KEY 0xca710001 #define RSH_RESET_CONTROL__DISABLE_SHIFT 32 #define RSH_RESET_CONTROL__DISABLE_WIDTH 1 #define RSH_RESET_CONTROL__DISABLE_RESET_VAL 0 #define RSH_RESET_CONTROL__DISABLE_RMASK 0x1 #define RSH_RESET_CONTROL__DISABLE_MASK _64bit(0x100000000) #define RSH_RESET_CONTROL__REQ_PND_SHIFT 33 #define RSH_RESET_CONTROL__REQ_PND_WIDTH 1 #define RSH_RESET_CONTROL__REQ_PND_RESET_VAL 0 #define RSH_RESET_CONTROL__REQ_PND_RMASK 0x1 #define RSH_RESET_CONTROL__REQ_PND_MASK _64bit(0x200000000) #define RSH_SCRATCHPAD1 0xc20 #define RSH_SCRATCHPAD6 0xc48 #define RSH_TM_HOST_TO_TILE_STS 0xa28 #define RSH_TM_HOST_TO_TILE_STS__LENGTH 0x0001 #define RSH_TM_HOST_TO_TILE_STS__COUNT_SHIFT 0 #define RSH_TM_HOST_TO_TILE_STS__COUNT_WIDTH 9 #define RSH_TM_HOST_TO_TILE_STS__COUNT_RESET_VAL 0 #define RSH_TM_HOST_TO_TILE_STS__COUNT_RMASK 0x1ff #define RSH_TM_HOST_TO_TILE_STS__COUNT_MASK 0x1ff #define RSH_TM_TILE_TO_HOST_STS 0xa48 #define RSH_TM_TILE_TO_HOST_STS__LENGTH 0x0001 #define RSH_TM_TILE_TO_HOST_STS__COUNT_SHIFT 0 #define RSH_TM_TILE_TO_HOST_STS__COUNT_WIDTH 9 #define RSH_TM_TILE_TO_HOST_STS__COUNT_RESET_VAL 0 #define RSH_TM_TILE_TO_HOST_STS__COUNT_RMASK 0x1ff #define RSH_TM_TILE_TO_HOST_STS__COUNT_MASK 0x1ff #define RSH_TM_HOST_TO_TILE_DATA 0xa20 #define RSH_TM_TILE_TO_HOST_DATA 0xa40 #define RSH_MMIO_ADDRESS_SPACE__LENGTH 0x10000000000 #define RSH_MMIO_ADDRESS_SPACE__STRIDE 0x8 #define RSH_MMIO_ADDRESS_SPACE__OFFSET_SHIFT 0 #define RSH_MMIO_ADDRESS_SPACE__OFFSET_WIDTH 16 #define RSH_MMIO_ADDRESS_SPACE__OFFSET_RESET_VAL 0 #define RSH_MMIO_ADDRESS_SPACE__OFFSET_RMASK 0xffff #define RSH_MMIO_ADDRESS_SPACE__OFFSET_MASK 0xffff #define RSH_MMIO_ADDRESS_SPACE__PROT_SHIFT 16 #define RSH_MMIO_ADDRESS_SPACE__PROT_WIDTH 3 #define RSH_MMIO_ADDRESS_SPACE__PROT_RESET_VAL 0 #define RSH_MMIO_ADDRESS_SPACE__PROT_RMASK 0x7 #define RSH_MMIO_ADDRESS_SPACE__PROT_MASK 0x70000 #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_SHIFT 23 #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_WIDTH 4 #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_RESET_VAL 0 #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_RMASK 0xf #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_MASK 0x7800000 #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_BOOT 0x0 #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_RSHIM 0x1 #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_UART0 0x2 #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_UART1 0x3 #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_DIAG_UART 0x4 #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_TYU 0x5 #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_TYU_EXT1 0x6 #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_TYU_EXT2 0x7 #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_TYU_EXT3 0x8 #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_TIMER 0x9 #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_USB 0xa #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_GPIO 0xb #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_MMC 0xc #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_TIMER_EXT 0xd #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_WDOG0 0xe #define RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_WDOG1 0xf /* TODO: Find equivalent registers in Mustang. Only used in PCIe */ #define RSH_BYTE_ACC_CTL 0x490 #define RSH_BYTE_ACC_WDAT 0x498 #define RSH_BYTE_ACC_RDAT 0x4a0 #define RSH_BYTE_ACC_ADDR 0x4a8 #define RSH_BYTE_ACC_INTERLOCK 0x04b0 #define RSH_UPTIME 0x0630 #define RSH_UPTIME_POR 0x0638 #define RSH_ARM_WDG_CONTROL_WCS 0x0000 #define RSH_SEMAPHORE0 0x0028 #define RSH_SCRATCH_BUF_DAT 0x0610 #define RSH_SCRATCH_BUF_CTL 0x0600 #define RSH_SCRATCH_BUF_CTL__LENGTH 0x0001 #define RSH_SCRATCH_BUF_CTL__IDX_SHIFT 0 #define RSH_SCRATCH_BUF_CTL__IDX_WIDTH 7 #define RSH_SCRATCH_BUF_CTL__IDX_RESET_VAL 0 #define RSH_SCRATCH_BUF_CTL__IDX_RMASK 0x7f #define RSH_SCRATCH_BUF_CTL__IDX_MASK 0x7f #define RSH_MEM_ACC_CTL 0x08f0 #define RSH_MEM_ACC_RSP_CNT 0x08f8 #define RSH_MEM_ACC_DATA__FIRST_WORD 0x0900 #define RSH_MEM_ACC_CTL__WRITE_SHIFT 48 #define RSH_MEM_ACC_CTL__WRITE_RMASK 0x1 #define RSH_MEM_ACC_CTL__ADDRESS_SHIFT 0 #define RSH_MEM_ACC_CTL__ADDRESS_RMASK _64bit(0xffffffffff) #define RSH_MEM_ACC_CTL__SIZE_SHIFT 52 #define RSH_MEM_ACC_CTL__SIZE_RMASK 0x7 #define RSH_MEM_ACC_CTL__SEND_SHIFT 63 #define RSH_MEM_ACC_CTL__SEND_RMASK 0x1 #define RSH_MEM_ACC_CTL__SIZE_VAL_SZ4 0x2 #define RSH_MEM_ACC_CTL__SIZE_VAL_SZ8 0x3 #define RSH_DEVICE_MSTR_PRIV_LVL 0x0550 #define RSH_DEVICE_MSTR_PRIV_LVL__MEM_ACC_LVL_SHIFT 48 #define RSH_FABRIC_DIM 0x0110 // Mustang-specific registers' addresses, masks, shifts #define BF3_RSH_BASE_ADDR 0x13000000 #define BF3_RSH_BOOT_FIFO_DATA 0x2000 #define BF3_RSH_BOOT_FIFO_COUNT 0x1000 #define BF3_RSH_BOOT_FIFO_COUNT__BOOT_FIFO_COUNT_MASK 0x3ff #define BF3_RSH_BOOT_CONTROL 0x528 #define BF3_RSH_RESET_CONTROL 0x500 #define BF3_RSH_SCRATCHPAD1 0xc40 #define BF3_RSH_SCRATCHPAD6 0xc68 #define BF3_RSH_TM_HOST_TO_TILE_STS 0x6000 #define BF3_RSH_TM_TILE_TO_HOST_STS 0x6100 #define BF3_RSH_TM_HOST_TO_TILE_DATA 0x4000 #define BF3_RSH_TM_TILE_TO_HOST_DATA 0x5000 #define BF3_RSH_SEMAPHORE0 0x00f10 #define BF3_RSH_MEM_ACC_CTL 0x0808 #define BF3_RSH_MEM_ACC_RSP_CNT 0x0810 #define BF3_RSH_MEM_ACC_DATA__FIRST_WORD 0x0828 #define BF3_RSH_DEVICE_MSTR_PRIV_LVL 0x0820 #define BF3_RSH_DEVICE_MSTR_PRIV_LVL__MEM_ACC_LVL_SHIFT 0 #define BF3_RSH_FABRIC_DIM 0x0110 #define BF3_RSH_UPTIME 0x0630 #define BF3_RSH_UPTIME_POR 0x0638 #define BF3_RSH_ARM_WDG_CONTROL_WCS 0x0000 #define BF3_RSH_SCRATCH_BUF_DAT 0x0610 #define BF3_RSH_SCRATCH_BUF_CTL 0x0600 #endif /* !defined(__DOXYGEN__) */ #endif /* !defined(__RSHIM_REGS_H__) */ ./Mellanox-rshim-user-space-0fd5d3f/src/rshim_usb.c0000664000175000017500000010205514563673752021571 0ustar tai271828tai271828// SPDX-License-Identifier: GPL-2.0-only /* * Copyright (C) 2019 Mellanox Technologies. All Rights Reserved. * */ #include #include #include #include #include #include "rshim.h" /* Our USB vendor/product IDs. */ #define USB_TILERA_VENDOR_ID 0x22dc /* Tilera Corporation */ #define USB_BLUEFIELD_1_PRODUCT_ID 0x0004 /* Mellanox Bluefield-1 */ #define USB_BLUEFIELD_2_PRODUCT_ID 0x0214 /* Mellanox Bluefield-2 */ #define USB_BLUEFIELD_3_PRODUCT_ID 0x021c /* Mellanox Bluefield-3 */ #define READ_RETRIES 5 #define WRITE_RETRIES 5 #define RSHIM_USB_TIMEOUT 20000 #define BF_MMIO_BASE 0x1000 /* Structure to hold all of our device specific stuff. */ typedef struct { /* RShim backend structure. */ rshim_backend_t bd; libusb_device_handle *handle; /* Control data. */ uint64_t ctrl_data; /* Interrupt data buffer. This is a USB DMA'able buffer. */ uint64_t *intr_buf; /* Read/interrupt urb, retries, and mode. */ struct libusb_transfer *read_or_intr_urb; int read_or_intr_retries; int read_urb_is_intr; /* Write urb and retries. */ struct libusb_transfer *write_urb; int write_retries; /* The address of the boot FIFO endpoint. */ uint8_t boot_fifo_ep; /* The address of the tile-monitor FIFO interrupt endpoint. */ uint8_t tm_fifo_int_ep; /* The address of the tile-monitor FIFO input endpoint. */ uint8_t tm_fifo_in_ep; /* The address of the tile-monitor FIFO output endpoint. */ uint8_t tm_fifo_out_ep; } rshim_usb_t; static libusb_context *rshim_usb_ctx; static int rshim_usb_epoll_fd; static bool rshim_usb_need_probe; static int rshim_usb_product_ids[] = { USB_BLUEFIELD_1_PRODUCT_ID, USB_BLUEFIELD_2_PRODUCT_ID, USB_BLUEFIELD_3_PRODUCT_ID }; static void rshim_usb_delete(rshim_backend_t *bd) { rshim_usb_t *dev = container_of(bd, rshim_usb_t, bd); rshim_deregister(bd); RSHIM_INFO("rshim %s deleted\n", bd->dev_name); if (dev->handle) { libusb_close(dev->handle); dev->handle = NULL; } free(dev); } struct rshim_usb_addr { uint16_t wvalue; uint16_t windex; }; struct rshim_usb_addr bf3_wval_widx_pair_map[] = { [RSHIM_CHANNEL] = { .wvalue = 0x0300, .windex = 0x0000, }, [UART0_CHANNEL] = { .wvalue = 0x0301, .windex = 0x0000, }, [UART1_CHANNEL] = { .wvalue = 0x0301, .windex = 0x1000, }, [DIAGUART_CHANNEL] = { .wvalue = 0x0301, .windex = 0x2000, }, [RSH_HUB_CHANNEL] = { .wvalue = 0x0301, .windex = 0x2400, }, [WDOG0_CHANNEL] = { .wvalue = 0x0302, .windex = 0x0000, }, [WDOG1_CHANNEL] = { .wvalue = 0x0304, .windex = 0x0000, }, [MCH_CORE_CHANNEL] = { .wvalue = 0x0306, .windex = 0x0000, }, [TIMER_ARM_CHANNEL] = { .wvalue = 0x0308, .windex = 0x0000, }, [TIMER_EXT_CHANNEL] = { .wvalue = 0x030a, .windex = 0x0000, }, [OOB_CHANNEL] = { .wvalue = 0x030a, .windex = 0x1000, }, [YU_CHANNEL] = { .wvalue = 0x0340, .windex = 0x0000, }, }; static struct rshim_usb_addr get_wvalue_windex(int chan, int addr, uint16_t ver_id) { struct rshim_usb_addr rsh_usb_addr; if (ver_id == RSHIM_BLUEFIELD_3) { if (chan <= 0xF) { rsh_usb_addr.wvalue = bf3_wval_widx_pair_map[chan].wvalue + BF_MMIO_BASE; rsh_usb_addr.windex = bf3_wval_widx_pair_map[chan].windex + addr; } else { rsh_usb_addr.wvalue = chan; rsh_usb_addr.windex = addr; } } else { rsh_usb_addr.wvalue = chan; rsh_usb_addr.windex = addr; } return rsh_usb_addr; } /* Rshim read/write routines */ static int rshim_usb_read_rshim(rshim_backend_t *bd, uint32_t chan, uint32_t addr, uint64_t *result, int size) { rshim_usb_t *dev = container_of(bd, rshim_usb_t, bd); struct rshim_usb_addr rsh_usb_addr; int rc; if (!bd->has_rshim) return -ENODEV; if ((bd->ver_id == RSHIM_BLUEFIELD_3) && (size <= RSHIM_REG_SIZE_4B)) size = RSHIM_REG_SIZE_4B; else size = RSHIM_REG_SIZE_8B; rsh_usb_addr = get_wvalue_windex(chan, addr, bd->ver_id); /* Do a blocking control read and endian conversion. */ dev->ctrl_data = 0; rc = libusb_control_transfer(dev->handle, LIBUSB_ENDPOINT_IN | LIBUSB_REQUEST_TYPE_VENDOR | LIBUSB_RECIPIENT_ENDPOINT, 0, rsh_usb_addr.wvalue, rsh_usb_addr.windex, (unsigned char *)&dev->ctrl_data, size, RSHIM_USB_TIMEOUT); /* * The RShim HW puts bytes on the wire in little-endian order * regardless of endianness settings either in the host or the ARM * cores. */ *result = le64toh(dev->ctrl_data); if (rc == size) return 0; /* * These are weird error codes, but we want to use something * the USB stack doesn't use so that we can identify short/long * reads. */ return rc >= 0 ? (rc > size ? -EINVAL : -ENXIO) : rc; } static int rshim_usb_write_rshim(rshim_backend_t *bd, uint32_t chan, uint32_t addr, uint64_t value, int size) { rshim_usb_t *dev = container_of(bd, rshim_usb_t, bd); struct rshim_usb_addr rsh_usb_addr; int rc; if (!bd->has_rshim) return -ENODEV; if ((bd->ver_id == RSHIM_BLUEFIELD_3) && (size <= RSHIM_REG_SIZE_4B)) size = RSHIM_REG_SIZE_4B; else size = RSHIM_REG_SIZE_8B; rsh_usb_addr = get_wvalue_windex(chan, addr, bd->ver_id); /* Convert the word to little endian and do blocking control write. */ dev->ctrl_data = htole64(value); rc = libusb_control_transfer(dev->handle, LIBUSB_ENDPOINT_OUT | LIBUSB_REQUEST_TYPE_VENDOR | LIBUSB_RECIPIENT_ENDPOINT, 0, rsh_usb_addr.wvalue, rsh_usb_addr.windex, (unsigned char *)&dev->ctrl_data, size, RSHIM_USB_TIMEOUT); if (rc == size) return 0; /* * These are weird error codes, but we want to use something * the USB stack doesn't use so that we can identify short/long * writes. */ return rc >= 0 ? (rc > size ? -EINVAL : -ENXIO) : rc; } static ssize_t rshim_usb_bf3_boot_write(rshim_backend_t *bd, const char *buf, size_t count) { rshim_usb_t *dev = container_of(bd, rshim_usb_t, bd); ssize_t avail_fifo_bytes, temp_count; int transferred = 0; int rc, tmp_tsfr; int size_addr; uint64_t reg; time_t t0, t1; int i = 0; size_addr = bd->regs->boot_fifo_count; temp_count = count; while (temp_count) { /* Check whether the BOOT FIFO is full. If it is, poll, until * there is free space. The BOOT FIFO has a max size of * is 0x400 (lines) * 8 = 8192 (bytes). */ do { time(&t0); rc = bd->read_rshim(bd, RSHIM_CHANNEL, size_addr, ®, RSHIM_REG_SIZE_8B); if (rc < 0) { RSHIM_ERR("read_rshim error %d\n", rc); return rc; } avail_fifo_bytes = BF3_MAX_BOOT_FIFO_SIZE - (reg * 8); if (avail_fifo_bytes > 0) break; usleep(50000); time(&t1); } while(difftime(t1, t0) < bd->boot_timeout); if (avail_fifo_bytes) { if (temp_count < avail_fifo_bytes) avail_fifo_bytes = temp_count; rc = libusb_bulk_transfer(dev->handle, dev->boot_fifo_ep, (void *)(buf + i), avail_fifo_bytes, &tmp_tsfr, RSHIM_USB_TIMEOUT); if (rc && (rc != LIBUSB_ERROR_TIMEOUT)) { RSHIM_ERR("Boot fifo bulk transfer failed\n"); return rc; } i += tmp_tsfr; temp_count -= tmp_tsfr; transferred += tmp_tsfr; } else { RSHIM_ERR("Timeout boot fifo count\n"); return transferred; } } return transferred; } /* Boot routines */ static ssize_t rshim_usb_boot_write(rshim_backend_t *bd, const char *buf, size_t count) { rshim_usb_t *dev = container_of(bd, rshim_usb_t, bd); int transferred; int rc; if (bd->ver_id == RSHIM_BLUEFIELD_3) { rc = rshim_usb_bf3_boot_write(bd, buf, count); return rc; } rc = libusb_bulk_transfer(dev->handle, dev->boot_fifo_ep, (void *)buf, count, &transferred, RSHIM_USB_TIMEOUT); if (!rc || rc == LIBUSB_ERROR_TIMEOUT) return transferred; else return rc; } /* FIFO routines */ static void rshim_usb_fifo_read_callback(struct libusb_transfer *urb) { rshim_usb_t *dev = urb->user_data; rshim_backend_t *bd = &dev->bd; RSHIM_DBG("fifo_read_callback: %s urb completed, status %d, " "actual length %d, intr buf 0x%x\n", dev->read_urb_is_intr ? "interrupt" : "read", urb->status, urb->actual_length, (int)*dev->intr_buf); pthread_mutex_lock(&bd->ringlock); bd->spin_flags &= ~RSH_SFLG_READING; switch (urb->status) { case LIBUSB_TRANSFER_COMPLETED: /* * If a read completed, clear the number of bytes available * from the last interrupt, and set up the new buffer for * processing. (If an interrupt completed, there's nothing * to do, since the number of bytes available was already * set by the I/O itself.) */ if (!dev->read_urb_is_intr) { *dev->intr_buf = 0; bd->read_buf_bytes = urb->actual_length; bd->read_buf_next = 0; } /* Process any data we got, and launch another I/O if needed. */ rshim_notify(bd, RSH_EVENT_FIFO_INPUT, 0); break; case LIBUSB_TRANSFER_NO_DEVICE: /* * The urb was explicitly cancelled. The only time we * currently do this is when we close the stream. If we * mark this as an error, tile-monitor --resume won't work, * so we just want to do nothing. */ break; case LIBUSB_TRANSFER_TIMED_OUT: case LIBUSB_TRANSFER_STALL: case LIBUSB_TRANSFER_OVERFLOW: if (dev->read_or_intr_retries < READ_RETRIES && urb->actual_length == 0) { /* * We got an error which could benefit from being retried. * Just submit the same urb again. Note that we don't * handle partial reads; it's hard, and we haven't really * seen them. */ int rc; dev->read_or_intr_retries++; rc = libusb_submit_transfer(urb); if (rc) { RSHIM_DBG("fifo_read_callback: resubmitted urb but got error %d\n", rc); /* * In this case, we won't try again; signal the * error to upper layers. */ rshim_notify(bd, RSH_EVENT_FIFO_ERR, rc > 0 ? -rc : rc); } else { bd->spin_flags |= RSH_SFLG_READING; } break; } case LIBUSB_TRANSFER_CANCELLED: break; default: /* * We got some error we don't know how to handle, or we got * too many errors. Either way we don't retry any more, * but we signal the error to upper layers. */ RSHIM_DBG("fifo_read_callback: %s urb completed abnormally, " "error %d\n", dev->read_urb_is_intr ? "interrupt" : "read", urb->status); rshim_notify(bd, RSH_EVENT_FIFO_ERR, urb->status > 0 ? -urb->status : urb->status); break; } pthread_mutex_unlock(&bd->ringlock); } static void rshim_usb_fifo_read(rshim_usb_t *dev, char *buffer, size_t count) { rshim_backend_t *bd = &dev->bd; struct libusb_transfer *urb; bool send_bulk_in = false; int rc; if (!bd->has_rshim || !bd->has_tm || bd->drop_mode) return; if (bd->ver_id == RSHIM_BLUEFIELD_3) { /* intr_buf is the number of "lines" that should be read from TMFIFO IN. * Each line is 8 bytes. * TMFIFO has a max of 256 lines so a max of 2048 bytes * Send a bulk request with the exact number of bytes returned * by the interrupt transfer. */ if ((int) *dev->intr_buf) { send_bulk_in = true; if (count > ((int) *dev->intr_buf * 8)) count = (int) *dev->intr_buf * 8; } } else { if ((int) *dev->intr_buf || bd->read_buf_bytes) send_bulk_in = true; } if (send_bulk_in) { /* We're doing a read. */ urb = dev->read_or_intr_urb; libusb_fill_bulk_transfer(urb, dev->handle, dev->tm_fifo_in_ep, (uint8_t *)buffer, count, rshim_usb_fifo_read_callback, dev, RSHIM_USB_TIMEOUT); dev->bd.spin_flags |= RSH_SFLG_READING; dev->read_urb_is_intr = 0; dev->read_or_intr_retries = 0; /* Submit the urb. */ rc = libusb_submit_transfer(urb); if (rc) { dev->bd.spin_flags &= ~RSH_SFLG_READING; RSHIM_ERR("usb_fifo_read: failed to submit read urb, error %d\n", rc); } RSHIM_DBG("usb_fifo_read: submit read urb\n"); } else { /* We're doing an interrupt. */ urb = dev->read_or_intr_urb; libusb_fill_interrupt_transfer(urb, dev->handle, dev->tm_fifo_int_ep, (unsigned char *)dev->intr_buf, sizeof(*dev->intr_buf), rshim_usb_fifo_read_callback, dev, #ifdef __linux__ -1 #else 0 #endif ); dev->bd.spin_flags |= RSH_SFLG_READING; dev->read_urb_is_intr = 1; dev->read_or_intr_retries = 0; /* Submit the urb */ rc = libusb_submit_transfer(urb); if (rc) { dev->bd.spin_flags &= ~RSH_SFLG_READING; RSHIM_DBG("usb_fifo_read: failed submitting interrupt urb %d\n", rc); } RSHIM_DBG("usb_fifo_read: submit interrupt urb\n"); } } static void rshim_usb_fifo_write_callback(struct libusb_transfer *urb) { rshim_usb_t *dev = urb->user_data; rshim_backend_t *bd = &dev->bd; RSHIM_DBG("usb_fifo_write_callback: urb completed, status %d, " "actual length %d, intr buf %d\n", urb->status, urb->actual_length, (int) *dev->intr_buf); pthread_mutex_lock(&bd->ringlock); bd->spin_flags &= ~RSH_SFLG_WRITING; switch (urb->status) { case LIBUSB_TRANSFER_COMPLETED: /* A write completed. */ pthread_cond_broadcast(&bd->fifo_write_complete_cond); rshim_notify(bd, RSH_EVENT_FIFO_OUTPUT, 0); break; case LIBUSB_TRANSFER_NO_DEVICE: /* * The urb was explicitly cancelled. The only time we * currently do this is when we close the stream. If we * mark this as an error, tile-monitor --resume won't work, * so we just want to do nothing. */ break; case LIBUSB_TRANSFER_TIMED_OUT: case LIBUSB_TRANSFER_STALL: case LIBUSB_TRANSFER_OVERFLOW: if (dev->write_retries < WRITE_RETRIES && urb->actual_length == 0) { /* * We got an error which could benefit from being retried. * Just submit the same urb again. Note that we don't * handle partial writes; it's hard, and we haven't really * seen them. */ int rc; dev->write_retries++; rc = libusb_submit_transfer(urb); if (rc) { RSHIM_ERR("usb_fifo_write_callback: resubmitted urb but " "got error %d\n", rc); /* * In this case, we won't try again; signal the * error to upper layers. */ rshim_notify(bd, RSH_EVENT_FIFO_ERR, rc > 0 ? -rc : rc); } else { bd->spin_flags |= RSH_SFLG_WRITING; } break; } case LIBUSB_TRANSFER_CANCELLED: break; default: /* * We got some error we don't know how to handle, or we got * too many errors. Either way we don't retry any more, * but we signal the error to upper layers. */ RSHIM_ERR("usb_fifo_write_callback: urb completed abnormally %d\n", urb->status); rshim_notify(bd, RSH_EVENT_FIFO_ERR, urb->status > 0 ? -urb->status : urb->status); break; } pthread_mutex_unlock(&bd->ringlock); } static int rshim_usb_fifo_write(rshim_usb_t *dev, const char *buffer, size_t count) { rshim_backend_t *bd = &dev->bd; int rc; if (!bd->has_rshim || !bd->has_tm) return -ENODEV; if (bd->drop_mode) return 0; if (count % 8) RSHIM_WARN("rshim write %d is not multiple of 8 bytes\n", (int)count); /* Initialize the urb properly. */ libusb_fill_bulk_transfer(dev->write_urb, dev->handle, dev->tm_fifo_out_ep, (uint8_t *)buffer, count, rshim_usb_fifo_write_callback, dev, RSHIM_USB_TIMEOUT); dev->write_retries = 0; /* Send the data out the bulk port. */ rc = libusb_submit_transfer(dev->write_urb); if (rc) { bd->spin_flags &= ~RSH_SFLG_WRITING; RSHIM_DBG("usb_fifo_write: failed submitting write urb, error %d\n", rc); return -1; } bd->spin_flags |= RSH_SFLG_WRITING; return 0; } /* Probe routines */ /* These make the endpoint test code in rshim_usb_probe() a lot cleaner. */ #define USB_ENDPOINT_XFERTYPE_MASK 0x03 #define USB_ENDPOINT_DIR_MASK 0x80 #define is_in_ep(ep) (((ep)->bEndpointAddress & USB_ENDPOINT_DIR_MASK) == \ LIBUSB_ENDPOINT_IN) #define is_bulk_ep(ep) (((ep)->bmAttributes & USB_ENDPOINT_XFERTYPE_MASK) == \ LIBUSB_TRANSFER_TYPE_BULK) #define is_int_ep(ep) (((ep)->bmAttributes & USB_ENDPOINT_XFERTYPE_MASK) == \ LIBUSB_TRANSFER_TYPE_INTERRUPT) #define max_pkt(ep) le16_to_cpu(ep->wMaxPacketSize) #define ep_addr(ep) (ep->bEndpointAddress) static ssize_t rshim_usb_backend_read(rshim_backend_t *bd, int devtype, char *buf, size_t count) { rshim_usb_t *dev = container_of(bd, rshim_usb_t, bd); switch (devtype) { case RSH_DEV_TYPE_TMFIFO: rshim_usb_fifo_read(dev, buf, count); return 0; default: RSHIM_ERR("bad devtype %d\n", devtype); return -EINVAL; } } static ssize_t rshim_usb_backend_write(rshim_backend_t *bd, int devtype, const char *buf, size_t count) { rshim_usb_t *dev = container_of(bd, rshim_usb_t, bd); switch (devtype) { case RSH_DEV_TYPE_TMFIFO: return rshim_usb_fifo_write(dev, buf, count); case RSH_DEV_TYPE_BOOT: return rshim_usb_boot_write(bd, buf, count); default: RSHIM_ERR("bad devtype %d\n", devtype); return -EINVAL; } } static void rshim_usb_backend_cancel_req(rshim_backend_t *bd, int devtype, bool is_write) { rshim_usb_t *dev = container_of(bd, rshim_usb_t, bd); if (!dev->handle) return; switch (devtype) { case RSH_DEV_TYPE_TMFIFO: if (is_write) { if (dev->write_urb) libusb_cancel_transfer(dev->write_urb); } else { if (dev->read_or_intr_urb) libusb_cancel_transfer(dev->read_or_intr_urb); } break; default: RSHIM_ERR("bad devtype %d\n", devtype); break; } } static int rshim_usb_probe_one(libusb_context *ctx, libusb_device *usb_dev, struct libusb_device_descriptor *desc) { const struct libusb_interface_descriptor *iface_desc; const struct libusb_endpoint_descriptor *ep; const struct libusb_interface *interface; struct libusb_config_descriptor *config; char dev_name[RSHIM_DEV_NAME_LEN], *p; uint8_t port_numbers[8] = {0}, bus; libusb_device_handle *handle; rshim_usb_t *dev = NULL; rshim_backend_t *bd; int i, rc; /* Check if already exists. */ rshim_lock(); bd = rshim_find_by_dev(usb_dev); rshim_unlock(); if (bd) return 0; /* Check bus number and the port path of the rshim device path. */ bus = libusb_get_bus_number(usb_dev); #if HAVE_LIBUSB_GET_PORT_NUMBERS || defined(__FreeBSD__) rc = libusb_get_port_numbers(usb_dev, port_numbers, sizeof(port_numbers)); #elif HAVE_LIBUSB_GET_DEVICE_ADDRESS port_numbers[0] = libusb_get_device_address(usb_dev); rc = 1; #else rc = -EINVAL; #endif if (rc <= 0) { RSHIM_ERR("Failed to get USB ports\n"); return -ENODEV; } sprintf(dev_name, "usb-%x", bus); p = dev_name + strlen(dev_name); for (i = 0; i < rc; i++) { sprintf(p, "%c%x", (i == rc - 1) ? '.' : '-', port_numbers[i]); p += strlen(p); } if (!rshim_allow_device(dev_name)) return -EACCES; RSHIM_INFO("Probing %s\n", dev_name); rc = libusb_get_active_config_descriptor(usb_dev, &config); if (rc) { RSHIM_ERR("Failed to get active config\n"); return -ENODEV; } rc = libusb_open(usb_dev, &handle); if (rc) { RSHIM_ERR("Failed to open USB device\n"); return rc; } for (i = 0; i < config->bNumInterfaces; i++) { rc = libusb_claim_interface(handle, i); if (rc < 0) { if (libusb_kernel_driver_active(handle, i) == 1) { RSHIM_ERR("Kernel driver is running. Please uninstall it first.\n"); exit(rc); } else { RSHIM_ERR("Failed to claim interface\n"); } libusb_close(handle); return rc; } } /* * Now see if we've previously seen this device. If so, we use the * same device number, otherwise we pick the first available one. */ rshim_lock(); /* Find the backend. */ bd = rshim_find_by_name(dev_name); if (bd) { RSHIM_INFO("found %s\n", dev_name); dev = container_of(bd, rshim_usb_t, bd); } else { RSHIM_INFO("create rshim %s\n", dev_name); dev = calloc(1, sizeof(*dev)); if (dev == NULL) { RSHIM_ERR("couldn't get memory for new device"); goto error; } bd = &dev->bd; strcpy(bd->dev_name, dev_name); bd->drop_mode = (rshim_drop_mode >= 0) ? rshim_drop_mode : 0; bd->read = rshim_usb_backend_read; bd->write = rshim_usb_backend_write; bd->cancel = rshim_usb_backend_cancel_req; bd->destroy = rshim_usb_delete; bd->read_rshim = rshim_usb_read_rshim; bd->write_rshim = rshim_usb_write_rshim; bd->has_reprobe = 1; pthread_mutex_init(&bd->mutex, NULL); } rshim_ref(bd); bd->dev = usb_dev; dev->handle = handle; switch (desc->idProduct) { case USB_BLUEFIELD_2_PRODUCT_ID: bd->ver_id = RSHIM_BLUEFIELD_2; bd->regs = &bf1_bf2_rshim_regs; break; case USB_BLUEFIELD_3_PRODUCT_ID: bd->ver_id = RSHIM_BLUEFIELD_3; bd->regs = &bf3_rshim_regs; break; default: bd->ver_id = RSHIM_BLUEFIELD_1; bd->regs = &bf1_bf2_rshim_regs; } bd->rev_id = desc->bcdDevice; if (rshim_has_usb_reset_delay || bd->ver_id < RSHIM_BLUEFIELD_3) bd->reset_delay = rshim_usb_reset_delay; else bd->reset_delay = 1; /* minimum delay for BF3 */ if (!dev->intr_buf) { dev->intr_buf = calloc(1, sizeof(*dev->intr_buf)); if (dev->intr_buf != NULL) *dev->intr_buf = 0; } if (!dev->read_or_intr_urb) dev->read_or_intr_urb = libusb_alloc_transfer(0); if (!dev->write_urb) dev->write_urb = libusb_alloc_transfer(0); if (!dev->read_or_intr_urb || !dev->write_urb) { RSHIM_ERR("can't allocate buffers or urbs\n"); goto error; } pthread_mutex_lock(&bd->mutex); for (i = 0; i < config->bNumInterfaces; i++) { interface = &config->interface[i]; if (interface->num_altsetting <= 0) continue; iface_desc = &interface->altsetting[0]; if (iface_desc->bInterfaceSubClass == 0) { RSHIM_DBG("Found rshim interface\n"); /* * We only expect one endpoint here, just make sure its * attributes match. */ if (iface_desc->bNumEndpoints != 1) { RSHIM_ERR("wrong number of endpoints for rshim interface\n"); pthread_mutex_unlock(&bd->mutex); goto error; } ep = &iface_desc->endpoint[0]; /* We expect a bulk out endpoint. */ if (!is_bulk_ep(ep) || is_in_ep(ep)) { pthread_mutex_unlock(&bd->mutex); goto error; } bd->has_rshim = 1; dev->boot_fifo_ep = ep_addr(ep); } else if (iface_desc->bInterfaceSubClass == 1) { RSHIM_DBG("Found tmfifo interface\n"); /* * We expect 3 endpoints here. Since they're listed in * random order we have to use their attributes to figure * out which is which. */ if (iface_desc->bNumEndpoints != 3) { RSHIM_ERR("wrong number of endpoints for tm interface\n"); pthread_mutex_unlock(&bd->mutex); goto error; } dev->tm_fifo_in_ep = 0; dev->tm_fifo_int_ep = 0; dev->tm_fifo_out_ep = 0; for (i = 0; i < iface_desc->bNumEndpoints; i++) { ep = &iface_desc->endpoint[i]; if (is_in_ep(ep)) { if (is_bulk_ep(ep)) { /* Bulk in endpoint. */ dev->tm_fifo_in_ep = ep_addr(ep); } else if (is_int_ep(ep)) { /* Interrupt in endpoint. */ dev->tm_fifo_int_ep = ep_addr(ep); } } else { if (is_bulk_ep(ep)) { /* Bulk out endpoint. */ dev->tm_fifo_out_ep = ep_addr(ep); } } } if (!dev->tm_fifo_in_ep || !dev->tm_fifo_int_ep || !dev->tm_fifo_out_ep) { RSHIM_ERR("could not find all required endpoints for tm interface\n"); pthread_mutex_unlock(&bd->mutex); goto error; } bd->has_tm = 1; } else { pthread_mutex_unlock(&bd->mutex); goto error; } } /* * Register rshim here since it needs to detect whether other backend * has already registered or not, which involves reading/writting rshim * registers and has assumption that the under layer is working. */ rc = rshim_register(bd); if (rc) { pthread_mutex_unlock(&bd->mutex); goto error; } /* Notify that device is attached. */ rc = rshim_notify(bd, RSH_EVENT_ATTACH, 0); pthread_mutex_unlock(&bd->mutex); if (rc) goto error; rshim_unlock(); return 0; error: if (dev) { libusb_free_transfer(dev->read_or_intr_urb); dev->read_or_intr_urb = NULL; libusb_free_transfer(dev->write_urb); dev->write_urb = NULL; free(dev->intr_buf); dev->intr_buf = NULL; rshim_deref(bd); } rshim_unlock(); return rc; } static void rshim_usb_disconnect(struct libusb_device *usb_dev) { rshim_backend_t *bd; rshim_usb_t *dev; if (rshim_trylock()) { RSHIM_ERR("rshim_trylock failed\n"); return; } bd = rshim_find_by_dev(usb_dev); if (!bd) { rshim_unlock(); return; } dev = container_of(bd, rshim_usb_t, bd); rshim_notify(bd, RSH_EVENT_DETACH, 0); /* * Clear this interface so we don't unregister our devices next * time. */ if (pthread_mutex_trylock(&bd->mutex)) { rshim_unlock(); return; } bd->has_rshim = 0; /* * We must make sure the console worker isn't running * before we free all these resources, and particularly * before we decrement our usage count, below. Most of the * time, if it's even enabled, it'll be scheduled to run at * some point in the future, and we can take care of that * by asking that it be canceled. * * However, it's possible that it's already started * running, but can't make progress because it's waiting * for the device mutex, which we currently have. We * handle this case by clearing the bit that says it's * enabled. The worker tests this bit as soon as it gets * the mutex, and if it's clear, it just returns without * rescheduling itself. Note that if we didn't * successfully cancel it, we flush the work entry below, * after we drop the mutex, to be sure it's done before we * decrement the device usage count. * * XXX This might be racy; what if something else which * would enable the worker runs after we drop the mutex * but before the worker itself runs? */ bd->has_cons_work = 0; libusb_cancel_transfer(dev->read_or_intr_urb); dev->read_or_intr_urb = NULL; libusb_cancel_transfer(dev->write_urb); dev->write_urb = NULL; free(dev->intr_buf); dev->intr_buf = NULL; if (!bd->has_rshim && !bd->has_tm) RSHIM_INFO("USB disconnected\n"); else RSHIM_INFO("USB partially disconnected\n"); pthread_mutex_unlock(&bd->mutex); if (dev->handle) { libusb_close(dev->handle); dev->handle = NULL; } rshim_deref(bd); rshim_unlock(); } static int rshim_usb_add_poll(libusb_context *ctx) { const struct libusb_pollfd **usb_pollfd = libusb_get_pollfds(ctx); struct epoll_event event; int i = 0, rc = -ENODEV; if (!usb_pollfd) return rc; memset(&event, 0, sizeof(event)); while (usb_pollfd[i]) { event.data.fd = usb_pollfd[i]->fd; event.events = 0; #define RSHIM_CONVERT(flag) do { \ if (usb_pollfd[i]->events & flag) \ event.events |= E##flag; \ } while (0) RSHIM_CONVERT(POLLIN); RSHIM_CONVERT(POLLOUT); #ifdef __linux__ RSHIM_CONVERT(POLLRDNORM); RSHIM_CONVERT(POLLRDBAND); RSHIM_CONVERT(POLLWRNORM); RSHIM_CONVERT(POLLWRBAND); RSHIM_CONVERT(POLLWRBAND); #endif RSHIM_CONVERT(POLLERR); RSHIM_CONVERT(POLLHUP); #undef RSHIM_CONVERT rc = epoll_ctl(rshim_usb_epoll_fd, EPOLL_CTL_ADD, usb_pollfd[i]->fd, &event); if (rc == -1 && errno != EEXIST) RSHIM_ERR("epoll_ctl failed; %m\n"); i++; } free(usb_pollfd); /* Notify the polling thread. */ rshim_work_signal(NULL); return rc; } #if LIBUSB_API_VERSION >= 0x01000102 static libusb_hotplug_callback_handle rshim_hotplug_handle; static int rshim_hotplug_callback(struct libusb_context *ctx, struct libusb_device *dev, libusb_hotplug_event event, void *user_data) { switch (event) { case LIBUSB_HOTPLUG_EVENT_DEVICE_ARRIVED: /* * The probe function would send control packet which could cause race * condition when calling from the hotplug callback function. Thus set * a flag and do it later in the main loop. */ RSHIM_INFO("USB device detected\n"); rshim_usb_need_probe = true; break; case LIBUSB_HOTPLUG_EVENT_DEVICE_LEFT: RSHIM_INFO("USB device leaving\n"); rshim_usb_disconnect(dev); break; default: break; } rshim_usb_add_poll(ctx); return 0; /* keep filter registered */ } #endif static bool rshim_usb_probe(void) { libusb_context *ctx = rshim_usb_ctx; libusb_device **devs, *dev; int rc, i = 0, j, num; rc = libusb_get_device_list(ctx, &devs); if (rc < 0) { RSHIM_ERR("USB Get Device Error\n"); return false; } while ((dev = devs[i++]) != NULL) { struct libusb_device_descriptor desc; rc = libusb_get_device_descriptor(dev, &desc); if (rc) continue; if (desc.idVendor != USB_TILERA_VENDOR_ID) continue; num = sizeof(rshim_usb_product_ids) / sizeof(rshim_usb_product_ids[0]); for (j = 0; j < num; j++) { if (desc.idProduct == rshim_usb_product_ids[j]) rshim_usb_probe_one(ctx, dev, &desc); } } rc = rshim_usb_add_poll(ctx); if (rc) return false; return true; } int rshim_usb_init(int epoll_fd) { libusb_context *ctx = NULL; int rc, i, num; rc = libusb_init(&ctx); if (rc < 0) { RSHIM_ERR("USB Init Error: %m\n"); return rc; } #ifdef LIBUSB_LOG_LEVEL_ERROR if (rshim_log_level > LOG_ERR) libusb_set_debug(ctx, LIBUSB_LOG_LEVEL_ERROR); #endif rshim_usb_ctx = ctx; rshim_usb_epoll_fd = epoll_fd; #if LIBUSB_API_VERSION >= 0x01000102 num = sizeof(rshim_usb_product_ids) / sizeof(rshim_usb_product_ids[0]); for (i = 0; i < num; i++) { /* * Register ARRIVED and LEFT separately to avoid the coverity 'mixed enum' * warnings. */ rc = libusb_hotplug_register_callback(ctx, LIBUSB_HOTPLUG_EVENT_DEVICE_ARRIVED, LIBUSB_HOTPLUG_ENUMERATE, USB_TILERA_VENDOR_ID, rshim_usb_product_ids[i], LIBUSB_HOTPLUG_MATCH_ANY, rshim_hotplug_callback, NULL, &rshim_hotplug_handle); if (rc != LIBUSB_SUCCESS) { RSHIM_ERR("failed to register hotplug callback\n"); libusb_exit(ctx); rshim_usb_ctx = NULL; return rc; } rc = libusb_hotplug_register_callback(ctx, LIBUSB_HOTPLUG_EVENT_DEVICE_LEFT, LIBUSB_HOTPLUG_ENUMERATE, USB_TILERA_VENDOR_ID, rshim_usb_product_ids[i], LIBUSB_HOTPLUG_MATCH_ANY, rshim_hotplug_callback, NULL, &rshim_hotplug_handle); if (rc != LIBUSB_SUCCESS) { RSHIM_ERR("failed to register hotplug callback\n"); libusb_exit(ctx); rshim_usb_ctx = NULL; return rc; } } #else rshim_usb_probe(); #endif return 0; } void rshim_usb_poll(bool blocking) { bool found = false; struct timeval tv = {0, 0}; if (!rshim_usb_ctx) return; if (rshim_usb_need_probe) { rshim_usb_need_probe = false; __sync_synchronize(); found = rshim_usb_probe(); } if (blocking && !found) { /* Handle USB events (like hot-plugin) in blocking mode. */ libusb_handle_events_completed(rshim_usb_ctx, NULL); /* Wake up the main thread. */ rshim_work_signal(NULL); } else { libusb_handle_events_timeout_completed(rshim_usb_ctx, &tv, NULL); } } ./Mellanox-rshim-user-space-0fd5d3f/src/rshim.h0000664000175000017500000004274614563673752020737 0ustar tai271828tai271828/* SPDX-License-Identifier: GPL-2.0 or BSD-3-Clause */ /* * Copyright (C) 2019-2023 Mellanox Technologies. All Rights Reserved. * */ #ifndef _RSHIM_H #define _RSHIM_H #ifdef __linux__ #include #else #include #endif #include #include #ifdef __linux__ #include #else #define VIRTIO_ID_NET 1 #define VIRTIO_ID_CONSOLE 3 #endif #include #include #include #include #include #include #include #ifdef HAVE_CONFIG_H #include #endif #ifdef HAVE_SYSLOG_H #include #endif #include "rshim_regs.h" /* Global variables. */ extern int rshim_log_level; extern bool rshim_daemon_mode; extern int rshim_drop_mode; extern int rshim_usb_reset_delay; extern bool rshim_has_usb_reset_delay; extern int rshim_pcie_reset_delay; extern bool rshim_has_pcie_reset_delay; extern int rshim_pcie_intr_poll_interval; extern int rshim_pcie_enable_vfio; extern int rshim_pcie_enable_uio; #ifndef offsetof #define offsetof(TYPE, MEMBER) ((size_t)&((TYPE *)0)->MEMBER) #endif #ifndef container_of #define container_of(ptr, type, member) ({ \ void *__mptr = (void *)(ptr); \ ((type *)(__mptr - offsetof(type, member))); }) #endif #ifdef HAVE_SYSLOG_H #define RSHIM_SYSLOG(level, fmt...) syslog(level, fmt) #else #define LOG_ERR 1 #define LOG_WARNING 2 #define LOG_NOTICE 3 #define LOG_DEBUG 4 #define RSHIM_SYSLOG(level, fmt...) #endif #define RSHIM_LOG(log_level, fmt...) do { \ if (rshim_log_level >= log_level) { \ if (rshim_daemon_mode) \ RSHIM_SYSLOG(log_level, fmt); \ else \ printf(fmt); \ } \ } while (0) #define RSHIM_ERR(fmt...) RSHIM_LOG(LOG_ERR, fmt) #define RSHIM_WARN(fmt...) RSHIM_LOG(LOG_WARNING, fmt) #define RSHIM_INFO(fmt...) RSHIM_LOG(LOG_NOTICE, fmt) #define RSHIM_DBG(fmt...) RSHIM_LOG(LOG_DEBUG, fmt) /* Spin flag values. */ #define RSH_SFLG_READING 0x1 /* read is active. */ #define RSH_SFLG_WRITING 0x2 /* write_urb is active. */ #define RSH_SFLG_CONS_OPEN 0x4 /* console stream is open. */ /* * Buffer/FIFO sizes. Note that the FIFO sizes must be powers of 2; also, * the read and write buffers must be no larger than the corresponding * FIFOs. */ #define READ_BUF_SIZE 2048 #define WRITE_BUF_SIZE 2048 #define READ_FIFO_SIZE (4 * 1024) #define WRITE_FIFO_SIZE (4 * 1024) #define BOOT_BUF_SIZE (16 * 1024) #define BF3_MAX_BOOT_FIFO_SIZE 8192 /* bytes */ #define RSHIM_BAD_CTRL_REG(v) \ (((v) == 0xbad00acce55) || ((v) == (uint64_t)-1) || ((v) == 0xbadacce55)) /* Sub-device types. */ enum { RSH_DEV_TYPE_RSHIM, RSH_DEV_TYPE_BOOT, RSH_DEV_TYPE_TMFIFO, RSH_DEV_TYPE_MISC, RSH_DEV_TYPES }; /* Event types used in rshim_notify(). */ enum { RSH_EVENT_FIFO_INPUT, /* fifo ready for input */ RSH_EVENT_FIFO_OUTPUT, /* fifo ready for output */ RSH_EVENT_FIFO_ERR, /* fifo error */ RSH_EVENT_ATTACH, /* backend attaching */ RSH_EVENT_DETACH, /* backend detaching */ }; /* Internal message types in addition to the standard VIRTIO_ID_xxx types. */ enum { TMFIFO_MSG_VLAN_ID = 0xFB, /* vlan id */ TMFIFO_MSG_PXE_ID = 0xFC, /* pxe client identifier */ TMFIFO_MSG_CTRL_REQ = 0xFD, /* ctrl request */ TMFIFO_MSG_MAC_1 = 0xFE, /* mac[0:2] */ TMFIFO_MSG_MAC_2 = 0xFF, /* mac[3:4] */ }; /* TMFIFO message header. */ typedef union { struct { uint8_t type; /* message type */ uint16_t len; /* payload length in network order */ union { uint8_t mac[3]; /* 3-bytes of the MAC address */ uint32_t pxe_id; /* pxe identifier in network order */ uint16_t vlan[2]; /* up to two vlan id */ }; uint8_t checksum; /* header checksum */ } __attribute__((packed)); uint64_t data; } rshim_tmfifo_msg_hdr_t; /* TMFIFO demux channels. */ enum { TMFIFO_CONS_CHAN, /* Console */ TMFIFO_NET_CHAN, /* Network */ TMFIFO_MAX_CHAN /* Number of channels */ }; #define RSH_BYTE_ACC_READ_TRIGGER 0x50 #define RSH_BYTE_ACC_SIZE_4BYTE 0x10 #define RSH_BYTE_ACC_PENDING 0x20 #define BOOT_CHANNEL RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_BOOT #define RSHIM_CHANNEL RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_RSHIM #define MMC_CHANNEL RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_MMC #define YU_CHANNEL RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_TYU #define UART0_CHANNEL RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_UART0 #define UART1_CHANNEL RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_UART1 #define DIAGUART_CHANNEL RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_DIAG_UART #define OOB_CHANNEL RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_TYU_EXT1 #define TIMER_ARM_CHANNEL RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_TIMER #define RSH_HUB_CHANNEL RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_USB #define TIMER_EXT_CHANNEL RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_TIMER_EXT #define WDOG0_CHANNEL RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_WDOG0 #define WDOG1_CHANNEL RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_WDOG1 #define GIC_CHANNEL RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_TYU_EXT2 #define MCH_CORE_CHANNEL RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_TYU_EXT3 /* Base RShim Address */ #define RSH_BASE_ADDR 0x80000000 #define RSH_CHANNEL_BASE(chan) (RSH_BASE_ADDR | (chan << 16)) #define RSH_BOOT_FIFO_SIZE 512 #define RSH_TM_FIFO_SIZE 256 /* Retry time in seconds. */ #define RSHIM_LOCK_RETRY_TIME 2 /* YU boot record OPN offset/size */ #define RSHIM_YU_BASE_ADDR 0x2800000 #define RSHIM_YU_BOOT_RECORD_OPN 0xfd8 #define RSHIM_YU_BOOT_RECORD_OPN_SIZE 16 #define RSHIM_YU_BF3_BOOT_RECORD_OPN 0x9bdc #define YU_RESET_MODE_TRIGGER 0x0011 #define YU_BOOT_DEVID 0x0014 #define YU_BOOT 0x0068 #define YU_POWER_CLK_DELAY 0x0c24 #define YU_RESET_8_CLK_EN 0x2800 #define YU_RESET_13_CLK_EN 0x2d00 #define YU_CLK_EN_COUNT 16 #define YU_RESET_8_RESET_EN 0x2840 #define YU_RESET_13_RESET_EN 0x2d40 #define YU_RESET_EN_COUNT 8 #define YU_RESET_8_POWERDOWN 0x2860 #define YU_RESET_13_POWERDOWN 0x2d60 #define YU_POWERDOWN_COUNT 8 #define YU_RESET_ACTIVATION_13 0x30b4 #define YU_MAIN_CLK_GATE_EN 0x30c8 /* FIFO structure. */ typedef struct { unsigned char *data; unsigned int head; unsigned int tail; pthread_cond_t operable; } rshim_fifo_t; /* RShim network packet. */ #define ETH_PKT_SIZE 1536 typedef struct { rshim_tmfifo_msg_hdr_t hdr; /* message header */ char buf[ETH_PKT_SIZE]; /* packet buffer */ } rshim_net_pkt_t; #define RSHIM_DEV_NAME_LEN 64 /* Bluefield Version. */ #define RSHIM_BLUEFIELD_1 1 #define RSHIM_BLUEFIELD_2 2 #define RSHIM_BLUEFIELD_3 3 /* Bluefield Revision ID. */ #define BLUEFIELD_REV0 0 #define BLUEFIELD_REV1 1 /* RShim backend. */ typedef struct rshim_backend rshim_backend_t; struct rshim_backend { /* Device name. */ char dev_name[RSHIM_DEV_NAME_LEN]; /* Backend device. */ void *dev; /* BlueField version / revision. */ uint16_t ver_id; uint16_t rev_id; /* FUSE sessions & poll handles. */ void *fuse_session[RSH_DEV_TYPES]; pthread_t fuse_thread[RSH_DEV_TYPES]; void *fuse_poll_handle[TMFIFO_MAX_CHAN]; /* Networking handler and packets. */ int net_fd, net_notify_fd[2]; rshim_net_pkt_t net_rx_pkt; rshim_net_pkt_t net_tx_pkt; int net_tx_len; int net_rx_len; bool net_rx_pending; /* State flags. */ uint32_t is_booting : 1; /* Waiting for device to come back. */ uint32_t is_boot_open : 1; /* Boot device is open. */ uint32_t is_net_open : 1; /* Network device is open. */ uint32_t is_cons_open : 1; /* Console device is open. */ uint32_t is_attach : 1; /* Service ready to attach. */ uint32_t is_in_boot_write : 1; /* A thread is in boot_write(). */ uint32_t has_cons_work : 1; /* Console worker thread running. */ uint32_t has_debug : 1; /* Debug enabled for this device. */ uint32_t has_tm : 1; /* TM FIFO found. */ uint32_t has_rshim : 1; /* RSHIM found. */ uint32_t has_fifo_work : 1; /* FIFO output to be done in worker. */ uint32_t has_reprobe : 1; /* Reprobe support after SW reset. */ uint32_t drop_pkt : 1; /* Drop the rest of the packet. */ uint32_t registered : 1; /* Backend has been registered. */ uint32_t keepalive : 1; /* A flag to update keepalive. */ uint32_t peer_ctrl_req : 1; /* A flag to send ctrl request. */ uint32_t peer_ctrl_resp : 1; /* A flag to indicate MAC response rcvd. */ uint32_t peer_mac_set : 1; /* A flag to send MAC-set request. */ uint32_t peer_pxe_id_set : 1; /* A flag to send pxe-id-set request. */ uint32_t peer_vlan_set : 1; /* A flag to set vlan IDs. */ uint32_t drop_mode : 1; /* A flag to drop all input/output. */ uint32_t skip_boot_reset : 1; /* Skip SW_RESET while pushing boot stream. */ /* reference count. */ volatile int ref; /* Last keepalive time. */ int last_keepalive; int net_init_time; /* timer. */ int timer; /* Last boot write time. */ time_t boot_write_time; /* State flag bits from RSH_SFLG_xxx (see above). */ int spin_flags; /* Total bytes in the read buffer. */ int read_buf_bytes; /* Offset of next unread byte in the read buffer. */ int read_buf_next; /* Bytes left in the current packet, or 0 if no current packet. */ int read_buf_pkt_rem; /* Padded bytes in the read buffer. */ int read_buf_pkt_padding; /* Bytes left in the current packet pending to write. */ int write_buf_pkt_rem; /* Current message header. */ rshim_tmfifo_msg_hdr_t msg_hdr; /* Read FIFOs. */ rshim_fifo_t read_fifo[TMFIFO_MAX_CHAN]; /* Write FIFOs. */ rshim_fifo_t write_fifo[TMFIFO_MAX_CHAN]; /* Read buffer. */ unsigned char *read_buf; /* Write buffer. */ unsigned char *write_buf; /* Current Tx FIFO channel. */ int tx_chan; /* Current Rx FIFO channel. */ int rx_chan; /* First error encountered during read or write. */ int tmfifo_error; /* Buffers used for boot writes. Allocated at startup. */ char *boot_buf[2]; /* Buffer to store the remaining data when it's not 8B unaligned. */ uint8_t boot_rem_cnt; uint64_t boot_rem_data; /* Debug code passed in scratchpad1. */ uint64_t debug_code; /* * This mutex is used to prevent the interface pointers and the * device pointer from disappearing while a driver entry point * is using them. It's held throughout a read or write operation * (at least the parts of those operations which depend upon those * pointers) and is also held whenever those pointers are modified. * It also protects state flags, and boot_complete_cond. */ pthread_mutex_t mutex; /* Mutex to protect the ring buffer. */ pthread_mutex_t ringlock; bool work_pending; /* We'll signal completion on this when FLG_BOOTING is turned off. */ pthread_cond_t boot_complete_cond; /* * This wait queue supports fsync; it's woken up whenever an * outstanding USB write URB is done. This will need to be more * complex if we start doing write double-buffering. */ pthread_cond_t fifo_write_complete_cond; /* State for our outstanding boot write. */ pthread_cond_t boot_write_complete_cond; /* Wait condition for control messages. */ pthread_cond_t ctrl_wait_cond; /* Current termios settings for the console. */ struct termios cons_termios; /* Pending boot & fifo request for the worker. */ uint8_t *boot_work_buf; uint32_t boot_work_buf_len; uint32_t boot_work_buf_actual_len; uint8_t *fifo_work_buf; uint32_t fifo_work_buf_len; int fifo_work_devtype; /* Number of open console files. */ long console_opens; /* Index in rshim_devs[]. */ int index; /* Display level in the misc output. */ int display_level; /* Boot timeout in seconds. */ int boot_timeout; /* Delay after reset. */ int reset_delay; /* Configured MAC address of the peer-side. */ uint8_t peer_mac[6]; /* Configured PXE client identifier. */ uint32_t pxe_client_id; /* Up to two VLAN IDs for PXE purpose. */ uint16_t vlan[2]; /* APIs provided by backend. */ /* API to write bulk data to RShim via the backend. */ ssize_t (*write)(rshim_backend_t *bd, int devtype, const char *buf, size_t count); /* API to read bulk data from RShim via the backend. */ ssize_t (*read)(rshim_backend_t *bd, int devtype, char *buf, size_t count); /* API to cancel a read / write request (optional). */ void (*cancel)(rshim_backend_t *bd, int devtype, bool is_write); /* API to destroy the backend. */ void (*destroy)(rshim_backend_t *bd); /* API to read bytes from RShim. */ int (*read_rshim)(rshim_backend_t *bd, uint32_t chan, uint32_t addr, uint64_t *value, int size); /* API to write bytes to RShim. */ int (*write_rshim)(rshim_backend_t *bd, uint32_t chan, uint32_t addr, uint64_t value, int size); /* API to enable the device. */ int (*enable_device)(rshim_backend_t *bd, bool enable); /* Platform specific register addresses */ const struct rshim_regs *regs; }; #define RSHIM_REG_SIZE_4B 4 #define RSHIM_REG_SIZE_8B 8 struct rshim_regs { uint32_t boot_fifo_data; uint32_t boot_fifo_count; uint32_t boot_fifo_count_mask; uint32_t boot_control; uint32_t reset_control; uint32_t scratchpad1; uint32_t scratchpad6; uint32_t tm_htt_sts; uint32_t tm_tth_sts; uint32_t tm_htt_data; uint32_t tm_tth_data; uint32_t semaphore0; uint32_t mem_acc_ctl; uint32_t mem_acc_rsp_cnt; uint32_t mem_acc_data_first_word; uint32_t device_mstr_priv_lvl; uint32_t device_mstr_priv_lvl_shift; uint32_t fabric_dim; uint32_t uptime; uint32_t uptime_por; uint32_t arm_wdg_control_wcs; uint32_t scratch_buf_dat; uint32_t scratch_buf_ctl; }; extern const struct rshim_regs bf1_bf2_rshim_regs; extern const struct rshim_regs bf3_rshim_regs; /* Global variables. */ extern int rshim_epoll_fd; extern volatile bool rshim_run; /* Common APIs. */ /* Register/unregister backend. */ int rshim_register(rshim_backend_t *bd); void rshim_deregister(rshim_backend_t *bd); /* Find backend by name. */ rshim_backend_t *rshim_find_by_name(char *dev_name); /* Find backend by device. */ rshim_backend_t *rshim_find_by_dev(void *dev); /* RShim global lock. */ void rshim_lock(void); int rshim_trylock(void); void rshim_unlock(void); /* Event notification. */ int rshim_notify(rshim_backend_t *bd, int event, int code); /* * FIFO APIs. * * FIFO is demuxed into two channels, one for network interface * (TMFIFO_NET_CHAN), one for console (TMFIFO_CONS_CHAN). */ /* Write / read some bytes to / from the FIFO via the backend. */ ssize_t rshim_fifo_read(rshim_backend_t *bd, char *buffer, size_t count, int chan, bool nonblock); ssize_t rshim_fifo_write(rshim_backend_t *bd, const char *buffer, size_t count, int chan, bool nonblock); /* Alloc/free the FIFO. */ int rshim_fifo_alloc(rshim_backend_t *bd); void rshim_fifo_free(rshim_backend_t *bd); /* Console APIs. */ /* Enable early console. */ int rshim_cons_early_enable(rshim_backend_t *bd); /* Network APIs. */ #ifdef HAVE_RSHIM_NET int rshim_net_init(rshim_backend_t *bd); int rshim_net_del(rshim_backend_t *bd); void rshim_net_rx(rshim_backend_t *bd); void rshim_net_tx(rshim_backend_t *bd); #else static inline int rshim_net_init(rshim_backend_t *bd) { return 0; } static inline int rshim_net_del(rshim_backend_t *bd) { return 0; } static inline void rshim_net_rx(rshim_backend_t *bd) { } static inline void rshim_net_tx(rshim_backend_t *bd) { } #endif void rshim_ref(rshim_backend_t *bd); void rshim_deref(rshim_backend_t *bd); int rshim_boot_open(rshim_backend_t *bd); int rshim_boot_write(rshim_backend_t *bd, const char *user_buffer, size_t count, int (*copy_in)(void *dest, const void *src, int count)); void rshim_boot_release(rshim_backend_t *bd); int rshim_console_open(rshim_backend_t *bd); int rshim_console_release(rshim_backend_t *bd, void (*poll_handle_destroy)(rshim_backend_t *bd, int chan)); void rshim_fifo_check_poll(rshim_backend_t *bd, int chan, bool *poll_rx, bool *poll_tx, bool *poll_err); int rshim_fifo_size(rshim_backend_t *bd, int chan, bool is_rx); void rshim_sig_hup(int sig); void rshim_fifo_reset(rshim_backend_t *bd); int rshim_reset_control(rshim_backend_t *bd); void rshim_work_signal(rshim_backend_t *bd); int rshim_fifo_fsync(rshim_backend_t *bd, int chan); /* Display the rshim logging buffer. */ int rshim_log_show(rshim_backend_t *bd, char *buf, int len); bool rshim_allow_device(const char *devname); /* USB backend APIs. */ #ifdef HAVE_RSHIM_USB int rshim_usb_init(int epoll_fd); void rshim_usb_poll(bool blocking); #else static inline int rshim_usb_init(int epoll_fd) { return -1; } static inline void rshim_usb_poll(bool blocking) { (void)blocking; } #endif /* PCIe & PCIe livefish backend APIs. */ #ifdef HAVE_RSHIM_PCIE int rshim_pcie_init(void); int rshim_pcie_lf_init(void); #else static inline int rshim_pcie_init(void) { return -1; } static inline int rshim_pcie_lf_init(void) { return -1; } #endif #ifdef HAVE_RSHIM_FUSE int rshim_fuse_init(rshim_backend_t *bd); int rshim_fuse_del(rshim_backend_t *bd); void rshim_fuse_input_notify(rshim_backend_t *bd); int rshim_fuse_got_peer_signal(void); #endif /* * Get/Set the OPN string from the YU boot record, which means setting * the value only persists during warm resets. */ int rshim_get_opn(rshim_backend_t *bd, char *opn, int len); int rshim_set_opn(rshim_backend_t *bd, const char *opn, int len); /* Check whether rshim backend is accessible or not. */ int rshim_access_check(rshim_backend_t *bd); #endif /* _RSHIM_H */ ./Mellanox-rshim-user-space-0fd5d3f/src/rshim_pcie.c0000664000175000017500000010443214563673752021721 0ustar tai271828tai271828// SPDX-License-Identifier: GPL-2.0-only /* * Copyright (C) 2019 Mellanox Technologies. All Rights Reserved. * */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef __FreeBSD__ #include #include #include #endif #ifdef __linux__ #include #include #include #include #include #include #endif #include "rshim.h" /* Our Vendor/Device IDs. */ #define TILERA_VENDOR_ID 0x15b3 #define BLUEFIELD1_DEVICE_ID 0xc2d2 #define BLUEFIELD2_DEVICE_ID 0xc2d3 #define BLUEFIELD3_DEVICE_ID 0xc2d4 #define BLUEFIELD3_DEVICE_ID2 0xc2d5 /* The size the RShim region. */ #define PCI_RSHIM_WINDOW_SIZE 0x100000 #define BF3_PCI_RSHIM_WINDOW_SIZE 0x800000 #define VFIO_GET_REGION_ADDR(x) ((uint64_t) x << 40ULL) #define SYS_CLASS_IOMMU_PATH "/sys/class/iommu" #define SYS_CLASS_VFIO_PCI_PATH "/sys/module/vfio_pci" #define SYS_BUS_PCI_PATH "/sys/bus/pci/devices" #define SYS_VFIO_PCI_PATH "/sys/bus/pci/drivers/vfio-pci" #define SYS_CLASS_UIO_PCI_PATH "/sys/module/uio_pci_generic" #define SYS_UIO_PCI_PATH "/sys/bus/pci/drivers/uio_pci_generic" #define RSHIM_PCI_COMMAND (PCI_COMMAND_MEMORY | PCI_COMMAND_MASTER) #define RSHIM_PATH_MAX 256 #define RSHIM_CMD_MAX 256 #define RSHIM_PCIE_NIC_RESET_WAIT 2 #define RSHIM_PCIE_NIC_IRQ_RATE 32 /* Different modes of memory map. */ typedef enum { RSHIM_PCIE_MMAP_DIRECT, RSHIM_PCIE_MMAP_UIO, RSHIM_PCIE_MMAP_VFIO } rshim_pcie_mmap_mode_t; /* Reset state stored in scratchpad6. */ enum { RSHIM_PCIE_RST_STATE_NONE, RSHIM_PCIE_RST_STATE_REQUEST, RSHIM_PCIE_RST_STATE_START, RSHIM_PCIE_RST_STATE_ABORT }; /* Reset reply stored in scratchpad6. */ enum { RSHIM_PCIE_RST_REPLY_NONE, RSHIM_PCIE_RST_REPLY_ACK, RSHIM_PCIE_RST_REPLY_NACK, RSHIM_PCIE_RST_START_ACK, }; /* Reset type stored in scratchpad6. */ enum { RSHIM_PCIE_RST_TYPE_NONE, RSHIM_PCIE_RST_TYPE_DPU_RESET, RSHIM_PCIE_RST_TYPE_NIC_RESET }; /* Min delay in seconds after RSHIM_PCIE_RST_START_ACK */ #define RSHIM_PCIE_RST_START_MIN_DELAY 4 /* Interrupt information between NIC_FW and rshim driver. */ typedef union { struct { uint64_t unused_0 : 32; /* RSHIM_PCIE_RST_STATE_xxx (RO) */ uint64_t rst_state : 4; uint64_t unused_1 : 2; /* RSHIM_PCIE_RST_REPLY_xxx (WO) */ uint64_t rst_reply : 2; /* 10’s of ms between the PCI link disable and PCI link enable (RO) */ uint64_t rst_downtime : 8; uint64_t unused_2 : 4; /* RSHIM_PCIE_RST_TYPE_xxx */ uint64_t rst_type : 3; uint64_t unused_3 : 9; }; uint64_t word; } rshim_pcie_intr_info_t; static rshim_pcie_mmap_mode_t rshim_pcie_mmap_mode = RSHIM_PCIE_MMAP_DIRECT; static const char *rshim_pcie_mmap_name[] = {"direct", "uio", "vfio"}; #ifdef __linux__ static const char *rshim_sys_pci_path; static bool rshim_pcie_has_uio(void); #endif static inline uint64_t readq(const volatile void *addr) { uint64_t value = *(const volatile uint64_t *)addr; __sync_synchronize(); return value; } static inline void writeq(uint64_t value, volatile void *addr) { __sync_synchronize(); *(volatile uint64_t *)addr = value; } static inline uint32_t readl(const volatile void *addr) { uint32_t value = *(const volatile uint32_t *)addr; __sync_synchronize(); return value; } static inline void writel(uint32_t value, volatile void *addr) { __sync_synchronize(); *(volatile uint32_t *)addr = value; } typedef struct { /* RShim backend structure. */ rshim_backend_t bd; /* Device info */ int domain; uint16_t device_id; uint8_t bus; uint8_t dev; uint8_t func; /* Address of the RShim registers. */ volatile uint8_t *rshim_regs; /* Keep track of number of 8-byte word writes */ uint8_t write_count; /* Device file handle */ int device_fd; /* VFIO container/group file handle */ int container_fd; int group_fd; /* Interrupt handle and read length */ volatile int intr_fd; volatile int intr_reset_seq; uint32_t intr_len; /* Interrupt thread */ pthread_t intr_thread; /* State to indicate NIC is resetting. */ volatile bool nic_reset; /* Last irq time */ time_t last_intr_time; /* Number of interrupts since last irq time */ uint32_t intr_cnt; /* Memory map and PCI sysfs path. */ int mmap_mode; const char *pci_path; /* BAR size */ uint32_t bar_size; } rshim_pcie_t; static const int bf3_rshim_pcie_chan_map[] = { [RSHIM_CHANNEL] = 0, [UART0_CHANNEL] = 0x10000, [UART1_CHANNEL] = 0x11000, [DIAGUART_CHANNEL] = 0x12000, [RSH_HUB_CHANNEL] = 0x12400, [WDOG0_CHANNEL] = 0x20000, [WDOG1_CHANNEL] = 0x40000, [MCH_CORE_CHANNEL] = 0x60000, [TIMER_ARM_CHANNEL] = 0x80000, [TIMER_EXT_CHANNEL] = 0xa0000, [OOB_CHANNEL] = 0xa1000, [YU_CHANNEL] = 0x400000, }; static bool rshim_is_bluefield1(uint16_t device_id) { return (device_id == BLUEFIELD1_DEVICE_ID); } static bool rshim_is_bluefield2(uint16_t device_id) { return (device_id == BLUEFIELD2_DEVICE_ID); } static bool rshim_is_bluefield3(uint16_t device_id) { return ((device_id == BLUEFIELD3_DEVICE_ID) || (device_id == BLUEFIELD3_DEVICE_ID2)); } #ifdef __linux__ static int rshim_pcie_enable_irq(rshim_pcie_t *dev, bool enable); static uint16_t rshim_pci_read_word(rshim_pcie_t *dev, int pos) { char path[RSHIM_PATH_MAX]; uint16_t data = 0xFFFF; int fd; snprintf(path, sizeof(path), "%s/%04x:%02x:%02x.%1u/config", SYS_BUS_PCI_PATH, dev->domain, dev->bus, dev->dev, dev->func); fd = open(path, O_RDWR | O_SYNC); if (fd != -1) { if (pread(fd, &data, sizeof(data), pos) != sizeof(data)) { data = 0xFFFF; } data = le16toh(data); close(fd); } else { RSHIM_WARN("Unable to open %s\n", path); } return data; } int rshim_pci_write_word(rshim_pcie_t *dev, int pos, uint16_t data) { char path[RSHIM_PATH_MAX]; int fd, len = 0; snprintf(path, sizeof(path), "%s/%04x:%02x:%02x.%1u/config", SYS_BUS_PCI_PATH, dev->domain, dev->bus, dev->dev, dev->func); fd = open(path, O_RDWR | O_SYNC); if (fd != -1) { data = htole16(data); len = pwrite(fd, &data, sizeof(data), pos); close(fd); } else { RSHIM_WARN("Unable to open %s\n", path); } return len; } /* Release pcie resource. */ static void rshim_pcie_mmap_release(rshim_pcie_t *dev) { volatile void *ptr; rshim_pcie_enable_irq(dev, false); ptr = dev->rshim_regs; if (ptr) { dev->rshim_regs = NULL; __sync_synchronize(); munmap((void *)ptr, dev->bar_size); } if (dev->device_fd >= 0) { close(dev->device_fd); dev->device_fd = -1; } if (dev->group_fd >= 0) { close(dev->group_fd); dev->group_fd = -1; } if (dev->container_fd >= 0) { close(dev->container_fd); dev->container_fd = -1; } } static void rshim_pcie_bind(rshim_pcie_t *dev, bool enable) { char cmd[RSHIM_CMD_MAX]; int rc; /* * Linux kernel prior 4.18 has a bug which could cause crash when uio is * unregistered (see commit 57c5f4df0a5a uio: fix crash after the device * is unregistered). Below is a workaround to avoid such crash for uio. * The rshim probing order is vfio->uio->direct. The uio unbind won't * affect the operation of direct mapping. */ if (!enable && (dev->mmap_mode == RSHIM_PCIE_MMAP_UIO)) return; if (dev->mmap_mode == RSHIM_PCIE_MMAP_VFIO || dev->mmap_mode == RSHIM_PCIE_MMAP_UIO) { if (!enable) { snprintf(cmd, sizeof(cmd), "echo %04x:%02x:%02x.%1u > %s/unbind 2>/dev/null", dev->domain, dev->bus, dev->dev, dev->func, dev->pci_path); if (system(cmd) == -1) RSHIM_DBG("Failed to unbind device\n"); } snprintf(cmd, sizeof(cmd), "echo '%x %x' > %s/%s 2>/dev/null", TILERA_VENDOR_ID, BLUEFIELD1_DEVICE_ID, dev->pci_path, enable ? "new_id" : "remove_id"); rc = system(cmd); if (rc == -1) RSHIM_DBG("Failed to write device id %m\n"); snprintf(cmd, sizeof(cmd), "echo '%x %x' > %s/%s 2>/dev/null", TILERA_VENDOR_ID, BLUEFIELD2_DEVICE_ID, dev->pci_path, enable ? "new_id" : "remove_id"); rc = system(cmd); if (rc == -1) RSHIM_DBG("Failed to write device id %m\n"); if (enable) { snprintf(cmd, sizeof(cmd), "echo %04x:%02x:%02x.%1u > %s/bind 2>/dev/null", dev->domain, dev->bus, dev->dev, dev->func, dev->pci_path); if (system(cmd) == -1) RSHIM_DBG("Failed to bind device\n"); } } else if (dev->mmap_mode == RSHIM_PCIE_MMAP_DIRECT) { if (enable) { snprintf(cmd, sizeof(cmd), "echo 1 > %s/%04x:%02x:%02x.%1u/enable", SYS_BUS_PCI_PATH, dev->domain, dev->bus, dev->dev, dev->func); if (system(cmd) == -1) RSHIM_DBG("Failed to enable pcie\n"); } /* * There is no driver in direct map mode. Set a faked driver name here * to prevent the "new_id" command from reassigning driver automatically * for this rshim PF. This is to avoid issues when there multiple rshim * devices exist with mixed mode. */ snprintf(cmd, sizeof(cmd), "echo %s > %s/%04x:%02x:%02x.%1u/driver_override 2>/dev/null", enable ? "rshim" : "", SYS_BUS_PCI_PATH, dev->domain, dev->bus, dev->dev, dev->func); if (system(cmd) == -1) RSHIM_DBG("Failed to enable pcie\n"); } } /* Memory map over sysfs. */ static int rshim_pcie_mmap_direct(rshim_pcie_t *dev) { char path[RSHIM_PATH_MAX]; uint16_t reg; snprintf(path, sizeof(path), "%s/%04x:%02x:%02x.%1u/resource0", SYS_BUS_PCI_PATH, dev->domain, dev->bus, dev->dev, dev->func); dev->device_fd = open(path, O_RDWR | O_SYNC); if (dev->device_fd < 0) { RSHIM_ERR("Failed to open %s\n", path); return -ENODEV; } dev->rshim_regs = mmap(NULL, dev->bar_size, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_LOCKED, dev->device_fd, 0); if (dev->rshim_regs == MAP_FAILED) { dev->rshim_regs = NULL; RSHIM_ERR("Failed to map RShim registers\n"); return -ENOMEM; } /* Set PCI bus mastering */ reg = rshim_pci_read_word(dev, PCI_COMMAND); if (reg != 0xFFFF) rshim_pci_write_word(dev, PCI_COMMAND, reg | RSHIM_PCI_COMMAND); return 0; } #define IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + sizeof(int)) struct { struct vfio_irq_set irq_set; int32_t intr_fd; } irq_set_buf; static int rshim_pcie_enable_irq(rshim_pcie_t *dev, bool enable) { struct vfio_irq_set *irq_set = (struct vfio_irq_set *) &irq_set_buf; int len, ret; uint16_t reg; if (dev->mmap_mode == RSHIM_PCIE_MMAP_UIO) { reg = rshim_pci_read_word(dev, PCI_COMMAND); if (reg != 0xFFFF) { if (enable && (reg & PCI_COMMAND_DISABLE_INTx)) rshim_pci_write_word(dev, PCI_COMMAND, reg & ~PCI_COMMAND_DISABLE_INTx); else if (!enable && !(reg & PCI_COMMAND_DISABLE_INTx)) rshim_pci_write_word(dev, PCI_COMMAND, reg | PCI_COMMAND_DISABLE_INTx); } return 0; } else if (dev->mmap_mode != RSHIM_PCIE_MMAP_VFIO) { return 0; } /* VFIO interrupt enable/disable */ if (dev->device_fd == -1) return 0; /* Mask interrupts before disabling. */ if (!enable) { len = sizeof(struct vfio_irq_set); memset(irq_set, 0, len); irq_set->argsz = len; irq_set->count = 1; irq_set->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_MASK; irq_set->index = VFIO_PCI_INTX_IRQ_INDEX; irq_set->start = 0; ret = ioctl(dev->device_fd, VFIO_DEVICE_SET_IRQS, irq_set); if (ret) { RSHIM_ERR("Failed to mask INTx\n"); return -1; } } /* Enable INTx */ irq_set->argsz = sizeof(irq_set_buf); irq_set->count = 1; irq_set->flags = enable ? VFIO_IRQ_SET_DATA_EVENTFD : VFIO_IRQ_SET_DATA_NONE; irq_set->flags |= VFIO_IRQ_SET_ACTION_TRIGGER; irq_set->index = VFIO_PCI_INTX_IRQ_INDEX; irq_set->start = 0; irq_set_buf.intr_fd = dev->intr_fd; ret = ioctl(dev->device_fd, VFIO_DEVICE_SET_IRQS, irq_set); if (ret) { RSHIM_ERR("Failed to enable INTx\n"); return -1; } /* Unmask INTx after enabling. */ if (enable) { len = sizeof(struct vfio_irq_set); memset(irq_set, 0, len); irq_set->argsz = len; irq_set->count = 1; irq_set->flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_UNMASK; irq_set->index = VFIO_PCI_INTX_IRQ_INDEX; irq_set->start = 0; ret = ioctl(dev->device_fd, VFIO_DEVICE_SET_IRQS, irq_set); if (ret) { RSHIM_ERR("Failed to unmask INTx\n"); return -1; } } return 0; } /* Memory map over VFIO. */ static int rshim_pcie_mmap_vfio(rshim_pcie_t *dev) { int rc, group_id, container_fd = -1, group_fd = -1, device_fd = -1; struct vfio_irq_info irq = { .argsz = sizeof(irq) }; char path[RSHIM_PATH_MAX], name[PATH_MAX], *p; struct vfio_group_status group_status = { .argsz = sizeof(group_status) }; struct vfio_device_info device_info = { .argsz = sizeof(device_info) }; struct vfio_region_info region_info = { .argsz = sizeof(region_info) }; container_fd = open("/dev/vfio/vfio", O_RDWR); if (container_fd < 0) { RSHIM_DBG("Failed to open /dev/vfio/vfio, %d (%s)\n", container_fd, strerror(errno)); rc = container_fd; goto fail; } if (ioctl(container_fd, VFIO_GET_API_VERSION) != VFIO_API_VERSION) { RSHIM_WARN("Unknown vfio API version\n"); rc = -EPROTONOSUPPORT; goto fail; } if (!ioctl(container_fd, VFIO_CHECK_EXTENSION, VFIO_TYPE1_IOMMU)) { RSHIM_WARN("IOMMU version not supported\n"); rc = -EPROTONOSUPPORT; goto fail; } /* Find the group_id. */ snprintf(path, sizeof(path), "%s/%04x:%02x:%02x.%1u/iommu_group", SYS_BUS_PCI_PATH, dev->domain, dev->bus, dev->dev, dev->func); rc = readlink(path, name, sizeof(name)); if (rc < 0 || !name[0] || rc >= sizeof(name)) { RSHIM_ERR("%s: failed to read iommu link\n", path); goto fail; } name[rc] = 0; p = strrchr(name, '/'); if (!p) { RSHIM_ERR("Failed to find vfio group\n"); rc = -ENOENT; goto fail; } group_id = atoi(p + 1); snprintf(path, sizeof(path), "/dev/vfio/%d", group_id); group_fd = open(path, O_RDWR); if (group_fd < 0) { RSHIM_DBG("Failed to open %s, %d (%s)\n", path, group_fd, strerror(errno)); rc = group_fd; goto fail; } rc = ioctl(group_fd, VFIO_GROUP_GET_STATUS, &group_status); if (rc) { RSHIM_ERR("VFIO_GROUP_GET_STATUS failed\n"); goto fail; } if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) { RSHIM_DBG("VFIO group not viable\n"); rc = -1; goto fail; } if (!(group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET)) { ioctl(group_fd, VFIO_GROUP_SET_CONTAINER, &container_fd); ioctl(container_fd, VFIO_SET_IOMMU, VFIO_TYPE1_IOMMU); } snprintf(path, sizeof(path), "%04x:%02x:%02x.%d", dev->domain, dev->bus, dev->dev, dev->func); device_fd = ioctl(group_fd, VFIO_GROUP_GET_DEVICE_FD, path); if (device_fd < 0) { RSHIM_ERR("Failed to get vfio device %s\n", path); rc = device_fd; goto fail; } rc = ioctl(device_fd, VFIO_DEVICE_GET_INFO, &device_info); if (rc) { RSHIM_ERR("Failed to get vfio device info\n"); goto fail; } if (!device_info.num_regions) { rc = -1; goto fail; } region_info.index = 0; rc = ioctl(device_fd, VFIO_DEVICE_GET_REGION_INFO, ®ion_info); if (rc) { RSHIM_ERR("Failed to get vfio region info\n"); goto fail; } if (region_info.flags & VFIO_REGION_INFO_FLAG_MMAP) { void *map = mmap(NULL, (size_t)region_info.size, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_LOCKED, device_fd, (off_t)region_info.offset); if (map == MAP_FAILED) { RSHIM_ERR("Vfio mmap failed %m\n"); rc = -1; goto fail; } else { uint16_t reg = 0; /* Set PCI bus mastering */ rc = pread(device_fd, ®, sizeof(reg), VFIO_GET_REGION_ADDR(VFIO_PCI_CONFIG_REGION_INDEX) + PCI_COMMAND); if (rc != sizeof(reg)) { RSHIM_ERR("Failed to read command from PCI config space!\n"); rc = -1; goto fail; } reg |= RSHIM_PCI_COMMAND; rc = pwrite(device_fd, ®, sizeof(reg), VFIO_GET_REGION_ADDR(VFIO_PCI_CONFIG_REGION_INDEX) + PCI_COMMAND); if (rc != sizeof(reg)) { RSHIM_ERR("Failed to set PCI bus mastering!\n"); rc = -1; goto fail; } dev->device_fd = device_fd; dev->group_fd = group_fd; dev->container_fd = container_fd; dev->rshim_regs = map; /* Enable interrupt */ irq.index = VFIO_PCI_INTX_IRQ_INDEX; rc = ioctl(device_fd, VFIO_DEVICE_GET_IRQ_INFO, &irq); if (rc < 0 || (irq.flags & VFIO_IRQ_INFO_EVENTFD) == 0) { RSHIM_WARN("Unable to get vfio IRQ\n"); } else { if (dev->intr_fd < 0) dev->intr_fd = eventfd(0, EFD_CLOEXEC); dev->intr_len = sizeof(uint64_t); if (dev->intr_fd >= 0) rshim_pcie_enable_irq(dev, true); } } } return 0; fail: if (device_fd >= 0) close(device_fd); if (group_fd >= 0) close(group_fd); if (container_fd >= 0) close(container_fd); return rc; } /* Memory map over UIO. */ static int rshim_pcie_mmap_uio(rshim_pcie_t *dev) { char dirname[RSHIM_PATH_MAX], devname[RSHIM_PATH_MAX], *str = NULL; struct dirent *e; int uio_num, rc; DIR *dir; /* Find the uio number. */ snprintf(dirname, sizeof(dirname), "%s/%04x:%02x:%02x.%1u/uio", SYS_BUS_PCI_PATH, dev->domain, dev->bus, dev->dev, dev->func); dir = opendir(dirname); if (!dir) { snprintf(dirname, sizeof(dirname), "%s/%04x:%02x:%02x.%1u", SYS_BUS_PCI_PATH, dev->domain, dev->bus, dev->dev, dev->func); dir = opendir(dirname); if (!dir) return -ENOENT; } while ((e = readdir(dir)) != NULL) { if ((str = strstr(e->d_name, "uio:uio")) != NULL) str += strlen("uio:uio"); else if ((str = strstr(e->d_name, "uio")) != NULL) str += strlen("uio"); if (str) break; } closedir(dir); if (!str) return -ENOENT; /* Memory map. */ rc = rshim_pcie_mmap_direct(dev); if (rc) return rc; /* Open the control fd to handle interrupt. */ uio_num = atoi(str); snprintf(devname, sizeof(devname), "/dev/uio%u", uio_num); if (dev->intr_fd >= 0) { dev->intr_reset_seq++; __sync_synchronize(); close(dev->intr_fd); } dev->intr_fd = open(devname, O_RDWR); dev->intr_len = sizeof(uint32_t); rshim_pcie_enable_irq(dev, true); return 0; } static int rshim_pcie_mmap(rshim_pcie_t *dev, bool enable) { int rc = -EINVAL; if (!enable) { rshim_pcie_mmap_release(dev); return 0; } if (dev->mmap_mode == RSHIM_PCIE_MMAP_VFIO) rc = rshim_pcie_mmap_vfio(dev); else if (dev->mmap_mode == RSHIM_PCIE_MMAP_UIO) rc = rshim_pcie_mmap_uio(dev); else if (dev->mmap_mode == RSHIM_PCIE_MMAP_DIRECT) rc = rshim_pcie_mmap_direct(dev); return rc; } static void rshim_pcie_intr(rshim_pcie_t *dev) { rshim_pcie_intr_info_t info = {.word = 0}; rshim_backend_t *bd = &dev->bd; int rc, drop_mode, delay; time_t t; /* Add some protection for interrupt flooding. */ time(&t); if (difftime(t, dev->last_intr_time) > 1) { dev->last_intr_time = t; dev->intr_cnt = 0; } dev->intr_cnt++; if (dev->intr_cnt > RSHIM_PCIE_NIC_IRQ_RATE) return; pthread_mutex_lock(&bd->mutex); rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->scratchpad6, &info.word, RSHIM_REG_SIZE_8B); if (rc || RSHIM_BAD_CTRL_REG(info.word)) { if (!bd->drop_mode) RSHIM_WARN("Failed to read irq request\n"); goto intr_done; } /* Only handles NIC reset for now. */ if (info.rst_type != RSHIM_PCIE_RST_TYPE_NIC_RESET && info.rst_type != RSHIM_PCIE_RST_TYPE_DPU_RESET) { goto intr_done; } RSHIM_INFO("Receive interrupt for %s reset\n", (info.rst_type == RSHIM_PCIE_RST_TYPE_NIC_RESET) ? "NIC" : ((info.rst_type == RSHIM_PCIE_RST_TYPE_DPU_RESET) ? "DPU" : "")); switch (info.rst_state) { case RSHIM_PCIE_RST_STATE_REQUEST: RSHIM_INFO("NIC reset ACK\n"); info.rst_reply = RSHIM_PCIE_RST_REPLY_ACK; __sync_synchronize(); bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->scratchpad6, info.word, RSHIM_REG_SIZE_8B); break; case RSHIM_PCIE_RST_STATE_ABORT: RSHIM_INFO("NIC reset ABORT\n"); info.word &= 0xFFFFFFFFUL; bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->scratchpad6, info.word, RSHIM_REG_SIZE_8B); break; case RSHIM_PCIE_RST_STATE_START: RSHIM_INFO("NIC reset START\n"); info.rst_reply = RSHIM_PCIE_RST_START_ACK; dev->nic_reset = true; __sync_synchronize(); bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->scratchpad6, info.word, RSHIM_REG_SIZE_8B); /* * Both NIC and ARM reset. * - Set drop_mode to prevent further read/write; * - Take the mutex so the main thread is not in Rx/Tx loop; * - Clear FIFO state; * - Add a small delay for ARM to be ready; */ __sync_synchronize(); drop_mode = bd->drop_mode; bd->drop_mode = 1; rshim_fifo_reset(bd); delay = (info.rst_downtime * 10 + 999) / 1000; if (delay < RSHIM_PCIE_RST_START_MIN_DELAY) delay = RSHIM_PCIE_RST_START_MIN_DELAY; sleep(delay); bd->drop_mode = drop_mode; dev->nic_reset = false; break; default: break; } if (!bd->drop_mode) rshim_pcie_enable_irq(dev, true); intr_done: pthread_mutex_unlock(&bd->mutex); } static void rshim_pcie_intr_poll(rshim_pcie_t *dev) { uint16_t reg; usleep(rshim_pcie_intr_poll_interval * 1000); reg = rshim_pci_read_word(dev, PCI_STATUS); if ((reg != 0xFFFF) && (reg & PCI_STATUS_INTx)) rshim_pcie_intr(dev); } static void *rshim_pcie_intr_thread(void *arg) { rshim_pcie_t *dev = arg; uint8_t intr_buf[16]; int rc, reset_seq; reset_seq = dev->intr_reset_seq; while (rshim_run) { if (dev->intr_fd < 0) { if (dev->mmap_mode == RSHIM_PCIE_MMAP_DIRECT) rshim_pcie_intr_poll(dev); else sleep(1); continue; } rc = read(dev->intr_fd, intr_buf, dev->intr_len); if (rc < 0) { if (errno == EINTR) continue; } else if (rc == 0) { sleep(1); continue; } __sync_synchronize(); if (reset_seq != dev->intr_reset_seq) { reset_seq = dev->intr_reset_seq; continue; } /* Interrupt handler. */ rshim_pcie_intr(dev); } return NULL; } #elif defined(__FreeBSD__) static int rshim_pcie_mmap(rshim_pcie_t *dev, bool enable) { struct pci_bar_mmap pbm = { .pbm_sel.pc_func = dev->func, .pbm_sel.pc_dev = dev->dev, .pbm_sel.pc_bus = dev->bus, .pbm_sel.pc_domain = dev->domain, .pbm_reg = 0x10, .pbm_flags = PCIIO_BAR_MMAP_RW, .pbm_memattr = VM_MEMATTR_UNCACHEABLE, }; int rc; if (!enable) { if (dev->device_fd >= 0) { close(dev->device_fd); dev->device_fd = -1; return 0; } } dev->device_fd = open("/dev/pci", O_RDWR, 0); if (dev->device_fd < 0) { RSHIM_ERR("Failed to open /dev/pci\n"); return -ENODEV; } if (ioctl(dev->device_fd, PCIOCBARMMAP, &pbm) < 0) { RSHIM_ERR("PCIOCBARMMAP IOCTL failed\n"); rc = -ENODEV; goto rshim_map_failed; } dev->rshim_regs = (void *)((uintptr_t)pbm.pbm_map_base + (uintptr_t)pbm.pbm_bar_off); if (pbm.pbm_bar_length < dev->bar_size) { dev->rshim_regs = NULL; RSHIM_ERR("BAR length is too small\n"); rc = -ENOMEM; goto rshim_map_failed; } return 0; rshim_map_failed: close(dev->device_fd); dev->device_fd = -1; return rc; } #else #error "Platform not supported" #endif /* __linux__ */ static uint32_t rshim_pcie_bf3_chan_addr_convert(uint32_t chan, uint32_t addr) { if (chan < 0xF) addr += bf3_rshim_pcie_chan_map[chan] + BF3_RSH_BASE_ADDR; else addr = (chan << 16) + addr; return addr; } /* RShim read/write routines */ static int __attribute__ ((noinline)) rshim_pcie_read(rshim_backend_t *bd, uint32_t chan, uint32_t addr, uint64_t *result, int size) { rshim_pcie_t *dev = container_of(bd, rshim_pcie_t, bd); int rc = 0; if (dev->nic_reset && (chan != RSHIM_CHANNEL || addr != bd->regs->scratchpad6)) sleep(RSHIM_PCIE_NIC_RESET_WAIT); if (bd->drop_mode) { *result = 0; return 0; } if (!bd->has_rshim || !bd->has_tm || !dev->rshim_regs) return -ENODEV; dev->write_count = 0; if (rshim_is_bluefield3(dev->device_id)) { addr = rshim_pcie_bf3_chan_addr_convert(chan, addr); if (addr < BF3_RSH_BASE_ADDR || addr >= (BF3_RSH_BASE_ADDR + BF3_PCI_RSHIM_WINDOW_SIZE)) return -EINVAL; addr -= BF3_RSH_BASE_ADDR; } else { addr = addr | (chan << 16); } if (size == 4) *result = readl(dev->rshim_regs + addr); else if (size == 8) *result = readq(dev->rshim_regs + addr); else rc = -EINVAL; return rc; } static int __attribute__ ((noinline)) rshim_pcie_write(rshim_backend_t *bd, uint32_t chan, uint32_t addr, uint64_t value, int size) { rshim_pcie_t *dev = container_of(bd, rshim_pcie_t, bd); uint64_t result; int rc = 0; if (dev->nic_reset && (chan != RSHIM_CHANNEL || addr != bd->regs->scratchpad6)) sleep(RSHIM_PCIE_NIC_RESET_WAIT); if (bd->drop_mode) return 0; if (!bd->has_rshim || !bd->has_tm || !dev->rshim_regs) return -ENODEV; /* * We cannot stream large numbers of PCIe writes to the RShim's BAR. * Instead, we must write no more than 15 8-byte words before * doing a read from another register within the BAR, * which forces previous writes to drain. */ if (rshim_is_bluefield1(dev->device_id)) { if (dev->write_count == 15) { __sync_synchronize(); rshim_pcie_read(bd, chan, RSH_SCRATCHPAD1, &result, rc); } dev->write_count++; } if (rshim_is_bluefield3(dev->device_id)) { addr = rshim_pcie_bf3_chan_addr_convert(chan, addr); if (addr < BF3_RSH_BASE_ADDR || addr >= (BF3_RSH_BASE_ADDR + BF3_PCI_RSHIM_WINDOW_SIZE)) return -EINVAL; addr -= BF3_RSH_BASE_ADDR; } else { addr = addr | (chan << 16); } if (size == 4) writel(value, dev->rshim_regs + addr); else if (size == 8) writeq(value, dev->rshim_regs + addr); else rc = -EINVAL; return rc; } static void rshim_pcie_delete(rshim_backend_t *bd) { rshim_pcie_t *dev = container_of(bd, rshim_pcie_t, bd); rshim_deregister(bd); free(dev); } /* Enable RSHIM PF and setup memory map. */ static int rshim_pcie_enable(rshim_backend_t *bd, bool enable) { rshim_pcie_t *dev = container_of(bd, rshim_pcie_t, bd); int rc = 0; #ifdef __linux__ if (!dev->device_id) return -ENODEV; /* * Clear scratchpad1 since it's checked by FW for rshim driver. * This needs to be done before the resources are unmapped. */ if (!enable) { rshim_pcie_write(bd, RSHIM_CHANNEL, bd->regs->scratchpad1, 0, RSHIM_REG_SIZE_8B); } /* Unmap existing resource first. */ rshim_pcie_mmap(dev, false); /* Bind/unbind the device. */ rshim_pcie_bind(dev, enable); /* Remap the resource. */ if (enable) { rc = rshim_pcie_mmap(dev, true); /* Fall-back to uio if failed. */ if (rc < 0 && dev->mmap_mode == RSHIM_PCIE_MMAP_VFIO && rshim_pcie_has_uio()) { RSHIM_INFO("Fall-back to uio\n"); rshim_pcie_bind(dev, false); dev->pci_path = SYS_UIO_PCI_PATH; dev->mmap_mode = RSHIM_PCIE_MMAP_UIO; rshim_pcie_bind(dev, true); rc = rshim_pcie_mmap(dev, true); } /* Fall-back to direct map if failed. */ if (rc < 0 && dev->mmap_mode != RSHIM_PCIE_MMAP_DIRECT) { RSHIM_INFO("Fall-back to direct io\n"); rshim_pcie_bind(dev, false); dev->pci_path = NULL; dev->mmap_mode = RSHIM_PCIE_MMAP_DIRECT; rshim_pcie_bind(dev, true); rc = rshim_pcie_mmap(dev, true); } } #else /* Unmap existing resource then remap it. */ rshim_pcie_mmap(dev, false); if (enable) rc = rshim_pcie_mmap(dev, true); #endif /* __linux__ */ RSHIM_INFO("rshim %s %s\n", bd->dev_name, enable ? "enable" : "disable"); return rc; } /* Probe routine */ static int rshim_pcie_probe(struct pci_dev *pci_dev) { char dev_name[RSHIM_DEV_NAME_LEN]; rshim_backend_t *bd; rshim_pcie_t *dev; int rc = 0; snprintf(dev_name, sizeof(dev_name) - 1, "pcie-%04x:%02x:%02x.%x", pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func); if (!rshim_allow_device(dev_name)) return -EACCES; RSHIM_INFO("Probing %s(%s)\n", dev_name, rshim_pcie_mmap_name[rshim_pcie_mmap_mode]); rshim_lock(); bd = rshim_find_by_name(dev_name); if (bd) { RSHIM_INFO("Found %s\n", dev_name); dev = container_of(bd, rshim_pcie_t, bd); } else { RSHIM_INFO("Create rshim %s\n", dev_name); dev = calloc(1, sizeof(*dev)); if (dev == NULL) { rshim_unlock(); return -ENOMEM; } bd = &dev->bd; strcpy(bd->dev_name, dev_name); bd->drop_mode = (rshim_drop_mode >= 0) ? rshim_drop_mode : 0; bd->read_rshim = rshim_pcie_read; bd->write_rshim = rshim_pcie_write; bd->destroy = rshim_pcie_delete; bd->enable_device = rshim_pcie_enable; dev->write_count = 0; dev->device_fd = -1; dev->group_fd = -1; dev->container_fd = -1; dev->intr_fd = -1; dev->mmap_mode = rshim_pcie_mmap_mode; #ifdef __linux__ dev->pci_path = rshim_sys_pci_path; #endif time(&dev->last_intr_time); pthread_mutex_init(&bd->mutex, NULL); } rshim_ref(bd); switch (pci_dev->device_id) { case BLUEFIELD3_DEVICE_ID: case BLUEFIELD3_DEVICE_ID2: bd->regs = &bf3_rshim_regs; bd->ver_id = RSHIM_BLUEFIELD_3; dev->bar_size = BF3_PCI_RSHIM_WINDOW_SIZE; break; case BLUEFIELD2_DEVICE_ID: bd->regs = &bf1_bf2_rshim_regs; bd->ver_id = RSHIM_BLUEFIELD_2; dev->bar_size = PCI_RSHIM_WINDOW_SIZE; break; default: bd->regs = &bf1_bf2_rshim_regs; bd->ver_id = RSHIM_BLUEFIELD_1; dev->bar_size = PCI_RSHIM_WINDOW_SIZE; break; } bd->rev_id = pci_read_byte(pci_dev, PCI_REVISION_ID); if (rshim_has_pcie_reset_delay || bd->ver_id < RSHIM_BLUEFIELD_3) bd->reset_delay = rshim_pcie_reset_delay; else bd->reset_delay = 1; /* minimum delay for BF3 */ /* Initialize object */ dev->device_id = pci_dev->device_id; dev->domain = pci_dev->domain; dev->bus = pci_dev->bus; dev->dev = pci_dev->dev; dev->func = pci_dev->func; /* Enable the device and setup memory map. */ if (!bd->drop_mode) { pthread_mutex_lock(&bd->mutex); rc = bd->enable_device(bd, true); pthread_mutex_unlock(&bd->mutex); if (rc) goto rshim_probe_failed; } pthread_mutex_lock(&bd->mutex); /* * Register rshim here since it needs to detect whether other backend * has already registered or not, which involves reading/writting rshim * registers and has assumption that the under layer is working. */ bd->has_rshim = 1; bd->has_tm = 1; rc = rshim_register(bd); /* Notify that the device is attached */ if (!rc) rc = rshim_notify(bd, RSH_EVENT_ATTACH, 0); pthread_mutex_unlock(&bd->mutex); if (rc) goto rshim_probe_failed; #ifdef __linux__ /* Create interrupt handling thread for BlueField-2 and above. */ if (pci_dev->device_id != BLUEFIELD1_DEVICE_ID) { rc = pthread_create(&dev->intr_thread, NULL, rshim_pcie_intr_thread, dev); if (rc) RSHIM_ERR("Failed to create intr thread\n"); } #endif rshim_unlock(); return 0; rshim_probe_failed: rshim_deref(bd); rshim_unlock(); return rc; } #ifdef __linux__ static bool rshim_pcie_has_vfio(void) { struct dirent* d; DIR* dir; int rc; if (!rshim_pcie_enable_vfio) return false; rc = system("modprobe vfio_pci"); if (rc == -1) RSHIM_DBG("Failed to load the vfio_pci module %m\n"); dir = opendir(SYS_CLASS_VFIO_PCI_PATH); if (!dir) return false; closedir(dir); dir = opendir(SYS_CLASS_IOMMU_PATH); if (!dir) return false; while ((d = readdir(dir)) != NULL) { if (strcmp(d->d_name, ".") && strcmp(d->d_name, "..")) break; } closedir(dir); return (d != NULL); } static bool rshim_pcie_has_uio(void) { DIR* dir; int rc; if (!rshim_pcie_enable_uio) return false; rc = system("modprobe uio_pci_generic"); if (rc == -1) RSHIM_DBG("Failed to load the uio_pci_generic module %m\n"); dir = opendir(SYS_CLASS_UIO_PCI_PATH); if (!dir) return false; closedir(dir); return true; } static bool kernel_lock_down_enabled(void) { char buf[64] = { 0 }, *p; FILE *file; file = fopen("/sys/kernel/security/lockdown", "rt"); if (!file) return false; p = fgets(buf, sizeof(buf), file); fclose(file); return (p != NULL && (strstr(buf, "[integrity]") != NULL || strstr(buf, "[confidentiality]") != NULL)); } #endif /* __linux__ */ int rshim_pcie_init(void) { bool dev_present = false; struct pci_access *pci; struct pci_dev *dev; int rc; #ifdef __linux__ if (rshim_pcie_has_vfio()) { rshim_pcie_mmap_mode = RSHIM_PCIE_MMAP_VFIO; rshim_sys_pci_path = SYS_VFIO_PCI_PATH; } else { /* Linux kernel lock_down requires VFIO. */ if (kernel_lock_down_enabled()) { RSHIM_ERR("Need to enable IOMMU/VFIO for kernel lock-down\n"); return -ENOTSUP; } if (rshim_pcie_has_uio()) { rshim_pcie_mmap_mode = RSHIM_PCIE_MMAP_UIO; rshim_sys_pci_path = SYS_UIO_PCI_PATH; } } #endif /* __linux__ */ pci = pci_alloc(); if (!pci) return -ENOMEM; pci_init(pci); pci_scan_bus(pci); /* Iterate over the devices */ for (dev = pci->devices; dev; dev = dev->next) { pci_fill_info(dev, PCI_FILL_IDENT | PCI_FILL_BASES | PCI_FILL_CLASS); if (dev->vendor_id != TILERA_VENDOR_ID || (!rshim_is_bluefield1(dev->device_id) && !rshim_is_bluefield2(dev->device_id) && !rshim_is_bluefield3(dev->device_id))) continue; rc = rshim_pcie_probe(dev); if (rc) continue; dev_present = true; } pci_cleanup(pci); if (!dev_present) return -ENODEV; return 0; } ./Mellanox-rshim-user-space-0fd5d3f/src/rshim_pcie_lf.c0000664000175000017500000005222714563673752022406 0ustar tai271828tai271828// SPDX-License-Identifier: GPL-2.0-only /* * Copyright (C) 2019 Mellanox Technologies. All Rights Reserved. * */ #include #include #include #include #include #include #include #include #include "rshim.h" /** Our Vendor/Device IDs. */ #define TILERA_VENDOR_ID 0x15b3 #define BLUEFIELD1_DEVICE_ID 0x0211 #define BLUEFIELD2_DEVICE_ID 0x0214 #define BLUEFIELD3_DEVICE_ID 0x021c /* Mellanox Address & Data Capabilities */ #define MELLANOX_ADDR 0x58 #define MELLANOX_DATA 0x5c #define MELLANOX_CAP_READ 0x1 /* TRIO_CR_GATEWAY registers */ #define TRIO_CR_GW_LOCK 0xe38a0 #define TRIO_CR_GW_LOCK_CPY 0xe38a4 #define TRIO_CR_GW_DATA_UPPER 0xe38ac #define TRIO_CR_GW_DATA_LOWER 0xe38b0 #define TRIO_CR_GW_CTL 0xe38b4 #define TRIO_CR_GW_ADDR_UPPER 0xe38b8 #define TRIO_CR_GW_ADDR_LOWER 0xe38bc #define TRIO_CR_GW_LOCK_ACQUIRED 0x80000000 #define TRIO_CR_GW_LOCK_RELEASE 0x0 #define TRIO_CR_GW_BUSY 0x60000000 #define TRIO_CR_GW_TRIGGER 0xe0000000 #define TRIO_CR_GW_READ_4BYTE 0x6 #define TRIO_CR_GW_WRITE_4BYTE 0x2 #define CRSPACE_RSH_CHANNEL1_BASE 0x310000 /* MSN GW_CR_BOOT registers */ #define MSN_GW_CR_64B_BOOT_REG0 0xae01040 #define MSN_GW_CR_64B_BOOT_REG0_COPY 0xae01044 #define MSN_GW_CR_64B_BOOT_LOCK 0x80000000 #define MSN_GW_CR_64B_BOOT_BUSY 0x40000000 #define MSN_GW_CR_64B_BOOT_ACC_TYPE 0xae0104c #define MSN_GW_CR_64B_BOOT_DATA_HIGH 0xae01050 #define MSN_GW_CR_64B_BOOT_DATA_LOW 0xae01054 #define MSN_GW_CR_64B_BOOT_ADDR 0xae01058 #define MSN_GW_CR_64B_BOOT_ACC_WIDTH 0xae0105c #define MSN_GW_CR_64B_BOOT_64BIT_LINE 0x20000 typedef struct { /* RShim backend structure. */ struct rshim_backend bd; struct pci_dev *pci_dev; /* Keep track of number of 8-byte word writes */ u8 write_count; } rshim_pcie_lf_t; int bf3_rshim_pcie_lf_chan_map[] = { [RSHIM_CHANNEL] = 0x3000000, [UART0_CHANNEL] = 0x3010000, [UART1_CHANNEL] = 0x3011000, [DIAGUART_CHANNEL] = 0x3012000, [RSH_HUB_CHANNEL] = 0x3012400, [WDOG0_CHANNEL] = 0x3020000, [WDOG1_CHANNEL] = 0x3040000, [MCH_CORE_CHANNEL] = 0x3060000, [TIMER_ARM_CHANNEL] = 0x3080000, [TIMER_EXT_CHANNEL] = 0x30a0000, [OOB_CHANNEL] = 0x30a1000, [YU_CHANNEL] = 0x3400000, }; /* Mechanism to access the CR space using hidden PCI capabilities */ static int pci_cap_read(struct pci_dev *pci_dev, int offset, uint32_t *result) { int rc; /* * Write target offset to MELLANOX_ADDR. * Set LSB to indicate a read operation. */ rc = pci_write_long(pci_dev, MELLANOX_ADDR, offset | MELLANOX_CAP_READ); if (rc < 0) return rc; /* Read result from MELLANOX_DATA */ *result = pci_read_long(pci_dev, MELLANOX_DATA); return 0; } static int pci_cap_write(struct pci_dev *pci_dev, int offset, uint32_t value) { int rc; /* Write data to MELLANOX_DATA */ rc = pci_write_long(pci_dev, MELLANOX_DATA, value); if (rc < 0) return rc; /* * Write target offset to MELLANOX_ADDR. * Leave LSB clear to indicate a write operation. */ rc = pci_write_long(pci_dev, MELLANOX_ADDR, offset); if (rc < 0) return rc; return 0; } /* Acquire and release the MSN GW_CR_64B lock */ static int msn_gw_lock_acquire(struct pci_dev *pci_dev) { uint32_t read_value; time_t t0, t1; int rc; /* Wait until the MSN GW_CR_64B lock is free */ time(&t0); do { rc = pci_cap_read(pci_dev, MSN_GW_CR_64B_BOOT_REG0, &read_value); if (rc) return rc; time(&t1); if (difftime(t1, t0) > RSHIM_LOCK_RETRY_TIME) return -ETIMEDOUT; } while (read_value & MSN_GW_CR_64B_BOOT_LOCK); return 0; } static int msn_gw_lock_release(struct pci_dev *pci_dev) { int rc; /* Release MSN GW_CR_64B lock */ rc = pci_cap_write(pci_dev, MSN_GW_CR_64B_BOOT_REG0, 0x0); return rc; } /* Poll the MSN GW_CR_64B */ static int msn_gw_poll_busy(struct pci_dev *pci_dev) { uint32_t read_value; time_t t0, t1; int rc; /* Wait until the MSN GW_CR_64B lock is free */ time(&t0); do { rc = pci_cap_read(pci_dev, MSN_GW_CR_64B_BOOT_REG0_COPY, &read_value); if (rc) return rc; time(&t1); if (difftime(t1, t0) > RSHIM_LOCK_RETRY_TIME) return -ETIMEDOUT; } while (read_value & MSN_GW_CR_64B_BOOT_BUSY); return 0; } /* * Mechanism to access RShim via the MSN GW_CR_64B gateway. */ static int msn_gw_read(struct pci_dev *pci_dev, int addr, uint64_t *result) { uint64_t read_result; uint32_t data; int rc = 0; /* Acquire MSN_GW_BOOT_LOCK */ rc = msn_gw_lock_acquire(pci_dev); if (rc) goto err; /* Write addr to MSN_GW_ADDR */ rc = pci_cap_write(pci_dev, MSN_GW_CR_64B_BOOT_ADDR, (addr >> 2)); if (rc) goto err; /* Set access width to 64 bits */ rc = pci_cap_write(pci_dev, MSN_GW_CR_64B_BOOT_ACC_WIDTH, MSN_GW_CR_64B_BOOT_64BIT_LINE); if (rc) goto err; /* Set access type to read */ rc = pci_cap_write(pci_dev, MSN_GW_CR_64B_BOOT_ACC_TYPE, 0x1); if (rc) goto err; /* Set BUSY bit to trigger MSN_GW to read from addr */ rc = pci_cap_read(pci_dev, MSN_GW_CR_64B_BOOT_REG0, &data); if (rc) goto err; data |= MSN_GW_CR_64B_BOOT_BUSY; rc = pci_cap_write(pci_dev, MSN_GW_CR_64B_BOOT_REG0, data); if (rc) goto err; /* Wait for MSN_GW_BOOT_BUSY to be cleared */ rc = msn_gw_poll_busy(pci_dev); if (rc) goto err; /* Read lower 32-bits of data */ rc = pci_cap_read(pci_dev, MSN_GW_CR_64B_BOOT_DATA_LOW, &data); if (rc) goto err; read_result = (uint64_t)data; /* Read upper 32-bits of data */ rc = pci_cap_read(pci_dev, MSN_GW_CR_64B_BOOT_DATA_HIGH, &data); if (rc) goto err; read_result |= ((uint64_t)data << 32); *result = read_result; err: /* Release MSN_GW_BOOT_LOCK */ if (msn_gw_lock_release(pci_dev)) RSHIM_ERR("Failed to release MSN GW lock\n"); return rc; } static int msn_gw_write(struct pci_dev *pci_dev, int addr, uint64_t value) { uint32_t data; int rc; /* Acquire MSN_GW_BOOT_LOCK */ rc = msn_gw_lock_acquire(pci_dev); if (rc) goto err; /* Write addr to MSN_GW_ADDR */ rc = pci_cap_write(pci_dev, MSN_GW_CR_64B_BOOT_ADDR, (addr >> 2)); if (rc) goto err; /* Set access width to 64 bits */ rc = pci_cap_write(pci_dev, MSN_GW_CR_64B_BOOT_ACC_WIDTH, MSN_GW_CR_64B_BOOT_64BIT_LINE); if (rc) goto err; /* Set access type to write */ rc = pci_cap_write(pci_dev, MSN_GW_CR_64B_BOOT_ACC_TYPE, 0x0); if (rc) goto err; /* Write lower 32 bits of data */ rc = pci_cap_write(pci_dev, MSN_GW_CR_64B_BOOT_DATA_LOW, (uint32_t)value); if (rc) goto err; value = value >> 32; /* Write higher 32 bits of data */ rc = pci_cap_write(pci_dev, MSN_GW_CR_64B_BOOT_DATA_HIGH, (uint32_t)value); if (rc) goto err; /* Set BUSY bit to trigger MSN_GW to write to addr */ rc = pci_cap_read(pci_dev, MSN_GW_CR_64B_BOOT_REG0, &data); if (rc) goto err; data |= MSN_GW_CR_64B_BOOT_BUSY; rc = pci_cap_write(pci_dev, MSN_GW_CR_64B_BOOT_REG0, data); if (rc) goto err; /* * The MSN block should not be accessed when the ARM is in reset * until BL1 can configure the crmaster table. * WA: Add a delay before resuming the flow. * TBD: RShim API should receive intimation when ARM reset happens */ if ((pci_dev->device_id == BLUEFIELD3_DEVICE_ID) && ((addr & 0xffff) == BF3_RSH_RESET_CONTROL)) sleep (10); /* Wait for MSN_GW_BOOT_BUSY to be cleared */ rc = msn_gw_poll_busy(pci_dev); if (rc) goto err; err: /* Release MSN_GW_BOOT_LOCK */ if (msn_gw_lock_release(pci_dev)) RSHIM_ERR("Failed to release MSN GW lock\n"); return rc; } /* Acquire and release the TRIO_CR_GW_LOCK. */ static int trio_cr_gw_lock_acquire(struct pci_dev *pci_dev) { uint32_t read_value; time_t t0, t1; int rc; /* Wait until TRIO_CR_GW_LOCK is free */ time(&t0); do { rc = pci_cap_read(pci_dev, TRIO_CR_GW_LOCK, &read_value); if (rc) return rc; time(&t1); if (difftime(t1, t0) > RSHIM_LOCK_RETRY_TIME) return -ETIMEDOUT; } while (read_value & TRIO_CR_GW_LOCK_ACQUIRED); /* Acquire TRIO_CR_GW_LOCK */ rc = pci_cap_write(pci_dev, TRIO_CR_GW_LOCK, TRIO_CR_GW_LOCK_ACQUIRED); if (rc) return rc; return 0; } static int trio_cr_gw_lock_release(struct pci_dev *pci_dev) { int rc; /* Release TRIO_CR_GW_LOCK */ rc = pci_cap_write(pci_dev, TRIO_CR_GW_LOCK, TRIO_CR_GW_LOCK_RELEASE); return rc; } /* * Mechanism to access the RShim from the CR space using the TRIO_CR_GATEWAY. */ static int crspace_rsh_gw_read(struct pci_dev *pci_dev, int addr, uint32_t *result) { int rc; if (pci_dev->device_id == BLUEFIELD2_DEVICE_ID) { addr = (addr & 0xffff) + CRSPACE_RSH_CHANNEL1_BASE; rc = pci_cap_read(pci_dev, addr, result); return rc; } addr += RSH_CHANNEL_BASE(RSHIM_CHANNEL); /* Acquire TRIO_CR_GW_LOCK */ rc = trio_cr_gw_lock_acquire(pci_dev); if (rc) return rc; /* Write addr to TRIO_CR_GW_ADDR_LOWER */ rc = pci_cap_write(pci_dev, TRIO_CR_GW_ADDR_LOWER, addr); if (rc) return rc; /* Set TRIO_CR_GW_READ_4BYTE */ rc = pci_cap_write(pci_dev, TRIO_CR_GW_CTL, TRIO_CR_GW_READ_4BYTE); if (rc) return rc; /* Trigger TRIO_CR_GW to read from addr */ rc = pci_cap_write(pci_dev, TRIO_CR_GW_LOCK, TRIO_CR_GW_TRIGGER); if (rc) return rc; /* Read 32-bit data from TRIO_CR_GW_DATA_LOWER */ rc = pci_cap_read(pci_dev, TRIO_CR_GW_DATA_LOWER, result); if (rc) return rc; *result = ntohl(*result); /* Release TRIO_CR_GW_LOCK */ rc = trio_cr_gw_lock_release(pci_dev); if (rc) return rc; return 0; } static int crspace_rsh_gw_write(struct pci_dev *pci_dev, int addr, uint32_t value) { int rc; if (pci_dev->device_id == BLUEFIELD2_DEVICE_ID) { addr = (addr & 0xffff) + CRSPACE_RSH_CHANNEL1_BASE; rc = pci_cap_write(pci_dev, addr, value); return rc; } /* * All Rshim accesses except writes to the BOOT_FIFO_DATA go through * the Byte Access Widget and hence need to use the RSHIM_CHANNEL. */ if ((addr & 0xffff) != RSH_BOOT_FIFO_DATA) addr += RSH_CHANNEL_BASE(RSHIM_CHANNEL); /* Acquire TRIO_CR_GW_LOCK */ rc = trio_cr_gw_lock_acquire(pci_dev); if (rc) return rc; /* Write 32-bit data to TRIO_CR_GW_DATA_LOWER */ rc = pci_cap_write(pci_dev, TRIO_CR_GW_DATA_LOWER, htonl(value)); if (rc) return rc; /* Write addr to TRIO_CR_GW_ADDR_LOWER */ rc = pci_cap_write(pci_dev, TRIO_CR_GW_ADDR_LOWER, addr); if (rc) return rc; /* Set TRIO_CR_GW_WRITE_4BYTE */ rc = pci_cap_write(pci_dev, TRIO_CR_GW_CTL, TRIO_CR_GW_WRITE_4BYTE); if (rc) return rc; /* Trigger CR gateway to write to RShim */ rc = pci_cap_write(pci_dev, TRIO_CR_GW_LOCK, TRIO_CR_GW_TRIGGER); if (rc) return rc; /* Release TRIO_CR_GW_LOCK */ rc = trio_cr_gw_lock_release(pci_dev); if (rc) return rc; return 0; } /* Wait until the RSH_BYTE_ACC_CTL pending bit is cleared */ static int rshim_byte_acc_pending_wait(struct pci_dev *pci_dev) { uint32_t read_value = 0; time_t t0, t1; int rc; time(&t0); do { rc = crspace_rsh_gw_read(pci_dev, RSH_BYTE_ACC_CTL, &read_value); if (rc) return rc; time(&t1); if (difftime(t1, t0) > RSHIM_LOCK_RETRY_TIME) return -ETIMEDOUT; } while (read_value & RSH_BYTE_ACC_PENDING); return 0; } /* Acquire BAW Interlock */ static int rshim_byte_acc_lock_acquire(struct pci_dev *pci_dev) { uint32_t read_value = 0; int rc; time_t t0, t1; time(&t0); do { time(&t1); if (difftime(t1, t0) > RSHIM_LOCK_RETRY_TIME) return -ETIMEDOUT; rc = crspace_rsh_gw_read(pci_dev, RSH_BYTE_ACC_INTERLOCK, &read_value); if (rc) return rc; } while (!(read_value & 0x1)); return 0; } /* Release BAW Interlock */ static int rshim_byte_acc_lock_release(struct pci_dev *pci_dev) { return crspace_rsh_gw_write(pci_dev, RSH_BYTE_ACC_INTERLOCK, 0); } /* * Mechanism to do an 8-byte access to the Rshim using * two 4-byte accesses through the Rshim Byte Access Widget. */ static int rshim_byte_acc_read(struct pci_dev *pci_dev, int addr, uint64_t *result) { uint64_t read_result; uint32_t read_value = 0; int rc; /* Wait for RSH_BYTE_ACC_CTL pending bit to be cleared */ rc = rshim_byte_acc_pending_wait(pci_dev); if (rc) return rc; /* Acquire RSH_BYTE_ACC_INTERLOCK */ if (pci_dev->device_id == BLUEFIELD2_DEVICE_ID) { rc = rshim_byte_acc_lock_acquire(pci_dev); if (rc) return rc; } /* Write target address to RSH_BYTE_ACC_ADDR */ rc = crspace_rsh_gw_write(pci_dev, RSH_BYTE_ACC_ADDR, addr); if (rc) goto exit_read; /* Write control and trigger bits to perform read */ rc = crspace_rsh_gw_write(pci_dev, RSH_BYTE_ACC_CTL, RSH_BYTE_ACC_READ_TRIGGER | RSH_BYTE_ACC_SIZE_4BYTE); if (rc) goto exit_read; /* Wait for RSH_BYTE_ACC_CTL pending bit to be cleared */ rc = rshim_byte_acc_pending_wait(pci_dev); if (rc) goto exit_read; /* Read RSH_BYTE_ACC_RDAT to read lower 32-bits of data */ rc = crspace_rsh_gw_read(pci_dev, RSH_BYTE_ACC_RDAT, &read_value); if (rc) goto exit_read; read_result = (uint64_t)read_value; /* Wait for RSH_BYTE_ACC_CTL pending bit to be cleared */ rc = rshim_byte_acc_pending_wait(pci_dev); if (rc) goto exit_read; /* Read RSH_BYTE_ACC_RDAT to read upper 32-bits of data */ rc = crspace_rsh_gw_read(pci_dev, RSH_BYTE_ACC_RDAT, &read_value); if (rc) goto exit_read; read_result |= ((u64)read_value << 32); *result = read_result; exit_read: /* Release RSH_BYTE_ACC_INTERLOCK */ if (pci_dev->device_id == BLUEFIELD2_DEVICE_ID) rc = rshim_byte_acc_lock_release(pci_dev); return rc; } static int rshim_byte_acc_write(struct pci_dev *pci_dev, int addr, uint64_t value) { int rc; /* Acquire RSH_BYTE_ACC_INTERLOCK */ if (pci_dev->device_id == BLUEFIELD2_DEVICE_ID) { rc = rshim_byte_acc_lock_acquire(pci_dev); if (rc) return rc; } /* Write target address to RSH_BYTE_ACC_ADDR */ rc = crspace_rsh_gw_write(pci_dev, RSH_BYTE_ACC_ADDR, addr); if (rc) return rc; /* Write control bits to RSH_BYTE_ACC_CTL */ rc = crspace_rsh_gw_write(pci_dev, RSH_BYTE_ACC_CTL, RSH_BYTE_ACC_SIZE_4BYTE); if (rc) goto exit_write; /* Write lower 32 bits of data to TRIO_CR_GW_DATA */ rc = crspace_rsh_gw_write(pci_dev, RSH_BYTE_ACC_WDAT, (uint32_t)value); if (rc) goto exit_write; /* Wait for RSH_BYTE_ACC_CTL pending bit to be cleared */ rc = rshim_byte_acc_pending_wait(pci_dev); if (rc) goto exit_write; /* Write upper 32 bits of data to TRIO_CR_GW_DATA */ rc = crspace_rsh_gw_write(pci_dev, RSH_BYTE_ACC_WDAT, (uint32_t)(value >> 32)); if (rc) goto exit_write; exit_write: /* Release RSH_BYTE_ACC_INTERLOCK */ if (pci_dev->device_id == BLUEFIELD2_DEVICE_ID) rc = rshim_byte_acc_lock_release(pci_dev); return rc; } /* * The RShim Boot FIFO has a holding register which can couple * two consecutive 4-byte writes into a single 8-byte write * before pushing the data into the FIFO. * Hence the RShim Byte Access Widget is not necessary to write * to the BOOT FIFO using 4-byte writes. */ static int rshim_boot_fifo_write(struct pci_dev *pci_dev, int addr, uint64_t value) { int rc; /* Write lower 32 bits of data to RSH_BOOT_FIFO_DATA */ rc = crspace_rsh_gw_write(pci_dev, addr, (uint32_t)value); if (rc) return rc; /* Write upper 32 bits of data to RSH_BOOT_FIFO_DATA */ rc = crspace_rsh_gw_write(pci_dev, addr, (uint32_t)(value >> 32)); if (rc) return rc; return 0; } /* RShim read/write routines */ static int __attribute__ ((noinline)) rshim_pcie_read(struct rshim_backend *bd, uint32_t chan, uint32_t addr, uint64_t *result, int size) { rshim_pcie_lf_t *dev = container_of(bd, rshim_pcie_lf_t, bd); struct pci_dev *pci_dev = dev->pci_dev; int rc = 0; if (!bd->has_rshim || !bd->has_tm) return -ENODEV; if (bd->drop_mode) { *result = 0; return 0; } if (pci_dev->device_id == BLUEFIELD3_DEVICE_ID) { rc = msn_gw_read(pci_dev, bf3_rshim_pcie_lf_chan_map[chan] + addr, result); } else { dev->write_count = 0; rc = rshim_byte_acc_read(pci_dev, RSH_CHANNEL_BASE(chan) + addr, result); } return rc; } static int __attribute__ ((noinline)) rshim_pcie_write(struct rshim_backend *bd, uint32_t chan, uint32_t addr, uint64_t value, int size) { rshim_pcie_lf_t *dev = container_of(bd, rshim_pcie_lf_t, bd); struct pci_dev *pci_dev = dev->pci_dev; bool is_boot_stream = (addr == RSH_BOOT_FIFO_DATA); uint64_t result; int rc = 0; if (!bd->has_rshim || !bd->has_tm) return -ENODEV; if (bd->drop_mode) return 0; if (pci_dev->device_id == BLUEFIELD3_DEVICE_ID) { rc = msn_gw_write(pci_dev, bf3_rshim_pcie_lf_chan_map[chan] + addr, value); } else { /* * Limitation in BlueField-1 * We cannot stream large numbers of PCIe writes to the RShim's BAR. * Instead, we must write no more than 15 8-byte words before * doing a read from another register within the BAR, * which forces previous writes to drain. * Note that we allow a max write_count of 7 since each 8-byte * write is done using 2 4-byte writes in the boot fifo case. */ if (pci_dev->device_id == BLUEFIELD1_DEVICE_ID) { if (dev->write_count == 7) { __sync_synchronize(); rshim_pcie_read(bd, chan, RSH_SCRATCHPAD1, &result, rc); } dev->write_count++; } if (is_boot_stream) rc = rshim_boot_fifo_write(pci_dev, RSH_CHANNEL_BASE(chan) + addr, value); else rc = rshim_byte_acc_write(pci_dev, RSH_CHANNEL_BASE(chan) + addr, value); } return rc; } static void rshim_pcie_delete(struct rshim_backend *bd) { rshim_pcie_lf_t *dev = container_of(bd, rshim_pcie_lf_t, bd); rshim_deregister(bd); free(dev); } /* Probe routine */ static int rshim_pcie_probe(struct pci_dev *pci_dev) { char dev_name[RSHIM_DEV_NAME_LEN]; struct rshim_backend *bd; rshim_pcie_lf_t *dev; int ret; snprintf(dev_name, sizeof(dev_name) - 1, "pcie-lf-%04x:%02x:%02x.%d", pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func); if (!rshim_allow_device(dev_name)) return -EACCES; RSHIM_INFO("Probing %s\n", dev_name); rshim_lock(); bd = rshim_find_by_name(dev_name); if (bd) { dev = container_of(bd, rshim_pcie_lf_t, bd); } else { dev = calloc(1, sizeof(*dev)); if (dev == NULL) { ret = -ENOMEM; goto error; } bd = &dev->bd; strcpy(bd->dev_name, dev_name); /* Enable drop mode by default if not configured. */ bd->drop_mode = (rshim_drop_mode >= 0) ? rshim_drop_mode : 1; bd->read_rshim = rshim_pcie_read; bd->write_rshim = rshim_pcie_write; bd->destroy = rshim_pcie_delete; dev->write_count = 0; pthread_mutex_init(&bd->mutex, NULL); } rshim_ref(bd); switch (pci_dev->device_id) { case BLUEFIELD3_DEVICE_ID: bd->regs = &bf3_rshim_regs; bd->ver_id = RSHIM_BLUEFIELD_3; break; case BLUEFIELD2_DEVICE_ID: bd->regs = &bf1_bf2_rshim_regs; bd->ver_id = RSHIM_BLUEFIELD_2; break; default: bd->regs = &bf1_bf2_rshim_regs; bd->ver_id = RSHIM_BLUEFIELD_1; break; } bd->rev_id = pci_read_byte(pci_dev, PCI_REVISION_ID); if (rshim_has_pcie_reset_delay || bd->ver_id < RSHIM_BLUEFIELD_3) bd->reset_delay = rshim_pcie_reset_delay; else bd->reset_delay = 1; /* minimum delay for BF3 */ /* Initialize object */ dev->pci_dev = pci_dev; pthread_mutex_lock(&bd->mutex); /* * Register rshim here since it needs to detect whether other backend * has already registered or not, which involves reading/writting rshim * registers and has assumption that the under layer is working. */ bd->has_rshim = 1; bd->has_tm = 1; ret = rshim_register(bd); if (ret) { pthread_mutex_unlock(&bd->mutex); goto rshim_map_failed; } /* Notify that the device is attached */ ret = rshim_notify(bd, RSH_EVENT_ATTACH, 0); pthread_mutex_unlock(&bd->mutex); if (ret) goto rshim_map_failed; rshim_unlock(); return 0; rshim_map_failed: rshim_deref(bd); error: rshim_unlock(); return ret; } int rshim_pcie_lf_init(void) { struct pci_access *pci; struct pci_dev *dev; bool dev_present = false; pci = pci_alloc(); if (!pci) return -ENOMEM; pci_init(pci); pci_scan_bus(pci); /* Iterate over the devices */ for (dev = pci->devices; dev; dev = dev->next) { pci_fill_info(dev, PCI_FILL_IDENT | PCI_FILL_BASES | PCI_FILL_CLASS); if (dev->vendor_id != TILERA_VENDOR_ID || (dev->device_id != BLUEFIELD1_DEVICE_ID && dev->device_id != BLUEFIELD2_DEVICE_ID && dev->device_id != BLUEFIELD3_DEVICE_ID)) continue; rshim_pcie_probe(dev); dev_present = true; } if (!dev_present) return -1; return 0; } ./Mellanox-rshim-user-space-0fd5d3f/src/rshim_regs.c0000664000175000017500000000442314563673752021740 0ustar tai271828tai271828// SPDX-License-Identifier: GPL-2.0 or BSD-3-Clause /* * Copyright (C) 2019-2023 Mellanox Technologies. All Rights Reserved. * */ #include "rshim_regs.h" #include "rshim.h" const struct rshim_regs bf1_bf2_rshim_regs = { .boot_fifo_data = RSH_BOOT_FIFO_DATA, .boot_fifo_count = RSH_BOOT_FIFO_COUNT, .boot_fifo_count_mask = RSH_BOOT_FIFO_COUNT__BOOT_FIFO_COUNT_MASK, .boot_control = RSH_BOOT_CONTROL, .reset_control = RSH_RESET_CONTROL, .scratchpad1 = RSH_SCRATCHPAD1, .scratchpad6 = RSH_SCRATCHPAD6, .tm_htt_sts = RSH_TM_HOST_TO_TILE_STS, .tm_tth_sts = RSH_TM_TILE_TO_HOST_STS, .tm_htt_data = RSH_TM_HOST_TO_TILE_DATA, .tm_tth_data = RSH_TM_TILE_TO_HOST_DATA, .semaphore0 = RSH_SEMAPHORE0, .mem_acc_ctl = RSH_MEM_ACC_CTL, .mem_acc_rsp_cnt = RSH_MEM_ACC_RSP_CNT, .mem_acc_data_first_word = RSH_MEM_ACC_DATA__FIRST_WORD, .device_mstr_priv_lvl = RSH_DEVICE_MSTR_PRIV_LVL, .device_mstr_priv_lvl_shift = RSH_DEVICE_MSTR_PRIV_LVL__MEM_ACC_LVL_SHIFT, .fabric_dim = RSH_FABRIC_DIM, .uptime = RSH_UPTIME, .uptime_por = RSH_UPTIME_POR, .arm_wdg_control_wcs = RSH_ARM_WDG_CONTROL_WCS, .scratch_buf_dat = RSH_SCRATCH_BUF_DAT, .scratch_buf_ctl = RSH_SCRATCH_BUF_CTL }; const struct rshim_regs bf3_rshim_regs = { .boot_fifo_data = BF3_RSH_BOOT_FIFO_DATA, .boot_fifo_count = BF3_RSH_BOOT_FIFO_COUNT, .boot_fifo_count_mask = BF3_RSH_BOOT_FIFO_COUNT__BOOT_FIFO_COUNT_MASK, .boot_control = BF3_RSH_BOOT_CONTROL, .reset_control = BF3_RSH_RESET_CONTROL, .scratchpad1 = BF3_RSH_SCRATCHPAD1, .scratchpad6 = BF3_RSH_SCRATCHPAD6, .tm_htt_sts = BF3_RSH_TM_HOST_TO_TILE_STS, .tm_tth_sts = BF3_RSH_TM_TILE_TO_HOST_STS, .tm_htt_data = BF3_RSH_TM_HOST_TO_TILE_DATA, .tm_tth_data = BF3_RSH_TM_TILE_TO_HOST_DATA, .semaphore0 = BF3_RSH_SEMAPHORE0, .mem_acc_ctl = BF3_RSH_MEM_ACC_CTL, .mem_acc_rsp_cnt = BF3_RSH_MEM_ACC_RSP_CNT, .mem_acc_data_first_word = BF3_RSH_MEM_ACC_DATA__FIRST_WORD, .device_mstr_priv_lvl = BF3_RSH_DEVICE_MSTR_PRIV_LVL, .device_mstr_priv_lvl_shift = BF3_RSH_DEVICE_MSTR_PRIV_LVL__MEM_ACC_LVL_SHIFT, .fabric_dim = BF3_RSH_FABRIC_DIM, .uptime = BF3_RSH_UPTIME, .uptime_por = BF3_RSH_UPTIME_POR, .arm_wdg_control_wcs = BF3_RSH_ARM_WDG_CONTROL_WCS, .scratch_buf_dat = BF3_RSH_SCRATCH_BUF_DAT, .scratch_buf_ctl = BF3_RSH_SCRATCH_BUF_CTL }; ./Mellanox-rshim-user-space-0fd5d3f/src/rshim.c0000664000175000017500000024125714563673752020730 0ustar tai271828tai271828// SPDX-License-Identifier: GPL-2.0 or BSD-3-Clause /* * Copyright (C) 2019-2023 Mellanox Technologies. All Rights Reserved. * */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "rshim.h" /* Maximum number of devices supported (currently it's limited to 64). */ #define RSHIM_MAX_DEV 64 /* RShim timer interval in milliseconds. */ #define RSHIM_TIMER_INTERVAL 1 /* Cycles to poll the network initialization before timeout. */ #define RSHIM_NET_INIT_DELAY (60000 / RSHIM_TIMER_INTERVAL) /* Reserve some space to indicate full. */ #define RSHIM_FIFO_SPACE_RESERV 3 /* Keepalive period in milliseconds. */ static int rshim_keepalive_period = 300; /* Keepalive magic number. */ #define RSHIM_KEEPALIVE_MAGIC_NUM 0x5089836482ULL /* Circular buffer macros. */ #define CIRC_SPACE(head, tail, size) CIRC_CNT((tail), ((head)+1), (size)) #define CIRC_SPACE_TO_END(head, tail, size) \ ({int end = (size) - 1 - (head); \ int n = (end + (tail)) & ((size)-1); \ n <= end ? n : end+1; }) #define CIRC_CNT(head, tail, size) (((head) - (tail)) & ((size)-1)) #define CIRC_CNT_TO_END(head, tail, size) \ ({int end = (size) - (tail); \ int n = ((head) + end) & ((size)-1); \ n < end ? n : end; }) #define read_empty(bd, chan) \ (CIRC_CNT((bd)->read_fifo[chan].head, \ (bd)->read_fifo[chan].tail, READ_FIFO_SIZE) == 0) #define read_full(bd, chan) \ (CIRC_SPACE((bd)->read_fifo[chan].head, \ (bd)->read_fifo[chan].tail, READ_FIFO_SIZE) == 0) #define read_space(bd, chan) \ CIRC_SPACE((bd)->read_fifo[chan].head, \ (bd)->read_fifo[chan].tail, READ_FIFO_SIZE) #define read_cnt(bd, chan) \ CIRC_CNT((bd)->read_fifo[chan].head, \ (bd)->read_fifo[chan].tail, READ_FIFO_SIZE) #define read_cnt_to_end(bd, chan) \ CIRC_CNT_TO_END((bd)->read_fifo[chan].head, \ (bd)->read_fifo[chan].tail, READ_FIFO_SIZE) #define read_data_ptr(bd, chan) \ ((bd)->read_fifo[chan].data + \ ((bd)->read_fifo[chan].tail & (READ_FIFO_SIZE - 1))) #define read_consume_bytes(bd, chan, nbytes) \ ((bd)->read_fifo[chan].tail = \ ((bd)->read_fifo[chan].tail + (nbytes)) & (READ_FIFO_SIZE - 1)) #define read_space_to_end(bd, chan) \ CIRC_SPACE_TO_END((bd)->read_fifo[chan].head, \ (bd)->read_fifo[chan].tail, READ_FIFO_SIZE) #define read_space_offset(bd, chan) \ ((bd)->read_fifo[chan].head & (READ_FIFO_SIZE - 1)) #define read_space_ptr(bd, chan) \ ((bd)->read_fifo[chan].data + read_space_offset(bd, (chan))) #define read_add_bytes(bd, chan, nbytes) \ ((bd)->read_fifo[chan].head = \ ((bd)->read_fifo[chan].head + (nbytes)) & (READ_FIFO_SIZE - 1)) #define read_reset(bd, chan) \ ((bd)->read_fifo[chan].head = (bd)->read_fifo[chan].tail = 0) #define write_empty(bd, chan) \ (CIRC_CNT((bd)->write_fifo[chan].head, \ (bd)->write_fifo[chan].tail, WRITE_FIFO_SIZE) == 0) #define write_full(bd, chan) \ (CIRC_SPACE((bd)->write_fifo[chan].head, \ (bd)->write_fifo[chan].tail, WRITE_FIFO_SIZE) == 0) #define write_space(bd, chan) \ CIRC_SPACE((bd)->write_fifo[chan].head, \ (bd)->write_fifo[chan].tail, WRITE_FIFO_SIZE) #define write_cnt(bd, chan) \ CIRC_CNT((bd)->write_fifo[chan].head, \ (bd)->write_fifo[chan].tail, WRITE_FIFO_SIZE) #define write_cnt_to_end(bd, chan) \ CIRC_CNT_TO_END((bd)->write_fifo[chan].head, \ (bd)->write_fifo[chan].tail, WRITE_FIFO_SIZE) #define write_data_offset(bd, chan) \ ((bd)->write_fifo[chan].tail & (WRITE_FIFO_SIZE - 1)) #define write_data_ptr(bd, chan) \ ((bd)->write_fifo[chan].data + write_data_offset(bd, (chan))) #define write_consume_bytes(bd, chan, nbytes) \ ((bd)->write_fifo[chan].tail = \ ((bd)->write_fifo[chan].tail + (nbytes)) & (WRITE_FIFO_SIZE - 1)) #define write_space_to_end(bd, chan) \ CIRC_SPACE_TO_END((bd)->write_fifo[chan].head, \ (bd)->write_fifo[chan].tail, WRITE_FIFO_SIZE) #define write_space_ptr(bd, chan) \ ((bd)->write_fifo[chan].data + \ ((bd)->write_fifo[chan].head & (WRITE_FIFO_SIZE - 1))) #define write_add_bytes(bd, chan, nbytes) \ ((bd)->write_fifo[chan].head = \ ((bd)->write_fifo[chan].head + (nbytes)) & (WRITE_FIFO_SIZE - 1)) #define write_reset(bd, chan) \ ((bd)->write_fifo[chan].head = (bd)->write_fifo[chan].tail = 0) /* * Tile-to-host bits (UART 0 scratchpad). */ /* * Output write pointer mask. Note that this is the maximum size; the * write pointer may be smaller if requested by the host. */ #define CONS_RSHIM_T2H_OUT_WPTR_MASK 0x3FF /* Tile is done mask. */ #define CONS_RSHIM_T2H_DONE_MASK 0x400 /* * Input read pointer mask. Note that this is the maximum size; the read * pointer may be smaller if requested by the host. */ #define CONS_RSHIM_T2H_IN_RPTR_MASK 0x1FF800 /* Input read pointer shift. */ #define CONS_RSHIM_T2H_IN_RPTR_SHIFT 11 /* Tile is done mask. */ #define CONS_RSHIM_T2H_DONE_MASK 0x400 /* Number of words to send as sync-data (calculated by packet MTU). */ #define TMFIFO_MAX_SYNC_WORDS (1536 / 8) /* Terminal characteristics for newly created consoles. */ #define INIT_C_CC "\003\034\177\025\004\0\1\0\021\023\032\0\022\017\027\026\0" static struct termios init_console_termios = { .c_iflag = INLCR | ICRNL, .c_oflag = OPOST | ONLCR, .c_cflag = B115200 | HUPCL | CLOCAL | CREAD | CS8, .c_lflag = ISIG | ICANON | ECHOE | ECHOK | ECHOCTL | ECHOKE | IEXTEN, .c_cc = INIT_C_CC, }; /* RShim global mutex. */ static pthread_mutex_t rshim_mutex = PTHREAD_MUTEX_INITIALIZER; /* RShim mutex for global fd read/write. */ static pthread_mutex_t rshim_fd_mutex = PTHREAD_MUTEX_INITIALIZER; /* Current timer ticks. */ static int rshim_timer_ticks; /* File handler for the worker function. */ static int rshim_work_fd[2]; /* Current RShim backend name. */ static char *rshim_backend_name; /* Global epoll handler. */ int rshim_epoll_fd; /* Static rshim index (/dev/rshim) and device name. */ int rshim_static_index = -1; char *rshim_static_dev_name; /* Default configuration file. */ const char *rshim_cfg_file = DEFAULT_RSHIM_CONFIG_FILE; static int rshim_display_level = 0; static int rshim_boot_timeout = 150; int rshim_drop_mode = -1; int rshim_usb_reset_delay = 1; bool rshim_has_usb_reset_delay = false; int rshim_pcie_reset_delay = 5; bool rshim_has_pcie_reset_delay = false; int rshim_pcie_enable_vfio = 1; int rshim_pcie_enable_uio = 1; int rshim_pcie_intr_poll_interval = 10; /* Interrupt polling in milliseconds */ /* Array of devices and device names. */ rshim_backend_t *rshim_devs[RSHIM_MAX_DEV]; char *rshim_dev_names[RSHIM_MAX_DEV]; char *rshim_blocked_dev_names[RSHIM_MAX_DEV]; /* Bitmask of the used rshim device id. */ #if RSHIM_MAX_DEV > 64 #error Need to fix the size of rshim_dev_bitmask. #endif uint64_t rshim_dev_bitmask; bool rshim_no_net = false; int rshim_log_level = LOG_NOTICE; bool rshim_daemon_mode = true; volatile bool rshim_run = true; static uint32_t rshim_timer_interval = RSHIM_TIMER_INTERVAL; static void rshim_fifo_msg_update_checksum(rshim_tmfifo_msg_hdr_t *hdr); /* Global lock / unlock. */ void rshim_lock(void) { pthread_mutex_lock(&rshim_mutex); } int rshim_trylock(void) { return pthread_mutex_trylock(&rshim_mutex); } void rshim_unlock(void) { pthread_mutex_unlock(&rshim_mutex); } static int rshim_fd_full_read(int fd, void *data, int len) { char *buf = (char *)data; int cc, total = 0; pthread_mutex_lock(&rshim_fd_mutex); while (len > 0) { cc = read(fd, buf, len); if (cc < 0) { if (errno == EINTR || errno == EAGAIN) { usleep(1000); continue; } pthread_mutex_unlock(&rshim_fd_mutex); return -1; } if (cc == 0) break; buf += cc; total += cc; len -= cc; } pthread_mutex_unlock(&rshim_fd_mutex); return total; } static int rshim_fd_full_write(int fd, void *data, int len) { int total = 0; char *buf = (char *)data; pthread_mutex_lock(&rshim_fd_mutex); while (len > 0) { ssize_t written = write(fd, buf, len); if (written < 0) { if (errno == EINTR || errno == EAGAIN) { usleep(1000); continue; } RSHIM_ERR("fd write error %d\n", (int)written); pthread_mutex_unlock(&rshim_fd_mutex); return written; } total += written; buf += written; len -= written; } pthread_mutex_unlock(&rshim_fd_mutex); return total; } /* Wake up the epoll loop or worker function. */ void rshim_work_signal(rshim_backend_t *bd) { uint8_t index = (uint8_t)-1; bool update = true; if (bd) { if (__sync_bool_compare_and_swap(&bd->work_pending, false, true)) index = (uint8_t)bd->index; else update = false; } if (update) rshim_fd_full_write(rshim_work_fd[1], &index, sizeof(index)); } /* * Read some bytes from RShim. * * The provided buffer size should be multiple of 8 bytes. If not, the * leftover bytes (which presumably were sent as NUL bytes by the sender) * will be discarded. */ static ssize_t rshim_read_default(rshim_backend_t *bd, int devtype, char *buf, size_t count) { int rc, total = 0, avail = 0; uint64_t reg; /* Read is only supported for RShim TMFIFO. */ if (devtype != RSH_DEV_TYPE_TMFIFO) { RSHIM_ERR("bad devtype %d\n", devtype); return -EINVAL; } while (total < count) { if (avail == 0) { reg = 0; rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->tm_tth_sts, ®, RSHIM_REG_SIZE_8B); if (rc < 0 || RSHIM_BAD_CTRL_REG(reg)) break; avail = reg & RSH_TM_TILE_TO_HOST_STS__COUNT_MASK; if (avail == 0) break; } rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->tm_tth_data, ®, RSHIM_REG_SIZE_8B); if (rc < 0) break; /* * Convert it to little endian before sending to RShim. The other side * should decode it as little endian as well which is usually the default * case. */ reg = le64toh(reg); if (total + sizeof(reg) <= count) { *(uint64_t *)buf = reg; buf += sizeof(reg); total += sizeof(reg); } else { /* Copy the rest data which is less than 8 bytes. */ memcpy(buf, ®, count - total); total = count; break; } avail--; } return total; } /* * Write some bytes to the RShim backend. * * If count is not multiple of 8-bytes, the data will be padded to 8-byte * aligned which is required by RShim HW. */ static ssize_t rshim_write_delayed(rshim_backend_t *bd, int devtype, const uint8_t *buf, size_t count) { int size_addr, size_mask, data_addr, max_size; uint8_t pad_buf[sizeof(uint64_t)] = { 0 }; int rc, avail = 0, byte_cnt = 0; time_t t0, t1; uint64_t reg; switch (devtype) { case RSH_DEV_TYPE_TMFIFO: if (bd->is_boot_open || bd->drop_mode) return count; size_addr = bd->regs->tm_htt_sts; size_mask = RSH_TM_HOST_TO_TILE_STS__COUNT_MASK; data_addr = bd->regs->tm_htt_data; max_size = RSH_TM_FIFO_SIZE; break; case RSH_DEV_TYPE_BOOT: size_addr = bd->regs->boot_fifo_count; size_mask = bd->regs->boot_fifo_count_mask; data_addr = bd->regs->boot_fifo_data; max_size = RSH_BOOT_FIFO_SIZE; break; default: RSHIM_ERR("bad devtype %d\n", devtype); return -EINVAL; } while (byte_cnt < count) { /* Check the boot cancel condition. */ if (devtype == RSH_DEV_TYPE_BOOT && !bd->boot_work_buf) break; /* Add padding if less than 8 bytes left. */ if (byte_cnt + sizeof(uint64_t) > count) { memcpy(pad_buf, buf, count - byte_cnt); buf = (const uint8_t *)pad_buf; } time(&t0); while (avail <= 0) { /* Calculate available space in words. */ rc = bd->read_rshim(bd, RSHIM_CHANNEL, size_addr, ®, RSHIM_REG_SIZE_8B); if (rc < 0 || RSHIM_BAD_CTRL_REG(reg)) { RSHIM_ERR("rshim%d read_rshim error addr=0x%x, reg=0x%lx, rc=%d\n", bd->index, size_addr, (long unsigned int)reg, rc); usleep(10000); return count; } avail = max_size - (int)(reg & size_mask) - RSHIM_FIFO_SPACE_RESERV; if (avail > 0) break; if (devtype == RSH_DEV_TYPE_BOOT) goto done; time(&t1); if (difftime(t1, t0) > 3) { if (devtype == RSH_DEV_TYPE_TMFIFO && bd->is_booting) return count; else return -ETIMEDOUT; } } reg = *(uint64_t *)buf; /* * Convert to little endian before sending to RShim. The * receiving side should call le64toh() to convert it back. */ reg = htole64(reg); rc = bd->write_rshim(bd, RSHIM_CHANNEL, data_addr, reg, RSHIM_REG_SIZE_8B); if (rc < 0) { RSHIM_ERR("write_rshim error %d\n", rc); break; } byte_cnt += sizeof(reg); if (buf == pad_buf) break; buf += sizeof(reg); avail--; } /* Return number shouldn't count the padded bytes. */ done: return (byte_cnt > count) ? count : byte_cnt; } static ssize_t rshim_write_default(rshim_backend_t *bd, int devtype, const char *buf, size_t count) { int rc; switch (devtype) { case RSH_DEV_TYPE_TMFIFO: if (bd->is_boot_open) return count; /* Set the flag so there is only one outstanding request. */ bd->spin_flags |= RSH_SFLG_WRITING; /* Wake up the worker. */ bd->fifo_work_buf = (uint8_t *)buf; bd->fifo_work_buf_len = count; bd->fifo_work_devtype = devtype; bd->has_fifo_work = 1; rshim_work_signal(bd); return 0; case RSH_DEV_TYPE_BOOT: bd->boot_work_buf_len = count; bd->boot_work_buf_actual_len = 0; bd->boot_work_buf = (uint8_t *)buf; rshim_work_signal(bd); rc = pthread_cond_wait(&bd->boot_write_complete_cond, &bd->mutex); /* Cancel the request if interrupted. */ if (rc) bd->boot_work_buf = NULL; return bd->boot_work_buf_actual_len; default: RSHIM_ERR("bad devtype %d\n", devtype); return -EINVAL; } } /* Boot file operations routines */ /* * Wait for boot to complete, if necessary. Return 0 if the boot is done * and it's safe to continue, an error code if something went wrong. Note * that this routine must be called with the device mutex held. If it * returns successfully, the mutex will still be held (although it may have * been dropped and reacquired); if it returns unsuccessfully the mutex * will have been dropped. */ static int wait_for_boot_done(rshim_backend_t *bd) { struct timespec ts; int rc; if (!bd->has_reprobe || bd->skip_boot_reset) { bd->is_booting = 0; return 0; } clock_gettime(CLOCK_REALTIME, &ts); ts.tv_sec += 20; if (!bd->has_rshim || bd->is_booting) { while (bd->is_booting) { RSHIM_INFO("boot write, waiting for re-probe\n"); /* * FIXME: might we want a timeout here, too? If the reprobe takes a very * long time, something's probably wrong. Maybe a couple of minutes? */ rc = pthread_cond_timedwait(&bd->boot_complete_cond, &bd->mutex, &ts); if (rc) { RSHIM_DBG("Failed to detect re-probe, continues anyway.\n"); bd->is_booting = 0; return 0; } /* * On some systems the USB up event comes too early while the system * is not fully ready yet. Add a delay here to avoid race codition. */ if (!bd->is_booting && bd->has_reprobe) sleep(bd->reset_delay); } if (!bd->has_rshim) return -ENODEV; } return 0; } static int rshim_reg_indirect_wait(rshim_backend_t *bd, uint64_t resp_count) { int rc, retries = 1000; uint64_t count; while (retries--) { rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->mem_acc_rsp_cnt, &count, RSHIM_REG_SIZE_8B); if (rc) return rc; if (count != resp_count) return 0; } RSHIM_DBG("Rshim byte access widget timeout\n"); return -1; } static int rshim_mmio_write_common(rshim_backend_t *bd, uintptr_t pa, uint8_t size, uint64_t data) { uint64_t reg, resp_count; bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->device_mstr_priv_lvl, ®, RSHIM_REG_SIZE_8B); reg |= 0x1ULL << bd->regs->device_mstr_priv_lvl_shift; bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->device_mstr_priv_lvl, reg, RSHIM_REG_SIZE_8B); bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->mem_acc_rsp_cnt, &resp_count, RSHIM_REG_SIZE_8B); bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->mem_acc_data_first_word, data, RSHIM_REG_SIZE_8B); reg = (((uint64_t)pa & RSH_MEM_ACC_CTL__ADDRESS_RMASK) << RSH_MEM_ACC_CTL__ADDRESS_SHIFT) | (((uint64_t)size & RSH_MEM_ACC_CTL__SIZE_RMASK) << RSH_MEM_ACC_CTL__SIZE_SHIFT) | (1ULL << RSH_MEM_ACC_CTL__WRITE_SHIFT) | (1ULL << RSH_MEM_ACC_CTL__SEND_SHIFT); bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->mem_acc_ctl, reg, RSHIM_REG_SIZE_8B); return rshim_reg_indirect_wait(bd, resp_count); } static int rshim_mmio_read_common(rshim_backend_t *bd, uintptr_t pa, uint8_t size, uint64_t *data) { uint64_t reg, resp_count; bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->device_mstr_priv_lvl, ®, RSHIM_REG_SIZE_8B); reg |= 0x1ULL << bd->regs->device_mstr_priv_lvl_shift; bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->device_mstr_priv_lvl, reg, RSHIM_REG_SIZE_8B); bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->mem_acc_rsp_cnt, &resp_count, RSHIM_REG_SIZE_8B); reg = (((uint64_t)pa & RSH_MEM_ACC_CTL__ADDRESS_RMASK) << RSH_MEM_ACC_CTL__ADDRESS_SHIFT) | (((uint64_t)size & RSH_MEM_ACC_CTL__SIZE_RMASK) << RSH_MEM_ACC_CTL__SIZE_SHIFT) | (1ULL << RSH_MEM_ACC_CTL__SEND_SHIFT); bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->mem_acc_ctl, reg, RSHIM_REG_SIZE_8B); if (rshim_reg_indirect_wait(bd, resp_count)) return -1; bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->mem_acc_data_first_word, ®, RSHIM_REG_SIZE_8B); *data = reg; return 0; } int rshim_mmio_write32(rshim_backend_t *bd, uintptr_t addr, uint32_t value) { return rshim_mmio_write_common(bd, addr, RSH_MEM_ACC_CTL__SIZE_VAL_SZ4, value); } int rshim_mmio_read32(rshim_backend_t *bd, uintptr_t addr, uint32_t *data) { uint64_t reg; if (rshim_mmio_read_common(bd, addr, RSH_MEM_ACC_CTL__SIZE_VAL_SZ4, ®)) { return -1; } else { *data = (uint32_t)reg; return 0; } } static bool rshim_is_livefish(rshim_backend_t *bd) { uint32_t yu_boot = 0, boot_status; int rc; /* No need to check livefish mode for pcie rshim driver. */ if (!strncmp(bd->dev_name, "pcie", 4) && strncmp(bd->dev_name + 4, "-lf", 3)) return false; /* * A value of 1 in yu_boot.boot_status indicates a successful FW * boot, any other value indicates livefish mode. */ rc = rshim_mmio_read32(bd, RSHIM_YU_BASE_ADDR + YU_BOOT, &yu_boot); boot_status = (yu_boot >> 17) & 3; RSHIM_DBG("yu_boot_status: %d\n", boot_status); return (!rc && boot_status != 1); } /* * Write to the RShim reset control register. */ int rshim_reset_control(rshim_backend_t *bd) { uint64_t reg, val; uint8_t shift; int rc; rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->reset_control, ®, RSHIM_REG_SIZE_8B); if (rc < 0) { RSHIM_ERR("failed to read rshim reset control error %d\n", rc); return rc; } val = RSH_RESET_CONTROL__RESET_CHIP_VAL_KEY; shift = RSH_RESET_CONTROL__RESET_CHIP_SHIFT; reg &= ~((uint64_t) RSH_RESET_CONTROL__RESET_CHIP_MASK); reg |= (val << shift); /* * The reset of the ARM can be blocked when the DISABLED bit * is set. The big assumption is that the DISABLED bit would * be hold high for a short period and only the platform code * can reset that bit. Thus the ARM reset can be delayed and * in theory this should not impact the behavior of the RShim * driver. */ rc = bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->reset_control, reg, RSHIM_REG_SIZE_8B); if (rc < 0) { RSHIM_ERR("failed to write rshim reset control error %d\n", rc); return rc; } if (bd->ver_id == RSHIM_BLUEFIELD_2 && bd->rev_id == BLUEFIELD_REV0 && rshim_is_livefish(bd)) { RSHIM_DBG("Apply reset type 13\n"); /* yu.reset_mode_control.reset_mode_control.sw_reset_event_activation13 */ rshim_mmio_write32(bd, RSHIM_YU_BASE_ADDR + YU_RESET_ACTIVATION_13, 1); } return 0; } int rshim_boot_open(rshim_backend_t *bd) { int rc; pthread_mutex_lock(&bd->mutex); if (bd->drop_mode) { RSHIM_INFO("rshim is in drop mode\n"); pthread_mutex_unlock(&bd->mutex); return -EINVAL; } if (bd->is_boot_open) { RSHIM_INFO("can't boot, boot file already open\n"); pthread_mutex_unlock(&bd->mutex); return -EBUSY; } if (!bd->has_rshim) { pthread_mutex_unlock(&bd->mutex); return -ENODEV; } RSHIM_INFO("rshim%d boot open\n", bd->index); bd->is_booting = 1; bd->boot_rem_cnt = 0; /* * Before we reset the chip, make sure we don't have any * outstanding writes, and flush the write and read FIFOs. (Note * that we can't have any outstanding reads, since we kill those * upon release of the TM FIFO file.) */ if (bd->cancel) bd->cancel(bd, RSH_DEV_TYPE_TMFIFO, true); /* Reset the TmFifo. */ rshim_fifo_reset(bd); /* Set RShim (external) boot mode. */ rc = bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->boot_control, RSH_BOOT_CONTROL__BOOT_MODE_VAL_NONE, RSHIM_REG_SIZE_8B); if (rc) { RSHIM_ERR("boot_open: error %d writing boot control\n", rc); bd->is_booting = 0; pthread_mutex_unlock(&bd->mutex); return rc; } bd->is_boot_open = 1; /* * Disable the watchdog. The channel and offset are the same on all * the BlueField SoC so far. */ bd->write_rshim(bd, RSH_MMIO_ADDRESS_SPACE__CHANNEL_VAL_WDOG1, bd->regs->arm_wdg_control_wcs, 0, RSHIM_REG_SIZE_8B); if (bd->skip_boot_reset) goto boot_open_done; /* SW reset. */ rc = rshim_reset_control(bd); /* * Note that occasionally, we get various errors on writing to * the reset register. This appears to be caused by the chip * actually resetting before the response goes out, or perhaps by * our noticing the device unplug before we've seen the response. * Either way, the chip _does_ actually reset, so we just ignore * the error. Should we ever start getting these errors without * the chip being reset, we'll have to figure out how to handle * this more intelligently. (One potential option is to not reset * directly, but to set up a down counter to do the reset, but that * seems kind of kludgy, especially since Tile software might also * be trying to use the down counter.) */ if (rc && rc != -EPROTO && rc != -ESHUTDOWN && rc != -ETIMEDOUT && rc != -EPIPE) { RSHIM_ERR("boot_open: error %d writing reset control\n", rc); bd->is_boot_open = 0; pthread_mutex_unlock(&bd->mutex); return rc; } if (rc) RSHIM_ERR("boot_open: got error %d on reset write\n", rc); boot_open_done: rshim_ref(bd); /* * PCIe doesn't have the disconnect/reconnect behavior. * Add a small delay for the reset. */ if (!bd->has_reprobe) sleep(bd->reset_delay); time(&bd->boot_write_time); pthread_mutex_unlock(&bd->mutex); return 0; } int rshim_boot_write(rshim_backend_t *bd, const char *user_buffer, size_t count, int (*copy_in)(void *dest, const void *src, int count)) { int rc = 0, whichbuf = 0, len; time_t tm; size_t bytes_written = 0; pthread_mutex_lock(&bd->mutex); if (bd->is_in_boot_write) { pthread_mutex_unlock(&bd->mutex); return -EBUSY; } rc = wait_for_boot_done(bd); if (rc) { RSHIM_ERR("boot_write: wait for boot failed, err %d\n", rc); pthread_mutex_unlock(&bd->mutex); return rc; } /* * We're going to drop the mutex while we wait for any outstanding * write to complete; this keeps another thread from getting in here * while we do that. */ bd->is_in_boot_write = 1; while (count + bd->boot_rem_cnt >= sizeof(uint64_t)) { size_t buf_bytes = MIN(BOOT_BUF_SIZE, (count + bd->boot_rem_cnt) & (-((size_t)8))); char *buf = bd->boot_buf[whichbuf]; whichbuf ^= 1; /* Copy the previous remaining data first. */ if (bd->boot_rem_cnt) memcpy(buf, &bd->boot_rem_data, bd->boot_rem_cnt); rc = copy_in(buf + bd->boot_rem_cnt, user_buffer, buf_bytes - bd->boot_rem_cnt); if (rc < 0) break; rc = bd->write(bd, RSH_DEV_TYPE_BOOT, buf, buf_bytes); if (rc > bd->boot_rem_cnt) { len = rc - bd->boot_rem_cnt; count -= len; user_buffer += len; bytes_written += len; bd->boot_rem_cnt = 0; } else if (rc == 0) { time(&tm); if (difftime(tm, bd->boot_write_time) > bd->boot_timeout) { rc = -ETIMEDOUT; RSHIM_INFO("rshim%d boot timeout\n", bd->index); } else { rc = -EINTR; } break; } time(&bd->boot_write_time); if (rc != buf_bytes) break; } /* Buffer the remaining data. */ if (count + bd->boot_rem_cnt < sizeof(bd->boot_rem_data)) { rc = copy_in((uint8_t *)&bd->boot_rem_data + bd->boot_rem_cnt, user_buffer, count); bd->boot_rem_cnt += count; bytes_written += count; } bd->is_in_boot_write = 0; pthread_mutex_unlock(&bd->mutex); if (bytes_written > 0 || count == 0) return bytes_written; else return rc; } void rshim_boot_release(rshim_backend_t *bd) { int rc; pthread_mutex_lock(&bd->mutex); /* Restore the boot mode register. */ rc = bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->boot_control, RSH_BOOT_CONTROL__BOOT_MODE_VAL_EMMC, RSHIM_REG_SIZE_8B); if (rc) RSHIM_ERR("couldn't set boot_control, err %d\n", rc); /* Flush the leftover data with zeros padded. */ if (bd->boot_rem_cnt) { memset((uint8_t *)&bd->boot_rem_data + bd->boot_rem_cnt, 0, sizeof(uint64_t) - bd->boot_rem_cnt); bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->boot_fifo_data, bd->boot_rem_data, RSHIM_REG_SIZE_8B); } bd->is_boot_open = 0; bd->boot_rem_cnt = 0; rshim_work_signal(bd); pthread_mutex_unlock(&bd->mutex); RSHIM_INFO("rshim%d boot close\n", bd->index); rshim_deref(bd); } /* FIFO common routines */ /* * Signal an error on the FIFO, and wake up anyone who might need to know * about it. */ static void rshim_fifo_err(rshim_backend_t *bd, int err) { int i; bd->tmfifo_error = err; pthread_cond_broadcast(&bd->fifo_write_complete_cond); for (i = 0; i < TMFIFO_MAX_CHAN; i++) { pthread_cond_broadcast(&bd->read_fifo[i].operable); pthread_cond_broadcast(&bd->write_fifo[i].operable); } } static int rshim_fifo_tx_avail(rshim_backend_t *bd) { uint64_t word; int rc, max_size, avail; /* Get FIFO max size. */ max_size = RSH_TM_FIFO_SIZE; /* Calculate available size. */ rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->tm_htt_sts, &word, RSHIM_REG_SIZE_8B); if (rc < 0 || RSHIM_BAD_CTRL_REG(word)) { RSHIM_ERR("rshim%d read_rshim error %d\n", bd->index, rc); usleep(10000); return -1; } avail = max_size - (int)(word & RSH_TM_HOST_TO_TILE_STS__COUNT_MASK) - RSHIM_FIFO_SPACE_RESERV; return avail; } static int rshim_fifo_sync(rshim_backend_t *bd) { rshim_tmfifo_msg_hdr_t hdr; int i, avail, rc; avail = rshim_fifo_tx_avail(bd); if (avail < 0) return avail; hdr.data = 0; hdr.type = VIRTIO_ID_NET; rshim_fifo_msg_update_checksum(&hdr); for (i = 0; i < avail; i++) { rc = bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->tm_htt_data, hdr.data, RSHIM_REG_SIZE_8B); if (rc) return rc; } return 0; } /* Just adds up all the bytes of the header. */ static uint8_t rshim_fifo_ctrl_checksum(rshim_tmfifo_msg_hdr_t *hdr) { uint8_t checksum = 0; int i; for (i = 0; i < sizeof(*hdr); i++) checksum += ((uint8_t *)hdr)[i]; return checksum; } static void rshim_fifo_msg_update_checksum(rshim_tmfifo_msg_hdr_t *hdr) { uint8_t checksum; hdr->checksum = 0; checksum = rshim_fifo_ctrl_checksum(hdr); hdr->checksum = ~checksum + 1; } static bool rshim_fifo_msg_verify_checksum(rshim_tmfifo_msg_hdr_t *hdr) { uint8_t checksum = 0; /* * hdr->checksum is either 0 (old version) or should have a valid checksum. */ if (hdr->checksum) checksum = rshim_fifo_ctrl_checksum(hdr); return checksum ? false : true; } static void rshim_fifo_ctrl_rx(rshim_backend_t *bd, rshim_tmfifo_msg_hdr_t *hdr) { if (!rshim_fifo_msg_verify_checksum(hdr)) return; switch (hdr->type) { case TMFIFO_MSG_MAC_1: memcpy(bd->peer_mac, hdr->mac, 3); break; case TMFIFO_MSG_MAC_2: memcpy(bd->peer_mac + 3, hdr->mac, 3); break; case TMFIFO_MSG_VLAN_ID: bd->vlan[0] = ntohs(hdr->vlan[0]); bd->vlan[1] = ntohs(hdr->vlan[1]); break; case TMFIFO_MSG_PXE_ID: bd->pxe_client_id = ntohl(hdr->pxe_id); /* Last info to receive, set the flag. */ bd->peer_ctrl_resp = 1; pthread_cond_broadcast(&bd->ctrl_wait_cond); break; default: return; } } static int rshim_fifo_ctrl_tx(rshim_backend_t *bd) { rshim_tmfifo_msg_hdr_t hdr; int len = 0; if (bd->peer_mac_set) { bd->peer_mac_set = 0; hdr.data = 0; hdr.type = TMFIFO_MSG_MAC_1; memcpy(hdr.mac, bd->peer_mac, 3); rshim_fifo_msg_update_checksum(&hdr); memcpy(bd->write_buf, &hdr.data, sizeof(hdr.data)); hdr.type = TMFIFO_MSG_MAC_2; memcpy(hdr.mac, bd->peer_mac + 3, 3); rshim_fifo_msg_update_checksum(&hdr); memcpy(bd->write_buf + sizeof(hdr.data), &hdr.data, sizeof(hdr.data)); len = sizeof(hdr.data) * 2; } else if (bd->peer_pxe_id_set) { bd->peer_pxe_id_set = 0; hdr.data = 0; hdr.type = TMFIFO_MSG_PXE_ID; hdr.pxe_id = htonl(bd->pxe_client_id); rshim_fifo_msg_update_checksum(&hdr); memcpy(bd->write_buf, &hdr.data, sizeof(hdr.data)); len = sizeof(hdr.data); } else if (bd->peer_vlan_set) { bd->peer_vlan_set = 0; hdr.data = 0; hdr.type = TMFIFO_MSG_VLAN_ID; hdr.vlan[0] = htons(bd->vlan[0]); hdr.vlan[1] = htons(bd->vlan[1]); rshim_fifo_msg_update_checksum(&hdr); memcpy(bd->write_buf, &hdr.data, sizeof(hdr.data)); len = sizeof(hdr.data); } else if (bd->peer_ctrl_req) { bd->peer_ctrl_req = 0; hdr.data = 0; hdr.type = TMFIFO_MSG_CTRL_REQ; rshim_fifo_msg_update_checksum(&hdr); memcpy(bd->write_buf, &hdr.data, sizeof(hdr.data)); len = sizeof(hdr.data); } return len; } static int rshim_got_peer_signal(void) { #ifdef HAVE_RSHIM_FUSE return rshim_fuse_got_peer_signal(); #else return -1; #endif } static void rshim_input_notify(rshim_backend_t *bd) { #ifdef HAVE_RSHIM_FUSE rshim_fuse_input_notify(bd); #endif } /* Drain the read buffer, and start another read/interrupt if needed. */ static void rshim_fifo_input(rshim_backend_t *bd) { rshim_tmfifo_msg_hdr_t *hdr; uint8_t rx_avail = 0; time_t t0, t1; int rc; if (!bd->has_rshim || !bd->has_tm) return; time(&t0); again: while (bd->read_buf_next < bd->read_buf_bytes) { int copysize; /* * If we're at the start of a packet, then extract the * header, and update our count of bytes remaining in the * packet. */ if (bd->read_buf_pkt_rem == 0) { /* Make sure header is received. */ if (bd->read_buf_next + sizeof(*hdr) > bd->read_buf_bytes) break; RSHIM_DBG("read_buf_next %d\n", bd->read_buf_next); hdr = (rshim_tmfifo_msg_hdr_t *)&bd->read_buf[bd->read_buf_next]; /* Verify message size. */ if ((hdr->type == VIRTIO_ID_NET) && (ntohs(hdr->len) + sizeof(*hdr) > sizeof(rshim_net_pkt_t))) { bd->read_buf_next += sizeof(*hdr); continue; } bd->read_buf_pkt_rem = ntohs(hdr->len) + sizeof(*hdr); bd->read_buf_pkt_padding = (8 - (bd->read_buf_pkt_rem & 7)) & 7; if (hdr->type == VIRTIO_ID_NET) bd->rx_chan = TMFIFO_NET_CHAN; else if (hdr->type == VIRTIO_ID_CONSOLE) { bd->rx_chan = TMFIFO_CONS_CHAN; /* Strip off the message header for console. */ bd->read_buf_next += sizeof(*hdr); bd->read_buf_pkt_rem -= sizeof(*hdr); if (bd->read_buf_pkt_rem == 0) continue; } else { RSHIM_DBG("bad type %d, drop it\n", hdr->type); bd->read_buf_pkt_rem = 0; bd->read_buf_pkt_padding = 0; if (hdr->len == 0) { bd->read_buf_next += sizeof(*hdr); rshim_fifo_ctrl_rx(bd, hdr); continue; } else { RSHIM_DBG("bad type %d, drop it", hdr->type); bd->read_buf_next = bd->read_buf_bytes; break; } } RSHIM_DBG("drain: hdr, nxt %d rem %d chn %d\n", bd->read_buf_next, bd->read_buf_pkt_rem, bd->rx_chan); bd->drop_pkt = 0; } if (bd->rx_chan == TMFIFO_CONS_CHAN && !(bd->spin_flags & RSH_SFLG_CONS_OPEN)) { /* * If data is coming in for a closed console channel, we want to just * throw it away. Resetting the channel every time through this loop is * a relatively cheap way to do that. Note that this works because the * read buffer is no larger than the read FIFO; thus, we know that if * we reset it here, we will always be able to drain the read buffer of * any console data, and will then launch another read. */ read_reset(bd, TMFIFO_CONS_CHAN); bd->drop_pkt = 1; } else if (bd->rx_chan == TMFIFO_NET_CHAN && bd->net_notify_fd[0] < 0) { /* Drop if networking is not enabled. */ read_reset(bd, TMFIFO_NET_CHAN); bd->drop_pkt = 1; } copysize = MIN(bd->read_buf_pkt_rem, bd->read_buf_bytes - bd->read_buf_next); copysize = MIN(copysize, read_space_to_end(bd, bd->rx_chan)); RSHIM_DBG("drain: copysize %d, head %d, tail %d, remaining %d\n", copysize, bd->read_fifo[bd->rx_chan].head, bd->read_fifo[bd->rx_chan].tail, bd->read_buf_pkt_rem); if (copysize == 0) { /* We have data, but no space to put it in, so we're done. */ RSHIM_DBG("drain: no more space in channel %d\n", bd->rx_chan); break; } if (!bd->drop_pkt) { memcpy(read_space_ptr(bd, bd->rx_chan), &bd->read_buf[bd->read_buf_next], copysize); read_add_bytes(bd, bd->rx_chan, copysize); } bd->read_buf_next += copysize; bd->read_buf_pkt_rem -= copysize; rshim_input_notify(bd); pthread_cond_broadcast(&bd->read_fifo[bd->rx_chan].operable); if (bd->read_buf_pkt_rem <= 0) { bd->read_buf_next = bd->read_buf_next + bd->read_buf_pkt_padding; rx_avail = 1; } } /* * We've processed all of the data we can, so now we decide if we * need to launch another I/O. If there's still data in the read * buffer, or if we're already reading, don't launch any new * operations. If an interrupt just completed, and said there was * data, or the last time we did a read we got some data, then do * another read. Otherwise, do an interrupt. */ if (bd->read_buf_next < bd->read_buf_bytes || (bd->spin_flags & RSH_SFLG_READING)) { /* We're doing nothing. */ RSHIM_DBG("fifo_input: no new read: %s\n", (bd->read_buf_next < bd->read_buf_bytes) ? "have data" : "already reading"); } else { int len; /* Process it if more data is received. */ len = bd->read(bd, RSH_DEV_TYPE_TMFIFO, (char *)bd->read_buf, READ_BUF_SIZE); if (len > 0) { bd->read_buf_bytes = len; bd->read_buf_next = 0; time(&t1); if (difftime(t1, t0) > 2) { /* Reschedule it in the work handler to avoid stuck. */ bd->has_cons_work = 1; rshim_work_signal(bd); } else { goto again; } } } if (rx_avail && bd->rx_chan == TMFIFO_NET_CHAN) { if (__sync_bool_compare_and_swap(&bd->net_rx_pending, false, true)) { do { rc = write(bd->net_notify_fd[1], &rx_avail, sizeof(rx_avail)); } while (rc == -1 && (errno == EINTR || errno == EAGAIN)); } } } ssize_t rshim_fifo_read(rshim_backend_t *bd, char *buffer, size_t count, int chan, bool nonblock) { struct timespec ts; size_t rd_cnt = 0; pthread_mutex_lock(&bd->mutex); while (count) { size_t readsize; int pass1; int pass2; RSHIM_DBG("fifo_read, top of loop, remaining count %zd\n", count); /* * We check this each time through the loop since the * device could get disconnected while we're waiting for * more data in the read FIFO. */ if (!bd->has_tm) { pthread_mutex_unlock(&bd->mutex); RSHIM_DBG("fifo_read: returning %zd/ENODEV\n", rd_cnt); return rd_cnt ? rd_cnt : -ENODEV; } if (bd->tmfifo_error) { pthread_mutex_unlock(&bd->mutex); RSHIM_DBG("fifo_read: returning %zd/%d\n", rd_cnt, bd->tmfifo_error); return rd_cnt ? rd_cnt : bd->tmfifo_error; } if (read_empty(bd, chan)) { RSHIM_DBG("fifo_read: fifo empty\n"); if (rd_cnt || nonblock) { if (rd_cnt == 0) { pthread_mutex_lock(&bd->ringlock); rshim_fifo_input(bd); pthread_mutex_unlock(&bd->ringlock); } pthread_mutex_unlock(&bd->mutex); RSHIM_DBG("fifo_read: returning %zd/EAGAIN\n", rd_cnt); return rd_cnt ? rd_cnt : -EAGAIN; } RSHIM_DBG("fifo_read: waiting for readable chan %d\n", chan); while (read_empty(bd, chan)) { clock_gettime(CLOCK_REALTIME, &ts); ts.tv_sec += 1; if (pthread_cond_timedwait(&bd->read_fifo[chan].operable, &bd->mutex, &ts)) { RSHIM_DBG("fifo_read: returning ERESTARTSYS\n"); pthread_mutex_unlock(&bd->mutex); return -EINTR; } if (rshim_got_peer_signal() == 0) { pthread_mutex_unlock(&bd->mutex); return -EINTR; } } /* * Since we dropped the mutex, we must make sure our interface is still * there before we do anything else. */ continue; } /* Figure out how many bytes we will transfer on this pass. */ pthread_mutex_lock(&bd->ringlock); readsize = MIN(count, (size_t)read_cnt(bd, chan)); pass1 = MIN(readsize, (size_t)read_cnt_to_end(bd, chan)); pass2 = readsize - pass1; RSHIM_DBG("fifo_read: readsize %zd, head %d, tail %d\n", readsize, bd->read_fifo[chan].head, bd->read_fifo[chan].tail); memcpy(buffer, read_data_ptr(bd, chan), pass1); if (pass2) memcpy(buffer + pass1, bd->read_fifo[chan].data, pass2); read_consume_bytes(bd, chan, readsize); /* Check if there is any more incoming data. */ rshim_fifo_input(bd); pthread_mutex_unlock(&bd->ringlock); count -= readsize; buffer += readsize; rd_cnt += readsize; RSHIM_DBG("fifo_read: transferred %zd bytes\n", readsize); } pthread_mutex_unlock(&bd->mutex); RSHIM_DBG("fifo_read: returning %zd\n", rd_cnt); return rd_cnt; } static void rshim_fifo_output(rshim_backend_t *bd) { int writesize, write_buf_next = 0, write_avail; int numchan = TMFIFO_MAX_CHAN; int chan, chan_offset, fifo_avail; /* If we're already writing, we have nowhere to put data. */ if (bd->spin_flags & RSH_SFLG_WRITING) return; if (bd->has_reprobe) fifo_avail = WRITE_BUF_SIZE; else fifo_avail = rshim_fifo_tx_avail(bd) * sizeof(uint64_t); write_avail = fifo_avail - write_buf_next; if (!bd->write_buf_pkt_rem) { /* Send control messages. */ writesize = rshim_fifo_ctrl_tx(bd); if (writesize > 0) { write_avail -= writesize; write_buf_next += writesize; } } /* Walk through all the channels, sending as much data as possible. */ for (chan_offset = 0; chan_offset < numchan && write_avail > 0; chan_offset++) { /* * Pick the current channel if not done, otherwise round-robin * to the next channel. */ if (bd->write_buf_pkt_rem > 0) chan = bd->tx_chan; else { rshim_tmfifo_msg_hdr_t *hdr = &bd->msg_hdr; uint16_t cur_len; chan = bd->tx_chan = (bd->tx_chan + 1) % numchan; cur_len = write_cnt(bd, chan); /* * Set up message header for console data which is byte * stream. Network packets already have the message * header included. */ if (chan == TMFIFO_CONS_CHAN) { if (cur_len == 0) continue; hdr->data = 0; hdr->type = VIRTIO_ID_CONSOLE; hdr->len = htons(cur_len); } else { int pass1; if (cur_len < sizeof(rshim_tmfifo_msg_hdr_t)) continue; pass1 = write_cnt_to_end(bd, chan); if (pass1 >= sizeof(*hdr)) { hdr = (rshim_tmfifo_msg_hdr_t *) write_data_ptr(bd, chan); } else { memcpy(hdr, write_data_ptr(bd, chan), pass1); memcpy((uint8_t *)hdr + pass1, bd->write_fifo[chan].data, sizeof(*hdr) - pass1); } } /* Calculate checksum for this header. */ rshim_fifo_msg_update_checksum(hdr); bd->write_buf_pkt_rem = ntohs(hdr->len) + sizeof(*hdr); } /* Send out the packet header for the console data. */ if (chan == TMFIFO_CONS_CHAN && bd->write_buf_pkt_rem > ntohs(bd->msg_hdr.len)) { rshim_tmfifo_msg_hdr_t *hdr = &bd->msg_hdr; int left = bd->write_buf_pkt_rem - ntohs(hdr->len); uint8_t *pos = (uint8_t *)hdr + sizeof(*hdr) - left; writesize = MIN(write_avail, left); memcpy(&bd->write_buf[write_buf_next], pos, writesize); write_buf_next += writesize; bd->write_buf_pkt_rem -= writesize; write_avail -= writesize; /* * Don't continue if no more space for the header. It'll be picked up * next time. */ if (left != writesize) break; } writesize = MIN(write_avail, (int)write_cnt(bd, chan)); writesize = MIN(writesize, bd->write_buf_pkt_rem); /* * The write size should be aligned to 8 bytes unless for the * last block, which will be padded at the end. */ if (bd->write_buf_pkt_rem != writesize) writesize &= -8; if (writesize > 0) { int pass1; int pass2; pass1 = MIN(writesize, (int)write_cnt_to_end(bd, chan)); pass2 = writesize - pass1; RSHIM_DBG("fifo_output: chan %d, writesize %d, next %d," " head %d, tail %d\n", chan, writesize, write_buf_next, bd->write_fifo[chan].head, bd->write_fifo[chan].tail); memcpy(&bd->write_buf[write_buf_next], write_data_ptr(bd, chan), pass1); memcpy(&bd->write_buf[write_buf_next + pass1], bd->write_fifo[chan].data, pass2); write_consume_bytes(bd, chan, writesize); write_buf_next += writesize; bd->write_buf_pkt_rem -= writesize; /* Add padding at the end. */ if (bd->write_buf_pkt_rem == 0) write_buf_next = (write_buf_next + 7) & -8; write_avail = fifo_avail - write_buf_next; pthread_cond_broadcast(&bd->write_fifo[chan].operable); RSHIM_DBG("fifo_output: woke up writable chan %d\n", chan); } } /* Drop the data if it is still booting. */ if (bd->is_boot_open || bd->drop_mode || !bd->has_rshim || !bd->has_tm) return; /* If we actually put anything in the buffer, send it. */ if (write_buf_next) bd->write(bd, RSH_DEV_TYPE_TMFIFO, (char *)bd->write_buf, write_buf_next); } int rshim_fifo_alloc(rshim_backend_t *bd) { int i; for (i = 0; i < TMFIFO_MAX_CHAN; i++) { if (!bd->read_fifo[i].data) bd->read_fifo[i].data = malloc(READ_FIFO_SIZE); if (!bd->write_fifo[i].data) bd->write_fifo[i].data = malloc(WRITE_FIFO_SIZE); } return 0; } void rshim_fifo_reset(rshim_backend_t *bd) { int i; bd->read_buf_bytes = 0; bd->read_buf_pkt_rem = 0; bd->read_buf_next = 0; bd->read_buf_pkt_padding = 0; bd->write_buf_pkt_rem = 0; bd->rx_chan = bd->tx_chan = 0; pthread_mutex_lock(&bd->ringlock); bd->spin_flags &= ~(RSH_SFLG_WRITING | RSH_SFLG_READING); for (i = 0; i < TMFIFO_MAX_CHAN; i++) { read_reset(bd, i); write_reset(bd, i); } pthread_mutex_unlock(&bd->ringlock); } void rshim_fifo_free(rshim_backend_t *bd) { int i; for (i = 0; i < TMFIFO_MAX_CHAN; i++) { free(bd->read_fifo[i].data); bd->read_fifo[i].data = NULL; free(bd->write_fifo[i].data); bd->write_fifo[i].data = NULL; } rshim_fifo_reset(bd); bd->has_tm = 0; } ssize_t rshim_fifo_write(rshim_backend_t *bd, const char *buffer, size_t count, int chan, bool nonblock) { size_t wr_cnt = 0; pthread_mutex_lock(&bd->mutex); while (count) { size_t writesize; int pass1; int pass2; /* * We check this each time through the loop since the * device could get disconnected while we're waiting for * more space in the write buffer. */ if (!bd->has_tm) { pthread_mutex_unlock(&bd->mutex); RSHIM_DBG("fifo_write: returning %zd/ENODEV\n", wr_cnt); return wr_cnt ? wr_cnt : -ENODEV; } if (bd->tmfifo_error) { pthread_mutex_unlock(&bd->mutex); RSHIM_DBG("fifo_write: returning %zd/%d\n", wr_cnt, bd->tmfifo_error); return wr_cnt ? wr_cnt : bd->tmfifo_error; } if (write_full(bd, chan)) { RSHIM_DBG("fifo_write: fifo full\n"); /* Try to send more data. */ pthread_mutex_lock(&bd->ringlock); rshim_fifo_output(bd); pthread_mutex_unlock(&bd->ringlock); if (nonblock) { pthread_mutex_unlock(&bd->mutex); RSHIM_DBG("fifo_write: returning %zd/EAGAIN\n", wr_cnt); return wr_cnt ? wr_cnt : -EAGAIN; } RSHIM_DBG("fifo_write: waiting for writable chan %d\n", chan); while (write_full(bd, chan)) { if (pthread_cond_wait(&bd->write_fifo[chan].operable, &bd->mutex)) { RSHIM_DBG("fifo_write: returning %zd/ERESTARTSYS\n", wr_cnt); pthread_mutex_unlock(&bd->mutex); return wr_cnt ? wr_cnt : -EAGAIN; } if (rshim_got_peer_signal() == 0) { pthread_mutex_unlock(&bd->mutex); return -EINTR; } } /* * Since we dropped the mutex, we must make sure our interface is still * there before we do anything else. */ continue; } pthread_mutex_lock(&bd->ringlock); writesize = MIN(count, (size_t)write_space(bd, chan)); pass1 = MIN(writesize, (size_t)write_space_to_end(bd, chan)); pass2 = writesize - pass1; pthread_mutex_unlock(&bd->ringlock); RSHIM_DBG("fifo_write: writesize %zd, head %d, tail %d\n", writesize, bd->write_fifo[chan].head, bd->write_fifo[chan].tail); memcpy(write_space_ptr(bd, chan), buffer, pass1); if (pass2) memcpy(bd->write_fifo[chan].data, buffer + pass1, pass2); pthread_mutex_lock(&bd->ringlock); write_add_bytes(bd, chan, writesize); /* We have some new bytes, let's see if we can write any. */ rshim_fifo_output(bd); pthread_mutex_unlock(&bd->ringlock); count -= writesize; buffer += writesize; wr_cnt += writesize; RSHIM_DBG("fifo_write: transferred %zd bytes this pass\n", writesize); } pthread_mutex_unlock(&bd->mutex); RSHIM_DBG("fifo_write: returning %zd\n", wr_cnt); return wr_cnt; } static void rshim_work_handler(rshim_backend_t *bd) { int rc; pthread_mutex_lock(&bd->mutex); bd->work_pending = false; if (bd->keepalive && bd->has_rshim && !bd->debug_code) { bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->scratchpad1, RSHIM_KEEPALIVE_MAGIC_NUM, RSHIM_REG_SIZE_8B); bd->keepalive = 0; } if (bd->boot_work_buf != NULL) { bd->boot_work_buf_actual_len = rshim_write_delayed(bd, RSH_DEV_TYPE_BOOT, bd->boot_work_buf, bd->boot_work_buf_len); bd->boot_work_buf = NULL; pthread_cond_broadcast(&bd->boot_write_complete_cond); } if (!rshim_no_net && bd->net_fd < 0 && (rshim_timer_ticks - bd->net_init_time) < RSHIM_NET_INIT_DELAY) { rc = rshim_net_init(bd); if (!rc) { bd->is_net_open = 1; pthread_mutex_lock(&bd->ringlock); rshim_fifo_input(bd); pthread_mutex_unlock(&bd->ringlock); } } if (bd->is_boot_open || bd->is_booting) { if (bd->is_boot_open && bd->has_cons_work) rshim_fifo_input(bd); pthread_mutex_unlock(&bd->mutex); return; } if (bd->has_fifo_work) { int len; len = rshim_write_delayed(bd, bd->fifo_work_devtype, bd->fifo_work_buf, bd->fifo_work_buf_len); bd->has_fifo_work = 0; pthread_mutex_lock(&bd->ringlock); bd->spin_flags &= ~RSH_SFLG_WRITING; if (len == bd->fifo_work_buf_len) { pthread_cond_broadcast(&bd->fifo_write_complete_cond); rshim_notify(bd, RSH_EVENT_FIFO_OUTPUT, 0); } else { RSHIM_DBG("fifo_write: completed abnormally (%d)\n", len); } pthread_mutex_unlock(&bd->ringlock); } if (bd->has_cons_work) { pthread_mutex_lock(&bd->ringlock); /* FIFO output. */ rshim_fifo_output(bd); /* FIFO input. */ rshim_fifo_input(bd); pthread_mutex_unlock(&bd->ringlock); bd->has_cons_work = 0; } if (!bd->has_reprobe && bd->is_cons_open) { bd->has_cons_work = 1; if (bd->timer - rshim_timer_ticks > 100) bd->timer = rshim_timer_ticks + 100; } pthread_mutex_unlock(&bd->mutex); } static int rshim_boot_done(rshim_backend_t *bd) { if (bd->has_rshim && bd->has_tm) { /* Clear any previous errors. */ bd->tmfifo_error = 0; /* * If someone might be waiting for the device to come up, * tell them it's ready. */ if (bd->is_booting) { bd->is_booting = 0; RSHIM_DBG("signaling booting complete\n"); pthread_cond_broadcast(&bd->boot_complete_cond); }; /* If the console device is open, start the worker. */ if (bd->is_cons_open && !bd->has_cons_work) { bd->has_cons_work = 1; RSHIM_DBG("probe: console_work submitted\n"); rshim_work_signal(bd); } /* Tell the user this device is now attached. */ RSHIM_INFO("rshim%d attached\n", bd->index); } return 0; } int rshim_fifo_fsync(rshim_backend_t *bd, int chan) { int rc = 0; pthread_mutex_lock(&bd->mutex); /* * To ensure that all of our data has actually made it to the * device, we first wait until the channel is empty, then we wait * until there is no outstanding write urb. */ while (!write_empty(bd, chan)) { if (pthread_cond_wait(&bd->write_fifo[chan].operable, &bd->mutex)) { rc = -EINTR; break; } if (rshim_got_peer_signal() == 0) { rc = -EINTR; break; } } while (!rc && (bd->spin_flags & RSH_SFLG_WRITING)) { if (pthread_cond_wait(&bd->fifo_write_complete_cond, &bd->mutex)) { rc = -EINTR; break; } if (rshim_got_peer_signal() == 0) { rc = -EINTR; break; } } pthread_mutex_unlock(&bd->mutex); return rc; } void rshim_fifo_check_poll(rshim_backend_t *bd, int chan, bool *poll_rx, bool *poll_tx, bool *poll_err) { pthread_mutex_lock(&bd->mutex); pthread_mutex_lock(&bd->ringlock); if (!read_empty(bd, chan)) *poll_rx = true; else *poll_rx = false; if (!write_full(bd, chan)) *poll_tx = true; else *poll_tx = false; /* * We don't report POLLERR on the console so that it doesn't get * automatically disconnected when it fails, and so that you can * connect to it in the error state before rebooting the target. * This is inconsistent, but being consistent turns out to be very * annoying. If someone tries to actually type on it, they'll * get an error. */ if (bd->tmfifo_error && chan != TMFIFO_CONS_CHAN) *poll_err = true; else *poll_err = false; pthread_mutex_unlock(&bd->ringlock); pthread_mutex_unlock(&bd->mutex); } static int rshim_fifo_release(rshim_backend_t *bd, int chan, void (*poll_handle_destroy)(rshim_backend_t *bd, int chan)) { pthread_mutex_lock(&bd->mutex); if (chan == TMFIFO_CONS_CHAN) { /* * If we aren't the last console file, nothing to do but * fix the reference count. */ bd->console_opens--; if (bd->console_opens) { pthread_mutex_unlock(&bd->mutex); return 0; } /* * We've told the host to stop using the TM FIFO console, * but there may be a lag before it does. Unless we * continue to read data from the console stream, the host * may spin forever waiting for the console to be drained * and not realize that it's time to stop using it. * Clearing the CONS_OPEN spin flag will discard any future * incoming console data, but if our input buffers are full * now, we might not be even reading from the hardware * FIFO. To avoid problems, clear the buffers and call the * drainer so that it knows there's space. */ pthread_mutex_lock(&bd->ringlock); bd->spin_flags &= ~RSH_SFLG_CONS_OPEN; read_reset(bd, TMFIFO_CONS_CHAN); write_reset(bd, TMFIFO_CONS_CHAN); rshim_fifo_input(bd); pthread_mutex_unlock(&bd->ringlock); } if (chan == TMFIFO_CONS_CHAN) bd->is_cons_open = 0; else bd->is_net_open = 0; if (!bd->is_net_open && !bd->is_cons_open) { if (bd->cancel) bd->cancel(bd, RSH_DEV_TYPE_TMFIFO, false); pthread_mutex_lock(&bd->ringlock); bd->spin_flags &= ~RSH_SFLG_READING; pthread_mutex_unlock(&bd->ringlock); } if (poll_handle_destroy) poll_handle_destroy(bd, chan); pthread_mutex_unlock(&bd->mutex); return 0; } /* Console operations */ int rshim_console_open(rshim_backend_t *bd) { pthread_mutex_lock(&bd->mutex); if (bd->is_cons_open) { pthread_mutex_unlock(&bd->mutex); return -EBUSY; } bd->is_cons_open = 1; pthread_mutex_lock(&bd->ringlock); bd->spin_flags |= RSH_SFLG_CONS_OPEN; pthread_mutex_unlock(&bd->ringlock); if (!bd->has_cons_work) { bd->has_cons_work = 1; rshim_work_signal(bd); } rshim_ref(bd); bd->console_opens++; pthread_mutex_unlock(&bd->mutex); return 0; } int rshim_console_release(rshim_backend_t *bd, void (*poll_handle_destroy)(rshim_backend_t *bd, int chan)) { int rc; rc = rshim_fifo_release(bd, TMFIFO_CONS_CHAN, poll_handle_destroy); rshim_deref(bd); return rc; } int rshim_notify(rshim_backend_t *bd, int event, int code) { int rc = 0; switch (event) { case RSH_EVENT_FIFO_INPUT: rshim_fifo_input(bd); break; case RSH_EVENT_FIFO_OUTPUT: rshim_fifo_output(bd); break; case RSH_EVENT_FIFO_ERR: rshim_fifo_err(bd, code); break; case RSH_EVENT_ATTACH: rshim_boot_done(bd); /* Sync-up the tmfifo if reprobe is not supported. */ if (!bd->has_reprobe && bd->has_rshim) rshim_fifo_sync(bd); __sync_synchronize(); bd->is_attach = 1; /* Init network interface. Moved to the work handler since it takes time. */ bd->net_init_time = rshim_timer_ticks; break; case RSH_EVENT_DETACH: /* Shutdown network interface. */ __sync_synchronize(); bd->is_attach = 0; rshim_net_del(bd); rshim_fifo_release(bd, TMFIFO_NET_CHAN, NULL); break; } return rc; } static int rshim_find_index(char *dev_name) { int i; /* Need to match static device name if configured. */ if (rshim_static_dev_name && strcmp(rshim_static_dev_name, dev_name)) return -1; /* Return static index if configured. */ if (rshim_static_index >= 0) return rshim_static_index; /* First look for a match with a previous device name. */ for (i = 0; i < RSHIM_MAX_DEV; i++) { if (rshim_dev_names[i] && !strcmp(dev_name, rshim_dev_names[i])) { RSHIM_DBG("found match with previous at index %d\n", i); return i; } } /* Then look for a never-used slot. */ for (i = 0; i < RSHIM_MAX_DEV; i++) { if (!rshim_dev_names[i]) return i; } /* Finally look for a currently-unused slot. */ for (i = 0; i < RSHIM_MAX_DEV; i++) { if (!rshim_devs[i]) { RSHIM_DBG("found unused slot %d\n", i); return i; } } return -1; } rshim_backend_t *rshim_find_by_name(char *dev_name) { int index = rshim_find_index(dev_name); /* If none of that worked, we fail. */ if (index < 0) { RSHIM_ERR("couldn't find slot for new device %s\n", dev_name); return NULL; } return rshim_devs[index]; } rshim_backend_t *rshim_find_by_dev(void *dev) { rshim_backend_t *bd; int index; for (index = 0; index < RSHIM_MAX_DEV; index++) { bd = rshim_devs[index]; if (bd && bd->dev == dev) return bd; } return NULL; } /* House-keeping timer. */ static void rshim_timer_func(rshim_backend_t *bd) { int period = rshim_keepalive_period; if (bd->has_cons_work) rshim_work_signal(bd); /* Request keepalive update and restart the ~300ms timer. */ if (rshim_timer_ticks - (bd->last_keepalive + period) > 0) { bd->keepalive = 1; bd->last_keepalive = rshim_timer_ticks; rshim_work_signal(bd); } bd->timer = rshim_timer_ticks + period; } static void rshim_timer_run(void) { rshim_backend_t *bd; int i; rshim_timer_ticks++; for (i = 0; i < RSHIM_MAX_DEV; i++) { bd = rshim_devs[i]; if (bd) { if (rshim_timer_ticks - bd->timer > 0) rshim_timer_func(bd); /* Push out remaining data if not sent out in the epoll loop. */ if (bd->net_fd >= 0) { rshim_net_tx(bd); rshim_net_rx(bd); } } } } /* * For some BF-1 SmartNIC cards with UART connected to the same RSim host, the * BOO_MODE comes up with 0 after power-cycle thus not able to boot from eMMC. * This function provides a workaround to detect such case and reset the card * with the correct boot mode. */ static void rshim_boot_workaround_check(rshim_backend_t *bd) { int rc; uint64_t value, uptime_sw, uptime_hw; /* This issue is only seen on BF-1 card. */ if (bd->ver_id != RSHIM_BLUEFIELD_1) return; /* Check boot mode 0, which supposes to be set externally. */ rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->boot_control, &value, RSHIM_REG_SIZE_8B); if (rc || value != RSH_BOOT_CONTROL__BOOT_MODE_VAL_NONE) return; /* * The logic below detects whether it's a hard reset. Register * RSH_UPTIME_POR has the value of cycles since hw reset, register * RSH_UPTIME has value of the most recent reset (sw or hard reset). * If the gap between these two values is less than 1G, we treat it * as hard reset. * * If boot mode is 0 after hard-reset, we update the boot mode and * initiate sw reset so the chip could boot up. */ rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->uptime_por, &uptime_hw, RSHIM_REG_SIZE_8B); if (rc) return; rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->uptime, &uptime_sw, RSHIM_REG_SIZE_8B); if (rc) return; if (uptime_sw - uptime_hw < 1000000000ULL) { rc = bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->boot_control, RSH_BOOT_CONTROL__BOOT_MODE_VAL_EMMC, RSHIM_REG_SIZE_8B); if (!rc) { /* SW reset. */ rc = rshim_reset_control(bd); if (!rc) usleep(100000); } } } static int rshim_bf2_a0_wa(rshim_backend_t *bd) { uint32_t devid = 0, clk_en, reset_en, powerdown; uint32_t main_clk_gate_en; int i, rc; rc = rshim_mmio_read32(bd, RSHIM_YU_BASE_ADDR + YU_BOOT_DEVID, &devid); RSHIM_DBG("yu_boot_devid: 0x%x\n", devid); if (rc) return -1; /* Workaround applies only to rev A0 in livefish mode */ if ((devid >> 16) > 0) return 0; RSHIM_INFO("Apply reset_mode_control WA\n"); if (rshim_mmio_read32(bd, RSHIM_YU_BASE_ADDR + YU_MAIN_CLK_GATE_EN, &main_clk_gate_en)) return -1; main_clk_gate_en |= (1 << 13); if (rshim_mmio_write32(bd, RSHIM_YU_BASE_ADDR + YU_MAIN_CLK_GATE_EN, main_clk_gate_en)) return -1; RSHIM_DBG("main_clk_gate_en: 0x%x\n", main_clk_gate_en); /* * yu.reset_mode_control.reset_modes.reset_mode[13].clk_en_bits[i].clk_en_bits = * yu.reset_mode_control.reset_modes.reset_mode[8].clk_en_bits[i].clk_en_bits */ for (i = 0; i < YU_CLK_EN_COUNT; ++i) { if (rshim_mmio_read32(bd, RSHIM_YU_BASE_ADDR + YU_RESET_8_CLK_EN + (i * 4), &clk_en)) return -1; if (rshim_mmio_write32(bd, RSHIM_YU_BASE_ADDR + YU_RESET_13_CLK_EN + (i * 4), clk_en)) return -1; } /* * yu.reset_mode_control.reset_modes.reset_mode[13].reset_en_bits[i].reset_en_bits = * yu.reset_mode_control.reset_modes.reset_mode[8].reset_en_bits[i].reset_en_bits */ for (i = 0; i < YU_RESET_EN_COUNT; ++i) { if (rshim_mmio_read32(bd, RSHIM_YU_BASE_ADDR + YU_RESET_8_RESET_EN + (i * 4), &reset_en)) return -1; if (rshim_mmio_write32(bd, RSHIM_YU_BASE_ADDR + YU_RESET_13_RESET_EN + (i * 4), reset_en)) return -1; } /* * yu.reset_mode_control.reset_modes.reset_mode[13].powerdown_bits[i].powerdown_bits = * yu.reset_mode_control.reset_modes.reset_mode[8].powerdown_bits[i].powerdown_bits */ for (i = 0; i < YU_POWERDOWN_COUNT; ++i) { if (rshim_mmio_read32(bd, RSHIM_YU_BASE_ADDR + YU_RESET_8_POWERDOWN + (i * 4), &powerdown)) return -1; if (rshim_mmio_write32(bd, RSHIM_YU_BASE_ADDR + YU_RESET_13_POWERDOWN + (i * 4), powerdown)) return -1; } /* * yu.bootrecord.nic.power.power_clock_up_delay = 0x9 * yu.bootrecord.nic.power.power_clock_down_delay = 0x9 */ if (rshim_mmio_write32(bd, RSHIM_YU_BASE_ADDR + YU_POWER_CLK_DELAY, 0x9 | (0x9 << 5))) return -1; return 0; } int rshim_access_check(rshim_backend_t *bd) { rshim_backend_t *other_bd; uint64_t value = 0; int i, rc; /* * Add a check and delay to make sure rshim is ready. * It's mainly used in BlueField-2+ where the rshim (like USB) access is * enabled in boot ROM which might happen after external host detects the * rshim device. */ for (i = 0; i < 10; i++) { rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->fabric_dim, &value, RSHIM_REG_SIZE_8B); if (!rc && value && !RSHIM_BAD_CTRL_REG(value)) break; usleep(100000); } if (RSHIM_BAD_CTRL_REG(value)) { RSHIM_ERR("Unable to read from rshim\n"); return -ETIMEDOUT; } rshim_boot_workaround_check(bd); /* BLUEFIELD_2 REV0 workaround. */ if (bd->ver_id == RSHIM_BLUEFIELD_2 && bd->rev_id == BLUEFIELD_REV0 && rshim_is_livefish(bd)) { if (rshim_bf2_a0_wa(bd) != 0) { /* * If the workaround fails it is likely because the mesh is already * stuck. Issue a soft reset and try one more time. Ignore the error code * since it's not reliable when doing reset. */ bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->reset_control, RSH_RESET_CONTROL__RESET_CHIP_VAL_KEY, RSHIM_REG_SIZE_8B); sleep(1); rshim_bf2_a0_wa(bd); } } /* Write value 0 to RSH_SCRATCHPAD1. */ rc = bd->write_rshim(bd, RSHIM_CHANNEL, bd->regs->scratchpad1, 0, RSHIM_REG_SIZE_8B); if (rc < 0) { RSHIM_ERR("failed to write rshim rc=%d\n", rc); return -ENODEV; } /* Write magic number to all the other backends. */ for (i = 0; i < RSHIM_MAX_DEV; i++) { other_bd = rshim_devs[i]; if (!other_bd || other_bd == bd) continue; pthread_mutex_lock(&other_bd->mutex); other_bd->write_rshim(other_bd, RSHIM_CHANNEL, bd->regs->scratchpad1, RSHIM_KEEPALIVE_MAGIC_NUM, RSHIM_REG_SIZE_8B); pthread_mutex_unlock(&other_bd->mutex); } /* * Poll RSH_SCRATCHPAD1 up to one second to check whether it's reset to * the keepalive magic value, which indicates another backend driver has * already attached to this target. */ value = 0; for (i = 0; i < 10; i++) { rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->scratchpad1, &value, RSHIM_REG_SIZE_8B); if (!rc && value == RSHIM_KEEPALIVE_MAGIC_NUM) { RSHIM_INFO("another backend already attached\n"); return -EEXIST; } usleep(100000); } /* One more read to make sure it's ready. */ rc = bd->read_rshim(bd, RSHIM_CHANNEL, bd->regs->scratchpad1, &value, RSHIM_REG_SIZE_8B); if (rc < 0) { RSHIM_ERR("access_check: failed to read rshim\n"); return -ENODEV; } return 0; } int rshim_register(rshim_backend_t *bd) { int i, rc, index; if (bd->registered) return 0; index = rshim_find_index(bd->dev_name); if (index < 0) return -ENODEV; if (!bd->read_rshim || !bd->write_rshim) { RSHIM_ERR("read_rshim/write_rshim missing\n"); return -EINVAL; } rc = rshim_access_check(bd); if (rc) return rc; if (!bd->write) bd->write = rshim_write_default; if (!bd->read) bd->read = rshim_read_default; pthread_mutex_init(&bd->ringlock, NULL); for (i = 0; i < TMFIFO_MAX_CHAN; i++) { pthread_cond_init(&bd->read_fifo[i].operable, NULL); pthread_cond_init(&bd->write_fifo[i].operable, NULL); } pthread_cond_init(&bd->fifo_write_complete_cond, NULL); pthread_cond_init(&bd->boot_complete_cond, NULL); pthread_cond_init(&bd->boot_write_complete_cond, NULL); pthread_cond_init(&bd->ctrl_wait_cond, NULL); memcpy(&bd->cons_termios, &init_console_termios, sizeof(init_console_termios)); bd->index = index; if (rshim_dev_names[index]) free(rshim_dev_names[index]); rshim_dev_names[index] = strdup(bd->dev_name); rshim_devs[index] = bd; for (i = 0; i < 2; i++) { bd->boot_buf[i] = malloc(BOOT_BUF_SIZE); if (!bd->boot_buf[i]) { if (i == 1) { free(bd->boot_buf[0]); bd->boot_buf[0] = NULL; } } } rshim_fifo_alloc(bd); if (!bd->read_buf) bd->read_buf = calloc(1, READ_BUF_SIZE); if (!bd->write_buf) bd->write_buf = calloc(1, WRITE_BUF_SIZE); bd->net_fd = -1; bd->net_notify_fd[0] = -1; bd->net_notify_fd[1] = -1; bd->registered = 1; bd->boot_timeout = rshim_boot_timeout; bd->display_level = rshim_display_level; /* Start the keepalive timer. */ bd->last_keepalive = rshim_timer_ticks; bd->timer = rshim_timer_ticks + 1; /* create character devices. */ #ifdef HAVE_RSHIM_FUSE rc = rshim_fuse_init(bd); if (rc) { rshim_deregister(bd); return rc; } #endif rshim_dev_bitmask |= (1ULL << index); return 0; } void rshim_deregister(rshim_backend_t *bd) { int i; if (!bd->registered) return; rshim_dev_bitmask &= ~(1ULL << bd->index); #ifdef HAVE_RSHIM_FUSE rshim_fuse_del(bd); #endif for (i = 0; i < 2; i++) { free(bd->boot_buf[i]); bd->boot_buf[i] = NULL; } free(bd->read_buf); bd->read_buf = NULL; free(bd->write_buf); bd->write_buf = NULL; rshim_fifo_free(bd); rshim_devs[bd->index] = NULL; bd->registered = 0; } void rshim_ref(rshim_backend_t *bd) { __sync_add_and_fetch(&bd->ref, 1); } void rshim_deref(rshim_backend_t *bd) { if (__sync_sub_and_fetch(&bd->ref, 1) == 0) { if (bd->destroy) bd->destroy(bd); } } bool rshim_allow_device(const char *devname) { int i; if (rshim_static_dev_name && strcmp(rshim_static_dev_name, devname)) return false; for (i = 0; i < RSHIM_MAX_DEV; i++) { if (rshim_blocked_dev_names[i] && !strcmp(rshim_blocked_dev_names[i], devname)) return false; } return true; } static void *rshim_stop_thread(void *arg) { /* Force to kill if not able to cleanup in time. */ sleep(3); kill(0, SIGKILL); return NULL; } static void rshim_stop(void) { rshim_backend_t *bd; pthread_t thread; int i, rc; rc = pthread_create(&thread, NULL, rshim_stop_thread, NULL); if (rc) { kill(0, SIGKILL); return; } rshim_lock(); for (i = 0; i < RSHIM_MAX_DEV; i++) { bd = rshim_devs[i]; if (!bd) continue; pthread_mutex_lock(&bd->mutex); if (bd->enable_device) bd->enable_device(bd, false); rshim_deregister(bd); pthread_mutex_unlock(&bd->mutex); } rshim_unlock(); } static void rshim_set_timer(int timer_fd, int interval) { struct itimerspec ts; ts.it_interval.tv_sec = 0; ts.it_interval.tv_nsec = (long)interval * 1000000; ts.it_value.tv_sec = 0; ts.it_value.tv_nsec = ts.it_interval.tv_nsec; rshim_timer_interval = interval; timerfd_settime(timer_fd, 0, &ts, NULL); } static void rshim_main(int argc, char *argv[]) { int i, fd, num, rc, epoll_fd, timer_fd; bool rshim_pcie_lf_init_done = false; uint8_t index; #ifdef __FreeBSD__ const int MAXEVENTS = 16; #else const int MAXEVENTS = 64; #endif struct epoll_event events[MAXEVENTS]; struct epoll_event event; rshim_backend_t *bd; time_t t0, t1; uint8_t tmp; memset(&event, 0, sizeof(event)); memset(events, 0, sizeof(events)); #ifdef HAVE_RSHIM_FUSE #ifdef __linux__ rc = system("modprobe cuse"); if (rc == -1) RSHIM_DBG("Failed the load cuse: %m\n"); #endif #ifdef __FreeBSD__ if (feature_present("cuse") == 0) if (system("kldload cuse") == -1) RSHIM_DBG("Failed the load cuse\n"); #endif #endif /* Create the epoll fd */ epoll_fd = epoll_create1(EPOLL_CLOEXEC); if (epoll_fd == -1) { RSHIM_ERR("epoll_create1 failed: %m\n"); exit(-1); } rshim_epoll_fd = epoll_fd; /* Create and add work fd. */ if (pipe(rshim_work_fd) == -1) { RSHIM_ERR("Failed to create pipe"); exit(-1); } if (fcntl(rshim_work_fd[0], F_SETFL, O_NONBLOCK) < 0) { RSHIM_ERR("failed to set nonblock pipe"); exit(-1); } event.data.fd = rshim_work_fd[0]; event.events = EPOLLIN; rc = epoll_ctl(epoll_fd, EPOLL_CTL_ADD, rshim_work_fd[0], &event); if (rc == -1) { RSHIM_ERR("epoll_ctl failed: %m\n"); exit(-1); } /* Add periodic timer. */ timer_fd = timerfd_create(CLOCK_MONOTONIC, 0); if (timer_fd == -1) { fprintf(stderr, "timerfd_create failed: %m\n"); exit(1); } rshim_set_timer(timer_fd, RSHIM_TIMER_INTERVAL); event.data.fd = timer_fd; event.events = EPOLLIN | EPOLLOUT; rc = epoll_ctl(epoll_fd, EPOLL_CTL_ADD, timer_fd, &event); if (rc == -1) { fprintf(stderr, "epoll_ctl failed: %m\n"); exit(1); } /* Scan rshim backends. */ rc = 0; if (!rshim_backend_name && rshim_static_dev_name) { if (!strncmp(rshim_static_dev_name, "usb", 3)) rshim_backend_name = "usb"; else if (!strncmp(rshim_static_dev_name, "pcie", 4)) rshim_backend_name = "pcie"; else if (!strncmp(rshim_static_dev_name, "pcie_lf", 7)) rshim_backend_name = "pcie_lf"; } if (!rshim_backend_name) { rshim_pcie_init(); rshim_usb_init(epoll_fd); } else { if (!strcmp(rshim_backend_name, "usb")) rc = rshim_usb_init(epoll_fd); else if (!strcmp(rshim_backend_name, "pcie")) rc = rshim_pcie_init(); else if (!strcmp(rshim_backend_name, "pcie_lf")) rc = rshim_pcie_lf_init(); } if (rc) { RSHIM_ERR("failed to initialize rshim backend\n"); exit(-1); } time(&t0); while (rshim_run) { num = epoll_wait(epoll_fd, events, MAXEVENTS, -1); if (num <= 0) { if (num < 0) RSHIM_DBG("epoll_wait failed; %m\n"); continue; } for (i = 0; i < num; i++) { fd = events[i].data.fd; if ((events[i].events & EPOLLERR) || (events[i].events & EPOLLHUP)) { RSHIM_DBG("epoll error\n"); epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, NULL); continue; } if (fd == timer_fd) { uint64_t res; rshim_fd_full_read(timer_fd, &res, sizeof(res)); rshim_timer_run(); } else if (fd == rshim_work_fd[0]) { rc = rshim_fd_full_read(rshim_work_fd[0], &index, sizeof(index)); if (rc == sizeof(index) && index >=0 && index < RSHIM_MAX_DEV) { bd = rshim_devs[index]; if (bd) rshim_work_handler(bd); } continue; } else { /* Network. */ for (index = 0; index < RSHIM_MAX_DEV; index++) { bd = rshim_devs[index]; if (!bd) continue; if (fd == bd->net_notify_fd[0]) { /* Rx. */ if (read(fd, &tmp, sizeof(tmp)) == sizeof(tmp)) rshim_net_rx(bd); break; } else if (fd == rshim_devs[index]->net_fd) { /* Tx. */ rshim_net_tx(bd); break; } } if (index != RSHIM_MAX_DEV) continue; } } /* Delayed initialization for livefish probe. */ if (!rshim_pcie_lf_init_done) { time(&t1); if (difftime(t1, t0) > 3) { if (!rshim_backend_name) rshim_pcie_lf_init(); rshim_pcie_lf_init_done = true; } } else { /* Disable the timer if no rshim devices are found. */ if (rshim_dev_bitmask) { if (!rshim_timer_interval) rshim_set_timer(timer_fd, RSHIM_TIMER_INTERVAL); } else if (rshim_timer_interval) { rshim_set_timer(timer_fd, 0); } /* Check USB for timeout or unhandled fd. */ rshim_usb_poll(rshim_dev_bitmask ? false : true); } } rshim_stop(); } int rshim_fifo_size(rshim_backend_t *bd, int chan, bool is_rx) { return is_rx ? read_cnt(bd, chan) : write_cnt(bd, chan); } int rshim_get_opn(rshim_backend_t *bd, char *opn, int len) { uint32_t value32 = 0; uint64_t value64 = 0; int i, rc; if (len) opn[0] = 0; switch (bd->ver_id) { case RSHIM_BLUEFIELD_2: for (i = 0; i < RSHIM_YU_BOOT_RECORD_OPN_SIZE && len >= 4; i += 4, len -= 4) { rc = rshim_mmio_read32(bd, RSHIM_YU_BASE_ADDR + RSHIM_YU_BOOT_RECORD_OPN + i, &value32); if (rc) return rc; value32 = le32toh(value32); opn[i] = (value32 >> 24) & 0xff; opn[i + 1] = (value32 >> 16) & 0xff; opn[i + 2] = (value32 >> 8) & 0xff; opn[i + 3] = value32 & 0xff; } break; case RSHIM_BLUEFIELD_3: for (i = 0; i < RSHIM_YU_BOOT_RECORD_OPN_SIZE && len >= 4; i += 4, len -= 4) { rc = bd->read_rshim(bd, YU_CHANNEL, RSHIM_YU_BF3_BOOT_RECORD_OPN + i, &value64, RSHIM_REG_SIZE_4B); if (rc) return rc; value32 = value64 & 0xFFFFFFFF; value32 = le32toh(value32); opn[i] = (value32 >> 24) & 0xff; opn[i + 1] = (value32 >> 16) & 0xff; opn[i + 2] = (value32 >> 8) & 0xff; opn[i + 3] = value32 & 0xff; } break; default: return -EOPNOTSUPP; } return 0; } int rshim_set_opn(rshim_backend_t *bd, const char *opn, int len) { uint32_t value32; uint64_t value64; int i, rc; switch (bd->ver_id) { case RSHIM_BLUEFIELD_2: for (i = 0; i < RSHIM_YU_BOOT_RECORD_OPN_SIZE && len >= 4; i += 4, len -= 4) { value32 = htole32((opn[i] << 24) | (opn[i + 1] << 16) | (opn[i + 2] << 8) | opn[i + 3]); rc = rshim_mmio_write32(bd, RSHIM_YU_BASE_ADDR + RSHIM_YU_BOOT_RECORD_OPN + i, value32); if (rc) return rc; } break; case RSHIM_BLUEFIELD_3: for (i = 0; i < RSHIM_YU_BOOT_RECORD_OPN_SIZE && len >= 4; i += 4, len -= 4) { value32 = htole32((opn[i] << 24) | (opn[i + 1] << 16) | (opn[i + 2] << 8) | opn[i + 3]); value64 = value32; rc = bd->write_rshim(bd, YU_CHANNEL, RSHIM_YU_BF3_BOOT_RECORD_OPN + i, value64, RSHIM_REG_SIZE_4B); if (rc) return rc; } break; default: return -EOPNOTSUPP; } return 0; } static int rshim_load_cfg(void) { char key[32] = "", value[64] = ""; char *buf = NULL; size_t n = 0; FILE *file; int index; file = fopen(rshim_cfg_file, "r"); if (!file) return -ENOENT; while (getline(&buf, &n, file) != -1) { if (sscanf(buf, "%31s%63s", key, value) != 2) continue; if (!strcmp(key, "DISPLAY_LEVEL")) { rshim_display_level = atoi(value); continue; } else if (!strcmp(key, "BOOT_TIMEOUT")) { rshim_boot_timeout = atoi(value); continue; } else if (!strcmp(key, "DROP_MODE")) { rshim_drop_mode = (atoi(value) > 0) ? 1 : 0; continue; } else if (!strcmp(key, "USB_RESET_DELAY")) { rshim_usb_reset_delay = atoi(value); rshim_has_usb_reset_delay = true; continue; } else if (!strcmp(key, "PCIE_RESET_DELAY")) { rshim_pcie_reset_delay = atoi(value); rshim_has_pcie_reset_delay = true; continue; } else if (!strcmp(key, "PCIE_INTR_POLL_INTERVAL")) { rshim_pcie_intr_poll_interval = atoi(value); continue; } else if (!strcmp(key, "PCIE_HAS_VFIO")) { rshim_pcie_enable_vfio = atoi(value); continue; } else if (!strcmp(key, "PCIE_HAS_UIO")) { rshim_pcie_enable_uio = atoi(value); continue; } if (strncmp(key, "rshim", 5) && strcmp(key, "none")) continue; if (strncmp(value, "usb-", 4) && strncmp(value, "pcie-", 5)) continue; /* Blocked devices. */ if (!strcmp(key, "none")) { for (index = 0; index < RSHIM_MAX_DEV; index++) { if (!rshim_blocked_dev_names[index]) { rshim_blocked_dev_names[index] = strdup(value); break; } } continue; } /* Static mapping of rshim device to index. */ index = atoi(key + 5); if (index < 0 || index >= RSHIM_MAX_DEV) continue; if (rshim_dev_names[index]) free(rshim_dev_names[index]); rshim_dev_names[index] = strdup(value); } if (buf) free(buf); fclose(file); return 0; } void rshim_sig_hup(int sig) { rshim_backend_t *bd; int index; int i; for (index = 0; index != RSHIM_MAX_DEV; index++) { bd = rshim_devs[index]; if (bd == NULL) continue; for (i = 0; i < TMFIFO_MAX_CHAN; i++) { pthread_cond_broadcast(&bd->read_fifo[i].operable); pthread_cond_broadcast(&bd->write_fifo[i].operable); } } } static void rshim_sig_handler(int sig) { switch (sig) { case SIGHUP: rshim_sig_hup(sig); break; case SIGTERM: rshim_run = false; rshim_work_signal(NULL); break; } } static void set_signals(void) { struct sigaction sa; memset(&sa, 0, sizeof(struct sigaction)); sa.sa_handler = rshim_sig_handler; sigaction(SIGHUP, &sa, NULL); sigaction(SIGINT, &sa, NULL); sigaction(SIGTERM, &sa, NULL); sigaction(SIGPIPE, &sa, NULL); } static void print_help(void) { printf("Usage: rshim [options]\n"); printf("\n"); printf("OPTIONS:\n"); printf(" -b, --backend backend name (usb, pcie or pcie_lf)\n"); printf(" -d, --device device to attach\n"); printf(" -f, --foreground run in foreground\n"); printf(" -i, --index use device path /dev/rshim/\n"); printf(" -l, --log-level log level"); printf("(0:none, 1:error, 2:warning, 3:notice, 4:debug)\n"); printf(" -n, --nonet no network interface\n"); printf(" -v, --version version\n"); } int main(int argc, char *argv[]) { static const char short_options[] = "b:d:fhi:l:nv"; static struct option long_options[] = { { "backend", required_argument, NULL, 'b' }, { "device", required_argument, NULL, 'd' }, { "foreground", no_argument, NULL, 'f' }, { "help", no_argument, NULL, 'h' }, { "index", required_argument, NULL, 'i' }, { "log-level", required_argument, NULL, 'l' }, { "nonet", no_argument, NULL, 'n' }, { "version", no_argument, NULL, 'v' }, { NULL, 0, NULL, 0 } }; int c; /* Parse arguments. */ while ((c = getopt_long(argc, argv, short_options, long_options, NULL)) != -1) { switch (c) { case 'b': rshim_backend_name = optarg; break; case 'd': rshim_static_dev_name = optarg; break; case 'f': rshim_daemon_mode = false; break; case 'i': rshim_static_index = atoi(optarg); if (rshim_static_index >= RSHIM_MAX_DEV) { fprintf(stderr, "Index exceeds max value %d\n", RSHIM_MAX_DEV - 1); return -EINVAL; } break; case 'l': rshim_log_level = atoi(optarg); if (rshim_log_level == 1) rshim_log_level = LOG_ERR; else if (rshim_log_level == 2) rshim_log_level = LOG_WARNING; else if (rshim_log_level == 3) rshim_log_level = LOG_NOTICE; else if (rshim_log_level >= 4) rshim_log_level = LOG_DEBUG; break; case 'n': rshim_no_net = true; break; case 'v': #if defined(PACKAGE_NAME) && defined(VERSION) printf(PACKAGE_NAME " " VERSION "\n"); #else printf("Rshim Driver for BlueField SoC 2.0\n"); #endif return 0; case 'h': default: print_help(); return 0; } } /* Put into daemon mode. */ if (rshim_daemon_mode) { int pid = fork(); if (pid < 0) { perror("fork failed: %m\n"); return -1; } else if (pid > 0) { return 0; } umask(0); if (setsid() < 0) { perror("setsid failed: %m\n"); return -1; } signal(SIGCHLD, SIG_IGN); if (chdir("/") == -1) perror("chdir failed: %m\n"); close(STDIN_FILENO); close(STDOUT_FILENO); close(STDERR_FILENO); } #ifdef HAVE_SYSLOG_H openlog("rshim", LOG_CONS, LOG_USER); #endif rshim_load_cfg(); set_signals(); rshim_main(argc, argv); closelog(); return 0; } ./Mellanox-rshim-user-space-0fd5d3f/freebsd/0000775000175000017500000000000014563673752020252 5ustar tai271828tai271828./Mellanox-rshim-user-space-0fd5d3f/freebsd/rshim.in0000775000175000017500000000035714563673752021734 0ustar tai271828tai271828#!/bin/sh # PROVIDE: rshim # REQUIRE: DAEMON # BEFORE: LOGIN # KEYWORD: nojail shutdown . /etc/rc.subr name=rshim desc="rshim driver for Mellanox BlueField SoC" command="@prefix@/sbin/${name}" load_rc_config $name run_rc_command "$1" ./Mellanox-rshim-user-space-0fd5d3f/rshim.spec.in0000664000175000017500000002421014563673752021242 0ustar tai271828tai271828# SPDX-License-Identifier: GPL-2.0-only # Copyright (C) 2019 Mellanox Technologies. All Rights Reserved. # Name: rshim Version: @VERSION@ Release: 0%{?dist} Summary: User-space driver for Mellanox BlueField SoC License: GPLv2 URL: https://github.com/mellanox/rshim-user-space Source0: https://github.com/mellanox/rshim-user-space/archive/%{name}-%{version}.tar.gz BuildRequires: gcc, autoconf, automake, pkgconfig, make BuildRequires: pkgconfig(libpci), pkgconfig(libusb-1.0) %if (0%{?rhel} >= 8 || 0%{?fedora} > 0) && "0%{?ctyunos}" == "0" Requires: kernel-modules-extra %endif %global with_systemd %(if (test -d "%{_unitdir}" > /dev/null); then echo -n '1'; else echo -n '0'; fi) %global debug_package %{nil} %description This is the user-space driver to access the BlueField SoC via the rshim interface. It provides ways to push boot stream, debug the target or login via the virtual console or network interface. %prep rm -fr %{name}-%{version} mkdir %{name}-%{version} tar -axf %{SOURCE0} -C %{name}-%{version} --strip-components 1 %setup -q -D -T %build ./bootstrap.sh %configure %if %{?make_build:1}%{!?make_build:0} %make_build %else make %endif %install %undefine _missing_build_ids_terminate_build %makeinstall -C src INSTALL_DIR="%{buildroot}%{_sbindir}" %if "%{with_systemd}" == "1" %{__install} -d %{buildroot}%{_unitdir} %{__install} -m 0644 rshim.service %{buildroot}%{_unitdir} %endif %{__install} -d %{buildroot}%{_mandir}/man8 %{__install} -m 0644 man/rshim.8 %{buildroot}%{_mandir}/man8 %{__install} -m 0644 man/bfb-install.8 %{buildroot}%{_mandir}/man8 %{__install} -d %{buildroot}%{_sysconfdir} %{__install} -m 0644 etc/rshim.conf %{buildroot}%{_sysconfdir} %{__install} -m 0755 scripts/bfb-install %{buildroot}%{_sbindir} %pre %if "%{with_systemd}" == "1" if systemctl is-active --quiet rshim ; then systemctl stop rshim fi %endif %post %if "%{with_systemd}" == "1" systemctl daemon-reload systemctl enable rshim systemctl start rshim %endif %preun if [ "$1" = "0" ]; then %if "%{with_systemd}" == "1" if systemctl is-active --quiet rshim ; then systemctl stop rshim fi %else killall -9 rshim %endif fi %files %{!?_licensedir:%global license %%doc} %license LICENSE %defattr(-,root,root,-) %doc README.md %config(noreplace) %{_sysconfdir}/rshim.conf %if "%{with_systemd}" == "1" %{_unitdir}/rshim.service %endif %{_sbindir}/rshim %{_sbindir}/bfb-install %{_mandir}/man8/rshim.8.gz %{_mandir}/man8/bfb-install.8.gz %changelog * Fri Feb 16 2024 Liming Sun - 2.0.20 - rshim_pci: adjust delay time for nic_fw reset - bfb-install: Exit on "Linux up" * Wed Jan 10 2024 Liming Sun - 2.0.19 - Fix incorrect console message drop - Allow runtime debug code for DK cards * Thu Dec 14 2023 Liming Sun - 2.0.18 - Clear scratchpad1 register when setting drop_mode * Wed Nov 22 2023 Liming Sun - 2.0.17 - bfb-install: Fix duplicate output * Thu Nov 16 2023 Liming Sun - 2.0.16 - Remove fuse build dependency * Tue Nov 14 2023 Liming Sun - 2.0.15 - Add BFB completion condition for enhanced NIC mode * Fri Nov 10 2023 Liming Sun - 2.0.14 - Fix 9f19cfb4a75687ae * Wed Nov 08 2023 Liming Sun - 2.0.13 - Several robust fixes - Add fuse3 support * Mon Oct 23 2023 Liming Sun - 2.0.12 - BF3: Add UPTIME display in seconds * Tue Sep 26 2023 Liming Sun - 2.0.11 - Remove version 0 support for NIC FW_RESET - bfb-install: Return failure code * Mon Sep 18 2023 Liming Sun - 2.0.10 - Fix interrupt handling for NIC FW_RESET * Sat Jun 17 2023 Liming Sun - 2.0.9 - rshim/usb/bf3: fix timeout logic * Tue May 16 2023 Liming Sun - 2.0.8 - Fix the fall-back logic of direct-mapping * Thu Mar 30 2023 Liming Sun - 2.0.7 - Avoid opening /dev/uio multiple times - Update common files to dual-license - Adjust rshim reset delay * Sun Nov 20 2022 Liming Sun - 2.0.6-19 - BF3: Support 4B access for PCIe * Tue Oct 25 2022 Liming Sun - 2.0.6-18 - pcie: fix initialization issue when setting DROP_MODE in rshim.conf * Thu Oct 20 2022 Liming Sun - 2.0.6-17 - pcie: Avoid using cached pci_dev - rshim_fuse: display misc file even when rshim is not accessible * Thu Oct 06 2022 Liming Sun - 2.0.6-16 - pcie: Support mixed vfio and direct mapping mode * Thu Sep 29 2022 Liming Sun - 2.0.6-15 - Add dependency of libfuse2 for .deb - rshim-pcie: add a new bad-access code - Fix a potential NULL pointer access during USB disconnect - Adjust default boot timeout to 150s * Tue Aug 16 2022 Liming Sun - 2.0.6-14 - Avoid potential race when stopping the rshim process - Add configuration option to enable/disable PCIe VFIO/UIO - Fix warnings for compiling on 32-bit BMC - Mustang rshim usb supports for 4B and 8B transactions * Sun Jul 17 2022 Liming Sun - 2.0.6-13 - BF3: Support 32-bit CR-space access via USB - Avoid kernel-modules-extra dependency on ctyunos * Thu Jun 16 2022 Liming Sun - 2.0.6-12 - Optimize the rshim_work_fd - Detect new USB/rshim hot plugin * Mon May 16 2022 Liming Sun - 2.0.6-11 - Avoid kernel crash when unbind rshim from uio * Mon May 02 2022 Liming Sun - 2.0.6-10 - Fix several compiling issues for FreeBSD * Thu Apr 28 2022 Liming Sun - 2.0.6-9 - Use per-device memory-map mode * Mon Apr 18 2022 Liming Sun - 2.0.6-8 - Add interrupt polling for direct mmap() mode - Fix several coverity warnings * Thu Apr 07 2022 Liming Sun - 2.0.6-7 - Keep intr_fd during rshim_pcie disable/enable - Mustang: Add support for rshim over pcie and pcie_lf * Wed Mar 30 2022 Liming Sun - 2.0.6-6 - Clear scratchpad1 to 0 before PCI resources are unmapped - Fallback to UIO if VFIO failed * Fri Mar 18 2022 Liming Sun - 2.0.6-5 - PCIe: Add UIO and IRQ support - PCIe: Remove 32-bit support * Mon Feb 28 2022 Liming Sun - 2.0.6-4 - VFIO support - Fix potential race in rshim_work_signal * Mon Nov 29 2021 Liming Sun - 2.0.6-3 - Adjust the defaul value of usb_reset_delay to 5 - Add a delay after USB probe - Make the reset delay configurable * Wed Nov 03 2021 Liming Sun - 2.0.6-2 - bfb-install: Handle new indications for installation completion - Clean up some un-needed register definition - Fix MTU of the tmfifo_net0 interface on FreeBSD - Several fixes to prevent hypervisor crash - Refine some BF-2 Rev0 workaround condition * Wed May 12 2021 Liming Sun - 2.0.6-1 - Disable the background timer if no rshim devices - Setting default path for rshim config file * Wed Mar 10 2021 Liming Sun - 2.0.5-10 - PCIe hotplug support - Reduce CPU utilization when there is no rshim device * Wed Jan 27 2021 Liming Sun - 2.0.5-9 - Fix potential tmfifo data loss - Add workaround checking for Bluefield-2 REV-0 - Fix network traffic stop issue when Tx buffer full * Fri Dec 11 2020 Liming Sun - 2.0.5-8 - Don't allow any register access when DROP_MODE is set - Avoid potential race in rshim_fifo_read * Wed Dec 09 2020 Liming Sun - 2.0.5-7 - Fix potential dead-lock when calling rshim_access_check - Ignore rshim access checking when global drop mode is enabled - Fix some secure boot related issue * Wed Dec 02 2020 Liming Sun - 2.0.5-6 - Add some default configuration in rshim.conf - Change the debug level of Rshim byte access widget timeout - Add bfb-install script * Thu Oct 29 2020 Liming Sun - 2.0.5-5 - Check rshim accessibility when re-enabling it - Enable console output during boot stream pushing - Add some delay for the pcie_lf probe - Auto-start rshim service after installation * Fri Sep 25 2020 Liming Sun - 2.0.5-4 - Some robust fixes for USB rshim - Fix a typo in pcie mmap * Mon Aug 17 2020 Liming Sun - 2.0.5-3 - Fix several coverity warnings - Add workaround to boot Viper rev A0 in LiveFish mode - Display/configure OPN string for BlueField-2 * Fri Jul 24 2020 Liming Sun - 2.0.5-2 - Add configuration file support - misc: Display device version / revision ID - Add service file for FreeBSD * Tue Jun 16 2020 Liming Sun - 2.0.5-1 - Improve response time to ctrl+c for boot stream - Fix a rpmbuild issue when make_build is not defined - Add DROP_MODE configuration in misc file - Avoid reading the fifo if still booting - Fix configure issue for FreeBSD 12.1-RELEASE - Add domain id to the DEV_NAME in the misc file - Fix the debian copyright format - Enhance rshim_pcie_enable function * Tue Apr 21 2020 Liming Sun - 2.0.4-1 - Update .spec file according to review comments - Fix the 'KillMode' in rshim.service - Support process termination by SIGTERM - Fix some compiling warnings and configure issue for FreeBSD - Fix a read()/write() issue in rshim_pcie.c caused by optimization * Tue Apr 14 2020 Liming Sun - 2.0.3-1 - Enable pci device during probing - Map the pci resource0 file instead of /dev/mem - Add copyright header in bootstrap.sh - Add 'Requires' tag check in the rpm .spec for kernel-modules-extra - Fix the 'rshim --version' output * Thu Apr 09 2020 Liming Sun - 2.0.2-1 - Remove unnecessary dependency in .spec and use make_build - Add package build for debian/ubuntu - Fix some format in the man page - Add check for syslog headers * Mon Mar 23 2020 Liming Sun - 2.0.1-1 - Rename bfrshim to rshim - Remove rshim.spec since it's auto-generated from rshim.spec.in - Fix some warnings reported by coverity - Add file rhel/rshim.spec.in for fedora - Move rshim to sbin and move man page to man8 * Fri Mar 13 2020 Liming Sun - 2.0-1 - Update the spec file according to fedora packaging-guidelines * Mon Dec 16 2019 Liming Sun - Initial packaging