pax_global_header00006660000000000000000000000064151632626260014523gustar00rootroot0000000000000052 comment=995685ac3ad8909cb7562e74dd13e3b450833d9c sanlock/000077500000000000000000000000001516326262600125215ustar00rootroot00000000000000sanlock/.gitattributes000066400000000000000000000001251516326262600154120ustar00rootroot00000000000000# Set the default behavior, in case people don't have core.autocrlf set. * text=auto sanlock/.gitignore000066400000000000000000000006051516326262600145120ustar00rootroot00000000000000*.co *.swp *.tar.gz .tox/ .vimdir cscope.* python/*.so python/build/ python/usr/ reset/sanlk-reset reset/sanlk-resetd sanlock.spec src/libsanlock.pc src/libsanlock.so* src/libsanlock_client.pc src/libsanlock_client.so* src/sanlock tests/*.pyc tests/__pycache__/ tests/devcount tests/killpath tests/sanlk_client tests/sanlk_load tests/sanlk_path wdmd/libwdmd.so* wdmd/wdmd wdmd/wdmd_client sanlock/.vimrc000066400000000000000000000001251516326262600136400ustar00rootroot00000000000000" Local vim configuration for sanlock set noexpandtab set shiftwidth=8 set tabstop=8 sanlock/Makefile000066400000000000000000000003361516326262600141630ustar00rootroot00000000000000version := $(shell cat VERSION) SUBDIRS = wdmd src .PHONY: all $(SUBDIRS) clean install all: $(SUBDIRS) $(SUBDIRS): $(MAKE) -C $@ src: wdmd clean install: for dir in $(SUBDIRS); do \ $(MAKE) -C $$dir $@; \ done sanlock/README.license000066400000000000000000000006021516326262600150200ustar00rootroot00000000000000LGPLv2+ src/libsanlock_client.so src/sanlock.h src/sanlock_rv.h src/sanlock_admin.h src/sanlock_resource.h src/sanlock_sock.h src/sanlock_sock.c src/client.c wdmd/libwdmd.so wdmd/wdmd.h wdmd/wdmd_sock.h wdmd/wdmd_sock.c wdmd/client.c GPLv2 src/list.h (copied from linux kernel) GPLv2+ src/crc32c.c (copied from btrfs-progs which copied from linux kernel) all other original files sanlock/README.md000066400000000000000000000600061516326262600140020ustar00rootroot00000000000000sanlock - shared storage lock manager ===================================== sanlock is a lock manager for shared storage environments. It allows applications running on multiple hosts to coordinate access to shared resources, such as data objects on shared storage systems like a SAN, preventing data corruption and ensuring data integrity. sanlock uses the shared storage itself to hold lock state. The lock state is typically located on the same storage as the application data that the locks are protecting. A distributed application wishing to use sanlock for coordination will reserve space on the shared storage to hold the lock state, and pass the location for lock storage to sanlock. Resources --------- A sanlock resource is an object which can be locked. It has a name and a location on disk. The resource exists as a 1MB area on shared storage, which holds the lock state for that resource. An application initializes a resource, giving it a name that corresponds to an application-specific entity. The application can acquire and release a lock on the resource to coordinate application activity. Within sanlock, the process of acquiring or releasing the lock involves reading and writing to the resource area on the shared storage. When sanlock acquires a lock on a resource, the local host ID is written as the lock owner in the first sector of the resource area. Lockspaces ---------- A sanlock lockspace is a collection of resources, and a group of hosts that can acquire locks on them. The lockspace has a unique name which is referenced by each resource in the lockspace. The lockspace exists as a 1MB area on shared storage, typically preceding the resource areas (also 1MB each.) The lockspace area consists of 2000 special "host_id leases" (one sector each). This supports up to 2000 hosts using the lockspace. The lease for host_id N exists in sector N-1 of the lockspace disk area. A lockspace also refers to a group of hosts with common access to the lockspace's shared storage, and using the lockspace's resources. Each host using the lockspace has a unique host name and a unique integer host ID. A host must "join" the lockspace before it can lock resources. The process of joining the lockspace involves acquiring a special "host_id lease" within the lockspace disk area. Acquiring the host_id lease involves writing the host name to the host_id lease. A host_id lease is renewed (once every 20 seconds by default). The host_id lease renewal applies to all resource locks held in the lockspace, i.e. renewing the host_id lease renews all resource locks. During renewal, a host also reads the state of other host_id leases in the lockspace. Monitoring lease renewals enables a host to determine if other hosts are alive. Hosts ----- A host is a machine with a unique name and host ID (an integer in the range 1-2000). The host ID is assigned by the application or user to each host. The host name can be configured by the user in sanlock.conf, otherwise it is taken from the product_uuid of the system. If a name is not assigned, and product_uuid is not available, sanlock generates a new random uuid for the host name each time sanlock is started. To use sanlock, a host must run the sanlock(8) and wdmd(8) daemons, and have a connection to the shared storage holding lockspaces and resources. Applications use libsanlock to connect to the sanlock daemon, and make API calls to initialize lockspaces and resources, join and leave lockspaces, and acquire and release locks on resources. When a host has joined a lockspace, and the sanlock daemon has begun renewing the lockspace host_id lease, the reported host state is "LIVE" for the given lockspace. The host's state in other lockspaces depends on whether the application on the host has joined them, and whether sanlock maintains access to the lockspace storage for renewals. If a host stops renewing its host_id lease, its resource locks in that lockspace will expire, and sanlock will eventually allow those locks to be acquired by other hosts. When a host fails to renew its host_id lease for a period of time (id_renewal_fail_seconds, default 80 seconds), it is initially placed into the "FAIL" state. In this state, sanlock attempts to stop the application from using resources in the lockspace, so that the resource locks are no longer needed. If the application shutdown is successful, resource locks and the host_id lease are dropped, and sanlock stops attempting to renew them. If not successful, the failed host will be fenced and then enter the "DEAD" state. When dead, a host's unreleased locks will be granted by sanlock on other hosts. sanlock includes built-in watchdog-based fencing, and also supports applications deploying other (often faster) fencing methods, e.g. removing persistent reservations. Lock algorithms --------------- Resource locks are implemented using two alternative methods, depending on the capabilities of the storage device: 1. The SCSI COMPARE AND WRITE (CAW) command. This special-purpose command, supported by certain storage devices, atomically updates a disk sector. sanlock uses this to atomically update the lock owner field in a resource. If multiple hosts attempt to become the lock owner simultaneously, the CAW command will succeed on only one of the hosts. 2. The Disk Paxos algorithm. All hosts which want to become the lock owner cooperatively determine, using a sequence of reads and writes, which host will become the new lock owner. The resource lock owner is updated with a new value in a way that all hosts recognize the same result. Lockspace host_id leases are also implemented using two alternative methods: 1. The SCSI COMPARE AND WRITE (CAW) command. 2. A time-delay algorithm. In theory, host_id leases in a lockspace should not be contended, because the host_id should be configured uniquely for all hosts. But, the algorithm for host_id lease ownership attempts to detect conflict in case of misconfiguration. The time delay method is particularly inefficient at this, which can cause long delays joining a lockspace. The CAW method has no delay. Within sanlock, all resource locks are referred to as "paxos leases" (even when CAW is used), and all lockspace host_id leases are referred to as "delta leases" (regardless of which algorithm is used.) When the shared storage supports the COMPARE AND WRITE (CAW) command (also known as atomic test and set), it is the most quick and efficient option, and is recommended. When CAW is not supported, the Disk Paxos algorithm is used. The use of CAW is determined when a resource or lockspace is initialized on disk by testing if the storage location for the lease supports CAW. The amount of I/O required to acquire/release a resource lock depends on: - the algorithm (CAW or Disk Paxos) - the lock mode (exclusive or shared) - the lock state (if the lock is already owned) - lock contention (are multiple hosts attempting to acquire the lock at the same time) The I/O required for CAW and Disk Paxos (DP), exclusive and shared locks, assuming the lock is unowned and uncontended: ``` acquire ex CAW resource lock: read (1MB), ioctl (512b) release ex CAW resource lock: ioctl (512b) acquire sh CAW resource lock: read (1MB), ioctl (512b); write (512b); ioctl (512b) release sh CAW resource lock: write (512b) (ioctl: ioctl submits an SG_IO COMPARE AND WRITE command to a single sector. Two sectors of data, previous and next, are sent to the device.) acquire ex DP resource lock: read (1MB), write (512b), read (1MB), write (512b), read (1MB), write (512b) release ex DP resource lock: read (512b), write (512b) acquire sh DP resource lock read (1MB), write (512b), read (1MB), write (512b), read (1MB), write (512b); write (512b); read (512b), write (512b) release sh DP resource lock: write (512b) ``` Disk sectors and lease areas ---------------------------- sanlock commonly uses atomic reads and writes of disk sectors. Storage with either 512 or 4096 byte sectors is supported. For 512 byte sectors, resource and lockspace areas are always 1MB (supporting 2000 hosts). With 4096 byte sectors, resource and lockspace areas default to 8MB (also supporting 2000 hosts.) Discussion elsewhere referring to 1MB disk areas should be translated to 8MB in this case. The disk area size of a lockspace or resource is called "align_size" in sanlock interfaces and internally. For disks with 4K sectors, applications can choose to initialize resource and lockspace areas as 1MB, 2MB, 4MB or 8MB. The reduced sizes of 1/2/4 MB limit the permitted host_id range (and the max number of hosts) to 250/500/1000. Accepted combinations of sector size and align size, and the corresponding max_hosts (and max host_id) are: ``` sector_size 512, align_size 1M, max_hosts 2000 sector_size 4096, align_size 1M, max_hosts 250 sector_size 4096, align_size 2M, max_hosts 500 sector_size 4096, align_size 4M, max_hosts 1000 sector_size 4096, align_size 8M, max_hosts 2000 ``` Using sanlock on shared block devices that do host based mirroring or replication is unlikely to work correctly. sanlock can also be used on shared files, e.g. over NFS, but this is not the primary use case. When using sanlock on shared files, all sanlock io should go to one file server. Disk area example ----------------- ``` /dev/sdx --------------------------------------- Lockspace | Resource | Resource | ... | --------------------------------------- 0 1MB 2MB 3MB Lockspace ------------------------------------------------------------ | delta lease | delta lease | ... | delta lease | unused | | LS name FOO | LS name FOO | ... | LS name FOO | | | host_id 1 | host_id 2 | ... | host_id 2000 | | | generation 1 | generation 1 | ... | generation 0 | | | hostname A | hostname B | ... | hostname - | | | timestamp 44 | timestamp 25 | ... | timestamp 0 | | ------------------------------------------------------------- 0 512 ... 1023488 1MB Resource ------------------------------------------------------------- | paxos lease | | LS name FOO | | RS name FILE0 | | owner: host_id 1 generation 1 | ------------------------------------------------------------- 1MB 2MB Resource ------------------------------------------------------------- | paxos lease | | LS name FOO | | RS name FILE1 | | owner: host_id 2 generation 1 | ------------------------------------------------------------- 2MB 3MB ``` - Lockspace "FOO" at the start of the shared disk is followed by two resources "FILE0" and "FILE1". - Two hosts: "A" using host_id 1 and "B" using host_id 2 have joined the lockspace. - In joining the lockspace, A and B have written their hostname to the host_id lease, and have begun updating the timestamp field in each renewal (current times on A/B were 44/25 at their last renewals.) - A has locked FILE0, and B has locked FILE1. The following sanlock commands will produce the state shown in the diagram. These steps illustrate, using command line interfaces to libsanlock APIs, what an application would typically perform using the libsanlock APIs. ``` Shared disk /dev/sdx is attached to hosts A and B. host A initializes disk areas for one lockspace and two resources: $ sanlock init -s FOO:0:/dev/sdx:0 $ sanlock init -r FOO:FILE0:/dev/sdx:1M $ sanlock init -r FOO:FILE1:/dev/sdx:2M host A joins lockspace FOO using host_id 1: $ sanlock add_lockspace -s FOO:1:/dev/sdx:0 host B joins lockspace FOO using host_id 2: $ sanlock add_lockspace -s FOO:2:/dev/sdx:0 host A acquires lock on FILE0 (lock associated with the sleep process): $ sanlock command -r FOO:FILE0:/dev/sdx:1M -c /bin/sleep 10 host B acquires lock on FILE1 (lock associated with the sleep process): $ sanlock command -r FOO:FILE1:/dev/sdx:2M -c /bin/sleep 10 host A fails to acquire lock on FILE1 (owned by host B): $ sanlock command -r FOO:FILE1:/dev/sdx:1M -c /bin/sleep 10 host B fails to acquire lock on FILE0 (owned by host A): $ sanlock command -r FOO:FILE0:/dev/sdx:1M -c /bin/sleep 10 When the sleep processes exit, their associated locks (FILE0 and FILE1) are automatically released. (The sleep process is a trivial replacement for what would typically be a long running application process holding locks.) host A and B leave the lockspace: $ sanlock rem_lockspace -s FOO:1:/dev/sdx:0 $ sanlock rem_lockspace -s FOO:2:/dev/sdx:0 ``` Failures -------- _application process failure_ If a process using sanlock exits, sanlock automatically drops the resource locks that the process was holding. This response assumes that if the process is not running, it can no longer write to the shared storage, so the locks it held are no longer required. However, if an application may be writing to shared storage from a process other than the one connected to sanlock, then PERSISTENT resource locks should be used, which sanlock will not automatically drop if the connection to the application process is closed. _sanlock daemon failure_ If the sanlock daemon process exits abnormally, e.g. it crashes or is killed, it will no longer be renewing host_id leases in lockspaces that had been joined. To other hosts this will appear to be, and will be handled as, a host failure. The host with the failed sanlock daemon will be reset by its watchdog (since watchdog renewals are tied to lockspace lease renewals), and/or other hosts will apply a form of external fencing, e.g. removing persistent reservations from the host with the crashed/killed sanlock daemon. _wdmd daemon failure_ sanlock uses the wdmd process to manage and update the local /dev/watchdog device. If the wdmd process exits abnormally, e.g. it crashes or is killed, it will no longer be updating the watchdog, which will lead to the host being reset by its watchdog. Other hosts wait until the watchdog has fired, or until external fencing has occurred, before considering the host dead. _host system freeze_ If a host system freezes, sanlock and wdmd will no longer be updating the watchdog, causing the system to be reset by the watchdog. Other hosts wait until the watchdog expiration time, or until external fencing has occurred, before considering the host dead. _host system failure_ If a host system fails, e.g. loss of power, other hosts cannot distinguish the failure from a host system freeze, and will wait for the watchdog expiration time, or for external fencing to occur, before considering the host dead. _renewal i/o failure_ Renewing host_id leases in a lockspace (periodically writing a new timestamp to the lease) is how other hosts determine liveness. If renewal I/O fails to the host_id lease on the shared storage, the host will no longer appear to be alive. Other hosts cannot distinguish this condition from a host system failure or freeze. Once sanlock has failed to renew a host_id lease for specific period of time (id_renewal_fail_seconds, default 80 seconds), it begins responding: - Graceful shutdown. sanlock will execute a "graceful shutdown" program if the application has specified one to be run. The shutdown program tells the application to shut down because its leases are expiring. The application must respond by stopping its activities and releasing its leases (or exit). - SIGTERM. If an application does not specify a graceful shutdown program, sanlock sends SIGTERM to the process instead. The process must release its leases or exit in a prescribed amount of time, or sanlock proceeds to the next method of stopping. (An application may specify that SIGTERM is not appropriate for shutdown and should not be tried.) - SIGKILL. sanlock sends SIGKILL to the process using the expiring leases. The process has a fixed amount of time to exit after receiving SIGKILL (note: a process could be stuck waiting for I/O in the kernel and not killable.) If it does not exit in time, sanlock will proceed to the next method. As with SIGTERM, this method may be disabled by applications for which SIGKILL is not appropriate, e.g. those that modify shared storage from processes other than the process connected to sanlock. - Watchdog reset. Eventually, if an application has not voluntarily released all of its locks, sanlock intentionally allows the local watchdog to do a forced reset of the host. sanlock carefully manages the timing of the watchdog device so that it fires shortly before any other host could take over the resource locks held by local processes. - External fencing. Other hosts may still use an external fencing method against a failed host. External fencing may finish faster than the watchdog reset, allowing faster takeover of locks/resources from a failed host. The watchdog reset can be optionally disabled with external fencing. Watchdog -------- sanlock uses the wdmd(8) daemon to access /dev/watchdog. wdmd multiplexes multiple timeouts onto the single watchdog timer. This is required because host_id leases for each lockspace are renewed and expire independently. sanlock maintains a wdmd connection for each lockspace host_id lease being renewed. Each connection has an expiry time for some seconds in the future. After each successful host_id lease renewal, the expiry time is renewed for the associated wdmd connection. If wdmd finds any connection expired, it will not renew the /dev/watchdog timer. Given enough successive failed renewals, the watchdog device will fire and reset the host. (Given the multiplexing nature of wdmd, shorter overlapping renewal failures from multiple lockspaces could cause spurious watchdog firing.) The direct link between host_id lease renewals and watchdog renewals provides a predictable watchdog firing time based on host_id lease renewal timestamps that are visible from other hosts. sanlock knows the time the watchdog on another host has fired based on the host_id lease time. Furthermore, if the watchdog device on another host fails to fire when it should, the continuation of host_id lease renewals from the other host will make this evident and prevent leases from being taken from the failed host. If sanlock is able to stop/kill all processing using an expiring lockspace, the associated wdmd connection for that lockspace is removed. The expired wdmd connection will no longer block /dev/watchdog renewals, and the host should avoid being reset. External fencing ---------------- External fencing allows an application to control when failed hosts are declared dead. When sanlock indicates that a host has entered the failed state (after missing renewals for id_renewal_fail_seconds), the application can detect this, and respond by performing a custom form of fencing that it manages itself. The custom fencing method would protect the shared storage that the application is using (sanlock itself is not aware of all the shared storage that the application may be writing to, it is only aware of the shared storage used by the lockspace and resources.) One example of custom fencing is the application removing persistent reservations of the failed host. After the application has successfully fenced the failed host, it notifies sanlock that the host is dead. sanlock records this in the host_id lease of the dead host (the DEAD_EXT flag), which in turn allows resource locks owned by the dead host to be granted by other hosts. If the application wants to use external fencing only, and does not want the watchdog expiration to place a host into the DEAD state, it can pass the NO_TIMEOUT flag when initializing the lockspace. With this flag encoded in the lockspace host_id leases, the host_id leases will enter the FAIL state based on missing renewals, but the host will only enter the DEAD state based on application's "external dead" API call. Timing ------ The sanlock timing parameters control host_id lease renewal frequency, failure detection time, and total recovery time. These parameters work together to balance responsiveness to failures (allowing other hosts to take over resources from failed hosts) against tolerance for temporary storage slowdowns (avoid resetting hosts that are temporarily slow.) *io_timeout* This is the maximum time allowed for a single I/O operation to complete. This includes I/O used to renew host_id leases, and I/O used to acquire resource locks. If I/O times out while acquiring a resource lock, an error is returned to the application's lock request. If I/O times out while renewing a host_id lease, the renewal I/O continues to be retried until successful or until the lockspace is removed. The default io_timeout is 10 seconds. This value can be configured by the application when initializing a lockspace, or can be configured in sanlock.conf. The io_timeout should be set based on the expected I/O performance of the shared storage, including potential transient delays. *id_renewal_seconds* This is the interval between successful host_id lease renewals. It represents the maximum time that a single host_id lease renewal can take (when each I/O operation takes the maximum io_timeout). id_renewal_seconds = 2 * io_timeout With the default io_timeout of 10 seconds, id_renewal_seconds is 20 seconds. This is the typical interval seen between renewal timestamps written to disk when I/O completes quickly. Each host_id lease renewal consists of one read and one write. If a renewal completes in less than id_renewal_seconds, sanlock will delay before beginning the next renewal attempt, so that renewals occur at approximately id_renewal_seconds intervals. If a renewal takes the full id_renewal_seconds to complete, the next renewal attempt will begin immediately. *id_renewal_fail_seconds* This is the time from the last successful host_id lease renewal until sanlock enters recovery mode and begins attempting to stop local processes using the expiring lockspace. id_renewal_fail_seconds = 8 * io_timeout With the default io_timeout of 10 seconds, id_renewal_fail_seconds is 80 seconds. When sanlock has been unable to renew a host_id lease for id_renewal_fail_seconds, it begins recovery procedures: running graceful shutdown programs, sending SIGTERM/SIGKILL, and ultimately allowing the watchdog to reset the host if the application using the lockspace does not shut down in time. *watchdog_fire_timeout* This is the time the /dev/watchdog device will wait without being renewed before firing and resetting the host. The watchdog_fire_timeout is a constant, typically 60 seconds. All hosts using sanlock must use the same watchdog_fire_timeout value. Mismatched values will break the protection provided by the watchdog. Some watchdog devices support configurable timeouts. When using a configurable watchdog, common alternative values are 30 seconds and 10 seconds (typically paired with io_timeout values of 5 and 2 seconds respectively.) *host_dead_seconds* (Note: host_dead_seconds does not apply to host_id leases with NO_TIMEOUT set. Hosts using NO_TIMEOUT rely on an external host dead call from the application before they are considered dead. An external host dead indicator may often be faster than waiting for host_dead_seconds.) This is the time from the last successful host_id lease renewal until other hosts can be certain that the host has been reset by its watchdog and is dead (no longer writing to storage). host_dead_seconds = id_renewal_fail_seconds + watchdog_fire_timeout With the default values of io_timeout 10 and watchdog_fire_timeout 60, host_dead_seconds is 140 seconds (80 + 60). When a host attempts to acquire a resource lock that is held by another host, it checks the host_id lease of the owner host. If the host_id lease has not been renewed for host_dead_seconds, the resource lock is considered expired and can be acquired. Before this time, the resource lock remains owned by the other host, even if that host has stopped renewing its host_id lease. Mailing list ------------ https://lists.fedorahosted.org/admin/lists/sanlock-devel.lists.fedorahosted.org/ sanlock/VERSION000066400000000000000000000000061516326262600135650ustar00rootroot000000000000005.1.0 sanlock/common.mk000066400000000000000000000012531516326262600143430ustar00rootroot00000000000000export CC ?= cc check = $(shell echo "int main() { return 0; }" \ | $(CC) $(1) -xc - >&/dev/null && echo $(1) ||:) export CFLAGS += -D_GNU_SOURCE -O2 -ggdb \ -Wall \ -Wformat \ -Wformat-security \ -Wmissing-prototypes \ -Wnested-externs \ -Wpointer-arith \ -Wextra \ -Wshadow \ -Wcast-align \ -Wwrite-strings \ -Waggregate-return \ -Wstrict-prototypes \ -Winline \ -Wredundant-decls \ -Wno-sign-compare \ -Wno-unused-parameter \ -Wp,-D_FORTIFY_SOURCE=2 \ -Wno-strict-overflow \ -fexceptions \ -fasynchronous-unwind-tables \ -fdiagnostics-show-option \ -Wp,-D_GLIBCXX_ASSERTIONS \ -fstack-protector-strong \ $(check -fstack-clash-protection) \ -Wl,-z,now sanlock/init.d/000077500000000000000000000000001516326262600137065ustar00rootroot00000000000000sanlock/init.d/sanlock.service000066400000000000000000000003661516326262600167270ustar00rootroot00000000000000[Unit] Description=Shared Storage Lease Manager Documentation=man:sanlock(8) After=syslog.socket Wants=wdmd.service [Service] Type=forking ExecStart=/usr/sbin/sanlock daemon SendSIGKILL=no LimitNOFILE=2048 [Install] WantedBy=multi-user.target sanlock/init.d/sanlock.sysconfig000066400000000000000000000006671516326262600172770ustar00rootroot00000000000000# SANLOCKOPTS -- set the command line options for the sanlock daemon # See sanlock man page for full list of command line options. # # Include "-U sanlock -G sanlock" in the option string unless # also changing the SANLOCKUSER above. # # To disable use of watchdog via wdmd #SANLOCKOPTS="-U sanlock -G sanlock -w 0" # # To disable use of watchdog via wdmd and disable high priority features #SANLOCKOPTS="-U sanlock -G sanlock -w 0 -h 0" sanlock/init.d/systemd-wdmd000066400000000000000000000012421516326262600162510ustar00rootroot00000000000000#!/bin/bash # # wdmd - watchdog multiplexing daemon # helper to load the softdog module if a watchdog is not enabled # exec="/usr/sbin/wdmd" watchdog_probe() { $exec --probe > /dev/null 2>&1 retval=$? return $retval } watchdog_check() { watchdog_probe retval=$? if [ $retval -ne 0 ]; then echo -n $"Loading the softdog kernel module: " modprobe softdog # handle delay in udev setting permission/label udevadm settle watchdog_probe retval=$? if [ $retval -ne 0 ]; then echo "failed" return 1 fi echo "success" return 0 fi } case "$1" in watchdog-check) watchdog_check ;; *) echo $"Usage $0 watchdog-check" exit 2 esac exit $? sanlock/init.d/wdmd.service000066400000000000000000000003731516326262600162260ustar00rootroot00000000000000[Unit] Description=Watchdog Multiplexing Daemon Documentation=man:wdmd(8) After=syslog.socket [Service] Type=forking ExecStartPre=/lib/systemd/systemd-wdmd watchdog-check ExecStart=/usr/sbin/wdmd SendSIGKILL=no [Install] WantedBy=multi-user.target sanlock/init.d/wdmd.sysconfig000066400000000000000000000004721516326262600165720ustar00rootroot00000000000000# WDMDOPTS -- set the command line options for the wdmd daemon # See wdmd man page for full list of command line options. # # Include "-G sanlock" in the option string. # # To enable use of test scripts #WDMDOPTS="-G sanlock -S 1" # # To select a specific watchdog device #WDMDOPTS="-G sanlock -w /dev/watchdog1" sanlock/src/000077500000000000000000000000001516326262600133105ustar00rootroot00000000000000sanlock/src/Makefile000066400000000000000000000100751516326262600147530ustar00rootroot00000000000000# Copyright 2010-2011 Red Hat, Inc. # # This copyrighted material is made available to anyone wishing to use, # modify, copy, or redistribute it subject to the terms and conditions # of the GNU General Public License v2 or (at your option) any later version. include ../common.mk CMD_TARGET = sanlock HEADER_TARGET = sanlock.h sanlock_rv.h sanlock_resource.h sanlock_admin.h sanlock_direct.h MAN_TARGET = sanlock.8 SOMAJOR=1 SOMINOR=0 LIB_ENTIRE_TARGET = libsanlock LIB_CLIENT_TARGET = libsanlock_client LIBPC_ENTIRE_TARGET = libsanlock.pc LIBPC_CLIENT_TARGET = libsanlock_client.pc LIBSO_ENTIRE_TARGET = $(LIB_ENTIRE_TARGET).so.$(SOMAJOR).$(SOMINOR) LIBSO_CLIENT_TARGET = $(LIB_CLIENT_TARGET).so.$(SOMAJOR).$(SOMINOR) CMD_SOURCE = \ crc32c.c \ delta_lease.c \ direct.c \ diskio.c \ ondisk.c \ sizeflags.c \ helper.c \ lockspace.c \ lockfile.c \ log.c \ main.c \ paxos_lease.c \ task.c \ timeouts.c \ resource.c \ watchdog.c \ monotime.c \ cmd.c \ client_cmd.c \ sanlock_sock.c \ env.c LIB_ENTIRE_SOURCE = \ client.c \ sanlock_sock.c \ crc32c.c \ diskio.c \ ondisk.c \ sizeflags.c \ delta_lease.c \ paxos_lease.c \ direct.c \ task.c \ timeouts.c \ direct_lib.c \ monotime.c \ env.c \ sanlock_internal.c LIB_CLIENT_SOURCE = \ client.c \ sanlock_sock.c \ env.c LIBPC_ENTIRE_SOURCE = libsanlock.pc.in LIBPC_CLIENT_SOURCE = libsanlock_client.pc.in VER=$(shell cat ../VERSION) CFLAGS += -DVERSION=\"$(VER)\" CMD_CFLAGS = $(CFLAGS) -fPIE -DPIE LIB_ENTIRE_CFLAGS = $(CFLAGS) -fPIC LIB_CLIENT_CFLAGS = $(CFLAGS) -fPIC CMD_LDFLAGS = $(LDFLAGS) -Wl,-z,relro -pie LIB_ENTIRE_LDFLAGS = $(LDFLAGS) -Wl,-z,relro -pie LIB_CLIENT_LDFLAGS = $(LDFLAGS) -Wl,-z,relro -pie CMD_LDADD = -lpthread -luuid -lrt -laio -lblkid -ldevmapper -lsanlock -L../wdmd -lwdmd LIB_ENTIRE_LDADD = -lpthread -lrt -laio -lblkid -ldevmapper -L../wdmd -lwdmd all: $(LIBSO_ENTIRE_TARGET) $(LIBSO_CLIENT_TARGET) $(CMD_TARGET) $(LIBPC_ENTIRE_TARGET) $(LIBPC_CLIENT_TARGET) $(LIBSO_ENTIRE_TARGET): $(LIB_ENTIRE_SOURCE) $(CC) $(LIB_ENTIRE_CFLAGS) $(LIB_ENTIRE_LDFLAGS) -shared -o $@ -Wl,-soname=$(LIB_ENTIRE_TARGET).so.$(SOMAJOR) $^ $(LIB_ENTIRE_LDADD) ln -sf $(LIBSO_ENTIRE_TARGET) $(LIB_ENTIRE_TARGET).so ln -sf $(LIBSO_ENTIRE_TARGET) $(LIB_ENTIRE_TARGET).so.$(SOMAJOR) $(LIBSO_CLIENT_TARGET): $(LIB_CLIENT_SOURCE) $(CC) $(LIB_CLIENT_CFLAGS) $(LIB_CLIENT_LDFLAGS) -shared -o $@ -Wl,-soname=$(LIB_CLIENT_TARGET).so.$(SOMAJOR) $^ ln -sf $(LIBSO_CLIENT_TARGET) $(LIB_CLIENT_TARGET).so ln -sf $(LIBSO_CLIENT_TARGET) $(LIB_CLIENT_TARGET).so.$(SOMAJOR) $(CMD_TARGET): $(LIBSO_ENTIRE_TARGET) $(CMD_SOURCE) $(CC) $(CMD_CFLAGS) $(CMD_LDFLAGS) $(CMD_SOURCE) $(CMD_LDADD) -o $@ -L. $(LIBPC_ENTIRE_TARGET): $(LIBPC_ENTIRE_SOURCE) sed -e "s/@VERSION@/$(VER)/" $(LIBPC_ENTIRE_SOURCE) > $(LIBPC_ENTIRE_TARGET) $(LIBPC_CLIENT_TARGET): $(LIBPC_CLIENT_SOURCE) sed -e "s/@VERSION@/$(VER)/" $(LIBPC_CLIENT_SOURCE) > $(LIBPC_CLIENT_TARGET) clean: rm -f *.o *.so *.so.* $(CMD_TARGET) $(LIBSO_ENTIRE_TARGET) $(LIBSO_CLIENT_TARGET) $(LIBPC_ENTIRE_TARGET) $(LIBPC_CLIENT_TARGET) INSTALL=install DESTDIR= BINDIR=/usr/sbin LIBDIR=/usr/lib64 HEADIR=/usr/include MANDIR=/usr/share/man VARDIR=/var/lib/sanlock .PHONY: install install: all $(INSTALL) -d $(DESTDIR)/$(BINDIR) $(INSTALL) -d $(DESTDIR)/$(LIBDIR) $(INSTALL) -d $(DESTDIR)/$(HEADIR) $(INSTALL) -d $(DESTDIR)/$(VARDIR) $(INSTALL) -d $(DESTDIR)/$(MANDIR)/man8 $(INSTALL) -d $(DESTDIR)/$(LIBDIR)/pkgconfig $(INSTALL) -c -m 755 $(CMD_TARGET) $(DESTDIR)/$(BINDIR) $(INSTALL) -c -m 755 $(LIBSO_ENTIRE_TARGET) $(DESTDIR)/$(LIBDIR) $(INSTALL) -c -m 755 $(LIBSO_CLIENT_TARGET) $(DESTDIR)/$(LIBDIR) $(INSTALL) -c -m 644 $(LIBPC_ENTIRE_TARGET) $(DESTDIR)/$(LIBDIR)/pkgconfig $(INSTALL) -c -m 644 $(LIBPC_CLIENT_TARGET) $(DESTDIR)/$(LIBDIR)/pkgconfig cp -a $(LIB_ENTIRE_TARGET).so $(DESTDIR)/$(LIBDIR) cp -a $(LIB_CLIENT_TARGET).so $(DESTDIR)/$(LIBDIR) cp -a $(LIB_ENTIRE_TARGET).so.$(SOMAJOR) $(DESTDIR)/$(LIBDIR) cp -a $(LIB_CLIENT_TARGET).so.$(SOMAJOR) $(DESTDIR)/$(LIBDIR) $(INSTALL) -c -m 644 $(HEADER_TARGET) $(DESTDIR)/$(HEADIR) $(INSTALL) -m 644 $(MAN_TARGET) $(DESTDIR)/$(MANDIR)/man8/ sanlock/src/client.c000066400000000000000000001466011516326262600147420ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock.h" #include "sanlock_internal.h" #include "sanlock_resource.h" #include "sanlock_admin.h" #include "sanlock_sock.h" #include "sanlock_rv.h" #include "env.h" #ifndef GNUC_UNUSED #define GNUC_UNUSED __attribute__((__unused__)) #endif static int connect_socket(int *sock_fd) { int rv, s; struct sockaddr_un addr; static const char *run_dir; *sock_fd = -1; s = socket(AF_LOCAL, SOCK_STREAM, 0); if (s < 0) return -errno; if (run_dir == NULL) run_dir = env_get("SANLOCK_RUN_DIR", DEFAULT_RUN_DIR); rv = sanlock_socket_address(run_dir, &addr); if (rv < 0) { close(s); return rv; } rv = connect(s, (struct sockaddr *) &addr, sizeof(struct sockaddr_un)); if (rv < 0) { rv = -errno; close(s); return rv; } *sock_fd = s; return 0; } static int send_header(int sock, int cmd, uint32_t cmd_flags, int datalen, uint32_t data, uint32_t data2) { struct sm_header header; size_t rem = sizeof(header); size_t off = 0; ssize_t rv; memset(&header, 0, sizeof(header)); header.magic = SM_MAGIC; header.version = SM_PROTO; header.cmd = cmd; header.cmd_flags = cmd_flags; header.length = sizeof(header) + datalen; header.data = data; header.data2 = data2; retry: rv = send(sock, (char *)&header + off, rem, 0); if (rv == -1 && errno == EINTR) goto retry; if (rv < 0) return -errno; if (rv < rem) { rem -= rv; off += rv; goto retry; } return 0; } static ssize_t send_data(int sockfd, const void *buf, size_t len, int flags) { size_t rem = len; size_t off = 0; ssize_t rv; retry: rv = send(sockfd, (char *)buf + off, rem, flags); if (rv == -1 && errno == EINTR) goto retry; if (rv < 0) return -errno; if (rv < rem) { rem -= rv; off += rv; goto retry; } return 0; } static ssize_t recv_data(int sockfd, void *buf, size_t len, int flags) { ssize_t rv; retry: rv = recv(sockfd, buf, len, flags); if (rv == -1 && errno == EINTR) goto retry; return rv; } int send_command(int cmd, uint32_t data); int send_command(int cmd, uint32_t data) { int rv, sock; rv = connect_socket(&sock); if (rv < 0) return rv; rv = send_header(sock, cmd, 0, 0, data, 0); if (rv < 0) { close(sock); return rv; } return sock; } static int recv_result(int fd) { struct sm_header h; ssize_t rv; memset(&h, 0, sizeof(h)); retry: rv = recv(fd, &h, sizeof(h), MSG_WAITALL); if (rv == -1 && errno == EINTR) goto retry; if (rv < 0) return -errno; if (rv != sizeof(h)) return -1; return (int)h.data; } static int recv_header(int fd, struct sm_header *h) { ssize_t rv; retry: rv = recv(fd, h, sizeof(struct sm_header), MSG_WAITALL); if (rv == -1 && errno == EINTR) goto retry; if (rv < 0) return -errno; if (rv != sizeof(struct sm_header)) return -1; return 0; } static int validate_res_disk_paths(struct sanlk_resource *res) { int i; if (!res) return -EINVAL; for (i = 0; i < res->num_disks; i++) { if (!memchr(res->disks[i].path, '\0', SANLK_PATH_LEN)) return -EINVAL; } return 0; } static int cmd_lockspace(int cmd, struct sanlk_lockspace *ls, uint32_t flags, uint32_t data) { int rv, fd; if (!ls || !ls->host_id_disk.path[0] || !memchr(ls->host_id_disk.path, '\0', SANLK_PATH_LEN)) return -EINVAL; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, cmd, flags, sizeof(struct sanlk_lockspace), data, 0); if (rv < 0) goto out; rv = send_data(fd, (void *)ls, sizeof(struct sanlk_lockspace), 0); if (rv < 0) { rv = -errno; goto out; } rv = recv_result(fd); out: close(fd); return rv; } int sanlock_add_lockspace(struct sanlk_lockspace *ls, uint32_t flags) { return cmd_lockspace(SM_CMD_ADD_LOCKSPACE, ls, flags, 0); } int sanlock_add_lockspace_timeout(struct sanlk_lockspace *ls, uint32_t flags, uint32_t io_timeout) { return cmd_lockspace(SM_CMD_ADD_LOCKSPACE, ls, flags, io_timeout); } int sanlock_inq_lockspace(struct sanlk_lockspace *ls, uint32_t flags) { return cmd_lockspace(SM_CMD_INQ_LOCKSPACE, ls, flags, 0); } int sanlock_rem_lockspace(struct sanlk_lockspace *ls, uint32_t flags) { return cmd_lockspace(SM_CMD_REM_LOCKSPACE, ls, flags, 0); } int sanlock_get_lockspaces(struct sanlk_lockspace **lss, int *lss_count, uint32_t flags) { struct sanlk_lockspace *lsbuf, *ls; struct sm_header h; int rv, fd, i, ret, recv_count; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_GET_LOCKSPACES, flags, 0, 0, 0); if (rv < 0) goto out; /* receive result and ls structs */ memset(&h, 0, sizeof(h)); rv = recv_data(fd, &h, sizeof(h), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(h)) { rv = -1; goto out; } /* -ENOSPC means that the daemon's send buffer ran out of space */ rv = (int)h.data; if (rv < 0 && rv != -ENOSPC) goto out; *lss_count = h.data2; recv_count = h.data2; if (!lss) goto out; lsbuf = malloc(recv_count * sizeof(struct sanlk_lockspace)); if (!lsbuf) goto out; ls = lsbuf; for (i = 0; i < recv_count; i++) { ret = recv_data(fd, ls, sizeof(struct sanlk_lockspace), MSG_WAITALL); if (ret < 0) { rv = -errno; free(lsbuf); goto out; } if (ret != sizeof(struct sanlk_lockspace)) { rv = -1; free(lsbuf); goto out; } ls++; } *lss = lsbuf; out: close(fd); return rv; } int sanlock_set_host(struct sanlk_lockspace *ls, uint32_t cmd, uint32_t flags, uint32_t io_timeout, void *host_data) { int rv, fd; if (!ls || !ls->name[0]) return -EINVAL; /* * SANLK_SET_HOST_DEAD_EXT: host_data is sanlk_host */ if (cmd == SANLK_SET_HOST_DEAD_EXT) { if (!host_data) return -EINVAL; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_SET_HOST, flags, sizeof(struct sanlk_lockspace) + sizeof(struct sanlk_host), cmd, io_timeout); if (rv < 0) goto out; rv = send_data(fd, ls, sizeof(struct sanlk_lockspace), 0); if (rv < 0) { rv = -errno; goto out; } rv = send_data(fd, host_data, sizeof(struct sanlk_host), 0); if (rv < 0) { rv = -errno; goto out; } rv = recv_result(fd); out: close(fd); return rv; } return -EINVAL; } int sanlock_get_hosts(const char *ls_name, uint64_t host_id, struct sanlk_host **hss, int *hss_count, uint32_t flags) { struct sm_header h; struct sanlk_lockspace ls; struct sanlk_host *hsbuf, *hs; int rv, fd, i, ret, recv_count; if (!ls_name) return -EINVAL; memset(&ls, 0, sizeof(struct sanlk_lockspace)); strncpy(ls.name, ls_name, SANLK_NAME_LEN); ls.host_id = host_id; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_GET_HOSTS, flags, sizeof(struct sanlk_lockspace), 0, 0); if (rv < 0) goto out; rv = send_data(fd, &ls, sizeof(struct sanlk_lockspace), 0); if (rv < 0) { rv = -errno; goto out; } /* receive result and ls structs */ memset(&h, 0, sizeof(h)); rv = recv_data(fd, &h, sizeof(h), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(h)) { rv = -1; goto out; } /* -ENOSPC means that the daemon's send buffer ran out of space */ rv = (int)h.data; if (rv < 0 && rv != -ENOSPC) goto out; *hss_count = h.data2; recv_count = h.data2; if (!hss) goto out; hsbuf = malloc(recv_count * sizeof(struct sanlk_host)); if (!hsbuf) goto out; hs = hsbuf; for (i = 0; i < recv_count; i++) { ret = recv_data(fd, hs, sizeof(struct sanlk_host), MSG_WAITALL); if (ret < 0) { rv = -errno; free(hsbuf); goto out; } if (ret != sizeof(struct sanlk_host)) { rv = -1; free(hsbuf); goto out; } hs++; } *hss = hsbuf; out: close(fd); return rv; } int sanlock_set_config(const char *ls_name, uint32_t flags, uint32_t cmd, void *config_data) { struct sanlk_lockspace ls; struct sm_header h; uint32_t h_data2 = 0; int rv, fd; if (!ls_name) return -EINVAL; /* * For SANLK_CONFIG_IO_TIMEOUT: * data arg points to 4 byte io_timeout value */ if (cmd == SANLK_CONFIG_IO_TIMEOUT) { if (!config_data) return -EINVAL; memcpy(&h_data2, config_data, 4); /* leader.io_timeout is uint16_t */ if (!h_data2 || (h_data2 > 0x000000FF)) return -EINVAL; } memset(&ls, 0, sizeof(struct sanlk_lockspace)); strncpy(ls.name, ls_name, SANLK_NAME_LEN); rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_SET_CONFIG, flags, sizeof(struct sanlk_lockspace), cmd, h_data2); if (rv < 0) goto out; rv = send_data(fd, &ls, sizeof(ls), 0); if (rv < 0) { rv = -errno; goto out; } memset(&h, 0, sizeof(h)); rv = recv_data(fd, &h, sizeof(h), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(h)) { rv = -1; goto out; } rv = (int)h.data; out: close(fd); return rv; } int sanlock_align(struct sanlk_disk *disk) { int rv, fd; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_ALIGN, 0, sizeof(struct sanlk_disk), 0, 0); if (rv < 0) goto out; rv = send_data(fd, (void *)disk, sizeof(struct sanlk_disk), 0); if (rv < 0) { rv = -errno; goto out; } rv = recv_result(fd); out: close(fd); return rv; } int sanlock_read_lockspace_host(struct sanlk_lockspace *ls, uint32_t flags, uint32_t *io_timeout, struct sanlk_host *hs) { struct sm_header h; int rv, fd; if (!ls || !ls->host_id_disk.path[0] || !memchr(ls->host_id_disk.path, '\0', SANLK_PATH_LEN)) return -EINVAL; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_READ_LOCKSPACE_HOST, flags, sizeof(struct sanlk_lockspace), 0, 0); if (rv < 0) goto out; rv = send_data(fd, ls, sizeof(struct sanlk_lockspace), 0); if (rv < 0) { rv = -errno; goto out; } memset(&h, 0, sizeof(h)); rv = recv_data(fd, &h, sizeof(h), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(h)) { rv = -1; goto out; } rv = (int)h.data; if (rv < 0) goto out; rv = recv_data(fd, ls, sizeof(struct sanlk_lockspace), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(struct sanlk_lockspace)) { rv = -1; goto out; } rv = recv_data(fd, hs, sizeof(struct sanlk_host), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(struct sanlk_host)) { rv = -1; goto out; } *io_timeout = h.data2; rv = (int)h.data; out: close(fd); return rv; } int sanlock_read_lockspace(struct sanlk_lockspace *ls, uint32_t flags, uint32_t *io_timeout) { struct sm_header h; int rv, fd; if (!ls || !ls->host_id_disk.path[0] || !memchr(ls->host_id_disk.path, '\0', SANLK_PATH_LEN)) return -EINVAL; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_READ_LOCKSPACE, flags, sizeof(struct sanlk_lockspace), 0, 0); if (rv < 0) goto out; rv = send_data(fd, ls, sizeof(struct sanlk_lockspace), 0); if (rv < 0) { rv = -errno; goto out; } /* receive result, io_timeout and ls struct */ memset(&h, 0, sizeof(h)); rv = recv_data(fd, &h, sizeof(h), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(h)) { rv = -1; goto out; } rv = (int)h.data; if (rv < 0) goto out; rv = recv_data(fd, ls, sizeof(struct sanlk_lockspace), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(struct sanlk_lockspace)) { rv = -1; goto out; } *io_timeout = h.data2; rv = (int)h.data; out: close(fd); return rv; } int sanlock_read_resource(struct sanlk_resource *res, uint32_t flags) { struct sm_header h; int rv, fd; if (!res || !res->num_disks || res->num_disks > SANLK_MAX_DISKS || !res->disks[0].path[0]) return -EINVAL; rv = validate_res_disk_paths(res); if (rv < 0) return rv; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_READ_RESOURCE, flags, sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk) * res->num_disks, 0, 0); if (rv < 0) goto out; rv = send_data(fd, res, sizeof(struct sanlk_resource), 0); if (rv < 0) { rv = -errno; goto out; } rv = send_data(fd, res->disks, sizeof(struct sanlk_disk) * res->num_disks, 0); if (rv < 0) { rv = -errno; goto out; } /* receive result and res struct */ memset(&h, 0, sizeof(h)); rv = recv_data(fd, &h, sizeof(h), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(h)) { rv = -1; goto out; } rv = recv_data(fd, res, sizeof(struct sanlk_resource), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(struct sanlk_resource)) { rv = -1; goto out; } rv = (int)h.data; out: close(fd); return rv; } int sanlock_init_lockspace_host(struct sanlk_lockspace *ls, char *our_host_name, uint64_t generation, uint64_t timestamp, uint32_t flags, uint32_t io_timeout) { struct leader_record leader = { 0 }; char name[NAME_ID_SIZE + 1] = { 0 }; int rv, fd; if (!ls || !ls->name[0] || !ls->host_id_disk.path[0] || !ls->host_id || !memchr(ls->host_id_disk.path, '\0', SANLK_PATH_LEN)) return -EINVAL; leader.owner_generation = generation; leader.timestamp = timestamp; leader.owner_id = ls->host_id; if (our_host_name) { strncpy(name, our_host_name, NAME_ID_SIZE); memcpy(leader.resource_name, name, NAME_ID_SIZE); } rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_INIT_LOCKSPACE_HOST, flags, sizeof(struct sanlk_lockspace) + sizeof(struct leader_record), 0, io_timeout); if (rv < 0) goto out; rv = send_data(fd, ls, sizeof(struct sanlk_lockspace), 0); if (rv < 0) { rv = -errno; goto out; } rv = send_data(fd, &leader, sizeof(struct leader_record), 0); if (rv < 0) { rv = -errno; goto out; } rv = recv_result(fd); out: close(fd); return rv; } int sanlock_write_lockspace(struct sanlk_lockspace *ls, int max_hosts, uint32_t flags, uint32_t io_timeout) { int rv, fd; if (!ls || !ls->host_id_disk.path[0] || !memchr(ls->host_id_disk.path, '\0', SANLK_PATH_LEN)) return -EINVAL; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_WRITE_LOCKSPACE, flags, sizeof(struct sanlk_lockspace), max_hosts, io_timeout); if (rv < 0) goto out; rv = send_data(fd, ls, sizeof(struct sanlk_lockspace), 0); if (rv < 0) { rv = -errno; goto out; } rv = recv_result(fd); out: close(fd); return rv; } int sanlock_write_resource(struct sanlk_resource *res, int max_hosts, int num_hosts, uint32_t flags) { int rv, fd; if (!res || !res->num_disks || res->num_disks > SANLK_MAX_DISKS || !res->disks[0].path[0]) return -EINVAL; rv = validate_res_disk_paths(res); if (rv < 0) return rv; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_WRITE_RESOURCE, flags, sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk) * res->num_disks, max_hosts, num_hosts); if (rv < 0) goto out; rv = send_data(fd, res, sizeof(struct sanlk_resource), 0); if (rv < 0) { rv = -errno; goto out; } rv = send_data(fd, res->disks, sizeof(struct sanlk_disk) * res->num_disks, 0); if (rv < 0) { rv = -errno; goto out; } rv = recv_result(fd); out: close(fd); return rv; } int sanlock_read_resource_owners(struct sanlk_resource *res, uint32_t flags, struct sanlk_host **hss, int *hss_count) { struct sm_header h; struct sanlk_host *hsbuf, *hs; int rv, fd, i, ret, recv_count; if (!res || !res->num_disks || res->num_disks > SANLK_MAX_DISKS || !res->disks[0].path[0]) return -EINVAL; rv = validate_res_disk_paths(res); if (rv < 0) return rv; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_READ_RESOURCE_OWNERS, flags, sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk) * res->num_disks, 0, 0); if (rv < 0) goto out; rv = send_data(fd, res, sizeof(struct sanlk_resource), 0); if (rv < 0) { rv = -errno; goto out; } rv = send_data(fd, res->disks, sizeof(struct sanlk_disk) * res->num_disks, 0); if (rv < 0) { rv = -errno; goto out; } /* receive result, res struct, and host structs */ memset(&h, 0, sizeof(h)); rv = recv_data(fd, &h, sizeof(h), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(h)) { rv = -1; goto out; } rv = (int)h.data; if (rv < 0) goto out; rv = recv_data(fd, res, sizeof(struct sanlk_resource), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(struct sanlk_resource)) { rv = -1; goto out; } rv = 0; *hss_count = h.data2; recv_count = h.data2; if (!hss) goto out; hsbuf = malloc(recv_count * sizeof(struct sanlk_host)); if (!hsbuf) goto out; hs = hsbuf; for (i = 0; i < recv_count; i++) { ret = recv_data(fd, hs, sizeof(struct sanlk_host), MSG_WAITALL); if (ret < 0) { rv = -errno; free(hsbuf); goto out; } if (ret != sizeof(struct sanlk_host)) { rv = -1; free(hsbuf); goto out; } hs++; } *hss = hsbuf; out: close(fd); return rv; } int sanlock_test_resource_owners(struct sanlk_resource *res GNUC_UNUSED, uint32_t flags GNUC_UNUSED, struct sanlk_host *owners, int owners_count, struct sanlk_host *hosts, int hosts_count, uint32_t *test_flags) { struct sanlk_host *owner, *host; int i, j, found, fail = 0; *test_flags = 0; owner = owners; for (i = 0; i < owners_count; i++) { found = 0; host = hosts; for (j = 0; j < hosts_count; j++) { if (owner->host_id != host->host_id) { host++; continue; } found = 1; break; } if (!found) goto next; if (host->generation > owner->generation) goto next; /* this should not be possible, and should never happen */ if (host->generation < owner->generation) return -EINVAL; switch (host->flags & SANLK_HOST_MASK) { case SANLK_HOST_FREE: case SANLK_HOST_DEAD: break; case SANLK_HOST_LIVE: case SANLK_HOST_FAIL: case SANLK_HOST_UNKNOWN: fail = 1; break; default: return -EINVAL; } next: owner++; } if (fail) *test_flags |= SANLK_TRF_FAIL; return 0; } int sanlock_reg_event(const char *ls_name, struct sanlk_host_event *he, uint32_t flags) { struct sm_header h; struct sanlk_lockspace ls; struct sanlk_host_event ev; int rv, reg_fd; if (!ls_name) return -EINVAL; memset(&ls, 0, sizeof(ls)); strncpy(ls.name, ls_name, SANLK_NAME_LEN); memset(&ev, 0, sizeof(ev)); if (he) memcpy(&ev, he, sizeof(ev)); rv = connect_socket(®_fd); if (rv < 0) return rv; rv = send_header(reg_fd, SM_CMD_REG_EVENT, flags, sizeof(ls) + sizeof(ev), 0, 0); if (rv < 0) goto fail; rv = send_data(reg_fd, &ls, sizeof(ls), 0); if (rv < 0) goto fail; rv = send_data(reg_fd, &ev, sizeof(ev), 0); if (rv < 0) goto fail; memset(&h, 0, sizeof(h)); rv = recv_data(reg_fd, &h, sizeof(h), MSG_WAITALL); if (rv < 0) { rv = -errno; goto fail; } if (rv != sizeof(h)) { rv = -1; goto fail; } rv = (int)h.data; if (rv < 0) goto fail; return reg_fd; fail: close(reg_fd); return rv; } int sanlock_end_event(int reg_fd, const char *ls_name, uint32_t flags) { struct sm_header h; struct sanlk_lockspace ls; int rv, fd; uint32_t end = 1; if (!ls_name) return -EINVAL; /* * write 4 bytes to the registered fd. sanlock attempts * a non-blocking read of 4 bytes from registered fds to * check if they have been unregistered. */ rv = send_data(reg_fd, &end, sizeof(end), 0); if (rv < 0) { close(reg_fd); return -EALREADY; } close(reg_fd); /* * sanlock does not poll registered event fds because * it receives nothing from them during normal operation, * only to indicate it's being closed. So, we need * to tell sanlock to check the registered event fds to * remove the one we've written to and closed above. */ memset(&ls, 0, sizeof(ls)); strncpy(ls.name, ls_name, SANLK_NAME_LEN); rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_END_EVENT, flags, sizeof(ls), 0, 0); if (rv < 0) goto out; rv = send_data(fd, &ls, sizeof(ls), 0); if (rv < 0) goto out; memset(&h, 0, sizeof(h)); rv = recv_data(fd, &h, sizeof(h), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(h)) { rv = -1; goto out; } rv = (int)h.data; if (rv < 0) goto out; rv = 0; out: close(fd); return rv; } int sanlock_set_event(const char *ls_name, struct sanlk_host_event *he, uint32_t flags) { struct sanlk_lockspace ls; struct sm_header h; int rv, fd; if (!ls_name || !he) return -EINVAL; memset(&ls, 0, sizeof(struct sanlk_lockspace)); strncpy(ls.name, ls_name, SANLK_NAME_LEN); rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_SET_EVENT, flags, sizeof(struct sanlk_lockspace) + sizeof(struct sanlk_host_event), 0, 0); if (rv < 0) goto out; rv = send_data(fd, &ls, sizeof(ls), 0); if (rv < 0) { rv = -errno; goto out; } rv = send_data(fd, he, sizeof(struct sanlk_host_event), 0); if (rv < 0) { rv = -errno; goto out; } memset(&h, 0, sizeof(h)); rv = recv_data(fd, &h, sizeof(h), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(h)) { rv = -1; goto out; } rv = (int)h.data; out: close(fd); return rv; } int sanlock_get_event(int reg_fd, GNUC_UNUSED uint32_t flags, struct sanlk_host_event *he, uint64_t *from_host_id, uint64_t *from_generation) { struct event_cb cb; int rv; /* * The caller's poll(2) indicates there's data, it doesn't know how * many events to read, and doesn't want to block, so they want to * get events until we return -EAGAIN to indicate there are no more. */ rv = recv_data(reg_fd, &cb, sizeof(cb), MSG_DONTWAIT); if (rv < 0) return -errno; if (rv != sizeof(cb)) return -1; memcpy(he, &cb.he, sizeof(struct sanlk_host_event)); if (from_host_id) *from_host_id = cb.from_host_id; if (from_generation) *from_generation = cb.from_generation; return 0; } /* old api */ int sanlock_init(struct sanlk_lockspace *ls, struct sanlk_resource *res, int max_hosts, int num_hosts) { if (ls) return sanlock_write_lockspace(ls, max_hosts, 0, 0); else return sanlock_write_resource(res, max_hosts, num_hosts, 0); } int sanlock_register(void) { int sock, rv; rv = connect_socket(&sock); if (rv < 0) return rv; rv = send_header(sock, SM_CMD_REGISTER, 0, 0, 0, 0); if (rv < 0) { close(sock); return rv; } return sock; } int sanlock_restrict(int sock, uint32_t flags) { int rv; rv = send_header(sock, SM_CMD_RESTRICT, flags, 0, 0, -1); if (rv < 0) return rv; rv = recv_result(sock); return rv; } int sanlock_version(uint32_t flags, uint32_t *version, uint32_t *proto) { struct sm_header h; int fd, rv; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_VERSION, flags, 0, 0, 0); if (rv < 0) goto out; memset(&h, 0, sizeof(h)); rv = recv_data(fd, &h, sizeof(h), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(h)) { rv = -1; goto out; } if (proto) *proto = h.version; rv = (int)h.data; if (rv < 0) goto out; *version = h.data2; rv = 0; out: close(fd); return rv; } int sanlock_killpath(int sock, uint32_t flags, const char *path, char *args) { char path_max[SANLK_HELPER_PATH_LEN]; char args_max[SANLK_HELPER_ARGS_LEN]; int rv, datalen; datalen = SANLK_HELPER_PATH_LEN + SANLK_HELPER_ARGS_LEN; memset(path_max, 0, sizeof(path_max)); memset(args_max, 0, sizeof(args_max)); snprintf(path_max, SANLK_HELPER_PATH_LEN-1, "%s", path); snprintf(args_max, SANLK_HELPER_ARGS_LEN-1, "%s", args); rv = send_header(sock, SM_CMD_KILLPATH, flags, datalen, 0, -1); if (rv < 0) return rv; rv = send_data(sock, path_max, SANLK_HELPER_PATH_LEN, 0); if (rv < 0) { rv = -errno; goto out; } rv = send_data(sock, args_max, SANLK_HELPER_ARGS_LEN, 0); if (rv < 0) { rv = -errno; goto out; } rv = recv_result(sock); out: return rv; } int sanlock_acquire2(int sock, int pid, uint32_t flags, struct sanlk_resource *res, struct sanlk_options *opt_in, struct sanlk_host *owner_host, char **owner_name) { struct sm_header h = { 0 }; struct sanlk_options opt = {0}; char namebuf[SANLK_NAME_LEN+1] = {0}; char *nameret; int result, rv, fd, data2; int datalen; if (res->num_disks != 1) return -EINVAL; rv = validate_res_disk_paths(res); if (rv < 0) return rv; if (opt_in) memcpy(&opt, opt_in, sizeof(struct sanlk_options)); datalen = sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk) + sizeof(struct sanlk_options); if (sock == -1) { /* connect to daemon and ask it to acquire a lease for another registered pid */ data2 = pid; rv = connect_socket(&fd); if (rv < 0) return rv; } else { /* use our own existing registered connection and ask daemon to acquire a lease for self */ data2 = -1; fd = sock; } rv = send_header(fd, SM_CMD_ACQUIRE2, flags, datalen, 1, data2); if (rv < 0) goto out; rv = send_data(fd, res, sizeof(struct sanlk_resource), 0); if (rv < 0) { rv = -1; goto out; } rv = send_data(fd, res->disks, sizeof(struct sanlk_disk), 0); if (rv < 0) { rv = -1; goto out; } rv = send_data(fd, &opt, sizeof(struct sanlk_options), 0); if (rv < 0) { rv = -1; goto out; } rv = recv_header(fd, &h); if (rv < 0) goto out; result = (int)h.data; if (h.length == sizeof(struct sm_header)) goto no_owner; rv = recv_data(fd, owner_host, sizeof(struct sanlk_host), MSG_WAITALL); if (rv < 0) goto out; rv = recv_data(fd, &namebuf, SANLK_NAME_LEN, MSG_WAITALL); if (rv < 0) goto out; if (owner_name && namebuf[0]) { int len = strlen(namebuf); if ((nameret = malloc(len+1))) { memcpy(nameret, namebuf, len); nameret[len] = '\0'; *owner_name = nameret; } } no_owner: rv = result; out: if (sock == -1) close(fd); return rv; } int sanlock_acquire(int sock, int pid, uint32_t flags, int res_count, struct sanlk_resource *res_args[], struct sanlk_options *opt_in) { struct sanlk_resource *res; struct sanlk_options opt; int rv, i, fd, data2; int datalen = 0; if (res_count > SANLK_MAX_RESOURCES) return -EINVAL; for (i = 0; i < res_count; i++) { res = res_args[i]; datalen += sizeof(struct sanlk_resource); if (res->num_disks > SANLK_MAX_DISKS) return -EINVAL; rv = validate_res_disk_paths(res); if (rv < 0) return rv; datalen += (res->num_disks * sizeof(struct sanlk_disk)); } datalen += sizeof(struct sanlk_options); if (opt_in) { memcpy(&opt, opt_in, sizeof(struct sanlk_options)); /* data beyond opt is not used */ /* datalen += opt_in->len; */ } else { memset(&opt, 0, sizeof(opt)); } if (sock == -1) { /* connect to daemon and ask it to acquire a lease for another registered pid */ data2 = pid; rv = connect_socket(&fd); if (rv < 0) return rv; } else { /* use our own existing registered connection and ask daemon to acquire a lease for self */ data2 = -1; fd = sock; } rv = send_header(fd, SM_CMD_ACQUIRE, flags, datalen, res_count, data2); if (rv < 0) goto out; for (i = 0; i < res_count; i++) { res = res_args[i]; rv = send_data(fd, res, sizeof(struct sanlk_resource), 0); if (rv < 0) { rv = -1; goto out; } rv = send_data(fd, res->disks, sizeof(struct sanlk_disk) * res->num_disks, 0); if (rv < 0) { rv = -1; goto out; } } rv = send_data(fd, &opt, sizeof(struct sanlk_options), 0); if (rv < 0) { rv = -1; goto out; } /* if (opt.len) { rv = send_data(fd, opt_in->str, opt.len, 0); if (rv < 0) { rv = -1; goto out; } } */ rv = recv_result(fd); out: if (sock == -1) close(fd); return rv; } int sanlock_inquire(int sock, int pid, uint32_t flags, int *res_count, char **res_state) { struct sm_header h; char *reply_data = NULL; int rv, fd, data2, len; *res_count = 0; if (res_state) *res_state = NULL; if (sock == -1) { /* connect to daemon and ask it to acquire a lease for another registered pid */ data2 = pid; rv = connect_socket(&fd); if (rv < 0) return rv; } else { /* use our own existing registered connection and ask daemon to acquire a lease for self */ data2 = -1; fd = sock; } rv = send_header(fd, SM_CMD_INQUIRE, flags, 0, 0, data2); if (rv < 0) goto out; /* get result */ memset(&h, 0, sizeof(h)); rv = recv_data(fd, &h, sizeof(h), MSG_WAITALL); if (rv != sizeof(h)) { rv = -1; goto out; } len = h.length - sizeof(h); if (!len) { rv = (int)h.data; goto out; } reply_data = malloc(len); if (!reply_data) { rv = -ENOMEM; goto out; } rv = recv_data(fd, reply_data, len, MSG_WAITALL); if (rv != len) { free(reply_data); rv = -1; goto out; } if (res_state) *res_state = reply_data; else free(reply_data); *res_count = (int)h.data2; rv = (int)h.data; out: if (sock == -1) close(fd); return rv; } int sanlock_convert(int sock, int pid, uint32_t flags, struct sanlk_resource *res) { int fd, rv, data2, datalen; if (!res) return -EINVAL; if (sock == -1) { /* connect to daemon and ask it to acquire a lease for another registered pid */ data2 = pid; rv = connect_socket(&fd); if (rv < 0) return rv; } else { /* use our own existing registered connection and ask daemon to acquire a lease for self */ data2 = -1; fd = sock; } datalen = sizeof(struct sanlk_resource); rv = send_header(fd, SM_CMD_CONVERT, flags, datalen, 0, data2); if (rv < 0) goto out; rv = send_data(fd, res, sizeof(struct sanlk_resource), 0); if (rv < 0) { rv = -errno; goto out; } rv = recv_result(fd); out: if (sock == -1) close(fd); return rv; } /* tell daemon to release lease(s) for given pid. I don't think the pid itself will usually tell sm to release leases, but it will be requested by a manager overseeing the pid */ int sanlock_release(int sock, int pid, uint32_t flags, int res_count, struct sanlk_resource *res_args[]) { int fd, rv, i, data2, datalen; if (sock == -1) { /* connect to daemon and ask it to acquire a lease for another registered pid */ data2 = pid; rv = connect_socket(&fd); if (rv < 0) return rv; } else { /* use our own existing registered connection and ask daemon to acquire a lease for self */ data2 = -1; fd = sock; } datalen = res_count * sizeof(struct sanlk_resource); rv = send_header(fd, SM_CMD_RELEASE, flags, datalen, res_count, data2); if (rv < 0) goto out; for (i = 0; i < res_count; i++) { rv = send_data(fd, res_args[i], sizeof(struct sanlk_resource), 0); if (rv < 0) { rv = -1; goto out; } } rv = recv_result(fd); out: if (sock == -1) close(fd); return rv; } int sanlock_request(uint32_t flags, uint32_t force_mode, struct sanlk_resource *res) { int fd, rv, datalen; if (!res) return -EINVAL; rv = validate_res_disk_paths(res); if (rv < 0) return rv; datalen = sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk) * res->num_disks; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_REQUEST, flags, datalen, force_mode, 0); if (rv < 0) goto out; rv = send_data(fd, res, sizeof(struct sanlk_resource), 0); if (rv < 0) { rv = -errno; goto out; } rv = send_data(fd, res->disks, sizeof(struct sanlk_disk) * res->num_disks, 0); if (rv < 0) { rv = -errno; goto out; } rv = recv_result(fd); out: close(fd); return rv; } int sanlock_examine(uint32_t flags, struct sanlk_lockspace *ls, struct sanlk_resource *res) { char *data; int rv, fd, cmd, datalen; if (!ls && !res) return -EINVAL; rv = connect_socket(&fd); if (rv < 0) return rv; if (ls && ls->host_id_disk.path[0]) { if (!memchr(ls->host_id_disk.path, '\0', SANLK_PATH_LEN)) return -EINVAL; cmd = SM_CMD_EXAMINE_LOCKSPACE; datalen = sizeof(struct sanlk_lockspace); data = (char *)ls; } else { cmd = SM_CMD_EXAMINE_RESOURCE; datalen = sizeof(struct sanlk_resource); data = (char *)res; } rv = send_header(fd, cmd, flags, datalen, 0, 0); if (rv < 0) goto out; rv = send_data(fd, data, datalen, 0); if (rv < 0) { rv = -errno; goto out; } rv = recv_result(fd); out: close(fd); return rv; } int sanlock_set_lvb(uint32_t flags, struct sanlk_resource *res, char *lvb, int lvblen) { int datalen = 0; int rv, fd; if (!res || !lvb || !lvblen) return -EINVAL; datalen = sizeof(struct sanlk_resource) + lvblen; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_SET_LVB, flags, datalen, 0, 0); if (rv < 0) return rv; rv = send_data(fd, res, sizeof(struct sanlk_resource), 0); if (rv < 0) { rv = -errno; goto out; } rv = send_data(fd, lvb, lvblen, 0); if (rv < 0) { rv = -1; goto out; } rv = recv_result(fd); out: close(fd); return rv; } int sanlock_get_lvb(uint32_t flags, struct sanlk_resource *res, char *lvb, int lvblen) { struct sm_header h; char *reply_data = NULL; int datalen = 0; int rv, fd, len; if (!res || !lvb || !lvblen) return -EINVAL; datalen = sizeof(struct sanlk_resource); rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_GET_LVB, flags, datalen, 0, 0); if (rv < 0) return rv; rv = send_data(fd, res, sizeof(struct sanlk_resource), 0); if (rv < 0) { rv = -1; goto out; } /* get result */ memset(&h, 0, sizeof(h)); rv = recv_data(fd, &h, sizeof(h), MSG_WAITALL); if (rv != sizeof(h)) { rv = -1; goto out; } len = h.length - sizeof(h); if (!len) { rv = (int)h.data; goto out; } reply_data = malloc(len); if (!reply_data) { rv = -ENOMEM; goto out; } rv = recv_data(fd, reply_data, len, MSG_WAITALL); if (rv != len) { free(reply_data); rv = -1; goto out; } if (lvblen < len) len = lvblen; memcpy(lvb, reply_data, len); free(reply_data); rv = (int)h.data; out: close(fd); return rv; } #if 0 int sanlock_format_rindex(struct sanlk_rindex *rx, uint32_t flags) { int rv, fd; if (!rx || !rx->lockspace_name[0] || !rx->disk.path[0] || !memchr(rx->disk.path, '\0', SANLK_PATH_LEN)) return -EINVAL; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_FORMAT_RINDEX, flags, sizeof(struct sanlk_rindex), 0, 0); if (rv < 0) goto out; rv = send_data(fd, rx, sizeof(struct sanlk_rindex), 0); if (rv < 0) { rv = -errno; goto out; } rv = recv_result(fd); out: close(fd); return rv; } int sanlock_rebuild_rindex(struct sanlk_rindex *rx, uint32_t flags) { int rv, fd; if (!rx || !rx->lockspace_name[0] || !rx->disk.path[0] || !memchr(rx->disk.path, '\0', SANLK_PATH_LEN)) return -EINVAL; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_REBUILD_RINDEX, flags, sizeof(struct sanlk_rindex), 0, 0); if (rv < 0) goto out; rv = send_data(fd, rx, sizeof(struct sanlk_rindex), 0); if (rv < 0) { rv = -errno; goto out; } rv = recv_result(fd); out: close(fd); return rv; } int sanlock_update_rindex(struct sanlk_rindex *rx, uint32_t flags, struct sanlk_rentry *re) { struct sanlk_rentry re_recv; int rv, fd; memset(&re_recv, 0, sizeof(re_recv)); if (!rx || !rx->lockspace_name[0] || !rx->disk.path[0] || !re || !memchr(rx->disk.path, '\0', SANLK_PATH_LEN)) return -EINVAL; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_UPDATE_RINDEX, flags, sizeof(struct sanlk_rindex) + sizeof(struct sanlk_rentry), 0, 0); if (rv < 0) goto out; rv = send_data(fd, rx, sizeof(struct sanlk_rindex), 0); if (rv < 0) { rv = -errno; goto out; } rv = send_data(fd, re, sizeof(struct sanlk_rentry), 0); if (rv < 0) { rv = -1; goto out; } rv = recv_result(fd); if (rv < 0) goto out; rv = recv_data(fd, &re_recv, sizeof(struct sanlk_rentry), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(struct sanlk_rentry)) { rv = -1; goto out; } memcpy(re, &re_recv, sizeof(struct sanlk_rentry)); rv = 0; out: close(fd); return rv; } int sanlock_lookup_rindex(struct sanlk_rindex *rx, uint32_t flags, struct sanlk_rentry *re) { struct sanlk_rentry re_recv; int rv, fd; memset(&re_recv, 0, sizeof(re_recv)); if (!rx || !rx->lockspace_name[0] || !rx->disk.path[0] || !re || !memchr(rx->disk.path, '\0', SANLK_PATH_LEN)) return -EINVAL; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_LOOKUP_RINDEX, flags, sizeof(struct sanlk_rindex) + sizeof(struct sanlk_resource), 0, 0); if (rv < 0) goto out; rv = send_data(fd, rx, sizeof(struct sanlk_rindex), 0); if (rv < 0) { rv = -errno; goto out; } rv = send_data(fd, re, sizeof(struct sanlk_rentry), 0); if (rv < 0) { rv = -1; goto out; } rv = recv_result(fd); if (rv < 0) goto out; rv = recv_data(fd, &re_recv, sizeof(struct sanlk_rentry), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(struct sanlk_rentry)) { rv = -1; goto out; } memcpy(re, &re_recv, sizeof(struct sanlk_rentry)); rv = 0; out: close(fd); return rv; } int sanlock_create_resource(struct sanlk_rindex *rx, uint32_t flags, struct sanlk_rentry *re, int max_hosts, int num_hosts) { struct sanlk_rentry re_recv; int rv, fd; memset(&re_recv, 0, sizeof(re_recv)); if (!rx || !rx->lockspace_name[0] || !rx->disk.path[0] || !re || !memchr(rx->disk.path, '\0', SANLK_PATH_LEN)) return -EINVAL; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_CREATE_RESOURCE, flags, sizeof(struct sanlk_rindex) + sizeof(struct sanlk_rentry), max_hosts, num_hosts); if (rv < 0) goto out; rv = send_data(fd, rx, sizeof(struct sanlk_rindex), 0); if (rv < 0) { rv = -errno; goto out; } rv = send_data(fd, re, sizeof(struct sanlk_rentry), 0); if (rv < 0) { rv = -1; goto out; } rv = recv_result(fd); if (rv < 0) goto out; rv = recv_data(fd, &re_recv, sizeof(struct sanlk_rentry), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(struct sanlk_rentry)) { rv = -1; goto out; } memcpy(re, &re_recv, sizeof(struct sanlk_rentry)); rv = 0; out: close(fd); return rv; } int sanlock_delete_resource(struct sanlk_rindex *rx, uint32_t flags, struct sanlk_rentry *re) { struct sanlk_rentry re_recv; int rv, fd; memset(&re_recv, 0, sizeof(re_recv)); if (!rx || !rx->lockspace_name[0] || !rx->disk.path[0] || !re || !memchr(rx->disk.path, '\0', SANLK_PATH_LEN)) return -EINVAL; rv = connect_socket(&fd); if (rv < 0) return rv; rv = send_header(fd, SM_CMD_DELETE_RESOURCE, flags, sizeof(struct sanlk_rindex) + sizeof(struct sanlk_rentry), 0, 0); if (rv < 0) goto out; rv = send_data(fd, rx, sizeof(struct sanlk_rindex), 0); if (rv < 0) { rv = -errno; goto out; } rv = send_data(fd, re, sizeof(struct sanlk_rentry), 0); if (rv < 0) { rv = -1; goto out; } rv = recv_result(fd); if (rv < 0) goto out; rv = recv_data(fd, &re_recv, sizeof(struct sanlk_rentry), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(struct sanlk_rentry)) { rv = -1; goto out; } rv = 0; out: close(fd); return rv; } #endif int sanlock_format_rindex(struct sanlk_rindex *rx, uint32_t flags) { return -EOPNOTSUPP; } int sanlock_rebuild_rindex(struct sanlk_rindex *rx, uint32_t flags) { return -EOPNOTSUPP; } int sanlock_update_rindex(struct sanlk_rindex *rx, uint32_t flags, struct sanlk_rentry *re) { return -EOPNOTSUPP; } int sanlock_lookup_rindex(struct sanlk_rindex *rx, uint32_t flags, struct sanlk_rentry *re) { return -EOPNOTSUPP; } int sanlock_create_resource(struct sanlk_rindex *rx, uint32_t flags, struct sanlk_rentry *re, int max_hosts, int num_hosts) { return -EOPNOTSUPP; } int sanlock_delete_resource(struct sanlk_rindex *rx, uint32_t flags, struct sanlk_rentry *re) { return -EOPNOTSUPP; } /* * src may have colons/spaces escaped (with backslash) or unescaped. * if unescaped colons/spaces are found, insert backslash before them. * * returns strlen of dst. */ size_t sanlock_path_export(char *dst, const char *src, size_t dstlen) { int i = 0; /* pos in src */ int j = 0; /* pos in dst */ memset(dst, 0, dstlen); for (i = 0; i < strlen(src); i++) { /* take an escape character plus whatever follows it. */ if (src[i] == '\\') { if (j > dstlen - 3) goto out; dst[j] = src[i]; j++; i++; dst[j] = src[i]; goto next_char; } /* add escape character before an unescaped space or colon. */ if ((src[i] == ' ') || (src[i] == ':')) { if (j > dstlen - 3) goto out; dst[j] = '\\'; j++; dst[j] = src[i]; goto next_char; } /* copy non-special char from src to dst. */ if (j > dstlen - 2) goto out; dst[j] = src[i]; next_char: if (dst[j] == '\0') goto out; j++; } out: return strlen(dst); } /* src has colons/spaces escaped with backslash, dst should have backslash removed */ size_t sanlock_path_import(char *dst, const char *src, size_t dstlen) { size_t j = 0; const char *p = src; while (j < dstlen) { if (*p == '\\') goto next_loop; dst[j] = *p; if (*p == '\0') return j; j++; next_loop: p++; } return 0; } /* * convert from struct sanlk_resource to string with format: * :::[::...]: */ int sanlock_res_to_str(struct sanlk_resource *res, char **str_ret) { char path[SANLK_PATH_LEN + 1]; char *str; int ret, len, pos, d; if (!res) return -EINVAL; str = malloc(SANLK_MAX_RES_STR + 1); if (!str) return -ENOMEM; memset(str, 0, SANLK_MAX_RES_STR + 1); len = SANLK_MAX_RES_STR; pos = 0; ret = snprintf(str + pos, len - pos, "%s:%s", res->lockspace_name, res->name); if (ret >= len - pos) goto fail; pos += ret; for (d = 0; d < res->num_disks; d++) { memset(path, 0, sizeof(path)); sanlock_path_export(path, res->disks[d].path, sizeof(path)); ret = snprintf(str + pos, len - pos, ":%s:%llu", path, (unsigned long long)res->disks[d].offset); if (ret >= len - pos) goto fail; pos += ret; } if (res->flags & SANLK_RES_SHARED) ret = snprintf(str + pos, len - pos, ":SH"); else ret = snprintf(str + pos, len - pos, ":%llu", (unsigned long long)res->lver); if (ret > len - pos) goto fail; pos += ret; if (pos > len) goto fail; *str_ret = str; return 0; fail: free(str); return -EINVAL; } /* * convert to struct sanlk_resource from string with format: * :::[::...][:] * * If str contains a backslash escape character, the backslash needs to be * excluded from the string in res struct. The path string in the res struct * needs to be suitable for passing to open(2), which means it should not * include escape characters. */ int sanlock_str_to_res(char *str, struct sanlk_resource **res_ret) { struct sanlk_resource *res; char sub[SANLK_PATH_LEN + 1]; int i, j, d, rv, len, sub_count, colons, num_disks, have_lver; if (strlen(str) < 3) return -ENXIO; colons = 0; for (i = 0; i < strlen(str); i++) { if (str[i] == '\\') { i++; continue; } if (str[i] == ':') colons++; } if (!colons || (colons == 2)) { return -1; } num_disks = (colons - 1) / 2; have_lver = (colons - 1) % 2; if (num_disks > SANLK_MAX_DISKS) return -2; len = sizeof(struct sanlk_resource) + num_disks * sizeof(struct sanlk_disk); res = malloc(len); if (!res) return -ENOMEM; memset(res, 0, len); res->num_disks = num_disks; d = 0; sub_count = 0; j = 0; memset(sub, 0, sizeof(sub)); len = strlen(str); for (i = 0; i < len + 1; i++) { if (str[i] == '\\') { if (i == (len - 1)) goto fail; if (j >= SANLK_PATH_LEN) goto fail; i++; sub[j++] = str[i]; continue; } if (i < len && str[i] != ':') { if (j >= SANLK_PATH_LEN) goto fail; sub[j++] = str[i]; continue; } /* do something with sub when we hit ':' or end of str, first and second subs are lockspace and resource names, then even sub is path, odd sub is offset */ if (sub_count < 2 && strlen(sub) > SANLK_NAME_LEN) goto fail; if (sub_count >= 2 && (strlen(sub) > SANLK_PATH_LEN-1 || strlen(sub) < 1)) goto fail; if (sub_count == 0) { strncpy(res->lockspace_name, sub, SANLK_NAME_LEN); } else if (sub_count == 1) { strncpy(res->name, sub, SANLK_NAME_LEN); } else if (!(sub_count % 2)) { if (have_lver && (d == num_disks)) { if (!strncmp(sub, "SH", 2)) { res->flags |= SANLK_RES_SHARED; } else { res->flags |= SANLK_RES_LVER; res->lver = strtoull(sub, NULL, 0); } } else { strncpy(res->disks[d].path, sub, SANLK_PATH_LEN - 1); } } else { rv = sscanf(sub, "%llu", (unsigned long long *)&res->disks[d].offset); if (rv != 1) goto fail; d++; } sub_count++; j = 0; memset(sub, 0, sizeof(sub)); } *res_ret = res; return 0; fail: free(res); return -1; } /* * convert from array of struct sanlk_resource * to state string with format: * "RESOURCE1 RESOURCE2 RESOURCE3 ..." * RESOURCE format in sanlock_res_to_str() comment */ int sanlock_args_to_state(int res_count, struct sanlk_resource *res_args[], char **res_state) { char *str, *state; int i, rv; state = malloc(res_count * (SANLK_MAX_RES_STR + 1)); if (!state) return -ENOMEM; memset(state, 0, res_count * (SANLK_MAX_RES_STR + 1)); for (i = 0; i < res_count; i++) { str = NULL; rv = sanlock_res_to_str(res_args[i], &str); if (rv < 0 || !str) { free(state); return rv; } if (strlen(str) > SANLK_MAX_RES_STR - 1) { free(str); free(state); return -EINVAL; } if (i) strcat(state, " "); strcat(state, str); free(str); } /* caller to free state */ *res_state = state; return 0; } /* * convert to array of struct sanlk_resource * from state string with format: * "RESOURCE1 RESOURCE2 RESOURCE3 ..." * RESOURCE format in sanlock_str_to_res() comment */ int sanlock_state_to_args(char *res_state, int *res_count, struct sanlk_resource ***res_args) { struct sanlk_resource **args; struct sanlk_resource *res; char str[SANLK_MAX_RES_STR + 1]; int count = 1, arg_count = 0; int escape = 0; int sep_colons = 0; int i, j, len, rv; for (i = 0; i < strlen(res_state); i++) { if (res_state[i] == '\\') { i++; continue; } if (res_state[i] == ' ') count++; } *res_count = count; args = malloc(count * sizeof(*args)); if (!args) return -ENOMEM; memset(args, 0, count * sizeof(*args)); j = 0; memset(str, 0, sizeof(str)); sep_colons = 0; len = strlen(res_state); for (i = 0; i < len + 1; i++) { if (i < len && res_state[i] == '\\') { str[j++] = res_state[i]; escape = 1; continue; } if (i < len && escape) { str[j++] = res_state[i]; escape = 0; continue; } if ((i < len) && (res_state[i] == ' ') && (sep_colons < 3)) { /* * This is a bit dubious. It's meant to detect when * a res string contains an unescaped space, and * inserts an escape char before it. An unescaped * space within a res string would otherwise be * misinterpreted as a separator between res strings. * If we've not yet seen three colons within a single * res string, then we should not be at the end yet. */ str[j++] = '\\'; str[j++] = res_state[i]; continue; } if (i < len && res_state[i] != ' ') { if (res_state[i] == ':') sep_colons++; str[j++] = res_state[i]; continue; } rv = sanlock_str_to_res(str, &res); if (rv < 0 || !res) goto fail_free; if (arg_count == count) goto fail_free; args[arg_count++] = res; j = 0; memset(str, 0, sizeof(str)); sep_colons = 0; } /* caller to free res_count res and args */ *res_count = arg_count; *res_args = args; return 0; fail_free: for (i = 0; i < count; i++) { if (args[i]) free(args[i]); } free(args); return rv; } /* * convert to struct sanlk_lockspace from string with format: * ::: */ int sanlock_str_to_lockspace(char *str, struct sanlk_lockspace *ls) { char *host_id = NULL; char *path = NULL; char *offset = NULL; int i; if (!str) return -EINVAL; for (i = 0; i < strlen(str); i++) { if (str[i] == '\\') { i++; continue; } if (str[i] == ':') { if (!host_id) host_id = &str[i]; else if (!path) path = &str[i]; else if (!offset) offset = &str[i]; } } if (host_id) { *host_id = '\0'; host_id++; } if (path) { *path = '\0'; path++; } if (offset) { *offset= '\0'; offset++; } strncpy(ls->name, str, SANLK_NAME_LEN); if (host_id) ls->host_id = atoll(host_id); if (path) sanlock_path_import(ls->host_id_disk.path, path, sizeof(ls->host_id_disk.path)); if (offset) ls->host_id_disk.offset = atoll(offset); return 0; } const char *sanlock_strerror(int rv) { switch (rv) { case SANLK_ERROR: return "General error"; case SANLK_AIO_TIMEOUT: return "IO timeout"; case SANLK_WD_ERROR: return "Watchdog device error"; case SANLK_DBLOCK_READ: return "Lease read error in dblock"; case SANLK_DBLOCK_WRITE: return "Lease write error in dblock"; case SANLK_DBLOCK_MBAL: return "Lease was acquired by another host in current ballot"; case SANLK_DBLOCK_LVER: return "Lease was acquired by another host in new ballot"; case SANLK_DBLOCK_CHECKSUM: return "Lease checksum error in dblock"; case SANLK_LEADER_READ: return "Lease read error in leader"; case SANLK_LEADER_WRITE: return "Lease write error in leader"; case SANLK_LEADER_DIFF: return "Lease read inconsistent"; case SANLK_LEADER_MAGIC: return "Lease does not exist on storage"; case SANLK_LEADER_VERSION: return "Lease format version on storage is not recognized"; case SANLK_LEADER_SECTORSIZE: return "Lease sector size is inconsistent"; case SANLK_LEADER_LOCKSPACE: return "Lease lockspace name is incorrect"; case SANLK_LEADER_RESOURCE: return "Lease resource name is incorrect"; case SANLK_LEADER_NUMHOSTS: return "Lease num_hosts is incorrect"; case SANLK_LEADER_CHECKSUM: return "Lease checksum error in leader"; case SANLK_ACQUIRE_LOCKSPACE: return "Lease lockspace is not found"; case SANLK_ACQUIRE_IDDISK: return "Lease lockspace disk cannot be opened"; case SANLK_ACQUIRE_IDLIVE: return "Lease is held by another host"; case SANLK_ACQUIRE_OWNED: return "Lease was acquired by another host in other ballot"; case SANLK_ACQUIRE_OTHER: return "Lease was acquired by another host in local commit"; case SANLK_ACQUIRE_SHRETRY: return "Lease is held by another host for shared acquire"; case SANLK_ACQUIRE_OWNED_RETRY: return "Lease is owned by another host"; case SANLK_ACQUIRE_OWNED_NO_TIMEOUT: return "Lease is owned by another notimeout host"; case SANLK_RELEASE_LVER: return "Lease release version is incorrect"; case SANLK_RELEASE_OWNER: return "Lease release owner is incorrect"; case SANLK_RENEW_OWNER: return "Lease renew owner is incorrect"; case SANLK_RENEW_DIFF: return "Lease renew data has changed"; case SANLK_HOSTID_BUSY: return "Lease host ID is being used by another host"; case SANLK_REQUEST_MAGIC: return "Lease request block has invalid data"; case SANLK_REQUEST_VERSION: return "Lease request block has invalid version"; case SANLK_REQUEST_OLD: return "Lease request has newer lease version"; case SANLK_REQUEST_LVER: return "Lease request block has newer version"; default: return "Unknown error"; }; } sanlock/src/client_cmd.c000066400000000000000000000352241516326262600155630ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock.h" #include "sanlock_sock.h" #include "client_cmd.h" #ifndef GNUC_UNUSED #define GNUC_UNUSED __attribute__((__unused__)) #endif extern int send_command(int cmd, uint32_t data); static void print_debug(char *str, int len) { char *p; int i; p = &str[0]; for (i = 0; i < len-1; i++) { if (str[i] == ' ') { str[i] = '\0'; printf(" %s\n", p); p = &str[i+1]; } } if (p) printf(" %s\n", p); } static ssize_t send_all(int sockfd, const void *buf, size_t len, int flags) { size_t rem = len; size_t off = 0; ssize_t rv; retry: rv = send(sockfd, (char *)buf + off, rem, flags); if (rv == -1 && errno == EINTR) goto retry; if (rv < 0) return -errno; if (rv < rem) { rem -= rv; off += rv; goto retry; } return 0; } static void status_daemon(struct sanlk_state *st, char *str, int debug) { printf("daemon %.48s\n", st->name); if (st->str_len && debug) print_debug(str, st->str_len); } static void status_client(struct sanlk_state *st, char *str, int debug) { printf("p %d ", st->data32); printf("%.48s\n", st->name); if (st->str_len && debug) print_debug(str, st->str_len); } static const char *add_rem_str(struct sanlk_state *st, char *str) { if (!st->str_len) return NULL; if (strstr(str, "list=add")) return "ADD"; if (strstr(str, "list=rem")) return "REM"; if (strstr(str, "list=orphan")) return "ORPHAN"; return NULL; } /* TODO: when path strings are exported, through status or inquire, we should export into a malloced buffer the size of the standard chars plus extra esc chars. */ static void status_lockspace(struct sanlk_state *st, char *str, char *bin, int debug) { struct sanlk_lockspace *ls = (struct sanlk_lockspace *)bin; char path[SANLK_PATH_LEN + 1]; const char *add_rem; memset(path, 0, sizeof(path)); sanlock_path_export(path, ls->host_id_disk.path, sizeof(path)); printf("s %.48s:%llu:%s:%llu", ls->name, (unsigned long long)ls->host_id, path, (unsigned long long)ls->host_id_disk.offset); add_rem = add_rem_str(st, str); if (add_rem) printf(" %s\n", add_rem); else printf("\n"); if (st->str_len && debug) print_debug(str, st->str_len); } static void status_resource(struct sanlk_state *st, char *str, char *bin, int debug) { struct sanlk_resource *res = (struct sanlk_resource *)bin; struct sanlk_disk *disk; char path[SANLK_PATH_LEN + 1]; const char *add_rem; int i; printf("r %.48s:%.48s", res->lockspace_name, res->name); for (i = 0; i < res->num_disks; i++) { disk = (struct sanlk_disk *)(bin + sizeof(struct sanlk_resource) + i * sizeof(struct sanlk_disk)); memset(path, 0, sizeof(path)); sanlock_path_export(path, disk->path, sizeof(path)); printf(":%s:%llu", path, (unsigned long long)disk->offset); } if (res->flags & SANLK_RES_SHARED) printf(":SH p %u", st->data32); else printf(":%llu p %u", (unsigned long long)st->data64, st->data32); add_rem = add_rem_str(st, str); if (add_rem) printf(" %s\n", add_rem); else printf("\n"); if (st->str_len && debug) print_debug(str, st->str_len); } static void status_host(struct sanlk_state *st, char *str, int debug) { printf("%u timestamp %llu\n", st->data32, (unsigned long long)st->data64); if (st->str_len && debug) print_debug(str, st->str_len); } static void status_worker(struct sanlk_state *st, char *str, int debug) { if (!debug) return; printf("worker %u\n", st->data32); if (st->str_len) print_debug(str, st->str_len); } static void status_rthread(struct sanlk_state *st, char *str, int debug) { if (!debug) return; printf("resource_thread\n"); if (st->str_len) print_debug(str, st->str_len); } static void print_st(struct sanlk_state *st, char *str, char *bin, int debug) { switch (st->type) { case SANLK_STATE_DAEMON: status_daemon(st, str, debug); break; case SANLK_STATE_CLIENT: status_client(st, str, debug); break; case SANLK_STATE_LOCKSPACE: status_lockspace(st, str, bin, debug); break; case SANLK_STATE_RESOURCE: status_resource(st, str, bin, debug); break; case SANLK_STATE_WORKER: status_worker(st, str, debug); break; case SANLK_STATE_RTHREAD: status_rthread(st, str, debug); break; } } #define MAX_SORT_ENTRIES 1024 static char *sort_bufs[MAX_SORT_ENTRIES]; static int sort_count; static int sort_done; static void print_type(int type, int debug) { struct sanlk_state *st; char *buf, *str, *bin; int i; for (i = 0; i < sort_count; i++) { buf = sort_bufs[i]; if (!buf) continue; st = (struct sanlk_state *)buf; str = buf + sizeof(struct sanlk_state); bin = buf + sizeof(struct sanlk_state) + SANLK_STATE_MAXSTR; if (!type || st->type == type) { print_st(st, str, bin, debug); free(buf); sort_bufs[i] = NULL; sort_done++; } } } static void print_p(int p, int debug) { struct sanlk_state *st; char *buf, *str, *bin; int i; for (i = 0; i < sort_count; i++) { buf = sort_bufs[i]; if (!buf) continue; st = (struct sanlk_state *)buf; str = buf + sizeof(struct sanlk_state); bin = buf + sizeof(struct sanlk_state) + SANLK_STATE_MAXSTR; if (st->type != SANLK_STATE_CLIENT) continue; if (st->data32 == p) { print_st(st, str, bin, debug); free(buf); sort_bufs[i] = NULL; sort_done++; } } } static int find_type(int type, int *sort_index) { struct sanlk_state *st; char *buf; int i; for (i = 0; i < sort_count; i++) { buf = sort_bufs[i]; if (!buf) continue; st = (struct sanlk_state *)buf; if (st->type == type) { *sort_index = i; return 0; } } return -1; } static void print_r(int p, char *s, int debug) { struct sanlk_resource *res; struct sanlk_state *st; char *buf, *str, *bin; int i; for (i = 0; i < sort_count; i++) { buf = sort_bufs[i]; if (!buf) continue; st = (struct sanlk_state *)buf; str = buf + sizeof(struct sanlk_state); bin = buf + sizeof(struct sanlk_state) + SANLK_STATE_MAXSTR; if (st->type != SANLK_STATE_RESOURCE) continue; res = (struct sanlk_resource *)bin; if ((p && st->data32 == p) || (s && !strncmp(s, res->lockspace_name, SANLK_NAME_LEN))) { print_st(st, str, bin, debug); free(buf); sort_bufs[i] = NULL; sort_done++; } } } static void print_r_by_p(int debug) { struct sanlk_state *st; char *buf, *str, *bin; int rv, i; while (1) { rv = find_type(SANLK_STATE_CLIENT, &i); if (rv < 0) return; buf = sort_bufs[i]; st = (struct sanlk_state *)buf; str = buf + sizeof(struct sanlk_state); bin = buf + sizeof(struct sanlk_state) + SANLK_STATE_MAXSTR; print_st(st, str, bin, debug); print_r(st->data32, NULL, debug); free(buf); sort_bufs[i] = NULL; sort_done++; } } static void print_r_by_s(int debug) { struct sanlk_state *st; char *buf, *str, *bin; int rv, i; while (1) { rv = find_type(SANLK_STATE_LOCKSPACE, &i); if (rv < 0) return; buf = sort_bufs[i]; st = (struct sanlk_state *)buf; str = buf + sizeof(struct sanlk_state); bin = buf + sizeof(struct sanlk_state) + SANLK_STATE_MAXSTR; print_st(st, str, bin, debug); print_r(0, st->name, debug); free(buf); sort_bufs[i] = NULL; sort_done++; } } static void recv_bin(int fd, struct sanlk_state *st, char *bin) { struct sanlk_resource *res; if (st->type == SANLK_STATE_LOCKSPACE) { recv(fd, bin, sizeof(struct sanlk_lockspace), MSG_WAITALL); } else if (st->type == SANLK_STATE_RESOURCE) { recv(fd, bin, sizeof(struct sanlk_resource), MSG_WAITALL); res = (struct sanlk_resource *)bin; recv(fd, bin+sizeof(struct sanlk_resource), res->num_disks * sizeof(struct sanlk_disk), MSG_WAITALL); } } int sanlock_status(int debug, char sort_arg) { struct sm_header h; struct sanlk_state state; char maxstr[SANLK_STATE_MAXSTR]; char maxbin[SANLK_STATE_MAXSTR]; struct sanlk_state *st; char *buf = NULL, *str, *bin; int fd, rv, len; int sort_p = 0, sort_s = 0; if (sort_arg == 'p') sort_p = 1; else if (sort_arg == 's') sort_s = 1; fd = send_command(SM_CMD_STATUS, 0); if (fd < 0) return fd; rv = recv(fd, &h, sizeof(h), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(h)) { rv = -1; goto out; } st = &state; str = maxstr; bin = maxbin; while (1) { if (sort_s || sort_p) { len = sizeof(struct sanlk_state) + SANLK_STATE_MAXSTR*4; buf = calloc(len, sizeof(char)); if (!buf) return -ENOMEM; st = (struct sanlk_state *)buf; str = buf + sizeof(struct sanlk_state); bin = buf + sizeof(struct sanlk_state) + SANLK_STATE_MAXSTR; } else { memset(&state, 0, sizeof(state)); memset(maxstr, 0, sizeof(maxstr)); memset(maxbin, 0, sizeof(maxbin)); } rv = recv(fd, st, sizeof(struct sanlk_state), MSG_WAITALL); if (!rv) break; if (rv != sizeof(struct sanlk_state)) break; if (st->str_len) { rv = recv(fd, str, st->str_len, MSG_WAITALL); if (rv != st->str_len) break; } recv_bin(fd, st, bin); if (sort_s || sort_p) { if ((sort_count == MAX_SORT_ENTRIES) || (!buf)) { printf("cannot sort over %d\n", MAX_SORT_ENTRIES); goto out; } sort_bufs[sort_count++] = buf; continue; } /* no sorting, print as received */ print_st(st, str, bin, debug); } if (sort_p) { print_type(SANLK_STATE_DAEMON, debug); print_p(-1, debug); print_type(SANLK_STATE_LOCKSPACE, debug); print_r_by_p(debug); if (sort_done < sort_count) { printf("-\n"); print_type(0, debug); } } else if (sort_s) { print_type(SANLK_STATE_DAEMON, debug); print_p(-1, debug); print_type(SANLK_STATE_CLIENT, debug); print_r_by_s(debug); if (sort_done < sort_count) { printf("-\n"); print_type(0, debug); } } rv = 0; out: close(fd); return rv; } static int lockspace_host_status(int debug, char *lockspace_name) { struct sm_header h; struct sanlk_state st; struct sanlk_lockspace lockspace; char str[SANLK_STATE_MAXSTR]; int fd, rv; if (!lockspace_name || !lockspace_name[0]) return -1; fd = send_command(SM_CMD_HOST_STATUS, 0); if (fd < 0) return fd; memset(&lockspace, 0, sizeof(lockspace)); snprintf(lockspace.name, SANLK_NAME_LEN, "%s", lockspace_name); rv = send_all(fd, &lockspace, sizeof(lockspace), 0); if (rv < 0) goto out; rv = recv(fd, &h, sizeof(h), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(h)) { rv = -1; goto out; } while (1) { rv = recv(fd, &st, sizeof(st), MSG_WAITALL); if (!rv) break; if (rv != sizeof(st)) break; if (st.str_len) { rv = recv(fd, str, st.str_len, MSG_WAITALL); if (rv != st.str_len) break; } switch (st.type) { case SANLK_STATE_HOST: status_host(&st, str, debug); break; } } rv = h.data; out: close(fd); return rv; } int sanlock_host_status(int debug, char *lockspace_name) { struct sm_header h; struct sanlk_state state; char maxstr[SANLK_STATE_MAXSTR]; char maxbin[SANLK_STATE_MAXSTR]; struct sanlk_state *st; char *str, *bin; struct sanlk_lockspace *ls; int fd, rv, i; if (lockspace_name && lockspace_name[0]) return lockspace_host_status(debug, lockspace_name); fd = send_command(SM_CMD_STATUS, SANLK_STATE_LOCKSPACE); if (fd < 0) return fd; rv = recv(fd, &h, sizeof(h), MSG_WAITALL); if (rv < 0) { rv = -errno; close(fd); return rv; } if (rv != sizeof(h)) { close(fd); return -1; } st = &state; str = maxstr; bin = maxbin; while (1) { memset(&state, 0, sizeof(state)); memset(maxstr, 0, sizeof(maxstr)); memset(maxbin, 0, sizeof(maxbin)); rv = recv(fd, st, sizeof(struct sanlk_state), MSG_WAITALL); if (!rv) break; if (rv != sizeof(struct sanlk_state)) break; if (st->str_len) { rv = recv(fd, str, st->str_len, MSG_WAITALL); if (rv != st->str_len) break; } recv_bin(fd, st, bin); if (st->type != SANLK_STATE_LOCKSPACE) continue; ls = (struct sanlk_lockspace *)bin; sort_bufs[sort_count++] = strdup(ls->name); } close(fd); for (i = 0; i < sort_count; i++) { printf("lockspace %s\n", sort_bufs[i]); lockspace_host_status(debug, sort_bufs[i]); free(sort_bufs[i]); } return 0; } int sanlock_renewal(char *lockspace_name) { struct sm_header h; struct sanlk_state st; struct sanlk_lockspace lockspace; char str[SANLK_STATE_MAXSTR]; int fd, rv; if (!lockspace_name || !lockspace_name[0]) return -1; fd = send_command(SM_CMD_RENEWAL, 0); if (fd < 0) return fd; memset(&lockspace, 0, sizeof(lockspace)); snprintf(lockspace.name, SANLK_NAME_LEN, "%s", lockspace_name); rv = send_all(fd, &lockspace, sizeof(lockspace), 0); if (rv < 0) goto out; rv = recv(fd, &h, sizeof(h), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(h)) { rv = -1; goto out; } while (1) { rv = recv(fd, &st, sizeof(st), MSG_WAITALL); if (!rv) break; if (rv != sizeof(st)) break; if (st.str_len) { rv = recv(fd, str, st.str_len, MSG_WAITALL); if (rv != st.str_len) break; } printf("%s\n", str); } rv = h.data; out: close(fd); return rv; } int sanlock_log_dump(int max_size) { struct sm_header h; char *buf; int fd, rv; buf = malloc(max_size); if (!buf) return -ENOMEM; memset(buf, 0, max_size); fd = send_command(SM_CMD_LOG_DUMP, 0); if (fd < 0) { free(buf); return fd; } memset(&h, 0, sizeof(h)); rv = recv(fd, &h, sizeof(h), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(h)) { rv = -1; goto out; } if (h.data <= 0 || h.data > max_size) goto out; rv = recv(fd, buf, h.data, MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (!rv) { rv = -1; goto out; } printf("%s", buf); printf("\n"); if (rv != h.data) printf("partial dump %d of %d\n", rv, h.data); out: close(fd); free(buf); return rv; } int sanlock_shutdown(uint32_t force, int wait_result) { struct sm_header h; int cmd; int fd; int rv = 0; if (wait_result) cmd = SM_CMD_SHUTDOWN_WAIT; else cmd = SM_CMD_SHUTDOWN; fd = send_command(cmd, force); if (fd < 0) return fd; if (cmd != SM_CMD_SHUTDOWN_WAIT) goto out; memset(&h, 0, sizeof(h)); rv = recv(fd, &h, sizeof(h), MSG_WAITALL); if (rv < 0) { rv = -errno; goto out; } if (rv != sizeof(h)) { rv = -1; goto out; } rv = h.data; out: close(fd); return rv; } sanlock/src/client_cmd.h000066400000000000000000000010711516326262600155610ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __CLIENT_CMD_H__ #define __CLIENT_CMD_H__ int sanlock_status(int debug, char sort_arg); int sanlock_host_status(int debug, char *lockspace_name); int sanlock_renewal(char *lockspace_name); int sanlock_log_dump(int max_size); int sanlock_shutdown(uint32_t force, int wait_result); #endif sanlock/src/cmd.c000066400000000000000000002632231516326262600142270ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock_internal.h" #include "sanlock_admin.h" #include "sanlock_sock.h" #include "diskio.h" #include "log.h" #include "paxos_lease.h" #include "delta_lease.h" #include "lockspace.h" #include "resource.h" #include "direct.h" #include "task.h" #include "cmd.h" /* from main.c */ void client_resume(int ci); void client_free(int ci); void client_recv_all(int ci, struct sm_header *h_recv, int pos); void client_pid_dead(int ci); void send_result(int ci, int fd, struct sm_header *h_recv, int result); static uint32_t token_id_counter = 1; static ssize_t recv_loop(int sockfd, void *buf, size_t len, int flags) { ssize_t rv; for (;;) { rv = recv(sockfd, buf, len, flags); if (rv == -1 && errno == EINTR) continue; return rv; } } static ssize_t send_all(int sockfd, const void *buf, size_t len, int flags) { size_t rem = len; size_t off = 0; ssize_t rv; retry: rv = send(sockfd, (char *)buf + off, rem, flags); if (rv == -1 && errno == EINTR) goto retry; if (rv < 0) return -errno; if (rv < rem) { rem -= rv; off += rv; goto retry; } return 0; } static void release_cl_tokens(struct task *task, struct client *cl) { struct token *token; int j; for (j = 0; j < cl->tokens_slots; j++) { token = cl->tokens[j]; if (!token) continue; release_token(task, token, NULL); free_token(token); } } static void release_new_tokens(struct task *task, struct token *new_tokens[], int alloc_count, int acquire_count) { int i; for (i = 0; i < acquire_count; i++) release_token(task, new_tokens[i], NULL); for (i = 0; i < alloc_count; i++) free(new_tokens[i]); } /* called with both spaces_mutex and cl->mutex held */ static int check_new_tokens_space(struct client *cl, struct token *new_tokens[], int new_tokens_count) { struct space_info spi; struct token *token; int i, rv, empty_slots = 0; for (i = 0; i < cl->tokens_slots; i++) { if (!cl->tokens[i]) empty_slots++; } if (empty_slots < new_tokens_count) { /* shouldn't ever happen */ log_error("check_new_tokens_space slots %d empty %d new_tokens %d", cl->tokens_slots, empty_slots, new_tokens_count); return -ENOENT; } /* space may have failed while new tokens were being acquired */ for (i = 0; i < new_tokens_count; i++) { token = new_tokens[i]; rv = _lockspace_info(token->r.lockspace_name, &spi); if (!rv && !spi.killing_pids && spi.host_id == token->host_id) continue; return -ENOSPC; } return 0; } static const char *acquire_error_str(int error) { switch (error) { case SANLK_ACQUIRE_IDLIVE: case SANLK_ACQUIRE_OWNED: case SANLK_ACQUIRE_OTHER: case SANLK_ACQUIRE_OWNED_RETRY: case SANLK_ACQUIRE_OWNED_NO_TIMEOUT: return "lease owned by other host"; case SANLK_ACQUIRE_SHRETRY: return "shared lease contention"; case SANLK_DBLOCK_READ: case SANLK_DBLOCK_WRITE: case SANLK_LEADER_READ: case SANLK_LEADER_WRITE: return "lease io error"; case SANLK_LEADER_DIFF: case SANLK_LEADER_VERSION: case SANLK_LEADER_SECTORSIZE: case SANLK_LEADER_LOCKSPACE: case SANLK_LEADER_RESOURCE: case SANLK_LEADER_NUMHOSTS: case SANLK_LEADER_CHECKSUM: return "lease data invalid"; case SANLK_LEADER_MAGIC: return "lease not found"; default: return ""; }; } static void cmd_acquire(struct task *task, struct cmd_args *ca, uint32_t cmd) { struct sm_header h = { 0 }; struct client *cl; struct token *token = NULL; struct token *new_tokens[SANLK_MAX_RESOURCES]; struct token **grow_tokens; struct sanlk_resource res; struct sanlk_options opt; struct space_info spi; struct sanlk_host owner_host = { 0 }; char killpath[SANLK_HELPER_PATH_LEN]; char killargs[SANLK_HELPER_ARGS_LEN]; char owner_host_name[SANLK_NAME_LEN+1] = { 0 }; int owner_host_send = 0; int token_len, disks_len; int fd, rv, i, j, empty_slots; int alloc_count = 0, acquire_count = 0; int pos = 0, pid_dead = 0; int new_tokens_count; int recv_done = 0; int result = 0; int grow_size; int cl_ci = ca->ci_target; int cl_fd = ca->cl_fd; int cl_pid = ca->cl_pid; cl = &client[cl_ci]; fd = client[ca->ci_in].fd; new_tokens_count = ca->header.data; log_cmd(cmd, "cmd_acquire %d,%d,%d ci_in %d fd %d count %d flags %x %c", cl_ci, cl_fd, cl_pid, ca->ci_in, fd, new_tokens_count, ca->header.cmd_flags, (cmd == SM_CMD_ACQUIRE2) ? '2' : ' '); if (new_tokens_count > SANLK_MAX_RESOURCES) { log_error("cmd_acquire %d,%d,%d new %d max %d", cl_ci, cl_fd, cl_pid, new_tokens_count, SANLK_MAX_RESOURCES); result = -E2BIG; goto done; } pthread_mutex_lock(&cl->mutex); if (cl->pid_dead) { result = -ESTALE; pthread_mutex_unlock(&cl->mutex); goto done; } empty_slots = 0; for (i = 0; i < cl->tokens_slots; i++) { if (!cl->tokens[i]) empty_slots++; } if (empty_slots < new_tokens_count) { log_debug("cmd_acquire grow tokens slots %d empty %d new %d", cl->tokens_slots, empty_slots, new_tokens_count); grow_size = (cl->tokens_slots + (SANLK_MAX_RESOURCES * 2)) * sizeof(struct token *); grow_tokens = malloc(grow_size); if (!grow_tokens) { log_error("cmd_acquire ENOMEM grow tokens slots %d empty %d new %d grow_size %d", cl->tokens_slots, empty_slots, new_tokens_count, grow_size); result = -ENOMEM; pthread_mutex_unlock(&cl->mutex); goto done; } else { memset(grow_tokens, 0, grow_size); memcpy(grow_tokens, cl->tokens, cl->tokens_slots * sizeof(struct token *)); free(cl->tokens); cl->tokens = grow_tokens; cl->tokens_slots += (SANLK_MAX_RESOURCES * 2); empty_slots += (SANLK_MAX_RESOURCES * 2); } } memcpy(killpath, cl->killpath, SANLK_HELPER_PATH_LEN); memcpy(killargs, cl->killargs, SANLK_HELPER_ARGS_LEN); pthread_mutex_unlock(&cl->mutex); if (empty_slots < new_tokens_count) { log_error("cmd_acquire %d,%d,%d new %d slots %d", cl_ci, cl_fd, cl_pid, new_tokens_count, empty_slots); result = -ENOENT; goto done; } /* * read resource input and allocate tokens for each */ for (i = 0; i < new_tokens_count; i++) { /* * receive sanlk_resource, create token for it */ rv = recv_loop(fd, &res, sizeof(struct sanlk_resource), MSG_WAITALL); if (rv > 0) pos += rv; if (rv != sizeof(struct sanlk_resource)) { log_error("cmd_acquire %d,%d,%d recv res %d %d", cl_ci, cl_fd, cl_pid, rv, errno); result = -ENOTCONN; goto done; } if (!res.num_disks || res.num_disks > SANLK_MAX_DISKS) { result = -ERANGE; goto done; } disks_len = res.num_disks * sizeof(struct sync_disk); token_len = sizeof(struct token) + disks_len; token = malloc(token_len); if (!token) { result = -ENOMEM; goto done; } memset(token, 0, token_len); token->disks = (struct sync_disk *)&token->r.disks[0]; /* shorthand */ token->r.num_disks = res.num_disks; memcpy(token->r.lockspace_name, res.lockspace_name, SANLK_NAME_LEN); memcpy(token->r.name, res.name, SANLK_NAME_LEN); if (res.flags & SANLK_RES_SHARED) token->r.flags |= SANLK_RES_SHARED; token->acquire_lver = res.lver; token->acquire_data64 = res.data64; token->acquire_data32 = res.data32; token->acquire_flags = res.flags; /* * receive sanlk_disk's / sync_disk's * * WARNING: as a shortcut, this requires that sync_disk and * sanlk_disk match; this is the reason for the pad fields * in sanlk_disk (TODO: let these differ?) */ rv = recv_loop(fd, token->disks, disks_len, MSG_WAITALL); if (rv > 0) pos += rv; if (rv != disks_len) { log_error("cmd_acquire %d,%d,%d recv disks %d %d", cl_ci, cl_fd, cl_pid, rv, errno); free_token(token); result = -ENOTCONN; goto done; } for (j = 0; j < token->r.num_disks; j++) { /* zero out pad1 and pad2, see WARNING above */ token->disks[j].sector_size = 0; token->disks[j].fd = -1; if (!token->disks[j].path[0] || !memchr(token->disks[j].path, '\0', SANLK_PATH_LEN)) { free_token(token); result = -ENODEV; goto done; } } token->token_id = token_id_counter++; new_tokens[i] = token; alloc_count++; } rv = recv_loop(fd, &opt, sizeof(struct sanlk_options), MSG_WAITALL); if (rv > 0) pos += rv; if (rv != sizeof(struct sanlk_options)) { log_error("cmd_acquire %d,%d,%d recv opt %d %d", cl_ci, cl_fd, cl_pid, rv, errno); result = -ENOTCONN; goto done; } memcpy(cl->owner_name, opt.owner_name, SANLK_NAME_LEN); /* data after opt is not used */ /* if (opt.len) { opt_str = malloc(opt.len); if (!opt_str) { result = -ENOMEM; goto done; } rv = recv_loop(fd, opt_str, opt.len, MSG_WAITALL); if (rv > 0) pos += rv; if (rv != opt.len) { log_error("cmd_acquire %d,%d,%d recv str %d %d", cl_ci, cl_fd, cl_pid, rv, errno); free(opt_str); result = -ENOTCONN; goto done; } } */ /* TODO: warn if header.length != sizeof(header) + pos ? */ recv_done = 1; /* * all command input has been received, start doing the acquire */ for (i = 0; i < new_tokens_count; i++) { token = new_tokens[i]; rv = lockspace_info(token->r.lockspace_name, &spi); if (rv < 0 || spi.killing_pids) { log_errot(token, "cmd_acquire %d,%d,%d invalid lockspace " "found %d failed %d name %.48s", cl_ci, cl_fd, cl_pid, rv, spi.killing_pids, token->r.lockspace_name); result = -ENOSPC; goto done; } token->host_id = spi.host_id; token->host_generation = spi.host_generation; token->space_id = spi.space_id; token->pid = cl_pid; token->client_id = cl_ci; token->io_timeout = spi.io_timeout; token->sector_size = spi.sector_size; /* starting hint, may be changed */ token->align_size = spi.align_size; /* starting hint, may be changed */ if (cl->restricted & SANLK_RESTRICT_SIGKILL) token->flags |= T_RESTRICT_SIGKILL; if (cl->restricted & SANLK_RESTRICT_SIGTERM) token->flags |= T_RESTRICT_SIGTERM; } for (i = 0; i < new_tokens_count; i++) { token = new_tokens[i]; rv = acquire_token(task, token, ca->header.cmd_flags, killpath, killargs, &owner_host); if (rv < 0) { switch (rv) { case -EAGAIN: case SANLK_ACQUIRE_IDLIVE: case SANLK_ACQUIRE_OWNED: case SANLK_ACQUIRE_OTHER: case SANLK_ACQUIRE_OWNED_RETRY: case SANLK_ACQUIRE_OWNED_NO_TIMEOUT: if ((cmd == SM_CMD_ACQUIRE2) && (owner_host.host_id > 0) && (owner_host.host_id <= DEFAULT_MAX_HOSTS)) owner_host_send = 1; break; } log_token(token, "cmd_acquire %d,%d,%d acquire_token error %d %s", cl_ci, cl_fd, cl_pid, rv, acquire_error_str(rv)); result = rv; goto done; } acquire_count++; } /* * Success acquiring the leases: * lock mutex, * 1. if pid is live, move new_tokens to cl->tokens, clear cmd_active, unlock mutex * 2. if pid is dead, clear cmd_active, unlock mutex, release new_tokens, release cl->tokens, client_free * * Failure acquiring the leases: * lock mutex, * 3. if pid is live, clear cmd_active, unlock mutex, release new_tokens * 4. if pid is dead, clear cmd_active, unlock mutex, release new_tokens, release cl->tokens, client_free * * client_pid_dead() won't touch cl->tokens while cmd_active is set. * As soon as we clear cmd_active and unlock the mutex, client_pid_dead * will attempt to clear cl->tokens itself. If we find client_pid_dead * has already happened when we look at pid_dead, then we know that it * won't be called again, and it's our responsibility to clear cl->tokens * and call client_free. */ /* * We hold both space_mutex and cl->mutex at once to create the crucial * linkage between the client pid and the lockspace. Once we release * these two mutexes, if the lockspace fails, this pid will be killed. * Prior to inserting the new_tokens into the client, if the lockspace * fails, kill_pids/client_using_pid would not find this pid (assuming * it doesn't already hold other tokens using the lockspace). If * the lockspace failed while we were acquiring the tokens, kill_pids * has already run and not found us, so we must revert what we've done * in acquire. * * Warning: * We could deadlock if we hold cl->mutex and take spaces_mutex, * because all_pids_dead() and kill_pids() hold spaces_mutex and take * cl->mutex. So, lock spaces_mutex first, then cl->mutex to avoid the * deadlock. * * Other approaches: * A solution may be to record in each sp all the pids/cis using it * prior to starting the acquire. Then we would not need to do this * check here to see if the lockspace has been killed (if it was, the * pid for this ci would have been killed in kill_pids), and * all_pids_dead() and kill_pids() would not need to go through each cl * and each cl->token to check if it's using the sp (it would know by * just looking at sp->pids[] and killing each). */ done: pthread_mutex_lock(&spaces_mutex); pthread_mutex_lock(&cl->mutex); log_cmd(cmd, "cmd_acquire %d,%d,%d result %d pid_dead %d owner_host_send %d", cl_ci, cl_fd, cl_pid, result, cl->pid_dead, owner_host_send); pid_dead = cl->pid_dead; cl->cmd_active = 0; if (!result && !pid_dead) { if (check_new_tokens_space(cl, new_tokens, new_tokens_count)) { /* case 1 becomes case 3 */ log_error("cmd_acquire %d,%d,%d invalid lockspace", cl_ci, cl_fd, cl_pid); result = -ENOSPC; } } /* 1. Success acquiring leases, and pid is live */ if (!result && !pid_dead) { for (i = 0; i < new_tokens_count; i++) { for (j = 0; j < cl->tokens_slots; j++) { if (!cl->tokens[j]) { cl->tokens[j] = new_tokens[i]; break; } } } /* goto reply after mutex unlock */ } /* * sanlock_acquire2() returns info about the owner when the * lease fails because it's held by another host. Most of * that owner info comes from the leader returned by * paxos_lease_acquire(), but two additional bits of info * are added here: the host status flag (indicating the * liveness state) and the owner name. * This is done here with spaces_mutex held because the * lookup/use of sp requires that. */ if (owner_host_send) { struct space *sp; struct host_status *hs; sp = find_lockspace(res.lockspace_name); if (sp) { hs = &sp->host_status[owner_host.host_id-1]; owner_host.flags = get_host_flag(sp->host_id, hs); memcpy(owner_host_name, hs->owner_name, NAME_ID_SIZE); } } pthread_mutex_unlock(&cl->mutex); pthread_mutex_unlock(&spaces_mutex); /* 1. Success acquiring leases, and pid is live */ if (!result && !pid_dead) { /* work done before mutex unlock */ goto reply; } /* 2. Success acquiring leases, and pid is dead */ if (!result && pid_dead) { release_new_tokens(task, new_tokens, alloc_count, acquire_count); release_cl_tokens(task, cl); client_free(cl_ci); result = -ENOTTY; goto reply; } /* 3. Failure acquiring leases, and pid is live */ if (result && !pid_dead) { release_new_tokens(task, new_tokens, alloc_count, acquire_count); goto reply; } /* 4. Failure acquiring leases, and pid is dead */ if (result && pid_dead) { release_new_tokens(task, new_tokens, alloc_count, acquire_count); release_cl_tokens(task, cl); client_free(cl_ci); goto reply; } reply: if (!recv_done) client_recv_all(ca->ci_in, &ca->header, pos); if (owner_host_send) { memcpy(&h, &ca->header, sizeof(struct sm_header)); h.version = SM_PROTO; h.data = result; h.length = sizeof(h) + sizeof(owner_host) + SANLK_NAME_LEN; send_all(fd, &h, sizeof(h), MSG_NOSIGNAL); send_all(fd, &owner_host, sizeof(owner_host), MSG_NOSIGNAL); send_all(fd, owner_host_name, SANLK_NAME_LEN, MSG_NOSIGNAL); } else { send_result(ca->ci_in, fd, &ca->header, result); } client_resume(ca->ci_in); } static void cmd_release(struct task *task, struct cmd_args *ca, uint32_t cmd) { struct client *cl; struct token *token; struct token *rem_tokens[SANLK_MAX_RESOURCES]; struct sanlk_resource res; struct sanlk_resource new; struct sanlk_resource *resrename = NULL; int fd, rv, i, j, found, pid_dead; int rem_tokens_count = 0; int result = 0; int cl_ci = ca->ci_target; int cl_fd = ca->cl_fd; int cl_pid = ca->cl_pid; cl = &client[cl_ci]; fd = client[ca->ci_in].fd; log_cmd(cmd, "cmd_release %d,%d,%d ci_in %d fd %d count %d flags %x", cl_ci, cl_fd, cl_pid, ca->ci_in, fd, ca->header.data, ca->header.cmd_flags); /* caller wants to release all resources */ if (ca->header.cmd_flags & SANLK_REL_ALL) { pthread_mutex_lock(&cl->mutex); for (j = 0; j < cl->tokens_slots; j++) { token = cl->tokens[j]; if (!token) continue; rem_tokens[rem_tokens_count++] = token; cl->tokens[j] = NULL; } pthread_mutex_unlock(&cl->mutex); goto do_remove; } if (ca->header.cmd_flags & SANLK_REL_ORPHAN) { rv = recv_loop(fd, &res, sizeof(struct sanlk_resource), MSG_WAITALL); if (rv != sizeof(struct sanlk_resource)) { log_error("cmd_release %d,%d,%d recv res %d %d", cl_ci, cl_fd, cl_pid, rv, errno); result = -ENOTCONN; goto do_remove; } result = release_orphan(&res); goto out; } if (ca->header.cmd_flags & SANLK_REL_RENAME) { rv = recv_loop(fd, &res, sizeof(struct sanlk_resource), MSG_WAITALL); if (rv != sizeof(struct sanlk_resource)) { log_error("cmd_release %d,%d,%d recv res %d %d", cl_ci, cl_fd, cl_pid, rv, errno); result = -ENOTCONN; goto do_remove; } /* second res struct has new name for first res */ rv = recv_loop(fd, &new, sizeof(struct sanlk_resource), MSG_WAITALL); if (rv != sizeof(struct sanlk_resource)) { log_error("cmd_release %d,%d,%d recv new %d %d", cl_ci, cl_fd, cl_pid, rv, errno); result = -ENOTCONN; goto do_remove; } found = 0; pthread_mutex_lock(&cl->mutex); for (j = 0; j < cl->tokens_slots; j++) { token = cl->tokens[j]; if (!token) continue; if (memcmp(token->r.lockspace_name, res.lockspace_name, NAME_ID_SIZE)) continue; if (memcmp(token->r.name, res.name, NAME_ID_SIZE)) continue; rem_tokens[rem_tokens_count++] = token; cl->tokens[j] = NULL; found = 1; break; } pthread_mutex_unlock(&cl->mutex); if (!found) { log_error("cmd_release %d,%d,%d no resource %.48s", cl_ci, cl_fd, cl_pid, res.name); result = -1; } resrename = &new; goto do_remove; } /* caller is specifying specific resources to release */ for (i = 0; i < ca->header.data; i++) { rv = recv_loop(fd, &res, sizeof(struct sanlk_resource), MSG_WAITALL); if (rv != sizeof(struct sanlk_resource)) { log_error("cmd_release %d,%d,%d recv res %d %d", cl_ci, cl_fd, cl_pid, rv, errno); result = -ENOTCONN; break; } found = 0; pthread_mutex_lock(&cl->mutex); for (j = 0; j < cl->tokens_slots; j++) { token = cl->tokens[j]; if (!token) continue; if (memcmp(token->r.lockspace_name, res.lockspace_name, NAME_ID_SIZE)) continue; if (memcmp(token->r.name, res.name, NAME_ID_SIZE)) continue; rem_tokens[rem_tokens_count++] = token; cl->tokens[j] = NULL; found = 1; break; } pthread_mutex_unlock(&cl->mutex); if (!found) { log_error("cmd_release %d,%d,%d no resource %.48s", cl_ci, cl_fd, cl_pid, res.name); result = -1; } } do_remove: for (i = 0; i < rem_tokens_count; i++) { token = rem_tokens[i]; if (ca->header.cmd_flags & SANLK_REL_NO_DISK) { rv = release_token_nodisk(task, token); } else { rv = release_token(task, token, resrename); } if (rv < 0) result = rv; free_token(token); } out: pthread_mutex_lock(&cl->mutex); log_cmd(cmd, "cmd_release %d,%d,%d result %d pid_dead %d count %d", cl_ci, cl_fd, cl_pid, result, cl->pid_dead, rem_tokens_count); pid_dead = cl->pid_dead; cl->cmd_active = 0; if (!pid_dead && cl->kill_count) { /* * If no tokens are left, clear all cl killing state. The * cl no longer needs to be killed, and the pid may continue * running, even if a failed lockspace it was using is * released. When the lockspace is re-added, the tokens * may be re-acquired for this same cl/pid. */ found = 0; for (j = 0; j < cl->tokens_slots; j++) { if (!cl->tokens[j]) continue; found = 1; break; } if (!found) { cl->kill_count = 0; cl->kill_last = 0; cl->flags &= ~CL_RUNPATH_SENT; log_cmd(cmd, "cmd_release %d,%d,%d clear kill state", cl_ci, cl_fd, cl_pid); } } pthread_mutex_unlock(&cl->mutex); if (pid_dead) { /* release any tokens not already released above */ release_cl_tokens(task, cl); client_free(cl_ci); } send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); } static void cmd_inquire(struct task *task, struct cmd_args *ca, uint32_t cmd) { struct sm_header h; struct token *token; struct client *cl; char *state = NULL, *str; int state_maxlen = 0, state_strlen = 0; int res_count = 0, cat_count = 0; int fd, i, rv, pid_dead; int result = 0; int cl_ci = ca->ci_target; int cl_fd = ca->cl_fd; int cl_pid = ca->cl_pid; cl = &client[cl_ci]; fd = client[ca->ci_in].fd; log_cmd(cmd, "cmd_inquire %d,%d,%d ci_in %d fd %d", cl_ci, cl_fd, cl_pid, ca->ci_in, fd); pthread_mutex_lock(&cl->mutex); if (cl->pid_dead) { result = -ESTALE; goto done; } for (i = 0; i < cl->tokens_slots; i++) { if (cl->tokens[i]) res_count++; } if (!res_count) { result = 0; goto done; } state_maxlen = res_count * (SANLK_MAX_RES_STR + 1); state = malloc(state_maxlen); if (!state) { result = -ENOMEM; goto done; } memset(state, 0, state_maxlen); /* should match sanlock_args_to_state() */ for (i = 0; i < cl->tokens_slots; i++) { token = cl->tokens[i]; if (!token) continue; /* check number of tokens hasn't changed since first count */ if (cat_count >= res_count) { log_error("cmd_inquire %d,%d,%d count changed %d %d", cl_ci, cl_fd, cl_pid, res_count, cat_count); result = -ENOENT; goto done; } str = NULL; rv = sanlock_res_to_str(&token->r, &str); if (rv < 0 || !str) { log_errot(token, "cmd_inquire %d,%d,%d res_to_str %d", cl_ci, cl_fd, cl_pid, rv); result = -ELIBACC; goto done; } if (strlen(str) > SANLK_MAX_RES_STR - 1) { log_errot(token, "cmd_inquire %d,%d,%d strlen %zu", cl_ci, cl_fd, cl_pid, strlen(str)); free(str); result = -ELIBBAD; goto done; } /* space is str separator, so it's invalid within each str */ if (strstr(str, " ")) { log_errot(token, "cmd_inquire %d,%d,%d str space", cl_ci, cl_fd, cl_pid); free(str); result = -ELIBSCN; goto done; } if (cat_count) strcat(state, " "); strcat(state, str); cat_count++; free(str); } state[state_maxlen - 1] = '\0'; state_strlen = strlen(state); result = 0; done: pid_dead = cl->pid_dead; cl->cmd_active = 0; pthread_mutex_unlock(&cl->mutex); log_cmd(cmd, "cmd_inquire %d,%d,%d result %d pid_dead %d res_count %d cat_count %d strlen %d", cl_ci, cl_fd, cl_pid, result, pid_dead, res_count, cat_count, state_strlen); if (pid_dead) { release_cl_tokens(task, cl); client_free(cl_ci); } memcpy(&h, &ca->header, sizeof(struct sm_header)); h.version = SM_PROTO; h.data = result; h.data2 = res_count; if (state) { h.length = sizeof(h) + state_strlen + 1; send_all(fd, &h, sizeof(h), MSG_NOSIGNAL); send_all(fd, state, state_strlen + 1, MSG_NOSIGNAL); free(state); } else { h.length = sizeof(h); send_all(fd, &h, sizeof(h), MSG_NOSIGNAL); } client_resume(ca->ci_in); } /* * The behavior may be a little iffy in the case where a pid is killed (due to * lockspace failure) while it is doing convert. If the pid responds by * exiting, then this cmd_convert will see pid_dead and release all tokens at * the end. If the pid wants to respond by explicitly releasing its leases, * then this convert should fail and return for the same reason the lockspace * failed. Once the convert returns, the pid can respond to the killpath by * releasing all the leases. * * This sets cmd_active, along with acquire/release/inquire, which means * that it is serialized along with all cmds that set cmd_active, and * cl->tokens will not change while the cmd is active. This also means * it has to handle pid_dead at the end in case the pid exited while the * cmd was active and cl->tokens need to be released. * (killpath also sets cmd_active so that tokens are not acquired * while it's being set.) */ static void cmd_convert(struct task *task, struct cmd_args *ca, uint32_t cmd) { struct sanlk_resource res; struct token *token; struct client *cl; int cl_ci = ca->ci_target; int cl_fd = ca->cl_fd; int cl_pid = ca->cl_pid; int pid_dead = 0; int result = 0; int found = 0; int fd, i, rv; cl = &client[cl_ci]; fd = client[ca->ci_in].fd; log_cmd(cmd, "cmd_convert %d,%d,%d ci_in %d fd %d", cl_ci, cl_fd, cl_pid, ca->ci_in, fd); rv = recv_loop(fd, &res, sizeof(struct sanlk_resource), MSG_WAITALL); if (rv != sizeof(struct sanlk_resource)) { result = -ENOTCONN; goto reply; } pthread_mutex_lock(&cl->mutex); for (i = 0; i < cl->tokens_slots; i++) { token = cl->tokens[i]; if (!token) continue; if (memcmp(token->r.lockspace_name, res.lockspace_name, NAME_ID_SIZE)) continue; if (memcmp(token->r.name, res.name, NAME_ID_SIZE)) continue; found = 1; break; } pthread_mutex_unlock(&cl->mutex); if (!found) { result = -ENOENT; goto cmd_done; } rv = convert_token(task, &res, token, ca->header.cmd_flags); if (rv < 0) result = rv; cmd_done: pthread_mutex_lock(&cl->mutex); pid_dead = cl->pid_dead; cl->cmd_active = 0; pthread_mutex_unlock(&cl->mutex); reply: log_cmd(cmd, "cmd_convert %d,%d,%d result %d pid_dead %d", cl_ci, cl_fd, cl_pid, result, pid_dead); if (pid_dead) { release_cl_tokens(task, cl); client_free(cl_ci); } send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); } static void cmd_request(struct task *task, struct cmd_args *ca, uint32_t cmd) { struct token *token; struct sanlk_resource res; struct space_info spi; uint64_t owner_id = 0; uint32_t force_mode; int token_len, disks_len; int j, fd, rv, error, result; fd = client[ca->ci_in].fd; force_mode = ca->header.data; /* receiving and setting up token copied from cmd_acquire */ rv = recv_loop(fd, &res, sizeof(struct sanlk_resource), MSG_WAITALL); if (rv != sizeof(struct sanlk_resource)) { log_error("cmd_request %d,%d recv %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } if (!res.num_disks || res.num_disks > SANLK_MAX_DISKS) { result = -ERANGE; goto reply; } disks_len = res.num_disks * sizeof(struct sync_disk); token_len = sizeof(struct token) + disks_len; token = malloc(token_len); if (!token) { result = -ENOMEM; goto reply; } memset(token, 0, token_len); token->disks = (struct sync_disk *)&token->r.disks[0]; /* shorthand */ token->r.num_disks = res.num_disks; memcpy(token->r.lockspace_name, res.lockspace_name, SANLK_NAME_LEN); memcpy(token->r.name, res.name, SANLK_NAME_LEN); token->acquire_lver = res.lver; token->acquire_data64 = res.data64; token->acquire_data32 = res.data32; token->acquire_flags = res.flags; /* * receive sanlk_disk's / sync_disk's * * WARNING: as a shortcut, this requires that sync_disk and * sanlk_disk match; this is the reason for the pad fields * in sanlk_disk (TODO: let these differ?) */ rv = recv_loop(fd, token->disks, disks_len, MSG_WAITALL); if (rv != disks_len) { result = -ENOTCONN; goto reply_free; } for (j = 0; j < token->r.num_disks; j++) { /* zero out pad1 and pad2, see WARNING above */ token->disks[j].sector_size = 0; token->disks[j].fd = -1; if (!token->disks[j].path[0] || !memchr(token->disks[j].path, '\0', SANLK_PATH_LEN)) { result = -ENODEV; goto reply_free; } } log_cmd(cmd, "cmd_request %d,%d force_mode %u %.48s:%.48s:%.256s:%llu", ca->ci_in, fd, force_mode, token->r.lockspace_name, token->r.name, token->disks[0].path, (unsigned long long)token->r.disks[0].offset); rv = lockspace_info(token->r.lockspace_name, &spi); if (rv < 0 || spi.killing_pids) { result = -ENOSPC; goto reply_free; } token->io_timeout = spi.io_timeout; token->sector_size = spi.sector_size; token->align_size = spi.align_size; error = request_token(task, token, force_mode, &owner_id, (ca->header.cmd_flags & SANLK_REQUEST_NEXT_LVER)); if (error < 0) { result = error; goto reply_free; } result = 0; if (!token->acquire_lver && !force_mode) goto reply_free; if (owner_id) host_status_set_bit(token->r.lockspace_name, owner_id); reply_free: free_token(token); reply: log_cmd(cmd, "cmd_request %d,%d done %d", ca->ci_in, fd, result); send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); } static void cmd_examine(struct task *task GNUC_UNUSED, struct cmd_args *ca, uint32_t cmd) { union { struct sanlk_resource r; struct sanlk_lockspace s; } buf; struct sanlk_resource *res = NULL; struct sanlk_lockspace *ls = NULL; char *space_name = NULL; char *res_name = NULL; int fd, rv, result, count = 0, datalen; fd = client[ca->ci_in].fd; if (ca->header.cmd == SM_CMD_EXAMINE_RESOURCE) { datalen = sizeof(struct sanlk_resource); res = &buf.r; } else { datalen = sizeof(struct sanlk_lockspace); ls = &buf.s; } rv = recv_loop(fd, &buf, datalen, MSG_WAITALL); if (rv != datalen) { log_error("cmd_examine %d,%d recv %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } if (res) { space_name = res->lockspace_name; res_name = res->name; } else { space_name = ls->name; } log_cmd(cmd, "cmd_examine %d,%d %.48s %.48s", ca->ci_in, fd, space_name, res_name ? res_name : ""); count = set_resource_examine(space_name, res_name); result = 0; reply: log_cmd(cmd, "cmd_examine %d,%d done %d", ca->ci_in, fd, count); send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); } static void cmd_set_host(struct task *task, struct cmd_args *ca, uint32_t cmd) { struct sanlk_lockspace lockspace; struct sanlk_host host; int fd, rv, result; fd = client[ca->ci_in].fd; rv = recv_loop(fd, &lockspace, sizeof(struct sanlk_lockspace), MSG_WAITALL); if (rv != sizeof(struct sanlk_lockspace)) { log_error("cmd_set_host %d,%d recv ls %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } rv = recv_loop(fd, &host, sizeof(struct sanlk_host), MSG_WAITALL); if (rv != sizeof(struct sanlk_host)) { log_error("cmd_set_host %d,%d recv host %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } if (lockspace.host_id_disk.path[0] && !memchr(lockspace.host_id_disk.path, '\0', SANLK_PATH_LEN)) { log_error("Invalid lockspace path."); result = -ENODEV; goto reply; } log_cmd(cmd, "cmd_set_host %d,%d %.48s host %llu %llu op %u", ca->ci_in, fd, lockspace.name, (unsigned long long)host.host_id, (unsigned long long)host.generation, ca->header.data); if (ca->header.data == SANLK_SET_HOST_DEAD_EXT) result = lockspace_set_host_dead_ext(task, &lockspace, &host, ca->header.data2); else result = -EINVAL; log_cmd(cmd, "cmd_set_host %d,%d result %d", ca->ci_in, fd, result); reply: send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); } static void cmd_set_lvb(struct task *task GNUC_UNUSED, struct cmd_args *ca, uint32_t cmd) { struct sanlk_resource res; char *lvb = NULL; int lvblen, rv, fd, result; fd = client[ca->ci_in].fd; rv = recv_loop(fd, &res, sizeof(struct sanlk_resource), MSG_WAITALL); if (rv != sizeof(struct sanlk_resource)) { log_error("cmd_set_lvb %d,%d recv %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } lvblen = ca->header.length - sizeof(struct sm_header) - sizeof(struct sanlk_resource); /* 4096 is the max sector size we handle, it is compared against the actual 512/4K sector size in res_set_lvb. */ if (lvblen > 4096) { log_error("cmd_set_lvb %d,%d lvblen %d too big", ca->ci_in, fd, lvblen); result = -E2BIG; goto reply; } lvb = malloc(lvblen); if (!lvb) { result = -ENOMEM; goto reply; } rv = recv_loop(fd, lvb, lvblen, MSG_WAITALL); if (rv != lvblen) { log_error("cmd_set_lvb %d,%d recv lvblen %d lvb %d %d", ca->ci_in, fd, lvblen, rv, errno); result = -ENOTCONN; goto reply; } result = res_set_lvb(&res, lvb, lvblen); log_cmd(cmd, "cmd_set_lvb ci %d fd %d result %d res %s:%s", ca->ci_in, fd, result, res.lockspace_name, res.name); reply: if (lvb) free(lvb); send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); } static void cmd_get_lvb(struct task *task GNUC_UNUSED, struct cmd_args *ca, uint32_t cmd) { struct sm_header h; struct sanlk_resource res; char *lvb = NULL; int lvblen = 0, rv, fd, result; fd = client[ca->ci_in].fd; rv = recv_loop(fd, &res, sizeof(struct sanlk_resource), MSG_WAITALL); if (rv != sizeof(struct sanlk_resource)) { log_error("cmd_get_lvb %d,%d recv %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } /* if 0 then we use the sector size as lvb len */ lvblen = ca->header.data2; result = res_get_lvb(&res, &lvb, &lvblen); log_cmd(cmd, "cmd_get_lvb ci %d fd %d result %d res %s:%s", ca->ci_in, fd, result, res.lockspace_name, res.name); reply: memcpy(&h, &ca->header, sizeof(struct sm_header)); h.version = SM_PROTO; h.data = result; h.data2 = 0; h.length = sizeof(h) + lvblen; send_all(fd, &h, sizeof(h), MSG_NOSIGNAL); if (lvb) { send_all(fd, lvb, lvblen, MSG_NOSIGNAL); free(lvb); } client_resume(ca->ci_in); } static int shutdown_reply_ci = -1; static int shutdown_reply_fd = -1; static int daemon_shutdown_start(int ci, int fd, int force) { int rv; if (force) { shutdown_reply_ci = ci; shutdown_reply_fd = fd; external_shutdown = 2; return 0; } pthread_mutex_lock(&spaces_mutex); if (list_empty(&spaces) && list_empty(&spaces_rem) && list_empty(&spaces_add)) { shutdown_reply_ci = ci; shutdown_reply_fd = fd; external_shutdown = 1; rv = 0; } else { rv = -EBUSY; } pthread_mutex_unlock(&spaces_mutex); return rv; } static void cmd_shutdown_wait(struct task *task GNUC_UNUSED, struct cmd_args *ca, uint32_t cmd) { int fd, result; fd = client[ca->ci_in].fd; result = daemon_shutdown_start(ca->ci_in, fd, ca->header.data); /* * daemon_shutdown_reply will send the result at the * end of main_loop. */ if (!result) return; send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); } void daemon_shutdown_reply(void) { struct sm_header h; /* shutdown wait was not used */ if (shutdown_reply_fd == -1) return; memset(&h, 0, sizeof(h)); h.magic = SM_MAGIC; h.version = SM_PROTO; h.length = sizeof(h); send_all(shutdown_reply_fd, &h, sizeof(h), MSG_NOSIGNAL); close(shutdown_reply_fd); client_resume(shutdown_reply_ci); } static int validate_lockspace_path(struct sanlk_lockspace *ls) { if (!ls->host_id_disk.path[0] || !memchr(ls->host_id_disk.path, '\0', SANLK_PATH_LEN)) { log_error("Invalid lockspace path."); return -ENODEV; } return 0; } static void cmd_add_lockspace(struct cmd_args *ca, uint32_t cmd) { struct sanlk_lockspace lockspace; struct space *sp; uint32_t io_timeout; int async = ca->header.cmd_flags & SANLK_ADD_ASYNC; int nodelay = (ca->header.cmd_flags & SANLK_ADD_NODELAY) ? 1 : 0; int fd, rv, result; fd = client[ca->ci_in].fd; rv = recv_loop(fd, &lockspace, sizeof(struct sanlk_lockspace), MSG_WAITALL); if (rv != sizeof(struct sanlk_lockspace)) { log_error("cmd_add_lockspace %d,%d recv %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } rv = validate_lockspace_path(&lockspace); if (rv < 0) { result = rv; goto reply; } log_cmd(cmd, "cmd_add_lockspace %d,%d %.48s:%llu:%s:%llu ls_flags 0x%x cmd_flags 0x%x timeout %u", ca->ci_in, fd, lockspace.name, (unsigned long long)lockspace.host_id, lockspace.host_id_disk.path, (unsigned long long)lockspace.host_id_disk.offset, lockspace.flags, ca->header.cmd_flags, ca->header.data); io_timeout = ca->header.data; if (!io_timeout) io_timeout = com.io_timeout; rv = add_lockspace_start(&lockspace, io_timeout, nodelay, &sp); if (rv < 0) { result = rv; goto reply; } if (async) { result = rv; log_cmd(cmd, "cmd_add_lockspace %d,%d async done %d", ca->ci_in, fd, result); send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); add_lockspace_wait(sp); return; } result = add_lockspace_wait(sp); reply: log_cmd(cmd, "cmd_add_lockspace %d,%d done %d", ca->ci_in, fd, result); send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); } static void cmd_inq_lockspace(struct cmd_args *ca, uint32_t cmd) { struct sanlk_lockspace lockspace; int waitrs = ca->header.cmd_flags & SANLK_INQ_WAIT; int fd, rv, result; fd = client[ca->ci_in].fd; rv = recv_loop(fd, &lockspace, sizeof(struct sanlk_lockspace), MSG_WAITALL); if (rv != sizeof(struct sanlk_lockspace)) { log_error("cmd_inq_lockspace %d,%d recv %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } rv = validate_lockspace_path(&lockspace); if (rv < 0) { result = rv; goto reply; } log_cmd(cmd, "cmd_inq_lockspace %d,%d %.48s:%llu:%s:%llu cmd_flags %x", ca->ci_in, fd, lockspace.name, (unsigned long long)lockspace.host_id, lockspace.host_id_disk.path, (unsigned long long)lockspace.host_id_disk.offset, ca->header.cmd_flags); while (1) { result = inq_lockspace(&lockspace); if ((result != -EINPROGRESS) || !(waitrs)) { break; } sleep(1); } reply: log_cmd(cmd, "cmd_inq_lockspace %d,%d done %d", ca->ci_in, fd, result); send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); } /* * TODO: rem_lockspace works like a renewal failure would, and abandons * resource leases (tokens) without releasing them. Unlike the renewal * failure case, rem_lockspace most likely releases the host_id. * * What might be nice is an option where rem_lockspace would try to * release resource leases before releasing the lockspace host_id. * (We don't really want to be releasing tokens after we've released * our host_id for the token's lockspace.) * * - kill all pids (by looking at struct resource pid?) * - wait for all pids to exit * o have us or other thread release their tokens/resources * o wait for tokens/resources to be released, although the release * may fail or time out, we don't want to wait too long * - set sp->external_remove * - main_loop sets sp->thread_stop (should find no pids) * - main_loop unlinks watchdog * - lockspace_thread releases host_id * * The aim is that we kill pids and wait for resources to be released * before main_loop gets involved and before the lockspace_thread is * told to stop. * * An alternative messy is to add another condition to the current * main_loop checks: * * if (sp->killing_pids && all_pids_dead(sp) && all_tokens_released(sp)) { * sp->thread_stop = 1; * deactivate_watchdog(sp); * list_move(spaces_rem); * } * * all_tokens_released would just return 1 in case we're not doing * the releases * * release_token_async would need to learn to put the resources onto * dispose list in this case * * consider using the resources/dispose_resources list for all_pids_dead * and kill_pids? instead of the clients[].tokens[] loops? actually, * could we remove tokens and cl->tokens altogether and just use the * resources list? */ static void cmd_rem_lockspace(struct cmd_args *ca, uint32_t cmd) { struct sanlk_lockspace lockspace; int async = ca->header.cmd_flags & SANLK_REM_ASYNC; int fd, rv, result; unsigned int space_id; fd = client[ca->ci_in].fd; rv = recv_loop(fd, &lockspace, sizeof(struct sanlk_lockspace), MSG_WAITALL); if (rv != sizeof(struct sanlk_lockspace)) { log_error("cmd_rem_lockspace %d,%d recv %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } rv = validate_lockspace_path(&lockspace); if (rv < 0) { result = rv; goto reply; } log_cmd(cmd, "cmd_rem_lockspace %d,%d %.48s cmd_flags %x", ca->ci_in, fd, lockspace.name, ca->header.cmd_flags); if (ca->header.cmd_flags & SANLK_REM_UNUSED) { if (lockspace_is_used(&lockspace)) { result = -EBUSY; goto reply; } } rv = rem_lockspace_start(&lockspace, &space_id); if (rv < 0) { result = rv; goto reply; } if (async) { result = rv; log_cmd(cmd, "cmd_rem_lockspace %d,%d async done %d", ca->ci_in, fd, result); send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); rem_lockspace_wait(&lockspace, space_id); return; } result = rem_lockspace_wait(&lockspace, space_id); reply: log_cmd(cmd, "cmd_rem_lockspace %d,%d done %d", ca->ci_in, fd, result); send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); } static void cmd_align(struct task *task GNUC_UNUSED, struct cmd_args *ca, uint32_t cmd) { struct sanlk_disk disk; struct sync_disk sd; int fd, rv, result; fd = client[ca->ci_in].fd; rv = recv_loop(fd, &disk, sizeof(struct sanlk_disk), MSG_WAITALL); if (rv != sizeof(struct sanlk_disk)) { log_error("cmd_align %d,%d recv %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } log_cmd(cmd, "cmd_align %d,%d", ca->ci_in, fd); if (!disk.path[0] || !memchr(disk.path, '\0', SANLK_PATH_LEN)) { result = -ENODEV; goto reply; } memset(&sd, 0, sizeof(struct sync_disk)); memcpy(&sd, &disk, sizeof(struct sanlk_disk)); sd.fd = -1; rv = open_disk(&sd); if (rv < 0) { result = -ENODEV; goto reply; } result = direct_align(&sd); close_disks(&sd, 1); reply: log_cmd(cmd, "cmd_align %d,%d done %d", ca->ci_in, fd, result); send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); } static void cmd_read_lockspace(struct task *task, struct cmd_args *ca, uint32_t cmd) { struct sm_header h; struct sanlk_lockspace lockspace; struct sanlk_host host = { 0 }; struct sync_disk sd; uint64_t host_id; int with_host = (cmd == SM_CMD_READ_LOCKSPACE_HOST); int sector_size = 0; int align_size = 0; int is_caw = 0; int fd, rv, result; fd = client[ca->ci_in].fd; rv = recv_loop(fd, &lockspace, sizeof(struct sanlk_lockspace), MSG_WAITALL); if (rv != sizeof(struct sanlk_lockspace)) { log_error("cmd_read_lockspace %d,%d recv %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } rv = validate_lockspace_path(&lockspace); if (rv < 0) { result = rv; goto reply; } if (!lockspace.host_id) host_id = 1; else host_id = lockspace.host_id; log_cmd(cmd, "cmd_read_lockspace%s %d,%d host_id %llu %s:%llu", with_host ? "_host" : "", ca->ci_in, fd, (unsigned long long)host_id, lockspace.host_id_disk.path, (unsigned long long)lockspace.host_id_disk.offset); memset(&sd, 0, sizeof(struct sync_disk)); memcpy(&sd, &lockspace.host_id_disk, sizeof(struct sanlk_disk)); sd.fd = -1; rv = open_disk(&sd); if (rv < 0) { result = -ENODEV; goto reply; } sector_size = sanlk_lsf_sector_flag_to_size(lockspace.flags); align_size = sanlk_lsf_align_flag_to_size(lockspace.flags); if (!sector_size) { result = delta_read_lockspace_sizes(task, &sd, host_id, task->use_aio, com.io_timeout, §or_size, &align_size, &is_caw); if (result < 0) { log_cmd(cmd, "cmd_read_lockspace%s %d,%d host_id %llu delta_read_lockspace_sizes error %d", with_host ? "_host" : "", ca->ci_in, fd, (unsigned long long)host_id, result); goto out_close; } if ((sector_size != 512) && (sector_size != 4096)) { log_cmd(cmd, "cmd_read_lockspace%s %d,%d host_id %llu invalid sector size %d", with_host ? "_host" : "", ca->ci_in, fd, (unsigned long long)host_id, sector_size); result = -EINVAL; goto out_close; } } result = delta_read_lockspace(task, &sd, sector_size, host_id, &lockspace, com.io_timeout, &host); if (result == SANLK_OK) { result = 0; } else { log_cmd(cmd, "cmd_read_lockspace%s %d,%d host_id %llu delta_read_lockspace error %d", with_host ? "_host" : "", ca->ci_in, fd, (unsigned long long)host_id, result); } out_close: close_disks(&sd, 1); reply: if (with_host) log_cmd(cmd, "cmd_read_lockspace_host %d,%d done %d %llu %llu %llu", ca->ci_in, fd, result, (unsigned long long)host.host_id, (unsigned long long)host.generation, (unsigned long long)host.timestamp); else log_cmd(cmd, "cmd_read_lockspace %d,%d done %d", ca->ci_in, fd, result); memcpy(&h, &ca->header, sizeof(struct sm_header)); h.version = SM_PROTO; h.data = result; h.data2 = host.io_timeout; h.length = sizeof(h) + sizeof(lockspace); if (with_host) h.length += sizeof(host); send_all(fd, &h, sizeof(h), MSG_NOSIGNAL); send_all(fd, &lockspace, sizeof(lockspace), MSG_NOSIGNAL); if (with_host) send_all(fd, &host, sizeof(host), MSG_NOSIGNAL); client_resume(ca->ci_in); } static void cmd_read_resource(struct task *task, struct cmd_args *ca, uint32_t cmd) { struct sm_header h; struct sanlk_resource res; struct token *token = NULL; int token_len, disks_len; int j, fd, rv, result; fd = client[ca->ci_in].fd; /* receiving and setting up token copied from cmd_acquire */ rv = recv_loop(fd, &res, sizeof(struct sanlk_resource), MSG_WAITALL); if (rv != sizeof(struct sanlk_resource)) { log_error("cmd_read_resource %d,%d recv %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } if (!res.num_disks || res.num_disks > SANLK_MAX_DISKS) { result = -ERANGE; goto reply; } disks_len = res.num_disks * sizeof(struct sync_disk); token_len = sizeof(struct token) + disks_len; token = malloc(token_len); if (!token) { result = -ENOMEM; goto reply; } memset(token, 0, token_len); token->disks = (struct sync_disk *)&token->r.disks[0]; /* shorthand */ token->r.num_disks = res.num_disks; memcpy(token->r.lockspace_name, res.lockspace_name, SANLK_NAME_LEN); memcpy(token->r.name, res.name, SANLK_NAME_LEN); /* * receive sanlk_disk's / sync_disk's * * WARNING: as a shortcut, this requires that sync_disk and * sanlk_disk match; this is the reason for the pad fields * in sanlk_disk (TODO: let these differ?) */ rv = recv_loop(fd, token->disks, disks_len, MSG_WAITALL); if (rv != disks_len) { result = -ENOTCONN; goto reply; } for (j = 0; j < token->r.num_disks; j++) { /* zero out pad1 and pad2, see WARNING above */ token->disks[j].sector_size = 0; token->disks[j].fd = -1; if (!token->disks[j].path[0] || !memchr(token->disks[j].path, '\0', SANLK_PATH_LEN)) { result = -ENODEV; goto reply; } } log_cmd(cmd, "cmd_read_resource %d,%d %.256s:%llu", ca->ci_in, fd, token->disks[0].path, (unsigned long long)token->r.disks[0].offset); rv = open_disks(token->disks, token->r.num_disks); if (rv < 0) { result = rv; goto reply; } token->io_timeout = com.io_timeout; /* * These may be zero, in which case paxos_read_resource reads a 4K sector * and gets the values from the leader record. */ token->sector_size = sanlk_res_sector_flag_to_size(res.flags); token->align_size = sanlk_res_align_flag_to_size(res.flags); /* sets res.lockspace_name, res.name, res.lver, res.flags */ result = paxos_read_resource(task, token, &res); if (result == SANLK_OK) result = 0; close_disks(token->disks, token->r.num_disks); reply: if (token) free_token(token); log_cmd(cmd, "cmd_read_resource %d,%d done %d", ca->ci_in, fd, result); memcpy(&h, &ca->header, sizeof(struct sm_header)); h.version = SM_PROTO; h.data = result; h.data2 = 0; h.length = sizeof(h) + sizeof(res); send_all(fd, &h, sizeof(h), MSG_NOSIGNAL); send_all(fd, &res, sizeof(res), MSG_NOSIGNAL); client_resume(ca->ci_in); } static void cmd_read_resource_owners(struct task *task, struct cmd_args *ca, uint32_t cmd) { struct sm_header h; struct sanlk_resource res; struct token *token = NULL; char *send_buf; int token_len, disks_len, send_len = 0; int j, fd, rv, result, count = 0; fd = client[ca->ci_in].fd; /* receiving and setting up token copied from cmd_acquire */ rv = recv_loop(fd, &res, sizeof(struct sanlk_resource), MSG_WAITALL); if (rv != sizeof(struct sanlk_resource)) { log_error("cmd_read_resource_owners %d,%d recv %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } if (!res.num_disks || res.num_disks > SANLK_MAX_DISKS) { result = -ERANGE; goto reply; } disks_len = res.num_disks * sizeof(struct sync_disk); token_len = sizeof(struct token) + disks_len; token = malloc(token_len); if (!token) { result = -ENOMEM; goto reply; } memset(token, 0, token_len); token->disks = (struct sync_disk *)&token->r.disks[0]; /* shorthand */ token->r.num_disks = res.num_disks; memcpy(token->r.lockspace_name, res.lockspace_name, SANLK_NAME_LEN); memcpy(token->r.name, res.name, SANLK_NAME_LEN); /* * receive sanlk_disk's / sync_disk's * * WARNING: as a shortcut, this requires that sync_disk and * sanlk_disk match; this is the reason for the pad fields * in sanlk_disk (TODO: let these differ?) */ rv = recv_loop(fd, token->disks, disks_len, MSG_WAITALL); if (rv != disks_len) { result = -ENOTCONN; goto reply; } for (j = 0; j < token->r.num_disks; j++) { /* zero out pad1 and pad2, see WARNING above */ token->disks[j].sector_size = 0; token->disks[j].fd = -1; if (!token->disks[j].path[0] || !memchr(token->disks[j].path, '\0', SANLK_PATH_LEN)) { result = -ENODEV; goto reply; } } log_cmd(cmd, "cmd_read_resource_owners %d,%d %.256s:%llu", ca->ci_in, fd, token->disks[0].path, (unsigned long long)token->r.disks[0].offset); rv = open_disks(token->disks, token->r.num_disks); if (rv < 0) { result = rv; goto reply; } token->io_timeout = com.io_timeout; /* * These may be zero, in which case paxos_read_resource reads a 4K sector * and gets the values from the leader record. */ token->sector_size = sanlk_res_sector_flag_to_size(res.flags); token->align_size = sanlk_res_align_flag_to_size(res.flags); send_buf = NULL; send_len = 0; result = read_resource_owners(task, token, &res, &send_buf, &send_len, &count); if (result == SANLK_OK) result = 0; close_disks(token->disks, token->r.num_disks); reply: if (token) free_token(token); log_cmd(cmd, "cmd_read_resource_owners %d,%d count %d done %d", ca->ci_in, fd, count, result); memcpy(&h, &ca->header, sizeof(struct sm_header)); h.version = SM_PROTO; h.data = result; h.data2 = count; h.length = sizeof(h) + sizeof(res) + send_len; send_all(fd, &h, sizeof(h), MSG_NOSIGNAL); send_all(fd, &res, sizeof(res), MSG_NOSIGNAL); if (send_len && send_buf) { send_all(fd, send_buf, send_len, MSG_NOSIGNAL); free(send_buf); } client_resume(ca->ci_in); } static void cmd_init_lockspace_host(struct task *task, struct cmd_args *ca, uint32_t cmd) { struct sanlk_lockspace lockspace; struct leader_record leader; struct sync_disk sd; uint32_t cmd_flags; int cmd_no_caw; int cmd_use_caw; int try_caw = 1; int fd, rv, result; int io_timeout = com.io_timeout; fd = client[ca->ci_in].fd; rv = recv_loop(fd, &lockspace, sizeof(struct sanlk_lockspace), MSG_WAITALL); if (rv != sizeof(struct sanlk_lockspace)) { log_error("cmd_init_lockspace_host %d,%d recv lockspace %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } rv = recv_loop(fd, &leader, sizeof(struct leader_record), MSG_WAITALL); if (rv != sizeof(struct leader_record)) { log_error("cmd_init_lockspace_host %d,%d recv leader %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } rv = validate_lockspace_path(&lockspace); if (rv < 0) { result = rv; goto reply; } log_cmd(cmd, "cmd_init_lockspace_host %d,%d %.48s:%llu:%s:%llu ls_flags 0x%x cmd_flags 0x%x", ca->ci_in, fd, lockspace.name, (unsigned long long)lockspace.host_id, lockspace.host_id_disk.path, (unsigned long long)lockspace.host_id_disk.offset, lockspace.flags, ca->header.cmd_flags); if (!lockspace.host_id) { result = -EINVAL; goto reply; } memset(&sd, 0, sizeof(struct sync_disk)); memcpy(&sd, &lockspace.host_id_disk, sizeof(struct sanlk_disk)); sd.fd = -1; rv = open_disk(&sd); if (rv < 0) { result = -ENODEV; goto reply; } if (ca->header.data2) io_timeout = ca->header.data2; cmd_flags = ca->header.cmd_flags; cmd_no_caw = (cmd_flags & SANLK_INIT_HOST_NO_CAW) ? 1 : 0; cmd_use_caw = (cmd_flags & SANLK_INIT_HOST_USE_CAW) ? 1 : 0; if (cmd_no_caw) try_caw = 0; else if (com.use_compare_and_write == USE_CAW_NO) try_caw = 0; else if ((com.use_compare_and_write == USE_CAW_ALLOW) && !cmd_use_caw) try_caw = 0; else if (com.create_delta_34 || com.create_delta_40) try_caw = 0; result = delta_lease_init_host(task, &lockspace, leader.resource_name[0] ? leader.resource_name : NULL, leader.owner_generation, leader.timestamp, io_timeout, try_caw, &sd); if (result == SANLK_OK) result = 0; close_disks(&sd, 1); reply: log_cmd(cmd, "cmd_init_lockspace_host %d,%d done %d", ca->ci_in, fd, result); send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); } static void cmd_write_lockspace(struct task *task, struct cmd_args *ca, uint32_t cmd) { struct sanlk_lockspace lockspace; struct sync_disk sd; uint32_t cmd_flags; int cmd_no_caw; int cmd_use_caw; int try_caw; int fd, rv, result; int io_timeout = com.io_timeout; fd = client[ca->ci_in].fd; rv = recv_loop(fd, &lockspace, sizeof(struct sanlk_lockspace), MSG_WAITALL); if (rv != sizeof(struct sanlk_lockspace)) { log_error("cmd_write_lockspace %d,%d recv %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } rv = validate_lockspace_path(&lockspace); if (rv < 0) { result = rv; goto reply; } log_cmd(cmd, "cmd_write_lockspace %d,%d %.48s:%llu:%s:%llu ls_flags 0x%x cmd_flags 0x%x", ca->ci_in, fd, lockspace.name, (unsigned long long)lockspace.host_id, lockspace.host_id_disk.path, (unsigned long long)lockspace.host_id_disk.offset, lockspace.flags, ca->header.cmd_flags); /* No longer used, max_hosts is derived from sector/align sizes. */ /* max_hosts = ca->header.data; */ memset(&sd, 0, sizeof(struct sync_disk)); memcpy(&sd, &lockspace.host_id_disk, sizeof(struct sanlk_disk)); sd.fd = -1; rv = open_disk(&sd); if (rv < 0) { result = -ENODEV; goto reply; } if (ca->header.data2) io_timeout = ca->header.data2; cmd_flags = ca->header.cmd_flags; cmd_no_caw = (cmd_flags & SANLK_WRITE_LS_NO_CAW) ? 1 : 0; cmd_use_caw = (cmd_flags & SANLK_WRITE_LS_USE_CAW) ? 1 : 0; cmd_flags &= ~SANLK_WRITE_LS_NO_CAW; cmd_flags &= ~SANLK_WRITE_LS_USE_CAW; if (cmd_flags & SANLK_WRITE_LS_FLAGS) { result = delta_lease_update_lockspace(task, &lockspace, io_timeout, &sd); } else if (!cmd_flags) { try_caw = 1; if (cmd_no_caw) try_caw = 0; else if (com.use_compare_and_write == USE_CAW_NO) try_caw = 0; else if ((com.use_compare_and_write == USE_CAW_ALLOW) && !cmd_use_caw) try_caw = 0; else if (com.create_delta_34 || com.create_delta_40) try_caw = 0; result = delta_lease_init(task, &lockspace, io_timeout, try_caw, &sd); } else { log_error("cmd_write_lockspace %d,%d unknown flags 0x%x", ca->ci_in, fd, ca->header.cmd_flags); result = -EINVAL; } close_disks(&sd, 1); reply: log_cmd(cmd, "cmd_write_lockspace %d,%d done %d", ca->ci_in, fd, result); send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); } static void cmd_write_resource(struct task *task, struct cmd_args *ca, uint32_t cmd) { struct token *token = NULL; struct sanlk_resource res; int token_len, disks_len; int num_hosts; int write_clear = 0; int try_caw = 1; int j, fd, rv, result; fd = client[ca->ci_in].fd; /* receiving and setting up token copied from cmd_acquire */ rv = recv_loop(fd, &res, sizeof(struct sanlk_resource), MSG_WAITALL); if (rv != sizeof(struct sanlk_resource)) { log_error("cmd_write_resource %d,%d recv %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } if (!res.num_disks || res.num_disks > SANLK_MAX_DISKS) { result = -ERANGE; goto reply; } disks_len = res.num_disks * sizeof(struct sync_disk); token_len = sizeof(struct token) + disks_len; token = malloc(token_len); if (!token) { result = -ENOMEM; goto reply; } memset(token, 0, token_len); token->disks = (struct sync_disk *)&token->r.disks[0]; /* shorthand */ token->r.num_disks = res.num_disks; memcpy(token->r.lockspace_name, res.lockspace_name, SANLK_NAME_LEN); memcpy(token->r.name, res.name, SANLK_NAME_LEN); token->r.flags = res.flags; /* * receive sanlk_disk's / sync_disk's * * WARNING: as a shortcut, this requires that sync_disk and * sanlk_disk match; this is the reason for the pad fields * in sanlk_disk (TODO: let these differ?) */ rv = recv_loop(fd, token->disks, disks_len, MSG_WAITALL); if (rv != disks_len) { result = -ENOTCONN; goto reply; } for (j = 0; j < token->r.num_disks; j++) { /* zero out pad1 and pad2, see WARNING above */ token->disks[j].sector_size = 0; token->disks[j].fd = -1; if (!token->disks[j].path[0] || !memchr(token->disks[j].path, '\0', SANLK_PATH_LEN)) { result = -ENODEV; goto reply; } } log_cmd(cmd, "cmd_write_resource %d,%d %.48s:%.48s:%.256s:%llu res_flags 0x%x", ca->ci_in, fd, token->r.lockspace_name, token->r.name, token->disks[0].path, (unsigned long long)token->r.disks[0].offset, res.flags); num_hosts = ca->header.data; /* No longer used, max_hosts is derived from sector/align sizes. */ /* max_hosts = ca->header.data2; */ if (ca->header.cmd_flags & SANLK_WRITE_NO_CAW) try_caw = 0; else if (com.use_compare_and_write == USE_CAW_NO) try_caw = 0; else if ((com.use_compare_and_write == USE_CAW_ALLOW) && !(ca->header.cmd_flags & SANLK_WRITE_USE_CAW)) try_caw = 0; else if (com.create_paxos_64) try_caw = 0; if (ca->header.cmd_flags & SANLK_WRITE_CLEAR) write_clear = 1; rv = open_disks(token->disks, token->r.num_disks); if (rv < 0) { result = rv; goto reply; } token->io_timeout = com.io_timeout; result = paxos_lease_init(task, token, num_hosts, write_clear, try_caw); close_disks(token->disks, token->r.num_disks); reply: if (token) free_token(token); send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); } /* N.B. the api doesn't support one client setting killpath for another pid/client */ static void cmd_killpath(struct task *task, struct cmd_args *ca, uint32_t cmd) { struct client *cl; int cl_ci = ca->ci_target; int cl_fd = ca->cl_fd; int cl_pid = ca->cl_pid; int rv, result, pid_dead; cl = &client[cl_ci]; log_cmd(cmd, "cmd_killpath %d,%d,%d cmd_flags 0x%x", cl_ci, cl_fd, cl_pid, ca->header.cmd_flags); rv = recv_loop(cl_fd, cl->killpath, SANLK_HELPER_PATH_LEN, MSG_WAITALL); if (rv != SANLK_HELPER_PATH_LEN) { log_error("cmd_killpath %d,%d,%d recv path %d %d", cl_ci, cl_fd, cl_pid, rv, errno); memset(cl->killpath, 0, SANLK_HELPER_PATH_LEN); memset(cl->killargs, 0, SANLK_HELPER_ARGS_LEN); result = -ENOTCONN; goto done; } rv = recv_loop(cl_fd, cl->killargs, SANLK_HELPER_ARGS_LEN, MSG_WAITALL); if (rv != SANLK_HELPER_ARGS_LEN) { log_error("cmd_killpath %d,%d,%d recv args %d %d", cl_ci, cl_fd, cl_pid, rv, errno); memset(cl->killpath, 0, SANLK_HELPER_PATH_LEN); memset(cl->killargs, 0, SANLK_HELPER_ARGS_LEN); result = -ENOTCONN; goto done; } cl->killpath[SANLK_HELPER_PATH_LEN - 1] = '\0'; cl->killargs[SANLK_HELPER_ARGS_LEN - 1] = '\0'; if (ca->header.cmd_flags & SANLK_KILLPATH_PID) cl->flags |= CL_KILLPATH_PID; result = 0; done: pthread_mutex_lock(&cl->mutex); pid_dead = cl->pid_dead; cl->cmd_active = 0; pthread_mutex_unlock(&cl->mutex); if (pid_dead) { /* release tokens in case a client sets/changes its killpath after it has acquired leases */ release_cl_tokens(task, cl); client_free(cl_ci); return; } send_result(ca->ci_in, cl_fd, &ca->header, result); client_resume(ca->ci_in); } static void cmd_set_event(struct task *task GNUC_UNUSED, struct cmd_args *ca, uint32_t cmd) { struct sanlk_lockspace lockspace; struct sanlk_host_event he; int rv, fd, result; fd = client[ca->ci_in].fd; rv = recv_loop(fd, &lockspace, sizeof(struct sanlk_lockspace), MSG_WAITALL); if (rv != sizeof(struct sanlk_lockspace)) { result = -ENOTCONN; goto reply; } rv = recv_loop(fd, &he, sizeof(struct sanlk_host_event), MSG_WAITALL); if (rv != sizeof(struct sanlk_host_event)) { result = -ENOTCONN; goto reply; } log_cmd(cmd, "cmd_set_event %.48s", lockspace.name); result = lockspace_set_event(&lockspace, &he, ca->header.cmd_flags); log_cmd(cmd, "cmd_set_event result %d", result); reply: send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); } #if 0 static void cmd_format_rindex(struct task *task, struct cmd_args *ca, uint32_t cmd) { struct sanlk_rindex ri; int fd, rv, result; fd = client[ca->ci_in].fd; rv = recv_loop(fd, &ri, sizeof(struct sanlk_rindex), MSG_WAITALL); if (rv != sizeof(struct sanlk_rindex)) { log_error("cmd_format_rindex %d,%d recv %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } log_cmd(cmd, "cmd_format_rindex %d,%d %.48s %s:%llu", ca->ci_in, fd, ri.lockspace_name, ri.disk.path, (unsigned long long)ri.disk.offset); result = rindex_format(task, &ri); reply: log_cmd(cmd, "cmd_format_rindex %d,%d done %d", ca->ci_in, fd, result); send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); } static void cmd_rebuild_rindex(struct task *task, struct cmd_args *ca, uint32_t cmd) { struct sanlk_rindex ri; int fd, rv, result; fd = client[ca->ci_in].fd; rv = recv_loop(fd, &ri, sizeof(struct sanlk_rindex), MSG_WAITALL); if (rv != sizeof(struct sanlk_rindex)) { log_error("cmd_rebuild_rindex %d,%d recv %d %d", ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } log_cmd(cmd, "cmd_rebuild_rindex %d,%d %.48s %s:%llu", ca->ci_in, fd, ri.lockspace_name, ri.disk.path, (unsigned long long)ri.disk.offset); result = rindex_rebuild(task, &ri, ca->header.cmd_flags); reply: log_cmd(cmd, "cmd_rebuild_rindex %d,%d done %d", ca->ci_in, fd, result); send_result(ca->ci_in, fd, &ca->header, result); client_resume(ca->ci_in); } static void rindex_op(struct task *task, struct cmd_args *ca, const char *ri_cmd_str, int op, uint32_t cmd) { struct sanlk_rindex ri; struct sanlk_rentry re; struct sanlk_rentry re_ret; struct sm_header h; int fd, rv, result; memset(&re_ret, 0, sizeof(re_ret)); fd = client[ca->ci_in].fd; rv = recv_loop(fd, &ri, sizeof(struct sanlk_rindex), MSG_WAITALL); if (rv != sizeof(struct sanlk_rindex)) { log_error("%s %d,%d recv %d %d", ri_cmd_str, ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } rv = recv_loop(fd, &re, sizeof(struct sanlk_rentry), MSG_WAITALL); if (rv != sizeof(struct sanlk_rentry)) { log_error("%s %d,%d recv %d %d", ri_cmd_str, ca->ci_in, fd, rv, errno); result = -ENOTCONN; goto reply; } log_cmd(cmd, "%s %d,%d %.48s %s:%llu", ri_cmd_str, ca->ci_in, fd, ri.lockspace_name, ri.disk.path, (unsigned long long)ri.disk.offset); if (op == RX_OP_LOOKUP) result = rindex_lookup(task, &ri, &re, &re_ret, ca->header.cmd_flags); else if (op == RX_OP_UPDATE) result = rindex_update(task, &ri, &re, &re_ret, ca->header.cmd_flags); else if (op == RX_OP_CREATE) result = rindex_create(task, &ri, &re, &re_ret, ca->header.data, ca->header.data2); else if (op == RX_OP_DELETE) result = rindex_delete(task, &ri, &re, &re_ret); else result = -EINVAL; reply: log_cmd(cmd, "%s %d,%d done %d", ri_cmd_str, ca->ci_in, fd, result); memcpy(&h, &ca->header, sizeof(struct sm_header)); h.version = SM_PROTO; h.data = result; h.data2 = 0; h.length = sizeof(h) + sizeof(re_ret); send_all(fd, &h, sizeof(h), MSG_NOSIGNAL); send_all(fd, &re_ret, sizeof(re), MSG_NOSIGNAL); client_resume(ca->ci_in); } #endif void call_cmd_thread(struct task *task, struct cmd_args *ca) { uint32_t cmd = ca->header.cmd; switch (cmd) { case SM_CMD_ACQUIRE: case SM_CMD_ACQUIRE2: cmd_acquire(task, ca, cmd); break; case SM_CMD_RELEASE: cmd_release(task, ca, cmd); break; case SM_CMD_INQUIRE: cmd_inquire(task, ca, cmd); break; case SM_CMD_CONVERT: cmd_convert(task, ca, cmd); break; case SM_CMD_REQUEST: cmd_request(task, ca, cmd); break; case SM_CMD_ADD_LOCKSPACE: strcpy(client[ca->ci_in].owner_name, "add_lockspace"); cmd_add_lockspace(ca, cmd); break; case SM_CMD_INQ_LOCKSPACE: strcpy(client[ca->ci_in].owner_name, "inq_lockspace"); cmd_inq_lockspace(ca, cmd); break; case SM_CMD_REM_LOCKSPACE: strcpy(client[ca->ci_in].owner_name, "rem_lockspace"); cmd_rem_lockspace(ca, cmd); break; case SM_CMD_ALIGN: cmd_align(task, ca, cmd); break; case SM_CMD_INIT_LOCKSPACE_HOST: cmd_init_lockspace_host(task, ca, cmd); break; case SM_CMD_WRITE_LOCKSPACE: cmd_write_lockspace(task, ca, cmd); break; case SM_CMD_WRITE_RESOURCE: cmd_write_resource(task, ca, cmd); break; case SM_CMD_READ_LOCKSPACE: case SM_CMD_READ_LOCKSPACE_HOST: cmd_read_lockspace(task, ca, cmd); break; case SM_CMD_READ_RESOURCE: cmd_read_resource(task, ca, cmd); break; case SM_CMD_READ_RESOURCE_OWNERS: cmd_read_resource_owners(task, ca, cmd); break; case SM_CMD_EXAMINE_LOCKSPACE: case SM_CMD_EXAMINE_RESOURCE: cmd_examine(task, ca, cmd); break; case SM_CMD_KILLPATH: cmd_killpath(task, ca, cmd); break; case SM_CMD_SET_HOST: cmd_set_host(task, ca, cmd); break; case SM_CMD_SET_LVB: cmd_set_lvb(task, ca, cmd); break; case SM_CMD_GET_LVB: cmd_get_lvb(task, ca, cmd); break; case SM_CMD_SHUTDOWN_WAIT: cmd_shutdown_wait(task, ca, cmd); break; case SM_CMD_SET_EVENT: cmd_set_event(task, ca, cmd); break; #if 0 case SM_CMD_FORMAT_RINDEX: cmd_format_rindex(task, ca, cmd); break; case SM_CMD_REBUILD_RINDEX: cmd_rebuild_rindex(task, ca, cmd); break; case SM_CMD_UPDATE_RINDEX: rindex_op(task, ca, "cmd_update_rindex", RX_OP_UPDATE, cmd); break; case SM_CMD_LOOKUP_RINDEX: rindex_op(task, ca, "cmd_lookup_rindex", RX_OP_LOOKUP, cmd); break; case SM_CMD_CREATE_RESOURCE: rindex_op(task, ca, "cmd_create_resource", RX_OP_CREATE, cmd); break; case SM_CMD_DELETE_RESOURCE: rindex_op(task, ca, "cmd_delete_resource", RX_OP_DELETE, cmd); break; #endif }; } static const char *use_compare_and_write_str(int use_caw) { switch (use_caw) { case USE_CAW_YES: return "yes"; case USE_CAW_NO: return "no"; case USE_CAW_ALLOW: return "allow"; }; return "unknown"; } /* * sanlock client status * * 1. send_state_daemon * * 2. for each cl in clients * send_state_client() [sanlk_state + str_len] * * 3. for each sp in spaces, spaces_add, spaces_rem * send_state_lockspace() [sanlk_state + str_len + sanlk_lockspace] * * 4. for each r in resources, dispose_resources * send_state_resource() [sanlk_state + str_len + sanlk_resource + sanlk_disk * num_disks] * * sanlock client host_status * * 1. for each hs in sp->host_status * send_state_host() */ static int print_state_daemon(char *str) { memset(str, 0, SANLK_STATE_MAXSTR); snprintf(str, SANLK_STATE_MAXSTR-1, "our_host_name=%s " "use_watchdog=%d " "high_priority=%d " "mlock_level=%d " "quiet_fail=%d " "debug_renew=%d " "debug_hosts=%d " "debug_clients=%d " "debug_cmds=0x%llx " "renewal_history_size=%d " "gid=%d " "uid=%d " "sh_retries=%d " "max_sectors_kb_ignore=%d " "max_sectors_kb_align=%d " "max_sectors_kb_num=%d " "max_worker_threads=%d " "num_worker_threads=%d " "use_hugepages=%s " "write_init_io_timeout=%u " "use_aio=%d " "io_timeout=%d " "use_compare_and_write=%s " "watchdog_fire_timeout=%d " "kill_grace_seconds=%d " "helper_pid=%d " "helper_kill_fd=%d " "helper_full_count=%u " "helper_last_status=%llu " "monotime=%llu " "version_str=%s " "version_num=%u.%u.%u " "version_hex=%08x " "smproto_hex=%08x " "create_delta_34=%d " "create_delta_40=%d " "create_paxos_64=%d", our_host_name_global, com.use_watchdog, com.high_priority, com.mlock_level, com.quiet_fail, com.debug_renew, com.debug_hosts, com.debug_clients, (unsigned long long)com.debug_cmds, com.renewal_history_size, com.gid, com.uid, com.sh_retries, com.max_sectors_kb_ignore, com.max_sectors_kb_align, com.max_sectors_kb_num, com.max_worker_threads, daemon_status_num_workers, com.use_hugepages ? "all" : "none", com.write_init_io_timeout, main_task.use_aio, com.io_timeout, use_compare_and_write_str(com.use_compare_and_write), com.watchdog_fire_timeout, com.kill_grace_seconds, helper_pid, helper_kill_fd, helper_full_count, (unsigned long long)helper_last_status, (unsigned long long)monotime(), VERSION, sanlock_version_major, sanlock_version_minor, sanlock_version_patch, sanlock_version_combined, SM_PROTO, com.create_delta_34, com.create_delta_40, com.create_paxos_64); return strlen(str) + 1; } static int print_state_client(struct client *cl, int ci, char *str) { memset(str, 0, SANLK_STATE_MAXSTR); snprintf(str, SANLK_STATE_MAXSTR-1, "ci=%d " "fd=%d " "pid=%d " "flags=%x " "restricted=%x " "cmd_active=%d " "cmd_last=%d " "pid_dead=%d " "kill_count=%d " "kill_last=%llu " "suspend=%d " "need_free=%d", ci, cl->fd, cl->pid, cl->flags, cl->restricted, cl->cmd_active, cl->cmd_last, cl->pid_dead, cl->kill_count, (unsigned long long)cl->kill_last, cl->suspend, cl->need_free); return strlen(str) + 1; } static int print_state_lockspace(struct space *sp, char *str, const char *list_name) { memset(str, 0, SANLK_STATE_MAXSTR); snprintf(str, SANLK_STATE_MAXSTR-1, "list=%s " "space_id=%u " "version=0x%08x " "io_timeout=%d " "sector_size=%d " "align_size=%d " "host_generation=%llu " "renew_fail=%d " "space_dead=%d " "killing_pids=%d " "used_retries=%u " "external_used=%d " "used_by_orphans=%d " "no_timeout=%d " "caw=%d " "caw_dev=%u:%u " "caw_off=%llu " "renewal_read_extend_sec=%u " "set_max_sectors_kb=%u " "corrupt_result=%d " "acquire_last_result=%d " "renewal_last_result=%d " "acquire_last_attempt=%llu " "acquire_last_success=%llu " "renewal_last_attempt=%llu " "renewal_last_success=%llu", list_name, sp->space_id, sp->version, sp->io_timeout, sp->sector_size, sp->align_size, (unsigned long long)sp->host_generation, sp->renew_fail, sp->space_dead, sp->killing_pids, sp->used_retries, (sp->flags & SP_EXTERNAL_USED) ? 1 : 0, (sp->flags & SP_USED_BY_ORPHANS) ? 1 : 0, sp->no_timeout, sp->caw, sp->caw_dev_major, sp->caw_dev_minor, (unsigned long long)sp->caw_dev_offset, sp->renewal_read_extend_sec, sp->set_max_sectors_kb, sp->lease_status.corrupt_result, sp->lease_status.acquire_last_result, sp->lease_status.renewal_last_result, (unsigned long long)sp->lease_status.acquire_last_attempt, (unsigned long long)sp->lease_status.acquire_last_success, (unsigned long long)sp->lease_status.renewal_last_attempt, (unsigned long long)sp->lease_status.renewal_last_success); return strlen(str) + 1; } static int print_state_resource(struct resource *r, char *str, const char *list_name, int client_id, uint32_t token_id) { memset(str, 0, SANLK_STATE_MAXSTR); snprintf(str, SANLK_STATE_MAXSTR-1, "list=%s " "flags=%x " "sector_size=%d " "align_size=%d " "lver=%llu " "reused=%u " "res_id=%u " "caw=%d " "caw_dev=%u:%u " "caw_off=%llu " "token_id=%u " "ci=%d", list_name, r->flags, r->sector_size, r->align_size, (unsigned long long)r->leader.lver, r->reused, r->res_id, (r->flags & R_CAW) ? 1 : 0, r->caw_major, r->caw_minor, (unsigned long long)r->caw_offset, token_id, client_id); return strlen(str) + 1; } static int print_state_host(struct host_status *hs, char *str) { uint32_t flag; memset(str, 0, SANLK_STATE_MAXSTR); flag = get_host_flag(0, hs); snprintf(str, SANLK_STATE_MAXSTR-1, "last_check=%llu " "last_live=%llu " "last_req=%llu " "owner_id=%llu " "owner_generation=%llu " "timestamp=%llu " "io_timeout=%u " "no_timeout=%u " "dead_ext=%u " "found_dead_ext_gen=%llu " "wrote_dead_ext_gen=%llu " "host_flags=0x%x " "host_state=%s " "owner_name=%.48s", (unsigned long long)hs->last_check, (unsigned long long)hs->last_live, (unsigned long long)hs->last_req, (unsigned long long)hs->owner_id, (unsigned long long)hs->owner_generation, (unsigned long long)hs->timestamp, hs->io_timeout, hs->no_timeout, hs->dead_ext, (unsigned long long)hs->found_dead_ext_gen, (unsigned long long)hs->wrote_dead_ext_gen, flag, host_state_str(flag), hs->owner_name); return strlen(str) + 1; } static int print_state_renewal(struct renewal_history *hi, char *str) { memset(str, 0, SANLK_STATE_MAXSTR); snprintf(str, SANLK_STATE_MAXSTR-1, "timestamp=%llu " "read_ms=%d " "write_ms=%d " "next_timeouts=%d " "next_errors=%d", (unsigned long long)hi->timestamp, hi->read_ms, hi->write_ms, hi->next_timeouts, hi->next_errors); return strlen(str) + 1; } static void send_state_daemon(int fd) { struct sanlk_state st; char str[SANLK_STATE_MAXSTR]; int str_len; memset(&st, 0, sizeof(st)); memcpy(st.name, our_host_name_global, NAME_ID_SIZE); st.type = SANLK_STATE_DAEMON; str_len = print_state_daemon(str); st.str_len = str_len; send_all(fd, &st, sizeof(st), MSG_NOSIGNAL); if (str_len) send_all(fd, str, str_len, MSG_NOSIGNAL); } int print_state_rthread(char *str); static void send_state_rthread(int fd) { struct sanlk_state st; char str[SANLK_STATE_MAXSTR]; int str_len; memset(&st, 0, sizeof(st)); st.type = SANLK_STATE_RTHREAD; str_len = print_state_rthread(str); if (!str_len) return; st.str_len = str_len; send_all(fd, &st, sizeof(st), MSG_NOSIGNAL); if (str_len) send_all(fd, str, str_len, MSG_NOSIGNAL); } int print_state_worker(int i, char *str); static void send_state_worker(int fd, int i) { struct sanlk_state st; char str[SANLK_STATE_MAXSTR]; int str_len; memset(&st, 0, sizeof(st)); st.type = SANLK_STATE_WORKER; st.data32 = i; str_len = print_state_worker(i, str); if (!str_len) return; st.str_len = str_len; send_all(fd, &st, sizeof(st), MSG_NOSIGNAL); if (str_len) send_all(fd, str, str_len, MSG_NOSIGNAL); } static void send_state_client(int fd, struct client *cl, int ci) { struct sanlk_state st; char str[SANLK_STATE_MAXSTR]; int str_len; memset(&st, 0, sizeof(st)); st.type = SANLK_STATE_CLIENT; st.data32 = cl->pid; memcpy(st.name, cl->owner_name, NAME_ID_SIZE); str_len = print_state_client(cl, ci, str); st.str_len = str_len; send_all(fd, &st, sizeof(st), MSG_NOSIGNAL); if (str_len) send_all(fd, str, str_len, MSG_NOSIGNAL); } static void send_state_lockspace(int fd, struct space *sp, const char *list_name) { struct sanlk_state st; struct sanlk_lockspace lockspace; char str[SANLK_STATE_MAXSTR]; int str_len; memset(&st, 0, sizeof(st)); st.type = SANLK_STATE_LOCKSPACE; st.data64 = sp->host_id; memcpy(st.name, sp->space_name, NAME_ID_SIZE); str_len = print_state_lockspace(sp, str, list_name); st.str_len = str_len; send_all(fd, &st, sizeof(st), MSG_NOSIGNAL); if (str_len) send_all(fd, str, str_len, MSG_NOSIGNAL); memset(&lockspace, 0, sizeof(struct sanlk_lockspace)); memcpy(lockspace.name, sp->space_name, NAME_ID_SIZE); lockspace.host_id = sp->host_id; memcpy(&lockspace.host_id_disk, &sp->host_id_disk, sizeof(struct sanlk_disk)); send_all(fd, &lockspace, sizeof(lockspace), MSG_NOSIGNAL); } void send_state_resource(int fd, struct resource *r, const char *list_name, int pid, int client_id, uint32_t token_id); void send_state_resource(int fd, struct resource *r, const char *list_name, int pid, int client_id, uint32_t token_id) { struct sanlk_state st; char str[SANLK_STATE_MAXSTR]; int str_len; int i; memset(&st, 0, sizeof(st)); st.type = SANLK_STATE_RESOURCE; st.data32 = pid; st.data64 = r->leader.lver; memcpy(st.name, r->r.name, NAME_ID_SIZE); str_len = print_state_resource(r, str, list_name, client_id, token_id); st.str_len = str_len; send_all(fd, &st, sizeof(st), MSG_NOSIGNAL); if (str_len) send_all(fd, str, str_len, MSG_NOSIGNAL); send_all(fd, &r->r, sizeof(struct sanlk_resource), MSG_NOSIGNAL); for (i = 0; i < r->r.num_disks; i++) { send_all(fd, &r->r.disks[i], sizeof(struct sanlk_disk), MSG_NOSIGNAL); } } static void send_state_host(int fd, struct host_status *hs, int host_id) { struct sanlk_state st; char str[SANLK_STATE_MAXSTR]; int str_len; memset(&st, 0, sizeof(st)); st.type = SANLK_STATE_HOST; st.data32 = host_id; st.data64 = hs->timestamp; str_len = print_state_host(hs, str); st.str_len = str_len; send_all(fd, &st, sizeof(st), MSG_NOSIGNAL); if (str_len) send_all(fd, str, str_len, MSG_NOSIGNAL); } static void send_state_renewal(int fd, struct renewal_history *hi) { struct sanlk_state st; char str[SANLK_STATE_MAXSTR]; int str_len; memset(&st, 0, sizeof(st)); st.type = SANLK_STATE_RENEWAL; st.data64 = hi->timestamp; str_len = print_state_renewal(hi, str); st.str_len = str_len; send_all(fd, &st, sizeof(st), MSG_NOSIGNAL); if (str_len) send_all(fd, str, str_len, MSG_NOSIGNAL); } static void cmd_status(int ci, int fd, struct sm_header *h_recv, int client_maxi, uint32_t cmd) { struct sm_header h; struct client *cl; struct space *sp; int ci_iter, i; log_cmd(cmd, "cmd_status %d,%d data %u", ci, fd, h_recv->data); memset(&h, 0, sizeof(h)); memcpy(&h, h_recv, sizeof(struct sm_header)); h.version = SM_PROTO; h.length = sizeof(h); h.data = 0; send_all(fd, &h, sizeof(h), MSG_NOSIGNAL); send_state_daemon(fd); if (h_recv->data == SANLK_STATE_DAEMON) return; for (i = 0; i < com.max_worker_threads; i++) send_state_worker(fd, i); if (h_recv->data == SANLK_STATE_WORKER) return; send_state_rthread(fd); if (h_recv->data == SANLK_STATE_RTHREAD) return; for (ci_iter = 0; ci_iter <= client_maxi; ci_iter++) { cl = &client[ci_iter]; if (!cl->used) continue; send_state_client(fd, cl, ci_iter); } if (h_recv->data == SANLK_STATE_CLIENT) return; /* N.B. the reporting function looks for the strings "add" and "rem", so if changed, the strings should be changed in both places. */ pthread_mutex_lock(&spaces_mutex); list_for_each_entry(sp, &spaces, list) send_state_lockspace(fd, sp, "spaces"); list_for_each_entry(sp, &spaces_add, list) send_state_lockspace(fd, sp, "add"); list_for_each_entry(sp, &spaces_rem, list) send_state_lockspace(fd, sp, "rem"); pthread_mutex_unlock(&spaces_mutex); if (h_recv->data == SANLK_STATE_LOCKSPACE) return; /* resource.c will iterate through private lists and call back here for each r */ send_state_resources(fd); } static void cmd_host_status(int ci, int fd, struct sm_header *h_recv, uint32_t cmd) { struct sm_header h; struct sanlk_lockspace lockspace; struct space *sp; struct host_status *hs, *status = NULL; int status_len; int i, rv; log_cmd(cmd, "cmd_host_status %d,%d", ci, fd); memset(&h, 0, sizeof(h)); memcpy(&h, h_recv, sizeof(struct sm_header)); h.version = SM_PROTO; h.length = sizeof(h); h.data = 0; status_len = sizeof(struct host_status) * DEFAULT_MAX_HOSTS; status = malloc(status_len); if (!status) { h.data = -ENOMEM; goto fail; } rv = recv_loop(fd, &lockspace, sizeof(struct sanlk_lockspace), MSG_WAITALL); if (rv != sizeof(struct sanlk_lockspace)) { h.data = -ENOTCONN; goto fail; } pthread_mutex_lock(&spaces_mutex); sp = find_lockspace(lockspace.name); if (sp) memcpy(status, &sp->host_status, status_len); pthread_mutex_unlock(&spaces_mutex); if (!sp) { h.data = -ENOSPC; goto fail; } send_all(fd, &h, sizeof(h), MSG_NOSIGNAL); for (i = 0; i < DEFAULT_MAX_HOSTS; i++) { hs = &status[i]; if (!hs->last_live && !hs->owner_id) continue; send_state_host(fd, hs, i+1); } if (status) free(status); return; fail: send_all(fd, &h, sizeof(h), MSG_NOSIGNAL); if (status) free(status); } static void cmd_renewal(int fd, struct sm_header *h_recv) { struct sm_header h; struct sanlk_lockspace lockspace; struct space *sp; uint32_t io_timeout = 0; struct renewal_history *history = NULL; struct renewal_history *hi; int history_size, history_prev, history_next; int i, rv, len; memset(&h, 0, sizeof(h)); memcpy(&h, h_recv, sizeof(struct sm_header)); h.version = SM_PROTO; h.length = sizeof(h); h.data = 0; if (!com.renewal_history_size) goto fail; len = sizeof(struct renewal_history) * com.renewal_history_size; history = malloc(len); if (!history) { h.data = -ENOMEM; goto fail; } rv = recv_loop(fd, &lockspace, sizeof(struct sanlk_lockspace), MSG_WAITALL); if (rv != sizeof(struct sanlk_lockspace)) { h.data = -ENOTCONN; goto fail; } pthread_mutex_lock(&spaces_mutex); sp = find_lockspace(lockspace.name); if (sp) { history_size = sp->renewal_history_size; history_prev = sp->renewal_history_prev; history_next = sp->renewal_history_next; io_timeout = sp->io_timeout; if (history_size != com.renewal_history_size) { log_error("mismatch history size"); history_size = 0; history_prev = 0; history_next = 0; } else { memcpy(history, sp->renewal_history, len); } } pthread_mutex_unlock(&spaces_mutex); if (!sp) { h.data = -ENOSPC; goto fail; } if (!history_size || (!history_prev && !history_next)) goto fail; h.data2 = io_timeout; send_all(fd, &h, sizeof(h), MSG_NOSIGNAL); /* If next slot is non-zero, then we've wrapped and should begin sending history from next to end before sending from 0 to prev. */ if (history[history_next].timestamp) { for (i = history_next; i < history_size; i++) { hi = &history[i]; send_state_renewal(fd, hi); } } for (i = 0; i < history_next; i++) { hi = &history[i]; send_state_renewal(fd, hi); } if (history) free(history); return; fail: send_all(fd, &h, sizeof(h), MSG_NOSIGNAL); if (history) free(history); } static char send_data_buf[LOG_DUMP_SIZE]; static void cmd_log_dump(int fd, struct sm_header *h_recv) { int len; copy_log_dump(send_data_buf, &len); h_recv->version = SM_PROTO; h_recv->data = len; send_all(fd, h_recv, sizeof(struct sm_header), MSG_NOSIGNAL); send_all(fd, send_data_buf, len, MSG_NOSIGNAL); } static void cmd_get_lockspaces(int ci, int fd, struct sm_header *h_recv, uint32_t cmd) { int count, len, rv; log_cmd(cmd, "cmd_get_lockspaces %d,%d", ci, fd); rv = get_lockspaces(send_data_buf, &len, &count, LOG_DUMP_SIZE); h_recv->version = SM_PROTO; h_recv->length = sizeof(struct sm_header) + len; h_recv->data = rv; h_recv->data2 = count; send_all(fd, h_recv, sizeof(struct sm_header), MSG_NOSIGNAL); send_all(fd, send_data_buf, len, MSG_NOSIGNAL); } static void cmd_get_hosts(int ci, int fd, struct sm_header *h_recv, uint32_t cmd) { struct sm_header h; struct sanlk_lockspace lockspace; struct sanlk_host host = { 0 }; void *data_buf; int data_len = 0; int count = 0, len = 0; int rv; log_cmd(cmd, "cmd_get_hosts %d,%d %x", ci, fd, h_recv->cmd_flags); memset(&h, 0, sizeof(h)); memcpy(&h, h_recv, sizeof(struct sm_header)); h.version = SM_PROTO; h.length = sizeof(h); h.data = 0; rv = recv_loop(fd, &lockspace, sizeof(struct sanlk_lockspace), MSG_WAITALL); if (rv != sizeof(struct sanlk_lockspace)) { h.data = -ENOTCONN; goto out; } if (h_recv->cmd_flags & SANLK_GET_HOST_LOCAL) { rv = get_host_local(&lockspace, &host); h.length = sizeof(struct sm_header) + sizeof(host); h.data = rv; h.data2 = 1; if (!rv) { data_buf = &host; data_len = sizeof(host); } } else { rv = get_hosts(&lockspace, send_data_buf, &len, &count, LOG_DUMP_SIZE); h.length = sizeof(struct sm_header) + len; h.data = rv; h.data2 = count; if (!rv) { data_buf = send_data_buf; data_len = len; } } out: send_all(fd, &h, sizeof(struct sm_header), MSG_NOSIGNAL); if (data_len) send_all(fd, data_buf, data_len, MSG_NOSIGNAL); } static void cmd_restrict(int ci, int fd, struct sm_header *h_recv, uint32_t cmd) { log_cmd(cmd, "cmd_restrict ci %d fd %d pid %d flags %x", ci, fd, client[ci].pid, h_recv->cmd_flags); client[ci].restricted = h_recv->cmd_flags; h_recv->version = SM_PROTO; send_result(ci, fd, h_recv, 0); } static void cmd_version(int ci GNUC_UNUSED, int fd, struct sm_header *h_recv) { h_recv->magic = SM_MAGIC; h_recv->version = SM_PROTO; h_recv->cmd = SM_CMD_VERSION; h_recv->cmd_flags = 0; h_recv->length = sizeof(struct sm_header); h_recv->seq = 0; h_recv->data = 0; h_recv->data2 = sanlock_version_combined; send_all(fd, h_recv, sizeof(struct sm_header), MSG_NOSIGNAL); } static void cmd_reg_event(int fd, struct sm_header *h_recv, uint32_t cmd) { struct sm_header h; struct sanlk_lockspace lockspace; struct sanlk_host_event he; int rv; memcpy(&h, h_recv, sizeof(struct sm_header)); h.version = SM_PROTO; h.length = sizeof(struct sm_header); rv = recv_loop(fd, &lockspace, sizeof(struct sanlk_lockspace), MSG_WAITALL); if (rv != sizeof(struct sanlk_lockspace)) { h.data = -ENOTCONN; goto out; } /* currently unused */ rv = recv_loop(fd, &he, sizeof(he), MSG_WAITALL); if (rv != sizeof(he)) { h.data = -ENOTCONN; goto out; } rv = lockspace_reg_event(&lockspace, fd, h_recv->cmd_flags); h.data = rv; out: log_cmd(cmd, "cmd_reg_event fd %d rv %d", fd, rv); send_all(fd, &h, sizeof(struct sm_header), MSG_NOSIGNAL); } static void cmd_end_event(int fd, struct sm_header *h_recv, uint32_t cmd) { struct sm_header h; struct sanlk_lockspace lockspace; int rv; memcpy(&h, h_recv, sizeof(struct sm_header)); h.version = SM_PROTO; h.length = sizeof(struct sm_header); rv = recv_loop(fd, &lockspace, sizeof(struct sanlk_lockspace), MSG_WAITALL); if (rv != sizeof(struct sanlk_lockspace)) { h.data = -ENOTCONN; goto out; } rv = lockspace_end_event(&lockspace); h.data = rv; out: log_cmd(cmd, "cmd_end_event fd %d rv %d", fd, rv); send_all(fd, &h, sizeof(struct sm_header), MSG_NOSIGNAL); } static void cmd_set_config(int fd, struct sm_header *h_recv, uint32_t cmd) { struct sm_header h; struct sanlk_lockspace lockspace; int rv; memcpy(&h, h_recv, sizeof(struct sm_header)); h.version = SM_PROTO; h.length = sizeof(struct sm_header); rv = recv_loop(fd, &lockspace, sizeof(struct sanlk_lockspace), MSG_WAITALL); if (rv != sizeof(struct sanlk_lockspace)) { h.data = -ENOTCONN; goto out; } rv = lockspace_set_config(&lockspace, h_recv->cmd_flags, h_recv->data, h_recv->data2); h.data = rv; out: log_cmd(cmd, "cmd_set_config fd %d rv %d", fd, rv); send_all(fd, &h, sizeof(struct sm_header), MSG_NOSIGNAL); } static int get_peer_pid(int fd, int *pid) { struct ucred cred; unsigned int len = sizeof(cred); if (getsockopt(fd, SOL_SOCKET, SO_PEERCRED, &cred, &len) != 0) return -1; *pid = cred.pid; return 0; } void call_cmd_daemon(int ci, struct sm_header *h_recv, int client_maxi) { int rv, pid, auto_close = 1; int fd = client[ci].fd; uint32_t cmd = h_recv->cmd; switch (cmd) { case SM_CMD_REGISTER: rv = get_peer_pid(fd, &pid); if (rv < 0) { log_error("cmd_register ci %d fd %d get pid failed", ci, fd); break; } log_cmd(cmd, "cmd_register ci %d fd %d pid %d", ci, fd, pid); snprintf(client[ci].owner_name, SANLK_NAME_LEN, "%d", pid); client[ci].pid = pid; client[ci].deadfn = client_pid_dead; if (client[ci].tokens) { log_error("cmd_register ci %d fd %d tokens exist slots %d", ci, fd, client[ci].tokens_slots); free(client[ci].tokens); } client[ci].tokens_slots = SANLK_MAX_RESOURCES; client[ci].tokens = malloc(sizeof(struct token *) * SANLK_MAX_RESOURCES); if (!client[ci].tokens) { rv = -ENOMEM; log_error("cmd_register ci %d fd %d ENOMEM", ci, fd); break; } memset(client[ci].tokens, 0, sizeof(struct token *) * SANLK_MAX_RESOURCES); auto_close = 0; break; case SM_CMD_RESTRICT: cmd_restrict(ci, fd, h_recv, cmd); auto_close = 0; break; case SM_CMD_VERSION: cmd_version(ci, fd, h_recv); auto_close = 0; break; case SM_CMD_SHUTDOWN: strcpy(client[ci].owner_name, "shutdown"); if (h_recv->data) { /* force */ external_shutdown = 2; } else { pthread_mutex_lock(&spaces_mutex); if (list_empty(&spaces) && list_empty(&spaces_rem) && list_empty(&spaces_add)) external_shutdown = 1; else log_debug("ignore shutdown, lockspace exists"); pthread_mutex_unlock(&spaces_mutex); } break; case SM_CMD_STATUS: strcpy(client[ci].owner_name, "status"); cmd_status(ci, fd, h_recv, client_maxi, cmd); break; case SM_CMD_HOST_STATUS: strcpy(client[ci].owner_name, "host_status"); cmd_host_status(ci, fd, h_recv, cmd); break; case SM_CMD_RENEWAL: strcpy(client[ci].owner_name, "renewal"); cmd_renewal(fd, h_recv); break; case SM_CMD_LOG_DUMP: strcpy(client[ci].owner_name, "log_dump"); cmd_log_dump(fd, h_recv); break; case SM_CMD_GET_LOCKSPACES: strcpy(client[ci].owner_name, "get_lockspaces"); cmd_get_lockspaces(ci, fd, h_recv, cmd); break; case SM_CMD_GET_HOSTS: strcpy(client[ci].owner_name, "get_hosts"); cmd_get_hosts(ci, fd, h_recv, cmd); break; case SM_CMD_REG_EVENT: strcpy(client[ci].owner_name, "reg_event"); cmd_reg_event(fd, h_recv, cmd); break; case SM_CMD_END_EVENT: strcpy(client[ci].owner_name, "end_event"); cmd_end_event(fd, h_recv, cmd); break; case SM_CMD_SET_CONFIG: strcpy(client[ci].owner_name, "set_config"); cmd_set_config(fd, h_recv, cmd); break; }; /* * Previously just called close(fd) and did not set client[ci].fd = -1. * This meant that a new client ci could get this fd and use it. * * When a poll error occurs because this ci was finished, then * client_free(ci) would be called for this ci. client_free would * see cl->fd was still set and call close() on it, even though that * fd was now in use by another ci. * * We could probably get by with just doing this here: * client[ci].fd = -1; * close(fd); * * and then handling the full client_free in response to * the poll error (as done previously), but I see no reason * to avoid the full client_free here. */ if (auto_close) client_free(ci); } sanlock/src/cmd.h000066400000000000000000000013021516326262600142200ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __CMD_H__ #define __CMD_H__ struct cmd_args { struct list_head list; /* thread_pool data */ int ci_in; int ci_target; int cl_fd; int cl_pid; struct sm_header header; }; /* cmds processed by thread pool */ void call_cmd_thread(struct task *task, struct cmd_args *ca); /* cmds processed by main loop */ void call_cmd_daemon(int ci, struct sm_header *h_recv, int client_maxi); void daemon_shutdown_reply(void); #endif sanlock/src/crc32c.c000066400000000000000000000103441516326262600145350ustar00rootroot00000000000000/* * Copied from the btrfs-progs source code, which... * Copied from the kernel source code, lib/libcrc32c.c. * * This program is free software; you can redistribute it and/or modify it * under the terms of the GNU General Public License as published by the Free * Software Foundation; either version 2 of the License, or (at your option) * any later version. */ #include #include /* * This is the CRC-32C table * Generated with: * width = 32 bits * poly = 0x1EDC6F41 * reflect input bytes = true * reflect output bytes = true */ static const uint32_t crc32c_table[256] = { 0x00000000L, 0xF26B8303L, 0xE13B70F7L, 0x1350F3F4L, 0xC79A971FL, 0x35F1141CL, 0x26A1E7E8L, 0xD4CA64EBL, 0x8AD958CFL, 0x78B2DBCCL, 0x6BE22838L, 0x9989AB3BL, 0x4D43CFD0L, 0xBF284CD3L, 0xAC78BF27L, 0x5E133C24L, 0x105EC76FL, 0xE235446CL, 0xF165B798L, 0x030E349BL, 0xD7C45070L, 0x25AFD373L, 0x36FF2087L, 0xC494A384L, 0x9A879FA0L, 0x68EC1CA3L, 0x7BBCEF57L, 0x89D76C54L, 0x5D1D08BFL, 0xAF768BBCL, 0xBC267848L, 0x4E4DFB4BL, 0x20BD8EDEL, 0xD2D60DDDL, 0xC186FE29L, 0x33ED7D2AL, 0xE72719C1L, 0x154C9AC2L, 0x061C6936L, 0xF477EA35L, 0xAA64D611L, 0x580F5512L, 0x4B5FA6E6L, 0xB93425E5L, 0x6DFE410EL, 0x9F95C20DL, 0x8CC531F9L, 0x7EAEB2FAL, 0x30E349B1L, 0xC288CAB2L, 0xD1D83946L, 0x23B3BA45L, 0xF779DEAEL, 0x05125DADL, 0x1642AE59L, 0xE4292D5AL, 0xBA3A117EL, 0x4851927DL, 0x5B016189L, 0xA96AE28AL, 0x7DA08661L, 0x8FCB0562L, 0x9C9BF696L, 0x6EF07595L, 0x417B1DBCL, 0xB3109EBFL, 0xA0406D4BL, 0x522BEE48L, 0x86E18AA3L, 0x748A09A0L, 0x67DAFA54L, 0x95B17957L, 0xCBA24573L, 0x39C9C670L, 0x2A993584L, 0xD8F2B687L, 0x0C38D26CL, 0xFE53516FL, 0xED03A29BL, 0x1F682198L, 0x5125DAD3L, 0xA34E59D0L, 0xB01EAA24L, 0x42752927L, 0x96BF4DCCL, 0x64D4CECFL, 0x77843D3BL, 0x85EFBE38L, 0xDBFC821CL, 0x2997011FL, 0x3AC7F2EBL, 0xC8AC71E8L, 0x1C661503L, 0xEE0D9600L, 0xFD5D65F4L, 0x0F36E6F7L, 0x61C69362L, 0x93AD1061L, 0x80FDE395L, 0x72966096L, 0xA65C047DL, 0x5437877EL, 0x4767748AL, 0xB50CF789L, 0xEB1FCBADL, 0x197448AEL, 0x0A24BB5AL, 0xF84F3859L, 0x2C855CB2L, 0xDEEEDFB1L, 0xCDBE2C45L, 0x3FD5AF46L, 0x7198540DL, 0x83F3D70EL, 0x90A324FAL, 0x62C8A7F9L, 0xB602C312L, 0x44694011L, 0x5739B3E5L, 0xA55230E6L, 0xFB410CC2L, 0x092A8FC1L, 0x1A7A7C35L, 0xE811FF36L, 0x3CDB9BDDL, 0xCEB018DEL, 0xDDE0EB2AL, 0x2F8B6829L, 0x82F63B78L, 0x709DB87BL, 0x63CD4B8FL, 0x91A6C88CL, 0x456CAC67L, 0xB7072F64L, 0xA457DC90L, 0x563C5F93L, 0x082F63B7L, 0xFA44E0B4L, 0xE9141340L, 0x1B7F9043L, 0xCFB5F4A8L, 0x3DDE77ABL, 0x2E8E845FL, 0xDCE5075CL, 0x92A8FC17L, 0x60C37F14L, 0x73938CE0L, 0x81F80FE3L, 0x55326B08L, 0xA759E80BL, 0xB4091BFFL, 0x466298FCL, 0x1871A4D8L, 0xEA1A27DBL, 0xF94AD42FL, 0x0B21572CL, 0xDFEB33C7L, 0x2D80B0C4L, 0x3ED04330L, 0xCCBBC033L, 0xA24BB5A6L, 0x502036A5L, 0x4370C551L, 0xB11B4652L, 0x65D122B9L, 0x97BAA1BAL, 0x84EA524EL, 0x7681D14DL, 0x2892ED69L, 0xDAF96E6AL, 0xC9A99D9EL, 0x3BC21E9DL, 0xEF087A76L, 0x1D63F975L, 0x0E330A81L, 0xFC588982L, 0xB21572C9L, 0x407EF1CAL, 0x532E023EL, 0xA145813DL, 0x758FE5D6L, 0x87E466D5L, 0x94B49521L, 0x66DF1622L, 0x38CC2A06L, 0xCAA7A905L, 0xD9F75AF1L, 0x2B9CD9F2L, 0xFF56BD19L, 0x0D3D3E1AL, 0x1E6DCDEEL, 0xEC064EEDL, 0xC38D26C4L, 0x31E6A5C7L, 0x22B65633L, 0xD0DDD530L, 0x0417B1DBL, 0xF67C32D8L, 0xE52CC12CL, 0x1747422FL, 0x49547E0BL, 0xBB3FFD08L, 0xA86F0EFCL, 0x5A048DFFL, 0x8ECEE914L, 0x7CA56A17L, 0x6FF599E3L, 0x9D9E1AE0L, 0xD3D3E1ABL, 0x21B862A8L, 0x32E8915CL, 0xC083125FL, 0x144976B4L, 0xE622F5B7L, 0xF5720643L, 0x07198540L, 0x590AB964L, 0xAB613A67L, 0xB831C993L, 0x4A5A4A90L, 0x9E902E7BL, 0x6CFBAD78L, 0x7FAB5E8CL, 0x8DC0DD8FL, 0xE330A81AL, 0x115B2B19L, 0x020BD8EDL, 0xF0605BEEL, 0x24AA3F05L, 0xD6C1BC06L, 0xC5914FF2L, 0x37FACCF1L, 0x69E9F0D5L, 0x9B8273D6L, 0x88D28022L, 0x7AB90321L, 0xAE7367CAL, 0x5C18E4C9L, 0x4F48173DL, 0xBD23943EL, 0xF36E6F75L, 0x0105EC76L, 0x12551F82L, 0xE03E9C81L, 0x34F4F86AL, 0xC69F7B69L, 0xD5CF889DL, 0x27A40B9EL, 0x79B737BAL, 0x8BDCB4B9L, 0x988C474DL, 0x6AE7C44EL, 0xBE2DA0A5L, 0x4C4623A6L, 0x5F16D052L, 0xAD7D5351L }; /* * Steps through buffer one byte at at time, calculates reflected * crc using table. */ uint32_t crc32c(uint32_t crc, uint8_t *data, size_t length); uint32_t crc32c(uint32_t crc, uint8_t *data, size_t length) { while (length--) crc = crc32c_table[(crc ^ *data++) & 0xFFL] ^ (crc >> 8); return crc; } sanlock/src/delta_lease.c000066400000000000000000001720531516326262600157260ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock_internal.h" #include "sanlock.h" #include "sanlock_admin.h" #include "diskio.h" #include "ondisk.h" #include "direct.h" #include "log.h" #include "paxos_lease.h" #include "delta_lease.h" #include "timeouts.h" /* Based on "Light-Weight Leases for Storage-Centric Coordination" by Gregory Chockler and Dahlia Malkhi */ /* delta_leases are a series max_hosts leader_records, one leader per sector, host N's delta_lease is the leader_record in sectors N-1 */ /* * variable names: * rv: success is 0, failure is < 0 * error: success is 1 (SANLK_OK), failure is < 0 */ static void log_leader_error(int result, char *space_name, uint64_t host_id, struct sync_disk *disk, struct leader_record *lr, const char *caller) { log_error("leader1 %s error %d lockspace %.48s host_id %llu", caller ? caller : "unknown", result, space_name, (unsigned long long)host_id); log_error("leader2 path %s offset %llu", disk->path, (unsigned long long)disk->offset); log_error("leader3 m %x v %x ss %u nh %llu mh %llu oi %llu og %llu lv %llu", lr->magic, lr->version, lr->sector_size, (unsigned long long)lr->num_hosts, (unsigned long long)lr->max_hosts, (unsigned long long)lr->owner_id, (unsigned long long)lr->owner_generation, (unsigned long long)lr->lver); log_error("leader4 sn %.48s rn %.48s ts %llu cs %x", lr->space_name, lr->resource_name, (unsigned long long)lr->timestamp, lr->checksum); } static int verify_leader(struct sync_disk *disk, char *space_name, uint64_t host_id, struct leader_record *lr, uint32_t checksum, const char *caller) { int result; if (lr->magic != DELTA_DISK_MAGIC) { log_error("verify_leader %llu wrong magic %x %s", (unsigned long long)host_id, lr->magic, disk->path); result = SANLK_LEADER_MAGIC; goto fail; } if ((lr->version & 0xFFFF0000) > DELTA_DISK_VERSION_MAJOR) { log_error("verify_leader %llu newer major version leader 0x%x local 0x%x %s", (unsigned long long)host_id, lr->version, DELTA_DISK_VERSION_MAJOR|DELTA_DISK_VERSION_MINOR, disk->path); result = SANLK_LEADER_VERSION; goto fail; } if (strncmp(lr->space_name, space_name, NAME_ID_SIZE)) { log_error("verify_leader %llu wrong space name %.48s %.48s %s", (unsigned long long)host_id, lr->space_name, space_name, disk->path); result = SANLK_LEADER_LOCKSPACE; goto fail; } if (lr->checksum != checksum) { log_error("verify_leader %llu wrong checksum %x %x %s", (unsigned long long)host_id, lr->checksum, checksum, disk->path); result = SANLK_LEADER_CHECKSUM; goto fail; } return SANLK_OK; fail: log_leader_error(result, space_name, host_id, disk, lr, caller); return result; } int delta_read_lockspace(struct task *task, struct sync_disk *disk, int sector_size, uint64_t host_id, struct sanlk_lockspace *ls, int io_timeout, struct sanlk_host *hs) { struct leader_record leader_end; struct leader_record leader; uint32_t checksum; char *space_name; int align_size; int rv, error; /* host_id N is block offset N-1 */ memset(&leader_end, 0, sizeof(struct leader_record)); rv = read_sector(disk, sector_size, host_id - 1, (char *)&leader_end, sizeof(struct leader_record), task, task->use_aio, io_timeout, "read_lockspace"); if (rv < 0) return rv; /* N.B. compute checksum before byte swapping */ checksum = leader_checksum(&leader_end); leader_record_in(&leader_end, &leader); if (!ls->name[0]) space_name = leader.space_name; else space_name = ls->name; error = verify_leader(disk, space_name, host_id, &leader, checksum, "read_lockspace"); if (error == SANLK_OK) { memcpy(ls->name, leader.space_name, SANLK_NAME_LEN); ls->host_id = host_id; align_size = leader_align_size_from_flag(leader.flags); if (!align_size) align_size = sector_size_to_align_size_old(leader.sector_size); /* The flags set by the user may not have been correct. */ sanlk_lsf_sector_flags_clear(&ls->flags); sanlk_lsf_align_flags_clear(&ls->flags); ls->flags |= sanlk_lsf_sector_size_to_flag(leader.sector_size); ls->flags |= sanlk_lsf_align_size_to_flag(align_size); if (leader.flags & LFL_NO_TIMEOUT) ls->flags |= SANLK_LSF_NO_TIMEOUT; if (leader.flags & LFL_CAW) ls->flags |= SANLK_LSF_USING_CAW; if (hs) { hs->host_id = leader.owner_id; hs->generation = leader.owner_generation; hs->timestamp = leader.timestamp; hs->io_timeout = leader.io_timeout; if (leader.flags & LFL_DEAD_EXT) hs->flags |= SANLK_HOST_DEAD_EXT; } } return error; } int delta_read_lockspace_sizes(struct task *task, struct sync_disk *disk, uint64_t host_id, int use_aio, int io_timeout, int *sector_size, int *align_size, int *is_caw) { struct leader_record leader_end; struct leader_record leader; int rv; memset(&leader_end, 0, sizeof(struct leader_record)); /* With no host_id, get sizes from any leader record. */ if (!host_id) host_id = 1; rv = read_sector(disk, 512, host_id - 1, (char *)&leader_end, sizeof(struct leader_record), task, use_aio, io_timeout, "read_lockspace_sector_size"); if (rv < 0) goto next; leader_record_in(&leader_end, &leader); if (leader.magic != DELTA_DISK_MAGIC) goto next; if ((leader.version & 0xFFFF0000) > DELTA_DISK_VERSION_MAJOR) goto next; if (leader.owner_id && (leader.owner_id != host_id)) goto next; if (leader.sector_size != 512) return SANLK_LEADER_SECTORSIZE; *sector_size = leader.sector_size; *align_size = leader_align_size_from_flag(leader.flags); if (!*align_size) *align_size = sector_size_to_align_size_old(leader.sector_size); if (leader.flags & LFL_CAW) *is_caw = 1; return SANLK_OK; next: rv = read_sector(disk, 4096, host_id - 1, (char *)&leader_end, sizeof(struct leader_record), task, use_aio, io_timeout, "read_lockspace_sector_size"); if (rv < 0) return rv; leader_record_in(&leader_end, &leader); if (leader.magic != DELTA_DISK_MAGIC) return SANLK_LEADER_MAGIC; if ((leader.version & 0xFFFF0000) > DELTA_DISK_VERSION_MAJOR) return SANLK_LEADER_VERSION; if (leader.owner_id && (leader.owner_id != host_id)) return SANLK_LEADER_DIFF; if (leader.sector_size != 4096) return SANLK_LEADER_SECTORSIZE; *sector_size = leader.sector_size; *align_size = leader_align_size_from_flag(leader.flags); if (!*align_size) *align_size = sector_size_to_align_size_old(leader.sector_size); if (leader.flags & LFL_CAW) *is_caw = 1; return SANLK_OK; } int delta_lease_leader_read(struct task *task, int sector_size, int io_timeout, struct sync_disk *disk, char *space_name, uint64_t host_id, struct leader_record *leader_ret, const char *caller) { struct leader_record leader_end; struct leader_record leader; uint32_t checksum; int rv, error; if (!sector_size) { log_error("delta_lease_leader_read with zero sector_size %s", space_name); return -EINVAL; } /* host_id N is block offset N-1 */ memset(&leader_end, 0, sizeof(struct leader_record)); memset(leader_ret, 0, sizeof(struct leader_record)); rv = read_sector(disk, sector_size, host_id - 1, (char *)&leader_end, sizeof(struct leader_record), task, task->use_aio, io_timeout, "delta_leader"); if (rv < 0) return rv; /* N.B. compute checksum before byte swapping */ checksum = leader_checksum(&leader_end); leader_record_in(&leader_end, &leader); error = verify_leader(disk, space_name, host_id, &leader, checksum, caller); memcpy(leader_ret, &leader, sizeof(struct leader_record)); return error; } int delta_lease_leader_read_buf(struct task *task, int sector_size, int use_aio, int io_timeout, struct sync_disk *disk, char *space_name, uint64_t host_id, struct leader_record *leader_ret, char *buf, const char *caller) { struct leader_record leader_end = { 0 }; struct leader_record leader = { 0 }; uint32_t checksum; int rv; rv = read_iobuf(disk->fd, disk->offset + ((host_id - 1) * sector_size), buf, sector_size, task, use_aio, io_timeout, NULL); if (rv < 0) return rv; memcpy(&leader_end, buf, sizeof(struct leader_record)); checksum = leader_checksum(&leader_end); leader_record_in(&leader_end, &leader); rv = verify_leader(disk, space_name, host_id, &leader, checksum, caller); if (rv < 0) return rv; memcpy(leader_ret, &leader, sizeof(struct leader_record)); return SANLK_OK; } int delta_caw_lease_leader_clobber(struct task *task, int io_timeout, struct sync_disk *disk, uint64_t host_id, struct leader_record *leader, int sector_size, uint32_t caw_major, uint32_t caw_minor, uint64_t caw_offset, char *caw_iobuf, const char *caller) { char caw_path[PATH_MAX] = { 0 }; uint64_t offset_bytes; struct leader_record leader_end = { 0 }; uint32_t checksum; int fd; int rv; if (devno_to_path(caw_major, caw_minor, caw_path) < 0) { log_error("delta_caw_lease_leader_clobber devno_to_path %u:%u error", caw_major, caw_minor); return -1; } if ((fd = open(caw_path, O_RDWR | O_DIRECT | O_SYNC, 0)) < 0) { log_error("delta_caw_lease_leader_clobber %u:%u %s open errno %d", caw_major, caw_minor, caw_path, errno); return -1; } leader_record_out(leader, &leader_end); checksum = leader_checksum(&leader_end); leader_end.checksum = cpu_to_le32(checksum); memmove(caw_iobuf + sector_size, caw_iobuf, sector_size); memcpy(caw_iobuf + sector_size, &leader_end, sizeof(struct leader_record)); offset_bytes = caw_offset + ((host_id - 1) * sector_size); rv = compare_and_write(task, fd, io_timeout, caw_major, caw_minor, sector_size, offset_bytes, caw_iobuf, NULL, NULL); close(fd); return rv; } /* * NB. this should not be used to write the leader record, it is meant only * for manually clobbering the disk to corrupt it for testing, or to manually * repair it after it's corrupted. */ int delta_lease_leader_clobber(struct task *task, int io_timeout, struct sync_disk *disk, uint64_t host_id, struct leader_record *leader, const char *caller) { struct leader_record leader_end = { 0 }; uint32_t checksum; int rv; leader_record_out(leader, &leader_end); checksum = leader_checksum(&leader_end); leader_end.checksum = cpu_to_le32(checksum); rv = write_sector(disk, leader->sector_size, host_id - 1, (char *)&leader_end, sizeof(struct leader_record), task, task->use_aio, io_timeout, caller); if (rv < 0) return rv; return SANLK_OK; } static void make_local_copy_path(char *path, int len, struct space *sp) { snprintf(path, len, "/var/lib/sanlock/delta_lease_%s_%llu", sp->space_name, (unsigned long long)sp->host_id); } static void delta_local_copy_read(struct space *sp, uint64_t *prev_gen) { char path[PATH_MAX]; FILE *file; int rv; *prev_gen = 0; make_local_copy_path(path, sizeof(path), sp); file = fopen(path, "r"); if (!file) { log_space(sp, "delta_local_copy_read open error %d %s", errno, path); return; } rv = fscanf(file, "%llu", (unsigned long long *)prev_gen); if (rv != 1) log_space(sp, "delta_local_copy_read read error %d %s", errno, path); fclose(file); } static void delta_local_copy_write(struct space *sp, uint64_t next_gen) { char path[PATH_MAX]; FILE *file; int rv; make_local_copy_path(path, sizeof(path), sp); file = fopen(path, "w"); if (!file) { log_space(sp, "delta_local_copy_write open error %d", errno); return; } rv = fprintf(file, "%llu\n", (unsigned long long)next_gen); if (rv < 0) log_erros(sp, "delta_local_copy_write write error %d %s", errno, path); if (fflush(file)) log_erros(sp, "delta_local_copy_write flush error %d %s", errno, path); if (fclose(file)) log_erros(sp, "delta_local_copy_write close error %d %s", errno, path); } int delta_compare_and_write(struct task *task, struct space *sp, uint64_t host_id, char *caw_data, int *wr_ms) { uint64_t offset_bytes; struct timespec begin, end, diff; int rv; if (wr_ms) clock_gettime(CLOCK_MONOTONIC_RAW, &begin); if (sp->caw_dev_use) { offset_bytes = sp->caw_dev_offset + ((host_id - 1) * sp->sector_size); rv = compare_and_write(task, sp->caw_dev_fd, sp->io_timeout, sp->caw_dev_major, sp->caw_dev_minor, sp->sector_size, offset_bytes, caw_data, NULL, NULL); } else { offset_bytes = sp->host_id_disk.offset + ((host_id - 1) * sp->sector_size); rv = compare_and_write(task, sp->host_id_disk.fd, sp->io_timeout, sp->disk_major, sp->disk_minor, sp->sector_size, offset_bytes, caw_data, NULL, NULL); } if (wr_ms) { clock_gettime(CLOCK_MONOTONIC_RAW, &end); ts_diff(&begin, &end, &diff); *wr_ms = (diff.tv_sec * 1000) + (diff.tv_nsec / 1000000); } return rv; } int delta_caw_lease_acquire(struct task *task, struct space *sp, char *our_host_name, struct leader_record *leader_ret) { struct leader_record prev_leader; struct leader_record next_leader; struct leader_record leader_end; uint32_t checksum; int retries = 0; int error; retry: /* * On host_id_disk, read host_id sector into caw_data buf, * copy leader out of that, and verify delta lease fields are valid. * (delta_read_lockspace_sizes has just read this, so reading again * here could be avoided with code reorg, but this is not frequent.) */ error = delta_lease_leader_read_buf(task, sp->sector_size, task->use_aio, sp->io_timeout, &sp->host_id_disk, sp->space_name, sp->host_id, &prev_leader, sp->caw_data, "caw_lease_acquire"); if (error == SANLK_AIO_TIMEOUT) { log_space(sp, "delta_caw_acquire leader_read error %d", error); sp->caw_data = NULL; return error; } if (error < 0) { log_space(sp, "delta_caw_acquire leader_read error %d", error); return error; } log_space(sp, "delta_caw_acquire read magic 0x%x version 0x%x flags 0x%x owner %llu %llu %llu %.48s %s", prev_leader.magic, prev_leader.version, prev_leader.flags, (unsigned long long)prev_leader.owner_id, (unsigned long long)prev_leader.owner_generation, (unsigned long long)prev_leader.timestamp, prev_leader.resource_name, (prev_leader.flags & LFL_DEAD_EXT) ? "DEAD_EXT" : ""); if (!(prev_leader.flags & LFL_CAW)) { /* shouldn't happen */ log_erros(sp, "delta_caw_acquire leader not caw"); return SANLK_WRONG_CAW; } if (retries && !(prev_leader.flags & LFL_DEAD_EXT)) { /* the only reason for a miscompare retry is if another host is setting our DEAD_EXT flag. */ log_erros(sp, "delta_caw_acquire retry did not find dead_ext flags 0x%x owner %llu %llu %llu %.48s", prev_leader.flags, (unsigned long long)prev_leader.owner_id, (unsigned long long)prev_leader.owner_generation, (unsigned long long)prev_leader.timestamp, prev_leader.resource_name); } memcpy(&next_leader, &prev_leader, sizeof(struct leader_record)); next_leader.flags &= ~LFL_DEAD_EXT; next_leader.timestamp = monotime(); next_leader.io_timeout = (sp->io_timeout & 0x00FF); next_leader.owner_id = sp->host_id; next_leader.owner_generation += 1; memcpy(next_leader.resource_name, our_host_name, NAME_ID_SIZE); next_leader.checksum = 0; /* set below */ log_space(sp, "delta_caw_acquire write owner %llu %llu %llu %.48s", (unsigned long long)next_leader.owner_id, (unsigned long long)next_leader.owner_generation, (unsigned long long)next_leader.timestamp, next_leader.resource_name); leader_record_out(&next_leader, &leader_end); checksum = leader_checksum(&leader_end); /* compute checksum on byte swapped data */ next_leader.checksum = checksum; leader_end.checksum = cpu_to_le32(checksum); /* * prev_data is the data currently on the disk, * and exists in the first half of caw_data. * next_data is a copy of prev_data updated with * the new leader record, and goes in second half * of caw_data. */ memmove(sp->caw_data + sp->sector_size, sp->caw_data, sp->sector_size); memcpy(sp->caw_data + sp->sector_size, &leader_end, sizeof(struct leader_record)); error = delta_compare_and_write(task, sp, sp->host_id, sp->caw_data, NULL); if (error == SANLK_CAW_MISCOMPARE) { /* can get miscompare if another host is setting dead_ext */ log_space(sp, "delta_caw_acquire compare_and_write error %d", error); if (!retries) { retries++; goto retry; } return error; } else if (error < 0) { log_erros(sp, "delta_caw_acquire compare_and_write error %d", error); return error; } memcpy(leader_ret, &next_leader, sizeof(struct leader_record)); return SANLK_OK; } /* * delta_lease_acquire: * set the owner of host_id to our_host_name. * * paxos_lease_acquire: * set the owner of resource_name to host_id. * * our_host_name is a unique host identifier used to detect when two different * hosts are trying to acquire the same host_id (since both will be using the * same host_id, that host_id won't work to distinguish between them.) We copy * our_host_name into leader.resource_name, so in a sense the owner_id and * resource_name fields of the leader_record switch functions: the common * resource is the ower_id, and the distinguishing id is the resource_name. */ int delta_lease_acquire(struct task *task, struct space *sp, struct sync_disk *disk, char *space_name, char *our_host_name, uint64_t host_id, int nodelay, struct leader_record *leader_ret) { struct leader_record leader; struct leader_record leader1; struct leader_record leader_end; uint64_t new_ts; uint64_t prev_generation; uint64_t next_generation; uint32_t checksum; uint32_t version; int other_io_timeout, other_host_dead_seconds, other_id_renewal_seconds; int i, error, rv, delay, delta_large_delay; int is_free, is_same; int no_delay = 0; log_space(sp, "delta_acquire begin %.48s:%llu", sp->space_name, (unsigned long long)host_id); /* * Keep a local copy of the last started generation number * in case the delta lease on disk is reverted by another * host writing the DEAD_EXT flag. The local copy of the * generation number ensures we don't reuse the same * generation number that we started last. */ delta_local_copy_read(sp, &prev_generation); error = delta_lease_leader_read(task, sp->sector_size, sp->io_timeout, disk, space_name, host_id, &leader, "delta_acquire_begin"); if (error < 0) { log_space(sp, "delta_acquire leader_read1 error %d", error); return error; } version = leader.version; other_io_timeout = leader.io_timeout; if (!other_io_timeout) other_io_timeout = sp->io_timeout; /* * If the delta lease is free, and the prev owner matches our host * name, then reacquire with no delay. * * If the delta lease is not free, and the prev owner matches our host * name, and our host name is from product_uuid, then reacquire with no * delay. Assumption here is that the delta lease is not free because * this host did not release it cleanly last time. The non-free lease * generally indicates it's being used by someone, and we should * monitor it for a renewal period for updates. But, we skip this * monitoring given the certainty that comes from the owner being * the product_uuid, and the assumption that our product_uuid will not * be used by another host. * * If the prev owner was not our name, we delay for a rewnewal period * to monitor for current use from another host. * * If the lease is not free, we delay and monitor for a renewal period * to monitor (except when owner is our product_uuid per above.) * * If the lease is not free, and the owner does not match our host * name, then use a long delay to monitor for other hosts using it, * or to ensure a prev host using this lease is dead. */ is_free = (leader.timestamp == LEASE_FREE); is_same = !memcmp(leader.resource_name, our_host_name, NAME_ID_SIZE); if (!is_same) log_debug("delta_acquire new owner %.48s old owner %.48s", our_host_name, leader.resource_name); if (is_same || is_free) { if (is_same && is_free) no_delay = 1; if (is_same && our_host_name_matches_product_uuid) no_delay = 1; if (nodelay) no_delay = 1; log_space(sp, "delta_acquire %s owner, %s free, %s our_product_uuid, %s delay, other_io_timeout %d, flag_nodelay %d", is_same ? "same" : "new", is_free ? "is" : "not", our_host_name_matches_product_uuid ? "is" : "not", no_delay ? "no" : "short", other_io_timeout, nodelay); goto write_new; } /* * we need to ensure that a host_id cannot be acquired and released * sooner than host_dead_seconds because the change in host_id * ownership affects the host_id "liveness" determination used by paxos * leases, and the ownership of paxos leases cannot change until after * host_dead_seconds to ensure that the watchdog has fired. So, I * think we want the delay here to be the max of host_dead_seconds and * the D+6d delay. * * Per the algorithm in the paper, a delta lease can change ownership * in the while loop below after the delta_delay of D+6d. However, * because we use the change of delta lease ownership to directly * determine the change in paxos lease ownership, we need the delta * delay to also meet the delay requirements of the paxos leases. The * paxos leases cannot change ownership until a min of * host_dead_seconds to ensure the watchdog has fired. So, the timeout * we use here must be the max of the delta delay (D+6d) and * host_dead_seconds. */ other_host_dead_seconds = calc_host_dead_seconds(other_io_timeout); other_id_renewal_seconds = calc_id_renewal_seconds(other_io_timeout); log_space(sp, "delta_acquire other_host %.48s, timestamp %llu, other_io_timeout %d", leader.resource_name, (unsigned long long)leader.timestamp, other_io_timeout); delay = other_host_dead_seconds; delta_large_delay = other_id_renewal_seconds + (6 * other_io_timeout); if (delta_large_delay > delay) delay = delta_large_delay; while (1) { memcpy(&leader1, &leader, sizeof(struct leader_record)); log_space(sp, "delta_acquire delta_large_delay %d delay %d", delta_large_delay, delay); /* TODO: we could reread every several seconds to see if it has changed, so we can abort more quickly if so */ for (i = 0; i < delay; i++) { if (sp->external_remove || external_shutdown) { log_space(sp, "delta_acquire abort1 remove %d shutdown %d", sp->external_remove, external_shutdown); return SANLK_ERROR; } sleep(1); } error = delta_lease_leader_read(task, sp->sector_size, sp->io_timeout, disk, space_name, host_id, &leader, "delta_acquire_wait"); if (error < 0) { log_space(sp, "delta_acquire leader_read2 error %d", error); return error; } if (!memcmp(&leader1, &leader, sizeof(struct leader_record))) break; if (leader.timestamp == LEASE_FREE) break; log_erros(sp, "delta_acquire host_id %llu busy1 %llu %llu %llu %.48s", (unsigned long long)host_id, (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation, (unsigned long long)leader.timestamp, leader.resource_name); return SANLK_HOSTID_BUSY; } write_new: if (leader.owner_generation && (prev_generation != leader.owner_generation)) { log_erros(sp, "delta_acquire %llu wrong prev_generation %llu leader %llu %llu %llu %s", (unsigned long long)host_id, (unsigned long long)prev_generation, (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation, (unsigned long long)leader.timestamp, (leader.flags & LFL_DEAD_EXT) ? "dead_ext" : ""); } if (leader.flags & LFL_DEAD_EXT) { leader.flags &= ~LFL_DEAD_EXT; if (!prev_generation) { /* * This shouldn't happen. If it does happen, * some extra caution could be to do * next = leader.gen + 2, but skipping a gen * unnecessarily could also complicate external * assumptions about the normal gen sequence. */ next_generation = leader.owner_generation + 1; log_space(sp, "delta_acquire %llu leader dead_ext gen %llu no prev_generation next %llu", (unsigned long long)host_id, (unsigned long long)leader.owner_generation, (unsigned long long)next_generation); } else if (leader.owner_generation + 1 == prev_generation) { /* * our last generation was clobbered by a DEAD_EXT * write from another host, and then we failed before * renewing and replacing that late DEAD_EXT write. * We must +2 to avoid reusing the previous gen. */ next_generation = leader.owner_generation + 2; log_space(sp, "delta_acquire %llu leader dead_ext gen %llu clobbered prev_generation %llu next %llu", (unsigned long long)host_id, (unsigned long long)leader.owner_generation, (unsigned long long)prev_generation, (unsigned long long)next_generation); } else { /* * normal case when we restart after DEAD_EXT * was set in our lease. */ next_generation = leader.owner_generation + 1; log_space(sp, "delta_acquire %llu leader dead_ext gen %llu normal prev_generation %llu next %llu", (unsigned long long)host_id, (unsigned long long)leader.owner_generation, (unsigned long long)prev_generation, (unsigned long long)next_generation); } } else { /* common case */ next_generation = leader.owner_generation + 1; } new_ts = monotime(); leader.version = version; leader.timestamp = new_ts; leader.io_timeout = (sp->io_timeout & 0x00FF); leader.owner_id = host_id; leader.owner_generation = next_generation; memcpy(leader.resource_name, our_host_name, NAME_ID_SIZE); leader.checksum = 0; /* set below */ log_space(sp, "delta_acquire write %llu %llu %llu %.48s", (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation, (unsigned long long)leader.timestamp, leader.resource_name); leader_record_out(&leader, &leader_end); /* * N.B. must compute checksum after the data has been byte swapped. */ checksum = leader_checksum(&leader_end); leader.checksum = checksum; leader_end.checksum = cpu_to_le32(checksum); rv = write_sector(disk, sp->sector_size, host_id - 1, (char *)&leader_end, sizeof(struct leader_record), task, task->use_aio, sp->io_timeout, "delta_leader"); if (rv < 0) { log_space(sp, "delta_acquire write error %d", rv); return rv; } delta_local_copy_write(sp, next_generation); memcpy(&leader1, &leader, sizeof(struct leader_record)); if (no_delay) { usleep(10000); goto reread; } delay = 2 * other_io_timeout; log_space(sp, "delta_acquire delta_short_delay %d", delay); for (i = 0; i < delay; i++) { if (sp->external_remove || external_shutdown) { log_space(sp, "delta_acquire abort2 remove %d shutdown %d", sp->external_remove, external_shutdown); return SANLK_ERROR; } sleep(1); } reread: error = delta_lease_leader_read(task, sp->sector_size, sp->io_timeout, disk, space_name, host_id, &leader, "delta_acquire_check"); if (error < 0) { log_space(sp, "delta_acquire leader_read3 error %d", error); return error; } if (memcmp(&leader1, &leader, sizeof(struct leader_record))) { log_erros(sp, "delta_acquire host_id %llu busy2 %llu %llu %llu %.48s", (unsigned long long)host_id, (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation, (unsigned long long)leader.timestamp, leader.resource_name); return SANLK_HOSTID_BUSY; } log_space(sp, "delta_acquire done %llu %llu %llu", (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation, (unsigned long long)leader.timestamp); memcpy(leader_ret, &leader, sizeof(struct leader_record)); return SANLK_OK; } int delta_caw_lease_renew(struct task *task, struct space *sp, char *bitmap, struct delta_extra *extra, int prev_result, int *read_result, int log_renewal_level, struct leader_record *leader_last, struct leader_record *leader_ret, int *rd_ms, int *wr_ms) { struct leader_record leader; struct leader_record leader_end; struct sync_disk *disk = &sp->host_id_disk; uint64_t id_offset; uint64_t new_ts; uint64_t now; uint64_t host_id = sp->host_id; int sector_size = sp->sector_size; int align_size = sp->align_size; uint32_t checksum; int rv; if (!leader_last) { log_erros(sp, "delta_caw_renew no leader_last"); return -EINVAL; } *rd_ms = -1; *wr_ms = -1; *read_result = SANLK_ERROR; /* offset in bytes of our leader_record in the lease area */ id_offset = (host_id - 1) * sector_size; if (id_offset > align_size) { log_erros(sp, "delta_caw_renew bad offset %llu align_size %d", (unsigned long long)id_offset, align_size); return -EINVAL; } /* * The first call to renew will allocate this special task->iobuf * into which the full lease area is read by each renew. * * NB. this task->iobuf is also copied by the lockspace thread * into renewal_read_buf, which is then copied in the main loop * by check_our_lease and passed to check_other_leases. */ if (!task->iobuf) { char **p_iobuf; int read_mem_align = getpagesize(); int is_huge = 0; int iobuf_len = sp->align_size; p_iobuf = &task->iobuf; /* * Minimum hugepage is 2MB, so for 1MB lease area we increase * the iobuf size and alignment to 2MB, wasting a MB of space * to more likely avoid splitting the i/o (read len remains * 1MB even though iobuf_len is extended to 2MB.) * iobuf_len begins being set to align_size (1MB, 2MB, 4MB, 8MB). */ if (com.use_hugepages && ((iobuf_len == ONE_MB_IN_BYTES) || !(iobuf_len % TWO_MB_IN_BYTES))) { read_mem_align = TWO_MB_IN_BYTES; if (iobuf_len < TWO_MB_IN_BYTES) iobuf_len = TWO_MB_IN_BYTES; is_huge = 1; } rv = posix_memalign((void *)p_iobuf, read_mem_align, iobuf_len); if (rv) { log_erros(sp, "dela_caw_renew memalign rv %d", rv); return -ENOMEM; } if (is_huge) { char *iobuf = task->iobuf; char *bp; madvise(iobuf, iobuf_len, MADV_HUGEPAGE); /* allocate the pages */ for (bp = iobuf; bp < iobuf+iobuf_len; bp += TWO_MB_IN_BYTES) memset(bp, 0, 1); } } if (log_renewal_level != -1) log_level(sp->space_id, 0, NULL, log_renewal_level, "delta_caw_renew begin read"); /* * Read the full lease area into task->iobuf. * * This is done with a sync io read on the host_id_disk * (not the underlying caw dev used by CAW SGIO.) * * Technically, renew would not need to read the lease * area, or the local host_id lease, and could just * issue a compare-and-write to update the leader * using the previous renewal buffer as the prev data. * But, reading the full lease acts as a monitor for * the liveness of all the other host_id leases. * * In the future, the delta caw lease renewal might * just use a single caw each time, and reading the * entire lease area (to monitor other leases) could * be done separately, e.g. by a different thread on * a different schedule, and not by the renewal thread. * This would let the renewal thread focus on the * critical updates and not be delayed by the less * important monitor reads. * * read len remains align_size even if iobuf_len * is the larger hugepage size. */ rv = read_iobuf(disk->fd, disk->offset, task->iobuf, align_size, task, NO_AIO, 0, rd_ms); if (rv) { /* This is NO_AIO so no need to check AIO_TIMEOUT and set task->iobuf = NULL */ log_erros(sp, "delta_caw_renew read rv %d offset %llu %s", rv, (unsigned long long)disk->offset, disk->path); return rv; } *read_result = SANLK_OK; memcpy(&leader_end, task->iobuf+id_offset, sizeof(struct leader_record)); checksum = leader_checksum(&leader_end); /* compute checksum before byte swapping */ leader_record_in(&leader_end, &leader); rv = verify_leader(disk, sp->space_name, host_id, &leader, checksum, "delta_caw_renew"); if (rv < 0) { log_erros(sp, "delta_caw_renew verify_leader error %d", rv); return rv; } /* * This can happen because another host can declare we have failed * (e.g. our renewals are delayed), but we are still trying to run * and renew our lease. The other host removes our persistent * reservation, and sets DEAD_EXT in our lease. We are able to * read our lease (still possible without a PR), and see here that * another host has said we're dead. Our PR has been removed, so * there's no hope of us continuing normally, so treat this ls as * if it is corrupt. * * I don't think there will be a scenario where leader.owner_generation * is less than leader_last->owner_generation, as is possible in the * non-CAW case, because CAW should resolve any race between us starting * the lockspace and another host setting DEAD_EXT. */ if (leader.flags & LFL_DEAD_EXT) { log_erros(sp, "delta_caw_renew read leader %llu %llu %llu dead_ext from 0x%llx last %llu %llu %llu", (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation, (unsigned long long)leader.timestamp, (unsigned long long)leader.unused1, (unsigned long long)leader_last->owner_id, (unsigned long long)leader_last->owner_generation, (unsigned long long)leader_last->timestamp); return SANLK_RENEW_DEAD_EXT; } /* We can't always memcmp(&leader, leader_last) because previous writes may have failed and we don't know if they were actually written or not. We can verify that we're still the owner, though, which is the main thing to verify. */ if (leader.owner_id != leader_last->owner_id || leader.owner_generation != leader_last->owner_generation || memcmp(leader.resource_name, leader_last->resource_name, NAME_ID_SIZE)) { log_erros(sp, "delta_caw_renew not owner"); log_leader_error(0, sp->space_name, host_id, &sp->host_id_disk, leader_last, "delta_caw_renew_last"); log_leader_error(0, sp->space_name, host_id, &sp->host_id_disk, &leader, "delta_caw_renew_read"); return SANLK_RENEW_OWNER; } if (prev_result == SANLK_OK && memcmp(&leader, leader_last, sizeof(struct leader_record))) { log_erros(sp, "delta_caw_renew reread mismatch"); log_leader_error(0, sp->space_name, host_id, &sp->host_id_disk, leader_last, "delta_caw_renew_last"); log_leader_error(0, sp->space_name, host_id, &sp->host_id_disk, &leader, "delta_caw_renew_read"); return SANLK_RENEW_DIFF; } if (leader.io_timeout != sp->io_timeout) { log_erros(sp, "delta_caw_renew io_timeout changed disk %d sp %d", leader.io_timeout, sp->io_timeout); leader.io_timeout = (sp->io_timeout & 0x00FF); } /* * Renewal updates the leader timestamp field. */ /* Copy our sector from iobuf into sp->caw_data (both next and prev). */ memcpy(sp->caw_data, task->iobuf+id_offset, sector_size); memcpy(sp->caw_data + sector_size, task->iobuf+id_offset, sector_size); new_ts = monotime(); if (log_renewal_level != -1) log_level(sp->space_id, 0, NULL, log_renewal_level, "delta_caw_renew begin write for new ts %llu", (unsigned long long)new_ts); if (leader.timestamp >= new_ts) log_erros(sp, "delta_caw_renew timestamp too small"); leader.timestamp = new_ts; leader.checksum = 0; /* set below */ /* TODO: rename the leader fields */ if (extra) { leader.write_id = extra->field1; leader.write_generation = extra->field2; leader.write_timestamp = extra->field3; } leader_record_out(&leader, &leader_end); /* * N.B. must compute checksum after the data has been byte swapped. */ checksum = leader_checksum(&leader_end); leader.checksum = checksum; leader_end.checksum = cpu_to_le32(checksum); memcpy(sp->caw_data + sector_size, &leader_end, sizeof(struct leader_record)); memcpy(sp->caw_data + sector_size + LEADER_RECORD_MAX, bitmap, HOSTID_BITMAP_SIZE); rv = delta_compare_and_write(task, sp, host_id, sp->caw_data, wr_ms); now = monotime(); if (rv < 0) { log_erros(sp, "delta_caw_renew write time %llu error %d", (unsigned long long)(now - new_ts), rv); return rv; } if (now - new_ts >= sp->io_timeout) log_erros(sp, "delta_caw_renew long write time %llu sec", (unsigned long long)(now - new_ts)); memcpy(leader_ret, &leader, sizeof(struct leader_record)); return SANLK_OK; } int delta_lease_renew(struct task *task, struct space *sp, struct sync_disk *disk, char *space_name, char *bitmap, struct delta_extra *extra, int prev_result, int *read_result, int log_renewal_level, struct leader_record *leader_last, struct leader_record *leader_ret, int *rd_ms, int *wr_ms) { struct leader_record leader; struct leader_record leader_end; char **p_iobuf; char **p_wbuf; char *wbuf; struct timespec begin, end, diff; uint32_t checksum; uint32_t reap_timeout_msec; uint64_t host_id, id_offset, new_ts, now; int sector_size = sp->sector_size; int align_size = sp->align_size; /* 1,2,4,8 MB depending on sector_size */ int iobuf_len = sp->align_size; /* may increase (from 1MB) for hugepage */ int read_len = sp->align_size; /* not increased (from 1MB) for hugepage */ int rv; if (!leader_last) { log_erros(sp, "delta_renew no leader_last"); return -EINVAL; } *rd_ms = -1; *wr_ms = -1; *read_result = SANLK_ERROR; host_id = leader_last->owner_id; /* offset of our leader_record */ id_offset = (host_id - 1) * sector_size; if (id_offset > align_size) { log_erros(sp, "delta_renew bad offset %llu align_size %d", (unsigned long long)id_offset, align_size); return -EINVAL; } /* if the previous renew timed out in this initial read, and that read is now complete, we can use that result here instead of discarding it and doing another. */ if (prev_result == SANLK_AIO_TIMEOUT) { if (!task->read_iobuf_timeout_aicb) { /* shouldn't happen, when do_linux_aio returned AIO_TIMEOUT it should have set read_iobuf_timeout_aicb */ log_erros(sp, "delta_renew reap no aicb"); goto skip_reap; } if (!task->iobuf) { /* shouldn't happen */ log_erros(sp, "delta_renew reap no iobuf"); goto skip_reap; } log_space(sp, "delta_renew begin reap"); if (!sp->renewal_read_extend_sec) { /* only wait .5 sec when trying to reap a prev io to clear it */ reap_timeout_msec = 500; } else { /* effectively continue/extend the read phase from the previous renewal */ reap_timeout_msec = sp->renewal_read_extend_sec * 1000; } clock_gettime(CLOCK_MONOTONIC_RAW, &begin); rv = read_iobuf_reap(disk->fd, disk->offset, task->iobuf, read_len, task, reap_timeout_msec); log_space(sp, "delta_renew reap %d", rv); if (!rv) { /* read time for this renewal is the io_timeout length for the previous read plus the time spent in reap. */ clock_gettime(CLOCK_MONOTONIC_RAW, &end); ts_diff(&begin, &end, &diff); *rd_ms = (diff.tv_sec * 1000) + (diff.tv_nsec / 1000000) + (sp->io_timeout * 1000); task->read_iobuf_timeout_aicb = NULL; goto read_done; } skip_reap: /* abandon the previous timed out read and try a new one from scratch. the current task->iobuf mem will freed when timeout_aicb completes sometime */ task->read_iobuf_timeout_aicb = NULL; task->iobuf = NULL; } if (task->read_iobuf_timeout_aicb) { /* this could happen get here if there was another read between renewal reads, which timed out and caused read_iobuf_timeout_aicb to be set; I don't think there are any cases where that would happen, though. we could avoid this confusion by passing back the timed out aicb along with SANLK_AIO_TIMEOUT, and only save the timed out aicb when we want to try to reap it later. */ log_space(sp, "delta_renew timeout_aicb is unexpectedly %p iobuf %p", task->read_iobuf_timeout_aicb, task->iobuf); task->read_iobuf_timeout_aicb = NULL; task->iobuf = NULL; } /* * NB. this task->iobuf is also copied by the lockspace thread * into renewal_read_buf, which is then copied in the main loop * by check_our_lease and passed to check_other_leases. */ if (!task->iobuf) { int read_mem_align = getpagesize(); int is_huge = 0; /* this will happen the first time renew is called, and after a timed out renewal read fails to be reaped (see task->iobuf = NULL above) */ p_iobuf = &task->iobuf; /* * Minimum hugepage is 2MB, so for 1MB lease area we increase * the iobuf size and alignment to 2MB, wasting a MB of space * to more likely avoid splitting the i/o (read_len remains * 1MB even though iobuf_len is extended to 2MB.) * iobuf_len begins being set to align_size (1MB, 2MB, 4MB, 8MB). */ if (com.use_hugepages && ((iobuf_len == ONE_MB_IN_BYTES) || !(iobuf_len % TWO_MB_IN_BYTES))) { read_mem_align = TWO_MB_IN_BYTES; if (iobuf_len < TWO_MB_IN_BYTES) iobuf_len = TWO_MB_IN_BYTES; is_huge = 1; } rv = posix_memalign((void *)p_iobuf, read_mem_align, iobuf_len); if (rv) { log_erros(sp, "dela_renew memalign rv %d", rv); rv = -ENOMEM; } if (is_huge) { char *iobuf = task->iobuf; char *bp; madvise(iobuf, iobuf_len, MADV_HUGEPAGE); /* allocate the pages */ for (bp = iobuf; bp < iobuf+iobuf_len; bp += TWO_MB_IN_BYTES) memset(bp, 0, 1); } } if (log_renewal_level != -1) log_level(sp->space_id, 0, NULL, log_renewal_level, "delta_renew begin read"); rv = read_iobuf(disk->fd, disk->offset, task->iobuf, read_len, task, task->use_aio, sp->io_timeout, rd_ms); if (rv) { /* the next time delta_lease_renew() is called, prev_result will be this rv. If this rv is SANLK_AIO_TIMEOUT, we'll try to reap the event */ if (rv == SANLK_AIO_TIMEOUT) log_erros(sp, "delta_renew read timeout %u sec offset %llu %s", sp->io_timeout, (unsigned long long)disk->offset, disk->path); else log_erros(sp, "delta_renew read rv %d offset %llu %s", rv, (unsigned long long)disk->offset, disk->path); return rv; } read_done: *read_result = SANLK_OK; memcpy(&leader_end, task->iobuf+id_offset, sizeof(struct leader_record)); /* N.B. compute checksum before byte swapping */ checksum = leader_checksum(&leader_end); leader_record_in(&leader_end, &leader); rv = verify_leader(disk, space_name, host_id, &leader, checksum, "delta_renew"); if (rv < 0) { log_erros(sp, "delta_renew verify_leader error %d", rv); return rv; } /* * Someone has set DEAD_EXT in our lease, likely along with * our previous generation. This would be a race between us * joining (acquiring our lease), and another host declaring * us dead (ext) for our prior failure. In this case, we want * to renew/write the values that we would have written if the * other host had not clobbered our previous write. * * If we find our current generation has had DEAD_EXT set, * it means that another host has seen us enter the fail state, * has removed our persistent reservation, and set DEAD_EXT. * This still allows us to read our lease, and see the DEAD_EXT * set here. There's no hope of continuing normally since we * cannot write, so handle this like our lease is corrupted. */ if (leader.flags & LFL_DEAD_EXT) { if (leader.owner_generation < leader_last->owner_generation) { log_erros(sp, "delta_renew %llu late dead_ext gen %llu from %llu", (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation, (unsigned long long)leader.unused1); goto write_renew; } else { log_erros(sp, "delta_renew %llu current gen %llu is dead_ext from %llu", (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation, (unsigned long long)leader.unused1); return SANLK_RENEW_DEAD_EXT; } } /* We can't always memcmp(&leader, leader_last) because previous writes may have timed out and we don't know if they were actually written or not. We can definitely verify that we're still the owner, though, which is the main thing we need to know. */ if (leader.owner_id != leader_last->owner_id || leader.owner_generation != leader_last->owner_generation || memcmp(leader.resource_name, leader_last->resource_name, NAME_ID_SIZE)) { log_erros(sp, "delta_renew not owner"); log_leader_error(0, space_name, host_id, disk, leader_last, "delta_renew_last"); log_leader_error(0, space_name, host_id, disk, &leader, "delta_renew_read"); return SANLK_RENEW_OWNER; } if (prev_result == SANLK_OK && memcmp(&leader, leader_last, sizeof(struct leader_record))) { log_erros(sp, "delta_renew reread mismatch"); log_leader_error(0, space_name, host_id, disk, leader_last, "delta_renew_last"); log_leader_error(0, space_name, host_id, disk, &leader, "delta_renew_read"); return SANLK_RENEW_DIFF; } if (leader.io_timeout != sp->io_timeout) { log_erros(sp, "delta_renew io_timeout changed disk %d sp %d", leader.io_timeout, sp->io_timeout); leader.io_timeout = (sp->io_timeout & 0x00FF); } write_renew: new_ts = monotime(); if (log_renewal_level != -1) log_level(sp->space_id, 0, NULL, log_renewal_level, "delta_renew begin write for new ts %llu", (unsigned long long)new_ts); if (leader.timestamp >= new_ts) log_erros(sp, "delta_renew timestamp too small"); leader.timestamp = new_ts; leader.flags &= ~LFL_DEAD_EXT; leader.checksum = 0; /* set below */ /* TODO: rename the leader fields */ if (extra) { leader.write_id = extra->field1; leader.write_generation = extra->field2; leader.write_timestamp = extra->field3; } p_wbuf = &wbuf; rv = posix_memalign((void *)p_wbuf, getpagesize(), sector_size); if (rv) { log_erros(sp, "dela_renew write memalign rv %d", rv); return -ENOMEM; } memset(wbuf, 0, sector_size); leader_record_out(&leader, &leader_end); /* * N.B. must compute checksum after the data has been byte swapped. */ checksum = leader_checksum(&leader_end); leader.checksum = checksum; leader_end.checksum = cpu_to_le32(checksum); memcpy(wbuf, &leader_end, sizeof(struct leader_record)); memcpy(wbuf+LEADER_RECORD_MAX, bitmap, HOSTID_BITMAP_SIZE); /* extend io timeout for this one write; we need to give this write every chance to succeed, and there's no point in letting it time out. there's nothing we would do but retry it, and timing out and retrying unnecessarily would probably be counter productive. */ rv = write_iobuf(disk->fd, disk->offset+id_offset, wbuf, sector_size, task, task->use_aio, calc_host_dead_seconds(sp->io_timeout), wr_ms); if (rv != SANLK_AIO_TIMEOUT) free(wbuf); now = monotime(); if (rv < 0) { log_erros(sp, "delta_renew write time %llu error %d", (unsigned long long)(now - new_ts), rv); return rv; } if (now - new_ts >= sp->io_timeout) log_erros(sp, "delta_renew long write time %llu sec", (unsigned long long)(now - new_ts)); /* the paper shows doing a delay and another read here, but it seems unnecessary since we do the same at the beginning of the next renewal */ memcpy(leader_ret, &leader, sizeof(struct leader_record)); return SANLK_OK; } int delta_caw_lease_release(struct task *task, struct space *sp, struct sync_disk *disk, char *space_name GNUC_UNUSED, struct leader_record *leader_last, struct leader_record *leader_ret) { struct leader_record leader; struct leader_record leader_end; struct leader_record prev_leader; char *rdbuf = NULL; char **p_rdbuf; uint64_t host_id; uint32_t checksum; int retries = 0; int error; int rv; if (!leader_last) return -EINVAL; host_id = leader_last->owner_id; log_space(sp, "delta_caw_release begin %.48s:%llu", sp->space_name, (unsigned long long)host_id); memcpy(&leader, leader_last, sizeof(struct leader_record)); leader.timestamp = LEASE_FREE; leader.checksum = 0; /* set below */ leader_record_out(&leader, &leader_end); /* compute checksum on byte swapped data */ checksum = leader_checksum(&leader_end); leader.checksum = checksum; leader_end.checksum = cpu_to_le32(checksum); /* * If our last compare_and_write (from acquire or renew) was * successful, then the second half of caw_data is what's currently on * disk, so copy that to the first half. * * If our last compare_and_write failed, then we don't know * what's currently on disk, so reread it (delta_lease_leader_read_buf), * then retry the compare_and_write with the new leader. * (We might avoid this by keeping a separate buffer of our * last successful compare_and_write data, and use that here * instead of the second half of sp->caw_data.) */ memmove(sp->caw_data, sp->caw_data + sp->sector_size, sp->sector_size); memcpy(sp->caw_data + sp->sector_size, &leader_end, sizeof(struct leader_record)); retry_caw: error = delta_compare_and_write(task, sp, host_id, sp->caw_data, NULL); if (error == SANLK_CAW_MISCOMPARE) { /* this is probably from initial acquire failure? other reasons? */ p_rdbuf = &rdbuf; rv = posix_memalign((void *)p_rdbuf, getpagesize(), sp->sector_size); if (rv) { log_erros(sp, "delta_caw_release miscompare reread nomem"); return SANLK_CAW_MISCOMPARE; } rv = delta_lease_leader_read_buf(task, sp->sector_size, NO_AIO, sp->io_timeout, &sp->host_id_disk, sp->space_name, sp->host_id, &prev_leader, rdbuf, "caw_lease_release"); if (rv == SANLK_AIO_TIMEOUT) { log_erros(sp, "delta_caw_release miscompare reread failed %d", rv); return SANLK_CAW_MISCOMPARE; } if (rv < 0) { log_erros(sp, "delta_caw_release miscompare reread failed %d", rv); free(rdbuf); return SANLK_CAW_MISCOMPARE; } log_erros(sp, "delta_caw_release miscompare reread %llu %llu %llu %x last %llu %llu %llu %x", (unsigned long long)prev_leader.owner_id, (unsigned long long)prev_leader.owner_generation, (unsigned long long)prev_leader.timestamp, prev_leader.flags, (unsigned long long)leader_last->owner_id, (unsigned long long)leader_last->owner_generation, (unsigned long long)leader_last->timestamp, leader_last->flags); memcpy(&leader, &prev_leader, sizeof(struct leader_record)); leader.timestamp = LEASE_FREE; leader.checksum = 0; leader_record_out(&leader, &leader_end); checksum = leader_checksum(&leader_end); leader.checksum = checksum; leader_end.checksum = cpu_to_le32(checksum); memcpy(sp->caw_data, rdbuf, sp->sector_size); memcpy(sp->caw_data + sp->sector_size, rdbuf, sp->sector_size); memcpy(sp->caw_data + sp->sector_size, &leader_end, sizeof(struct leader_record)); free(rdbuf); if (!retries) { retries++; goto retry_caw; } } if (error < 0) { log_erros(sp, "delta_caw_release compare_and_write error %d", error); return error; } log_space(sp, "delta_caw_release done %llu %llu %llu", (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation, (unsigned long long)leader.timestamp); memcpy(leader_ret, &leader, sizeof(struct leader_record)); return SANLK_OK; } int delta_lease_release(struct task *task, struct space *sp, struct sync_disk *disk, char *space_name GNUC_UNUSED, struct leader_record *leader_last, struct leader_record *leader_ret) { struct leader_record leader; struct leader_record leader_end; uint64_t host_id; uint32_t checksum; int rv; if (!leader_last) return -EINVAL; host_id = leader_last->owner_id; log_space(sp, "delta_release begin %.48s:%llu", sp->space_name, (unsigned long long)host_id); memcpy(&leader, leader_last, sizeof(struct leader_record)); leader.timestamp = LEASE_FREE; leader.checksum = 0; /* set below */ leader_record_out(&leader, &leader_end); /* * N.B. must compute checksum after the data has been byte swapped. */ checksum = leader_checksum(&leader_end); leader.checksum = checksum; leader_end.checksum = cpu_to_le32(checksum); rv = write_sector(disk, sp->sector_size, host_id - 1, (char *)&leader_end, sizeof(struct leader_record), task, task->use_aio, sp->io_timeout, "delta_leader"); if (rv < 0) { log_space(sp, "delta_release write error %d", rv); return rv; } log_space(sp, "delta_release done %llu %llu %llu", (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation, (unsigned long long)leader.timestamp); memcpy(leader_ret, &leader, sizeof(struct leader_record)); return SANLK_OK; } int delta_lease_init_host(struct task *task, struct sanlk_lockspace *ls, char *our_host_name, uint64_t generation, uint64_t timestamp, int io_timeout, int try_caw, struct sync_disk *disk) { struct leader_record leader_end; struct leader_record leader; char *iobuf, **p_iobuf; int iobuf_len; int sector_size = 0; int align_size = 0; int max_hosts = 0; int write_io_timeout; int no_timeout = (ls->flags & SANLK_LSF_NO_TIMEOUT) ? 1 : 0; int use_caw = try_caw; uint32_t checksum; uint64_t new_ts; uint64_t offset; int rv; if (no_timeout && com.create_delta_34) return -EINVAL; if (!io_timeout) io_timeout = com.io_timeout; if (timestamp == 0) new_ts = LEASE_FREE; else if (timestamp == 1) new_ts = monotime(); else new_ts = timestamp; rv = sizes_from_flags(ls->flags, §or_size, &align_size, &max_hosts, "LSF"); if (rv) return rv; if (!sector_size) { /* sector/align flags were not set, use historical defaults */ sector_size = disk->sector_size; align_size = sector_size_to_align_size_old(sector_size); max_hosts = DEFAULT_MAX_HOSTS; } iobuf_len = sector_size; p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), iobuf_len); if (rv) return rv; memset(iobuf, 0, iobuf_len); memset(&leader, 0, sizeof(struct leader_record)); leader.magic = DELTA_DISK_MAGIC; if (com.create_delta_34) leader.version = DELTA_DISK_VERSION_34; else if (com.create_delta_40) leader.version = DELTA_DISK_VERSION_40; else leader.version = DELTA_DISK_VERSION_MAJOR | DELTA_DISK_VERSION_MINOR; leader.flags = leader_align_flag_from_size(align_size); if (no_timeout) leader.flags |= LFL_NO_TIMEOUT; if (use_caw) leader.flags |= LFL_CAW; leader.sector_size = sector_size; leader.owner_id = ls->host_id; leader.owner_generation = generation; leader.max_hosts = 1; leader.timestamp = new_ts; leader.io_timeout = (io_timeout & 0x00FF); memcpy(leader.space_name, ls->name, NAME_ID_SIZE); if (our_host_name) memcpy(leader.resource_name, our_host_name, NAME_ID_SIZE); leader.checksum = 0; /* set below */ leader_record_out(&leader, &leader_end); /* * N.B. must compute checksum after the data has been byte swapped. */ checksum = leader_checksum(&leader_end); leader.checksum = checksum; leader_end.checksum = cpu_to_le32(checksum); memcpy(iobuf, &leader_end, sizeof(struct leader_record)); /* * The io_timeout arg is a part of the lockspace logic, and * determines how the lockspace times out. The process of * initializing the lease on disk can to use a longer timeout * than the algorithm uses. */ if (com.write_init_io_timeout) write_io_timeout = com.write_init_io_timeout; else write_io_timeout = io_timeout; offset = disk->offset + (sector_size * (ls->host_id - 1)); rv = write_iobuf(disk->fd, offset, iobuf, iobuf_len, task, task->use_aio, write_io_timeout, NULL); if (rv < 0) goto out; out: if (rv != SANLK_AIO_TIMEOUT) free(iobuf); return rv; } /* the host_id lease area begins disk->offset bytes from the start of block device disk->path */ int delta_lease_init(struct task *task, struct sanlk_lockspace *ls, int io_timeout, int try_caw, struct sync_disk *disk) { struct leader_record leader_first; struct leader_record leader_end; struct leader_record leader; char *iobuf, **p_iobuf; int iobuf_len; int sector_size = 0; int align_size = 0; int max_hosts = 0; int write_io_timeout; int no_timeout = (ls->flags & SANLK_LSF_NO_TIMEOUT) ? 1 : 0; int use_caw = try_caw; int i, rv, err; uint32_t checksum; if (no_timeout && com.create_delta_34) return -EINVAL; if (!io_timeout) io_timeout = com.io_timeout; rv = sizes_from_flags(ls->flags, §or_size, &align_size, &max_hosts, "LSF"); if (rv) return rv; if (!sector_size) { /* sector/align flags were not set, use historical defaults */ sector_size = disk->sector_size; align_size = sector_size_to_align_size_old(sector_size); max_hosts = DEFAULT_MAX_HOSTS; } if (try_caw) { err = compare_and_write_test(task, disk->path, disk->offset, sector_size, com.delta_caw_verify); if (err < 0) use_caw = 0; /* This likely indicates the device supports CAW, but not the FUA/DPO flags */ if ((err == SANLK_CAW_ILLEGAL) && (com.caw_dpo || com.caw_fua)) log_warn("Check CAW FUA/DPO settings"); } iobuf_len = align_size; p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), iobuf_len); if (rv) return rv; memset(iobuf, 0, iobuf_len); /* host_id N is block offset N-1 */ for (i = 0; i < max_hosts; i++) { memset(&leader, 0, sizeof(struct leader_record)); leader.magic = DELTA_DISK_MAGIC; if (com.create_delta_34) leader.version = DELTA_DISK_VERSION_34; else if (com.create_delta_40) leader.version = DELTA_DISK_VERSION_40; else leader.version = DELTA_DISK_VERSION_MAJOR | DELTA_DISK_VERSION_MINOR; leader.flags = leader_align_flag_from_size(align_size); if (no_timeout) leader.flags |= LFL_NO_TIMEOUT; if (use_caw) leader.flags |= LFL_CAW; leader.sector_size = sector_size; leader.max_hosts = 1; leader.timestamp = LEASE_FREE; leader.io_timeout = (io_timeout & 0x00FF); memcpy(leader.space_name, ls->name, NAME_ID_SIZE); leader.checksum = 0; /* set below */ /* make the first record invalid so we can do a single atomic write below to commit the whole thing */ if (!i) { leader.magic = 0; memcpy(&leader_first, &leader, sizeof(struct leader_record)); } leader_record_out(&leader, &leader_end); /* * N.B. must compute checksum after the data has been byte swapped. */ checksum = leader_checksum(&leader_end); leader.checksum = checksum; leader_end.checksum = cpu_to_le32(checksum); memcpy(iobuf + (i * sector_size), &leader_end, sizeof(struct leader_record)); } /* * The io_timeout arg is a part of the lockspace logic, and * determines how the lockspace times out. The process of * initializing the lease on disk can to use a longer timeout * than the algorithm uses. */ if (com.write_init_io_timeout) write_io_timeout = com.write_init_io_timeout; else write_io_timeout = io_timeout; rv = write_iobuf(disk->fd, disk->offset, iobuf, iobuf_len, task, task->use_aio, write_io_timeout, NULL); if (rv < 0) goto out; /* commit the whole lockspace by making the first record valid */ leader_first.magic = DELTA_DISK_MAGIC; leader_first.checksum = 0; /* set below */ leader_record_out(&leader_first, &leader_end); /* * N.B. must compute checksum after the data has been byte swapped. */ checksum = leader_checksum(&leader_end); leader_first.checksum = checksum; leader_end.checksum = cpu_to_le32(checksum); memcpy(iobuf, &leader_end, sizeof(struct leader_record)); rv = write_iobuf(disk->fd, disk->offset, iobuf, sector_size, task, task->use_aio, write_io_timeout, NULL); out: if (rv != SANLK_AIO_TIMEOUT) free(iobuf); return rv; } /* * Rewrites all delta leases in the lockspace like delta_lease_init(), * but keeping existing leader values while modifying leader flags. * * TODO: not yet able to change the caw flag on lockspace delta leases. * (Also updating resource paxos leases in the lockspace.) */ int delta_lease_update_lockspace(struct task *task, struct sanlk_lockspace *ls, int io_timeout, struct sync_disk *disk) { struct leader_record leader_end; struct leader_record leader; char *iobuf, **p_iobuf; int iobuf_len; int sector_size = 0; int align_size = 0; int max_hosts = 0; int write_io_timeout; int no_timeout = (ls->flags & SANLK_LSF_NO_TIMEOUT) ? 1 : 0; int i, rv; uint32_t checksum; if (no_timeout && com.create_delta_34) return -EINVAL; if (!io_timeout) io_timeout = com.io_timeout; rv = sizes_from_flags(ls->flags, §or_size, &align_size, &max_hosts, "LSF"); if (rv) return rv; if (!sector_size || !align_size || !max_hosts) { log_error("delta_lease_update_lockspace requires LSF flags for size and alignment."); return -EINVAL; } iobuf_len = align_size; p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), iobuf_len); if (rv) return rv; memset(iobuf, 0, iobuf_len); rv = read_iobuf(disk->fd, disk->offset, iobuf, iobuf_len, task, task->use_aio, io_timeout, NULL); if (rv < 0) goto out; /* host_id N is block offset N-1 */ for (i = 0; i < max_hosts; i++) { memcpy(&leader_end, iobuf + (i * sector_size), sizeof(struct leader_record)); leader_record_in(&leader_end, &leader); if (leader.magic != DELTA_DISK_MAGIC) continue; if (no_timeout) leader.flags |= LFL_NO_TIMEOUT; else leader.flags &= ~LFL_NO_TIMEOUT; leader.checksum = 0; /* set below */ leader_record_out(&leader, &leader_end); /* * N.B. must compute checksum after the data has been byte swapped. */ checksum = leader_checksum(&leader_end); leader.checksum = checksum; leader_end.checksum = cpu_to_le32(checksum); memcpy(iobuf + (i * sector_size), &leader_end, sizeof(struct leader_record)); } if (com.write_init_io_timeout) write_io_timeout = com.write_init_io_timeout; else write_io_timeout = io_timeout; rv = write_iobuf(disk->fd, disk->offset, iobuf, iobuf_len, task, task->use_aio, write_io_timeout, NULL); if (rv < 0) goto out; out: if (rv != SANLK_AIO_TIMEOUT) free(iobuf); return rv; } sanlock/src/delta_lease.h000066400000000000000000000112111516326262600157170ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __DELTA_LEASE_H__ #define __DELTA_LEASE_H__ int delta_lease_leader_read(struct task *task, int sector_size, int io_timeout, struct sync_disk *disk, char *space_name, uint64_t host_id, struct leader_record *leader_ret, const char *caller); int delta_lease_leader_read_buf(struct task *task, int sector_size, int use_aio, int io_timeout, struct sync_disk *disk, char *space_name, uint64_t host_id, struct leader_record *leader_ret, char *buf, const char *caller); int delta_lease_acquire(struct task *task, struct space *sp, struct sync_disk *disk, char *space_name, char *our_host_name, uint64_t host_id, int nodelay, struct leader_record *leader_ret); int delta_lease_renew(struct task *task, struct space *sp, struct sync_disk *disk, char *space_name, char *bitmap, struct delta_extra *extra, int prev_result, int *read_result, int log_renewal_level, struct leader_record *leader_last, struct leader_record *leader_ret, int *rd_ms, int *wr_ms); int delta_lease_release(struct task *task, struct space *sp, struct sync_disk *disk, char *space_name GNUC_UNUSED, struct leader_record *leader_last, struct leader_record *leader_ret); int delta_lease_init(struct task *task, struct sanlk_lockspace *ls, int io_timeout, int try_caw, struct sync_disk *disk); int delta_lease_init_host(struct task *task, struct sanlk_lockspace *ls, char *our_host_name, uint64_t generation, uint64_t timestamp, int io_timeout, int try_caw, struct sync_disk *disk); int delta_read_lockspace(struct task *task, struct sync_disk *disk, int sector_size, uint64_t host_id, struct sanlk_lockspace *ls, int io_timeout, struct sanlk_host *hs); int delta_read_lockspace_sizes(struct task *task, struct sync_disk *disk, uint64_t host_id, int use_aio, int io_timeout, int *sector_size, int *align_size, int *is_caw); int delta_lease_leader_clobber(struct task *task, int io_timeout, struct sync_disk *disk, uint64_t host_id, struct leader_record *leader, const char *caller); int delta_lease_update_lockspace(struct task *task, struct sanlk_lockspace *ls, int io_timeout, struct sync_disk *disk); int delta_caw_lease_acquire(struct task *task, struct space *sp, char *our_host_name, struct leader_record *leader_ret); int delta_caw_lease_renew(struct task *task, struct space *sp, char *bitmap, struct delta_extra *extra, int prev_result, int *read_result, int log_renewal_level, struct leader_record *leader_last, struct leader_record *leader_ret, int *rd_ms, int *wr_ms); int delta_caw_lease_release(struct task *task, struct space *sp, struct sync_disk *disk, char *space_name GNUC_UNUSED, struct leader_record *leader_last, struct leader_record *leader_ret); int delta_caw_lease_leader_clobber(struct task *task, int io_timeout, struct sync_disk *disk, uint64_t host_id, struct leader_record *leader, int sector_size, uint32_t caw_major, uint32_t caw_minor, uint64_t caw_offset, char *caw_iobuf, const char *caller); int delta_compare_and_write(struct task *task, struct space *sp, uint64_t host_id, char *caw_data, int *wr_ms); #endif sanlock/src/direct.c000066400000000000000000000624521516326262600147370ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock_internal.h" #include "sanlock_admin.h" #include "diskio.h" #include "ondisk.h" #include "log.h" #include "resource.h" #include "direct.h" #include "paxos_lease.h" #include "delta_lease.h" #include "timeouts.h" /* * the caller sets sd.offset to the location from the start of disk (in bytes) where * a data struct should be read and checked for sector/align sizes. */ static int direct_read_leader_sizes(struct task *task, struct sync_disk *sd, int *sector_size, int *align_size) { struct leader_record *lr_end; struct leader_record lr_in; char *data; int datalen; int rv; datalen = 4096; data = malloc(datalen); if (!data) return -ENOMEM; memset(data, 0, datalen); rv = read_sector(sd, 4096, 0, data, datalen, task, task->use_aio, com.io_timeout, "read_sector_size"); if (rv < 0) { free(data); return rv; } lr_end = (struct leader_record *)data; leader_record_in(lr_end, &lr_in); free(data); if ((lr_in.magic == DELTA_DISK_MAGIC) || (lr_in.magic == PAXOS_DISK_MAGIC)) { *sector_size = lr_in.sector_size; *align_size = leader_align_size_from_flag(lr_in.flags); if (!*align_size) *align_size = sector_size_to_align_size_old(*sector_size); return 0; } return -1; } /* * cli: sanlock direct init * cli: sanlock direct read_leader * cli: sanlock direct acquire * cli: sanlock direct release * lib: sanlock_direct_init() * * direct.c: * direct_init() * direct_read_leader() * direct_acquire() * direct_release() * do_paxos_action() * paxos_lease.c: * paxos_lease_init() * paxos_lease_leader_read() * paxos_lease_acquire() * paxos_lease_release() * * cli: sanlock direct init * cli: sanlock direct read_leader * cli: sanlock direct acquire_id * cli: sanlock direct release_id * cli: sanlock direct renew_id * lib: sanlock_direct_init() * * direct.c: * direct_init() * direct_read_leader() * direct_acquire_id() * direct_release_id() * direct_renew_id() * do_delta_action() * delta_lease.c: * delta_lease_init() * delta_lease_leader_read() * delta_lease_acquire() * delta_lease_release() * delta_lease_renew() */ static int do_paxos_action(int action, struct task *task, int io_timeout, struct sanlk_resource *res, int num_hosts, int write_clear, int try_caw, uint64_t local_host_id, uint64_t local_host_generation, struct leader_record *leader_in, struct leader_record *leader_ret) { struct token *token; struct leader_record leader; struct paxos_dblock dblock; int sector_size = 0; int align_size = 0; int max_hosts = 0; int disks_len, token_len; int j, rv = 0; uint64_t max_mbal; if (!io_timeout) io_timeout = com.io_timeout; if (!io_timeout) io_timeout = DEFAULT_IO_TIMEOUT; rv = sizes_from_flags(res->flags, §or_size, &align_size, &max_hosts, "RES"); if (rv) return -1; disks_len = res->num_disks * sizeof(struct sync_disk); token_len = sizeof(struct token) + disks_len; token = malloc(token_len); if (!token) return -ENOMEM; memset(token, 0, token_len); token->io_timeout = io_timeout; token->disks = (struct sync_disk *)&token->r.disks[0]; token->r.num_disks = res->num_disks; memcpy(token->r.lockspace_name, res->lockspace_name, SANLK_NAME_LEN); memcpy(token->r.name, res->name, SANLK_NAME_LEN); token->r.flags = res->flags; /* WARNING sync_disk == sanlk_disk */ memcpy(token->disks, &res->disks, disks_len); for (j = 0; j < token->r.num_disks; j++) { token->disks[j].sector_size = 0; token->disks[j].fd = -1; } rv = open_disks(token->disks, token->r.num_disks); if (rv < 0) { free_token(token); return rv; } if (!sector_size && com.sector_size) sector_size = com.sector_size; if (!align_size && com.align_size) align_size = com.align_size; switch (action) { case ACT_DIRECT_INIT: /* paxos_lease_init looks at token->r.flags for sector/align flags */ rv = paxos_lease_init(task, token, num_hosts, write_clear, try_caw); break; case ACT_ACQUIRE: if (!sector_size || !align_size) { rv = direct_read_leader_sizes(task, &token->disks[0], §or_size, &align_size); if (rv < 0) break; } token->sector_size = sector_size; token->align_size = align_size; token->host_id = local_host_id; token->host_generation = local_host_generation; rv = paxos_lease_acquire(task, token, 0, leader_ret, &dblock, 0, 0); break; case ACT_RELEASE: if (!sector_size) sector_size = 4096; if (!align_size) align_size = sector_size_to_align_size_old(sector_size); token->sector_size = sector_size; token->align_size = align_size; rv = paxos_lease_leader_read(task, task->use_aio, token, &leader, "direct_release"); if (rv < 0) break; sector_size = leader.sector_size; align_size = leader_align_size_from_flag(leader.flags); if (!align_size) align_size = sector_size_to_align_size_old(sector_size); token->sector_size = sector_size; token->align_size = align_size; rv = paxos_lease_release(task, token, NULL, &leader, leader_ret); break; case ACT_READ: if (!sector_size || !align_size) { rv = direct_read_leader_sizes(task, &token->disks[0], §or_size, &align_size); if (rv < 0) break; } token->sector_size = sector_size; token->align_size = align_size; token->host_id = 1; /* read all the initial values needed to start disk paxos, including leader record and all dblocks. TODO: reading the owners of resource will be implemented in the future. */ rv = paxos_lease_read(task, task->use_aio, token, 0, &leader, &max_mbal, "direct_read", 1, NULL); break; case ACT_READ_LEADER: if (!sector_size) sector_size = 4096; if (!align_size) align_size = sector_size_to_align_size_old(sector_size); token->sector_size = sector_size; token->align_size = align_size; rv = paxos_lease_leader_read(task, task->use_aio, token, &leader, "direct_read_leader"); break; case ACT_WRITE_LEADER: sector_size = leader_in->sector_size; align_size = leader_align_size_from_flag(leader_in->flags); if (!align_size) align_size = sector_size_to_align_size_old(sector_size); token->sector_size = sector_size; token->align_size = align_size; rv = paxos_lease_leader_clobber(task, token, leader_in, "direct_clobber"); break; } close_disks(token->disks, token->r.num_disks); free_token(token); if (rv == SANLK_OK) rv = 0; if (leader_ret) memcpy(leader_ret, &leader, sizeof(struct leader_record)); return rv; } /* * sanlock direct acquire -i -g -r RESOURCE * sanlock direct release -r RESOURCE */ int direct_acquire(struct task *task, int io_timeout, struct sanlk_resource *res, int num_hosts, uint64_t local_host_id, uint64_t local_host_generation, struct leader_record *leader_ret) { return do_paxos_action(ACT_ACQUIRE, task, io_timeout, res, num_hosts, 0, 0, local_host_id, local_host_generation, NULL, leader_ret); } int direct_release(struct task *task, int io_timeout, struct sanlk_resource *res, struct leader_record *leader_ret) { return do_paxos_action(ACT_RELEASE, task, io_timeout, res, 0, 0, 0, 0, 0, NULL, leader_ret); } static int do_delta_action(int action, struct task *task, int io_timeout, struct sanlk_lockspace *ls, int try_caw, uint64_t generation, uint64_t timestamp, char *our_host_name, struct leader_record *leader_in, struct leader_record *leader_ret) { struct leader_record leader; struct sync_disk sd; struct space space; char bitmap[HOSTID_BITMAP_SIZE]; int sector_size = 0; int align_size = 0; int max_hosts = 0; int read_result; int rd_ms, wr_ms; int rv; memset(bitmap, 0, sizeof(bitmap)); if (!io_timeout) io_timeout = com.io_timeout; if (!io_timeout) io_timeout = DEFAULT_IO_TIMEOUT; rv = sizes_from_flags(ls->flags, §or_size, &align_size, &max_hosts, "LSF"); if (rv) return -1; memset(&leader, 0, sizeof(leader)); /* for log_space in delta functions */ memset(&space, 0, sizeof(space)); space.io_timeout = io_timeout; if (!ls->host_id_disk.path[0]) return -ENODEV; if ((action != ACT_DIRECT_INIT) && !ls->host_id) return -EINVAL; memset(&sd, 0, sizeof(struct sync_disk)); memcpy(&sd, &ls->host_id_disk, sizeof(struct sanlk_disk)); sd.fd = -1; rv = open_disk(&sd); if (rv < 0) return -ENODEV; if (!sector_size && com.sector_size) sector_size = com.sector_size; if (!align_size && com.align_size) align_size = com.align_size; switch (action) { case ACT_DIRECT_INIT: /* delta_lease_init looks at ls->flags for sector/align sizes */ rv = delta_lease_init(task, ls, io_timeout, try_caw, &sd); break; case ACT_DIRECT_INIT_HOST: rv = delta_lease_init_host(task, ls, our_host_name, generation, timestamp, io_timeout, try_caw, &sd); break; case ACT_ACQUIRE_ID: if (!sector_size || !align_size) { rv = direct_read_leader_sizes(task, &sd, §or_size, &align_size); if (rv < 0) break; } space.sector_size = sector_size; space.align_size = align_size; rv = delta_lease_acquire(task, &space, &sd, ls->name, our_host_name, ls->host_id, 0, &leader); break; case ACT_RENEW_ID: if (!sector_size || !align_size) { rv = direct_read_leader_sizes(task, &sd, §or_size, &align_size); if (rv < 0) break; } space.sector_size = sector_size; space.align_size = align_size; rv = delta_lease_leader_read(task, sector_size, io_timeout, &sd, ls->name, ls->host_id, &leader, "direct_renew"); if (rv < 0) return rv; rv = delta_lease_renew(task, &space, &sd, ls->name, bitmap, NULL, -1, &read_result, 0, &leader, &leader, &rd_ms, &wr_ms); break; case ACT_RELEASE_ID: if (!sector_size || !align_size) { rv = direct_read_leader_sizes(task, &sd, §or_size, &align_size); if (rv < 0) break; } space.sector_size = sector_size; space.align_size = align_size; rv = delta_lease_leader_read(task, sector_size, io_timeout, &sd, ls->name, ls->host_id, &leader, "direct_release"); if (rv < 0) return rv; rv = delta_lease_release(task, &space, &sd, ls->name, &leader, &leader); break; case ACT_READ_LEADER: case ACT_READ: if (!sector_size || !align_size) { rv = direct_read_leader_sizes(task, &sd, §or_size, &align_size); if (rv < 0) break; } rv = delta_lease_leader_read(task, sector_size, io_timeout, &sd, ls->name, ls->host_id, &leader, "direct_read"); break; case ACT_WRITE_LEADER: rv = delta_lease_leader_clobber(task, io_timeout, &sd, ls->host_id, leader_in, "direct_clobber"); } close_disks(&sd, 1); if (rv == SANLK_OK) rv = 0; if (leader_ret) memcpy(leader_ret, &leader, sizeof(struct leader_record)); return rv; } /* * sanlock direct acquire_id|release_id|renew_id -s LOCKSPACE * * should be the equivalent of what the daemon would do for * sanlock client add_lockspace|rem_lockspace -s LOCKSPACE */ int direct_acquire_id(struct task *task, int io_timeout, struct sanlk_lockspace *ls, char *our_host_name) { return do_delta_action(ACT_ACQUIRE_ID, task, io_timeout, ls, 0, 0, 0, our_host_name, NULL, NULL); } int direct_release_id(struct task *task, int io_timeout, struct sanlk_lockspace *ls) { return do_delta_action(ACT_RELEASE_ID, task, io_timeout, ls, 0, 0, 0, NULL, NULL, NULL); } int direct_renew_id(struct task *task, int io_timeout, struct sanlk_lockspace *ls) { return do_delta_action(ACT_RENEW_ID, task, io_timeout, ls, 0, 0, 0, NULL, NULL, NULL); } int direct_align(struct sync_disk *disk) { if (disk->sector_size == 512) return 1024 * 1024; else if (disk->sector_size == 4096) return 8 * 1024 * 1024; else return -EINVAL; } /* io_timeout is written to leader record and used for the write call itself */ int direct_write_lockspace(struct task *task, struct sanlk_lockspace *ls, uint32_t io_timeout, int try_caw) { if (!ls) return -1; return do_delta_action(ACT_DIRECT_INIT, task, io_timeout, ls, try_caw, 0, 0, NULL, NULL, NULL); } int direct_write_lockspace_host(struct task *task, struct sanlk_lockspace *ls, char *our_host_name, uint64_t generation, uint64_t timestamp, uint32_t io_timeout, int try_caw) { if (!ls) return -1; return do_delta_action(ACT_DIRECT_INIT_HOST, task, io_timeout, ls, try_caw, generation, timestamp, our_host_name, NULL, NULL); } int direct_write_resource(struct task *task, struct sanlk_resource *res, int num_hosts, int write_clear, int try_caw) { if (!res) return -1; if (!res->num_disks) return -ENODEV; if (!res->disks[0].path[0]) return -ENODEV; return do_paxos_action(ACT_DIRECT_INIT, task, 0, res, num_hosts, write_clear, try_caw, 0, 0, NULL, NULL); } int direct_read(struct task *task, int io_timeout, struct sanlk_lockspace *ls, struct sanlk_resource *res, struct leader_record *leader_ret) { int rv = -1; if (ls && ls->host_id_disk.path[0]) rv = do_delta_action(ACT_READ, task, io_timeout, ls, 0, 0, 0, NULL, NULL, leader_ret); else if (res) rv = do_paxos_action(ACT_READ, task, io_timeout, res, 0, 0, 0, 0, 0, NULL, leader_ret); return rv; } int direct_read_leader(struct task *task, int io_timeout, struct sanlk_lockspace *ls, struct sanlk_resource *res, struct leader_record *leader_ret) { int rv = -1; if (ls && ls->host_id_disk.path[0]) rv = do_delta_action(ACT_READ_LEADER, task, io_timeout, ls, 0, 0, 0, NULL, NULL, leader_ret); else if (res) rv = do_paxos_action(ACT_READ_LEADER, task, io_timeout, res, 0, 0, 0, 0, 0, NULL, leader_ret); return rv; } int direct_write_leader(struct task *task, int io_timeout, struct sanlk_lockspace *ls, struct sanlk_resource *res, struct leader_record *leader) { int rv = -1; if (ls && ls->host_id_disk.path[0]) { rv = do_delta_action(ACT_WRITE_LEADER, task, io_timeout, ls, 0, 0, 0, NULL, leader, NULL); } else if (res) { rv = do_paxos_action(ACT_WRITE_LEADER, task, io_timeout, res, 0, 0, 0, 0, 0, leader, NULL); } return rv; } int test_id_bit(int host_id, char *bitmap); int direct_dump(struct task *task, char *dump_path, int force_mode) { char *data, *bitmap; char *colon1 = NULL, *colon2 = NULL, *off_str = NULL, *size_str = NULL, *m; uint32_t magic; struct leader_record *lr_end; struct leader_record *lr; struct leader_record lr_in; struct request_record rr; struct mode_block mb; struct sync_disk sd; struct paxos_dblock dblock; char sname[NAME_ID_SIZE+1]; char rname[NAME_ID_SIZE+1]; uint64_t sector_nr; uint64_t start_offset = 0; uint64_t dump_size = 0; uint64_t end_sector_nr = 0; int sector_size = 0; int align_size = 0; int sector_count, datalen, max_hosts; int i, rv, b; #if 0 struct rindex_header *rh_end; struct rindex_header *rh; struct rindex_header rh_in; struct rindex_entry *re_end; struct rindex_entry *re; struct rindex_entry re_in; int j; #endif memset(&sd, 0, sizeof(struct sync_disk)); /* * /path[:[:]] * * If path contains a colon, the user would escape it with \\, e.g. * device named /dev/foo:32 using offset 0 and length 1M would be * /dev/foo\\:32:0:1M */ for (i = 0; i < strlen(dump_path); i++) { if (dump_path[i] == '\\') { i++; continue; } if (dump_path[i] == ':') { if (!colon1) colon1 = &dump_path[i]; else if (!colon2) colon2 = &dump_path[i]; } } if (colon1) { *colon1 = '\0'; off_str = colon1 + 1; if (colon2) { *colon2 = '\0'; size_str = colon2 + 1; } if ((m = strchr(off_str, 'M'))) { *m = '\0'; start_offset = atoll(off_str) * 1024 * 1024; } else { start_offset = atoll(off_str); } if (size_str) { if ((m = strchr(size_str, 'M'))) { *m = '\0'; dump_size = atoll(size_str) * 1024 * 1024; } else { dump_size = atoll(size_str); } } } if (start_offset % 1048576) printf("WARNING: dump offset should be a multiple of 1048576 bytes.\n"); sanlock_path_import(sd.path, dump_path, sizeof(sd.path)); sd.fd = -1; rv = open_disk(&sd); if (rv < 0) { printf("Device %s not found.\n", sd.path); return -ENODEV; } if (com.sector_size) sector_size = com.sector_size; if (com.align_size) align_size = com.align_size; if (!sector_size || !align_size) { sd.offset = start_offset; for (i = 0; i < 1024; i++) { rv = direct_read_leader_sizes(task, &sd, §or_size, &align_size); if (sector_size && align_size) break; /* * search for a data structure that contains sector_size/align_size * every 1MB, up to 1GB or dump_size. */ sd.offset += 1048576; if (dump_size && (sd.offset >= (start_offset + dump_size))) break; } if (!sector_size || !align_size) { printf("Cannot find sector_size and align_size, set with -A and -Z.\n"); goto out_close; } } max_hosts = size_to_max_hosts(sector_size, align_size); sector_count = align_size / sector_size; datalen = align_size; data = malloc(datalen); if (!data) { rv = -ENOMEM; goto out_close; } memset(data, 0, datalen); printf("%8s %36s %48s %10s %4s %4s %s", "offset", "lockspace", "resource", "timestamp", "own", "gen", "lver"); if (force_mode) printf("/req/mode"); printf("\n"); sd.offset = start_offset; sector_nr = 0; if (dump_size) end_sector_nr = dump_size / sector_size; while (end_sector_nr == 0 || sector_nr < end_sector_nr) { memset(sname, 0, sizeof(rname)); memset(rname, 0, sizeof(rname)); memset(data, 0, sector_size); rv = read_sectors(&sd, sector_size, sector_nr, sector_count, data, datalen, task, task->use_aio, com.io_timeout, "dump"); magic_in(data, &magic); if (magic == DELTA_DISK_MAGIC) { lr_end = (struct leader_record *)data; leader_record_in(lr_end, &lr_in); lr = &lr_in; for (i = 0; i < sector_count; i++) { lr_end = (struct leader_record *)(data + (i * sector_size)); if (!lr_end->magic) continue; leader_record_in(lr_end, &lr_in); lr = &lr_in; /* has never been acquired, don't print */ if (!lr->owner_id && !lr->owner_generation) continue; memcpy(sname, lr->space_name, NAME_ID_SIZE); memcpy(rname, lr->resource_name, NAME_ID_SIZE); printf("%08llu %36s %48s %010llu %04llu %04llu 0x%08llx", (unsigned long long)(start_offset + ((sector_nr + i) * sector_size)), sname, rname, (unsigned long long)lr->timestamp, (unsigned long long)lr->owner_id, (unsigned long long)lr->owner_generation, (unsigned long long)lr->flags); if (force_mode) { bitmap = (char *)lr_end + LEADER_RECORD_MAX; for (b = 0; b < max_hosts; b++) { if (test_id_bit(b+1, bitmap)) printf(" %d", b+1); } } printf("\n"); } } else if (magic == PAXOS_DISK_MAGIC) { lr_end = (struct leader_record *)data; leader_record_in(lr_end, &lr_in); lr = &lr_in; memcpy(sname, lr->space_name, NAME_ID_SIZE); memcpy(rname, lr->resource_name, NAME_ID_SIZE); printf("%08llu %36s %48s %010llu %04llu %04llu %llu", (unsigned long long)(start_offset + (sector_nr * sector_size)), sname, rname, (unsigned long long)lr->timestamp, (unsigned long long)lr->owner_id, (unsigned long long)lr->owner_generation, (unsigned long long)lr->lver); if (force_mode) { struct request_record *rr_end = (struct request_record *)(data + sector_size); request_record_in(rr_end, &rr); printf("/%llu/%u", (unsigned long long)rr.lver, rr.force_mode); } printf("\n"); for (i = 0; i < lr->num_hosts; i++) { char *pd_end = data + ((2 + i) * sector_size); struct mode_block *mb_end = (struct mode_block *)(pd_end + MBLOCK_OFFSET); if (force_mode > 1) { paxos_dblock_in((struct paxos_dblock *)pd_end, &dblock); if (dblock.mbal || dblock.inp || dblock.lver) { printf("dblock[%04d] mbal %llu bal %llu inp %llu inp2 %llu inp3 %llu lver %llu sum %x\n", i, (unsigned long long)dblock.mbal, (unsigned long long)dblock.bal, (unsigned long long)dblock.inp, (unsigned long long)dblock.inp2, (unsigned long long)dblock.inp3, (unsigned long long)dblock.lver, dblock.checksum); } } mode_block_in(mb_end, &mb); if (!(mb.flags & MBLOCK_SHARED)) continue; printf(" "); printf("%04u %04llu SH\n", i+1, (unsigned long long)mb.generation); } #if 0 else if (magic == RINDEX_DISK_MAGIC) { rh_end = (struct rindex_header *)data; rindex_header_in(rh_end, &rh_in); rh = &rh_in; memcpy(sname, rh->lockspace_name, NAME_ID_SIZE); printf("%08llu %36s rindex_header 0x%x %d %u %llu\n", (unsigned long long)(start_offset + (sector_nr * sector_size)), sname, rh->flags, rh->sector_size, rh->max_resources, (unsigned long long)rh->rx_offset); if (!force_mode) goto next; /* i begins with 1 to skip the first sector of the rindex which holds the header */ for (i = 1; i < sector_count; i++) { int entry_size = sizeof(struct rindex_entry); int entries_per_sector = sector_size / entry_size; for (j = 0; j < entries_per_sector; j++) { re_end = (struct rindex_entry *)(data + (i * sector_size) + (j * entry_size)); rindex_entry_in(re_end, &re_in); re = &re_in; if (!re->res_offset && !re->name[0]) continue; printf("%08llu %36s rentry %s %llu\n", (unsigned long long)(start_offset + ((sector_nr * sector_size) + (i * sector_size) + (j * entry_size))), sname, re->name, (unsigned long long)re->res_offset); } } #endif } else { if (end_sector_nr == 0) break; } /* next: */ sector_nr += sector_count; } rv = 0; free(data); out_close: close_disks(&sd, 1); return rv; } int direct_next_free(struct task *task, char *path) { char *data; char *colon, *off_str; struct leader_record *lr_end; struct leader_record lr; struct sync_disk sd; uint64_t sector_nr; int sector_size = 0, sector_count, datalen, align_size = 0; int rv; memset(&sd, 0, sizeof(struct sync_disk)); colon = strstr(path, ":"); if (colon) { off_str = colon + 1; *colon = '\0'; sd.offset = atoll(off_str); } strncpy(sd.path, path, SANLK_PATH_LEN-1); sd.fd = -1; rv = open_disk(&sd); if (rv < 0) return -ENODEV; if (com.sector_size) sector_size = com.sector_size; if (com.align_size) align_size = com.align_size; if (!sector_size || !align_size) { rv = direct_read_leader_sizes(task, &sd, §or_size, &align_size); if (rv < 0) return rv; } sector_count = align_size / sector_size; datalen = sector_size; data = malloc(datalen); if (!data) { rv = -ENOMEM; goto out_close; } sector_nr = 0; rv = -ENOSPC; while (1) { memset(data, 0, sector_size); rv = read_sector(&sd, sector_size, sector_nr, data, datalen, task, task->use_aio, com.io_timeout, "next_free"); lr_end = (struct leader_record *)data; leader_record_in(lr_end, &lr); if (lr.magic != DELTA_DISK_MAGIC && lr.magic != PAXOS_DISK_MAGIC && lr.magic != RINDEX_DISK_MAGIC) { printf("%llu\n", (unsigned long long)(sector_nr * sector_size)); rv = 0; goto out_free; } sector_nr += sector_count; } out_free: free(data); out_close: close_disks(&sd, 1); return rv; } #if 0 int direct_rindex_format(struct task *task, struct sanlk_rindex *ri) { return rindex_format(task, ri); } int direct_rindex_rebuild(struct task *task, struct sanlk_rindex *ri, uint32_t cmd_flags) { return rindex_rebuild(task, ri, cmd_flags | SANLK_RX_NO_LOCKSPACE); } int direct_rindex_lookup(struct task *task, struct sanlk_rindex *ri, struct sanlk_rentry *re, uint32_t cmd_flags) { struct sanlk_rentry re_ret; int rv; rv = rindex_lookup(task, ri, re, &re_ret, cmd_flags | SANLK_RX_NO_LOCKSPACE); if (!rv) memcpy(re, &re_ret, sizeof(re_ret)); return rv; } int direct_rindex_update(struct task *task, struct sanlk_rindex *ri, struct sanlk_rentry *re, uint32_t cmd_flags) { struct sanlk_rentry re_ret; int rv; rv = rindex_update(task, ri, re, &re_ret, cmd_flags | SANLK_RX_NO_LOCKSPACE); if (!rv) memcpy(re, &re_ret, sizeof(re_ret)); return rv; } #endif sanlock/src/direct.h000066400000000000000000000055131516326262600147370ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __DIRECT_H__ #define __DIRECT_H__ int direct_acquire(struct task *task, int io_timeout, struct sanlk_resource *res, int num_hosts, uint64_t local_host_id, uint64_t local_host_generation, struct leader_record *leader_ret); int direct_release(struct task *task, int io_timeout, struct sanlk_resource *res, struct leader_record *leader_ret); int direct_acquire_id(struct task *task, int io_timeout, struct sanlk_lockspace *ls, char *our_host_name); int direct_release_id(struct task *task, int io_timeout, struct sanlk_lockspace *ls); int direct_renew_id(struct task *task, int io_timeout, struct sanlk_lockspace *ls); int direct_align(struct sync_disk *disk); /* io_timeout is written in the leader record and used for the write call itself */ int direct_write_lockspace(struct task *task, struct sanlk_lockspace *ls, uint32_t io_timeout, int try_caw); int direct_write_lockspace_host(struct task *task, struct sanlk_lockspace *ls, char *our_host_name, uint64_t generation, uint64_t timestamp, uint32_t io_timeout, int try_caw); int direct_write_resource(struct task *task, struct sanlk_resource *res, int num_hosts, int write_clear, int try_caw); int direct_read_leader(struct task *task, int io_timeout, struct sanlk_lockspace *ls, struct sanlk_resource *res, struct leader_record *leader_ret); int direct_write_leader(struct task *task, int io_timeout, struct sanlk_lockspace *ls, struct sanlk_resource *res, struct leader_record *leader); int direct_read(struct task *task, int io_timeout, struct sanlk_lockspace *ls, struct sanlk_resource *res, struct leader_record *leader_ret); int direct_dump(struct task *task, char *dump_path, int force_mode); int direct_next_free(struct task *task, char *path); int direct_rindex_format(struct task *task, struct sanlk_rindex *ri); int direct_rindex_rebuild(struct task *task, struct sanlk_rindex *ri, uint32_t cmd_flags); int direct_rindex_lookup(struct task *task, struct sanlk_rindex *ri, struct sanlk_rentry *re, uint32_t cmd_flags); int direct_rindex_update(struct task *task, struct sanlk_rindex *ri, struct sanlk_rentry *re, uint32_t cmd_flags); #endif sanlock/src/direct_lib.c000066400000000000000000000077421516326262600155660ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #define EXTERN #include "sanlock_internal.h" #include "sanlock_direct.h" #include "sanlock_admin.h" #include "diskio.h" #include "direct.h" #include "task.h" #include "timeouts.h" void log_level(uint32_t space_id GNUC_UNUSED, uint32_t res_id GNUC_UNUSED, char *name GNUC_UNUSED, int level GNUC_UNUSED, const char *fmt GNUC_UNUSED, ...); void log_level(uint32_t space_id GNUC_UNUSED, uint32_t res_id GNUC_UNUSED, char *name GNUC_UNUSED, int level GNUC_UNUSED, const char *fmt GNUC_UNUSED, ...) { } #if 0 int lockspace_begin_rindex_op(char *space_name GNUC_UNUSED, int rindex_op GNUC_UNUSED, struct space_info *spi GNUC_UNUSED); int lockspace_begin_rindex_op(char *space_name GNUC_UNUSED, int rindex_op GNUC_UNUSED, struct space_info *spi GNUC_UNUSED) { return -1; } int lockspace_clear_rindex_op(char *space_name GNUC_UNUSED); int lockspace_clear_rindex_op(char *space_name GNUC_UNUSED) { return -1; } #endif int lockspace_disk(char *space_name GNUC_UNUSED, struct sync_disk *disk GNUC_UNUSED); int lockspace_disk(char *space_name GNUC_UNUSED, struct sync_disk *disk GNUC_UNUSED) { return -1; } int host_info(char *space_name, uint64_t host_id, struct host_status *hs_out); int host_info(char *space_name GNUC_UNUSED, uint64_t host_id GNUC_UNUSED, struct host_status *hs_out GNUC_UNUSED) { return -1; } struct token; void check_mode_block(struct token *token GNUC_UNUSED, int q GNUC_UNUSED, char *dblock GNUC_UNUSED); void check_mode_block(struct token *token GNUC_UNUSED, int q GNUC_UNUSED, char *dblock GNUC_UNUSED) { } /* copied from host_id.c */ int test_id_bit(int host_id, char *bitmap); int test_id_bit(int host_id, char *bitmap) { char *byte = bitmap + ((host_id - 1) / 8); unsigned int bit = (host_id - 1) % 8; char mask; mask = 1 << bit; return (*byte & mask); } int get_rand(int a, int b); int get_rand(int a, int b) { return a + (int) (((float)(b - a + 1)) * random() / (RAND_MAX+1.0)); } static void setup_task_lib(struct task *task, int use_aio) { memset(task, 0, sizeof(struct task)); setup_task_aio(task, use_aio, LIB_AIO_CB_SIZE); sprintf(task->name, "%s", "lib"); } int sanlock_direct_write_lockspace(struct sanlk_lockspace *ls, int max_hosts_unused, uint32_t flags, uint32_t io_timeout) { struct task task; int try_caw = 0; int rv; setup_task_lib(&task, 1); if (flags & SANLK_WRITE_LS_USE_CAW) try_caw = 1; rv = direct_write_lockspace(&task, ls, io_timeout, try_caw); close_task_aio(&task); return rv; } int sanlock_direct_write_resource(struct sanlk_resource *res, int max_hosts_unused, int num_hosts, uint32_t flags) { struct task task; int try_caw = 0; int rv; setup_task_lib(&task, 1); if (flags & SANLK_WRITE_USE_CAW) try_caw = 1; rv = direct_write_resource(&task, res, num_hosts, (flags & SANLK_WRITE_CLEAR) ? 1 : 0, try_caw); close_task_aio(&task); return rv; } int sanlock_direct_init(struct sanlk_lockspace *ls, struct sanlk_resource *res, int max_hosts_unused, int num_hosts, int use_aio) { struct task task; int rv; setup_task_lib(&task, use_aio); if (ls) rv = direct_write_lockspace(&task, ls, 0, 0); else rv = direct_write_resource(&task, res, num_hosts, 0, 0); close_task_aio(&task); return rv; } int sanlock_direct_align(struct sanlk_disk *disk_in) { struct sync_disk disk; int align_size, rv; memset(&disk, 0, sizeof(disk)); memcpy(disk.path, disk_in->path, SANLK_PATH_LEN); rv = open_disk(&disk); if (rv < 0) return rv; align_size = direct_align(&disk); close(disk.fd); return align_size; } sanlock/src/diskio.c000066400000000000000000001404451516326262600147460ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include /* SG_IO */ #include /* posix aio */ #include #include "sanlock_internal.h" #include "diskio.h" #include "direct.h" #include "log.h" int read_sysfs_uint(char *path, unsigned int *val) { char buf[32] = { 0 }; int len; int fd; int rv = -1; fd = open(path, O_RDONLY, 0); if (fd < 0) return -1; rv = read(fd, buf, sizeof(buf)); if (rv < 0) { close(fd); return -1; } if ((len = strlen(buf)) && buf[len - 1] == '\n') buf[--len] = '\0'; if (strlen(buf)) { *val = atoi(buf); rv = 0; } close(fd); return rv; } int write_sysfs_uint(char *path, unsigned int val) { char buf[32] = { 0 }; int fd; int rv; fd = open(path, O_RDWR, 0); if (fd < 0) { log_debug("write_sysfs_uint open error %d %s", errno, path); return -1; } snprintf(buf, sizeof(buf), "%u", val); rv = write(fd, buf, strlen(buf)); if (rv < 0) { log_debug("write_sysfs_uint write %s error %d %s", buf, errno, path); close(fd); return -1; } close(fd); return 0; } int read_sysfs_size(const char *disk_path, const char *name, unsigned int *val) { char sysfs_path[PATH_MAX] = { 0 }; struct stat st; int ma, mi; int rv; rv = stat(disk_path, &st); if (rv < 0) return -1; ma = (int)major(st.st_rdev); mi = (int)minor(st.st_rdev); snprintf(sysfs_path, sizeof(sysfs_path), "/sys/dev/block/%d:%d/queue/%s", ma, mi, name); sysfs_path[PATH_MAX-1] = '\0'; rv = read_sysfs_uint(sysfs_path, val); return rv; } static int write_sysfs_size(const char *disk_path, const char *name, unsigned int val) { char sysfs_path[PATH_MAX] = { 0 }; struct stat st; int major, minor; int rv; rv = stat(disk_path, &st); if (rv < 0) { log_debug("write_sysfs_size stat error %d %s", errno, disk_path); return -1; } major = (int)major(st.st_rdev); minor = (int)minor(st.st_rdev); snprintf(sysfs_path, sizeof(sysfs_path), "/sys/dev/block/%d:%d/queue/%s", major, minor, name); sysfs_path[PATH_MAX-1] = '\0'; rv = write_sysfs_uint(sysfs_path, val); return rv; } /* * The default max_sectors_kb is 512 (KB), so a 1MB read is split into two * 512KB reads. Adjust this to at least do 1MB io's. */ int set_max_sectors_kb(struct sync_disk *disk, uint32_t set_kb) { unsigned int cur_kb = 0; int rv; rv = read_sysfs_size(disk->path, "max_sectors_kb", &cur_kb); if (rv < 0) { log_debug("set_max_sectors_kb read error %d %s", rv, disk->path); return rv; } if (cur_kb >= set_kb) return 0; rv = write_sysfs_size(disk->path, "max_sectors_kb", set_kb); if (rv < 0) { log_debug("set_max_sectors_kb write %u error %d %s", set_kb, rv, disk->path); return rv; } return 0; } int get_max_sectors_kb(struct sync_disk *disk, uint32_t *max_sectors_kb) { unsigned int max = 0; int rv; rv = read_sysfs_size(disk->path, "max_sectors_kb", &max); if (!rv) *max_sectors_kb = max; return rv; } static int set_disk_properties(struct sync_disk *disk) { blkid_probe probe; uint32_t sector_size; probe = blkid_new_probe_from_filename(disk->path); if (!probe) { log_error("cannot get blkid probe %s", disk->path); return -1; } sector_size = blkid_probe_get_sectorsize(probe); blkid_free_probe(probe); disk->sector_size = sector_size; return 0; } void close_disks(struct sync_disk *disks, int num_disks) { int d; for (d = 0; d < num_disks; d++) { if (disks[d].fd == -1) continue; close(disks[d].fd); disks[d].fd = -1; } } int majority_disks(int num_disks, int num) { if (num_disks == 1 && !num) return 0; /* odd number of disks */ if (num_disks % 2) return num >= ((num_disks / 2) + 1); /* even number of disks */ if (num > (num_disks / 2)) return 1; if (num < (num_disks / 2)) return 0; /* TODO: half of disks are majority if tiebreaker disk is present */ return 0; } /* * set fd in each disk * returns 0 if majority of disks were opened successfully, -EXXX otherwise */ int open_disks_fd(struct sync_disk *disks, int num_disks) { struct sync_disk *disk; int num_opens = 0; int d, fd, rv = -1; for (d = 0; d < num_disks; d++) { disk = &disks[d]; if (disk->fd != -1) { log_error("open fd %d exists %s", disk->fd, disk->path); rv = -1; goto fail; } fd = open(disk->path, O_RDWR | O_DIRECT | O_SYNC, 0); if (fd < 0) { rv = -errno; if (rv == -EACCES) { log_error("open error %d EACCES: no permission to open %s", rv, disk->path); log_error("check that daemon user %s %d group %s %d has access to disk or file.", com.uname, com.uid, com.gname, com.gid); } else log_error("open error %d %s", fd, disk->path); continue; } disk->fd = fd; num_opens++; } if (!majority_disks(num_disks, num_opens)) { /* rv is open errno */ goto fail; } return 0; fail: close_disks(disks, num_disks); return rv; } /* * set fd and sector_size * verify offset is correctly aligned * returns 0 for success or -EXXX */ int open_disk(struct sync_disk *disk) { struct stat st; int fd, rv; fd = open(disk->path, O_RDWR | O_DIRECT | O_SYNC, 0); if (fd < 0) { rv = -errno; if (rv == -EACCES) { log_error("open error %d EACCES: no permission to open %s", rv, disk->path); log_error("check that daemon user %s %d group %s %d has access to disk or file.", com.uname, com.uid, com.gname, com.gid); } else log_error("open error %d %s", rv, disk->path); goto fail; } if (fstat(fd, &st) < 0) { rv = -errno; log_error("fstat error %d %s", rv, disk->path); close(fd); goto fail; } if (S_ISREG(st.st_mode)) { disk->sector_size = 512; } else { rv = set_disk_properties(disk); if (rv < 0) { close(fd); goto fail; } } disk->fd = fd; return 0; fail: if (rv >= 0) rv = -1; return rv; } /* * set fd and sector_size in each disk * verify all sector_size's match * returns 0 if majority of disks were opened successfully, -EXXX otherwise */ int open_disks(struct sync_disk *disks, int num_disks) { struct sync_disk *disk; int num_opens = 0; int d, err, rv = -1; uint32_t ss = 0; for (d = 0; d < num_disks; d++) { disk = &disks[d]; if (disk->fd != -1) { log_error("open fd %d exists %s", disk->fd, disk->path); rv = -ENOTEMPTY; goto fail; } err = open_disk(disk); if (err < 0) { rv = err; continue; } if (!ss) { ss = disk->sector_size; } else if (ss != disk->sector_size) { log_error("inconsistent sector sizes %u %u %s", ss, disk->sector_size, disk->path); } num_opens++; } if (!majority_disks(num_disks, num_opens)) { /* rv is from open err */ goto fail; } return 0; fail: close_disks(disks, num_disks); return rv; } static int do_write(int fd, uint64_t offset, const char *buf, int len, struct task *task, int *wr_ms) { off_t ret; int rv; int pos = 0; int sys_error = 0; int save_errno = 0; struct timespec begin, end, diff; const char *len_str; char off_str[16]; char ms_str[8]; if (task) task->io_count++; if (com.debug_io_submit) { len_str = align_size_debug_str(len); offset_to_str(offset, sizeof(off_str), off_str); if (len_str) log_taskd(task, "WR %s at %s", len_str, off_str); else log_taskd(task, "WR %d at %s", len, off_str); } ret = lseek(fd, offset, SEEK_SET); if (ret == -1) { log_taskw(task, "WR %d at %s seek errno %d", len, off_str, errno); return -1; } if (ret != offset) { log_taskw(task, "WR %d at %s seek error %llu", len, off_str, (unsigned long long)ret); return -1; } if (wr_ms) clock_gettime(CLOCK_MONOTONIC_RAW, &begin); retry: rv = write(fd, buf + pos, len); if (rv == -1 && errno == EINTR) goto retry; if (rv < 0) { sys_error = 1; save_errno = errno; goto out; } /* if (rv != len && len == sector_size) return error? partial sector writes should not happen AFAIK, and some uses depend on atomic single sector writes */ if (rv != len) { len -= rv; pos += rv; goto retry; } if (sys_error) rv = -1; else rv = 0; out: if (wr_ms) { clock_gettime(CLOCK_MONOTONIC_RAW, &end); ts_diff(&begin, &end, &diff); *wr_ms = (diff.tv_sec * 1000) + (diff.tv_nsec / 1000000); } if (com.debug_io_complete || sys_error) { len_str = align_size_debug_str(len); offset_to_str(offset, sizeof(off_str), off_str); if (wr_ms) { memset(ms_str, 0, sizeof(ms_str)); snprintf(ms_str, 7, "%u", *wr_ms); } if (len_str) log_taskd(task, "WR %s at %s done %s", len_str, off_str, wr_ms ? ms_str : ""); else log_taskd(task, "WR %d at %s done %s", len, off_str, wr_ms ? ms_str : ""); if (sys_error) log_taskw(task, "WR %d at %s error %d %s", len, off_str, save_errno, wr_ms ? ms_str : ""); } return rv; } static int do_read(int fd, uint64_t offset, char *buf, int len, struct task *task, int *rd_ms) { off_t ret; int rv, pos = 0; int sys_error = 0; int save_errno = 0; struct timespec begin, end, diff; const char *len_str; char off_str[16]; char ms_str[8]; if (task) task->io_count++; if (com.debug_io_submit) { len_str = align_size_debug_str(len); offset_to_str(offset, sizeof(off_str), off_str); if (len_str) log_taskd(task, "RD %s at %s", len_str, off_str); else log_taskd(task, "RD %d at %s", len, off_str); } ret = lseek(fd, offset, SEEK_SET); if (ret != offset) { log_taskw(task, "RD %d at %s seek error %llu", len, off_str, (unsigned long long)ret); return -1; } if (rd_ms) clock_gettime(CLOCK_MONOTONIC_RAW, &begin); while (pos < len) { rv = read(fd, buf + pos, len - pos); if (rv == 0) { sys_error = 1; save_errno = errno; break; } if (rv == -1 && errno == EINTR) continue; if (rv < 0) { sys_error = 1; save_errno = errno; break; } pos += rv; } if (sys_error) rv = -1; else rv = 0; if (rd_ms) { clock_gettime(CLOCK_MONOTONIC_RAW, &end); ts_diff(&begin, &end, &diff); *rd_ms = (diff.tv_sec * 1000) + (diff.tv_nsec / 1000000); } if (com.debug_io_complete || sys_error) { len_str = align_size_debug_str(len); offset_to_str(offset, sizeof(off_str), off_str); if (rd_ms) { memset(ms_str, 0, sizeof(ms_str)); snprintf(ms_str, 7, "%u", *rd_ms); } if (len_str) log_taskd(task, "RD %s at %s done %s", len_str, off_str, rd_ms ? ms_str : ""); else log_taskd(task, "RD %d at %s done %s", len, off_str, rd_ms ? ms_str : ""); if (sys_error) log_taskw(task, "RD %d at %s error %d %s", len, off_str, save_errno, rd_ms ? ms_str : ""); } return rv; } static struct aicb *find_callback_slot(struct task *task, int ioto) { struct timespec ts; struct io_event event; int cleared = 0; int rv; int i; find: for (i = 0; i < task->cb_size; i++) { if (task->callbacks[i].used) continue; return &task->callbacks[i]; } if (cleared++) return NULL; memset(&ts, 0, sizeof(struct timespec)); ts.tv_sec = ioto; retry: memset(&event, 0, sizeof(event)); rv = io_getevents(task->aio_ctx, 1, 1, &event, &ts); if (rv == -EINTR) goto retry; if (rv < 0) return NULL; if (rv == 1) { struct iocb *ev_iocb = event.obj; struct aicb *ev_aicb = container_of(ev_iocb, struct aicb, iocb); int op = ev_iocb ? ev_iocb->aio_lio_opcode : -1; const char *op_str; if (op == IO_CMD_PREAD) op_str = "RD"; else if (op == IO_CMD_PWRITE) op_str = "WR"; else op_str = "UK"; log_taskw(task, "aio collect %s %p:%p:%p result %ld:%ld old free", op_str, ev_aicb, ev_iocb, ev_aicb->buf, event.res, event.res2); ev_aicb->used = 0; free(ev_aicb->buf); ev_aicb->buf = NULL; goto find; } return NULL; } void offset_to_str(unsigned long long offset, int buflen, char *off_str) { uint64_t num_mb; if (!offset) { strncpy(off_str, "0", buflen); } else if (!(offset % 1048576)) { num_mb = offset / 1048576; snprintf(off_str, buflen, "%uM", (uint32_t)num_mb); } else { snprintf(off_str, buflen, "%llu", (unsigned long long)offset); } } /* * If this function returns SANLK_AIO_TIMEOUT, it means the io has timed out * and the event for the timed out io has not been reaped; the caller cannot * free the buf it passed in. It will be freed by a subsequent call when the * event is reaped. (Using my own error value here because I'm not certain * what values we might return from event.res.) */ static int do_linux_aio(int fd, uint64_t offset, char *buf, int len, struct task *task, int ioto, int cmd, int *ms) { struct timespec ts; struct aicb *aicb; struct iocb *iocb; struct io_event event; struct timespec begin, end, diff; const char *op_str; const char *len_str; char ms_str[8]; char off_str[16]; int rv; if (!ioto) { log_taske(task, "aio %d zero io timeout", cmd); return -EINVAL; } /* I expect this pre-emptively catches the io_submit EAGAIN case */ aicb = find_callback_slot(task, ioto); if (!aicb) return -ENOENT; iocb = &aicb->iocb; memset(iocb, 0, sizeof(struct iocb)); iocb->aio_fildes = fd; iocb->aio_lio_opcode = cmd; iocb->u.c.buf = buf; iocb->u.c.nbytes = len; iocb->u.c.offset = offset; if (cmd == IO_CMD_PREAD) op_str = "RD"; else if (cmd == IO_CMD_PWRITE) op_str = "WR"; else op_str = "UK"; if (com.debug_io_submit) { len_str = align_size_debug_str(len); offset_to_str(offset, sizeof(off_str), off_str); if (len_str) log_taskd(task, "%s %s at %s", op_str, len_str, off_str); else log_taskd(task, "%s %d at %s", op_str, len, off_str); } if (ms) clock_gettime(CLOCK_MONOTONIC_RAW, &begin); rv = io_submit(task->aio_ctx, 1, &iocb); if (rv < 0) { log_taske(task, "aio submit %d %p:%p:%p rv %d fd %d", cmd, aicb, iocb, buf, rv, fd); goto out; } task->io_count++; /* don't reuse aicb->iocb or free the buf until we reap the event */ aicb->used = 1; aicb->buf = buf; memset(&ts, 0, sizeof(struct timespec)); ts.tv_sec = ioto; retry: memset(&event, 0, sizeof(event)); rv = io_getevents(task->aio_ctx, 1, 1, &event, &ts); if (rv == -EINTR) goto retry; if (rv < 0) { log_taske(task, "aio getevent %p:%p:%p rv %d", aicb, iocb, buf, rv); goto out; } if (rv == 1) { struct iocb *ev_iocb = event.obj; struct aicb *ev_aicb = container_of(ev_iocb, struct aicb, iocb); int op = ev_iocb ? ev_iocb->aio_lio_opcode : -1; if (op == IO_CMD_PREAD) op_str = "RD"; else if (op == IO_CMD_PWRITE) op_str = "WR"; else op_str = "UK"; if (ms) { clock_gettime(CLOCK_MONOTONIC_RAW, &end); ts_diff(&begin, &end, &diff); *ms = (diff.tv_sec * 1000) + (diff.tv_nsec / 1000000); } ev_aicb->used = 0; if (ev_iocb != iocb) { log_taskw(task, "aio collect %s %p:%p:%p result %ld:%ld other free", op_str, ev_aicb, ev_iocb, ev_aicb->buf, event.res, event.res2); free(ev_aicb->buf); ev_aicb->buf = NULL; goto retry; } if ((int)event.res < 0) { log_taskw(task, "aio collect %s %p:%p:%p result %ld:%ld match res", op_str, ev_aicb, ev_iocb, ev_aicb->buf, event.res, event.res2); rv = event.res; goto out; } if (event.res != len) { log_taskw(task, "aio collect %s %p:%p:%p result %ld:%ld match len %d", op_str, ev_aicb, ev_iocb, ev_aicb->buf, event.res, event.res2, len); rv = -EMSGSIZE; goto out; } /* standard success case */ if (com.debug_io_complete) { len_str = align_size_debug_str(len); offset_to_str(offset, sizeof(off_str), off_str); if (ms) { memset(ms_str, 0, sizeof(ms_str)); snprintf(ms_str, 7, "%u", *ms); } if (len_str) log_taskd(task, "%s %s at %s done %s", op_str, len_str, off_str, ms ? ms_str : ""); else log_taskd(task, "%s %d at %s done %s", op_str, len, off_str, ms ? ms_str : ""); } rv = 0; goto out; } /* Timed out waiting for result. If cancel fails, we could try retry io_getevents indefinitely, but that removes the whole point of using aio, which is the timeout. So, we need to be prepared to reap the event the next time we call io_getevents for a different i/o. We can't reuse the iocb for this timed out io until we get an event for it because we need to compare the iocb to event.obj to distinguish events for separate submissions. dct: io_cancel doesn't work, in general. you are very likely going to get -EINVAL from that call */ task->to_count++; if (cmd == IO_CMD_PREAD) op_str = "RD"; else if (cmd == IO_CMD_PWRITE) op_str = "WR"; else op_str = "UK"; log_taskw(task, "aio timeout %s %p:%p:%p ioto %d to_count %d", op_str, aicb, iocb, buf, ioto, task->to_count); rv = io_cancel(task->aio_ctx, iocb, &event); if (!rv) { aicb->used = 0; rv = -ECANCELED; } else { /* aicb->used and aicb->buf both remain set */ rv = SANLK_AIO_TIMEOUT; if (cmd == IO_CMD_PREAD) task->read_iobuf_timeout_aicb = aicb; } out: return rv; } static int do_write_aio_linux(int fd, uint64_t offset, char *buf, int len, struct task *task, int ioto, int *wr_ms) { return do_linux_aio(fd, offset, buf, len, task, ioto, IO_CMD_PWRITE, wr_ms); } static int do_read_aio_linux(int fd, uint64_t offset, char *buf, int len, struct task *task, int ioto, int *rd_ms) { return do_linux_aio(fd, offset, buf, len, task, ioto, IO_CMD_PREAD, rd_ms); } int write_iobuf(int fd, uint64_t offset, char *iobuf, int iobuf_len, struct task *task, int use_aio, int ioto, int *wr_ms) { if (use_aio == USE_AIO) return do_write_aio_linux(fd, offset, iobuf, iobuf_len, task, ioto, wr_ms); else if (use_aio == NO_AIO) return do_write(fd, offset, iobuf, iobuf_len, task, wr_ms); else return SANLK_IOTYPE; } static int _write_sectors(const struct sync_disk *disk, int sector_size, uint64_t sector_nr, uint32_t sector_count GNUC_UNUSED, const char *data, int data_len, int iobuf_len, struct task *task, int use_aio, int ioto, const char *blktype) { char *iobuf, **p_iobuf; uint64_t offset; int rv; offset = disk->offset + (sector_nr * sector_size); p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), iobuf_len); if (rv) { log_error("write_sectors %s posix_memalign rv %d %s", blktype, rv, disk->path); rv = -ENOMEM; goto out; } memset(iobuf, 0, iobuf_len); memcpy(iobuf, data, data_len); rv = write_iobuf(disk->fd, offset, iobuf, iobuf_len, task, use_aio, ioto, NULL); if (rv < 0) { log_error("write_sectors %s offset %llu rv %d %s", blktype, (unsigned long long)offset, rv, disk->path); } if (rv != SANLK_AIO_TIMEOUT) free(iobuf); out: return rv; } /* sector_nr is logical sector number within the sync_disk. the sync_disk itself begins at disk->offset (in bytes) from the start of the block device identified by disk->path, data_len must be <= sector_size */ int write_sector(const struct sync_disk *disk, int sector_size, uint64_t sector_nr, const char *data, int data_len, struct task *task, int use_aio, int ioto, const char *blktype) { int iobuf_len = sector_size; if ((sector_size != 4096) && (sector_size != 512)) { log_error("write_sector bad sector_size %d", sector_size); return -EINVAL; } if (data_len > iobuf_len) { log_error("write_sector %s data_len %d max %d %s", blktype, data_len, iobuf_len, disk->path); return -1; } return _write_sectors(disk, sector_size, sector_nr, 1, data, data_len, iobuf_len, task, use_aio, ioto, blktype); } /* write multiple complete sectors, data_len must be multiple of sector size */ int write_sectors(const struct sync_disk *disk, int sector_size, uint64_t sector_nr, uint32_t sector_count, const char *data, int data_len, struct task *task, int use_aio, int ioto, const char *blktype) { int iobuf_len = data_len; if ((sector_size != 4096) && (sector_size != 512)) { log_error("write_sectors bad sector_size %d", sector_size); return -EINVAL; } if (data_len != sector_count * sector_size) { log_error("write_sectors %s data_len %d sector_count %d %s", blktype, data_len, sector_count, disk->path); return -1; } return _write_sectors(disk, sector_size, sector_nr, sector_count, data, data_len, iobuf_len, task, use_aio, ioto, blktype); } int read_iobuf(int fd, uint64_t offset, char *iobuf, int len, struct task *task, int use_aio, int ioto, int *rd_ms) { if (use_aio == USE_AIO) return do_read_aio_linux(fd, offset, iobuf, len, task, ioto, rd_ms); else if (use_aio == NO_AIO) return do_read(fd, offset, iobuf, len, task, rd_ms); else return SANLK_IOTYPE; } /* read sector_count sectors starting with sector_nr, where sector_nr is a logical sector number within the sync_disk. the caller will generally want to look at the first N bytes of each sector. when reading multiple sectors, data_len will generally equal iobuf_len, but when reading one sector, data_len may be less than iobuf_len. */ int read_sectors(const struct sync_disk *disk, int sector_size, uint64_t sector_nr, uint32_t sector_count, char *data, int data_len, struct task *task, int use_aio, int ioto, const char *blktype) { char *iobuf, **p_iobuf; uint64_t offset; int iobuf_len; int rv; if ((sector_size != 512) && (sector_size != 4096)) { log_error("read_sectors %s bad sector_size %d", blktype, sector_size); return -EINVAL; } iobuf_len = sector_count * sector_size; offset = disk->offset + (sector_nr * sector_size); p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), iobuf_len); if (rv) { log_error("read_sectors %s posix_memalign rv %d %s", blktype, rv, disk->path); rv = -ENOMEM; goto out; } memset(iobuf, 0, iobuf_len); rv = read_iobuf(disk->fd, offset, iobuf, iobuf_len, task, use_aio, ioto, NULL); if (!rv) { memcpy(data, iobuf, data_len); } else if (rv == -EINVAL) { /* when disk sector size is not yet known, this happens when initially reading 512 but the disk has 4k sectors */ log_debug("read_sectors %s offset %llu rv %d %s", blktype, (unsigned long long)offset, rv, disk->path); } else { log_error("read_sectors %s offset %llu rv %d %s", blktype, (unsigned long long)offset, rv, disk->path); } if (rv != SANLK_AIO_TIMEOUT) free(iobuf); out: return rv; } int read_sector(const struct sync_disk *disk, int sector_size, uint64_t sector_nr, char *data, int data_len, struct task *task, int use_aio, int ioto, const char *blktype) { return read_sectors(disk, sector_size, sector_nr, 1, data, data_len, task, use_aio, ioto, blktype); } /* Try to reap the event of a previously timed out read_iobuf. The aicb used in a task's last timed out read_iobuf is task->read_iobuf_timeout_aicb . */ int read_iobuf_reap(int fd, uint64_t offset, char *iobuf, int read_len, struct task *task, uint32_t ioto_msec) { struct timespec ts; struct aicb *aicb; struct iocb *iocb; struct io_event event; int rv; aicb = task->read_iobuf_timeout_aicb; iocb = &aicb->iocb; if (!aicb->used) return -EINVAL; if (iocb->aio_fildes != fd) return -EINVAL; if (iocb->u.c.buf != iobuf) return -EINVAL; if (iocb->u.c.nbytes != read_len) return -EINVAL; if (iocb->u.c.offset != offset) return -EINVAL; if (iocb->aio_lio_opcode != IO_CMD_PREAD) return -EINVAL; memset(&ts, 0, sizeof(struct timespec)); ts.tv_sec = ioto_msec / 1000; ts.tv_nsec = (ioto_msec % 1000) * 1000000; retry: memset(&event, 0, sizeof(event)); rv = io_getevents(task->aio_ctx, 1, 1, &event, &ts); if (rv == -EINTR) goto retry; if (rv < 0) { log_taske(task, "aio getevent %p:%p:%p rv %d r", aicb, iocb, iobuf, rv); goto out; } if (rv == 1) { struct iocb *ev_iocb = event.obj; struct aicb *ev_aicb = container_of(ev_iocb, struct aicb, iocb); int op = ev_iocb ? ev_iocb->aio_lio_opcode : -1; const char *op_str; if (op == IO_CMD_PREAD) op_str = "RD"; else if (op == IO_CMD_PWRITE) op_str = "WR"; else op_str = "UK"; ev_aicb->used = 0; if (ev_iocb != iocb) { log_taskw(task, "aio collect %s %p:%p:%p result %ld:%ld other free r", op_str, ev_aicb, ev_iocb, ev_aicb->buf, event.res, event.res2); free(ev_aicb->buf); ev_aicb->buf = NULL; goto retry; } if ((int)event.res < 0) { log_taskw(task, "aio collect %s %p:%p:%p result %ld:%ld match res r", op_str, ev_aicb, ev_iocb, ev_aicb->buf, event.res, event.res2); rv = event.res; goto out; } if (event.res != read_len) { log_taskw(task, "aio collect %s %p:%p:%p result %ld:%ld match len %d r", op_str, ev_aicb, ev_iocb, ev_aicb->buf, event.res, event.res2, read_len); rv = -EMSGSIZE; goto out; } log_taskw(task, "aio collect %s %p:%p:%p result %ld:%ld match reap", op_str, ev_aicb, ev_iocb, ev_aicb->buf, event.res, event.res2); rv = 0; goto out; } /* timed out again */ rv = SANLK_AIO_TIMEOUT; out: return rv; } int lv_to_target(uint32_t major, uint32_t minor, uint64_t offset, uint32_t *target_major, uint32_t *target_minor, uint64_t *target_offset) { struct dm_task *dmt; struct dm_info info; void *next = NULL; char *target_type, *params; uint64_t start, length; uint64_t tgt_offset; int tgt_major, tgt_minor; int ret = -1; if (!target_major || !target_minor || !target_offset) return -1; if (!(dmt = dm_task_create(DM_DEVICE_TABLE))) { log_debug("lv_to_target dm_task_create error"); return -1; } if (!dm_task_set_major_minor(dmt, major, minor, 0)) { log_debug("lv_to_target dm_task_set_major_minor error"); goto out; } if (!dm_task_run(dmt)) { log_debug("dm_task_run error"); goto out; } if (!dm_task_get_info(dmt, &info)) { log_debug("dm_task_info error"); goto out; } if (!info.exists) { log_debug("dm_task_info does not exist"); goto out; } /* Iterate through targets to find the one containing our offset */ do { next = dm_get_next_target(dmt, next, &start, &length, &target_type, ¶ms); /* Check if offset falls within this target's range */ if (offset >= start && offset < (start + length)) { if (!target_type || strcmp(target_type, "linear") != 0) { log_debug("lv_to_target dm target at offset %" PRIu64 " is not linear (type: %s)", offset, target_type ? target_type : "unknown"); goto out; } /* Parse linear target parameters: "major:minor offset" */ if (!params || sscanf(params, "%d:%d %" SCNu64, &tgt_major, &tgt_minor, &tgt_offset) != 3) { log_debug("lv_to_target failed to parse linear target parameters: %s", params ? params : "(null)"); goto out; } /* Calculate the actual offset in the target device */ *target_major = (uint32_t)tgt_major; *target_minor = (uint32_t)tgt_minor; *target_offset = tgt_offset + (offset - start); ret = 0; break; } } while (next); if (ret) log_debug("lv_to_target offset %" PRIu64 " not found in device %u:%u table", offset, major, minor); out: dm_task_destroy(dmt); return ret; } int dev_is_lv(uint32_t major, uint32_t minor) { char path[PATH_MAX] = { 0 }; char uuid[256] = { 0 }; ssize_t num; int fd; snprintf(path, PATH_MAX-1, "/sys/dev/block/%u:%u/dm/uuid", major, minor); if ((fd = open(path, O_RDONLY, 0)) < 0) return 0; num = read(fd, uuid, sizeof(uuid)-1); close(fd); if ((num > 4) && !strncmp(uuid, "LVM-", 4)) return 1; return 0; } int dev_is_mpath(uint32_t major, uint32_t minor) { char path[PATH_MAX] = { 0 }; char uuid[256] = { 0 }; ssize_t num; int fd; snprintf(path, PATH_MAX-1, "/sys/dev/block/%u:%u/dm/uuid", major, minor); if ((fd = open(path, O_RDONLY, 0)) < 0) return 0; num = read(fd, uuid, sizeof(uuid)-1); close(fd); if ((num > 6) && !strncmp(uuid, "mpath-", 6)) return 1; return 0; } int dev_is_scsi(uint32_t major, uint32_t minor) { char path[PATH_MAX] = { 0 }; struct stat st; snprintf(path, PATH_MAX-1, "/sys/dev/block/%u:%u/device/scsi_device", major, minor); if (stat(path, &st) < 0) return 0; return 1; } int devno_to_path(uint32_t major, uint32_t minor, char *path) { char name[128] = { 0 }; if (!dm_device_get_name(major, minor, 1, name, sizeof(name))) return -1; snprintf(path, PATH_MAX-1, "/dev/%s", name); return 0; } /* static void log_debug_data(unsigned char *data, int len) { int i; for (i = 0; i < len ; i++) { if (!(i % 32)) printf("\n"); printf("%02x ", data[i]); } printf("\n"); } */ /* SCSI sense key definitions */ #define SENSE_KEY_NO_SENSE 0x00 #define SENSE_KEY_RECOVERED_ERROR 0x01 #define SENSE_KEY_NOT_READY 0x02 #define SENSE_KEY_MEDIUM_ERROR 0x03 #define SENSE_KEY_HARDWARE_ERROR 0x04 #define SENSE_KEY_ILLEGAL_REQUEST 0x05 #define SENSE_KEY_UNIT_ATTENTION 0x06 #define SENSE_KEY_DATA_PROTECT 0x07 #define SENSE_KEY_ABORTED_COMMAND 0x0B #define SENSE_KEY_MISCOMPARE 0x0E /* SCSI host status definitions */ #define DID_OK 0x00 #define DID_NO_CONNECT 0x01 #define DID_BUS_BUSY 0x02 #define DID_TIME_OUT 0x03 #define DID_BAD_TARGET 0x04 #define DID_ABORT 0x05 #define DID_ERROR 0x07 #define DID_RESET 0x08 #define DID_TRANSPORT_DISRUPTED 0x0e #define DID_TARGET_FAILURE 0x10 #define DID_NEXUS_FAILURE 0x11 /* SCSI driver status definitions */ #define DRIVER_OK 0x00 #define DRIVER_BUSY 0x01 #define DRIVER_SOFT 0x02 #define DRIVER_MEDIA 0x03 #define DRIVER_ERROR 0x04 #define DRIVER_INVALID 0x05 #define DRIVER_TIMEOUT 0x06 #define DRIVER_SENSE 0x08 #define FLAG_FUA 0x8 #define FLAG_DPO 0x10 #define CAW_CDB_SIZE 16 /* * Log SCSI sense key with detailed information * Returns error code to use: * 0 for success * errors SANLK_CAW_MISCOMPARE/SANLK_CAW_ERROR/SANLK_CAW_ILLEGAL */ static int check_scsi_sense_key(struct task *task, uint32_t major, uint32_t minor, uint64_t lba, int sector_size, unsigned char sense_key, unsigned char asc, unsigned char ascq, uint64_t info, int info_valid, int may_fail) { int level = may_fail ? LOG_DEBUG : LOG_ERR; switch (sense_key) { case SENSE_KEY_NO_SENSE: case SENSE_KEY_RECOVERED_ERROR: /* No error or recovered error - treat as success */ return 0; case SENSE_KEY_MISCOMPARE: if (info_valid) { log_debug("scsi_compare_and_write %u:%u lba %llu miscompare at byte offset %llu ASC/ASCQ 0x%02x/0x%02x", major, minor, (unsigned long long)lba, (unsigned long long)info, asc, ascq); } else { log_debug("scsi_compare_and_write %u:%u lba %llu miscompare ASC/ASCQ 0x%02x/0x%02x", major, minor, (unsigned long long)lba, asc, ascq); } return SANLK_CAW_MISCOMPARE; case SENSE_KEY_ILLEGAL_REQUEST: log_level(0, 0, NULL, level, "scsi_compare_and_write %u:%u lba %llu illegal request ASC/ASCQ 0x%02x/0x%02x", major, minor, (unsigned long long)lba, asc, ascq); return SANLK_CAW_ILLEGAL; case SENSE_KEY_NOT_READY: log_level(0, 0, NULL, level, "scsi_compare_and_write %u:%u lba %llu device not ready ASC/ASCQ 0x%02x/0x%02x", major, minor, (unsigned long long)lba, asc, ascq); return SANLK_CAW_ERROR; case SENSE_KEY_MEDIUM_ERROR: case SENSE_KEY_HARDWARE_ERROR: if (info_valid) { log_level(0, 0, NULL, level, "scsi_compare_and_write %u:%u lba %llu %s error at lba %llu ASC/ASCQ 0x%02x/0x%02x", major, minor, (unsigned long long)lba, (sense_key == SENSE_KEY_MEDIUM_ERROR) ? "medium" : "hardware", (unsigned long long)info, asc, ascq); } else { log_level(0, 0, NULL, level, "scsi_compare_and_write %u:%u lba %llu %s error ASC/ASCQ 0x%02x/0x%02x", major, minor, (unsigned long long)lba, (sense_key == SENSE_KEY_MEDIUM_ERROR) ? "medium" : "hardware", asc, ascq); } return SANLK_CAW_ERROR; case SENSE_KEY_DATA_PROTECT: log_level(0, 0, NULL, level, "scsi_compare_and_write %u:%u lba %llu data protect error ASC/ASCQ 0x%02x/0x%02x", major, minor, (unsigned long long)lba, asc, ascq); return SANLK_CAW_ERROR; case SENSE_KEY_UNIT_ATTENTION: log_level(0, 0, NULL, level, "scsi_compare_and_write %u:%u lba %llu unit attention ASC/ASCQ 0x%02x/0x%02x", major, minor, (unsigned long long)lba, asc, ascq); return SANLK_CAW_ERROR; case SENSE_KEY_ABORTED_COMMAND: log_level(0, 0, NULL, level, "scsi_compare_and_write %u:%u lba %llu command aborted ASC/ASCQ 0x%02x/0x%02x", major, minor, (unsigned long long)lba, asc, ascq); return SANLK_CAW_ERROR; default: log_level(0, 0, NULL, level, "scsi_compare_and_write %u:%u lba %llu sense key 0x%02x ASC/ASCQ 0x%02x/0x%02x", major, minor, (unsigned long long)lba, sense_key, asc, ascq); return SANLK_CAW_ERROR; } } static void log_scsi_status(uint32_t major, uint32_t minor, uint64_t lba, int sector_size, uint32_t scsi_status, int may_fail) { int level = may_fail ? LOG_DEBUG : LOG_ERR; const char *status_str; switch (scsi_status) { case 0x02: status_str = "check condition"; break; case 0x08: status_str = "busy"; break; case 0x18: status_str = "reservation conflict"; break; case 0x28: status_str = "task set full"; break; default: log_level(0, 0, NULL, level, "scsi_compare_and_write %u:%u lba %llu ss %d scsi_status 0x%02x", major, minor, (unsigned long long)lba, sector_size, scsi_status); return; } log_level(0, 0, NULL, level, "scsi_compare_and_write %u:%u lba %llu ss %d %s", major, minor, (unsigned long long)lba, sector_size, status_str); } static void log_scsi_host_status(uint32_t major, uint32_t minor, uint64_t lba, int sector_size, uint32_t host_status, int may_fail) { int level = may_fail ? LOG_DEBUG : LOG_ERR; const char *status_str; switch (host_status) { case DID_OK: return; /* No error */ case DID_NO_CONNECT: status_str = "no connect"; break; case DID_BUS_BUSY: status_str = "bus busy"; break; case DID_TIME_OUT: status_str = "timeout"; break; case DID_BAD_TARGET: status_str = "bad target"; break; case DID_ABORT: status_str = "abort"; break; case DID_ERROR: status_str = "internal error"; break; case DID_RESET: status_str = "reset"; break; case DID_TRANSPORT_DISRUPTED: status_str = "transport disrupted"; break; case DID_TARGET_FAILURE: status_str = "target failure"; break; case DID_NEXUS_FAILURE: status_str = "nexus failure"; break; default: log_level(0, 0, NULL, level, "scsi_compare_and_write %u:%u lba %llu ss %d host_status 0x%02x", major, minor, (unsigned long long)lba, sector_size, host_status); return; } log_level(0, 0, NULL, level, "scsi_compare_and_write %u:%u lba %llu ss %d host %s", major, minor, (unsigned long long)lba, sector_size, status_str); } static void log_scsi_driver_status(uint32_t major, uint32_t minor, uint64_t lba, int sector_size, uint32_t driver_status, int may_fail) { int level = may_fail ? LOG_DEBUG : LOG_ERR; const char *status_str; switch (driver_status) { case DRIVER_OK: return; /* No error */ case DRIVER_BUSY: status_str = "busy"; break; case DRIVER_SOFT: status_str = "soft error"; break; case DRIVER_MEDIA: status_str = "media error"; break; case DRIVER_ERROR: status_str = "error"; break; case DRIVER_INVALID: status_str = "invalid"; break; case DRIVER_TIMEOUT: status_str = "timeout"; break; case DRIVER_SENSE: status_str = "sense"; break; default: log_level(0, 0, NULL, level, "scsi_compare_and_write %u:%u lba %llu ss %d driver_status 0x%02x", major, minor, (unsigned long long)lba, sector_size, driver_status); return; } log_level(0, 0, NULL, level, "scsi_compare_and_write %u:%u lba %llu ss %d driver %s", major, minor, (unsigned long long)lba, sector_size, status_str); } int scsi_compare_and_write(struct task *task, int fd, int io_timeout, uint32_t major, uint32_t minor, int sector_size, uint64_t lba, const char *both_data, const char *prev_data, const char *next_data, int may_fail) { sg_io_hdr_t io_hdr; unsigned char sense_buf[32] = { 0 }; unsigned char cdb[CAW_CDB_SIZE]; unsigned char **p_caw_data; unsigned char *caw_data = (unsigned char *)both_data; unsigned short num_blocks = 1; /* number of blocks to compare/write */ uint32_t scsi_status, host_status, driver_status; unsigned char sense_key; unsigned char asc = 0, ascq = 0; int sense_len; int ret = -1; int rv; if (!both_data) { p_caw_data = &caw_data; rv = posix_memalign((void *)p_caw_data, getpagesize(), 2 * sector_size); if (rv) return -ENOMEM; /* first sector is prev data, second sector is next data */ memcpy(caw_data, prev_data, sector_size); memcpy(caw_data + sector_size, next_data, sector_size); } if (!io_timeout) { log_warn("scsi_compare_and_write zero io_timeout using 1"); io_timeout = 1; } memset(cdb, 0, sizeof(cdb)); cdb[0] = 0x89; /* COMPARE AND WRITE */ if (com.caw_dpo) cdb[1] |= FLAG_DPO; if (com.caw_fua) cdb[1] |= FLAG_FUA; /* LBA (Logical Block Address) is a 64-bit value, stored in bytes 2-9 (Big Endian) */ cdb[2] = (unsigned char)((lba >> 56) & 0xFF); cdb[3] = (unsigned char)((lba >> 48) & 0xFF); cdb[4] = (unsigned char)((lba >> 40) & 0xFF); cdb[5] = (unsigned char)((lba >> 32) & 0xFF); cdb[6] = (unsigned char)((lba >> 24) & 0xFF); cdb[7] = (unsigned char)((lba >> 16) & 0xFF); cdb[8] = (unsigned char)((lba >> 8) & 0xFF); cdb[9] = (unsigned char)(lba & 0xFF); cdb[13] = (uint8_t)(num_blocks & 0xFF); memset(&io_hdr, 0, sizeof(sg_io_hdr_t)); io_hdr.interface_id = 'S'; io_hdr.cmd_len = sizeof(cdb); io_hdr.mx_sb_len = sizeof(sense_buf); io_hdr.dxfer_direction = SG_DXFER_TO_DEV; io_hdr.dxfer_len = (2 * sector_size); io_hdr.dxferp = caw_data; io_hdr.cmdp = cdb; io_hdr.sbp = sense_buf; io_hdr.timeout = io_timeout * 1000; if (ioctl(fd, SG_IO, &io_hdr) < 0) { log_debug("scsi_compare_and_write %u:%u lba %llu ss %d ioctl errno %d", major, minor, (unsigned long long)lba, sector_size, errno); if (!both_data) free(caw_data); return SANLK_CAW_IOCTL; } scsi_status = io_hdr.status & 0x7e; host_status = io_hdr.host_status; driver_status = io_hdr.driver_status & 0x0f; sense_len = io_hdr.sb_len_wr; /* success */ if (!scsi_status && !host_status && !driver_status) { ret = 0; goto out; } /* generic error, may be replaced with specific error */ ret = SANLK_CAW_ERROR; if (scsi_status && (scsi_status != 0x02)) log_scsi_status(major, minor, lba, sector_size, scsi_status, may_fail); if (scsi_status && (scsi_status == 0x02) && (sense_len < 1)) log_error("scsi_compare_and_write check condition without sense data"); if (scsi_status && (scsi_status == 0x02) && (sense_len > 1)) { /* check condition (0x02) with sense data */ unsigned char response_code; int info_valid = 0; uint64_t info = 0; response_code = sense_buf[0] & 0x7f; if (response_code == 0x70 || response_code == 0x71) { /* Fixed format sense data */ if (sense_len < 14) { log_error("scsi_compare_and_write sense buffer too short: %d", sense_len); goto other_status; } sense_key = sense_buf[2] & 0x0f; asc = sense_buf[12]; ascq = sense_buf[13]; /* Extract INFORMATION field if valid bit is set */ if (sense_buf[0] & 0x80) { info_valid = 1; info = ((uint64_t)sense_buf[3] << 24) | ((uint64_t)sense_buf[4] << 16) | ((uint64_t)sense_buf[5] << 8) | (uint64_t)sense_buf[6]; } } else if (response_code == 0x72 || response_code == 0x73) { if (sense_len < 8) { log_error("scsi_compare_and_write descriptor sense buffer too short: %d", sense_len); goto other_status; } sense_key = sense_buf[1] & 0x0f; asc = sense_buf[2]; ascq = sense_buf[3]; } else { log_error("scsi_compare_and_write unknown sense format 0x%02x", response_code); goto other_status; } ret = check_scsi_sense_key(task, major, minor, lba, sector_size, sense_key, asc, ascq, info, info_valid, may_fail); if (ret == SANLK_CAW_MISCOMPARE) { task->miscompares++; goto out; } } other_status: if (host_status) log_scsi_host_status(major, minor, lba, sector_size, host_status, may_fail); if (driver_status) log_scsi_driver_status(major, minor, lba, sector_size, driver_status, may_fail); out: if (!both_data) free(caw_data); return ret; } int compare_and_write(struct task *task, int fd, int io_timeout, uint32_t major, uint32_t minor, int sector_size, uint64_t offset_bytes, const char *both_data, const char *prev_data, const char *next_data) { if (!fd || !major || !sector_size || !io_timeout) { log_error("compare_and_write invalid args %d %u %u %d %d", fd, major, minor, sector_size, io_timeout); return -1; } /* scsi compare-and-write units are sector size of the LUN */ return scsi_compare_and_write(task, fd, io_timeout, major, minor, sector_size, offset_bytes / sector_size, both_data, prev_data, next_data, 0); } /* * test verification levels: * 1. read sector, compare-and-write same data back * 2. read sector, compare-and-write new data back * 3. read sector, compare-and-write new data back, * reread sector, verify new data is read * 4. read sector, compare-and-write new data back, * reread sector, verify new data is read, * compare-and-write with modified prev_data, * verify miscompare error * 5. read sector, compare-and-write new data back, * reread sector, verify new data is read, * compare-and-write with modified prev_data, * verify miscompare error, * read same data through LV, verify correct */ int compare_and_write_test(struct task *task, char *disk_path, uint64_t disk_offset_bytes, int sector_size, int test_verify) { char test_path[PATH_MAX] = { 0 }; char **p_prev_data = NULL; char **p_next_data = NULL; char *prev_data = NULL; char *next_data = NULL; struct stat st; uint32_t disk_major, disk_minor; uint32_t test_major, test_minor; uint64_t disk_offset_sect512; uint64_t disk_offset_sectors; uint64_t test_offset_sect512; uint64_t test_offset_sectors; uint64_t test_offset_bytes; ssize_t num; int disk_fd = -1; int test_fd; int is_lv; int err; int ret = -1; int rv; if (!test_verify) return 0; if (stat(disk_path, &st) < 0) { log_debug("compare_and_write_test %s disk stat error %d", disk_path, errno); return -1; } /* * offset units: * sanlock uses units of bytes, * libdm uses units of 512 bytes, * sg_io uses units of disk sector size */ disk_major = (uint32_t)major(st.st_rdev); disk_minor = (uint32_t)minor(st.st_rdev); disk_offset_sect512 = disk_offset_bytes / 512; disk_offset_sectors = disk_offset_bytes / sector_size; /* sector_size is either 512 or 4096 */ /* * If "disk" is an LV, then it is translated to "test" * which is the PV used by the LV at the given offset. * SG_IO must be submitted to the PV. */ if ((is_lv = dev_is_lv(disk_major, disk_minor))) { err = lv_to_target(disk_major, disk_minor, disk_offset_sect512, &test_major, &test_minor, &test_offset_sect512); if (err < 0) { log_debug("compare_and_write_test %s %u:%u lv_to_target error %d", disk_path, disk_major, disk_minor, err); return -1; } test_offset_bytes = test_offset_sect512 * 512; test_offset_sectors = test_offset_bytes / sector_size; if (!dev_is_scsi(test_major, test_minor) && !dev_is_mpath(test_major, test_minor)) { log_debug("compare_and_write_test %s %u:%u off %llu test %u:%u off %llu unsupported dev type", disk_path, disk_major, disk_minor, (unsigned long long)disk_offset_bytes, test_major, test_minor, (unsigned long long)test_offset_bytes); return -1; } /* get path of device with test_major:test_minor */ if (devno_to_path(test_major, test_minor, test_path) < 0) { log_debug("compare_and_write_test %s %u:%u devno_to_path %u:%u error", disk_path, disk_major, disk_minor, test_major, test_minor); return -1; } } else { test_major = disk_major; test_minor = disk_minor; test_offset_bytes = disk_offset_bytes; test_offset_sect512 = disk_offset_sect512; test_offset_sectors = disk_offset_sectors; strncpy(test_path, disk_path, PATH_MAX-1); } /* log_debug("compare_and_write_test disk %s %u:%u off %llu %llu %llu test %s %u:%u off %llu %llu %llu", disk_path, disk_major, disk_minor, (unsigned long long)disk_offset_bytes, (unsigned long long)disk_offset_sect512, (unsigned long long)disk_offset_sectors, test_path, test_major, test_minor, (unsigned long long)test_offset_bytes, (unsigned long long)test_offset_sect512, (unsigned long long)test_offset_sectors); */ if ((test_fd = open(test_path, O_RDWR | O_DIRECT | O_SYNC, 0)) < 0) { log_debug("compare_and_write_test %s %u:%u %u:%u open error %d test_path %s", disk_path, disk_major, disk_minor, test_major, test_minor, errno, test_path); return -1; } p_prev_data = &prev_data; rv = posix_memalign((void *)p_prev_data, getpagesize(), sector_size); if (rv) goto out; p_next_data = &next_data; rv = posix_memalign((void *)p_next_data, getpagesize(), sector_size); if (rv) goto out; num = pread(test_fd, prev_data, sector_size, test_offset_bytes); if (num != sector_size) { log_debug("compare_and_write_test %s %u:%u %s %u:%u %d pread error %d %zd test_offset_bytes %llu", disk_path, disk_major, disk_minor, test_path, test_major, test_minor, sector_size, errno, num, (unsigned long long)test_offset_bytes); goto out; } memset(next_data, 0, sector_size); if (test_verify == 1) { /* write the same data back */ memcpy(next_data, prev_data, sector_size); } else { struct utsname name = { 0 }; uname(&name); /* write some random data to disk, making it self identifying to some degree in case there's some need to diagnose where the data came from if it goes wrong. */ snprintf(next_data, 511, "sanlock_test_compare_and_write_from_host_%s_at_%llu_to_%s_%llu_%s_%llu", name.nodename, (unsigned long long)time(NULL), disk_path, (unsigned long long)disk_offset_bytes, test_path, (unsigned long long)test_offset_bytes); } err = scsi_compare_and_write(task, test_fd, 1, test_major, test_minor, sector_size, test_offset_sectors, NULL, prev_data, next_data, 1); if (err) { log_debug("compare_and_write_test %s %u:%u %s %u:%u %d scsi_caw error %d test_offset_sectors %llu", disk_path, disk_major, disk_minor, test_path, test_major, test_minor, sector_size, err, (unsigned long long)test_offset_sectors); ret = err; goto out; } if (test_verify > 2) { num = pread(test_fd, prev_data, sector_size, test_offset_bytes); if (num != sector_size) { log_debug("compare_and_write_test %s %u:%u %s %u:%u %d pread2 error %d %zd test_offset_bytes %llu", disk_path, disk_major, disk_minor, test_path, test_major, test_minor, sector_size, errno, num, (unsigned long long)test_offset_bytes); goto out; } if (memcmp(prev_data, next_data, sector_size)) { log_debug("compare_and_write_test %s %u:%u %s %u:%u %d memcmp error test_offset_bytes %llu", disk_path, disk_major, disk_minor, test_path, test_major, test_minor, sector_size, (unsigned long long)test_offset_bytes); goto out; } } if (test_verify > 3) { prev_data[0] = 'X'; /* unmatching prev_data should cause failure to write */ next_data[1] = 'Y'; /* slightly modify new data, this shouldn't be written */ err = scsi_compare_and_write(task, test_fd, 1, test_major, test_minor, sector_size, test_offset_sectors, NULL, prev_data, next_data, 1); if (err != SANLK_CAW_MISCOMPARE) { /* compare-and-write failed to report expected error */ log_debug("compare_and_write_test %s %u:%u %s %u:%u %d scsi_caw result %d test_offset_sectors %llu", disk_path, disk_major, disk_minor, test_path, test_major, test_minor, sector_size, err, (unsigned long long)test_offset_sectors); goto out; } } if ((test_verify > 4) && is_lv) { if ((disk_fd = open(disk_path, O_RDWR | O_DIRECT | O_SYNC, 0)) < 0) { log_debug("compare_and_write_test %s %u:%u open disk error %d", disk_path, disk_major, disk_minor, errno); disk_fd = -1; goto out; } memset(prev_data, 0, sector_size); next_data[1] = 'a'; /* correct data from 'Y' above */ num = pread(disk_fd, prev_data, sector_size, disk_offset_bytes); if (num != sector_size) { log_debug("compare_and_write_test %s %u:%u %s %u:%u %d pread3 error %d %zd disk_offset_bytes %llu", disk_path, disk_major, disk_minor, test_path, test_major, test_minor, sector_size, errno, num, (unsigned long long)disk_offset_bytes); goto out; } if (memcmp(prev_data, next_data, sector_size)) { /* TODO: write the original data back to test_fd? */ log_debug("compare_and_write_test %s %u:%u %s %u:%u %d memcmp2 error disk_offset_bytes %llu", disk_path, disk_major, disk_minor, test_path, test_major, test_minor, sector_size, (unsigned long long)disk_offset_bytes); goto out; } } ret = 0; out: if (next_data) free(next_data); if (prev_data) free(prev_data); if (disk_fd != -1) close(disk_fd); close(test_fd); return ret; } sanlock/src/diskio.h000066400000000000000000000065011516326262600147450ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __DISKIO_H__ #define __DISKIO_H__ void offset_to_str(unsigned long long offset, int buflen, char *off_str); void close_disks(struct sync_disk *disks, int num_disks); int open_disk(struct sync_disk *disks); int open_disks(struct sync_disk *disks, int num_disks); int open_disks_fd(struct sync_disk *disks, int num_disks); int majority_disks(int num_disks, int num); int read_sysfs_size(const char *path, const char *name, unsigned int *val); int set_max_sectors_kb(struct sync_disk *disk, uint32_t max_sectors_kb); int get_max_sectors_kb(struct sync_disk *disk, uint32_t *max_sectors_kb); int read_sysfs_uint(char *path, unsigned int *val); int write_sysfs_uint(char *path, unsigned int val); /* * iobuf functions require the caller to allocate iobuf using posix_memalign * and pass it into the function */ int write_iobuf(int fd, uint64_t offset, char *iobuf, int iobuf_len, struct task *task, int use_aio, int ioto, int *wr_ms); int read_iobuf(int fd, uint64_t offset, char *iobuf, int iobuf_len, struct task *task, int use_aio, int ioto, int *rd_ms); int read_iobuf_reap(int fd, uint64_t offset, char *iobuf, int iobuf_len, struct task *task, uint32_t ioto_msec); /* * sector functions allocate an iobuf themselves, copy into it for read, use it * for io, copy out of it for write, and free it */ int write_sector(const struct sync_disk *disk, int sector_size, uint64_t sector_nr, const char *data, int data_len, struct task *task, int use_aio, int ioto, const char *blktype); int write_sectors(const struct sync_disk *disk, int sector_size, uint64_t sector_nr, uint32_t sector_count, const char *data, int data_len, struct task *task, int use_aio, int ioto, const char *blktype); int read_sectors(const struct sync_disk *disk, int sector_size, uint64_t sector_nr, uint32_t sector_count, char *data, int data_len, struct task *task, int use_aio, int ioto, const char *blktype); int read_sector(const struct sync_disk *disk, int sector_size, uint64_t sector_nr, char *data, int data_len, struct task *task, int use_aio, int ioto, const char *blktype); int lv_to_target(uint32_t major, uint32_t minor, uint64_t offset, uint32_t *target_major, uint32_t *target_minor, uint64_t *target_offset); int dev_is_lv(uint32_t major, uint32_t minor); int dev_is_mpath(uint32_t major, uint32_t minor); int dev_is_scsi(uint32_t major, uint32_t minor); int devno_to_path(uint32_t major, uint32_t minor, char *path); int scsi_compare_and_write(struct task *task, int fd, int timeout, uint32_t major, uint32_t minor, int sector_size, uint64_t lba, const char *both_data, const char *prev_data, const char *next_data, int may_fail); int compare_and_write(struct task *task, int fd, int io_timeout, uint32_t major, uint32_t minor, int sector_size, uint64_t offset_bytes, const char *both_data, const char *prev_data, const char *next_data); int compare_and_write_test(struct task *task, char *disk_path, uint64_t disk_offset_bytes, int sector_size, int test_verify); #endif sanlock/src/env.c000066400000000000000000000011731516326262600142460ustar00rootroot00000000000000/* * Copyright 2018 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include "env.h" const char *env_get(const char *key, const char *defval) { const char *val; val = getenv(key); if (val == NULL) return defval; return val; } int env_get_bool(const char *key, int defval) { const char *val; val = getenv(key); if (val == NULL) return defval; return strcmp(val, "1") ? 0 : 1; } sanlock/src/env.h000066400000000000000000000006411516326262600142520ustar00rootroot00000000000000/* * Copyright 2018 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __ENV_H__ #define __ENV_H__ const char *env_get(const char *key, const char *defval); int env_get_bool(const char *key, int defval); #endif sanlock/src/helper.c000066400000000000000000000124401516326262600147340ustar00rootroot00000000000000/* * Copyright 2012 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock.h" #include "monotime.h" #include "helper.h" /* only need sysfs functions */ struct sync_disk; struct task; #include "diskio.h" #define MAX_AV_COUNT 8 static void run_path(struct helper_msg *hm) { char arg[SANLK_HELPER_ARGS_LEN]; char *args = hm->args; char *av[MAX_AV_COUNT + 1]; /* +1 for NULL */ int av_count = 0; int i, arg_len, args_len; for (i = 0; i < MAX_AV_COUNT + 1; i++) av[i] = NULL; av[av_count++] = strdup(hm->path); if (!args[0]) goto pid_arg; /* this should already be done, but make sure */ args[SANLK_HELPER_ARGS_LEN - 1] = '\0'; memset(&arg, 0, sizeof(arg)); arg_len = 0; args_len = strlen(args); for (i = 0; i < args_len; i++) { if (!args[i]) break; if (av_count == MAX_AV_COUNT) break; if (args[i] == '\\') { if (i == (args_len - 1)) break; i++; if (args[i] == '\\') { arg[arg_len++] = args[i]; continue; } if (isspace(args[i])) { arg[arg_len++] = args[i]; continue; } else { break; } } if (isalnum(args[i]) || ispunct(args[i])) { arg[arg_len++] = args[i]; } else if (isspace(args[i])) { if (arg_len) av[av_count++] = strdup(arg); memset(arg, 0, sizeof(arg)); arg_len = 0; } else { break; } } if ((av_count < MAX_AV_COUNT) && arg_len) { av[av_count++] = strdup(arg); } pid_arg: if ((av_count < MAX_AV_COUNT) && hm->pid) { memset(arg, 0, sizeof(arg)); snprintf(arg, sizeof(arg)-1, "%d", hm->pid); av[av_count++] = strdup(arg); } execvp(av[0], av); } static int read_hm(int fd, struct helper_msg *hm) { int rv; retry: rv = read(fd, hm, sizeof(struct helper_msg)); if (rv == -1 && errno == EINTR) goto retry; if (rv != sizeof(struct helper_msg)) return -1; return 0; } static int send_status(int fd) { struct helper_status hs; int rv; memset(&hs, 0, sizeof(hs)); hs.type = HELPER_STATUS; rv = write(fd, &hs, sizeof(hs)); if (rv == sizeof(hs)) return 0; return -1; } #define log_debug(fmt, args...) \ do { \ if (log_stderr) \ fprintf(stderr, "helper %ld " fmt "\n", time(NULL), ##args); \ } while (0) #define STANDARD_TIMEOUT_MS (HELPER_STATUS_INTERVAL*1000) #define RECOVERY_TIMEOUT_MS 1000 int run_helper(int in_fd, int out_fd, int log_stderr) { unsigned int sysfs_val; char name[16]; struct pollfd pollfd; struct helper_msg hm; unsigned int fork_count = 0; unsigned int wait_count = 0; time_t now, last_send, last_good = 0; int timeout = STANDARD_TIMEOUT_MS; int rv, pid, status; memset(name, 0, sizeof(name)); sprintf(name, "%s", "sanlock-helper"); prctl(PR_SET_NAME, (unsigned long)name, 0, 0, 0); rv = setgroups(0, NULL); if (rv < 0) log_debug("error clearing helper groups errno %i", errno); memset(&pollfd, 0, sizeof(pollfd)); pollfd.fd = in_fd; pollfd.events = POLLIN; now = monotime(); last_send = now; rv = send_status(out_fd); if (!rv) last_good = now; while (1) { rv = poll(&pollfd, 1, timeout); if (rv == -1 && errno == EINTR) continue; if (rv < 0) exit(0); now = monotime(); if (now - last_good >= HELPER_STATUS_INTERVAL && now - last_send >= 2) { last_send = now; rv = send_status(out_fd); if (!rv) last_good = now; } memset(&hm, 0, sizeof(hm)); if (pollfd.revents & POLLIN) { rv = read_hm(in_fd, &hm); if (rv) continue; /* terminated at sender, but confirm for checker */ hm.path[SANLK_HELPER_PATH_LEN-1] = '\0'; hm.args[SANLK_HELPER_ARGS_LEN-1] = '\0'; if (hm.type == HELPER_MSG_RUNPATH) { pid = fork(); if (!pid) { run_path(&hm); exit(-1); } fork_count++; /* log_debug("helper fork %d count %d %d %s %s", pid, fork_count, wait_count, hm.path, hm.args); */ } else if (hm.type == HELPER_MSG_KILLPID) { kill(hm.pid, hm.sig); } else if (hm.type == HELPER_MSG_WRITE_SYSFS) { sysfs_val = atoi(hm.args); rv = write_sysfs_uint(hm.path, sysfs_val); log_debug("write_sysfs %s %u rv %d", hm.path, sysfs_val, rv); } } if (pollfd.revents & (POLLERR | POLLHUP | POLLNVAL)) exit(0); /* collect child exits until no more children exist (ECHILD) or none are ready (WNOHANG) */ while (1) { rv = waitpid(-1, &status, WNOHANG); if (rv > 0) { wait_count++; /* log_debug("helper wait %d count %d %d", rv, fork_count, wait_count); */ continue; } /* no more children to wait for or no children have exited */ if (rv < 0 && errno == ECHILD) { if (timeout == RECOVERY_TIMEOUT_MS) { log_debug("helper no children count %d %d", fork_count, wait_count); } timeout = STANDARD_TIMEOUT_MS; } else { timeout = RECOVERY_TIMEOUT_MS; } break; } } return 0; } sanlock/src/helper.h000066400000000000000000000016661516326262600147510ustar00rootroot00000000000000/* * Copyright 2012 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __HELPER_H__ #define __HELPER_H__ /* * helper process * recvs 512 byte helper_msg on in_fd * sends 4 byte helper_status on out_fd */ #define SANLK_HELPER_MSG_LEN 512 #define HELPER_MSG_RUNPATH 1 #define HELPER_MSG_KILLPID 2 #define HELPER_MSG_WRITE_SYSFS 3 struct helper_msg { uint8_t type; uint8_t pad1; uint16_t pad2; uint32_t flags; int pid; int sig; char path[SANLK_HELPER_PATH_LEN]; /* 128 */ char args[SANLK_HELPER_ARGS_LEN]; /* 128 */ char pad[240]; }; #define HELPER_STATUS_INTERVAL 30 #define HELPER_STATUS 1 struct helper_status { uint8_t type; uint8_t status; uint16_t len; }; int run_helper(int in_fd, int out_fd, int log_stderr); #endif sanlock/src/leader.h000066400000000000000000000075531516326262600147270ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __LEADER_H__ #define __LEADER_H__ /* does not include terminating null byte */ /* NB NAME_ID_SIZE must match SANLK_NAME_LEN */ /* NB NAME_ID_SIZE is part of ondisk format */ #define NAME_ID_SIZE 48 /* * paxos version changes: * 6.4: add ALIGN leader flags * 7.0: add CAW leader flag * breaks compat with older versions, because compare-and-write * only works correctly when all hosts use it to write leader. * * delta version changes: * 3.4: add ALIGN leader flags * 4.0: add NO_TIMEOUT and DEAD_EXT leader flags * breaks compat with older versions, because they would take * a NO_TIMEOUT lease after dead timeout when they shouldn't. * 5.0: add CAW leader flag * breaks compat with older versions, because compare-and-write * only works correctly when all hosts use it to write leader. */ #define PAXOS_DISK_MAGIC 0x06152010 #define PAXOS_DISK_CLEAR 0x11282016 #define PAXOS_DISK_VERSION_MAJOR 0x00070000 #define PAXOS_DISK_VERSION_MINOR 0x00000000 /* 6.4 does not support CAW leader flag. */ # define PAXOS_DISK_VERSION_64 0x00060004 #define DELTA_DISK_MAGIC 0x12212010 #define DELTA_DISK_VERSION_MAJOR 0x00050000 #define DELTA_DISK_VERSION_MINOR 0x00000000 /* 3.4 does not support NO_TIMEOUT leader flag. */ #define DELTA_DISK_VERSION_34 0x00030004 /* 4.0 does not support CAW leader flag. */ #define DELTA_DISK_VERSION_40 0x00040000 /* for all disk structures: uint64 aligned on 8 byte boundaries, uint32 aligned on 4 byte boundaries, etc */ /* NB. adjust LEADER_COMPARE_LEN and LEADER_CHECKSUM_LEN when changing this struct. LEADER_CHECKSUM_LEN should end just before the checksum field. LEADER_COMPARE_LEN should end just before timestamp. The checksum field should follow the timestamp field. The leader may be partially through updating the timestamp on multiple leader blocks in a lease, but for the purpose of counting repetitions of a leader block owned by a single host they should be counted together, so COMPARE_LEN should exclude timestamp. */ #define LEADER_COMPARE_LEN 152 #define LEADER_CHECKSUM_LEN 168 #define LEASE_FREE 0 /* leader_record flags */ #define LFL_SHORT_HOLD 0x00000001 #define LFL_NO_TIMEOUT 0x00000002 #define LFL_DEAD_EXT 0x00000004 #define LFL_CAW 0x00000008 /* note: these align flag values match other defines */ #define LFL_ALIGN_1M 0x00000010 #define LFL_ALIGN_2M 0x00000020 #define LFL_ALIGN_4M 0x00000040 #define LFL_ALIGN_8M 0x00000080 struct leader_record { uint32_t magic; uint32_t version; uint32_t flags; uint32_t sector_size; uint64_t num_hosts; uint64_t max_hosts; uint64_t owner_id; /* host_id of owner */ uint64_t owner_generation; uint64_t lver; char space_name[NAME_ID_SIZE] __attribute__ ((nonstring)); /* lockspace for resource */ char resource_name[NAME_ID_SIZE] __attribute__ ((nonstring)); /* resource being locked */ uint64_t timestamp; uint64_t unused1; uint32_t checksum; uint16_t unused2; uint16_t io_timeout; uint64_t write_id; /* for extra info, debug */ uint64_t write_generation; /* for extra info, debug */ uint64_t write_timestamp; /* for extra info, debug */ }; /* leader_record can use first 256 bytes of a sector, bitmap uses the last 256 bytes */ #define LEADER_RECORD_MAX 256 #define HOSTID_BITMAP_OFFSET 256 #define HOSTID_BITMAP_SIZE 256 /* the request record is in the sector following the leader record for a paxos lease. */ #define REQ_DISK_MAGIC 0x08292011 #define REQ_DISK_VERSION_MAJOR 0x00010000 #define REQ_DISK_VERSION_MINOR 0x00000001 struct request_record { uint32_t magic; uint32_t version; uint64_t lver; uint32_t force_mode; }; #endif sanlock/src/libsanlock.pc.in000066400000000000000000000004011516326262600163550ustar00rootroot00000000000000prefix=/usr exec_prefix=${prefix} includedir=${prefix}/include libdir=${exec_prefix}/lib64 Name: libsanlock Description: The sanlock library Version: @VERSION@ Cflags: -I${includedir} Libs: -L${libdir} -lpthread -lrt -laio -lblkid -luuid -lwdmd -lsanlock sanlock/src/libsanlock_client.pc.in000066400000000000000000000003531516326262600177210ustar00rootroot00000000000000prefix=/usr exec_prefix=${prefix} includedir=${prefix}/include libdir=${exec_prefix}/lib64 Name: libsanlock_client Description: The sanlock client library Version: @VERSION@ Cflags: -I${includedir} Libs: -L${libdir} -lsanlock_client sanlock/src/list.h000066400000000000000000000421421516326262600144370ustar00rootroot00000000000000/* Copied from linux kernel */ #ifndef _LINUX_LIST_H #define _LINUX_LIST_H /* * Simple doubly linked list implementation. * * Some of the internal functions ("__xxx") are useful when * manipulating whole lists rather than single entries, as * sometimes we already know the next/prev entries and we can * generate better code by using them directly rather than * using the generic single-entry routines. */ /** * container_of - cast a member of a structure out to the containing structure * * @ptr: the pointer to the member. * @type: the type of the container struct this is embedded in. * @member: the name of the member within the struct. * */ #define container_of(ptr, type, member) ({ \ const typeof( ((type *)0)->member ) *__mptr = (ptr); \ (type *)( (char *)__mptr - offsetof(type,member) );}) #define LIST_POISON1 ((void *) 0x00100100) #define LIST_POISON2 ((void *) 0x00200200) struct list_head { struct list_head *next, *prev; }; #define LIST_HEAD_INIT(name) { &(name), &(name) } #define LIST_HEAD(name) \ struct list_head name = LIST_HEAD_INIT(name) static inline void INIT_LIST_HEAD(struct list_head *list) { list->next = list; list->prev = list; } /* * Insert a new entry between two known consecutive entries. * * This is only for internal list manipulation where we know * the prev/next entries already! */ static inline void __list_add(struct list_head *new, struct list_head *prev, struct list_head *next) { next->prev = new; new->next = next; new->prev = prev; prev->next = new; } /** * list_add - add a new entry * @new: new entry to be added * @head: list head to add it after * * Insert a new entry after the specified head. * This is good for implementing stacks. */ static inline void list_add(struct list_head *new, struct list_head *head) { __list_add(new, head, head->next); } /** * list_add_tail - add a new entry * @new: new entry to be added * @head: list head to add it before * * Insert a new entry before the specified head. * This is useful for implementing queues. */ static inline void list_add_tail(struct list_head *new, struct list_head *head) { __list_add(new, head->prev, head); } /* * Delete a list entry by making the prev/next entries * point to each other. * * This is only for internal list manipulation where we know * the prev/next entries already! */ static inline void __list_del(struct list_head * prev, struct list_head * next) { next->prev = prev; prev->next = next; } /** * list_del - deletes entry from list. * @entry: the element to delete from the list. * Note: list_empty() on entry does not return true after this, the entry is * in an undefined state. */ static inline void list_del(struct list_head *entry) { __list_del(entry->prev, entry->next); entry->next = LIST_POISON1; entry->prev = LIST_POISON2; } /** * list_replace - replace old entry by new one * @old : the element to be replaced * @new : the new element to insert * * If @old was empty, it will be overwritten. */ static inline void list_replace(struct list_head *old, struct list_head *new) { new->next = old->next; new->next->prev = new; new->prev = old->prev; new->prev->next = new; } static inline void list_replace_init(struct list_head *old, struct list_head *new) { list_replace(old, new); INIT_LIST_HEAD(old); } /** * list_del_init - deletes entry from list and reinitialize it. * @entry: the element to delete from the list. */ static inline void list_del_init(struct list_head *entry) { __list_del(entry->prev, entry->next); INIT_LIST_HEAD(entry); } /** * list_move - delete from one list and add as another's head * @list: the entry to move * @head: the head that will precede our entry */ static inline void list_move(struct list_head *list, struct list_head *head) { __list_del(list->prev, list->next); list_add(list, head); } /** * list_move_tail - delete from one list and add as another's tail * @list: the entry to move * @head: the head that will follow our entry */ static inline void list_move_tail(struct list_head *list, struct list_head *head) { __list_del(list->prev, list->next); list_add_tail(list, head); } /** * list_is_last - tests whether @list is the last entry in list @head * @list: the entry to test * @head: the head of the list */ static inline int list_is_last(const struct list_head *list, const struct list_head *head) { return list->next == head; } /** * list_empty - tests whether a list is empty * @head: the list to test. */ static inline int list_empty(const struct list_head *head) { return head->next == head; } /** * list_empty_careful - tests whether a list is empty and not being modified * @head: the list to test * * Description: * tests whether a list is empty _and_ checks that no other CPU might be * in the process of modifying either member (next or prev) * * NOTE: using list_empty_careful() without synchronization * can only be safe if the only activity that can happen * to the list entry is list_del_init(). Eg. it cannot be used * if another CPU could re-list_add() it. */ static inline int list_empty_careful(const struct list_head *head) { struct list_head *next = head->next; return (next == head) && (next == head->prev); } /** * list_rotate_left - rotate the list to the left * @head: the head of the list */ static inline void list_rotate_left(struct list_head *head) { struct list_head *first; if (!list_empty(head)) { first = head->next; list_move_tail(first, head); } } /** * list_is_singular - tests whether a list has just one entry. * @head: the list to test. */ static inline int list_is_singular(const struct list_head *head) { return !list_empty(head) && (head->next == head->prev); } static inline void __list_cut_position(struct list_head *list, struct list_head *head, struct list_head *entry) { struct list_head *new_first = entry->next; list->next = head->next; list->next->prev = list; list->prev = entry; entry->next = list; head->next = new_first; new_first->prev = head; } /** * list_cut_position - cut a list into two * @list: a new list to add all removed entries * @head: a list with entries * @entry: an entry within head, could be the head itself * and if so we won't cut the list * * This helper moves the initial part of @head, up to and * including @entry, from @head to @list. You should * pass on @entry an element you know is on @head. @list * should be an empty list or a list you do not care about * losing its data. * */ static inline void list_cut_position(struct list_head *list, struct list_head *head, struct list_head *entry) { if (list_empty(head)) return; if (list_is_singular(head) && (head->next != entry && head != entry)) return; if (entry == head) INIT_LIST_HEAD(list); else __list_cut_position(list, head, entry); } static inline void __list_splice(const struct list_head *list, struct list_head *prev, struct list_head *next) { struct list_head *first = list->next; struct list_head *last = list->prev; first->prev = prev; prev->next = first; last->next = next; next->prev = last; } /** * list_splice - join two lists, this is designed for stacks * @list: the new list to add. * @head: the place to add it in the first list. */ static inline void list_splice(const struct list_head *list, struct list_head *head) { if (!list_empty(list)) __list_splice(list, head, head->next); } /** * list_splice_tail - join two lists, each list being a queue * @list: the new list to add. * @head: the place to add it in the first list. */ static inline void list_splice_tail(struct list_head *list, struct list_head *head) { if (!list_empty(list)) __list_splice(list, head->prev, head); } /** * list_splice_init - join two lists and reinitialise the emptied list. * @list: the new list to add. * @head: the place to add it in the first list. * * The list at @list is reinitialised */ static inline void list_splice_init(struct list_head *list, struct list_head *head) { if (!list_empty(list)) { __list_splice(list, head, head->next); INIT_LIST_HEAD(list); } } /** * list_splice_tail_init - join two lists and reinitialise the emptied list * @list: the new list to add. * @head: the place to add it in the first list. * * Each of the lists is a queue. * The list at @list is reinitialised */ static inline void list_splice_tail_init(struct list_head *list, struct list_head *head) { if (!list_empty(list)) { __list_splice(list, head->prev, head); INIT_LIST_HEAD(list); } } /** * list_entry - get the struct for this entry * @ptr: the &struct list_head pointer. * @type: the type of the struct this is embedded in. * @member: the name of the list_struct within the struct. */ #define list_entry(ptr, type, member) \ container_of(ptr, type, member) /** * list_first_entry - get the first element from a list * @ptr: the list head to take the element from. * @type: the type of the struct this is embedded in. * @member: the name of the list_struct within the struct. * * Note, that list is expected to be not empty. */ #define list_first_entry(ptr, type, member) \ list_entry((ptr)->next, type, member) /** * list_last_entry - get the last element from a list * @ptr: the list head to take the element from. * @type: the type of the struct this is embedded in. * @member: the name of the list_head within the struct. * * Note, that list is expected to be not empty. */ #define list_last_entry(ptr, type, member) \ list_entry((ptr)->prev, type, member) /** * list_for_each - iterate over a list * @pos: the &struct list_head to use as a loop cursor. * @head: the head for your list. */ #define list_for_each(pos, head) \ for (pos = (head)->next; pos != (head); pos = pos->next) /** * __list_for_each - iterate over a list * @pos: the &struct list_head to use as a loop cursor. * @head: the head for your list. * * This variant differs from list_for_each() in that it's the * simplest possible list iteration code, no prefetching is done. * Use this for code that knows the list to be very short (empty * or 1 entry) most of the time. */ #define __list_for_each(pos, head) \ for (pos = (head)->next; pos != (head); pos = pos->next) /** * list_for_each_prev - iterate over a list backwards * @pos: the &struct list_head to use as a loop cursor. * @head: the head for your list. */ #define list_for_each_prev(pos, head) \ for (pos = (head)->prev; pos != (head); pos = pos->prev) /** * list_for_each_safe - iterate over a list safe against removal of list entry * @pos: the &struct list_head to use as a loop cursor. * @n: another &struct list_head to use as temporary storage * @head: the head for your list. */ #define list_for_each_safe(pos, n, head) \ for (pos = (head)->next, n = pos->next; pos != (head); \ pos = n, n = pos->next) /** * list_for_each_prev_safe - iterate over a list backwards safe against removal of list entry * @pos: the &struct list_head to use as a loop cursor. * @n: another &struct list_head to use as temporary storage * @head: the head for your list. */ #define list_for_each_prev_safe(pos, n, head) \ for (pos = (head)->prev, n = pos->prev; \ pos != (head); \ pos = n, n = pos->prev) /** * list_for_each_entry - iterate over list of given type * @pos: the type * to use as a loop cursor. * @head: the head for your list. * @member: the name of the list_struct within the struct. */ #define list_for_each_entry(pos, head, member) \ for (pos = list_entry((head)->next, typeof(*pos), member); \ &pos->member != (head); \ pos = list_entry(pos->member.next, typeof(*pos), member)) /** * list_for_each_entry_reverse - iterate backwards over list of given type. * @pos: the type * to use as a loop cursor. * @head: the head for your list. * @member: the name of the list_struct within the struct. */ #define list_for_each_entry_reverse(pos, head, member) \ for (pos = list_entry((head)->prev, typeof(*pos), member); \ &pos->member != (head); \ pos = list_entry(pos->member.prev, typeof(*pos), member)) /** * list_prepare_entry - prepare a pos entry for use in list_for_each_entry_continue() * @pos: the type * to use as a start point * @head: the head of the list * @member: the name of the list_struct within the struct. * * Prepares a pos entry for use as a start point in list_for_each_entry_continue(). */ #define list_prepare_entry(pos, head, member) \ ((pos) ? : list_entry(head, typeof(*pos), member)) /** * list_for_each_entry_continue - continue iteration over list of given type * @pos: the type * to use as a loop cursor. * @head: the head for your list. * @member: the name of the list_struct within the struct. * * Continue to iterate over list of given type, continuing after * the current position. */ #define list_for_each_entry_continue(pos, head, member) \ for (pos = list_entry(pos->member.next, typeof(*pos), member); \ &pos->member != (head); \ pos = list_entry(pos->member.next, typeof(*pos), member)) /** * list_for_each_entry_continue_reverse - iterate backwards from the given point * @pos: the type * to use as a loop cursor. * @head: the head for your list. * @member: the name of the list_struct within the struct. * * Start to iterate over list of given type backwards, continuing after * the current position. */ #define list_for_each_entry_continue_reverse(pos, head, member) \ for (pos = list_entry(pos->member.prev, typeof(*pos), member); \ &pos->member != (head); \ pos = list_entry(pos->member.prev, typeof(*pos), member)) /** * list_for_each_entry_from - iterate over list of given type from the current point * @pos: the type * to use as a loop cursor. * @head: the head for your list. * @member: the name of the list_struct within the struct. * * Iterate over list of given type, continuing from current position. */ #define list_for_each_entry_from(pos, head, member) \ for (; &pos->member != (head); \ pos = list_entry(pos->member.next, typeof(*pos), member)) /** * list_for_each_entry_safe - iterate over list of given type safe against removal of list entry * @pos: the type * to use as a loop cursor. * @n: another type * to use as temporary storage * @head: the head for your list. * @member: the name of the list_struct within the struct. */ #define list_for_each_entry_safe(pos, n, head, member) \ for (pos = list_entry((head)->next, typeof(*pos), member), \ n = list_entry(pos->member.next, typeof(*pos), member); \ &pos->member != (head); \ pos = n, n = list_entry(n->member.next, typeof(*n), member)) /** * list_for_each_entry_safe_continue - continue list iteration safe against removal * @pos: the type * to use as a loop cursor. * @n: another type * to use as temporary storage * @head: the head for your list. * @member: the name of the list_struct within the struct. * * Iterate over list of given type, continuing after current point, * safe against removal of list entry. */ #define list_for_each_entry_safe_continue(pos, n, head, member) \ for (pos = list_entry(pos->member.next, typeof(*pos), member), \ n = list_entry(pos->member.next, typeof(*pos), member); \ &pos->member != (head); \ pos = n, n = list_entry(n->member.next, typeof(*n), member)) /** * list_for_each_entry_safe_from - iterate over list from current point safe against removal * @pos: the type * to use as a loop cursor. * @n: another type * to use as temporary storage * @head: the head for your list. * @member: the name of the list_struct within the struct. * * Iterate over list of given type from current point, safe against * removal of list entry. */ #define list_for_each_entry_safe_from(pos, n, head, member) \ for (n = list_entry(pos->member.next, typeof(*pos), member); \ &pos->member != (head); \ pos = n, n = list_entry(n->member.next, typeof(*n), member)) /** * list_for_each_entry_safe_reverse - iterate backwards over list safe against removal * @pos: the type * to use as a loop cursor. * @n: another type * to use as temporary storage * @head: the head for your list. * @member: the name of the list_struct within the struct. * * Iterate backwards over list of given type, safe against removal * of list entry. */ #define list_for_each_entry_safe_reverse(pos, n, head, member) \ for (pos = list_entry((head)->prev, typeof(*pos), member), \ n = list_entry(pos->member.prev, typeof(*pos), member); \ &pos->member != (head); \ pos = n, n = list_entry(n->member.prev, typeof(*n), member)) /** * list_safe_reset_next - reset a stale list_for_each_entry_safe loop * @pos: the loop cursor used in the list_for_each_entry_safe loop * @n: temporary storage used in list_for_each_entry_safe * @member: the name of the list_struct within the struct. * * list_safe_reset_next is not safe to use in general if the list may be * modified concurrently (eg. the lock is dropped in the loop body). An * exception to this is if the cursor element (pos) is pinned in the list, * and list_safe_reset_next is called after re-taking the lock and before * completing the current iteration of the loop body. */ #define list_safe_reset_next(pos, n, member) \ n = list_entry(pos->member.next, typeof(*pos), member) #endif sanlock/src/lockfile.c000066400000000000000000000037671516326262600152610ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock_internal.h" #include "log.h" #include "lockfile.h" int lockfile(const char *dir, const char *name, int uid, int gid) { char path[PATH_MAX]; char buf[16]; struct flock lock; mode_t old_umask; int fd, rv; /* Make rundir group writable, allowing creation of the lockfile when * starting as root. */ old_umask = umask(0002); rv = mkdir(dir, 0775); if (rv < 0 && errno != EEXIST) { umask(old_umask); return rv; } umask(old_umask); rv = chown(dir, uid, gid); if (rv < 0) { log_error("lockfile chown error %s: %s", dir, strerror(errno)); return rv; } snprintf(path, PATH_MAX, "%s/%s", dir, name); fd = open(path, O_CREAT|O_WRONLY|O_CLOEXEC, 0644); if (fd < 0) { log_error("lockfile open error %s: %s", path, strerror(errno)); return -1; } lock.l_type = F_WRLCK; lock.l_start = 0; lock.l_whence = SEEK_SET; lock.l_len = 0; rv = fcntl(fd, F_SETLK, &lock); if (rv < 0) { log_error("lockfile setlk error %s: %s", path, strerror(errno)); goto fail; } rv = ftruncate(fd, 0); if (rv < 0) { log_error("lockfile truncate error %s: %s", path, strerror(errno)); goto fail; } memset(buf, 0, sizeof(buf)); snprintf(buf, sizeof(buf), "%d\n", getpid()); rv = write(fd, buf, strlen(buf)); if (rv <= 0) { log_error("lockfile write error %s: %s", path, strerror(errno)); goto fail; } return fd; fail: close(fd); return -1; } sanlock/src/lockfile.h000066400000000000000000000006121516326262600152500ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __LOCKFILE_H__ #define __LOCKFILE_H__ int lockfile(const char *dir, const char *name, int uid, int gid); #endif sanlock/src/lockspace.c000066400000000000000000002222571516326262600154320ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock_internal.h" #include "sanlock_admin.h" #include "sanlock_sock.h" #include "diskio.h" #include "ondisk.h" #include "log.h" #include "delta_lease.h" #include "lockspace.h" #include "resource.h" #include "watchdog.h" #include "task.h" #include "timeouts.h" #include "direct.h" #include "helper.h" static uint32_t space_id_counter = 1; /* * When the sanlock daemon is not root, set_max_sectors_kb() needs to use the * root helper process to write to sysfs. */ static int helper_set_max_sectors_kb(struct sync_disk *disk, uint32_t set_kb) { char sysfs_path[SANLK_HELPER_PATH_LEN] = { 0 }; struct helper_msg hm; struct stat st; int ma, mi; unsigned int cur_kb = 0; int rv; rv = stat(disk->path, &st); if (rv < 0) { log_debug("helper_set_max_sectors_kb stat error %d %s", errno, disk->path); return -1; } ma = (int)major(st.st_rdev); mi = (int)minor(st.st_rdev); snprintf(sysfs_path, sizeof(sysfs_path), "/sys/dev/block/%d:%d/queue/max_sectors_kb", ma, mi); sysfs_path[SANLK_HELPER_PATH_LEN-1] = '\0'; rv = read_sysfs_uint(sysfs_path, &cur_kb); if (rv < 0) { /* max_sectors_kb setting may not exist */ return -1; } if (cur_kb >= set_kb) return 0; if (helper_kill_fd == -1) return -1; memset(&hm, 0, sizeof(hm)); hm.type = HELPER_MSG_WRITE_SYSFS; memcpy(hm.path, sysfs_path, SANLK_HELPER_PATH_LEN-1); snprintf(hm.args, sizeof(hm.args)-1, "%u", set_kb); retry: rv = write(helper_kill_fd, &hm, sizeof(hm)); if (rv == -1 && errno == EINTR) goto retry; /* pipe is full, we'll try again in a second */ if (rv == -1 && errno == EAGAIN) { log_error("helper_set_max_sectors_kb send EAGAIN"); return -1; } /* helper exited or closed fd, quit using helper */ if (rv == -1 && errno == EPIPE) { log_error("helper_set_max_sectors_kb send EPIPE"); return -1; } if (rv == -1) { log_error("helper_set_max_sectors_kb send errno %d", errno); return rv; } /* We don't try to wait for the helper process to do the write, although we could probably do something with the status msg. It shouldn't matter when the sysfs field is written wrt reading/writing the device. Add a slight delay here which should usually let the sysfs update happen first. */ usleep(2000); return 0; } static struct space *_search_space(const char *name, struct sync_disk *disk, uint64_t host_id, struct list_head *head1, struct list_head *head2, struct list_head *head3, int *listnum) { int i; struct space *sp; struct list_head *heads[] = {head1, head2, head3}; for (i = 0; i < 3; i++) { if (!heads[i]) { continue; } list_for_each_entry(sp, heads[i], list) { if (name && strncmp(sp->space_name, name, NAME_ID_SIZE)) continue; if (disk && strncmp(sp->host_id_disk.path, disk->path, SANLK_PATH_LEN)) continue; if (disk && sp->host_id_disk.offset != disk->offset) continue; if (host_id && sp->host_id != host_id) continue; if (listnum) *listnum = i+1; return sp; } } return NULL; } struct space *find_lockspace(const char *name) { return _search_space(name, NULL, 0, &spaces, &spaces_rem, &spaces_add, NULL); } static struct space *find_lockspace_id(uint32_t space_id) { struct space *sp; list_for_each_entry(sp, &spaces, list) { if (sp->space_id == space_id) return sp; } return NULL; } static void _set_space_info(struct space *sp, struct space_info *spi) { /* keep this in sync with any new fields added to struct space_info */ spi->space_id = sp->space_id; spi->io_timeout = sp->io_timeout; spi->sector_size = sp->sector_size; spi->align_size = sp->align_size; spi->host_id = sp->host_id; spi->host_generation = sp->host_generation; spi->killing_pids = sp->killing_pids; } int _lockspace_info(const char *space_name, struct space_info *spi) { struct space *sp; list_for_each_entry(sp, &spaces, list) { if (strncmp(sp->space_name, space_name, NAME_ID_SIZE)) continue; _set_space_info(sp, spi); return 0; } return -1; } int lockspace_info(const char *space_name, struct space_info *spi) { int rv; pthread_mutex_lock(&spaces_mutex); rv = _lockspace_info(space_name, spi); pthread_mutex_unlock(&spaces_mutex); return rv; } int lockspace_disk(char *space_name, struct sync_disk *disk, int *sector_size) { struct space *sp; int rv = -1; pthread_mutex_lock(&spaces_mutex); list_for_each_entry(sp, &spaces, list) { if (strncmp(sp->space_name, space_name, NAME_ID_SIZE)) continue; memcpy(disk, &sp->host_id_disk, sizeof(struct sync_disk)); *sector_size = sp->sector_size; disk->fd = -1; rv = 0; } pthread_mutex_unlock(&spaces_mutex); return rv; } #if 0 static void clear_bit(int host_id, char *bitmap) { char *byte = bitmap + ((host_id - 1) / 8); unsigned int bit = host_id % 8; *byte &= ~bit; } #endif /* FIXME: another copy in direct_lib.c */ int test_id_bit(int host_id, char *bitmap) { char *byte = bitmap + ((host_id - 1) / 8); unsigned int bit = (host_id - 1) % 8; char mask; mask = 1 << bit; return (*byte & mask); } int host_status_set_bit(char *space_name, uint64_t host_id) { struct space *sp; int found = 0; if (!host_id || host_id > DEFAULT_MAX_HOSTS) return -EINVAL; pthread_mutex_lock(&spaces_mutex); list_for_each_entry(sp, &spaces, list) { if (strncmp(sp->space_name, space_name, NAME_ID_SIZE)) continue; found = 1; break; } pthread_mutex_unlock(&spaces_mutex); if (!found) return -ENOSPC; if (host_id > sp->max_hosts) return -EINVAL; pthread_mutex_lock(&sp->mutex); sp->host_status[host_id-1].set_bit_time = monotime(); pthread_mutex_unlock(&sp->mutex); return 0; } int host_info(char *space_name, uint64_t host_id, struct host_status *hs_out) { struct space *sp; int found = 0; int toobig = 0; int notready = 0; if (!host_id || host_id > DEFAULT_MAX_HOSTS) return -EINVAL; pthread_mutex_lock(&spaces_mutex); list_for_each_entry(sp, &spaces, list) { if (strncmp(sp->space_name, space_name, NAME_ID_SIZE)) continue; if (host_id > sp->max_hosts) { toobig = 1; break; } memcpy(hs_out, &sp->host_status[host_id-1], sizeof(struct host_status)); found = 1; if (!hs_out->owner_id || !hs_out->first_check || !hs_out->last_check) { /* happens when host_info() is used before local lockspace has run long enough to check other leases. */ log_space(sp, "host_info %llu: not ready %llu,%llu,%llu,%llu,%llu,%llu,%u", (unsigned long long)host_id, (unsigned long long)hs_out->owner_id, (unsigned long long)hs_out->owner_generation, (unsigned long long)hs_out->timestamp, (unsigned long long)hs_out->first_check, (unsigned long long)hs_out->last_check, (unsigned long long)hs_out->last_live, hs_out->dead_ext); notready = 1; } else { log_space(sp, "host_info %llu: %llu %llu %llu first_check %llu last_check %llu last_live %llu dead_ext %u", (unsigned long long)host_id, (unsigned long long)hs_out->owner_id, (unsigned long long)hs_out->owner_generation, (unsigned long long)hs_out->timestamp, (unsigned long long)hs_out->first_check, (unsigned long long)hs_out->last_check, (unsigned long long)hs_out->last_live, hs_out->dead_ext); if (!hs_out->io_timeout) { log_erros(sp, "host_info %llu use own io_timeout %d", (unsigned long long)host_id, sp->io_timeout); hs_out->io_timeout = sp->io_timeout; } } break; } pthread_mutex_unlock(&spaces_mutex); if (notready) return -EAGAIN; if (toobig) return -EINVAL; if (!found) return -ENOSPC; return 0; } static void create_bitmap_and_extra(struct space *sp, char *bitmap, struct delta_extra *extra) { uint64_t now; int i; char c; now = monotime(); pthread_mutex_lock(&sp->mutex); for (i = 0; i < sp->max_hosts; i++) { if (i+1 == sp->host_id) continue; if (!sp->host_status[i].set_bit_time) continue; if (now - sp->host_status[i].set_bit_time > sp->set_bitmap_seconds) { /* log_space(sp, "bitmap clear host_id %d", i+1); */ sp->host_status[i].set_bit_time = 0; } else { set_id_bit(i+1, bitmap, &c); /* log_space(sp, "bitmap set host_id %d byte %x", i+1, c); */ } } extra->field1 = sp->host_event.generation; extra->field2 = sp->host_event.event; extra->field3 = sp->host_event.data; pthread_mutex_unlock(&sp->mutex); } /* * Called from main thread to look through the lease data collected in * the last renewal. Records liveness history about other hosts in the * lockspace, checks if another host is notifying us (through their bitmap) * to look at resource requests or an event they've written. * * NB. the way that this gets the copy of all leases to look at is * unfortunately very subtle and convoluted. * * The lockspace thread makes a copy of task iobuf, which holds all * delta leases that were read in the last renewal, into * sp->lease_status.renewal_read_buf. Then check_our_lease() called * by the main loop makes a copy of sp->lease_status.renewal_read_buf * to pass to this function. */ void check_other_leases(struct space *sp, char *buf) { struct leader_record leader_in; struct leader_record *leader_end; struct leader_record *leader; struct host_status *hs; struct sanlk_host_event he; char *bitmap; uint64_t now; uint64_t last; uint32_t other_io_timeout; int other_host_fail_seconds; int other_host_dead_seconds; int i, new; now = monotime(); new = 0; for (i = 0; i < sp->max_hosts; i++) { hs = &sp->host_status[i]; hs->last_check = now; if (!hs->first_check) hs->first_check = now; leader_end = (struct leader_record *)(buf + (i * sp->sector_size)); leader_record_in(leader_end, &leader_in); leader = &leader_in; if (!hs->owner_id && !leader->owner_id) continue; /* * If this lease has invalid fields, log an error. Limit the logging * frequency to avoid blowing up logs if some lease is left in a bad * state for a long time. Should we check other fields in addition * to magic and space_name? */ if ((leader->magic != DELTA_DISK_MAGIC) || (strncmp(leader->space_name, sp->space_name, NAME_ID_SIZE))) { if (!hs->lease_bad || !(hs->lease_bad % 100)) { log_erros(sp, "check_other_lease invalid for host %llu %llu ts %llu name %.48s in %.48s", (unsigned long long)hs->owner_id, (unsigned long long)hs->owner_generation, (unsigned long long)hs->timestamp, hs->owner_name, sp->space_name); log_erros(sp, "check_other_lease leader %x owner %llu %llu ts %llu sn %.48s rn %.48s", leader->magic, (unsigned long long)leader->owner_id, (unsigned long long)leader->owner_generation, (unsigned long long)leader->timestamp, leader->space_name, leader->resource_name); } hs->lease_bad++; if (!hs->lease_bad) hs->lease_bad++; } else { if (hs->lease_bad) { log_erros(sp, "check_other_lease corrected for host %llu %llu ts %llu name %.48s in %.48s", (unsigned long long)hs->owner_id, (unsigned long long)hs->owner_generation, (unsigned long long)hs->timestamp, hs->owner_name, sp->space_name); } hs->lease_bad = 0; } /* * Save a record of each new host instance to help with debugging. */ if (!hs->lease_bad && (strncmp(hs->owner_name, leader->resource_name, NAME_ID_SIZE) || (hs->owner_generation != leader->owner_generation))) { log_warns(sp, "host %llu %llu %llu %.48s", (unsigned long long)leader->owner_id, (unsigned long long)leader->owner_generation, (unsigned long long)leader->timestamp, leader->resource_name); memcpy(hs->owner_name, leader->resource_name, NAME_ID_SIZE); /* log warning for each generation of a host_id */ hs->warned_fail = 0; hs->warned_dead = 0; hs->warned_dead_ext = 0; } if (!hs->lease_bad) { hs->no_timeout = (leader->flags & LFL_NO_TIMEOUT) ? 1 : 0; hs->dead_ext = (leader->flags & LFL_DEAD_EXT) ? 1 : 0; if ((hs->no_timeout || hs->dead_ext) && (leader->version <= DELTA_DISK_VERSION_34)) { hs->no_timeout = 0; hs->dead_ext = 0; if (!hs->warned_dead_ext) log_erros(sp, "check_other_lease %llu %llu version 0x%x should not have flags 0x%x", (unsigned long long)leader->owner_id, (unsigned long long)leader->owner_generation, leader->version, leader->flags); hs->warned_dead_ext = 1; } if (hs->dead_ext && !hs->warned_dead_ext) { log_warns(sp, "host %llu %llu %llu is dead (ext)", (unsigned long long)leader->owner_id, (unsigned long long)leader->owner_generation, (unsigned long long)leader->timestamp); hs->warned_dead_ext = 1; } } /* * Other host's timestamp has not changed since we last checked it. */ if (hs->owner_id == leader->owner_id && hs->owner_generation == leader->owner_generation && hs->timestamp == leader->timestamp) { /* * No host state to update since the lease is unchanged. * If host check debugging is enabled, we may want to * log info about another host lease fail/dead state. */ if (!com.debug_hosts) continue; /* * Our own renewal info is logged elsewhere, this * is about other hosts lease renewals. */ if (sp->host_id == hs->owner_id) continue; /* * Other host lease is released, not failing or dead. */ if (!hs->timestamp) continue; /* * Don't bother logging info about other host leases if * we are quitting/failing ourself. */ if (sp->renew_fail || sp->space_dead || sp->killing_pids) continue; other_io_timeout = hs->io_timeout; other_host_fail_seconds = calc_id_renewal_fail_seconds(other_io_timeout); other_host_dead_seconds = calc_host_dead_seconds(other_io_timeout); last = hs->last_live ? hs->last_live : hs->first_check; /* * Informational/debug only, these fail/dead * determinations are not used anywhere. */ if (!hs->warned_fail && (com.debug_hosts == 2) && (hs->last_check - last > other_host_fail_seconds)) { log_space(sp, "host %llu %llu %llu is failing", (unsigned long long)hs->owner_id, (unsigned long long)hs->owner_generation, (unsigned long long)hs->timestamp); hs->warned_fail = 1; } if (!hs->warned_dead && !hs->no_timeout && (hs->last_check - last > other_host_dead_seconds)) { log_warns(sp, "host %llu %llu %llu is dead", (unsigned long long)hs->owner_id, (unsigned long long)hs->owner_generation, (unsigned long long)hs->timestamp); hs->warned_dead = 1; } continue; } /* * Log debug messages for each renewal seen from another host. */ if ((com.debug_hosts == 2) && hs->owner_id) { /* hs->owner_id: skip first time set */ if ((hs->owner_id != leader->owner_id) || (hs->owner_generation != leader->owner_generation)) { log_space(sp, "host %llu %llu %llu updated to %llu %llu %llu", (unsigned long long)hs->owner_id, (unsigned long long)hs->owner_generation, (unsigned long long)hs->timestamp, (unsigned long long)leader->owner_id, (unsigned long long)leader->owner_generation, (unsigned long long)leader->timestamp); } else if (hs->timestamp != leader->timestamp) { /* Normal renewal, or release if new timestamp is 0. */ log_space(sp, "host %llu %llu %llu new timestamp %llu", (unsigned long long)hs->owner_id, (unsigned long long)hs->owner_generation, (unsigned long long)hs->timestamp, (unsigned long long)leader->timestamp); } } /* * Replacing good values with potentially bad values * would have no purpose, and would confuse things, so * don't replace these fields if the lease is bad. * But, continue to update the timestamp because we don't * want to consider the host to be dead if the lease * is being renewed, even if the lease has bad fields. */ if (!hs->lease_bad) { hs->owner_id = leader->owner_id; hs->owner_generation = leader->owner_generation; memcpy(hs->owner_name, leader->resource_name, NAME_ID_SIZE); hs->io_timeout = leader->io_timeout; hs->warned_fail = 0; hs->warned_dead = 0; hs->warned_dead_ext = 0; } hs->timestamp = leader->timestamp; hs->last_live = now; if (i+1 == sp->host_id) continue; bitmap = (char *)leader_end + HOSTID_BITMAP_OFFSET; if (!test_id_bit(sp->host_id, bitmap)) continue; /* * Our bit is set in the bitmap, so this host is * notifying us of a host_event or resource request. */ memset(&he, 0, sizeof(he)); he.host_id = sp->host_id; he.generation = leader->write_id; he.event = leader->write_generation; he.data = leader->write_timestamp; /* * Pass an event to the resource_thread which is a * convenient place to do callbacks (we don't want * the main thread to be delayed with that.) */ if (he.event) { /* * lock order: spaces_mutex (main_loop), then * resource_mutex (add_host_event). */ log_space(sp, "host event from host_id %d", i+1); add_host_event(sp->space_id, &he, hs->owner_id, hs->owner_generation); } /* this host has made a resource request for us, we won't take a new request from this host for another set_bitmap_seconds */ if (now - hs->last_req < sp->set_bitmap_seconds) continue; log_space(sp, "request from host_id %d", i+1); hs->last_req = now; new = 1; } /* * Have the resource_thread check the request records of resources * in this lockspace. */ if (new) set_resource_examine(sp->space_name, NULL); } /* * check if our_host_id_thread has renewed within timeout */ int check_our_lease(struct space *sp, int *check_all, char *check_buf) { int id_renewal_fail_seconds, id_renewal_warn_seconds; uint64_t last_success; int corrupt_result; int gap; pthread_mutex_lock(&sp->mutex); last_success = sp->lease_status.renewal_last_success; corrupt_result = sp->lease_status.corrupt_result; if (sp->lease_status.renewal_read_count > sp->lease_status.renewal_read_check) { /* * NB. it's unfortunate how subtle this is. * main loop will pass this buf to check_other_leases next */ sp->lease_status.renewal_read_check = sp->lease_status.renewal_read_count; *check_all = 1; if (check_buf) memcpy(check_buf, sp->lease_status.renewal_read_buf, sp->align_size); } pthread_mutex_unlock(&sp->mutex); if (corrupt_result) { log_erros(sp, "check_our_lease corrupt %d", corrupt_result); return -1; } gap = monotime() - last_success; id_renewal_fail_seconds = calc_id_renewal_fail_seconds(sp->io_timeout); id_renewal_warn_seconds = calc_id_renewal_warn_seconds(sp->io_timeout); if (gap >= id_renewal_fail_seconds) { log_erros(sp, "check_our_lease failed %d%s", gap, sp->no_timeout ? " (no_timeout)" : ""); if (sp->no_timeout) return 0; return -1; } if (gap >= id_renewal_warn_seconds) { log_erros(sp, "check_our_lease warning %d last_success %llu", gap, (unsigned long long)last_success); } if (com.debug_renew > 1) { log_space(sp, "check_our_lease good %d %llu", gap, (unsigned long long)last_success); } return 0; } /* If a renewal result is one of the listed errors, it means our delta lease has been corrupted/overwritten/reinitialized out from under us, and we should stop using it immediately. There's no point in retrying the renewal. */ static int corrupt_result(int result) { switch (result) { case SANLK_RENEW_OWNER: case SANLK_RENEW_DIFF: case SANLK_LEADER_MAGIC: case SANLK_LEADER_VERSION: case SANLK_LEADER_SECTORSIZE: case SANLK_LEADER_LOCKSPACE: case SANLK_LEADER_CHECKSUM: case SANLK_RENEW_DEAD_EXT: return result; default: return 0; } } static void close_event_fds(struct space *sp) { int i; pthread_mutex_lock(&sp->mutex); for (i = 0; i < MAX_EVENT_FDS; i++) { if (sp->event_fds[i] == -1) continue; close(sp->event_fds[i]); sp->event_fds[i] = -1; } pthread_mutex_unlock(&sp->mutex); } /* * if delta_result is success: * new record saving last_success (timestamp in renewal), rd_ms, wr_ms * if delta_result is timeout: * increment next_timeouts in prev record * if delta_result is failure: * increment next_errors in prev record */ static void save_renewal_history(struct space *sp, int delta_result, uint64_t last_success, int rd_ms, int wr_ms) { struct renewal_history *hi; if (!sp->renewal_history_size || !sp->renewal_history) return; if (delta_result == SANLK_OK) { hi = &sp->renewal_history[sp->renewal_history_next]; hi->timestamp = last_success; hi->read_ms = rd_ms; hi->write_ms = wr_ms; hi->next_timeouts = 0; hi->next_errors = 0; sp->renewal_history_prev = sp->renewal_history_next; sp->renewal_history_next++; if (sp->renewal_history_next >= sp->renewal_history_size) sp->renewal_history_next = 0; } else { hi = &sp->renewal_history[sp->renewal_history_prev]; if (delta_result == SANLK_AIO_TIMEOUT) hi->next_timeouts++; else hi->next_errors++; } } static void set_lockspace_max_sectors_kb(struct space *sp, int sector_size, int align_size) { struct stat st; int align_size_kb = align_size / 1024; /* align_size is in bytes */ unsigned int hw_kb = 0; unsigned int cur_kb = 0; unsigned int set_kb = 0; int rv; if (fstat(sp->host_id_disk.fd, &st) < 0) { log_erros(sp, "set_lockspace_max_sectors_kb fstat error %d", errno); return; } /* file not device */ if (S_ISREG(st.st_mode)) return; if (com.max_sectors_kb_ignore) return; else if (com.max_sectors_kb_align) set_kb = align_size_kb; else if (com.max_sectors_kb_num) set_kb = com.max_sectors_kb_num; else return; rv = read_sysfs_size(sp->host_id_disk.path, "max_hw_sectors_kb", &hw_kb); if (rv < 0 || !hw_kb) { log_space(sp, "set_lockspace_max_sectors_kb max_hw_sectors_kb unknown %d %u", rv, hw_kb); return; } rv = read_sysfs_size(sp->host_id_disk.path, "max_sectors_kb", &cur_kb); if (rv < 0 || !cur_kb) { log_space(sp, "set_lockspace_max_sectors_kb max_sectors_kb unknown %d %u", rv, cur_kb); return; } if (cur_kb >= set_kb) { log_space(sp, "set_lockspace_max_sectors_kb keep current %u for set %u", cur_kb, set_kb); return; } if (hw_kb < set_kb) { /* * If the hardware won't support requested size, try setting 1MB. */ if (hw_kb < ONE_MB_IN_KB) { log_space(sp, "set_lockspace_max_sectors_kb small hw_kb %u req_kb %u", hw_kb, set_kb); return; } if (set_kb < 1024) { log_space(sp, "set_lockspace_max_sectors_kb small hw_kb %u small req_kb %u", hw_kb, set_kb); return; } set_kb = ONE_MB_IN_KB; log_space(sp, "set_lockspace_max_sectors_kb small hw_kb %u using 1024", hw_kb); if (!com.uid) rv = set_max_sectors_kb(&sp->host_id_disk, set_kb); else rv = helper_set_max_sectors_kb(&sp->host_id_disk, set_kb); if (rv < 0) { log_space(sp, "set_lockspace_max_sectors_kb small hw_kb %u set 1024 error %d", hw_kb, rv); return; } } else { /* * Tell the kernel to send hardware io's as large as the lease size. */ log_space(sp, "set_lockspace_max_sectors_kb hw_kb %u setting %u", hw_kb, set_kb); if (!com.uid) rv = set_max_sectors_kb(&sp->host_id_disk, set_kb); else rv = helper_set_max_sectors_kb(&sp->host_id_disk, set_kb); if (rv < 0) { log_space(sp, "set_lockspace_max_sectors_kb hw_kb %u set %u error %d", hw_kb, set_kb, rv); return; } } sp->set_max_sectors_kb = set_kb; } int open_space_caw(struct space *sp) { char caw_path[PATH_MAX] = { 0 }; uint32_t major = sp->disk_major; uint32_t minor = sp->disk_minor; uint32_t caw_major; uint32_t caw_minor; uint64_t disk_offset_sect512; uint64_t caw_offset_sect512; int err, caw_fd; if (!dev_is_lv(major, minor)) { if (!dev_is_scsi(major, minor) && !dev_is_mpath(major, minor)) { log_error("open_space_caw unsupported dev type %u:%u", major, minor); return -1; } /* submit caw to host_id_disk */ sp->caw_dev_use = 0; return 0; } /* * SGIO compare-and-write must be submitted to a scsi LUN or mpath device. * dm rejects SGIO to an LV. */ /* dm units are always 512 byte sectors */ disk_offset_sect512 = sp->host_id_disk.offset / 512; err = lv_to_target(major, minor, disk_offset_sect512, &caw_major, &caw_minor, &caw_offset_sect512); if (err < 0) { log_error("open_space_caw unsupported LV %u:%u", major, minor); return -1; } if (devno_to_path(caw_major, caw_minor, caw_path) < 0) { log_error("open_space_caw disk %u:%u devno_to_path %u:%u error", major, minor, caw_major, caw_minor); return -1; } if ((caw_fd = open(caw_path, O_RDWR | O_DIRECT | O_SYNC, 0)) < 0) { log_error("open_space_caw disk %u:%u caw %u:%u %s open errno %d", major, minor, caw_major, caw_minor, caw_path, errno); return -1; } if (!caw_fd) { log_error("open_space_caw disk %u:%u caw %u:%u %s open fd %d", major, minor, caw_major, caw_minor, caw_path, caw_fd); return -1; } sp->caw_dev_use = 1; sp->caw_dev_fd = caw_fd; sp->caw_dev_offset = caw_offset_sect512 * 512; sp->caw_dev_major = caw_major; sp->caw_dev_minor = caw_minor; log_space(sp, "disk %u:%u offset %llu is caw_dev %u:%u offset %llu path %s fd %d", major, minor, (unsigned long long)sp->host_id_disk.offset, caw_major, caw_minor, (unsigned long long)sp->caw_dev_offset, caw_path, caw_fd); return 0; } int open_space(struct space *sp) { struct stat st; int rv; if (stat(sp->host_id_disk.path, &st) < 0) { log_erros(sp, "open_space stat %s errno %d", sp->host_id_disk.path, errno); return -ENODEV; } sp->disk_major = (uint32_t)major(st.st_rdev); sp->disk_minor = (uint32_t)minor(st.st_rdev); rv = open_disk(&sp->host_id_disk); if (rv < 0) { log_erros(sp, "open_space open_disk %s %u:%u error %d", sp->host_id_disk.path, sp->disk_major, sp->disk_minor, rv); return -ENODEV; } return 0; } /* * This thread must not be stopped unless all pids that may be using any * resources in it are dead/gone. (The USED flag in the lockspace represents * pids using resources in the lockspace, when those pids are not using actual * sanlock resources. So the USED flag must also prevent this thread from * stopping.) */ static void *lockspace_thread(void *arg_in) { char bitmap[HOSTID_BITMAP_SIZE]; struct delta_extra extra; struct task task; struct space *sp; struct leader_record leader; uint64_t delta_begin, last_success = 0; int sector_size = 0; int align_size = 0; int max_hosts = 0; int is_caw = 0; int log_renewal_level = -1; int cur_lockspace_io_timeout; int delta_length, renewal_interval = 0; int id_renewal_seconds, id_renewal_fail_seconds; int acquire_result, delta_result, read_result; int rd_ms, wr_ms; int opened_disk = 0; int opened_caw = 0; int stop = 0; int wd_con; int rv; if (com.debug_renew) log_renewal_level = LOG_DEBUG; sp = (struct space *)arg_in; memset(&task, 0, sizeof(struct task)); setup_task_aio(&task, main_task.use_aio, HOSTID_AIO_CB_SIZE); memcpy(task.name, sp->space_name, NAME_ID_SIZE); cur_lockspace_io_timeout = sp->io_timeout; id_renewal_seconds = calc_id_renewal_seconds(sp->io_timeout); id_renewal_fail_seconds = calc_id_renewal_fail_seconds(sp->io_timeout); delta_begin = monotime(); rv = open_space(sp); if (rv < 0) { acquire_result = -ENODEV; delta_result = -1; goto set_status; } opened_disk = 1; /* * Using aio (if configured) for this first read is * helpful if the lockspace storage is slow, in * which case this can time out and return an error * in lockspace start, where it's most useful to fail, * as opposed to failing once the lockspace has begun * to be used, and failing is more messy. */ rv = delta_read_lockspace_sizes(&task, &sp->host_id_disk, sp->host_id, task.use_aio, sp->io_timeout, §or_size, &align_size, &is_caw); if (rv < 0) { log_erros(sp, "failed to read device to find sector size for host_id %llu error %d %s", (unsigned long long)sp->host_id, rv, sp->host_id_disk.path); acquire_result = rv; delta_result = -1; goto set_status; } if ((sector_size != 512) && (sector_size != 4096)) { log_erros(sp, "failed to get valid sector size %d for host_id %llu %s", sector_size, (unsigned long long)sp->host_id, sp->host_id_disk.path); acquire_result = SANLK_LEADER_SECTORSIZE; delta_result = -1; goto set_status; } max_hosts = size_to_max_hosts(sector_size, align_size); if (!max_hosts) { log_erros(sp, "invalid combination of sector size %d and align_size %d", sector_size, align_size); acquire_result = SANLK_ADDLS_SIZES; delta_result = -1; goto set_status; } if (sp->host_id > max_hosts) { log_erros(sp, "host_id %llu too large for max_hosts %d", (unsigned long long)sp->host_id, max_hosts); acquire_result = SANLK_ADDLS_INVALID_HOSTID; delta_result = -1; goto set_status; } if (is_caw && (com.use_compare_and_write == USE_CAW_NO)) { log_erros(sp, "use_compare_and_write=no prevents use of caw lease"); acquire_result = SANLK_CAW_CONFIG; delta_result = -1; goto set_status; } log_space(sp, "delta leader caw %d sector_size %d align_size %d", is_caw, sector_size, align_size); sp->caw = is_caw; sp->sector_size = sector_size; sp->align_size = align_size; sp->max_hosts = max_hosts; set_lockspace_max_sectors_kb(sp, sector_size, align_size); sp->lease_status.renewal_read_buf = malloc(sp->align_size); if (!sp->lease_status.renewal_read_buf) { acquire_result = -ENOMEM; delta_result = -1; goto set_status; } /* * When using CAW, and host_id_disk is an LV, translate * the LV and offset to a target device and offset for * submitting the SGIO (SGIO cannot be done on an LV). * Also allocate a buffer that can be used for doing all * CAW SGIO on the delta lease. */ if (sp->caw) { char *iobuf = NULL; char **p_iobuf; p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), sector_size * 2); if (rv) { acquire_result = -ENOMEM; delta_result = -1; goto set_status; } rv = open_space_caw(sp); if (rv < 0) { log_erros(sp, "open_space_caw failed"); acquire_result = SANLK_CAW_OPEN; delta_result = -1; free(iobuf); goto set_status; } opened_caw = sp->caw_dev_use; /* * two-sector buffer used for SGIO compare-and-write. * this buffer is not passed to aio read/write functions * so that it does not need to be set to NULL on AIO_TIMEOUT. */ sp->caw_data = iobuf; } /* * Connect first so we can fail quickly if wdmd is not running. */ wd_con = connect_watchdog(sp); if (wd_con < 0) { log_erros(sp, "connect_watchdog failed %d", wd_con); acquire_result = SANLK_WD_ERROR; delta_result = -1; goto set_status; } /* * Tell wdmd to open the watchdog device, set the fire timeout and * begin the keepalive loop that regularly pets the watchdog. This * only happens for the first client/lockspace. This fails if the * watchdog device cannot be opened by wdmd or does not support the * requested fire timeout. * * For later clients/lockspaces, when wdmd already has the watchdog * open, this does nothing (just verifies that fire timeout matches * what's in use.) */ rv = open_watchdog(wd_con, com.watchdog_fire_timeout); if (rv < 0) { log_erros(sp, "open_watchdog with fire_timeout %d failed %d", com.watchdog_fire_timeout, wd_con); acquire_result = SANLK_WD_ERROR; delta_result = -1; disconnect_watchdog(sp); goto set_status; } /* * acquire the delta lease */ delta_begin = monotime(); if (sp->caw) delta_result = delta_caw_lease_acquire(&task, sp, our_host_name_global, &leader); else delta_result = delta_lease_acquire(&task, sp, &sp->host_id_disk, sp->space_name, our_host_name_global, sp->host_id, sp->nodelay, &leader); delta_length = monotime() - delta_begin; if (delta_result == SANLK_OK) last_success = leader.timestamp; acquire_result = delta_result; /* we need to start the watchdog after we acquire the host_id but before we allow any pid's to begin running */ if (delta_result == SANLK_OK) { rv = activate_watchdog(sp, last_success, id_renewal_fail_seconds, wd_con); if (rv < 0) { log_erros(sp, "activate_watchdog failed %d", rv); acquire_result = SANLK_WD_ERROR; } } else { if (com.use_watchdog) close(wd_con); } set_status: pthread_mutex_lock(&sp->mutex); sp->lease_status.acquire_last_result = acquire_result; sp->lease_status.acquire_last_attempt = delta_begin; if (delta_result == SANLK_OK) sp->lease_status.acquire_last_success = last_success; sp->lease_status.renewal_last_result = acquire_result; sp->lease_status.renewal_last_attempt = delta_begin; if (delta_result == SANLK_OK) sp->lease_status.renewal_last_success = last_success; /* First renewal entry shows the acquire time with 0 latencies. */ save_renewal_history(sp, delta_result, last_success, 0, 0); if (acquire_result >= 0) { sp->host_generation = leader.owner_generation; sp->version = leader.version; sp->no_timeout = (leader.flags & LFL_NO_TIMEOUT) ? 1 : 0; } pthread_mutex_unlock(&sp->mutex); if (acquire_result < 0) goto out; while (1) { pthread_mutex_lock(&sp->mutex); stop = sp->thread_stop; /* sp->io_timeout can be updated while running by set_config */ if (cur_lockspace_io_timeout != sp->io_timeout) { cur_lockspace_io_timeout = sp->io_timeout; id_renewal_seconds = calc_id_renewal_seconds(sp->io_timeout); id_renewal_fail_seconds = calc_id_renewal_fail_seconds(sp->io_timeout); } pthread_mutex_unlock(&sp->mutex); if (stop) break; /* * wait between each renewal */ if (monotime() - last_success < id_renewal_seconds) { sleep(1); continue; } else { /* don't spin too quickly if renew is failing immediately and repeatedly */ usleep(500000); } /* * do a renewal, measuring length of time spent in renewal, * and the length of time between successful renewals */ memset(bitmap, 0, sizeof(bitmap)); memset(&extra, 0, sizeof(extra)); create_bitmap_and_extra(sp, bitmap, &extra); delta_begin = monotime(); if (sp->caw) { delta_result = delta_caw_lease_renew(&task, sp, bitmap, &extra, delta_result, &read_result, log_renewal_level, &leader, &leader, &rd_ms, &wr_ms); } else { delta_result = delta_lease_renew(&task, sp, &sp->host_id_disk, sp->space_name, bitmap, &extra, delta_result, &read_result, log_renewal_level, &leader, &leader, &rd_ms, &wr_ms); } delta_length = monotime() - delta_begin; if (delta_result == SANLK_OK) { renewal_interval = leader.timestamp - last_success; last_success = leader.timestamp; } /* * publish the results */ pthread_mutex_lock(&sp->mutex); sp->lease_status.renewal_last_result = delta_result; sp->lease_status.renewal_last_attempt = delta_begin; if (delta_result == SANLK_OK) sp->lease_status.renewal_last_success = last_success; if (delta_result != SANLK_OK && !sp->lease_status.corrupt_result) sp->lease_status.corrupt_result = corrupt_result(delta_result); if (read_result == SANLK_OK && task.iobuf) { /* NB. be careful with how this iobuf escapes */ memcpy(sp->lease_status.renewal_read_buf, task.iobuf, sp->align_size); sp->lease_status.renewal_read_count++; } /* * pet the watchdog * (don't update on thread_stop because it's probably unlinked) */ if (delta_result == SANLK_OK && !sp->thread_stop) update_watchdog(sp, last_success, id_renewal_fail_seconds); save_renewal_history(sp, delta_result, last_success, rd_ms, wr_ms); pthread_mutex_unlock(&sp->mutex); /* * log the results */ if (delta_result != SANLK_OK) { log_erros(sp, "renewal error %d delta_length %d last_success %llu", delta_result, delta_length, (unsigned long long)last_success); } else if (delta_length > id_renewal_seconds) { log_erros(sp, "renewed %llu delta_length %d too long", (unsigned long long)last_success, delta_length); } else { if (com.debug_renew) { log_space(sp, "renewed %llu delta_length %d interval %d", (unsigned long long)last_success, delta_length, renewal_interval); } } } /* watchdog unlink was done in main_loop when thread_stop was set, to get it done as quickly as possible in case the wd is about to fire. */ disconnect_watchdog(sp); out: if (delta_result == SANLK_OK) { if (sp->caw) delta_caw_lease_release(&task, sp, &sp->host_id_disk, sp->space_name, &leader, &leader); else delta_lease_release(&task, sp, &sp->host_id_disk, sp->space_name, &leader, &leader); } if (opened_caw) close(sp->caw_dev_fd); if (opened_disk) close(sp->host_id_disk.fd); /* * TODO: are there cases where struct resources for this lockspace * still exist on resource_held/resource_add/resource_rem? Is that ok? * Should we purge all of them here? When a lockspace is removed and * pids are killed, their resources go through release_token_async, * which will see token->space_dead, and those resources are freed * directly. resources that may have already been on resources_rem and * the resource_thread may be in the middle of releasing one of them. * For any further async releases, resource_thread will see that the * lockspace is going away and will just free the resource. */ purge_resource_orphans(sp->space_name); purge_resource_free(sp->space_name); close_event_fds(sp); close_task_aio(&task); return NULL; } static void free_sp(struct space *sp) { if (sp->lease_status.renewal_read_buf) free(sp->lease_status.renewal_read_buf); if (sp->renewal_history) free(sp->renewal_history); if (sp->caw_data) free(sp->caw_data); free(sp); } int add_lockspace_start(struct sanlk_lockspace *ls, uint32_t io_timeout, int nodelay, struct space **sp_out) { struct space *sp, *sp2; int listnum = 0; int rv; int i; if (!ls->name[0] || !ls->host_id || !ls->host_id_disk.path[0]) { log_error("add_lockspace bad args host_id %llu name %s path %s", (unsigned long long)ls->host_id, ls->name[0] ? "set" : "empty", ls->host_id_disk.path[0] ? "set" : "empty"); return -EINVAL; } sp = malloc(sizeof(struct space)); if (!sp) return -ENOMEM; memset(sp, 0, sizeof(struct space)); memcpy(sp->space_name, ls->name, NAME_ID_SIZE); memcpy(&sp->host_id_disk, &ls->host_id_disk, sizeof(struct sanlk_disk)); sp->host_id_disk.sector_size = 0; sp->host_id_disk.fd = -1; sp->host_id = ls->host_id; sp->io_timeout = io_timeout; sp->nodelay = nodelay; sp->set_bitmap_seconds = calc_set_bitmap_seconds(io_timeout); pthread_mutex_init(&sp->mutex, NULL); if (com.renewal_read_extend_sec_set) sp->renewal_read_extend_sec = com.renewal_read_extend_sec; else sp->renewal_read_extend_sec = io_timeout; for (i = 0; i < MAX_EVENT_FDS; i++) sp->event_fds[i] = -1; if (com.renewal_history_size) { sp->renewal_history = malloc(sizeof(struct renewal_history) * com.renewal_history_size); if (sp->renewal_history) { sp->renewal_history_size = com.renewal_history_size; memset(sp->renewal_history, 0, sizeof(struct renewal_history) * com.renewal_history_size); } } pthread_mutex_lock(&spaces_mutex); /* search all lists for an identical lockspace */ sp2 = _search_space(sp->space_name, &sp->host_id_disk, sp->host_id, &spaces, NULL, NULL, NULL); if (sp2) { pthread_mutex_unlock(&spaces_mutex); rv = -EEXIST; goto fail_free; } sp2 = _search_space(sp->space_name, &sp->host_id_disk, sp->host_id, &spaces_add, NULL, NULL, NULL); if (sp2) { pthread_mutex_unlock(&spaces_mutex); rv = -EINPROGRESS; goto fail_free; } sp2 = _search_space(sp->space_name, &sp->host_id_disk, sp->host_id, &spaces_rem, NULL, NULL, NULL); if (sp2) { pthread_mutex_unlock(&spaces_mutex); rv = -EAGAIN; goto fail_free; } /* search all lists for a lockspace with the same name */ sp2 = _search_space(sp->space_name, NULL, 0, &spaces, &spaces_add, &spaces_rem, &listnum); if (sp2) { log_error("add_lockspace %.48s:%llu:%.256s:%llu conflicts with name of list%d s%d %.48s:%llu:%.256s:%llu", sp->space_name, (unsigned long long)sp->host_id, sp->host_id_disk.path, (unsigned long long)sp->host_id_disk.offset, listnum, sp2->space_id, sp2->space_name, (unsigned long long)sp2->host_id, sp2->host_id_disk.path, (unsigned long long)sp2->host_id_disk.offset); pthread_mutex_unlock(&spaces_mutex); rv = -EINVAL; goto fail_free; } /* search all lists for a lockspace with the same host_id_disk */ sp2 = _search_space(NULL, &sp->host_id_disk, 0, &spaces, &spaces_add, &spaces_rem, &listnum); if (sp2) { log_error("add_lockspace %.48s:%llu:%.256s:%llu conflicts with path of list%d s%d %.48s:%llu:%.256s:%llu", sp->space_name, (unsigned long long)sp->host_id, sp->host_id_disk.path, (unsigned long long)sp->host_id_disk.offset, listnum, sp2->space_id, sp2->space_name, (unsigned long long)sp2->host_id, sp2->host_id_disk.path, (unsigned long long)sp2->host_id_disk.offset); pthread_mutex_unlock(&spaces_mutex); rv = -EINVAL; goto fail_free; } sp->space_id = space_id_counter++; list_add(&sp->list, &spaces_add); pthread_mutex_unlock(&spaces_mutex); /* save a record of what this space_id is for later debugging */ log_warns(sp, "lockspace %.48s:%llu:%.256s:%llu", sp->space_name, (unsigned long long)sp->host_id, sp->host_id_disk.path, (unsigned long long)sp->host_id_disk.offset); rv = pthread_create(&sp->thread, NULL, lockspace_thread, sp); if (rv) { log_erros(sp, "add_lockspace create thread failed %d", rv); rv = -1; goto fail_del; } *sp_out = sp; return 0; fail_del: pthread_mutex_lock(&spaces_mutex); list_del(&sp->list); pthread_mutex_unlock(&spaces_mutex); fail_free: free_sp(sp); return rv; } int add_lockspace_wait(struct space *sp) { int rv, result; while (1) { pthread_mutex_lock(&sp->mutex); result = sp->lease_status.acquire_last_result; pthread_mutex_unlock(&sp->mutex); if (result) break; if (sp->caw) usleep(200000); else sleep(1); } if (result != SANLK_OK) { /* the thread exits right away if acquire fails */ pthread_join(sp->thread, NULL); rv = result; log_erros(sp, "add_lockspace fail result %d", result); goto fail_del; } /* Once we move sp to spaces list, tokens can begin using it, the main loop will begin monitoring its renewals, and will handle removing it. */ pthread_mutex_lock(&spaces_mutex); if (sp->external_remove || external_shutdown) { pthread_mutex_unlock(&spaces_mutex); log_space(sp, "add_lockspace undo remove %d shutdown %d", sp->external_remove, external_shutdown); /* We've caught a remove/shutdown just before completing the add process. Don't complete it, but reverse the add, leaving the sp on spaces_add while reversing. Do the same thing that main_loop would do, except we don't have to go through killing_pids and checking for all_pids_dead since this lockspace has never been on the spaces list, so it could not have been used yet. */ pthread_mutex_lock(&sp->mutex); sp->thread_stop = 1; deactivate_watchdog(sp); pthread_mutex_unlock(&sp->mutex); pthread_join(sp->thread, NULL); rv = -1; log_space(sp, "add_lockspace undo complete"); goto fail_del; } else { list_move(&sp->list, &spaces); log_space(sp, "add_lockspace done"); pthread_mutex_unlock(&spaces_mutex); return 0; } fail_del: pthread_mutex_lock(&spaces_mutex); list_del(&sp->list); pthread_mutex_unlock(&spaces_mutex); free_sp(sp); return rv; } int inq_lockspace(struct sanlk_lockspace *ls) { int rv; struct space *sp; pthread_mutex_lock(&spaces_mutex); sp = _search_space(ls->name, (struct sync_disk *)&ls->host_id_disk, ls->host_id, &spaces, NULL, NULL, NULL); if (sp) { rv = 0; goto out; } else { rv = -ENOENT; } sp = _search_space(ls->name, (struct sync_disk *)&ls->host_id_disk, ls->host_id, &spaces_add, &spaces_rem, NULL, NULL); if (sp) rv = -EINPROGRESS; out: pthread_mutex_unlock(&spaces_mutex); return rv; } int rem_lockspace_start(struct sanlk_lockspace *ls, unsigned int *space_id) { struct space *sp; unsigned int id; int rv; pthread_mutex_lock(&spaces_mutex); sp = _search_space(ls->name, (struct sync_disk *)&ls->host_id_disk, ls->host_id, &spaces_rem, NULL, NULL, NULL); if (sp) { pthread_mutex_unlock(&spaces_mutex); rv = -EINPROGRESS; goto out; } sp = _search_space(ls->name, (struct sync_disk *)&ls->host_id_disk, ls->host_id, &spaces_add, NULL, NULL, NULL); if (sp) { /* add_lockspace will be aborted and undone and the sp will not be moved to the spaces list */ sp->external_remove = 1; id = sp->space_id; pthread_mutex_unlock(&spaces_mutex); *space_id = id; rv = 0; goto out; } sp = _search_space(ls->name, (struct sync_disk *)&ls->host_id_disk, ls->host_id, &spaces, NULL, NULL, NULL); if (!sp) { pthread_mutex_unlock(&spaces_mutex); rv = -ENOENT; goto out; } if (sp->rindex_op) { log_space(sp, "rem_lockspace ignored for rindex_op %d", sp->rindex_op); pthread_mutex_unlock(&spaces_mutex); rv = -EBUSY; goto out; } if (sp->flags & SP_USED_BY_ORPHANS) { /* lock ordering: spaces_mutex, then resource_mutex (resource_orphan_count) */ int orphans = resource_orphan_count(sp->space_name); if (orphans) { log_space(sp, "rem_lockspace ignored for orphan count %d", orphans); pthread_mutex_unlock(&spaces_mutex); rv = -EBUSY; goto out; } } /* * Removal happens in a round about way: * - we set external_remove * - main_loop sees external_remove and sets space_dead, killing_pids * - main_loop sees killing_pids and all pids dead, sets thread_stop, * and moves sp from spaces to spaces_rem * - main_loop calls free_lockspaces(0), which joins any * lockspace_thread that is done, and then frees sp * * Once we release spaces_mutex, the sp could be freed any time, * so we can't touch it. Use its space_id to check for completion. */ sp->external_remove = 1; id = sp->space_id; pthread_mutex_unlock(&spaces_mutex); *space_id = id; rv = 0; out: return rv; } /* check for matching space_id in case the lockspace is added again */ int rem_lockspace_wait(struct sanlk_lockspace *ls, unsigned int space_id) { struct space *sp; int done; while (1) { pthread_mutex_lock(&spaces_mutex); sp = _search_space(ls->name, (struct sync_disk *)&ls->host_id_disk, ls->host_id, &spaces, &spaces_rem, &spaces_add, NULL); if (sp && (sp->space_id == space_id)) done = 0; else done = 1; pthread_mutex_unlock(&spaces_mutex); if (done) break; usleep(500000); } return 0; } int get_lockspaces(char *buf, int *len, int *count, int maxlen) { struct sanlk_lockspace *ls; struct space *sp; struct list_head *heads[] = {&spaces, &spaces_rem, &spaces_add}; int i, rv, sp_count = 0; rv = 0; *len = 0; *count = 0; ls = (struct sanlk_lockspace *)buf; pthread_mutex_lock(&spaces_mutex); for (i = 0; i < 3; i++) { list_for_each_entry(sp, heads[i], list) { sp_count++; if (*len + sizeof(struct sanlk_lockspace) > maxlen) { rv = -ENOSPC; continue; } memcpy(ls->name, sp->space_name, NAME_ID_SIZE); memcpy(&ls->host_id_disk, &sp->host_id_disk, sizeof(struct sync_disk)); ls->host_id_disk.pad1 = 0; ls->host_id_disk.pad2 = 0; ls->host_id = sp->host_id; ls->flags = 0; if (i == 1) ls->flags |= SANLK_LSF_REM; else if (i == 2) ls->flags |= SANLK_LSF_ADD; if (sp->no_timeout) ls->flags |= SANLK_LSF_NO_TIMEOUT; *len += sizeof(struct sanlk_lockspace); ls++; } } pthread_mutex_unlock(&spaces_mutex); *count = sp_count; return rv; } /* * After the lockspace starts, there is a limited amount of * time that we've been watching the other hosts. This means * we can't make an accurate assessment of their state, because * the state is based on monitoring the hosts for host_fail_seconds * and host_dead_seconds, or seeing a renewal. When none of * those are true (not enough time monitoring and not seeing a * renewal), we return UNKNOWN. * * (Example number of seconds below are based on hosts using the * default 10 second io timeout.) * * * For hosts that are alive when we start, we return: * UNKNOWN then LIVE * * UNKNOWN would typically last for 10-20 seconds, but it's possible that * UNKNOWN could persist for up to 80 seconds before LIVE is returned. * LIVE is returned after we see the timestamp change once. * * * For hosts that are dead when we start, we'd return: * UNKNOWN then FAIL then DEAD * * UNKNOWN would last for 80 seconds before we return FAIL. * FAIL would last for 60 more seconds before we return DEAD. * * * Hosts that are failing and don't recover would be the same as prev. * * * For hosts that are failing but recover, we'd return: * UNKNOWN then FAIL then LIVE * * * For another host that is alive when we start, * the sequence of values is: * * 0: we have not yet called check_other_leases() * first_check = 0, last_check = 0, last_live = 0 * * other host renews its lease * * 10: we call check_other_leases() for the first time, * first_check = 10, last_check = 10, last_live = 10 * * other host renews its lease * * 20: we call check_other_leases() for the second time, * first_check = 10, last_check = 20, last_live = 20 * * At 10, we have not yet seen a renewal from the other host, i.e. we have * not seen its timestamp change (we only have one sample). The host could * be dead or alive, so we set the state to UNKNOWN. The way we know * that we have not yet observed the timestamp change is that * first_check == last_live, (10 == 10). * * At 20, we have seen a renewal, i.e. the timestamp changed between checks, * so we return LIVE. * * In the other case, if the host was actually dead, not alive, it would not * have renewed between 10 and 20. So at 20 we would continue to see * first_check == last_live, and would return UNKNOWN. If the host remains * dead, we'd continue to report UNKNOWN for the first 80 seconds. * After 80 seconds, we'd return FAIL. After 140 seconds we'd return DEAD. */ uint32_t get_host_flag(uint64_t local_host_id, struct host_status *hs) { uint32_t other_io_timeout = hs->io_timeout; int other_host_fail_seconds = calc_id_renewal_fail_seconds(other_io_timeout); int other_host_dead_seconds = calc_host_dead_seconds(other_io_timeout); if (hs->dead_ext || (hs->found_dead_ext_gen == hs->owner_generation) || (hs->wrote_dead_ext_gen == hs->owner_generation)) { uint32_t flag = SANLK_HOST_DEAD | SANLK_HOST_DEAD_EXT; if (hs->no_timeout) return flag; /* a host can be DEAD from both DEAD_EXT flag and host_dead_sec timeout, copy the timeout-based DEAD checks from below. */ if (!hs->last_live) { if (hs->last_check - hs->first_check > other_host_dead_seconds) flag |= SANLK_HOST_DEAD_SEC; } else { if (hs->last_check - hs->last_live > other_host_dead_seconds) flag |= SANLK_HOST_DEAD_SEC; } return flag; } /* See sh_holder_live() for same/similar checks */ if (!hs->first_check) return SANLK_HOST_UNKNOWN; if (!hs->last_check) return SANLK_HOST_UNKNOWN; /* The most recent data is older than the fail time. */ if (monotime() - hs->last_check > other_host_fail_seconds) return SANLK_HOST_UNKNOWN | SANLK_HOST_UNKNOWN_OLD; if (!hs->timestamp) return SANLK_HOST_FREE; if (hs->first_check == hs->last_check) return SANLK_HOST_UNKNOWN; if (hs->last_check == hs->last_live) return SANLK_HOST_LIVE; if (!hs->last_live) { if (hs->last_check - hs->first_check <= other_host_fail_seconds) return SANLK_HOST_UNKNOWN; if (!hs->no_timeout && (hs->last_check - hs->first_check > other_host_dead_seconds)) return SANLK_HOST_DEAD; if (hs->last_check - hs->first_check > other_host_fail_seconds) return SANLK_HOST_FAIL; /* shouldn't be possible */ return SANLK_HOST_UNKNOWN; } else { if (!hs->no_timeout && (hs->last_check - hs->last_live > other_host_dead_seconds)) return SANLK_HOST_DEAD; if (hs->last_check - hs->last_live > other_host_fail_seconds) return SANLK_HOST_FAIL; /* last_check == last_live is the common case, covered above */ /* last_check > last_live, but not yet fail_seconds, is live */ return SANLK_HOST_LIVE; } } int get_host_local(struct sanlk_lockspace *ls, struct sanlk_host *host) { struct space *sp; struct host_status *hs; int rv = 0; pthread_mutex_lock(&spaces_mutex); sp = _search_space(ls->name, NULL, 0, &spaces, NULL, NULL, NULL); if (!sp) { rv = -ENOENT; goto out; } hs = &sp->host_status[sp->host_id - 1]; host->host_id = sp->host_id; host->generation = sp->host_generation; host->timestamp = hs->timestamp; host->io_timeout = hs->io_timeout; host->flags = get_host_flag(sp->host_id, hs); out: pthread_mutex_unlock(&spaces_mutex); return rv; } int get_hosts(struct sanlk_lockspace *ls, char *buf, int *len, int *count, int maxlen) { struct space *sp; struct host_status *hs; struct sanlk_host *host; int host_count = 0; int i, rv; rv = 0; *len = 0; *count = 0; host = (struct sanlk_host *)buf; pthread_mutex_lock(&spaces_mutex); sp = _search_space(ls->name, NULL, 0, &spaces, NULL, NULL, NULL); if (!sp) { rv = -ENOENT; goto out; } /* * Between add_lockspace completing and the first * time we call check_other_leases, we don't have * any data on other hosts, so return this error * to indicate this to the caller. */ if (!sp->host_status[0].last_check) { rv = -EAGAIN; goto out; } for (i = 0; i < sp->max_hosts; i++) { hs = &sp->host_status[i]; if (ls->host_id && (ls->host_id != (i + 1))) continue; if (!ls->host_id && !hs->timestamp) continue; host_count++; if (*len + sizeof(struct sanlk_host) > maxlen) { rv = -ENOSPC; continue; } host->host_id = i + 1; host->generation = hs->owner_generation; host->timestamp = hs->timestamp; host->io_timeout = hs->io_timeout; host->flags = get_host_flag(sp->host_id, hs); *len += sizeof(struct sanlk_host); host++; } out: pthread_mutex_unlock(&spaces_mutex); *count = host_count; return rv; } /* A starts PR key A.1 (no local prev_gen) A writes delta lease A.1 (sets local prev_gen 1) A lock R1 writes paxos lease owner A.1 A.1 fails (enters host state FAIL) B lock R1 reads paxos lease owner A.1 (lock fails, owner A.1) B sees A.1 is in FAIL state B removes PR key A.1 A is restarting A starts PR key A.2 (sees local prev_gen 1) B set_host reads delta lease A.1 A start writes delta lease A.2 (sets local prev_gen 2) A lock R2 writes paxos lease owner A.2 A.2 fails (enters host state FAIL) B set_host writes delta lease A.1.DEAD_EXT (delta lease A.2 lost) B lock R1 retry succeeds because paxos is A.1 and delta is A.1.DEAD_EXT B lock R2 reads paxos lease with owner A.2 (lock fails, owner A.2) B sees A.2 is in FAIL state, or sees delta A.1.DEAD_EXT (does this matter?) B removes PR key A.2 A is restarting A starts PR key A.3 (sees local prev_gen 2) A start reads delta lease A.1.DEAD_EXT B set_host reads delta lease A.1.DEAD_EXT A start writes delta lease A.3 (sees local prev_gen 2) (without local prev_gen, it would write delta lease A.2) A lock R3 writes paxos lease owner A.3 B set_host writes delta lease A.2.DEAD_EXT B lock R2 retry succeeds because paxos is A.2 and delta is A.2.DEAD_EXT (if B lock R2 retry was after A renew A.3, it would succeed because paxos is A.2 and delta is A.3) A renew writes delta lease A.3 */ int lockspace_set_host_dead_ext(struct task *task, struct sanlk_lockspace *ls, struct sanlk_host *hs, uint32_t io_timeout) { char *iobuf = NULL; char **p_iobuf; struct space *sp; struct sync_disk sd; struct leader_record leader; uint64_t local_host_id; uint64_t local_generation; uint64_t caw_offset = 0; uint32_t caw_major = 0; uint32_t caw_minor = 0; int is_caw; int max_hosts; int sector_size; int iobuf_len; int found_dead_ext = 0; int wrote_dead_ext = 0; int rv = -1; if (!ls->host_id) return -EINVAL; pthread_mutex_lock(&spaces_mutex); sp = _search_space(ls->name, NULL, 0, &spaces, NULL, NULL, NULL); if (!sp) { pthread_mutex_unlock(&spaces_mutex); return -ENOENT; } max_hosts = sp->max_hosts; sector_size = sp->sector_size; local_host_id = sp->host_id; local_generation = sp->host_generation; if (!io_timeout) io_timeout = sp->io_timeout; memcpy(&sd, &sp->host_id_disk, sizeof(struct sync_disk)); is_caw = sp->caw; if (is_caw) { caw_major = sp->caw_dev_major; caw_minor = sp->caw_dev_minor; caw_offset = sp->caw_dev_offset; } pthread_mutex_unlock(&spaces_mutex); log_debug("lockspace_set_host_dead_ext %.48s host %llu %llu", ls->name, (unsigned long long)hs->host_id, (unsigned long long)hs->generation); if (ls->host_id_disk.path[0] && strcmp(sd.path, ls->host_id_disk.path)) return -EINVAL; if (ls->host_id && (ls->host_id != local_host_id)) return -EINVAL; if (hs->host_id > max_hosts) return -EINVAL; rv = open_disk(&sd); if (rv < 0) { log_error("lockspace_set_host_dead_ext %.48s host %llu %llu open_disk %d", ls->name, (unsigned long long)hs->host_id, (unsigned long long)hs->generation, rv); return rv; } if (is_caw) iobuf_len = sector_size * 2; else iobuf_len = sector_size; p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), iobuf_len); if (rv) { close_disks(&sd, 1); return rv; } rv = delta_lease_leader_read_buf(task, sector_size, task->use_aio, io_timeout, &sd, ls->name, hs->host_id, &leader, iobuf, "set_host_dead_ext"); if (rv < 0) { log_error("lockspace_set_host_dead_ext %.48s host %llu %llu read %d", ls->name, (unsigned long long)hs->host_id, (unsigned long long)hs->generation, rv); goto out; } if (leader.version <= DELTA_DISK_VERSION_34) { log_error("lockspace_set_host_dead_ext %.48s host %llu %llu version incompat 0x%x", ls->name, (unsigned long long)hs->host_id, (unsigned long long)hs->generation, leader.version); rv = SANLK_VERSION_INCOMPAT; goto out; } if (hs->generation && (leader.owner_generation > hs->generation)) { log_error("lockspace_set_host_dead_ext %.48s host %llu %llu leader gen %llu", ls->name, (unsigned long long)hs->host_id, (unsigned long long)hs->generation, (unsigned long long)leader.owner_generation); rv = SANLK_WRONG_GENERATION; goto out; } if (is_caw && !(leader.flags & LFL_CAW)) { log_error("lockspace_set_host_dead_ext %.48s %llu is_caw no leader caw", ls->name, (unsigned long long)hs->host_id); rv = -EINVAL; goto out; } if (!is_caw && (leader.flags & LFL_CAW)) { log_error("lockspace_set_host_dead_ext %.48s %llu !is_caw with leader caw", ls->name, (unsigned long long)hs->host_id); rv = -EINVAL; goto out; } if (leader.flags & LFL_DEAD_EXT) { found_dead_ext = 1; goto save; } leader.flags |= LFL_DEAD_EXT; if (hs->generation) leader.owner_generation = hs->generation; /* for debugging only */ leader.unused1 = (local_host_id << 16) | (local_generation & 0x0000FFFF); if (is_caw) rv = delta_caw_lease_leader_clobber(task, io_timeout, &sd, hs->host_id, &leader, sector_size, caw_major, caw_minor, caw_offset, iobuf, "set_host_dead_ext"); else rv = delta_lease_leader_clobber(task, io_timeout, &sd, hs->host_id, &leader, "set_host_dead_ext"); if (rv < 0) { log_error("lockspace_set_host_dead_ext %.48s host %llu %llu write %d", ls->name, (unsigned long long)hs->host_id, (unsigned long long)hs->generation, rv); goto out; } wrote_dead_ext = 1; /* * Note: multiple hosts may all clobber this lease to set DEAD_EXT * at the same time, with the last one winning. So multiple hosts * may show wrote_dead_ext_gen. * * If the target host is restarting at the same time, this may * clobber the delta acquire with the next gen from the owner. * In that case the owner's next renewal will see its own delta * lease go backward to the prev gen with DEAD_EXT set. The * delta renewal can ignore that and go ahead with its next * renewal update. */ save: pthread_mutex_lock(&spaces_mutex); sp = _search_space(ls->name, NULL, 0, &spaces, NULL, NULL, NULL); if (!sp) { pthread_mutex_unlock(&spaces_mutex); log_error("lockspace_set_host_dead_ext %.48s host %llu %llu sp removed", ls->name, (unsigned long long)hs->host_id, (unsigned long long)hs->generation); rv = -ENOENT; goto out; } if (found_dead_ext) sp->host_status[hs->host_id-1].found_dead_ext_gen = leader.owner_generation; if (wrote_dead_ext) sp->host_status[hs->host_id-1].wrote_dead_ext_gen = leader.owner_generation; pthread_mutex_unlock(&spaces_mutex); rv = 0; out: if (iobuf) free(iobuf); close_disks(&sd, 1); return rv; } int lockspace_set_config(struct sanlk_lockspace *ls, GNUC_UNUSED uint32_t flags, uint32_t cmd, uint32_t data) { struct space *sp; int rv; pthread_mutex_lock(&spaces_mutex); sp = _search_space(ls->name, NULL, 0, &spaces, NULL, NULL, NULL); if (!sp) { pthread_mutex_unlock(&spaces_mutex); rv = -ENOENT; goto out; } pthread_mutex_unlock(&spaces_mutex); pthread_mutex_lock(&sp->mutex); switch (cmd) { case SANLK_CONFIG_USED: if (sp->space_dead) { rv = -ENOSPC; } else { sp->flags |= SP_EXTERNAL_USED; rv = 0; } break; case SANLK_CONFIG_UNUSED: sp->flags &= ~SP_EXTERNAL_USED; rv = 0; break; case SANLK_CONFIG_USED_BY_ORPHANS: if (sp->space_dead) { rv = -ENOSPC; } else { sp->flags |= SP_USED_BY_ORPHANS; rv = 0; } break; case SANLK_CONFIG_UNUSED_BY_ORPHANS: sp->flags &= ~SP_USED_BY_ORPHANS; rv = 0; break; case SANLK_CONFIG_IO_TIMEOUT: if (sp->space_dead) { rv = -ENOSPC; break; } if (!data) { log_space(sp, "set_config %.48s zero io timeout is invalid", ls->name); rv = -EINVAL; break; } if (sp->io_timeout == data) { log_space(sp, "set_config %.48s io_timeout is already %u", ls->name, sp->io_timeout); rv = 0; break; } log_space(sp, "set_config %.48s update io_timeout from %u to %u", ls->name, sp->io_timeout, data); sp->io_timeout = data; rv = 0; break; default: rv = -EINVAL; } pthread_mutex_unlock(&sp->mutex); out: return rv; } int lockspace_begin_rindex_op(char *space_name, int rindex_op, struct space_info *spi) { struct space *sp; int rv = 0; pthread_mutex_lock(&spaces_mutex); sp = _search_space(space_name, NULL, 0, &spaces, NULL, NULL, NULL); if (!sp) { rv = -ENOENT; goto out; } /* space_dead and thread_stop are only set while spaces_mutex is held, so we don't need to lock sp->mutex */ if (sp->space_dead || sp->thread_stop) { rv = -ENOSPC; goto out; } if (sp->rindex_op) { log_debug("being_rindex_op busy with %d", sp->rindex_op); rv = -EBUSY; goto out; } sp->rindex_op = rindex_op; _set_space_info(sp, spi); out: pthread_mutex_unlock(&spaces_mutex); return rv; } int lockspace_clear_rindex_op(char *space_name) { struct space *sp; int rv = 0; pthread_mutex_lock(&spaces_mutex); sp = _search_space(space_name, NULL, 0, &spaces, NULL, NULL, NULL); if (!sp) rv = -ENOENT; else sp->rindex_op = 0; pthread_mutex_unlock(&spaces_mutex); return rv; } static int _clean_event_fds(struct space *sp) { uint32_t end; int count = 0; int old_fd; int i, rv; pthread_mutex_lock(&sp->mutex); for (i = 0; i < MAX_EVENT_FDS; i++) { if (sp->event_fds[i] == -1) continue; old_fd = sp->event_fds[i]; rv = recv(old_fd, &end, sizeof(end), MSG_DONTWAIT); if (rv == -1 && errno == EAGAIN) continue; if ((rv == sizeof(end)) && (end != 1)) { log_erros(sp, "clean_event_fds ignore end value %u on event fd %d", end, old_fd); continue; } if ((rv == sizeof(end)) || !rv || (rv < 0)) log_space(sp, "clean_event_fds unregister event fd %d recv %d", old_fd, rv); else log_erros(sp, "clean_event_fds close event fd %d recv %d", old_fd, rv); close(old_fd); sp->event_fds[i] = -1; count++; } pthread_mutex_unlock(&sp->mutex); return count; } /* * end_event/reg_event don't need to worry about sp being freed * because the main daemon thread processes end_event/reg_event, * and the main thread is also the only thread that will free * sp structs. */ int lockspace_end_event(struct sanlk_lockspace *ls) { struct space *sp; if (!ls->name[0]) return -EINVAL; pthread_mutex_lock(&spaces_mutex); sp = _search_space(ls->name, NULL, 0, &spaces, NULL, NULL, NULL); pthread_mutex_unlock(&spaces_mutex); if (!sp) return -ENOENT; _clean_event_fds(sp); return 0; } int lockspace_reg_event(struct sanlk_lockspace *ls, int fd, GNUC_UNUSED uint32_t flags) { struct space *sp; int retried = 0; int cleaned = 0; int new_fd = -1; int i; if (!ls->name[0]) return -EINVAL; pthread_mutex_lock(&spaces_mutex); sp = _search_space(ls->name, NULL, 0, &spaces, NULL, NULL, NULL); if (!sp) { pthread_mutex_unlock(&spaces_mutex); log_error("lockspace_reg_event %.48s not found", ls->name); return -ENOENT; } pthread_mutex_unlock(&spaces_mutex); retry: pthread_mutex_lock(&sp->mutex); for (i = 0; i < MAX_EVENT_FDS; i++) { if (sp->event_fds[i] != -1) continue; /* _client_free closes the fd when the reg_event call is done, so we dup it here instead of adding a special case in _client free to keep it open. */ new_fd = dup(fd); sp->event_fds[i] = new_fd; break; } pthread_mutex_unlock(&sp->mutex); log_space(sp, "lockspace_reg_event new_fd %d from client fd %d", new_fd, fd); if (new_fd < 0) { if (retried) return -ENOCSI; cleaned = _clean_event_fds(sp); if (!cleaned) return -ENOCSI; retried = 1; goto retry; } return 0; } int lockspace_set_event(struct sanlk_lockspace *ls, struct sanlk_host_event *he, uint32_t flags) { struct space *sp; struct host_status *hs; uint64_t now; int i, rv = 0; if (!ls->name[0] || !he->host_id || he->host_id > DEFAULT_MAX_HOSTS) { log_error("set_event invalid args host_id %llu name %.48s", (unsigned long long)he->host_id, ls->name); return -EINVAL; } pthread_mutex_lock(&spaces_mutex); sp = _search_space(ls->name, NULL, 0, &spaces, NULL, NULL, NULL); if (!sp) { pthread_mutex_unlock(&spaces_mutex); return -ENOENT; } pthread_mutex_unlock(&spaces_mutex); if (he->host_id > sp->max_hosts) { log_error("set_event host_id %llu too large max %u %s", (unsigned long long)he->host_id, sp->max_hosts, ls->name); return -EINVAL; } if (!he->generation && (flags & SANLK_SETEV_CUR_GENERATION)) { hs = &(sp->host_status[he->host_id-1]); he->generation = hs->owner_generation; } now = monotime(); pthread_mutex_lock(&sp->mutex); if (flags & SANLK_SETEV_CLEAR_EVENT) { memset(&sp->host_event, 0, sizeof(struct sanlk_host_event)); sp->set_event_time = now; goto out; } if (flags & SANLK_SETEV_CLEAR_HOSTID) { sp->host_status[he->host_id-1].set_bit_time = 0; goto out; } if (flags & SANLK_SETEV_REPLACE_EVENT) goto set; /* log a warning if one non-zero event clobbers another non-zero event */ if ((now - sp->set_event_time < sp->set_bitmap_seconds) && sp->host_event.event && he->event && (sp->host_event.event != he->event)) { log_warns(sp, "event %llu %llu %llu %llu replaced by %llu %llu %llu %llu t %llu", (unsigned long long)sp->host_event.host_id, (unsigned long long)sp->host_event.generation, (unsigned long long)sp->host_event.event, (unsigned long long)sp->host_event.data, (unsigned long long)he->host_id, (unsigned long long)he->generation, (unsigned long long)he->event, (unsigned long long)he->data, (unsigned long long)sp->set_event_time); rv = -EBUSY; goto out; } set: sp->set_event_time = now; sp->host_status[he->host_id-1].set_bit_time = now; memcpy(&sp->host_event, he, sizeof(struct sanlk_host_event)); if (flags & SANLK_SETEV_ALL_HOSTS) { for (i = 0; i < sp->max_hosts; i++) sp->host_status[i].set_bit_time = now; } out: pthread_mutex_unlock(&sp->mutex); return rv; } int send_event_callbacks(uint32_t space_id, uint64_t from_host_id, uint64_t from_generation, struct sanlk_host_event *he) { struct space *sp; struct event_cb cb; int fd, i; int rv = 0; memset(&cb, 0, sizeof(cb)); cb.h.magic = SM_MAGIC; cb.h.version = SM_CB_PROTO; cb.h.cmd = SM_CB_GET_EVENT; cb.h.length = sizeof(cb); memcpy(&cb.he, he, sizeof(struct sanlk_host_event)); cb.from_host_id = from_host_id; cb.from_generation = from_generation; pthread_mutex_lock(&spaces_mutex); sp = find_lockspace_id(space_id); if (!sp) { pthread_mutex_unlock(&spaces_mutex); rv = -ENOSPC; goto ret; } pthread_mutex_unlock(&spaces_mutex); pthread_mutex_lock(&sp->mutex); for (i = 0; i < MAX_EVENT_FDS; i++) { if (sp->event_fds[i] == -1) continue; fd = sp->event_fds[i]; retry: rv = send(fd, &cb, sizeof(cb), MSG_NOSIGNAL | MSG_DONTWAIT); if (rv == -1 && errno == EINTR) goto retry; if (rv < 0) { log_erros(sp, "send_event_callbacks error %d %d close fd %d", rv, errno, fd); close(fd); sp->event_fds[i] = -1; } log_space(sp, "sent event to fd %d", fd); } pthread_mutex_unlock(&sp->mutex); ret: return rv; } /* * we call stop_host_id() when all pids are gone and we're in a safe state, so * it's safe to unlink the watchdog right away here. We want to sp the unlink * as soon as it's safe, so we can reduce the chance we get killed by the * watchdog (we could actually call this in main_loop just before the break). * Getting this unlink done quickly is more important than doing at the more * "logical" point commented above in host_id_thread. */ static int stop_lockspace_thread(struct space *sp, int wait) { int stop, rv; pthread_mutex_lock(&sp->mutex); stop = sp->thread_stop; sp->thread_stop = 1; pthread_mutex_unlock(&sp->mutex); if (!stop) { /* should never happen */ log_erros(sp, "stop_lockspace_thread zero thread_stop"); return -EINVAL; } if (wait) rv = pthread_join(sp->thread, NULL); else rv = pthread_tryjoin_np(sp->thread, NULL); return rv; } /* * locking/lifetime rules for a struct space * * multiple factors: * . spaces_mutex * . sp->mutex * . the specific thread: main daemon thread, lockspace thread, worker thread * * spaces, spaces_add, spaces_rem lists are protected by spaces_mutex * * sp->mutex protects info that is exchanged between the lockspace thread * (for the sp) and the main thread. This is primarily sp->thread_stop, * and sp->lease_status (although it seems a couple other bits of info * have been added over time that are communicated between the lockspace * thread and the main thread). * * add_lockspace_start(), called by worker thread, creates sp, * adds it to spaces_add list under spaces_mutex, creates lockspace_thread * for the sp. * * lockspace_thread never has to worry about sp going away and can access * sp directly any time. The sp will not be freed until lockspace_thread * has exited. * * The main thread never has to worry about sp going away, because the * main thread is the only context in which sp structs are freed * (and that only happens in free_lockspaces). * * add_lockspace_wait(), called by worker thread, can access sp directly * because sp won't go away while it's on spaces_add. Only add_lockspace_wait * can do something with sp while it's on spaces_add. _wait uses sp->mutex * to exchange lease status with lockspace_thread. Once the host_id lease * is acquired, _wait moves sp from spaces_add to spaces under spaces_mutex. * After sp is moved to spaces list, its lifetime is owned by the main thread. * * While sp is on spaces list, its lifetime is controlled by the main thread. * Apart from lockspace_thread, any other thread, e.g. worker thread, must * lock spaces_mutex, look up sp on spaces list, access sp fields, then unlock * spaces_mutex. After releasing spaces_mutex, it can't access sp struct * because the main thread could dispose of it. If the worker thread wants * to look at info that's being updated by the lockspace_thread, it should * also take sp->mutex before copying it. * * I currently see some violations of proper sp access that should be fixed. * The bad pattern in each case is: lock spaces_mutex, find sp, * unlock spaces_mutex, lock sp->mutex. The sp could in theory go away between * unlock spaces_mutex and lock sp->mutex. (In practice this would * likely never happen.) * * . worker_thread lockspace_set_event() * (reg_event and end_event are ok since they are called from * the main thread) * * . worker_thread host_status_set_bit() * * . resource_thread send_event_callbacks() does the same. * * I'm not sure what the best solution would be: lock sp->mutex before * unlocking spaces_mutex? Do everything under spaces_mutex? Add * a simple ref count to sp for these cases of using sp from other * threads? * * cmd_rem_lockspace() is run by a worker_thread. rem_lockspace_start() * locks spaces_mutex, finds sp, sets sp->external_remove, unlocks spaces_mutex. * Then the main thread, which owns the sp structs, sees sp->external_remove, * kills any pids using the sp, and when the sp is no longer used, it sets * sp->thread_stop, and moves sp from spaces list to spaces_rem list. * The main thread then runs free_lockspaces() which stops the lockspace_thread * for sp's on spaces_rem. When the lockspace_thread exits, the main thread * then removes sp from spaces_rem and frees sp. */ void free_lockspaces(int wait) { struct space *sp, *safe; int rv; pthread_mutex_lock(&spaces_mutex); list_for_each_entry_safe(sp, safe, &spaces_rem, list) { rv = stop_lockspace_thread(sp, wait); if (!rv) { log_space(sp, "free lockspace"); list_del(&sp->list); free_sp(sp); } } pthread_mutex_unlock(&spaces_mutex); } sanlock/src/lockspace.h000066400000000000000000000061271516326262600154330ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __LOCKSPACE_H__ #define __LOCKSPACE_H__ /* See resource.h for lock ordering between spaces_mutex and resource_mutex. */ /* no locks */ struct space *find_lockspace(const char *name); /* no locks */ int _lockspace_info(const char *space_name, struct space_info *spi); /* locks spaces_mutex */ int lockspace_info(const char *space_name, struct space_info *spi); /* locks spaces_mutex */ int lockspace_disk(char *space_name, struct sync_disk *disk, int *sector_size); /* locks spaces_mutex */ int host_info(char *space_name, uint64_t host_id, struct host_status *hs_out); /* locks spaces_mutex, locks sp */ int host_status_set_bit(char *space_name, uint64_t host_id); /* no locks */ int test_id_bit(int host_id, char *bitmap); /* locks sp */ int check_our_lease(struct space *sp, int *check_all, char *check_buf); /* locks resource_mutex (add_host_event), locks resource_mutex (set_resource_examine) */ void check_other_leases(struct space *sp, char *buf); /* locks spaces_mutex */ int add_lockspace_start(struct sanlk_lockspace *ls, uint32_t io_timeout, int nodelay, struct space **sp_out); /* locks sp, locks spaces_mutex */ int add_lockspace_wait(struct space *sp); /* locks spaces_mutex */ int inq_lockspace(struct sanlk_lockspace *ls); /* locks spaces_mutex */ int rem_lockspace_start(struct sanlk_lockspace *ls, unsigned int *space_id); /* locks spaces_mutex */ int rem_lockspace_wait(struct sanlk_lockspace *ls, unsigned int space_id); /* locks spaces_mutex, locks sp */ void free_lockspaces(int wait); /* locks spaces_mutex */ int get_lockspaces(char *buf, int *len, int *count, int maxlen); /* locks spaces_mutex */ int get_hosts(struct sanlk_lockspace *ls, char *buf, int *len, int *count, int maxlen); int get_host_local(struct sanlk_lockspace *ls, struct sanlk_host *host); /* locks spaces_mutex, locks sp */ int lockspace_set_event(struct sanlk_lockspace *ls, struct sanlk_host_event *he, uint32_t flags); /* locks spaces_mutex, locks sp */ int lockspace_reg_event(struct sanlk_lockspace *ls, int fd, uint32_t flags); /* locks spaces_mutex, locks sp */ int lockspace_end_event(struct sanlk_lockspace *ls); /* locks spaces_mutex, locks sp */ int send_event_callbacks(uint32_t space_id, uint64_t from_host_id, uint64_t from_generation, struct sanlk_host_event *he); /* locks spaces_mutex, locks sp */ int lockspace_set_config(struct sanlk_lockspace *ls, uint32_t flags, uint32_t cmd, uint32_t data); int lockspace_begin_rindex_op(char *space_name, int rindex_op, struct space_info *spi); int lockspace_clear_rindex_op(char *space_name); uint32_t get_host_flag(uint64_t local_host_id, struct host_status *hs); int lockspace_set_host_dead_ext(struct task *task, struct sanlk_lockspace *ls, struct sanlk_host *hs, uint32_t io_timeout); int open_space_caw(struct space *sp); int open_space(struct space *sp); #endif sanlock/src/log.c000066400000000000000000000165151516326262600142450ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock_internal.h" #include "log.h" #define LOG_STR_LEN 512 static char log_str[LOG_STR_LEN]; static pthread_t thread_handle; static pthread_mutex_t log_mutex = PTHREAD_MUTEX_INITIALIZER; static pthread_cond_t log_cond = PTHREAD_COND_INITIALIZER; static char log_dump[LOG_DUMP_SIZE]; static unsigned int log_point; static unsigned int log_wrap; struct entry { int level; char str[LOG_STR_LEN]; }; #define LOG_DEFAULT_ENTRIES 4096 static struct entry *log_ents; static unsigned int log_num_ents = LOG_DEFAULT_ENTRIES; static unsigned int log_head_ent; /* add at head */ static unsigned int log_tail_ent; /* remove from tail */ static unsigned int log_dropped; static unsigned int log_pending_ents; static unsigned int log_thread_done; static char logfile_path[PATH_MAX]; static FILE *logfile_fp; extern int log_logfile_priority; extern int log_logfile_use_utc; extern int log_syslog_priority; extern int log_stderr_priority; static void _log_save_dump(int level GNUC_UNUSED, int len) { int i; if (len < LOG_DUMP_SIZE - log_point) { memcpy(log_dump+log_point, log_str, len); log_point += len; if (log_point == LOG_DUMP_SIZE) { log_point = 0; log_wrap = 1; } return; } for (i = 0; i < len; i++) { log_dump[log_point++] = log_str[i]; if (log_point == LOG_DUMP_SIZE) { log_point = 0; log_wrap = 1; } } } static void _log_save_ent(int level, int len) { struct entry *e; if (!log_ents) return; if (log_pending_ents == log_num_ents) { log_dropped++; return; } e = &log_ents[log_head_ent++]; log_head_ent = log_head_ent % log_num_ents; log_pending_ents++; e->level = level; memcpy(e->str, log_str, len); } /* * This log function: * 1. formats the log message in the log_str buffer * 2. copies log_str into the log_dump circular buffer * 3. copies log_str into the log_ents circular array to be written to * logfile and/or syslog (so callers don't block writing messages to files) * * N.B. level as "int" instead of "uint32_t" is needed because * of comparison with int log_stderr_priority which can be -1. */ void log_level(uint32_t space_id, uint32_t res_id, char *name_in, int level, const char *fmt, ...) { va_list ap; char name[NAME_ID_SIZE + 1]; int ret, pos = 0; int len = LOG_STR_LEN - 2; /* leave room for \n\0 */ struct timeval cur_time; struct tm time_info; pid_t tid; if (is_helper) return; memset(name, 0, sizeof(name)); if (level == LOG_CLIENT) { int log_ci = 0, log_fd = 0; if (!com.debug_clients) return; level = LOG_DEBUG; log_ci = space_id; log_fd = res_id; if (!name_in) snprintf(name, NAME_ID_SIZE, "cl %d:%d ", log_ci, log_fd); else snprintf(name, NAME_ID_SIZE, "cl %d:%d %.8s ", log_ci, log_fd, name_in); } else if (level == LOG_CMD) { uint32_t cmd = space_id; if (!is_cmd_debug(cmd)) return; space_id = 0; level = LOG_DEBUG; } else { if (space_id && !res_id) snprintf(name, NAME_ID_SIZE, "s%u ", space_id); else if (!space_id && res_id) snprintf(name, NAME_ID_SIZE, "r%u ", res_id); else if (space_id && res_id) snprintf(name, NAME_ID_SIZE, "s%u:r%u ", space_id, res_id); else if (name_in) snprintf(name, NAME_ID_SIZE, "%.8s ", name_in); } pthread_mutex_lock(&log_mutex); gettimeofday(&cur_time, NULL); if (log_logfile_use_utc) gmtime_r(&cur_time.tv_sec, &time_info); else localtime_r(&cur_time.tv_sec, &time_info); ret = strftime(log_str + pos, len - pos, "%Y-%m-%d %H:%M:%S ", &time_info); pos += ret; tid = syscall(SYS_gettid); ret = snprintf(log_str + pos, len - pos, "%llu [%u]: %s", (unsigned long long) monotime(), tid, name); pos += ret; va_start(ap, fmt); ret = vsnprintf(log_str + pos, len - pos, fmt, ap); va_end(ap); if (ret >= len - pos) pos = len - 1; else pos += ret; log_str[pos++] = '\n'; log_str[pos++] = '\0'; /* * save all messages in circular buffer "log_dump" that can be * sent over unix socket */ _log_save_dump(level, pos - 1); /* * save some messages in circular array "log_ents" that a thread * writes to logfile/syslog */ if (level <= log_logfile_priority || level <= log_syslog_priority) _log_save_ent(level, pos); if (level <= log_stderr_priority) fprintf(stderr, "%s", log_str); pthread_cond_signal(&log_cond); pthread_mutex_unlock(&log_mutex); } static void write_entry(int level, char *str) { if ((level <= log_logfile_priority) && logfile_fp) { fprintf(logfile_fp, "%s", str); fflush(logfile_fp); } if (level <= log_syslog_priority) syslog(level, "%s", str); } static void write_dropped(int level, int num) { char str[LOG_STR_LEN]; sprintf(str, "dropped %d entries", num); write_entry(level, str); } void copy_log_dump(char *buf, int *len) { int tail_len; pthread_mutex_lock(&log_mutex); if (!log_wrap && !log_point) { *len = 0; } else if (log_wrap) { tail_len = LOG_DUMP_SIZE - log_point; memcpy(buf, log_dump+log_point, tail_len); if (log_point) memcpy(buf+tail_len, log_dump, log_point); *len = LOG_DUMP_SIZE; } else { memcpy(buf, log_dump, log_point-1); *len = log_point-1; } pthread_mutex_unlock(&log_mutex); } static void *log_thread_fn(void *arg GNUC_UNUSED) { char str[LOG_STR_LEN]; struct entry *e; int level, prev_dropped = 0; while (1) { pthread_mutex_lock(&log_mutex); while (log_head_ent == log_tail_ent) { if (log_thread_done) { pthread_mutex_unlock(&log_mutex); goto out; } pthread_cond_wait(&log_cond, &log_mutex); } e = &log_ents[log_tail_ent++]; log_tail_ent = log_tail_ent % log_num_ents; log_pending_ents--; memcpy(str, e->str, LOG_STR_LEN); level = e->level; prev_dropped = log_dropped; log_dropped = 0; pthread_mutex_unlock(&log_mutex); if (prev_dropped) { write_dropped(level, prev_dropped); prev_dropped = 0; } write_entry(level, str); } out: pthread_exit(NULL); } int setup_logging(void) { int fd, rv; snprintf(logfile_path, PATH_MAX, "%s/%s", SANLK_LOG_DIR, SANLK_LOGFILE_NAME); logfile_fp = fopen(logfile_path, "a+"); if (logfile_fp) { fd = fileno(logfile_fp); fcntl(fd, F_SETFD, fcntl(fd, F_GETFD, 0) | FD_CLOEXEC); } log_ents = malloc(log_num_ents * sizeof(struct entry)); if (!log_ents) { fclose(logfile_fp); logfile_fp = NULL; return -1; } memset(log_ents, 0, log_num_ents * sizeof(struct entry)); openlog(DAEMON_NAME, LOG_CONS | LOG_PID, LOG_DAEMON); rv = pthread_create(&thread_handle, NULL, log_thread_fn, NULL); if (rv) return -1; return 0; } void close_logging(void) { pthread_mutex_lock(&log_mutex); log_thread_done = 1; pthread_cond_signal(&log_cond); pthread_mutex_unlock(&log_mutex); pthread_join(thread_handle, NULL); pthread_mutex_lock(&log_mutex); closelog(); if (logfile_fp) { fclose(logfile_fp); logfile_fp = NULL; } pthread_mutex_unlock(&log_mutex); } sanlock/src/log.h000066400000000000000000000056421516326262600142510ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __LOG_H__ #define __LOG_H__ #define LOG_CLIENT LOG_LOCAL0 #define LOG_CMD LOG_LOCAL1 /* * Log levels are used mainly to indicate where the message * should be recorded: * * log_error() write to /var/log/messages and /var/log/sanlock.log * log_level(WARNING) write to /var/log/sanlock.log * log_debug() write to incore buffer, not to file * * Anything in /var/log/messages should not happen and should be reported. * So anything we want to visible and reported should be LOG_ERR. * * If we want to log something to assist in debugging, but not be reported, * it should be LOG_WARNING (goes only to sanlock.log) */ void log_level(uint32_t space_id, uint32_t res_id, char *name_in, int level, const char *fmt, ...) __attribute__((format(printf, 5, 6))); int setup_logging(void); void close_logging(void); void copy_log_dump(char *buf, int *len); #define log_debug(fmt, args...) log_level(0, 0, NULL, LOG_DEBUG, fmt, ##args) #define log_space(space, fmt, args...) log_level(space->space_id, 0, NULL, LOG_DEBUG, fmt, ##args) #define log_token(token, fmt, args...) log_level(token->space_id, token->res_id, NULL, LOG_DEBUG, fmt, ##args) #define log_sid(space_id, fmt, args...) log_level(space_id, 0, NULL, LOG_DEBUG, fmt, ##args) #define log_rid(res_id, fmt, args...) log_level(res_id, 0, NULL, LOG_DEBUG, fmt, ##args) #define log_warn(fmt, args...) log_level(0, 0, NULL, LOG_WARNING, fmt, ##args) #define log_warns(space, fmt, args...) log_level(space->space_id, 0, NULL, LOG_WARNING, fmt, ##args) #define log_warnt(token, fmt, args...) log_level(token->space_id, token->res_id, NULL, LOG_WARNING, fmt, ##args) #define log_error(fmt, args...) log_level(0, 0, NULL, LOG_ERR, fmt, ##args) #define log_erros(space, fmt, args...) log_level(space->space_id, 0, NULL, LOG_ERR, fmt, ##args) #define log_errot(token, fmt, args...) log_level(token->space_id, token->res_id, NULL, LOG_ERR, fmt, ##args) #define log_taske(task, fmt, args...) log_level(0, 0, task->name, LOG_ERR, fmt, ##args) #define log_taskw(task, fmt, args...) log_level(0, 0, task->name, LOG_WARNING, fmt, ##args) #define log_taskd(task, fmt, args...) log_level(0, 0, task->name, LOG_DEBUG, fmt, ##args) #define log_client(ci, fd, fmt, args...) log_level(ci, fd, NULL, LOG_CLIENT, fmt, ##args) #define log_cmd(cmd, fmt, args...) log_level(cmd, 0, NULL, LOG_CMD, fmt, ##args) /* use log_tool for tool actions (non-daemon), and for daemon until logging is set up */ #define log_tool(fmt, args...) \ do { \ printf(fmt "\n", ##args); \ } while (0) #endif sanlock/src/logrotate.sanlock000066400000000000000000000002571516326262600166700ustar00rootroot00000000000000/var/log/sanlock.log { rotate 3 missingok copytruncate size 10M compress compresscmd /usr/bin/xz uncompresscmd /usr/bin/unxz compressext .xz } sanlock/src/main.c000066400000000000000000003621621516326262600144120ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #define EXTERN #include "sanlock_internal.h" #include "sanlock_sock.h" #include "sanlock_resource.h" #include "sanlock_admin.h" #include "diskio.h" #include "log.h" #include "lockspace.h" #include "resource.h" #include "direct.h" #include "lockfile.h" #include "watchdog.h" #include "task.h" #include "client_cmd.h" #include "cmd.h" #include "helper.h" #include "timeouts.h" #include "paxos_lease.h" #include "env.h" #define SIGRUNPATH 100 /* anything that's not SIGTERM/SIGKILL */ struct thread_pool { int num_workers; int max_workers; int free_workers; int quit; struct list_head work_data; pthread_mutex_t mutex; pthread_cond_t cond; pthread_cond_t quit_wait; struct worker_info *info; }; /* priorities are LOG_* from syslog.h */ int log_logfile_priority = LOG_WARNING; int log_logfile_use_utc = 0; int log_syslog_priority = LOG_ERR; int log_stderr_priority = -1; /* -D sets this to LOG_DEBUG */ #define CLIENT_NALLOC 1024 static int client_maxi; static int client_size = 0; static struct pollfd *pollfd; static char command[COMMAND_MAX]; static int cmd_argc; static char **cmd_argv; static struct thread_pool pool; static char rand_state[32]; static pthread_mutex_t rand_mutex = PTHREAD_MUTEX_INITIALIZER; static const char *run_dir = NULL; static int privileged = 1; static void close_helper(void) { close(helper_kill_fd); close(helper_status_fd); helper_kill_fd = -1; helper_status_fd = -1; pollfd[helper_ci].fd = -1; pollfd[helper_ci].events = 0; helper_ci = -1; /* don't set helper_pid = -1 until we've tried waitpid */ } /* * We cannot block the main thread on this write, so the pipe is * NONBLOCK, and write fails with EAGAIN when the pipe is full. * With 512 msg size and 64k default pipe size, the pipe will be full * if we quickly send kill messages for 128 pids. We retry * the kill once a second, so we'll retry the write again in * a second. * * By setting the pipe size to 1MB in setup_helper, we could quickly send 2048 * msgs before getting EAGAIN. */ static void send_helper_kill(struct space *sp, struct client *cl, int sig) { struct helper_msg hm; int rv; /* * We come through here once a second while the pid still has * leases. We only send a single RUNPATH message, so after * the first RUNPATH goes through we set CL_RUNPATH_SENT to * avoid further RUNPATH's. */ if ((cl->flags & CL_RUNPATH_SENT) && (sig == SIGRUNPATH)) return; if (helper_kill_fd == -1) { log_error("send_helper_kill pid %d no fd", cl->pid); return; } memset(&hm, 0, sizeof(hm)); if (sig == SIGRUNPATH) { hm.type = HELPER_MSG_RUNPATH; memcpy(hm.path, cl->killpath, SANLK_HELPER_PATH_LEN-1); memcpy(hm.args, cl->killargs, SANLK_HELPER_ARGS_LEN-1); /* only include pid if it's requested as a killpath arg */ if (cl->flags & CL_KILLPATH_PID) hm.pid = cl->pid; } else { hm.type = HELPER_MSG_KILLPID; hm.sig = sig; hm.pid = cl->pid; } log_erros(sp, "kill %d sig %d count %d", cl->pid, sig, cl->kill_count); retry: rv = write(helper_kill_fd, &hm, sizeof(hm)); if (rv == -1 && errno == EINTR) goto retry; /* pipe is full, we'll try again in a second */ if (rv == -1 && errno == EAGAIN) { helper_full_count++; log_space(sp, "send_helper_kill pid %d sig %d full_count %u", cl->pid, sig, helper_full_count); return; } /* helper exited or closed fd, quit using helper */ if (rv == -1 && errno == EPIPE) { log_erros(sp, "send_helper_kill EPIPE"); close_helper(); return; } if (rv != sizeof(hm)) { /* this shouldn't happen */ log_erros(sp, "send_helper_kill pid %d error %d %d", cl->pid, rv, errno); close_helper(); return; } if (sig == SIGRUNPATH) cl->flags |= CL_RUNPATH_SENT; } /* FIXME: add a mutex for client array so we don't try to expand it while a cmd thread is using it. Or, with a thread pool we know when cmd threads are running and can expand when none are. */ static int client_alloc(void) { int i; /* pollfd is one element longer as we use an additional element for the * eventfd notification mechanism */ client = malloc(CLIENT_NALLOC * sizeof(struct client)); pollfd = malloc((CLIENT_NALLOC+1) * sizeof(struct pollfd)); if (!client || !pollfd) { log_error("can't alloc for client or pollfd array"); return -ENOMEM; } for (i = 0; i < CLIENT_NALLOC; i++) { memset(&client[i], 0, sizeof(struct client)); memset(&pollfd[i], 0, sizeof(struct pollfd)); pthread_mutex_init(&client[i].mutex, NULL); client[i].fd = -1; client[i].pid = -1; pollfd[i].fd = -1; pollfd[i].events = 0; } client_size = CLIENT_NALLOC; return 0; } static void _client_free(int ci) { struct client *cl = &client[ci]; if (cl->cmd_active || cl->pid_dead) log_client(ci, cl->fd, "free cmd %d dead %d", cl->cmd_active, cl->pid_dead); else log_client(ci, cl->fd, "free"); if (!cl->used) { /* should never happen */ log_error("client_free ci %d not used", ci); goto out; } if (cl->pid != -1) { /* client_pid_dead() should have set pid to -1 */ /* should never happen */ log_error("client_free ci %d live pid %d", ci, cl->pid); goto out; } if (cl->fd == -1) { /* should never happen */ log_error("client_free ci %d is free", ci); goto out; } if (cl->need_free) log_debug("client_free ci %d already need_free", ci); if (cl->suspend) { log_debug("client_free ci %d is suspended", ci); cl->need_free = 1; goto out; } if (cl->fd != -1) close(cl->fd); cl->used = 0; cl->fd = -1; cl->pid = -1; cl->cmd_active = 0; cl->pid_dead = 0; cl->suspend = 0; cl->need_free = 0; cl->kill_count = 0; cl->kill_last = 0; cl->restricted = 0; cl->flags = 0; memset(cl->owner_name, 0, sizeof(cl->owner_name)); memset(cl->killpath, 0, SANLK_HELPER_PATH_LEN); memset(cl->killargs, 0, SANLK_HELPER_ARGS_LEN); cl->workfn = NULL; cl->deadfn = NULL; if (cl->tokens) free(cl->tokens); cl->tokens = NULL; cl->tokens_slots = 0; /* make poll() ignore this connection */ pollfd[ci].fd = -1; pollfd[ci].events = 0; pollfd[ci].revents = 0; out: return; } void client_free(int ci); void client_free(int ci) { struct client *cl = &client[ci]; pthread_mutex_lock(&cl->mutex); _client_free(ci); pthread_mutex_unlock(&cl->mutex); } /* the connection that we suspend and resume may or may not be the same connection as the target client where we set cmd_active */ static int client_suspend(int ci) { struct client *cl = &client[ci]; int rv = 0; pthread_mutex_lock(&cl->mutex); if (!cl->used) { /* should never happen */ log_error("client_suspend ci %d not used", ci); rv = -1; goto out; } if (cl->fd == -1) { /* should never happen */ log_error("client_suspend ci %d is free", ci); rv = -1; goto out; } if (cl->suspend) { /* should never happen */ log_error("client_suspend ci %d is suspended", ci); rv = -1; goto out; } log_client(ci, cl->fd, "suspend"); cl->suspend = 1; /* make poll() ignore this connection */ pollfd[ci].fd = -1; pollfd[ci].events = 0; out: pthread_mutex_unlock(&cl->mutex); return rv; } void client_resume(int ci); void client_resume(int ci) { struct client *cl = &client[ci]; pthread_mutex_lock(&cl->mutex); if (!cl->used) { /* should never happen */ log_error("client_resume ci %d not used", ci); goto out; } if (cl->fd == -1) { /* should never happen */ log_error("client_resume ci %d is free", ci); goto out; } if (!cl->suspend) { /* should never happen */ log_error("client_resume ci %d not suspended", ci); goto out; } log_client(ci, cl->fd, "resume"); cl->suspend = 0; if (cl->need_free) { log_debug("client_resume ci %d need_free", ci); _client_free(ci); } else { /* make poll() watch this connection */ pollfd[ci].fd = cl->fd; pollfd[ci].events = POLLIN; /* interrupt any poll() that might already be running */ eventfd_write(efd, 1); } out: pthread_mutex_unlock(&cl->mutex); } static int client_add(int fd, void (*workfn)(int ci), void (*deadfn)(int ci)) { struct client *cl; int i; for (i = 0; i < client_size; i++) { cl = &client[i]; pthread_mutex_lock(&cl->mutex); if (!cl->used) { cl->used = 1; cl->fd = fd; cl->workfn = workfn; cl->deadfn = deadfn ? deadfn : client_free; /* make poll() watch this connection */ pollfd[i].fd = fd; pollfd[i].events = POLLIN; if (i > client_maxi) client_maxi = i; pthread_mutex_unlock(&cl->mutex); log_client(i, fd, "add"); return i; } pthread_mutex_unlock(&cl->mutex); } return -1; } /* clear the unreceived portion of an aborted command */ void client_recv_all(int ci, struct sm_header *h_recv, int pos); void client_recv_all(int ci, struct sm_header *h_recv, int pos) { char trash[64]; int rem = h_recv->length - sizeof(struct sm_header) - pos; int rv, error = 0, total = 0, retries = 0; if (!rem) return; while (1) { rv = recv(client[ci].fd, trash, sizeof(trash), MSG_DONTWAIT); if (rv == -1 && errno == EAGAIN) { usleep(1000); if (retries < 20) { retries++; continue; } } if (rv == -1) error = errno; if (rv <= 0) break; total += rv; if (total >= rem) break; } log_debug("client recv_all %d,%d,%d pos %d rv %d error %d retries %d rem %d total %d", ci, client[ci].fd, client[ci].pid, pos, rv, error, retries, rem, total); } void send_result(int ci, int fd, struct sm_header *h_recv, int result); void send_result(int ci, int fd, struct sm_header *h_recv, int result) { struct sm_header h; size_t rem = sizeof(h); size_t off = 0; ssize_t rv; log_client(ci, fd, "send %d", result); memcpy(&h, h_recv, sizeof(struct sm_header)); h.version = SM_PROTO; h.length = sizeof(h); h.data = result; h.data2 = 0; retry: rv = send(fd, (char *)&h + off, rem, MSG_NOSIGNAL); if (rv == -1 && errno == EINTR) goto retry; if (rv < 0) return; if (rv < rem) { rem -= rv; off += rv; goto retry; } } void client_pid_dead(int ci); void client_pid_dead(int ci) { struct client *cl = &client[ci]; int cmd_active; int i, pid; /* cmd_acquire_thread may still be waiting for the tokens to be acquired. if it is, cl->pid_dead tells it to release them when finished. Similarly, cmd_release_thread, cmd_inquire_thread are accessing cl->tokens */ pthread_mutex_lock(&cl->mutex); if (!cl->used || cl->fd == -1 || cl->pid == -1) { /* should never happen */ pthread_mutex_unlock(&cl->mutex); log_error("client_pid_dead %d,%d,%d u %d a %d s %d bad state", ci, cl->fd, cl->pid, cl->used, cl->cmd_active, cl->suspend); return; } log_debug("client_pid_dead %d,%d,%d cmd_active %d suspend %d", ci, cl->fd, cl->pid, cl->cmd_active, cl->suspend); if (cl->kill_count) log_error("dead %d ci %d count %d", cl->pid, ci, cl->kill_count); cmd_active = cl->cmd_active; pid = cl->pid; cl->pid = -1; cl->pid_dead = 1; /* when cmd_active is set and cmd_a,r,i_thread is done and takes cl->mutex to set cl->cmd_active to 0, it will see cl->pid_dead is 1 and know they need to release cl->tokens and call client_free */ /* make poll() ignore this connection */ pollfd[ci].fd = -1; pollfd[ci].events = 0; pthread_mutex_unlock(&cl->mutex); /* it would be nice to do this SIGKILL as a confirmation that the pid is really gone (i.e. didn't just close the fd) if we always had root permission to do it */ /* kill(pid, SIGKILL); */ if (cmd_active) { log_debug("client_pid_dead %d,%d,%d defer to cmd %d", ci, cl->fd, pid, cmd_active); return; } /* use async release here because this is the main thread that we don't want to block doing disk lease i/o */ pthread_mutex_lock(&cl->mutex); for (i = 0; i < cl->tokens_slots; i++) { if (cl->tokens[i]) { release_token_async(cl->tokens[i]); free(cl->tokens[i]); } } _client_free(ci); pthread_mutex_unlock(&cl->mutex); } /* At some point we may want to keep a record of each pid using a lockspace in the sp struct to avoid walking through each client's cl->tokens to see if it's using the lockspace. It should be the uncommon situation where a lockspace renewal fails and we need to walk through all client tokens like this. i.e. we'd probably not want to optimize for this case at the expense of the more common case where a pid exits, but we do want it to be robust. The locking is also made a bit ugly by these three routines that need to correlate which clients are using which lockspaces. (client_using_space, kill_pids, all_pids_dead) spaces_mutex is held when they are called, and they need to take cl->mutex. This means that cmd_acquire_thread has to lock both spaces_mutex and cl->mutex when adding new tokens to the client. (It needs to check that the lockspace for the new tokens hasn't failed while the tokens were being acquired.) In kill_pids and all_pids_dead could we check cl->pid <= 0 without taking cl->mutex, since client_pid_dead in the main thread is the only place that changes that? */ static int client_using_space(struct client *cl, struct space *sp) { struct token *token; int i, rv = 0; for (i = 0; i < cl->tokens_slots; i++) { token = cl->tokens[i]; if (!token) continue; if (strncmp(token->r.lockspace_name, sp->space_name, NAME_ID_SIZE)) continue; if (!cl->kill_count) log_token(token, "client_using_space pid %d", cl->pid); if (sp->space_dead) token->space_dead = sp->space_dead; rv = 1; } return rv; } static void kill_pids(struct space *sp) { struct client *cl; uint64_t now, last_success; int id_renewal_fail_seconds; int ci, sig; int do_kill, in_grace; /* * all remaining pids using sp are stuck, we've made max attempts to * kill all, don't bother cycling through them */ if (sp->killing_pids > 1) return; id_renewal_fail_seconds = calc_id_renewal_fail_seconds(sp->io_timeout); /* * If we happen to renew our lease after we've started killing pids, * the period we allow for graceful shutdown will be extended. This * is an incidental effect, although it may be nice. The previous * behavior would still be ok, where we only ever allow up to * kill_grace_seconds for graceful shutdown before moving to sigkill. */ pthread_mutex_lock(&sp->mutex); last_success = sp->lease_status.renewal_last_success; pthread_mutex_unlock(&sp->mutex); now = monotime(); for (ci = 0; ci <= client_maxi; ci++) { do_kill = 0; cl = &client[ci]; pthread_mutex_lock(&cl->mutex); if (!cl->used) goto unlock; if (cl->pid <= 0) goto unlock; /* NB this cl may not be using sp, but trying to avoid the expensive client_using_space check */ if (cl->kill_count >= kill_count_max) goto unlock; if (cl->kill_count && (now - cl->kill_last < 1)) goto unlock; if (!client_using_space(cl, sp)) goto unlock; cl->kill_last = now; cl->kill_count++; /* * the transition from using killpath/sigterm to sigkill * is when now >= * last successful lease renewal + * id_renewal_fail_seconds + * kill_grace_seconds */ in_grace = now < (last_success + id_renewal_fail_seconds + com.kill_grace_seconds); if (sp->external_remove || (external_shutdown > 1)) { sig = SIGKILL; } else if ((com.kill_grace_seconds > 0) && in_grace && cl->killpath[0]) { sig = SIGRUNPATH; } else if (in_grace) { sig = SIGTERM; } else { sig = SIGKILL; } /* * sigterm will be used in place of sigkill if restricted * sigkill will be used in place of sigterm if restricted */ if ((sig == SIGKILL) && (cl->restricted & SANLK_RESTRICT_SIGKILL)) sig = SIGTERM; if ((sig == SIGTERM) && (cl->restricted & SANLK_RESTRICT_SIGTERM)) sig = SIGKILL; do_kill = 1; unlock: pthread_mutex_unlock(&cl->mutex); if (!do_kill) continue; send_helper_kill(sp, cl, sig); } } static int all_pids_dead(struct space *sp) { struct client *cl; int stuck = 0, check = 0; int ci; for (ci = 0; ci <= client_maxi; ci++) { cl = &client[ci]; pthread_mutex_lock(&cl->mutex); if (!cl->used) goto unlock; if (cl->pid <= 0) goto unlock; if (!client_using_space(cl, sp)) goto unlock; if (cl->kill_count >= kill_count_max) stuck++; else check++; unlock: pthread_mutex_unlock(&cl->mutex); } if (stuck && !check && sp->killing_pids < 2) { log_erros(sp, "killing pids stuck %d", stuck); /* cause kill_pids to give up */ sp->killing_pids = 2; } if (stuck || check) return 0; if (sp->flags & SP_EXTERNAL_USED) { if (!sp->used_retries || !(sp->used_retries % 1000)) log_erros(sp, "used external blocking lockspace removal"); sp->used_retries++; return 0; } if (sp->flags & SP_USED_BY_ORPHANS) { /* * lock ordering: spaces_mutex (main_loop), then * resource_mutex (resource_orphan_count) */ int orphans = resource_orphan_count(sp->space_name); if (orphans) { if (!sp->used_retries || !(sp->used_retries % 1000)) log_erros(sp, "used by orphan %d blocking lockspace removal", orphans); sp->used_retries++; return 0; } } if (sp->renew_fail || sp->used_retries) log_erros(sp, "all pids clear"); else log_space(sp, "all pids clear"); return 1; } static unsigned int time_diff(struct timeval *begin, struct timeval *end) { struct timeval result; timersub(end, begin, &result); return (result.tv_sec * 1000) + (result.tv_usec / 1000); } #define STANDARD_CHECK_INTERVAL 1000 /* milliseconds */ #define RECOVERY_CHECK_INTERVAL 200 /* milliseconds */ static int main_loop(void) { void (*workfn) (int ci); void (*deadfn) (int ci); struct space *sp, *safe; struct timeval now, last_check; int poll_timeout, check_interval; unsigned int ms; int i, rv, empty, check_all; char *check_buf = NULL; int check_buf_len = 0; uint64_t ebuf; gettimeofday(&last_check, NULL); poll_timeout = STANDARD_CHECK_INTERVAL; check_interval = STANDARD_CHECK_INTERVAL; while (1) { /* as well as the clients, check the eventfd */ pollfd[client_maxi+1].fd = efd; pollfd[client_maxi+1].events = POLLIN; rv = poll(pollfd, client_maxi + 2, poll_timeout); if (rv == -1 && errno == EINTR) continue; if (rv < 0) { /* not sure */ log_client(0, 0, "poll err %d", rv); } for (i = 0; i <= client_maxi + 1; i++) { /* * This index for efd has no client array entry. Its * only purpose is to wake up this poll loop in which * case we just clear any data and continue looking * for other client entries that need processing. */ if (pollfd[i].fd == efd) { if (pollfd[i].revents & POLLIN) { log_client(i, efd, "efd wake"); /* N.B. i is not a ci */ eventfd_read(efd, &ebuf); } continue; } /* * FIXME? client_maxi is never reduced so over time we * end up checking and skipping some number of unused * client entries here which seems inefficient. */ if (client[i].fd < 0) continue; if (pollfd[i].revents & POLLIN) { workfn = client[i].workfn; if (workfn) workfn(i); } if (pollfd[i].revents & (POLLERR | POLLHUP | POLLNVAL)) { log_client(i, client[i].fd, "poll dead"); deadfn = client[i].deadfn; if (deadfn) deadfn(i); } } gettimeofday(&now, NULL); ms = time_diff(&last_check, &now); if (ms < check_interval) { poll_timeout = check_interval - ms; continue; } last_check = now; check_interval = STANDARD_CHECK_INTERVAL; /* * check the condition of each lockspace, * if pids are being killed, have pids all exited? * is its host_id being renewed?, if not kill pids */ pthread_mutex_lock(&spaces_mutex); list_for_each_entry_safe(sp, safe, &spaces, list) { if (sp->killing_pids && all_pids_dead(sp)) { /* * move sp to spaces_rem so main_loop * will no longer see it. */ log_space(sp, "set thread_stop"); pthread_mutex_lock(&sp->mutex); sp->thread_stop = 1; deactivate_watchdog(sp); pthread_mutex_unlock(&sp->mutex); list_move(&sp->list, &spaces_rem); continue; } if (sp->killing_pids) { /* * continue to kill the pids with increasing * levels of severity until they all exit */ kill_pids(sp); check_interval = RECOVERY_CHECK_INTERVAL; continue; } /* * check host_id lease renewal */ if (sp->align_size > check_buf_len) { if (check_buf) free(check_buf); check_buf_len = sp->align_size; check_buf = malloc(check_buf_len); } if (check_buf) memset(check_buf, 0, check_buf_len); /* * check_our_lease will return check_all=1 if * all ls leases have been successfully read * since it was last called. They are copied * into check_buf for check_other_leases. * So, check_other_leases will be called once * after each successful read of other leases. */ check_all = 0; rv = check_our_lease(sp, &check_all, check_buf); if (rv) sp->renew_fail = 1; if (rv || sp->external_remove || (external_shutdown > 1)) { log_space(sp, "set killing_pids check %d remove %d", rv, sp->external_remove); sp->space_dead = 1; sp->killing_pids = 1; kill_pids(sp); check_interval = RECOVERY_CHECK_INTERVAL; } else if (check_all) { check_other_leases(sp, check_buf); } } empty = list_empty(&spaces); pthread_mutex_unlock(&spaces_mutex); if (external_shutdown && empty) break; if (external_shutdown == 1) { log_debug("ignore shutdown, lockspace exists"); external_shutdown = 0; } free_lockspaces(0); rem_resources(); gettimeofday(&now, NULL); ms = time_diff(&last_check, &now); if (ms < check_interval) poll_timeout = check_interval - ms; else poll_timeout = 1; } free_lockspaces(1); daemon_shutdown_reply(); return 0; } int print_state_worker(int i, char *str); int print_state_worker(int i, char *str) { if (!pool.info[i].work_count) return 0; memset(str, 0, SANLK_STATE_MAXSTR); snprintf(str, SANLK_STATE_MAXSTR-1, "cmd_last=%s " "busy=%u " "thread_id=%u " "work_count=%u " "io_count=%u " "to_count=%u " "miscompares=%u " "paxos_areabuf_created=%u " "paxos_areabuf_used=%u", cmd_num_to_str(pool.info[i].cmd_last), pool.info[i].busy, pool.info[i].thread_id, pool.info[i].work_count, pool.info[i].io_count, pool.info[i].to_count, pool.info[i].miscompares, pool.info[i].paxos_areabuf_created, pool.info[i].paxos_areabuf_used); return strlen(str) + 1; } static void *thread_pool_worker(void *data) { struct task task; struct cmd_args *ca; long workerid = (long)data; memset(&task, 0, sizeof(struct task)); setup_task_aio(&task, main_task.use_aio, WORKER_AIO_CB_SIZE); snprintf(task.name, NAME_ID_SIZE, "worker%ld", workerid); pool.info[workerid].thread_id = gettid(); pthread_mutex_lock(&pool.mutex); while (1) { while (!pool.quit && list_empty(&pool.work_data)) { pool.free_workers++; pthread_cond_wait(&pool.cond, &pool.mutex); pool.free_workers--; } while (!list_empty(&pool.work_data)) { ca = list_first_entry(&pool.work_data, struct cmd_args, list); list_del(&ca->list); pool.info[workerid].busy = 1; pool.info[workerid].cmd_last = ca->header.cmd; pthread_mutex_unlock(&pool.mutex); call_cmd_thread(&task, ca); free(ca); pthread_mutex_lock(&pool.mutex); pool.info[workerid].busy = 0; pool.info[workerid].work_count++; pool.info[workerid].io_count = task.io_count; pool.info[workerid].to_count = task.to_count; pool.info[workerid].miscompares = task.miscompares; pool.info[workerid].paxos_areabuf_created = task.paxos_areabuf_created; pool.info[workerid].paxos_areabuf_used = task.paxos_areabuf_used; } if (pool.quit) break; } pool.num_workers--; daemon_status_num_workers--; /* for status reporting */ if (!pool.num_workers) pthread_cond_signal(&pool.quit_wait); pthread_mutex_unlock(&pool.mutex); close_task_aio(&task); return NULL; } static int thread_pool_add_work(struct cmd_args *ca) { pthread_t th; int rv; pthread_mutex_lock(&pool.mutex); if (pool.quit) { pthread_mutex_unlock(&pool.mutex); return -1; } list_add_tail(&ca->list, &pool.work_data); if (!pool.free_workers && pool.num_workers < pool.max_workers) { rv = pthread_create(&th, NULL, thread_pool_worker, (void *)(long)pool.num_workers); if (rv) { log_error("thread_pool_add_work ci %d error %d", ca->ci_in, rv); list_del(&ca->list); pthread_mutex_unlock(&pool.mutex); rv = -1; return rv; } pool.num_workers++; daemon_status_num_workers++; /* for status reporting */ } pthread_cond_signal(&pool.cond); pthread_mutex_unlock(&pool.mutex); return 0; } static void thread_pool_free(void) { pthread_mutex_lock(&pool.mutex); pool.quit = 1; if (pool.num_workers > 0) { pthread_cond_broadcast(&pool.cond); pthread_cond_wait(&pool.quit_wait, &pool.mutex); } pthread_mutex_unlock(&pool.mutex); } static int thread_pool_create(int min_workers, int max_workers) { pthread_t th; int i, rv = 0; memset(&pool, 0, sizeof(pool)); INIT_LIST_HEAD(&pool.work_data); pthread_mutex_init(&pool.mutex, NULL); pthread_cond_init(&pool.cond, NULL); pthread_cond_init(&pool.quit_wait, NULL); pool.max_workers = max_workers; pool.info = malloc(max_workers * sizeof(struct worker_info)); if (!pool.info) return -ENOMEM; memset(pool.info, 0, max_workers * sizeof(struct worker_info)); for (i = 0; i < min_workers; i++) { rv = pthread_create(&th, NULL, thread_pool_worker, (void *)(long)i); if (rv) { log_error("thread_pool_create failed %d", rv); rv = -1; break; } pool.num_workers++; daemon_status_num_workers++; /* for status reporting */ } if (rv < 0) thread_pool_free(); return rv; } /* * cmd comes from a transient client/fd set up just to pass the cmd, * and is not being done on behalf of another registered client/fd. * The command is processed independently of the lifetime of a specific * client or the tokens held by a specific client. */ static void process_cmd_thread_unregistered(int ci_in, struct sm_header *h_recv) { struct cmd_args *ca; int rv; ca = malloc(sizeof(struct cmd_args)); if (!ca) { rv = -ENOMEM; goto fail; } ca->ci_in = ci_in; memcpy(&ca->header, h_recv, sizeof(struct sm_header)); snprintf(client[ci_in].owner_name, SANLK_NAME_LEN, "cmd%d", h_recv->cmd); log_client(ci_in, client[ci_in].fd, "process cmd %u", h_recv->cmd); rv = thread_pool_add_work(ca); if (rv < 0) goto fail_free; return; fail_free: free(ca); fail: log_error("cmd %d %d:%d process_unreg error %d", h_recv->cmd, ci_in, client[ci_in].fd, rv); client_recv_all(ci_in, h_recv, 0); send_result(ci_in, client[ci_in].fd, h_recv, rv); client_resume(ci_in); } /* * cmd either comes from a registered client/fd, or is targeting a registered * client/fd. The processing of the cmd is closely coordinated with the * lifetime of a specific client and to tokens held by that client. Handling * of the client's death or changing of the client's tokens will be serialized * with the processing of this command. This means that the end of processing * this command needs to check if the client failed during the command * processing and handle the cleanup of the client if so. */ static void process_cmd_thread_registered(int ci_in, struct sm_header *h_recv) { struct cmd_args *ca; struct client *cl; int result = 0; int for_pid = -1, for_client_id = -1; int rv, i, ci_target = -1; ca = malloc(sizeof(struct cmd_args)); if (!ca) { result = -ENOMEM; goto fail; } if (h_recv->data2 != -1) { /* lease for another registered client with pid or client_id specified by data2 */ if (h_recv->cmd_flags & SANLK_FOR_CLIENT_ID) for_client_id = h_recv->data2; else for_pid = h_recv->data2; for (i = 0; i < client_size; i++) { cl = &client[i]; pthread_mutex_lock(&cl->mutex); if (for_pid > -1) { if (cl->pid != for_pid) { pthread_mutex_unlock(&cl->mutex); continue; } } else if (for_client_id > -1) { if (i != for_client_id) { pthread_mutex_unlock(&cl->mutex); continue; } } ci_target = i; break; } if (ci_target < 0) { if (h_recv->cmd != SM_CMD_INQUIRE) { /* inquire can be used to check if a pid or client_id exists */ log_debug("cmd %d target %d not found", h_recv->cmd, h_recv->data2); } result = -ESRCH; goto fail; } log_client(ci_in, client[ci_in].fd, "process reg cmd %u for target ci %d pid %d", h_recv->cmd, ci_target, client[ci_target].pid); } else { /* lease for this registered client */ log_client(ci_in, client[ci_in].fd, "process reg cmd %u", h_recv->cmd); ci_target = ci_in; cl = &client[ci_target]; pthread_mutex_lock(&cl->mutex); } if (!cl->used) { log_error("cmd %d %d,%d,%d not used", h_recv->cmd, ci_target, cl->fd, cl->pid); result = -EBUSY; goto out; } if (cl->pid <= 0) { log_error("cmd %d %d,%d,%d no pid", h_recv->cmd, ci_target, cl->fd, cl->pid); result = -EBUSY; goto out; } if (cl->pid_dead) { log_error("cmd %d %d,%d,%d pid_dead", h_recv->cmd, ci_target, cl->fd, cl->pid); result = -EBUSY; goto out; } if (cl->need_free) { log_error("cmd %d %d,%d,%d need_free", h_recv->cmd, ci_target, cl->fd, cl->pid); result = -EBUSY; goto out; } if (cl->kill_count && (h_recv->cmd == SM_CMD_ACQUIRE || h_recv->cmd == SM_CMD_ACQUIRE2)) { /* when pid is being killed, we want killpath to be able to inquire and release for it */ log_error("cmd %d %d,%d,%d kill_count %d", h_recv->cmd, ci_target, cl->fd, cl->pid, cl->kill_count); result = -EBUSY; goto out; } if (cl->cmd_active) { if (com.quiet_fail && (cl->cmd_active == SM_CMD_ACQUIRE || cl->cmd_active == SM_CMD_ACQUIRE2)) { result = -EBUSY; goto out; } log_error("cmd %d %d,%d,%d cmd_active %d", h_recv->cmd, ci_target, cl->fd, cl->pid, cl->cmd_active); result = -EBUSY; goto out; } cl->cmd_active = h_recv->cmd; /* once cmd_active is set, client_pid_dead() will not clear cl->tokens or call client_free, so it's the responsibility of cmd_a,r,i_thread to check if pid_dead when clearing cmd_active, and doing the cleanup if pid is dead */ out: pthread_mutex_unlock(&cl->mutex); if (result < 0) goto fail; ca->ci_in = ci_in; ca->ci_target = ci_target; ca->cl_pid = cl->pid; ca->cl_fd = cl->fd; memcpy(&ca->header, h_recv, sizeof(struct sm_header)); rv = thread_pool_add_work(ca); if (rv < 0) { /* we don't have to worry about client_pid_dead having been called while mutex was unlocked with cmd_active set, because client_pid_dead is called from the main thread which is running this function */ log_error("create cmd thread failed"); pthread_mutex_lock(&cl->mutex); cl->cmd_active = 0; pthread_mutex_unlock(&cl->mutex); result = rv; goto fail; } return; fail: log_error("process_cmd_thread_reg failed ci %d fd %d cmd %u", ci_in, client[ci_in].fd, h_recv->cmd); client_recv_all(ci_in, h_recv, 0); send_result(ci_in, client[ci_in].fd, h_recv, result); client_resume(ci_in); if (ca) free(ca); } static void process_connection(int ci) { struct sm_header h; void (*deadfn)(int ci); int rv; memset(&h, 0, sizeof(h)); retry: rv = recv(client[ci].fd, &h, sizeof(h), MSG_WAITALL); if (rv == -1 && errno == EINTR) goto retry; if (!rv) goto dead; log_client(ci, client[ci].fd, "recv %d %u", rv, h.cmd); if (rv < 0) { log_error("client connection %d %d %d recv msg header rv %d errno %d", ci, client[ci].fd, client[ci].pid, rv, errno); goto bad; } if (rv != sizeof(h)) { log_error("client connection %d %d %d recv msg header rv %d cmd %u len %u", ci, client[ci].fd, client[ci].pid, rv, h.cmd, h.length); goto bad; } if (h.magic != SM_MAGIC) { log_error("client connection %d %d %d recv msg header rv %d cmd %u len %u magic %x vs %x", ci, client[ci].fd, client[ci].pid, rv, h.cmd, h.length, h.magic, SM_MAGIC); goto bad; } if (client[ci].restricted & SANLK_RESTRICT_ALL) { log_error("client connection %d %d %d recv msg header rv %d cmd %u len %u restrict all", ci, client[ci].fd, client[ci].pid, rv, h.cmd, h.length); goto bad; } if (h.version && (h.cmd != SM_CMD_VERSION) && (h.version & 0xFFFF0000) > (SM_PROTO & 0xFFFF0000)) { log_error("client connection %d %d %d recv msg header rv %d cmd %u len %u version %x", ci, client[ci].fd, client[ci].pid, rv, h.cmd, h.length, h.version); goto bad; } client[ci].cmd_last = h.cmd; switch (h.cmd) { case SM_CMD_REGISTER: case SM_CMD_RESTRICT: case SM_CMD_VERSION: case SM_CMD_SHUTDOWN: case SM_CMD_STATUS: case SM_CMD_HOST_STATUS: case SM_CMD_RENEWAL: case SM_CMD_LOG_DUMP: case SM_CMD_GET_LOCKSPACES: case SM_CMD_GET_HOSTS: case SM_CMD_REG_EVENT: case SM_CMD_END_EVENT: case SM_CMD_SET_CONFIG: call_cmd_daemon(ci, &h, client_maxi); break; case SM_CMD_ADD_LOCKSPACE: case SM_CMD_INQ_LOCKSPACE: case SM_CMD_REM_LOCKSPACE: case SM_CMD_REQUEST: case SM_CMD_EXAMINE_RESOURCE: case SM_CMD_EXAMINE_LOCKSPACE: case SM_CMD_ALIGN: case SM_CMD_WRITE_LOCKSPACE: case SM_CMD_WRITE_RESOURCE: case SM_CMD_READ_LOCKSPACE: case SM_CMD_READ_LOCKSPACE_HOST: case SM_CMD_INIT_LOCKSPACE_HOST: case SM_CMD_READ_RESOURCE: case SM_CMD_READ_RESOURCE_OWNERS: case SM_CMD_SET_LVB: case SM_CMD_GET_LVB: case SM_CMD_SHUTDOWN_WAIT: case SM_CMD_SET_EVENT: case SM_CMD_SET_HOST: case SM_CMD_FORMAT_RINDEX: case SM_CMD_REBUILD_RINDEX: case SM_CMD_UPDATE_RINDEX: case SM_CMD_LOOKUP_RINDEX: case SM_CMD_CREATE_RESOURCE: case SM_CMD_DELETE_RESOURCE: rv = client_suspend(ci); if (rv < 0) goto bad; process_cmd_thread_unregistered(ci, &h); break; case SM_CMD_ACQUIRE: case SM_CMD_RELEASE: case SM_CMD_INQUIRE: case SM_CMD_CONVERT: case SM_CMD_KILLPATH: case SM_CMD_ACQUIRE2: /* the main_loop needs to ignore this connection while the thread is working on it */ rv = client_suspend(ci); if (rv < 0) goto bad; process_cmd_thread_registered(ci, &h); break; default: log_error("client connection ci %d fd %d pid %d cmd %d unknown", ci, client[ci].fd, client[ci].pid, h.cmd); goto bad; }; return; bad: return; dead: log_client(ci, client[ci].fd, "recv dead"); deadfn = client[ci].deadfn; if (deadfn) deadfn(ci); } static void process_listener(int ci GNUC_UNUSED) { int fd; int on = 1; fd = accept(client[ci].fd, NULL, NULL); if (fd < 0) return; setsockopt(fd, SOL_SOCKET, SO_PASSCRED, &on, sizeof(on)); client_add(fd, process_connection, NULL); } static int setup_listener(void) { struct sockaddr_un addr; int rv, fd, ci; rv = sanlock_socket_address(run_dir, &addr); if (rv < 0) return rv; fd = socket(AF_LOCAL, SOCK_STREAM, 0); if (fd < 0) return fd; unlink(addr.sun_path); rv = bind(fd, (struct sockaddr *) &addr, sizeof(struct sockaddr_un)); if (rv < 0) goto exit_fail; rv = chmod(addr.sun_path, DEFAULT_SOCKET_MODE); if (rv < 0) goto exit_fail; rv = chown(addr.sun_path, com.uid, com.gid); if (rv < 0) { log_error("could not set socket %s permissions: %s", addr.sun_path, strerror(errno)); goto exit_fail; } rv = listen(fd, 5); if (rv < 0) goto exit_fail; fcntl(fd, F_SETFL, fcntl(fd, F_GETFL, 0) | O_NONBLOCK); ci = client_add(fd, process_listener, NULL); if (ci < 0) goto exit_fail; strcpy(client[ci].owner_name, "listener"); return 0; exit_fail: close(fd); return -1; } static void sigterm_handler(int sig GNUC_UNUSED, siginfo_t *info GNUC_UNUSED, void *ctx GNUC_UNUSED) { external_shutdown = 1; } static void setup_priority(void) { struct sched_param sched_param; int rv = 0; if (com.mlock_level == 1) rv = mlockall(MCL_CURRENT); else if (com.mlock_level == 2) rv = mlockall(MCL_CURRENT | MCL_FUTURE); if (rv < 0) { log_error("mlockall %d failed: %s", com.mlock_level, strerror(errno)); } if (!com.high_priority) return; rv = sched_get_priority_max(SCHED_RR); if (rv < 0) { log_error("could not get max scheduler priority err %d", errno); return; } sched_param.sched_priority = rv; rv = sched_setscheduler(0, SCHED_RR|SCHED_RESET_ON_FORK, &sched_param); if (rv < 0) { log_error("set scheduler RR|RESET_ON_FORK priority %d failed: %s", sched_param.sched_priority, strerror(errno)); } } /* return a random int between a and b inclusive */ int get_rand(int a, int b); int get_rand(int a, int b) { long int rv; pthread_mutex_lock(&rand_mutex); rv = random(); pthread_mutex_unlock(&rand_mutex); if (rv < 0) return rv; return a + (int) (((float)(b - a + 1)) * rv / (RAND_MAX+1.0)); } static void read_product_uuid(void) { FILE *fp; char full[256] = { 0 }; int len; int i, j; memset(our_product_uuid, 0, sizeof(our_product_uuid)); memset(our_product_uuid_compact, 0, sizeof(our_product_uuid_compact)); if (!(fp = fopen("/sys/devices/virtual/dmi/id/product_uuid", "r"))) return; if (!fgets(full, sizeof(full), fp)) { fclose(fp); return; } fclose(fp); full[sizeof(full)-1] = '\0'; len = strlen(full); if (len && full[len - 1] == '\n') full[--len] = '\0'; len = strlen(full); /* * Randomly pick 16 as a minimum legitimate size for a product_uuid * (expected to be 36 for a proper uuid including dashes) */ if (len < 16) { log_debug("Ignore product_uuid that is too short %d (%s)", len, full); return; } /* * We can't trust that only a portion of the product_uuid * would be unique. */ if (len > SANLK_NAME_LEN) { log_debug("Ignore product_uuid that is too long %d (%s)", len, full); return; } memcpy(our_product_uuid, full, SANLK_NAME_LEN); if (strchr(our_product_uuid, '-')) { for (i = 0, j = 0; i < len; i++) { if (our_product_uuid[i] == '-') continue; our_product_uuid_compact[j++] = our_product_uuid[i]; } if (strlen(our_product_uuid_compact) < 16) { log_debug("Ignore compact product_uuid that is too short (%s)", our_product_uuid_compact); memset(our_product_uuid_compact, 0, sizeof(our_product_uuid_compact)); } } } static void setup_host_name(void) { char our_host_name_long[1024] = { 0 }; /* temp buf for snprintf, then memcpy to _global */ char rand_uuid[37] = { 0 }; struct utsname name = { 0 }; uuid_t uu; read_product_uuid(); uname(&name); /* * Get host name value from: * 1. command line * 2. config file * 3. /sys/devices/virtual/dmi/id/product_uuid * 4. generate random uuid that won't collide with another host */ if (com.our_host_name_opt[0]) { memcpy(our_host_name_global, com.our_host_name_opt, SANLK_NAME_LEN); /* * user could configure our_host_name using product_uuid, * in which case we can enable the no delay optimization * when using product_uuid. */ if (our_product_uuid[0] && !strncmp(our_host_name_global, our_product_uuid, strlen(our_product_uuid))) our_host_name_matches_product_uuid = 1; if (our_product_uuid_compact[0] && !strncmp(our_host_name_global, our_product_uuid_compact, strlen(our_product_uuid_compact))) our_host_name_matches_product_uuid = 1; if (strlen(com.our_host_name_opt) > SANLK_NAME_LEN) log_warn("our_host_name shortened from config %s len %ld to: %s", com.our_host_name_opt, strlen(com.our_host_name_opt), our_host_name_global); else log_debug("our_host_name set from config: %s", our_host_name_global); if (our_host_name_matches_product_uuid) log_debug("our_host_name uses product_uuid"); return; } /* * A typical 36 char uuid, plus a '.' separator, leaves space for the first * 11 characters of the local nodename, which is included at the end mainly * to help with readability/debugging. */ if (our_product_uuid[0]) { snprintf(our_host_name_long, sizeof(our_host_name_long), "%s.%s", our_product_uuid, name.nodename); memcpy(our_host_name_global, our_host_name_long, SANLK_NAME_LEN); our_host_name_matches_product_uuid = 1; log_debug("our_host_name set from product_uuid: %s", our_host_name_global); } else { memset(rand_state, 0, sizeof(rand_state)); initstate(time(NULL), rand_state, sizeof(rand_state)); uuid_generate(uu); uuid_unparse_lower(uu, rand_uuid); snprintf(our_host_name_long, sizeof(our_host_name_long), "%s.%s", rand_uuid, name.nodename); memcpy(our_host_name_global, our_host_name_long, SANLK_NAME_LEN); log_debug("our_host_name set from uuid_generate: %s", our_host_name_global); } } static void setup_limits(void) { int rv; struct rlimit rlim = { .rlim_cur = -1, .rlim_max= -1 }; if (!privileged) return; rv = setrlimit(RLIMIT_MEMLOCK, &rlim); if (rv < 0) { log_error("cannot set the limits for memlock %i", errno); exit(EXIT_FAILURE); } rv = setrlimit(RLIMIT_RTPRIO, &rlim); if (rv < 0) { log_error("cannot set the limits for rtprio %i", errno); exit(EXIT_FAILURE); } rv = setrlimit(RLIMIT_CORE, &rlim); if (rv < 0) { log_error("cannot set the limits for core dumps %i", errno); exit(EXIT_FAILURE); } } static void setup_groups(void) { int rv; if (!com.uname || !com.gname || !privileged) return; rv = initgroups(com.uname, com.gid); if (rv < 0) { log_error("error initializing groups errno %i", errno); } } static void setup_uid_gid(void) { int rv; if (!com.uname || !com.gname || !privileged) return; rv = setgid(com.gid); if (rv < 0) { log_error("cannot set group id to %i errno %i", com.gid, errno); } rv = setuid(com.uid); if (rv < 0) { log_error("cannot set user id to %i errno %i", com.uid, errno); } /* When a program is owned by a user (group) other than the real user * (group) ID of the process, the PR_SET_DUMPABLE option gets cleared. * See RLIMIT_CORE in setup_limits and man 5 core. */ rv = prctl(PR_SET_DUMPABLE, 1, 0, 0, 0); if (rv < 0) { log_error("cannot set dumpable process errno %i", errno); } } static void setup_signals(void) { struct sigaction act; int rv, i, sig_list[] = { SIGHUP, SIGINT, SIGTERM, 0 }; memset(&act, 0, sizeof(act)); act.sa_flags = SA_SIGINFO; act.sa_sigaction = sigterm_handler; for (i = 0; sig_list[i] != 0; i++) { rv = sigaction(sig_list[i], &act, NULL); if (rv < 0) { log_error("cannot set the signal handler for: %i", sig_list[i]); exit(EXIT_FAILURE); } } } /* * first pipe for daemon to send requests to helper; they are not acknowledged * and the daemon does not get any result back for the requests. * * second pipe for helper to send general status/heartbeat back to the daemon * every so often to confirm it's not dead/hung. If the helper gets stuck or * killed, the daemon will not get the status and won't bother sending requests * to the helper, and use SIGTERM instead */ static int setup_helper(void) { int pid; int pw_fd = -1; /* parent write */ int cr_fd = -1; /* child read */ int pr_fd = -1; /* parent read */ int cw_fd = -1; /* child write */ int pfd[2]; /* we can't allow the main daemon thread to block */ if (pipe2(pfd, O_NONBLOCK | O_CLOEXEC)) return -errno; /* uncomment for rhel7 where this should be available */ /* fcntl(pfd[1], F_SETPIPE_SZ, 1024*1024); */ cr_fd = pfd[0]; pw_fd = pfd[1]; if (pipe2(pfd, O_NONBLOCK | O_CLOEXEC)) { close(cr_fd); close(pw_fd); return -errno; } pr_fd = pfd[0]; cw_fd = pfd[1]; pid = fork(); if (pid < 0) { close(cr_fd); close(pw_fd); close(pr_fd); close(cw_fd); return -errno; } if (pid) { close(cr_fd); close(cw_fd); helper_kill_fd = pw_fd; helper_status_fd = pr_fd; helper_pid = pid; return 0; } else { close(pr_fd); close(pw_fd); is_helper = 1; run_helper(cr_fd, cw_fd, (log_stderr_priority == LOG_DEBUG)); exit(0); } } static void process_helper(int ci) { struct helper_status hs; int rv; memset(&hs, 0, sizeof(hs)); rv = read(client[ci].fd, &hs, sizeof(hs)); if (!rv || rv == -EAGAIN) return; if (rv < 0) { log_error("process_helper rv %d errno %d", rv, errno); goto fail; } if (rv != sizeof(hs)) { log_error("process_helper recv size %d", rv); goto fail; } if (hs.type == HELPER_STATUS && !hs.status) helper_last_status = monotime(); return; fail: close_helper(); } static void helper_dead(int ci GNUC_UNUSED) { int pid = helper_pid; int rv, status; close_helper(); helper_pid = -1; rv = waitpid(pid, &status, WNOHANG); if (rv != pid) { /* should not happen */ log_error("helper pid %d dead wait %d", pid, rv); return; } if (WIFEXITED(status)) { log_error("helper pid %d exit status %d", pid, WEXITSTATUS(status)); return; } if (WIFSIGNALED(status)) { log_error("helper pid %d term signal %d", pid, WTERMSIG(status)); return; } /* should not happen */ log_error("helper pid %d state change", pid); } static int do_daemon(void) { struct utsname nodename; int fd, rv; run_dir = env_get(SANLOCK_RUN_DIR, DEFAULT_RUN_DIR); privileged = env_get_bool(SANLOCK_PRIVILEGED, 1); /* This can take a while so do it before forking. */ setup_groups(); if (!com.debug) { /* TODO: copy comprehensive daemonization method from libvirtd */ if (daemon(0, 0) < 0) { log_tool("cannot fork daemon\n"); exit(EXIT_FAILURE); } } setup_limits(); setup_timeouts(); setup_helper(); /* main task never does disk io, so we don't really need to set * it up, but other tasks get their use_aio value by copying * the main_task settings */ sprintf(main_task.name, "%s", "main"); setup_task_aio(&main_task, com.aio_arg, 0); rv = client_alloc(); if (rv < 0) return rv; helper_ci = client_add(helper_status_fd, process_helper, helper_dead); if (helper_ci < 0) return rv; strcpy(client[helper_ci].owner_name, "helper"); setup_signals(); setup_logging(); if (strcmp(run_dir, DEFAULT_RUN_DIR)) log_warn("Using non-standard run directory '%s'", run_dir); if (!privileged) log_warn("Running in unprivileged mode"); /* If we run as root, make run_dir owned by root, so we can create the * lockfile when selinux disables DAC_OVERRIDE. * See https://danwalsh.livejournal.com/79643.html */ fd = lockfile(run_dir, SANLK_LOCKFILE_NAME, com.uid, privileged ? 0 : com.gid); if (fd < 0) { close_logging(); return fd; } setup_host_name(); setup_uid_gid(); uname(&nodename); if (com.io_timeout != DEFAULT_IO_TIMEOUT || com.watchdog_fire_timeout != DEFAULT_WATCHDOG_FIRE_TIMEOUT) { log_warn("sanlock daemon started %s host %s (%s) io_timeout %u watchdog_fire_timeout %u", VERSION, our_host_name_global, nodename.nodename, com.io_timeout, com.watchdog_fire_timeout); syslog(LOG_INFO, "sanlock daemon started %s host %s (%s) io_timeout %u watchdog_fire_timeout %u", VERSION, our_host_name_global, nodename.nodename, com.io_timeout, com.watchdog_fire_timeout); } else { log_warn("sanlock daemon started %s host %s (%s)", VERSION, our_host_name_global, nodename.nodename); syslog(LOG_INFO, "sanlock daemon started %s host %s (%s)", VERSION, our_host_name_global, nodename.nodename); } setup_priority(); rv = thread_pool_create(DEFAULT_MIN_WORKER_THREADS, com.max_worker_threads); if (rv < 0) goto out; rv = setup_listener(); if (rv < 0) goto out_threads; setup_token_manager(); if (rv < 0) goto out_threads; /* initialize global eventfd for client_resume notification */ if ((efd = eventfd(0, EFD_CLOEXEC | EFD_NONBLOCK)) == -1) { log_error("couldn't create eventfd"); goto out_threads; } main_loop(); close_token_manager(); out_threads: thread_pool_free(); out: /* order reversed from setup so lockfile is last */ close_logging(); close(fd); return rv; } static int user_to_uid(char *arg) { struct passwd *pw; pw = getpwnam(arg); if (pw == NULL) { log_error("user '%s' not found, " "using uid: %i", arg, DEFAULT_SOCKET_UID); return DEFAULT_SOCKET_UID; } return pw->pw_uid; } static int group_to_gid(char *arg) { struct group *gr; gr = getgrnam(arg); if (gr == NULL) { log_error("group '%s' not found, " "using uid: %i", arg, DEFAULT_SOCKET_GID); return DEFAULT_SOCKET_GID; } return gr->gr_gid; } static int parse_arg_rentry(char *str) { char *name = NULL; char *offset = NULL; if (!str) return -EINVAL; /* "-r :1M" can be used to specify only an offset */ if (str[0] != ':') name = str; if ((offset = strchr(str, ':'))) { uint64_t offnum; char *m ; *offset = '\0'; offset++; if ((m = strchr(offset, 'M'))) { *m = '\0'; offnum = atoll(offset) * 1024 * 1024; } else { offnum = atoll(offset); } com.rentry.offset = offnum; } if (name) { char tmps[SANLK_NAME_LEN+1] = { 0 }; strncpy(tmps, name, SANLK_NAME_LEN); memcpy(com.rentry.name, tmps, SANLK_NAME_LEN); } return 0; } static int parse_arg_rindex(char *str) { char *ls_name = NULL; char *path = NULL; char *offset = NULL; int i; if (!str) return -EINVAL; ls_name = &str[0]; for (i = 0; i < strlen(str); i++) { if (str[i] == '\\') { i++; continue; } if (str[i] == ':') { if (!path) path = &str[i]; else if (!offset) offset = &str[i]; } } if (path) { *path = '\0'; path++; } if (offset) { *offset= '\0'; offset++; } if (ls_name) strncpy(com.rindex.lockspace_name, ls_name, SANLK_NAME_LEN); if (path) sanlock_path_import(com.rindex.disk.path, path, sizeof(com.rindex.disk.path)); if (offset) { uint64_t offnum; char *m ; if ((m = strchr(offset, 'M'))) { *m = '\0'; offnum = atoll(offset) * 1024 * 1024; } else { offnum = atoll(offset); } com.rindex.disk.offset = offnum; } return 0; } /* ::: */ static int parse_arg_lockspace(char *arg) { char offstr[16]; char *colon1, *colon2, *colon3, *m, *p; uint64_t offnum = 0; char *arg2 = NULL; int len = strlen(arg); int len2 = 0; int i; /* * If the arg string uses an offset with the 'M' suffix, then * convert it to a string without 'M'. */ if ((colon1 = strchr(arg, ':'))) { if ((colon2 = strchr(colon1+1, ':'))) { if ((colon3 = strchr(colon2+1, ':'))) { if ((m = strchr(colon3+1, 'M'))) { p = colon3+1; i = 0; while (1) { offstr[i++] = *p; p++; if (p == m) break; } offnum = atoll(offstr) * 1024 * 1024; /* terminate 'arg' before offset */ *colon3 = '\0'; len2 = len + 64; arg2 = malloc(len2); if (!arg2) return -1; memset(arg2, 0, len2); snprintf(arg2, len2, "%s:%llu", arg, (unsigned long long)offnum); } } } } if (arg2) sanlock_str_to_lockspace(arg2, &com.lockspace); else sanlock_str_to_lockspace(arg, &com.lockspace); log_debug("lockspace %s host_id %llu path %s offset %llu", com.lockspace.name, (unsigned long long)com.lockspace.host_id, com.lockspace.host_id_disk.path, (unsigned long long)com.lockspace.host_id_disk.offset); return 0; } /* :::[:] */ static int parse_arg_resource(char *arg) { struct sanlk_resource *res; char offstr[16]; char *colon1, *colon2, *colon3, *colon4, *m, *p; uint64_t offnum = 0; char *arg2 = NULL; int len = strlen(arg); int len2 = 0; int rv, i; if (com.res_count >= SANLK_MAX_RESOURCES) { log_tool("resource args over max %d", SANLK_MAX_RESOURCES); return -1; } memset(offstr, 0, sizeof(offstr)); /* * If the arg string uses an offset with the 'M' suffix, then * convert it to a string without 'M'. */ if ((colon1 = strchr(arg, ':'))) { if ((colon2 = strchr(colon1+1, ':'))) { if ((colon3 = strchr(colon2+1, ':'))) { colon4 = strchr(colon3+1, ':'); /* optional */ if ((m = strchr(colon3+1, 'M'))) { p = colon3+1; i = 0; while (1) { offstr[i++] = *p; p++; if (p == m) break; } offnum = atoll(offstr) * 1024 * 1024; /* terminate 'arg' before offset */ *colon3 = '\0'; len2 = len + 64; arg2 = malloc(len2); if (!arg2) return -1; memset(arg2, 0, len2); if (!colon4) snprintf(arg2, len2, "%s:%llu", arg, (unsigned long long)offnum); else snprintf(arg2, len2, "%s:%llu%s", arg, (unsigned long long)offnum, colon4); } } } } if (arg2) rv = sanlock_str_to_res(arg2, &res); else rv = sanlock_str_to_res(arg, &res); if (rv < 0) { log_tool("resource arg parse error %d\n", rv); return rv; } com.res_args[com.res_count] = res; com.res_count++; log_debug("resource %s %s num_disks %d flags %x lver %llu", res->lockspace_name, res->name, res->num_disks, res->flags, (unsigned long long)res->lver); for (i = 0; i < res->num_disks; i++) { log_debug("resource disk %s %llu", res->disks[i].path, (unsigned long long)res->disks[i].offset); } return 0; } /* * daemon: acquires leases for the local host_id, associates them with a local * pid, and releases them when the associated pid exits. * * client: ask daemon to acquire/release leases associated with a given pid. * * direct: acquires and releases leases directly for the local host_id by * reading and writing storage directly. */ static void print_usage(void) { printf("Usage:\n"); printf("sanlock ...\n\n"); printf("commands:\n"); printf(" daemon start daemon\n"); printf(" client send request to daemon (default type if none given)\n"); printf(" direct access storage directly (no coordination with daemon)\n"); printf(" help print this usage (defaults in parens)\n"); printf(" version print version\n"); printf("\n"); printf("sanlock daemon [options]\n"); printf(" -D no fork and print all logging to stderr\n"); printf(" -Q 0|1 quiet error messages for common lock contention (%d)\n", DEFAULT_QUIET_FAIL); printf(" -R 0|1 renewal debugging, log debug info about renewals (0)\n"); printf(" -H renewal history size (%d)\n", DEFAULT_RENEWAL_HISTORY_SIZE); printf(" -L write logging at priority level and up to logfile (4 LOG_WARNING)\n"); printf(" (use -1 for none)\n"); printf(" -S write logging at priority level and up to syslog (3 LOG_ERR)\n"); printf(" (use -1 for none)\n"); printf(" -U user id\n"); printf(" -G group id\n"); printf(" -t max worker threads (%d)\n", DEFAULT_MAX_WORKER_THREADS); printf(" -g seconds for graceful recovery (%d)\n", DEFAULT_GRACE_SEC); printf(" -o io timeout (%d)\n", DEFAULT_IO_TIMEOUT); printf(" -w 0|1 use watchdog through wdmd (%d)\n", DEFAULT_USE_WATCHDOG); printf(" -h 0|1 use high priority (RR) scheduling (%d)\n", DEFAULT_HIGH_PRIORITY); printf(" -l use mlockall (0 none, 1 current, 2 current and future) (%d)\n", DEFAULT_MLOCK_LEVEL); printf(" -b seconds a host id bit will remain set in delta lease bitmap\n"); printf(" (default: 6 * io_timeout)\n"); printf(" -e local host name used in delta leases, max len 48\n"); printf(" (default: product_uuid or randomly generated uuid)\n"); printf("\n"); printf("sanlock client [options]\n"); printf("sanlock client status [-D] [-o p|s]\n"); printf("sanlock client gets [-h 0|1]\n"); printf("sanlock client host_status -s LOCKSPACE [-D]\n"); printf("sanlock client renewal -s LOCKSPACE\n"); printf("sanlock client set_host -s LOCKSPACE -i [-g gen] -F \n"); printf("sanlock client set_event -s LOCKSPACE -i [-g gen] -e -d \n"); printf("sanlock client set_config -s LOCKSPACE [-u 0|1] [-O 0|1]\n"); printf("sanlock client log_dump\n"); printf("sanlock client shutdown [-f 0|1] [-w 0|1]\n"); printf("sanlock client init -s LOCKSPACE | -r RESOURCE [-z 0|1] [-Z 512|4096 -A 1M|2M|4M|8M] [-C 0|1]\n"); printf("sanlock client init_host -s LOCKSPACE [-g ] [-t ] [-N 1] [-e ] [-Z 512|4096 -A 1M|2M|4M|8M] [-C 0|1]\n"); printf("sanlock client read -s LOCKSPACE | -r RESOURCE [-D]\n"); printf("sanlock client add_lockspace -s LOCKSPACE [-N 1]\n"); printf("sanlock client inq_lockspace -s LOCKSPACE\n"); printf("sanlock client rem_lockspace -s LOCKSPACE\n"); printf("sanlock client command -r RESOURCE [-h 0|1] -c \n"); printf("sanlock client spawn -r RESOURCE [-h 0|1] [-O 0|1] [-P 0|1] [-d 0|1] -c COUNT CMD [ARG...] [-c COUNT CMD [ARG...]]\n"); printf("sanlock client acquire -r RESOURCE -p|-C \n"); printf("sanlock client convert -r RESOURCE -p|-C \n"); printf("sanlock client release -r RESOURCE -p|-C \n"); printf("sanlock client inquire -p|-C \n"); printf("sanlock client request -r RESOURCE -f \n"); printf("sanlock client examine -r RESOURCE | -s LOCKSPACE\n"); #if 0 printf("sanlock client format -x RINDEX [-Z 512|4096 -A 1M|2M|4M|8M]\n"); printf("sanlock client create -x RINDEX -e \n"); printf("sanlock client delete -x RINDEX -e [:]\n"); printf("sanlock client lookup -x RINDEX [-e :]\n"); printf("sanlock client update -x RINDEX -e [:] [-z 0|1]\n"); printf("sanlock client rebuild -x RINDEX\n"); #endif printf("\n"); printf("sanlock direct [-a 0|1] [-o 0|1] [-Z 512|4096 -A 1M|2M|4M|8M]\n"); printf("sanlock direct init -s LOCKSPACE | -r RESOURCE [-Z 512|4096 -A 1M|2M|4M|8M] [-C 0|1]\n"); printf("sanlock direct init_host -s LOCKSPACE [-g ] [-t ] [-N 1] [-Z 512|4096 -A 1M|2M|4M|8M] [-C 0|1]\n"); printf("sanlock direct read_leader -s LOCKSPACE | -r RESOURCE\n"); printf("sanlock direct dump [:[:]]\n"); #if 0 printf("sanlock direct format -x RINDEX [-Z 512|4096 -A 1M|2M|4M|8M]\n"); printf("sanlock direct lookup -x RINDEX [-e :]\n"); printf("sanlock direct update -x RINDEX -e [:] [-z 0|1]\n"); printf("sanlock direct rebuild -x RINDEX\n"); #endif printf("\n"); printf("LOCKSPACE = :::\n"); printf(" name of lockspace\n"); printf(" local host identifier in lockspace\n"); printf(" path to storage reserved for leases\n"); printf(" offset on path (bytes)\n"); printf("\n"); printf("RESOURCE = :::[:]\n"); printf(" name of lockspace\n"); printf(" name of resource\n"); printf(" path to storage reserved for leases\n"); printf(" offset on path (bytes)\n"); printf(" optional leader version or SH for shared lease\n"); printf("\n"); #if 0 printf("RINDEX = ::\n"); printf(" name of lockspace\n"); printf(" path to storage reserved for leases\n"); printf(" offset on path (bytes)\n"); printf("\n"); #endif printf("Limits:\n"); printf("valid sector/align size combinations: 512/1M, 4K/1M, 4K/2M, 4K/4M, 4K/8M\n"); printf("maximum host_id for sector/align sizes: 2000, 250, 500, 1000, 2000\n"); printf("maximum name length for lockspaces and resources: %d\n", SANLK_NAME_LEN); printf("maximum path length: %d\n", SANLK_PATH_LEN); printf("maximum client process connections: 1000\n"); /* NALLOC */ printf("\n"); } /* * read_spawn_command - parse a single "-c COUNT CMD [ARG...]" option group * * Called with I pointing at the COUNT argument (the option argument * that follows -c). Parses COUNT and the subsequent CMD [ARG...] words, * appends a new entry to com.spawn_args, and returns the updated index * pointing at the next option. The caller can then check argv[i] to decide * whether another -c group follows. * * Returns the updated index (>= 0) on success, or a negative error code. */ static int read_spawn_command(int i, int argc, char *argv[]) { int count, k; if (i >= argc) { log_tool("spawn -c requires COUNT and CMD"); return -EINVAL; } count = strtol(argv[i], NULL, 0); if (i + count >= argc) { log_tool("spawn -c COUNT past the end of arguments"); return -EINVAL; } com.spawn_args = realloc(com.spawn_args, (com.spawn_count + 1) * sizeof(struct spawn_cmd)); if (!com.spawn_args) return -ENOMEM; com.spawn_args[com.spawn_count].argc = count; com.spawn_args[com.spawn_count].argv = malloc((count + 1) * sizeof(char *)); if (!com.spawn_args[com.spawn_count].argv) return -ENOMEM; for (k = 0; k < count; k++) { i += 1; com.spawn_args[com.spawn_count].argv[k] = strdup(argv[i]); if (!com.spawn_args[com.spawn_count].argv[k]) return -ENOMEM; } com.spawn_args[com.spawn_count].argv[count] = NULL; com.spawn_count += 1; return i + 1; } static int read_command_line(int argc, char *argv[]) { char optchar; char *optionarg; char *p; char *arg1 = argv[1]; char *act; int i, j, len, sec, val, begin_command = 0; if (argc < 2 || !strcmp(arg1, "help") || !strcmp(arg1, "--help") || !strcmp(arg1, "-h")) { print_usage(); exit(EXIT_SUCCESS); } if (!strcmp(arg1, "version")) { printf("%u.%u.%u\n", sanlock_version_major, sanlock_version_minor, sanlock_version_patch); exit(EXIT_SUCCESS); } if (!strcmp(arg1, "--version") || !strcmp(arg1, "-V")) { printf("%s %s (built %s %s)\n", argv[0], VERSION, __DATE__, __TIME__); exit(EXIT_SUCCESS); } if (!strcmp(arg1, "daemon")) { com.type = COM_DAEMON; i = 2; } else if (!strcmp(arg1, "direct")) { com.type = COM_DIRECT; if (argc < 3) { print_usage(); exit(EXIT_FAILURE); } act = argv[2]; i = 3; } else if (!strcmp(arg1, "client")) { com.type = COM_CLIENT; if (argc < 3) { print_usage(); exit(EXIT_FAILURE); } act = argv[2]; i = 3; } else { com.type = COM_CLIENT; act = argv[1]; i = 2; } switch (com.type) { case COM_DAEMON: break; case COM_CLIENT: if (!strcmp(act, "status")) com.action = ACT_STATUS; else if (!strcmp(act, "host_status")) com.action = ACT_HOST_STATUS; else if (!strcmp(act, "renewal")) com.action = ACT_RENEWAL; else if (!strcmp(act, "gets")) com.action = ACT_GETS; else if (!strcmp(act, "log_dump")) com.action = ACT_LOG_DUMP; else if (!strcmp(act, "shutdown")) com.action = ACT_SHUTDOWN; else if (!strcmp(act, "add_lockspace")) com.action = ACT_ADD_LOCKSPACE; else if (!strcmp(act, "inq_lockspace")) com.action = ACT_INQ_LOCKSPACE; else if (!strcmp(act, "rem_lockspace")) com.action = ACT_REM_LOCKSPACE; else if (!strcmp(act, "command")) com.action = ACT_COMMAND; else if (!strcmp(act, "spawn")) com.action = ACT_SPAWN; else if (!strcmp(act, "acquire")) com.action = ACT_ACQUIRE; else if (!strcmp(act, "convert")) com.action = ACT_CONVERT; else if (!strcmp(act, "release")) com.action = ACT_RELEASE; else if (!strcmp(act, "inquire")) com.action = ACT_INQUIRE; else if (!strcmp(act, "request")) com.action = ACT_REQUEST; else if (!strcmp(act, "examine")) com.action = ACT_EXAMINE; else if (!strcmp(act, "align")) com.action = ACT_CLIENT_ALIGN; else if (!strcmp(act, "init_host")) com.action = ACT_CLIENT_INIT_HOST; else if (!strcmp(act, "init")) com.action = ACT_CLIENT_INIT; else if (!strcmp(act, "write")) com.action = ACT_CLIENT_INIT; else if (!strcmp(act, "read")) com.action = ACT_CLIENT_READ; else if (!strcmp(act, "version")) com.action = ACT_VERSION; else if (!strcmp(act, "set_event")) com.action = ACT_SET_EVENT; else if (!strcmp(act, "set_config")) com.action = ACT_SET_CONFIG; else if (!strcmp(act, "set_host")) com.action = ACT_SET_HOST; else if (!strcmp(act, "format")) { com.action = ACT_FORMAT; com.rindex_op = RX_OP_FORMAT; } else if (!strcmp(act, "rebuild")) { com.action = ACT_REBUILD; com.rindex_op = RX_OP_REBUILD; } else if (!strcmp(act, "create")) { com.action = ACT_CREATE; com.rindex_op = RX_OP_CREATE; } else if (!strcmp(act, "delete")) { com.action = ACT_DELETE; com.rindex_op = RX_OP_DELETE; } else if (!strcmp(act, "lookup")) { com.action = ACT_LOOKUP; com.rindex_op = RX_OP_LOOKUP; } else if (!strcmp(act, "update")) { com.action = ACT_UPDATE; com.rindex_op = RX_OP_UPDATE; } else { log_tool("client action \"%s\" is unknown", act); exit(EXIT_FAILURE); } break; case COM_DIRECT: if (!strcmp(act, "init_host")) com.action = ACT_DIRECT_INIT_HOST; else if (!strcmp(act, "init")) com.action = ACT_DIRECT_INIT; else if (!strcmp(act, "dump")) com.action = ACT_DUMP; else if (!strcmp(act, "next_free")) com.action = ACT_NEXT_FREE; else if (!strcmp(act, "read_leader")) com.action = ACT_READ_LEADER; else if (!strcmp(act, "write_leader")) com.action = ACT_WRITE_LEADER; else if (!strcmp(act, "read")) com.action = ACT_READ; else if (!strcmp(act, "acquire")) com.action = ACT_ACQUIRE; else if (!strcmp(act, "release")) com.action = ACT_RELEASE; else if (!strcmp(act, "acquire_id")) com.action = ACT_ACQUIRE_ID; else if (!strcmp(act, "release_id")) com.action = ACT_RELEASE_ID; else if (!strcmp(act, "renew_id")) com.action = ACT_RENEW_ID; else if (!strcmp(act, "format")) { com.action = ACT_FORMAT; com.rindex_op = RX_OP_FORMAT; } else if (!strcmp(act, "rebuild")) { com.action = ACT_REBUILD; com.rindex_op = RX_OP_REBUILD; } else if (!strcmp(act, "lookup")) { com.action = ACT_LOOKUP; com.rindex_op = RX_OP_LOOKUP; } else if (!strcmp(act, "update")) { com.action = ACT_UPDATE; com.rindex_op = RX_OP_UPDATE; } else { log_tool("direct action \"%s\" is unknown", act); exit(EXIT_FAILURE); } break; }; /* actions that have an option without dash-letter prefix */ if (com.action == ACT_DUMP || com.action == ACT_NEXT_FREE) { if (argc < 4) exit(EXIT_FAILURE); optionarg = argv[i++]; com.dump_path = strdup(optionarg); } for (; i < argc; ) { p = argv[i]; if ((p[0] != '-') || (strlen(p) != 2)) { log_tool("unknown option %s", p); log_tool("space required before option value"); exit(EXIT_FAILURE); } optchar = p[1]; i++; /* the only option that does not have optionarg */ if (optchar == 'D') { com.debug = 1; log_stderr_priority = LOG_DEBUG; continue; } if (i >= argc) { log_tool("option '%c' requires arg", optchar); exit(EXIT_FAILURE); } optionarg = argv[i]; switch (optchar) { case 'Q': com.quiet_fail = atoi(optionarg); break; case 'R': com.debug_renew = atoi(optionarg); break; case 'H': com.renewal_history_size = atoi(optionarg); break; case 'L': log_logfile_priority = atoi(optionarg); break; case 'S': log_syslog_priority = atoi(optionarg); break; case 'F': if (com.action == ACT_SET_HOST) strncpy(com.flag_name, optionarg, SANLK_NAME_LEN); else com.file_path = strdup(optionarg); break; case 'a': com.all = atoi(optionarg); com.aio_arg = atoi(optionarg); if (com.aio_arg && com.aio_arg != 1) com.aio_arg = 1; break; case 't': if (com.action == ACT_DIRECT_INIT_HOST) com.timestamp = strtoull(optionarg, NULL, 0); else { com.max_worker_threads = atoi(optionarg); if (com.max_worker_threads < DEFAULT_MIN_WORKER_THREADS) com.max_worker_threads = DEFAULT_MIN_WORKER_THREADS; } break; case 'w': com.use_watchdog = atoi(optionarg); com.wait = atoi(optionarg); break; case 'h': if (com.action == ACT_GETS || com.action == ACT_CLIENT_READ || com.action == ACT_COMMAND || com.action == ACT_SPAWN) com.get_hosts = atoi(optionarg); else com.high_priority = atoi(optionarg); break; case 'l': com.mlock_level = atoi(optionarg); break; case 'o': if (com.action == ACT_STATUS) { com.sort_arg = *optionarg; } else { val = atoi(optionarg); if (val > 0) { com.io_timeout = val; com.io_timeout_set = 1; } } break; case 'N': com.no_timeout = atoi(optionarg); break; case 'b': com.set_bitmap_seconds = atoi(optionarg); break; case 'n': com.num_hosts = atoi(optionarg); break; case 'm': com.max_hosts = atoi(optionarg); break; case 'p': com.pid = atoi(optionarg); break; case 'C': val = atoi(optionarg); com.cid = val; if (val == 1) com.compare_and_write_opt1 = 1; if (val == 0) com.compare_and_write_opt0 = 1; break; case 'd': if (com.action == ACT_SET_EVENT) { com.he_data = strtoull(optionarg, NULL, 0); } else { com.no_disk = atoi(optionarg); } break; case 'e': if (com.rindex_op) { parse_arg_rentry(optionarg); } else { strncpy(com.our_host_name_opt, optionarg, sizeof(com.our_host_name_opt)-1); com.he_event = strtoull(optionarg, NULL, 0); } break; case 'i': com.host_id = strtoull(optionarg, NULL, 0); break; case 'g': if (com.type == COM_DAEMON) { sec = atoi(optionarg); if (sec <= 60 && sec >= 0) { com.kill_grace_seconds = sec; com.kill_grace_set = 1; } } else { com.host_generation = strtoull(optionarg, NULL, 0); } break; case 'f': com.force_mode = strtoul(optionarg, NULL, 0); break; case 's': parse_arg_lockspace(optionarg); /* com.lockspace */ break; case 'r': parse_arg_resource(optionarg); /* com.res_args[] */ break; case 'U': com.uname = optionarg; com.uid = user_to_uid(optionarg); break; case 'G': com.gname = optionarg; com.gid = group_to_gid(optionarg); break; case 'O': com.orphan_set = 1; com.orphan = atoi(optionarg); break; case 'P': com.persistent = atoi(optionarg); break; case 'u': com.used_set = 1; com.used = atoi(optionarg); break; case 'x': parse_arg_rindex(optionarg); break; case 'z': com.clear_arg = 1; break; case 'c': if (com.action == ACT_SPAWN) { /* * repeated -c COUNT CMD [ARG...] where COUNT includes the * CMD itself. This lets the parser consume arguments with no * ambiguity regardless of their content. */ do { i = read_spawn_command(i, argc, argv); if (i < 0) return i; } while (i < argc && !strcmp(argv[i], "-c") && ++i); /* i already points at the next option, so continue the loop */ continue; } else { begin_command = 1; } break; case 'A': if (!strcmp(optionarg, "1M")) com.align_size = ALIGN_SIZE_1M; else if (!strcmp(optionarg, "2M")) com.align_size = ALIGN_SIZE_2M; else if (!strcmp(optionarg, "4M")) com.align_size = ALIGN_SIZE_4M; else if (!strcmp(optionarg, "8M")) com.align_size = ALIGN_SIZE_8M; break; case 'Z': com.sector_size = atoi(optionarg); break; default: log_tool("unknown option: %c", optchar); exit(EXIT_FAILURE); }; if (begin_command) break; i++; } if (!com.sector_size && !com.align_size) { } else if (com.sector_size && !com.align_size) { } else if (((com.sector_size == 512) && (com.align_size == ALIGN_SIZE_1M)) || ((com.sector_size == 4096) && (com.align_size == ALIGN_SIZE_1M)) || ((com.sector_size == 4096) && (com.align_size == ALIGN_SIZE_2M)) || ((com.sector_size == 4096) && (com.align_size == ALIGN_SIZE_4M)) || ((com.sector_size == 4096) && (com.align_size == ALIGN_SIZE_8M))) { } else { log_tool("Invalid sector_size/align_size combination (%d/%d)", com.sector_size, com.align_size); log_tool("Use one of: 512/1M, 4096/1M, 4096/2M, 4096/4M, 4096/8M."); return -EINVAL; } /* * the remaining args are for the command * * sanlock -r foo -n 2 -d bar:0 -c /bin/cmd -X -Y -Z * argc = 12 * loop above breaks with i = 8, argv[8] = "/bin/cmd" * * cmd_argc = 4 = argc (12) - i (8) * cmd_argv[0] = "/bin/cmd" * cmd_argv[1] = "-X" * cmd_argv[2] = "-Y" * cmd_argv[3] = "-Z" * cmd_argv[4] = NULL (required by execv) */ if (begin_command) { cmd_argc = argc - i; if (cmd_argc < 1) { log_tool("command option (-c) requires an arg"); return -EINVAL; } len = (cmd_argc + 1) * sizeof(char *); /* +1 for final NULL */ cmd_argv = malloc(len); if (!cmd_argv) return -ENOMEM; memset(cmd_argv, 0, len); for (j = 0; j < cmd_argc; j++) { cmd_argv[j] = strdup(argv[i++]); if (!cmd_argv[j]) return -ENOMEM; } strncpy(command, cmd_argv[0], COMMAND_MAX - 1); } return 0; } const char *cmd_num_to_str(int cmd) { switch (cmd) { case SM_CMD_INQ_LOCKSPACE: return "inq_lockspace"; case SM_CMD_READ_RESOURCE_OWNERS: return "read_resource_owners"; case SM_CMD_GET_LOCKSPACES: return "get_lockspaces"; case SM_CMD_GET_HOSTS: return "get_hosts"; case SM_CMD_REGISTER: return "register"; case SM_CMD_ADD_LOCKSPACE: return "add_lockspace"; case SM_CMD_REM_LOCKSPACE: return "rem_lockspace"; case SM_CMD_SHUTDOWN: return "shutdown"; case SM_CMD_STATUS: return "status"; case SM_CMD_ACQUIRE: return "acquire"; case SM_CMD_ACQUIRE2: return "acquire2"; case SM_CMD_RELEASE: return "release"; case SM_CMD_INQUIRE: return "inquire"; case SM_CMD_RESTRICT: return "restrict"; case SM_CMD_REQUEST: return "request"; case SM_CMD_ALIGN: return "align"; case SM_CMD_EXAMINE_LOCKSPACE: return "examine_lockspace"; case SM_CMD_EXAMINE_RESOURCE: return "examine_resource"; case SM_CMD_HOST_STATUS: return "host_status"; case SM_CMD_KILLPATH: return "killpath"; case SM_CMD_WRITE_LOCKSPACE: return "write_lockspace"; case SM_CMD_WRITE_RESOURCE: return "write_resource"; case SM_CMD_READ_LOCKSPACE: return "read_lockspace"; case SM_CMD_READ_LOCKSPACE_HOST: return "read_lockspace_host"; case SM_CMD_READ_RESOURCE: return "read_resource"; case SM_CMD_SET_LVB: return "set_lvb"; case SM_CMD_GET_LVB: return "get_lvb"; case SM_CMD_CONVERT: return "convert"; case SM_CMD_VERSION: return "version"; case SM_CMD_SHUTDOWN_WAIT: return "shutdown_wait"; case SM_CMD_REG_EVENT: return "reg_event"; case SM_CMD_END_EVENT: return "end_event"; case SM_CMD_SET_EVENT: return "set_event"; case SM_CMD_SET_CONFIG: return "set_config"; case SM_CMD_SET_HOST: return "set_host"; case SM_CMD_RENEWAL: return "renewal"; case SM_CMD_FORMAT_RINDEX: return "format_rindex"; case SM_CMD_UPDATE_RINDEX: return "update_rindex"; case SM_CMD_LOOKUP_RINDEX: return "lookup_rindex"; case SM_CMD_CREATE_RESOURCE: return "create_resource"; case SM_CMD_DELETE_RESOURCE: return "delete_resource"; case SM_CMD_REBUILD_RINDEX: return "rebuild_rindex"; case SM_CMD_LOG_DUMP: return "log_dump"; default: return "unknown"; } } uint32_t cmd_str_to_num(const char *str) { if (!strcmp(str, "inq_lockspace")) return SM_CMD_INQ_LOCKSPACE; if (!strcmp(str, "read_resource_owners")) return SM_CMD_READ_RESOURCE_OWNERS; if (!strcmp(str, "get_lockspaces")) return SM_CMD_GET_LOCKSPACES; if (!strcmp(str, "get_hosts")) return SM_CMD_GET_HOSTS; if (!strcmp(str, "register")) return SM_CMD_REGISTER; if (!strcmp(str, "add_lockspace")) return SM_CMD_ADD_LOCKSPACE; if (!strcmp(str, "rem_lockspace")) return SM_CMD_REM_LOCKSPACE; if (!strcmp(str, "shutdown")) return SM_CMD_SHUTDOWN; if (!strcmp(str, "status")) return SM_CMD_STATUS; if (!strcmp(str, "acquire")) return SM_CMD_ACQUIRE; if (!strcmp(str, "acquire2")) return SM_CMD_ACQUIRE2; if (!strcmp(str, "release")) return SM_CMD_RELEASE; if (!strcmp(str, "inquire")) return SM_CMD_INQUIRE; if (!strcmp(str, "restrict")) return SM_CMD_RESTRICT; if (!strcmp(str, "request")) return SM_CMD_REQUEST; if (!strcmp(str, "align")) return SM_CMD_ALIGN; if (!strcmp(str, "examine_lockspace")) return SM_CMD_EXAMINE_LOCKSPACE; if (!strcmp(str, "examine_resource")) return SM_CMD_EXAMINE_RESOURCE; if (!strcmp(str, "host_status")) return SM_CMD_HOST_STATUS; if (!strcmp(str, "killpath")) return SM_CMD_KILLPATH; if (!strcmp(str, "write_lockspace")) return SM_CMD_WRITE_LOCKSPACE; if (!strcmp(str, "write_resource")) return SM_CMD_WRITE_RESOURCE; if (!strcmp(str, "read_lockspace")) return SM_CMD_READ_LOCKSPACE; if (!strcmp(str, "read_lockspace_host")) return SM_CMD_READ_LOCKSPACE_HOST; if (!strcmp(str, "read_resource")) return SM_CMD_READ_RESOURCE; if (!strcmp(str, "set_lvb")) return SM_CMD_SET_LVB; if (!strcmp(str, "get_lvb")) return SM_CMD_GET_LVB; if (!strcmp(str, "convert")) return SM_CMD_CONVERT; if (!strcmp(str, "version")) return SM_CMD_VERSION; if (!strcmp(str, "shutdown_wait")) return SM_CMD_SHUTDOWN_WAIT; if (!strcmp(str, "reg_event")) return SM_CMD_REG_EVENT; if (!strcmp(str, "end_event")) return SM_CMD_END_EVENT; if (!strcmp(str, "set_event")) return SM_CMD_SET_EVENT; if (!strcmp(str, "set_config")) return SM_CMD_SET_CONFIG; if (!strcmp(str, "set_host")) return SM_CMD_SET_HOST; if (!strcmp(str, "renewal")) return SM_CMD_RENEWAL; if (!strcmp(str, "format_rindex")) return SM_CMD_FORMAT_RINDEX; if (!strcmp(str, "update_rindex")) return SM_CMD_UPDATE_RINDEX; if (!strcmp(str, "lookup_rindex")) return SM_CMD_LOOKUP_RINDEX; if (!strcmp(str, "create_resource")) return SM_CMD_CREATE_RESOURCE; if (!strcmp(str, "delete_resource")) return SM_CMD_DELETE_RESOURCE; if (!strcmp(str, "rebuild_rindex")) return SM_CMD_REBUILD_RINDEX; if (!strcmp(str, "log_dump")) return SM_CMD_LOG_DUMP; log_debug("unknown cmd string %.16s", str); return 0; } uint64_t cmd_num_to_debug_flag(uint32_t cmd) { return ((uint64_t)1 << cmd); } int is_cmd_debug(uint32_t cmd) { uint64_t flag = cmd_num_to_debug_flag(cmd); if (com.debug_cmds & flag) return 1; return 0; } void set_cmd_debug(uint32_t cmd) { uint64_t flag = cmd_num_to_debug_flag(cmd); com.debug_cmds |= flag; } void clear_cmd_debug(uint32_t cmd) { uint64_t flag = cmd_num_to_debug_flag(cmd); com.debug_cmds &= ~flag; } #define MAX_CONF_LINE 128 static void get_val_int(char *line, int *val_out) { char key[MAX_CONF_LINE]; char val[MAX_CONF_LINE]; int rv; rv = sscanf(line, "%[^=]=%s", key, val); if (rv != 2) return; *val_out = atoi(val); } static void get_val_str(char *line, char *val_out) { char key[MAX_CONF_LINE]; char val[MAX_CONF_LINE]; int rv; rv = sscanf(line, "%[^=]=%s", key, val); if (rv != 2) return; strcpy(val_out, val); } static void read_config_file(void) { FILE *file; struct stat buf; char line[MAX_CONF_LINE]; char str[MAX_CONF_LINE]; uint32_t cmd; int i, val; if (stat(SANLK_CONF_PATH, &buf) < 0) { if (errno != ENOENT) log_error("%s stat failed: %d", SANLK_CONF_PATH, errno); return; } file = fopen(SANLK_CONF_PATH, "r"); if (!file) return; while (fgets(line, MAX_CONF_LINE, file)) { if (line[0] == '#') continue; if (line[0] == '\n') continue; memset(str, 0, sizeof(str)); for (i = 0; i < MAX_CONF_LINE; i++) { if (line[i] == ' ') break; if (line[i] == '=') break; if (line[i] == '\0') break; if (line[i] == '\n') break; if (line[i] == '\t') break; str[i] = line[i]; } if (!strcmp(str, "quiet_fail")) { get_val_int(line, &val); com.quiet_fail = val; } else if (!strcmp(str, "debug_renew")) { get_val_int(line, &val); com.debug_renew = val; } else if (!strcmp(str, "debug_hosts")) { get_val_int(line, &val); com.debug_hosts = val; } else if (!strcmp(str, "use_aio")) { get_val_int(line, &val); com.aio_arg = val; } else if (!strcmp(str, "logfile_priority")) { get_val_int(line, &val); log_logfile_priority = val; } else if (!strcmp(str, "logfile_use_utc")) { get_val_int(line, &val); log_logfile_use_utc = val; } else if (!strcmp(str, "syslog_priority")) { get_val_int(line, &val); log_syslog_priority = val; } else if (!strcmp(str, "names_log_priority")) { get_val_int(line, &val); com.names_log_priority = val; } else if (!strcmp(str, "use_watchdog")) { get_val_int(line, &val); com.use_watchdog = val; } else if (!strcmp(str, "io_timeout")) { get_val_int(line, &val); if (val > 0) com.io_timeout = val; } else if (!strcmp(str, "watchdog_fire_timeout")) { get_val_int(line, &val); if (val > 0) com.watchdog_fire_timeout = val; } else if (!strcmp(str, "kill_grace_seconds")) { get_val_int(line, &val); if (val <= 60 && val >= 0) { com.kill_grace_seconds = val; com.kill_grace_set = 1; } } else if (!strcmp(str, "high_priority")) { get_val_int(line, &val); com.high_priority = val; } else if (!strcmp(str, "mlock_level")) { get_val_int(line, &val); com.mlock_level = val; } else if (!strcmp(str, "sh_retries")) { get_val_int(line, &val); com.sh_retries = val; } else if (!strcmp(str, "uname")) { memset(str, 0, sizeof(str)); get_val_str(line, str); com.uname = strdup(str); com.uid = user_to_uid(str); } else if (!strcmp(str, "gname")) { memset(str, 0, sizeof(str)); get_val_str(line, str); com.gname = strdup(str); com.gid = group_to_gid(str); } else if (!strcmp(str, "our_host_name")) { memset(str, 0, sizeof(str)); get_val_str(line, str); strncpy(com.our_host_name_opt, str, sizeof(com.our_host_name_opt)-1); } else if (!strcmp(str, "renewal_read_extend_sec")) { /* zero is a valid setting so we need the _set field to say it's set */ get_val_int(line, &val); com.renewal_read_extend_sec_set = 1; com.renewal_read_extend_sec = val; } else if (!strcmp(str, "write_init_io_timeout")) { get_val_int(line, &val); if (val > 0) com.write_init_io_timeout = val; } else if (!strcmp(str, "renewal_history_size")) { get_val_int(line, &val); com.renewal_history_size = val; } else if (!strcmp(str, "paxos_debug_all")) { get_val_int(line, &val); com.paxos_debug_all = val; } else if (!strcmp(str, "debug_io")) { memset(str, 0, sizeof(str)); get_val_str(line, str); if (strstr(str, "submit")) com.debug_io_submit = 1; if (strstr(str, "complete")) com.debug_io_complete = 1; } else if (!strcmp(str, "debug_clients")) { get_val_int(line, &val); com.debug_clients = val; } else if (!strcmp(str, "debug_cmd")) { get_val_str(line, str); if (!strcmp(str, "+all")) com.debug_cmds = ~0LL; else if (!strcmp(str, "-all")) com.debug_cmds = 0LL; else { cmd = cmd_str_to_num(str+1); if (cmd && (str[0] == '+')) set_cmd_debug(cmd); else if (cmd && (str[0] == '-')) clear_cmd_debug(cmd); } } else if (!strcmp(str, "max_sectors_kb")) { memset(str, 0, sizeof(str)); get_val_str(line, str); if (strstr(str, "ignore")) { com.max_sectors_kb_ignore = 1; com.max_sectors_kb_align = 0; com.max_sectors_kb_num = 0; } else if (strstr(str, "align")) { com.max_sectors_kb_ignore = 0; com.max_sectors_kb_align = 1; com.max_sectors_kb_num = 0; } else if (isdigit(str[0])) { int num = atoi(str); if (!num || (num % 2) || (num > 8192)) { log_error("ignore invalid num max_sectors_kb %s", str); } else { com.max_sectors_kb_ignore = 0; com.max_sectors_kb_align = 0; com.max_sectors_kb_num = num; } } else { log_error("ignore unknown max_sectors_kb %s", str); } } else if (!strcmp(str, "max_worker_threads")) { get_val_int(line, &val); if (val < DEFAULT_MIN_WORKER_THREADS) val = DEFAULT_MIN_WORKER_THREADS; com.max_worker_threads = val; } else if (!strcmp(str, "use_hugepages")) { memset(str, 0, sizeof(str)); get_val_str(line, str); if (!strcmp(str, "none")) com.use_hugepages = 0; else if (!strcmp(str, "all")) com.use_hugepages = 1; } else if (!strcmp(str, "create_old_delta_disk_version")) { memset(str, 0, sizeof(str)); get_val_str(line, str); if (!strcmp(str, "3.4")) com.create_delta_34 = 1; else if (!strcmp(str, "4.0")) com.create_delta_40 = 1; else log_error("ignore unknown create_old_delta_disk_version %s", str); } else if (!strcmp(str, "create_old_paxos_disk_version")) { memset(str, 0, sizeof(str)); get_val_str(line, str); if (!strcmp(str, "6.4")) com.create_paxos_64 = 1; else log_error("ignore unknown create_old_paxos_disk_version %s", str); } else if (!strcmp(str, "use_compare_and_write")) { memset(str, 0, sizeof(str)); get_val_str(line, str); if (!strcmp(str, "allow")) com.use_compare_and_write = USE_CAW_ALLOW; else if (!strcmp(str, "yes")) com.use_compare_and_write = USE_CAW_YES; else if (!strcmp(str, "no")) com.use_compare_and_write = USE_CAW_NO; else log_error("ignore unknown use_compare_and_write %s", str); } else if (!strcmp(str, "delta_caw_verify")) { get_val_int(line, &val); com.delta_caw_verify = val; } else if (!strcmp(str, "paxos_caw_verify")) { get_val_int(line, &val); com.delta_caw_verify = val; } else if (!strcmp(str, "caw_dpo")) { get_val_int(line, &val); com.caw_dpo = val; } else if (!strcmp(str, "caw_fua")) { get_val_int(line, &val); com.caw_fua = val; } } fclose(file); } /* only used by do_client */ static char *lsf_to_str(uint32_t flags) { static char lsf_str[32] = {0}; memset(lsf_str, 0, sizeof(lsf_str)); if (flags & SANLK_LSF_ADD) strcat(lsf_str, "ADD "); if (flags & SANLK_LSF_REM) strcat(lsf_str, "REM "); if (flags & SANLK_LSF_NO_TIMEOUT) strcat(lsf_str, "NO_TIMEOUT "); return lsf_str; } const char *host_state_str(uint32_t flags) { int val = flags & SANLK_HOST_MASK; if (val == SANLK_HOST_FREE) return "FREE"; if (val == SANLK_HOST_LIVE) return "LIVE"; if (val == SANLK_HOST_FAIL) return "FAIL"; if (val == SANLK_HOST_DEAD) return "DEAD"; if (val == SANLK_HOST_UNKNOWN) return "UNKNOWN"; return "ERROR"; } static int do_client_gets(void) { struct sanlk_lockspace *lss = NULL, *ls; struct sanlk_host *hss = NULL, *hs; int ls_count = 0, hss_count = 0; int i, j, rv; rv = sanlock_get_lockspaces(&lss, &ls_count, 0); if (rv < 0) log_tool("gets error %d", rv); if (rv < 0 && rv != -ENOSPC) { if (lss) free(lss); return rv; } if (!lss) return 0; ls = lss; for (i = 0; i < ls_count; i++) { log_tool("s %.48s:%llu:%s:%llu %s", ls->name, (unsigned long long)ls->host_id, ls->host_id_disk.path, (unsigned long long)ls->host_id_disk.offset, !ls->flags ? "" : lsf_to_str(ls->flags)); if (!com.get_hosts) goto next; hss = NULL; hss_count = 0; rv = sanlock_get_hosts(ls->name, 0, &hss, &hss_count, 0); if (rv == -EAGAIN) { log_tool("hosts not ready"); goto next; } if (rv < 0) { log_tool("hosts error %d", rv); goto next; } if (!hss) goto next; hs = hss; for (j = 0; j < hss_count; j++) { log_tool("h %llu gen %llu timestamp %llu %s", (unsigned long long)hs->host_id, (unsigned long long)hs->generation, (unsigned long long)hs->timestamp, host_state_str(hs->flags)); hs++; } free(hss); next: ls++; } free(lss); return 0; } static int do_direct_read(void) { struct leader_record leader; char *res_str = NULL; int rv; rv = direct_read(&main_task, com.io_timeout, &com.lockspace, com.res_args[0], &leader); log_tool("read done %d", rv); if (rv < 0) { return rv; } if (com.lockspace.host_id_disk.path[0]) { log_tool("s %.48s:%llu:%s:%llu", com.lockspace.name, (unsigned long long)com.lockspace.host_id, com.lockspace.host_id_disk.path, (unsigned long long)com.lockspace.host_id_disk.offset); } else { rv = sanlock_res_to_str(com.res_args[0], &res_str); if (rv < 0) { log_tool("res_to_str error %d", rv); return rv; } log_tool("r %s", res_str); free(res_str); if (com.get_hosts) { /* TODO: reading the owners of resource will be implemented in the future. */ } } return rv; } static int do_client_read(void) { struct sanlk_host host; struct sanlk_host *hss = NULL, *hs; char *res_str = NULL; uint32_t io_timeout = 0; int rv, i, hss_count = 0; if (com.lockspace.host_id_disk.path[0]) { if (com.sector_size) com.lockspace.flags |= sanlk_lsf_sector_size_to_flag(com.sector_size); if (com.align_size) com.lockspace.flags |= sanlk_lsf_align_size_to_flag(com.align_size); if (!com.get_hosts) rv = sanlock_read_lockspace(&com.lockspace, 0, &io_timeout); else rv = sanlock_read_lockspace_host(&com.lockspace, 0, &io_timeout, &host); } else { if (com.sector_size) com.res_args[0]->flags |= sanlk_res_sector_size_to_flag(com.sector_size); if (com.align_size) com.res_args[0]->flags |= sanlk_res_align_size_to_flag(com.align_size); if (!com.get_hosts) { rv = sanlock_read_resource(com.res_args[0], 0); } else { rv = sanlock_read_resource_owners(com.res_args[0], 0, &hss, &hss_count); } } if (rv < 0) { log_tool("read error %d", rv); goto out; } if (com.lockspace.host_id_disk.path[0]) { log_tool("s %.48s:%llu:%s:%llu", com.lockspace.name, (unsigned long long)com.lockspace.host_id, com.lockspace.host_id_disk.path, (unsigned long long)com.lockspace.host_id_disk.offset); if (com.debug) { log_tool("io_timeout %u", io_timeout); log_tool("sector_size %d", sanlk_lsf_sector_flag_to_size(com.lockspace.flags)); log_tool("align_size %d", sanlk_lsf_align_flag_to_size(com.lockspace.flags)); log_tool("no_timeout %d", (com.lockspace.flags & SANLK_LSF_NO_TIMEOUT) ? 1 :0); } if (com.debug && com.get_hosts) { log_tool("host_id %llu", (unsigned long long)host.host_id); log_tool("generation %llu", (unsigned long long)host.generation); log_tool("timestamp %llu", (unsigned long long)host.timestamp); log_tool("flags 0x%x", host.flags); } goto out; } rv = sanlock_res_to_str(com.res_args[0], &res_str); if (rv < 0) { log_tool("res_to_str error %d", rv); goto out; } log_tool("r %s", res_str); if (com.debug) { log_tool("sector_size %d", sanlk_res_sector_flag_to_size(com.res_args[0]->flags)); log_tool("align_size %d", sanlk_res_align_flag_to_size(com.res_args[0]->flags)); } free(res_str); if (!hss) goto out; hs = hss; for (i = 0; i < hss_count; i++) { if (hs->timestamp) log_tool("h %llu gen %llu timestamp %llu", (unsigned long long)hs->host_id, (unsigned long long)hs->generation, (unsigned long long)hs->timestamp); else log_tool("h %llu gen %llu", (unsigned long long)hs->host_id, (unsigned long long)hs->generation); hs++; } out: if (hss) free(hss); return rv; } static void do_client_version(void) { uint32_t version = 0; uint32_t proto = 0; int rv; rv = sanlock_version(0, &version, &proto); if (rv < 0) { log_tool("daemon version error %d", rv); } log_tool("client version %u.%u.%u (0x%08x)", sanlock_version_major, sanlock_version_minor, sanlock_version_patch, sanlock_version_combined); log_tool("daemon version %u.%u.%u (0x%08x)", (version & 0xFF000000) >> 24, (version & 0x00FF0000) >> 16, (version & 0x0000FF00) >> 8, version); log_tool("client socket protocol %u.%u", (SM_PROTO & 0xFFFF0000) >> 16, (SM_PROTO & 0x0000FFFF)); log_tool("daemon socket protocol %u.%u", (proto & 0xFFFF0000) >> 16, (proto & 0x0000FFFF)); } /* * log_owner - for ACT_COMMAND and ACT_SPAWN, log the owner when failing to acquire * * Only logs when option "-h 1" is used */ static void log_owner(struct sanlk_host *owner, char *owner_name) { if (com.get_hosts && (owner->host_id || owner_name)) { log_tool("owner: host_id %llu generation %llu timestamp %llu state %s name %s", (unsigned long long)owner->host_id, (unsigned long long)owner->generation, (unsigned long long)owner->timestamp, host_state_str(owner->flags), owner_name ?: "none"); } } /* * do_spawn - implementation of "sanlock client spawn" * * Acquires the resource lease, runs a sequence of commands specified via * repeated "-c COUNT CMD [ARG...]" options, then releases the lease explicitly * and synchronously before returning. */ static int do_spawn(void) { struct sanlk_host owner = { 0 }; char *owner_name = NULL; uint32_t flags = 0; pid_t pid; int status, exit_code; int i, fd, rv; if (!com.spawn_count) { log_tool("spawn requires at least one -c command"); return -EINVAL; } log_tool("register"); fd = sanlock_register(); log_tool("register done %d", fd); if (fd < 0) return fd; /* * Do not allow the daemon to use SIGTERM or SIGKILL on this process * for recovery. If the lockspace lease cannot be renewed, the daemon * must fence the node via the watchdog instead. This prevents the * unsafe scenario of a killed supervisor with a still-running writer. */ sanlock_restrict(fd, SANLK_RESTRICT_SIGKILL | SANLK_RESTRICT_SIGTERM); flags |= com.orphan ? SANLK_ACQUIRE_ORPHAN : 0; log_tool("acquire fd %d", fd); if (com.get_hosts) rv = sanlock_acquire2(fd, -1, flags, com.res_args[0], NULL, &owner, &owner_name); else rv = sanlock_acquire(fd, -1, flags, com.res_count, com.res_args, NULL); log_tool("acquire done %d", rv); if (rv < 0) { log_owner(&owner, owner_name); if (owner_name) free(owner_name); close(fd); return rv; } /* * Run each command sequentially under the held lease, stop on failure. */ exit_code = 0; for (i = 0; i < com.spawn_count; i++) { log_tool("spawn: %s", com.spawn_args[i].argv[0]); pid = fork(); if (pid < 0) { log_tool("spawn fork failed: %s", strerror(errno)); exit_code = -errno; break; } else if (pid == 0) { /* child: close the lease fd so it is not inherited */ close(fd); execv(com.spawn_args[i].argv[0], com.spawn_args[i].argv); perror("spawn execv failed"); exit(1); } /* parent: wait for child to complete before continuing */ if (waitpid(pid, &status, 0) < 0) { log_tool("spawn waitpid failed: %s", strerror(errno)); exit_code = -errno; break; } exit_code = WIFEXITED(status) ? WEXITSTATUS(status) : 1; log_tool("spawn done %d", exit_code); if (exit_code != 0) break; } /* * Explicit synchronous release. When sanlock_release() returns the * daemon has already processed the release, unlike ACT_COMMAND which * relies on POLLHUP as an asynchronous method to release the lease. */ log_tool("release"); flags = SANLK_REL_ALL; if (exit_code == 0) { flags |= com.no_disk ? SANLK_REL_NO_DISK : 0; } rv = sanlock_release(fd, -1, flags, 0, NULL); log_tool("release done %d", rv); close(fd); if (rv < 0) { return rv; } return exit_code; } static int do_client(void) { struct sanlk_host_event he; struct sanlk_resource **res_args = NULL; struct sanlk_resource *res; struct sanlk_host owner = { 0 }; struct sanlk_host host = { 0 }; char *owner_name = NULL; char *res_state = NULL; void *config_data = NULL; uint32_t flags = 0; uint32_t config_cmd = 0; uint32_t io_timeout_data = com.io_timeout; int id = -1; int i, fd; int rv = 0; if (com.action == ACT_COMMAND || com.action == ACT_SPAWN || com.action == ACT_ACQUIRE) { for (i = 0; i < com.res_count; i++) { res = com.res_args[i]; if (com.num_hosts) { res->flags |= SANLK_RES_NUM_HOSTS; res->data32 = com.num_hosts; } if (com.persistent) res->flags |= SANLK_RES_PERSISTENT; } } switch (com.action) { case ACT_STATUS: rv = sanlock_status(com.debug, com.sort_arg); break; case ACT_HOST_STATUS: rv = sanlock_host_status(com.debug, com.lockspace.name); break; case ACT_RENEWAL: rv = sanlock_renewal(com.lockspace.name); break; case ACT_GETS: rv = do_client_gets(); break; case ACT_LOG_DUMP: rv = sanlock_log_dump(LOG_DUMP_SIZE); break; case ACT_SHUTDOWN: log_tool("shutdown force %d wait %d", com.force_mode, com.wait); rv = sanlock_shutdown(com.force_mode, com.wait); log_tool("shutdown done %d", rv); break; case ACT_SPAWN: rv = do_spawn(); break; case ACT_COMMAND: log_tool("register"); fd = sanlock_register(); log_tool("register done %d", fd); if (fd < 0) goto out; flags |= com.orphan ? SANLK_ACQUIRE_ORPHAN : 0; log_tool("acquire fd %d", fd); if (com.get_hosts) rv = sanlock_acquire2(fd, -1, flags, com.res_args[0], NULL, &owner, &owner_name); else rv = sanlock_acquire(fd, -1, flags, com.res_count, com.res_args, NULL); log_tool("acquire done %d", rv); if (rv < 0) { log_owner(&owner, owner_name); if (owner_name) free(owner_name); goto out; } if (!command[0]) { while (1) sleep(10); } execv(command, cmd_argv); perror("execv failed"); /* release happens automatically when pid exits and daemon detects POLLHUP on registered connection */ break; case ACT_ADD_LOCKSPACE: if (com.io_timeout != DEFAULT_IO_TIMEOUT) { log_tool("add_lockspace_timeout %d", com.io_timeout); rv = sanlock_add_lockspace_timeout(&com.lockspace, 0, com.io_timeout); log_tool("add_lockspace_timeout done %d", rv); } else { log_tool("add_lockspace"); rv = sanlock_add_lockspace(&com.lockspace, 0); log_tool("add_lockspace done %d", rv); } break; case ACT_CLIENT_INIT_HOST: log_tool("init_host"); if (com.sector_size) com.lockspace.flags |= sanlk_lsf_sector_size_to_flag(com.sector_size); if (com.align_size) com.lockspace.flags |= sanlk_lsf_align_size_to_flag(com.align_size); if (com.no_timeout) com.lockspace.flags |= SANLK_LSF_NO_TIMEOUT; rv = sanlock_init_lockspace_host(&com.lockspace, com.our_host_name_opt[0] ? com.our_host_name_opt : NULL, com.host_generation, com.timestamp, 0, com.io_timeout); log_tool("init_host done %d", rv); break; case ACT_INQ_LOCKSPACE: log_tool("inq_lockspace"); rv = sanlock_inq_lockspace(&com.lockspace, 0); log_tool("inq_lockspace done %d", rv); break; case ACT_REM_LOCKSPACE: log_tool("rem_lockspace"); rv = sanlock_rem_lockspace(&com.lockspace, 0); log_tool("rem_lockspace done %d", rv); break; case ACT_ACQUIRE: if (com.pid > -1) { log_tool("acquire pid %d", com.pid); id = com.pid; } else if (com.cid > -1) { log_tool("acquire client_id %d", com.cid); id = com.cid; flags |= SANLK_FOR_CLIENT_ID; } flags |= com.orphan ? SANLK_ACQUIRE_ORPHAN : 0; rv = sanlock_acquire(-1, id, flags, com.res_count, com.res_args, NULL); log_tool("acquire done %d", rv); break; case ACT_CONVERT: if (com.pid > -1) { log_tool("convert pid %d", com.pid); id = com.pid; } else if (com.cid > -1) { log_tool("convert client_id %d", com.cid); id = com.cid; flags |= SANLK_FOR_CLIENT_ID; } rv = sanlock_convert(-1, id, flags, com.res_args[0]); log_tool("convert done %d", rv); break; case ACT_RELEASE: if (com.pid > -1) { log_tool("release pid %d", com.pid); id = com.pid; } else if (com.cid > -1) { log_tool("release client_id %d", com.cid); id = com.cid; flags |= SANLK_FOR_CLIENT_ID; } /* * Odd case to specify: release all orphan resources for the named lockspace. * Uses -s lockspace_name instead of using -r, but the function takes a * struct resource, so we take the lockspace arg and copy the name into * a resource struct. When releasing one named orphan resource, the * usual -r lockspace_name:resource_name arg is used. */ if (com.orphan && !com.res_count && com.lockspace.name[0]) { struct sanlk_resource *res_ls = malloc(sizeof(struct sanlk_resource)); if (!res_ls) break; memset(res_ls, 0, sizeof(struct sanlk_resource)); memcpy(res_ls->lockspace_name, com.lockspace.name, SANLK_NAME_LEN); com.res_args[0] = res_ls; com.res_count = 1; } flags |= com.orphan ? SANLK_REL_ORPHAN : 0; flags |= com.all ? SANLK_REL_ALL: 0; flags |= com.no_disk ? SANLK_REL_NO_DISK : 0; rv = sanlock_release(-1, id, flags, com.res_count, com.res_args); log_tool("release done %d", rv); break; case ACT_INQUIRE: if (com.pid > -1) { log_tool("inquire pid %d", com.pid); id = com.pid; } else if (com.cid > -1) { log_tool("inquire client_id %d", com.cid); id = com.cid; flags |= SANLK_FOR_CLIENT_ID; } rv = sanlock_inquire(-1, id, flags, &com.res_count, &res_state); log_tool("inquire done %d res_count %d", rv, com.res_count); if (rv < 0) break; log_tool("\"%s\"", res_state); if (!com.debug) break; com.res_count = 0; rv = sanlock_state_to_args(res_state, &com.res_count, &res_args); log_tool("\nstate_to_args done %d res_count %d", rv, com.res_count); if (rv < 0) break; free(res_state); res_state = NULL; for (i = 0; i < com.res_count; i++) { res = res_args[i]; log_tool("\"%s:%s:%s:%llu:%llu\"", res->lockspace_name, res->name, res->disks[0].path, (unsigned long long)res->disks[0].offset, (unsigned long long)res->lver); } rv = sanlock_args_to_state(com.res_count, res_args, &res_state); log_tool("\nargs_to_state done %d", rv); if (rv < 0) break; log_tool("\"%s\"", res_state); break; case ACT_REQUEST: log_tool("request"); rv = sanlock_request(0, com.force_mode, com.res_args[0]); log_tool("request done %d", rv); break; case ACT_EXAMINE: log_tool("examine"); if (com.lockspace.host_id_disk.path[0]) rv = sanlock_examine(0, &com.lockspace, NULL); else rv = sanlock_examine(0, NULL, com.res_args[0]); log_tool("examine done %d", rv); break; case ACT_CLIENT_ALIGN: log_tool("align"); rv = sanlock_align(&com.lockspace.host_id_disk); log_tool("align done %d", rv); break; case ACT_CLIENT_INIT: log_tool("init"); if (com.lockspace.host_id_disk.path[0]) { if (com.sector_size) com.lockspace.flags |= sanlk_lsf_sector_size_to_flag(com.sector_size); if (com.align_size) com.lockspace.flags |= sanlk_lsf_align_size_to_flag(com.align_size); if (com.no_timeout) com.lockspace.flags |= SANLK_LSF_NO_TIMEOUT; if (com.compare_and_write_opt1) flags = SANLK_WRITE_LS_USE_CAW; else if (com.compare_and_write_opt0) flags = SANLK_WRITE_LS_NO_CAW; rv = sanlock_write_lockspace(&com.lockspace, com.max_hosts, flags, com.io_timeout); } else { if (com.sector_size) com.res_args[0]->flags |= sanlk_res_sector_size_to_flag(com.sector_size); if (com.align_size) com.res_args[0]->flags |= sanlk_res_align_size_to_flag(com.align_size); if (com.compare_and_write_opt1) flags = SANLK_WRITE_USE_CAW; else if (com.compare_and_write_opt0) flags = SANLK_WRITE_NO_CAW; if (com.clear_arg) flags |= SANLK_WRITE_CLEAR; rv = sanlock_write_resource(com.res_args[0], com.max_hosts, com.num_hosts, flags); } log_tool("init done %d", rv); break; case ACT_CLIENT_READ: rv = do_client_read(); break; case ACT_VERSION: do_client_version(); break; case ACT_SET_EVENT: log_tool("set_event %llu %llu event 0x%llx data 0x%llx", (unsigned long long)com.host_id, (unsigned long long)com.host_generation, (unsigned long long)com.he_event, (unsigned long long)com.he_data); he.host_id = com.host_id; he.generation = com.host_generation; he.event = com.he_event; he.data = com.he_data; rv = sanlock_set_event(com.lockspace.name, &he, 0); log_tool("set_event done %d", rv); break; case ACT_SET_CONFIG: if (com.orphan_set) { config_cmd = com.orphan ? SANLK_CONFIG_USED_BY_ORPHANS : SANLK_CONFIG_UNUSED_BY_ORPHANS; log_tool("set_config %.48s %s", com.lockspace.name, com.orphan ? "USED_BY_ORPHANS" : "UNUSED_BY_ORPHANS"); } else if (com.used_set) { config_cmd = com.used ? SANLK_CONFIG_USED : SANLK_CONFIG_UNUSED; log_tool("set_config %.48s %s", com.lockspace.name, com.used ? "USED" : "UNUSED"); } else if (com.io_timeout_set) { config_data = &io_timeout_data; config_cmd = SANLK_CONFIG_IO_TIMEOUT; log_tool("set_config %.48s io_timeout %u", com.lockspace.name, io_timeout_data); } rv = sanlock_set_config(com.lockspace.name, 0, config_cmd, config_data); log_tool("set_config done %d", rv); break; case ACT_SET_HOST: if (!strcmp(com.flag_name, "DEAD_EXT")) { log_tool("set_host %llu %llu DEAD_EXT", (unsigned long long)com.host_id, (unsigned long long)com.host_generation); host.host_id = com.host_id; host.generation = com.host_generation; rv = sanlock_set_host(&com.lockspace, SANLK_SET_HOST_DEAD_EXT, 0, com.io_timeout, &host); log_tool("set_host done %d", rv); } else { log_tool("set_host unknown operation"); } break; case ACT_FORMAT: if (com.sector_size) com.rindex.flags |= sanlk_rif_sector_size_to_flag(com.sector_size); if (com.align_size) com.rindex.flags |= sanlk_rif_align_size_to_flag(com.align_size); rv = sanlock_format_rindex(&com.rindex, 0); log_tool("format done %d", rv); break; case ACT_REBUILD: rv = sanlock_rebuild_rindex(&com.rindex, 0); log_tool("rebuild done %d", rv); break; case ACT_CREATE: rv = sanlock_create_resource(&com.rindex, 0, &com.rentry, 0, 0); log_tool("create_resource done %d", rv); if (!rv) log_tool("offset %llu", (unsigned long long)com.rentry.offset); break; case ACT_DELETE: rv = sanlock_delete_resource(&com.rindex, 0, &com.rentry); log_tool("delete_resource done %d", rv); break; case ACT_LOOKUP: rv = sanlock_lookup_rindex(&com.rindex, 0, &com.rentry); log_tool("lookup done %d", rv); if (!rv) log_tool("name %.48s offset %llu", com.rentry.name[0] ? com.rentry.name : "-", (unsigned long long)com.rentry.offset); break; case ACT_UPDATE: rv = sanlock_update_rindex(&com.rindex, com.clear_arg ? SANLK_RXUP_REM : SANLK_RXUP_ADD, &com.rentry); log_tool("update done %d", rv); break; default: log_tool("action not implemented"); rv = -1; } out: return rv; } #define MAX_LINE 128 static int read_file_leader(struct leader_record *leader, int is_ls) { FILE *file; char line[MAX_LINE]; char field[MAX_LINE]; char val[MAX_LINE]; uint32_t checksum = 0; struct leader_record lr; int rv; file = fopen(com.file_path, "r"); if (!file) { log_tool("open error %d %s", errno, com.file_path); return -1; } memcpy(&lr, leader, sizeof(lr)); memset(line, 0, sizeof(line)); while (fgets(line, MAX_LINE, file)) { memset(field, 0, sizeof(field)); memset(val, 0, sizeof(val)); rv = sscanf(line, "%s %s", field, val); if (rv != 2) { log_tool("ignoring empty field \"%s\"", field); continue; } if (!strcmp(field, "magic")) { sscanf(val, "0x%x", &lr.magic); } else if (!strcmp(field, "version")) { sscanf(val, "0x%x", &lr.version); } else if (!strcmp(field, "flags")) { sscanf(val, "0x%x", &lr.flags); } else if (!strcmp(field, "sector_size")) { sscanf(val, "%u", &lr.sector_size); } else if (!strcmp(field, "num_hosts")) { sscanf(val, "%llu", (unsigned long long *)&lr.num_hosts); } else if (!strcmp(field, "max_hosts")) { sscanf(val, "%llu", (unsigned long long *)&lr.max_hosts); } else if (!strcmp(field, "owner_id")) { sscanf(val, "%llu", (unsigned long long *)&lr.owner_id); } else if (!strcmp(field, "owner_generation")) { sscanf(val, "%llu", (unsigned long long *)&lr.owner_generation); } else if (!strcmp(field, "lver")) { sscanf(val, "%llu", (unsigned long long *)&lr.lver); } else if (!strcmp(field, "space_name")) { strncpy(lr.space_name, val, NAME_ID_SIZE); } else if (!strcmp(field, "resource_name")) { strncpy(lr.resource_name, val, NAME_ID_SIZE); } else if (!strcmp(field, "timestamp")) { sscanf(val, "%llu", (unsigned long long *)&lr.timestamp); } else if (!strcmp(field, "checksum")) { sscanf(val, "0x%x", &checksum); log_tool("ignoring checksum, will be recalculated."); } else if (!strcmp(field, "io_timeout")) { sscanf(val, "%hu", &lr.io_timeout); } else if (is_ls && !strcmp(field, "extra1")) { sscanf(val, "%llu", (unsigned long long *)&lr.write_id); } else if (is_ls && !strcmp(field, "extra2")) { sscanf(val, "%llu", (unsigned long long *)&lr.write_generation); } else if (is_ls && !strcmp(field, "extra3")) { sscanf(val, "%llu", (unsigned long long *)&lr.write_timestamp); } else if (!is_ls && !strcmp(field, "write_id")) { sscanf(val, "%llu", (unsigned long long *)&lr.write_id); } else if (!is_ls && !strcmp(field, "write_generation")) { sscanf(val, "%llu", (unsigned long long *)&lr.write_generation); } else if (!is_ls && !strcmp(field, "write_timestamp")) { sscanf(val, "%llu", (unsigned long long *)&lr.write_timestamp); } else { log_tool("ignoring unknown field \"%s\"", field); } memset(line, 0, sizeof(line)); } fclose(file); memcpy(leader, &lr, sizeof(lr)); return 0; } static char *leader_flags_str(uint32_t flags) { static char flags_str[256]; if (!flags) return (char *)""; memset(flags_str, 0, sizeof(flags_str)); strcat(flags_str, " ["); if (flags & LFL_SHORT_HOLD) strcat(flags_str, " SHORT_HOLD"); if (flags & LFL_NO_TIMEOUT) strcat(flags_str, " NO_TIMEOUT"); if (flags & LFL_DEAD_EXT) strcat(flags_str, " DEAD_EXT"); if (flags & LFL_ALIGN_1M) strcat(flags_str, " ALIGN_1M"); if (flags & LFL_ALIGN_2M) strcat(flags_str, " ALIGN_2M"); if (flags & LFL_ALIGN_4M) strcat(flags_str, " ALIGN_4M"); if (flags & LFL_ALIGN_8M) strcat(flags_str, " ALIGN_8M"); if (flags & LFL_CAW) strcat(flags_str, " CAW"); strcat(flags_str, " ]"); return flags_str; } static void print_leader(struct leader_record *leader, int is_ls) { log_tool("magic 0x%0x", leader->magic); log_tool("version 0x%x", leader->version); log_tool("flags 0x%x%s", leader->flags, leader_flags_str(leader->flags)); log_tool("sector_size %u", leader->sector_size); log_tool("num_hosts %llu", (unsigned long long)leader->num_hosts); log_tool("max_hosts %llu", (unsigned long long)leader->max_hosts); log_tool("owner_id %llu", (unsigned long long)leader->owner_id); log_tool("owner_generation %llu", (unsigned long long)leader->owner_generation); log_tool("lver %llu", (unsigned long long)leader->lver); log_tool("space_name %.48s", leader->space_name); log_tool("resource_name %.48s", leader->resource_name); log_tool("timestamp %llu", (unsigned long long)leader->timestamp); log_tool("unused1 0x%llx", (unsigned long long)leader->unused1); log_tool("checksum 0x%0x", leader->checksum); log_tool("io_timeout %u", leader->io_timeout); if (!is_ls) { log_tool("write_id %llu", (unsigned long long)leader->write_id); log_tool("write_generation %llu", (unsigned long long)leader->write_generation); log_tool("write_timestamp %llu", (unsigned long long)leader->write_timestamp); } else { log_tool("extra1 %llu", (unsigned long long)leader->write_id); log_tool("extra2 %llu", (unsigned long long)leader->write_generation); log_tool("extra3 %llu", (unsigned long long)leader->write_timestamp); } } static int do_direct_read_leader(void) { struct leader_record leader; int rv; rv = direct_read_leader(&main_task, com.io_timeout, &com.lockspace, com.res_args[0], &leader); log_tool("read_leader done %d", rv); print_leader(&leader, com.res_args[0] ? 0 : 1); return rv; } /* * read the current leader record from disk, override any values found in * the file, write back the result. */ static int do_direct_write_leader(void) { struct leader_record leader; char *res_str = NULL; int is_ls = com.res_args[0] ? 0 : 1; int rv; memset(&leader, 0, sizeof(leader)); direct_read_leader(&main_task, com.io_timeout, &com.lockspace, com.res_args[0], &leader); rv = read_file_leader(&leader, is_ls); if (rv < 0) return rv; /* make a record in the logs that this has been done */ if (is_ls) { syslog(LOG_WARNING, "write_leader lockspace %.48s:%llu:%s:%llu", com.lockspace.name, (unsigned long long)com.lockspace.host_id, com.lockspace.host_id_disk.path, (unsigned long long)com.lockspace.host_id_disk.offset); } else { rv = sanlock_res_to_str(com.res_args[0], &res_str); if (rv < 0) goto out; syslog(LOG_WARNING, "write_leader resource %s", res_str); } rv = direct_write_leader(&main_task, com.io_timeout, &com.lockspace, com.res_args[0], &leader); out: log_tool("write_leader done %d", rv); if (!rv) print_leader(&leader, is_ls); if (res_str) free(res_str); return rv; } /* * sanlock.conf use_compare_and_write = yes|no|allow * yes: use caw if supported, and api does not disable (opt-out) * allow: use caw if supported, and api enables (opt-in) * no: never use caw (ignore api enable) * * sanlock.conf create_old_delta_disk_version = 3.4 or 4.0 * sanlock.conf create_old_paxos_disk_version = 6.4 * create_delta_34=1, create_delta_40=1, create_paxos_64=1 * tells sanlock to create leases with versions that don't support caw * (to remain compatible with other hosts that have an old sanlock version) */ static int direct_init_lockspace_try_caw(void) { if (com.create_delta_34 || com.create_delta_40) return 0; switch (com.use_compare_and_write) { case USE_CAW_NO: return 0; case USE_CAW_YES: /* command line api can disable via -C 0 */ if (com.compare_and_write_opt0) return 0; return 1; case USE_CAW_ALLOW: /* command line api can enable via -C 1 */ if (com.compare_and_write_opt1) return 1; return 0; }; return 0; } static int direct_init_resource_try_caw(void) { if (com.create_paxos_64) return 0; switch (com.use_compare_and_write) { case USE_CAW_NO: return 0; case USE_CAW_YES: /* command line api can disable via -C 0 */ if (com.compare_and_write_opt0) return 0; return 1; case USE_CAW_ALLOW: /* command line api can enable via -C 1 */ if (com.compare_and_write_opt1) return 1; return 0; }; return 0; } static int do_direct_init(int action) { char *res_str = NULL; int rv = -EINVAL; int try_caw = 0; if (com.lockspace.host_id_disk.path[0]) { if (com.sector_size) com.lockspace.flags |= sanlk_lsf_sector_size_to_flag(com.sector_size); if (com.align_size) com.lockspace.flags |= sanlk_lsf_align_size_to_flag(com.align_size); if (com.no_timeout) com.lockspace.flags |= SANLK_LSF_NO_TIMEOUT; if (action == ACT_DIRECT_INIT_HOST) { if (!com.lockspace.host_id || com.lockspace.host_id > DEFAULT_MAX_HOSTS) goto out; setup_host_name(); syslog(LOG_WARNING, "init_host lockspace %.48s:%llu:%s:%llu 0x%x gen %llu ts %llu %s", com.lockspace.name, (unsigned long long)com.lockspace.host_id, com.lockspace.host_id_disk.path, (unsigned long long)com.lockspace.host_id_disk.offset, com.lockspace.flags, (unsigned long long)com.host_generation, (unsigned long long)com.timestamp, our_host_name_global); try_caw = direct_init_lockspace_try_caw(); rv = direct_write_lockspace_host(&main_task, &com.lockspace, our_host_name_global, com.host_generation, com.timestamp, com.io_timeout, try_caw); } else { syslog(LOG_WARNING, "init lockspace %.48s:%llu:%s:%llu 0x%x", com.lockspace.name, (unsigned long long)com.lockspace.host_id, com.lockspace.host_id_disk.path, (unsigned long long)com.lockspace.host_id_disk.offset, com.lockspace.flags); try_caw = direct_init_lockspace_try_caw(); rv = direct_write_lockspace(&main_task, &com.lockspace, com.io_timeout, try_caw); } } else if (com.res_args[0]) { if (com.sector_size) com.res_args[0]->flags |= sanlk_res_sector_size_to_flag(com.sector_size); if (com.align_size) com.res_args[0]->flags |= sanlk_res_align_size_to_flag(com.align_size); rv = sanlock_res_to_str(com.res_args[0], &res_str); if (rv < 0) { log_tool("res_to_str parsing error %d", rv); goto out; } else { syslog(LOG_WARNING, "init resource %s", res_str); } try_caw = direct_init_resource_try_caw(); rv = direct_write_resource(&main_task, com.res_args[0], com.num_hosts, com.clear_arg, try_caw); } out: log_tool("init done %d", rv); if (res_str) free(res_str); return rv; } static int do_direct(void) { struct leader_record leader; int rv; /* * We want a record of any out-of-band changes to disk in the system * log. If /dev/log is missing, drop the message so it would not be * logged to stderr. */ openlog("sanlock-direct", LOG_PID, LOG_DAEMON); setup_task_aio(&main_task, com.aio_arg, DIRECT_AIO_CB_SIZE); sprintf(main_task.name, "%s", "main_direct"); switch (com.action) { case ACT_DIRECT_INIT: case ACT_DIRECT_INIT_HOST: rv = do_direct_init(com.action); break; case ACT_DUMP: rv = direct_dump(&main_task, com.dump_path, com.force_mode); break; case ACT_NEXT_FREE: rv = direct_next_free(&main_task, com.dump_path); break; case ACT_READ_LEADER: rv = do_direct_read_leader(); break; case ACT_WRITE_LEADER: rv = do_direct_write_leader(); break; case ACT_READ: rv = do_direct_read(); break; #if 0 case ACT_FORMAT: if (com.sector_size) com.rindex.flags |= sanlk_rif_sector_size_to_flag(com.sector_size); if (com.align_size) com.rindex.flags |= sanlk_rif_align_size_to_flag(com.align_size); syslog(LOG_WARNING, "format rindex %.48s:%s:%llu 0x%x", com.rindex.lockspace_name, com.rindex.disk.path, (unsigned long long)com.rindex.disk.offset, com.rindex.flags); rv = direct_rindex_format(&main_task, &com.rindex); log_tool("format done %d", rv); break; case ACT_REBUILD: rv = direct_rindex_rebuild(&main_task, &com.rindex, 0); log_tool("rebuild done %d", rv); break; case ACT_LOOKUP: rv = direct_rindex_lookup(&main_task, &com.rindex, &com.rentry, 0); log_tool("lookup done %d", rv); if (!rv) log_tool("name %.48s offset %llu", com.rentry.name[0] ? com.rentry.name : "-", (unsigned long long)com.rentry.offset); break; case ACT_UPDATE: uint32_t cmd_flags = 0; if (com.clear_arg) cmd_flags |= SANLK_RXUP_REM; else cmd_flags |= SANLK_RXUP_ADD; rv = direct_rindex_update(&main_task, &com.rindex, &com.rentry, cmd_flags); log_tool("update done %d", rv); break; #endif case ACT_ACQUIRE: syslog(LOG_WARNING, "acquire"); rv = direct_acquire(&main_task, com.io_timeout, com.res_args[0], com.num_hosts, com.host_id, com.host_generation, &leader); log_tool("acquire done %d", rv); break; case ACT_RELEASE: syslog(LOG_WARNING, "release"); rv = direct_release(&main_task, com.io_timeout, com.res_args[0], &leader); log_tool("release done %d", rv); break; case ACT_ACQUIRE_ID: syslog(LOG_WARNING, "acquire_id"); setup_host_name(); rv = direct_acquire_id(&main_task, com.io_timeout, &com.lockspace, our_host_name_global); log_tool("acquire_id done %d", rv); break; case ACT_RELEASE_ID: syslog(LOG_WARNING, "release_id"); rv = direct_release_id(&main_task, com.io_timeout, &com.lockspace); log_tool("release_id done %d", rv); break; case ACT_RENEW_ID: syslog(LOG_WARNING, "renew_id"); rv = direct_renew_id(&main_task, com.io_timeout, &com.lockspace); log_tool("rewew_id done %d", rv); break; default: log_tool("direct action %d not known", com.action); rv = -1; } close_task_aio(&main_task); closelog(); return rv; } static void set_sanlock_version(void) { char version_str[64]; char *major_str, *minor_str, *patch_str; char *d1, *d2; strncpy(version_str, VERSION, 64); d1 = strstr(version_str, "."); if (!d1) return; d2 = strstr(d1 + 1, "."); if (!d2) return; major_str = version_str; minor_str = d1 + 1; patch_str = d2 + 1; *d1 = '\0'; *d2 = '\0'; sanlock_version_major = atoi(major_str); sanlock_version_minor = atoi(minor_str); sanlock_version_patch = atoi(patch_str); sanlock_version_build = 0; /* TODO */ sanlock_version_combined = 0; sanlock_version_combined |= sanlock_version_major << 24; sanlock_version_combined |= sanlock_version_minor << 16; sanlock_version_combined |= sanlock_version_patch << 8; sanlock_version_combined |= sanlock_version_build; } int main(int argc, char *argv[]) { int rv; BUILD_BUG_ON(sizeof(struct sanlk_disk) != sizeof(struct sync_disk)); BUILD_BUG_ON(sizeof(struct leader_record) > LEADER_RECORD_MAX); BUILD_BUG_ON(sizeof(struct helper_msg) != SANLK_HELPER_MSG_LEN); /* initialize global EXTERN variables */ set_sanlock_version(); kill_count_max = 100; helper_ci = -1; helper_pid = -1; helper_kill_fd = -1; helper_status_fd = -1; pthread_mutex_init(&spaces_mutex, NULL); INIT_LIST_HEAD(&spaces); INIT_LIST_HEAD(&spaces_rem); INIT_LIST_HEAD(&spaces_add); memset(&com, 0, sizeof(com)); com.use_watchdog = DEFAULT_USE_WATCHDOG; com.watchdog_fire_timeout = DEFAULT_WATCHDOG_FIRE_TIMEOUT; com.kill_grace_seconds = DEFAULT_GRACE_SEC; com.high_priority = DEFAULT_HIGH_PRIORITY; com.mlock_level = DEFAULT_MLOCK_LEVEL; com.names_log_priority = LOG_INFO; com.max_worker_threads = DEFAULT_MAX_WORKER_THREADS; com.io_timeout = DEFAULT_IO_TIMEOUT; com.write_init_io_timeout = DEFAULT_WRITE_INIT_IO_TIMEOUT; com.aio_arg = DEFAULT_USE_AIO; com.pid = -1; com.cid = -1; com.sh_retries = DEFAULT_SH_RETRIES; com.quiet_fail = DEFAULT_QUIET_FAIL; com.renewal_read_extend_sec_set = 0; com.renewal_read_extend_sec = 0; com.renewal_history_size = DEFAULT_RENEWAL_HISTORY_SIZE; com.paxos_debug_all = 0; com.max_sectors_kb_ignore = DEFAULT_MAX_SECTORS_KB_IGNORE; com.max_sectors_kb_align = DEFAULT_MAX_SECTORS_KB_ALIGN; com.max_sectors_kb_num = DEFAULT_MAX_SECTORS_KB_NUM; com.use_hugepages = 1; com.debug_cmds = ~0LL; com.debug_hosts = 1; com.use_compare_and_write = DEFAULT_USE_COMPARE_AND_WRITE; com.paxos_caw_verify = 1; com.delta_caw_verify = 1; com.caw_fua = 1; com.caw_dpo = 1; /* By default disable cmds that often cause too much logging. */ clear_cmd_debug(SM_CMD_INQ_LOCKSPACE); clear_cmd_debug(SM_CMD_GET_LOCKSPACES); clear_cmd_debug(SM_CMD_GET_HOSTS); clear_cmd_debug(SM_CMD_READ_LOCKSPACE); clear_cmd_debug(SM_CMD_READ_LOCKSPACE_HOST); clear_cmd_debug(SM_CMD_READ_RESOURCE); clear_cmd_debug(SM_CMD_READ_RESOURCE_OWNERS); clear_cmd_debug(SM_CMD_WRITE_RESOURCE); if (getgrnam("sanlock") && getpwnam("sanlock")) { com.uname = (char *)"sanlock"; com.gname = (char *)"sanlock"; com.uid = user_to_uid(com.uname); com.gid = group_to_gid(com.uname); } else { com.uname = NULL; com.gname = NULL; com.uid = DEFAULT_SOCKET_UID; com.gid = DEFAULT_SOCKET_GID; } memset(&main_task, 0, sizeof(main_task)); /* * read_config_file() overrides com default settings, * read_command_line() overrides com default settings and * config file settings. */ read_config_file(); rv = read_command_line(argc, argv); if (rv < 0) goto out; /* * compare and write requires using libdm and SGIO ioctl * which both require root. */ if ((com.use_compare_and_write == USE_CAW_YES) || (com.use_compare_and_write == USE_CAW_ALLOW)) { com.uname = NULL; com.gname = NULL; com.uid = DEFAULT_SOCKET_UID; com.gid = DEFAULT_SOCKET_GID; } switch (com.type) { case COM_DAEMON: rv = do_daemon(); break; case COM_CLIENT: rv = do_client(); break; case COM_DIRECT: rv = do_direct(); break; }; out: return rv == 0 ? EXIT_SUCCESS : EXIT_FAILURE; } sanlock/src/mode_block.h000066400000000000000000000010221516326262600155520ustar00rootroot00000000000000/* * Copyright 2012 Red Hat, Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. */ #ifndef __MODE_BLOCK_H__ #define __MODE_BLOCK_H__ #define MBLOCK_OFFSET 128 /* include paxos_dblock plus padding */ #define MBLOCK_SHARED 0x00000001 struct mode_block { uint32_t flags; uint64_t generation; }; #endif sanlock/src/monotime.c000066400000000000000000000014541516326262600153070ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include "monotime.h" uint64_t monotime(void) { struct timespec ts; clock_gettime(CLOCK_MONOTONIC, &ts); return ts.tv_sec; } void ts_diff(struct timespec *begin, struct timespec *end, struct timespec *diff) { if ((end->tv_nsec - begin->tv_nsec) < 0) { diff->tv_sec = end->tv_sec - begin->tv_sec - 1; diff->tv_nsec = end->tv_nsec - begin->tv_nsec + 1000000000; } else { diff->tv_sec = end->tv_sec - begin->tv_sec; diff->tv_nsec = end->tv_nsec - begin->tv_nsec; } } sanlock/src/monotime.h000066400000000000000000000006561516326262600153170ustar00rootroot00000000000000/* * Copyright 2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __MONOTIME_H__ #define __MONOTIME_H__ uint64_t monotime(void); void ts_diff(struct timespec *begin, struct timespec *end, struct timespec *diff); #endif sanlock/src/ondisk.c000066400000000000000000000140531516326262600147460ustar00rootroot00000000000000/* * Copyright 2014 Red Hat, Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. */ #include #include #include #include #include #include "sanlock_internal.h" #include "ondisk.h" /* * "end" variables point to ondisk format (endian converted) structures. */ void magic_in(char *end, uint32_t *magic) { uint32_t magic_end; memcpy(&magic_end, end, sizeof(uint32_t)); *magic = le32_to_cpu(magic_end); } void leader_record_in(struct leader_record *end, struct leader_record *lr) { lr->magic = le32_to_cpu(end->magic); lr->version = le32_to_cpu(end->version); lr->flags = le32_to_cpu(end->flags); lr->sector_size = le32_to_cpu(end->sector_size); lr->num_hosts = le64_to_cpu(end->num_hosts); lr->max_hosts = le64_to_cpu(end->max_hosts); lr->owner_id = le64_to_cpu(end->owner_id); lr->owner_generation = le64_to_cpu(end->owner_generation); lr->lver = le64_to_cpu(end->lver); memcpy(lr->space_name, end->space_name, NAME_ID_SIZE); memcpy(lr->resource_name, end->resource_name, NAME_ID_SIZE); lr->timestamp = le64_to_cpu(end->timestamp); lr->unused1 = le64_to_cpu(end->unused1); lr->checksum = le32_to_cpu(end->checksum); lr->unused2 = le16_to_cpu(end->unused2); lr->io_timeout = le16_to_cpu(end->io_timeout); lr->write_id = le64_to_cpu(end->write_id); lr->write_generation = le64_to_cpu(end->write_generation); lr->write_timestamp = le64_to_cpu(end->write_timestamp); } void leader_record_out(struct leader_record *lr, struct leader_record *end) { end->magic = cpu_to_le32(lr->magic); end->version = cpu_to_le32(lr->version); end->flags = cpu_to_le32(lr->flags); end->sector_size = cpu_to_le32(lr->sector_size); end->num_hosts = cpu_to_le64(lr->num_hosts); end->max_hosts = cpu_to_le64(lr->max_hosts); end->owner_id = cpu_to_le64(lr->owner_id); end->owner_generation = cpu_to_le64(lr->owner_generation); end->lver = cpu_to_le64(lr->lver); memcpy(end->space_name, lr->space_name, NAME_ID_SIZE); memcpy(end->resource_name, lr->resource_name, NAME_ID_SIZE); end->timestamp = cpu_to_le64(lr->timestamp); end->unused1 = cpu_to_le64(lr->unused1); /* N.B. the checksum must be computed after the byte swapping */ /* leader_record_out(lr, end); checksum = compute(end); end->checksum = cpu_to_le32(checksum); */ end->unused2 = cpu_to_le16(lr->unused2); end->io_timeout = cpu_to_le16(lr->io_timeout); end->write_id = cpu_to_le64(lr->write_id); end->write_generation = cpu_to_le64(lr->write_generation); end->write_timestamp = cpu_to_le64(lr->write_timestamp); } void request_record_in(struct request_record *end, struct request_record *rr) { rr->magic = le32_to_cpu(end->magic); rr->version = le32_to_cpu(end->version); rr->lver = le64_to_cpu(end->lver); rr->force_mode = le32_to_cpu(end->force_mode); } void request_record_out(struct request_record *rr, struct request_record *end) { end->magic = cpu_to_le32(rr->magic); end->version = cpu_to_le32(rr->version); end->lver = cpu_to_le64(rr->lver); end->force_mode = cpu_to_le32(rr->force_mode); } void paxos_dblock_in(struct paxos_dblock *end, struct paxos_dblock *pd) { pd->mbal = le64_to_cpu(end->mbal); pd->bal = le64_to_cpu(end->bal); pd->inp = le64_to_cpu(end->inp); pd->inp2 = le64_to_cpu(end->inp2); pd->inp3 = le64_to_cpu(end->inp3); pd->lver = le64_to_cpu(end->lver); pd->checksum = le32_to_cpu(end->checksum); pd->flags = le32_to_cpu(end->flags); } void paxos_dblock_out(struct paxos_dblock *pd, struct paxos_dblock *end) { end->mbal = cpu_to_le64(pd->mbal); end->bal = cpu_to_le64(pd->bal); end->inp = cpu_to_le64(pd->inp); end->inp2 = cpu_to_le64(pd->inp2); end->inp3 = cpu_to_le64(pd->inp3); end->lver = cpu_to_le64(pd->lver); /* N.B. the checksum must be computed after the byte swapping */ /* paxos_dblock_out(pd, end); checksum = compute(end), end->checksum = cpu_to_le32(checksum); */ end->flags = cpu_to_le32(pd->flags); } void mode_block_in(struct mode_block *end, struct mode_block *mb) { mb->flags = le32_to_cpu(end->flags); mb->generation = le64_to_cpu(end->generation); } void mode_block_out(struct mode_block *mb, struct mode_block *end) { end->flags = cpu_to_le32(mb->flags); end->generation = cpu_to_le64(mb->generation); } void rindex_header_in(struct rindex_header *end, struct rindex_header *rh) { rh->magic = le32_to_cpu(end->magic); rh->version = le32_to_cpu(end->version); rh->flags = le32_to_cpu(end->flags); rh->sector_size = le32_to_cpu(end->sector_size); rh->max_resources = le32_to_cpu(end->max_resources); rh->unused = le32_to_cpu(end->unused); rh->rx_offset = le64_to_cpu(end->rx_offset); memcpy(rh->lockspace_name, end->lockspace_name, NAME_ID_SIZE); } void rindex_header_out(struct rindex_header *rh, struct rindex_header *end) { end->magic = cpu_to_le32(rh->magic); end->version = cpu_to_le32(rh->version); end->flags = cpu_to_le32(rh->flags); end->sector_size = cpu_to_le32(rh->sector_size); end->max_resources = cpu_to_le32(rh->max_resources); end->unused = cpu_to_le32(rh->unused); end->rx_offset = cpu_to_le64(rh->rx_offset); memcpy(end->lockspace_name, rh->lockspace_name, NAME_ID_SIZE); } void rindex_entry_in(struct rindex_entry *end, struct rindex_entry *re) { re->res_offset = le64_to_cpu(end->res_offset); re->flags = le32_to_cpu(end->flags); re->unused = le32_to_cpu(end->unused); memcpy(re->name, end->name, NAME_ID_SIZE); } void rindex_entry_out(struct rindex_entry *re, struct rindex_entry *end) { end->res_offset = cpu_to_le64(re->res_offset); end->flags = cpu_to_le32(re->flags); end->unused = cpu_to_le32(re->unused); memcpy(end->name, re->name, NAME_ID_SIZE); } sanlock/src/ondisk.h000066400000000000000000000035131516326262600147520ustar00rootroot00000000000000/* * Copyright 2014 Red Hat, Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. */ #ifndef __ONDISK_H__ #define __ONDISK_H__ #include #include /* * sanlock ondisk format is little endian. */ #if __BYTE_ORDER == __BIG_ENDIAN #define le16_to_cpu(x) (bswap_16((x))) #define le32_to_cpu(x) (bswap_32((x))) #define le64_to_cpu(x) (bswap_64((x))) #define cpu_to_le16(x) (bswap_16((x))) #define cpu_to_le32(x) (bswap_32((x))) #define cpu_to_le64(x) (bswap_64((x))) #endif #if __BYTE_ORDER == __LITTLE_ENDIAN #define le16_to_cpu(x) (x) #define le32_to_cpu(x) (x) #define le64_to_cpu(x) (x) #define cpu_to_le16(x) (x) #define cpu_to_le32(x) (x) #define cpu_to_le64(x) (x) #endif void magic_in(char *end, uint32_t *magic); void leader_record_in(struct leader_record *end, struct leader_record *lr); void leader_record_out(struct leader_record *lr, struct leader_record *end); void request_record_in(struct request_record *end, struct request_record *rr); void request_record_out(struct request_record *rr, struct request_record *end); void paxos_dblock_in(struct paxos_dblock *end, struct paxos_dblock *pd); void paxos_dblock_out(struct paxos_dblock *pd, struct paxos_dblock *end); void mode_block_in(struct mode_block *end, struct mode_block *mb); void mode_block_out(struct mode_block *mb, struct mode_block *end); void rindex_header_in(struct rindex_header *end, struct rindex_header *rh); void rindex_header_out(struct rindex_header *rh, struct rindex_header *end); void rindex_entry_in(struct rindex_entry *end, struct rindex_entry *re); void rindex_entry_out(struct rindex_entry *re, struct rindex_entry *end); #endif sanlock/src/paxos_dblock.h000066400000000000000000000020131516326262600161250ustar00rootroot00000000000000/* * Copyright 2014 Red Hat, Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. */ #ifndef __PAXOS_DBLOCK_H__ #define __PAXOS_DBLOCK_H__ /* The first dblock (for host_id 1) is in the third sector of a paxos lease. The first sector holds the leader record, and the second sector holds the request record. */ #define DBLOCK_CHECKSUM_LEN 48 /* ends before checksum field */ #define DBLOCK_FL_RELEASED 0x00000001 struct paxos_dblock { uint64_t mbal; uint64_t bal; uint64_t inp; /* host_id */ uint64_t inp2; /* host_id generation */ uint64_t inp3; /* host_id's timestamp */ uint64_t lver; uint32_t checksum; uint32_t flags; /* DBLOCK_FL_ */ }; /* * This struct cannot grow any larger than MBLOCK_OFFSET (128) * because the mode_block starts at that offset in the same sector. */ #endif sanlock/src/paxos_lease.c000066400000000000000000003065011516326262600157640ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock_internal.h" #include "diskio.h" #include "ondisk.h" #include "direct.h" #include "log.h" #include "lockspace.h" #include "delta_lease.h" #include "paxos_lease.h" #include "resource.h" #include "timeouts.h" uint32_t crc32c(uint32_t crc, uint8_t *data, size_t length); int get_rand(int a, int b); /* * BK_DEBUG_SIZE: size of buffer to hold ballot debug info, * this can't be larger than LOG_STR_LEN 512 * BK_STR_SIZE: the max length of a dblock string for one host * BK_DEBUG_COUNT: the max number of hosts for which we'll copy * dblock info * * BK_DEBUG_COUNT * BK_STR_SIZE + extra debug text that comes before * the dblock info needs to be less than BK_DEBUG_SIZE. * Be very careful about increasing BK_DEBUG_COUNT because the use * of strncat depends on it. */ #define BK_DEBUG_SIZE 512 #define BK_DEBUG_COUNT 4 #define BK_STR_SIZE 80 static void bk_debug_append(char *bk_debug, char *bk_str) { int off = 0; int i, j; /* bk_debug buffer is BK_DEBUG_SIZE, bk_str is BK_STR_SIZE */ /* include as much debugging text in bk_debug as will fit */ for (i = 0; i < BK_DEBUG_SIZE - 1; i++) { if (!bk_debug[i]) { off = i; break; } } if (off >= (BK_DEBUG_SIZE-1)) { bk_debug[BK_DEBUG_SIZE-1] = '\0'; return; } for (j = 0; j < strlen(bk_str); j++) { bk_debug[off++] = bk_str[j]; if (off >= BK_DEBUG_SIZE) { bk_debug[BK_DEBUG_SIZE-1] = '\0'; return; } } } /* static uint32_t roundup_power_of_two(uint32_t val) { val--; val |= val >> 1; val |= val >> 2; val |= val >> 4; val |= val >> 8; val |= val >> 16; val++; return val; } */ uint32_t leader_checksum(struct leader_record *lr) { return crc32c((uint32_t)~1, (uint8_t *)lr, LEADER_CHECKSUM_LEN); } uint32_t dblock_checksum(struct paxos_dblock *pd) { return crc32c((uint32_t)~1, (uint8_t *)pd, DBLOCK_CHECKSUM_LEN); } int paxos_lease_request_read(struct task *task, struct token *token, struct request_record *rr) { struct request_record rr_end; int rv; /* 1 = request record is second sector */ rv = read_sector(&token->disks[0], token->sector_size, 1, (char *)&rr_end, sizeof(struct request_record), task, task->use_aio, token->io_timeout, "request"); if (rv < 0) return rv; request_record_in(&rr_end, rr); return SANLK_OK; } int paxos_lease_request_write(struct task *task, struct token *token, struct request_record *rr) { struct request_record rr_end; int rv; request_record_out(rr, &rr_end); rv = write_sector(&token->disks[0], token->sector_size, 1, (char *)&rr_end, sizeof(struct request_record), task, task->use_aio, token->io_timeout, "request"); if (rv < 0) return rv; return SANLK_OK; } static int write_dblock(struct task *task, struct token *token, struct sync_disk *disk, uint64_t host_id, struct paxos_dblock *pd); int paxos_erase_dblock(struct task *task, struct token *token, uint64_t host_id) { struct paxos_dblock dblock_end; int num_disks = token->r.num_disks; int num_writes = 0; int d, rv, error = -1; memset(&dblock_end, 0, sizeof(struct paxos_dblock)); for (d = 0; d < num_disks; d++) { rv = write_dblock(task, token, &token->disks[d], host_id, &dblock_end); if (rv < 0) { error = rv; continue; } num_writes++; } if (!majority_disks(num_disks, num_writes)) return error; return SANLK_OK; } /* * Write a combined dblock and mblock. This is an odd case that doesn't fit * well with the way the code has been written. It's used when we want to * convert sh to ex, which requires acquiring the lease owner, but we don't * want to clobber our SHARED mblock by writing a plain dblock in the process * in case there's a problem with the acquiring, we don't want to lose our * shared mode lease. * * NB. this assumes the only mblock flag we want is MBLOCK_SHARED and that * the generation we want is token->host_generation. This is currently * the case, but could change in the future. */ static int write_dblock_mblock_sh(struct task *task, struct token *token, struct sync_disk *disk, uint64_t host_id, struct paxos_dblock *pd) { struct paxos_dblock pd_end; struct mode_block mb; struct mode_block mb_end; char *iobuf, **p_iobuf; uint64_t offset; uint32_t checksum; int iobuf_len, rv, sector_size; memset(&mb, 0, sizeof(mb)); mb.flags = MBLOCK_SHARED; mb.generation = token->host_generation; sector_size = token->sector_size; iobuf_len = sector_size; if (!iobuf_len) return -EINVAL; p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), iobuf_len); if (rv) return -ENOMEM; offset = disk->offset + ((2 + host_id - 1) * sector_size); paxos_dblock_out(pd, &pd_end); /* * N.B. must compute checksum after the data has been byte swapped. */ checksum = dblock_checksum(&pd_end); pd->checksum = checksum; pd_end.checksum = cpu_to_le32(checksum); mode_block_out(&mb, &mb_end); memcpy(iobuf, (char *)&pd_end, sizeof(struct paxos_dblock)); memcpy(iobuf + MBLOCK_OFFSET, (char *)&mb_end, sizeof(struct mode_block)); rv = write_iobuf(disk->fd, offset, iobuf, iobuf_len, task, task->use_aio, token->io_timeout, NULL); if (rv < 0) { log_errot(token, "write_dblock_mblock_sh host_id %llu gen %llu rv %d", (unsigned long long)host_id, (unsigned long long)token->host_generation, rv); } if (rv != SANLK_AIO_TIMEOUT) free(iobuf); return rv; } static int write_dblock(struct task *task, struct token *token, struct sync_disk *disk, uint64_t host_id, struct paxos_dblock *pd) { struct paxos_dblock pd_end; uint32_t checksum; int rv; if (token->flags & T_WRITE_DBLOCK_MBLOCK_SH) { /* special case to preserve our SH mode block within the dblock */ return write_dblock_mblock_sh(task, token, disk, host_id, pd); } /* 1 leader block + 1 request block; host_id N is block offset N-1 */ paxos_dblock_out(pd, &pd_end); /* * N.B. must compute checksum after the data has been byte swapped. */ checksum = dblock_checksum(&pd_end); pd->checksum = checksum; pd_end.checksum = cpu_to_le32(checksum); rv = write_sector(disk, token->sector_size, 2 + host_id - 1, (char *)&pd_end, sizeof(struct paxos_dblock), task, task->use_aio, token->io_timeout, "dblock"); return rv; } static int write_leader(struct task *task, struct token *token, struct sync_disk *disk, struct leader_record *lr) { struct leader_record lr_end; uint32_t checksum; int rv; leader_record_out(lr, &lr_end); /* * N.B. must compute checksum after the data has been byte swapped. */ checksum = leader_checksum(&lr_end); lr->checksum = checksum; lr_end.checksum = cpu_to_le32(checksum); rv = write_sector(disk, token->sector_size, 0, (char *)&lr_end, sizeof(struct leader_record), task, task->use_aio, token->io_timeout, "leader"); return rv; } /* * NB. this should not be used to write the leader record, it is meant only * for manually clobbering the disk to corrupt it for testing, or to manually * repair it after it's corrupted. */ int paxos_lease_leader_clobber(struct task *task, struct token *token, struct leader_record *leader, const char *caller) { struct leader_record lr_end; uint32_t checksum; int rv; leader_record_out(leader, &lr_end); /* * N.B. must compute checksum after the data has been byte swapped. */ checksum = leader_checksum(&lr_end); leader->checksum = checksum; lr_end.checksum = cpu_to_le32(checksum); rv = write_sector(&token->disks[0], token->sector_size, 0, (char *)&lr_end, sizeof(struct leader_record), task, task->use_aio, token->io_timeout, caller); return rv; } static int read_dblock(struct task *task, struct token *token, struct sync_disk *disk, uint64_t host_id, struct paxos_dblock *pd) { struct paxos_dblock pd_end; int rv; /* 1 leader block + 1 request block; host_id N is block offset N-1 */ rv = read_sector(disk, token->sector_size, 2 + host_id - 1, (char *)&pd_end, sizeof(struct paxos_dblock), task, task->use_aio, token->io_timeout, "dblock"); paxos_dblock_in(&pd_end, pd); return rv; } #if 0 static int read_dblocks(struct task *task, struct sync_disk *disk, struct paxos_dblock *pds, int pds_count) { struct paxos_dblock pd_end; char *data; int data_len, rv, i; data_len = pds_count * sector_size; data = malloc(data_len); if (!data) { log_error("read_dblocks malloc %d %s", data_len, disk->path); rv = -ENOMEM; goto out; } /* 2 = 1 leader block + 1 request block */ rv = read_sectors(disk, token->sector_size, 2, pds_count, data, data_len, task, task->use_aio, token->io_timeout, "dblocks"); if (rv < 0) goto out_free; /* copy the first N bytes from each sector, where N is size of paxos_dblock */ for (i = 0; i < pds_count; i++) { memcpy(&pd_end, data + (i * sector_size), sizeof(struct paxos_dblock)); paxos_dblock_in(&pd_end, &pd); memcpy(&pds[i], &pd, sizeof(struct paxos_dblock)); } rv = 0; out_free: free(data); out: return rv; } #endif static int read_leader(struct task *task, int use_aio, struct token *token, struct sync_disk *disk, struct leader_record *lr, uint32_t *checksum) { struct leader_record lr_end; int rv; if (!token->sector_size) { log_errot(token, "paxos_read_leader with zero sector_size"); return -EINVAL; } /* 0 = leader record is first sector */ rv = read_sector(disk, token->sector_size, 0, (char *)&lr_end, sizeof(struct leader_record), task, use_aio, token->io_timeout, "leader"); /* N.B. checksum is computed while the data is in ondisk format. */ *checksum = leader_checksum(&lr_end); leader_record_in(&lr_end, lr); return rv; } static int verify_dblock(struct token *token, struct paxos_dblock *pd, uint32_t checksum) { if (!pd->checksum && !pd->mbal && !pd->bal && !pd->inp && !pd->lver) return SANLK_OK; if (pd->checksum != checksum) { log_errot(token, "verify_dblock wrong checksum %x %x", pd->checksum, checksum); return SANLK_DBLOCK_CHECKSUM; } return SANLK_OK; } /* * It's possible that we pick a bk_max from another host which has our own * inp values in it, and we can end up committing our own inp values, copied * from another host's dblock: * * host2 leader free * host2 phase1 mbal 14002 * host2 writes dblock[1] mbal 14002 * host2 reads no higher mbal * host2 choose own inp 2,1 * host2 phase2 mbal 14002 bal 14002 inp 2,1 * host2 writes dblock[1] bal 14002 inp 2,1 * host1 leader free * host1 phase1 mbal 20001 * host1 writes dblock[0] mbal 20001 * host1 reads no higher mbal * host1 choose dblock[1] bal 14002 inp 2,1 * host1 phase2 mbal 20001 bal 20001 inp 2,1 * host1 writes dblock[0] bal 20001 inp 2,1 * host2 reads dblock[0] mbal 20001 > 14002 * abort2, retry * host2 leader free * host2 phase1 mbal 16002 * host2 writes dblock[1] mbal 16002 * host2 reads dblock[0] mbal 20001 > 16002 * abort1 retry * host2 leader free * host2 phase1 mbal 18002 * host2 writes dblock[1] mbal 18002 * host2 reads dblock[0] mbal 20001 > 18002 * abort1 retry * host2 leader free * host2 phase1 mbal 20002 * host2 writes dblock[1] mbal 20002 * host2 reads no higher mbal * host2 choose dblock[0] bal 20001 inp 2,1 * host1 reads dblock[1] mbal 20002 > 20001 * abort2 retry * host2 phase2 mbal 20002 bal 20002 inp 2,1 * host2 writes dblock[1] bal 20002 inp 2,1 * host2 reads no higher mbal * host2 commit inp 2,1 * host2 success * host1 leader owner 2,1 * host1 fail */ static int run_ballot(struct task *task, struct token *token, uint32_t flags, int num_hosts, uint64_t next_lver, uint64_t our_mbal, struct paxos_dblock *dblock_out, uint64_t *other_max_mbal, uint64_t *larger_lver) { char bk_debug[BK_DEBUG_SIZE]; char bk_str[BK_STR_SIZE]; int bk_debug_count; struct paxos_dblock dblock; struct paxos_dblock bk_in; struct paxos_dblock bk_max; struct paxos_dblock *bk_end; struct paxos_dblock *bk; struct sync_disk *disk; char **p_iobuf = NULL; char *iobuf = NULL; uint32_t checksum; int num_disks = token->r.num_disks; int num_writes, num_reads; int sector_size; int read_len; int iobuf_len; int use_huge = 0; int read_mem_align; int phase2 = 0; int d, q, rv = 0; int q_max = -1; int error; if (num_disks != 1) return -1; sector_size = token->sector_size; /* token->align_size: 1,2,4,8 MB depending on sector_size */ read_len = token->align_size; /* remains at 1MB with hugepage */ iobuf_len = token->align_size; /* increases to 2MB with hugepage, even with 1MB align_size */ read_mem_align = getpagesize(); /* increases to 2MB with hugepage, even with 1MB align_size */ /* Reuse a previously allocated buffer. */ if (task->paxos_areabuf && (task->paxos_areabuf_size >= iobuf_len)) { memset(task->paxos_areabuf, 0, task->paxos_areabuf_size); task->paxos_areabuf_used++; iobuf = task->paxos_areabuf; goto mem_done; } /* Replace previously allocated buffer with a larger one. */ if (task->paxos_areabuf && (task->paxos_areabuf_size < iobuf_len)) { log_taskw(task, "replacing areabuf with size %u smaller than iobuf_len %u", task->paxos_areabuf_size, iobuf_len); free(task->paxos_areabuf); task->paxos_areabuf = NULL; task->paxos_areabuf_size = 0; } /* Determine if hugepages is applicable. */ if (com.use_hugepages && ((iobuf_len == ONE_MB_IN_BYTES) || !(iobuf_len % TWO_MB_IN_BYTES))) { read_mem_align = TWO_MB_IN_BYTES; if (iobuf_len < TWO_MB_IN_BYTES) iobuf_len = TWO_MB_IN_BYTES; use_huge = 1; } /* Allocate a new buffer. */ p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, read_mem_align, iobuf_len); if (rv) return rv; /* Make the new buffer use hugepages. */ if (use_huge) { char *bp; madvise(iobuf, iobuf_len, MADV_HUGEPAGE); /* allocate the pages */ for (bp = iobuf; bp < iobuf+iobuf_len; bp += TWO_MB_IN_BYTES) memset(bp, 0, 1); } /* Save the new buffer to reuse if the task doesn't have one. */ if (!task->paxos_areabuf) { task->paxos_areabuf = iobuf; task->paxos_areabuf_size = iobuf_len; task->paxos_areabuf_created++; log_taskd(task, "created areabuf %p size %u %u", iobuf, iobuf_len, task->paxos_areabuf_created); } mem_done: /* * phase 1 * * "For each disk d, it tries first to write dblock[p] to disk[d][p] * and then to read disk[d][q] for all other processors q. It aborts * the ballot if, for any d and q, it finds disk[d][q].mbal > * dblock[p].mbal. The phase completes when p has written and read a * majority of the disks, without reading any block whose mbal * component is greater than dblock[p].mbal." */ log_token(token, "ballot %llu phase1 write mbal %llu", (unsigned long long)next_lver, (unsigned long long)our_mbal); memset(&dblock, 0, sizeof(struct paxos_dblock)); dblock.mbal = our_mbal; dblock.lver = next_lver; dblock.checksum = 0; /* set after paxos_dblock_out */ memset(&bk_max, 0, sizeof(struct paxos_dblock)); num_writes = 0; for (d = 0; d < num_disks; d++) { /* acquire io: write 1 (sector) */ rv = write_dblock(task, token, &token->disks[d], token->host_id, &dblock); if (rv < 0) continue; num_writes++; } if (!majority_disks(num_disks, num_writes)) { log_errot(token, "ballot %llu dblock write error %d", (unsigned long long)next_lver, rv); error = SANLK_DBLOCK_WRITE; goto out; } memset(bk_debug, 0, sizeof(bk_debug)); bk_debug_count = 0; num_reads = 0; for (d = 0; d < num_disks; d++) { disk = &token->disks[d]; memset(iobuf, 0, read_len); /* acquire io: read 2 (lease area) */ rv = read_iobuf(disk->fd, disk->offset, iobuf, read_len, task, task->use_aio, token->io_timeout, NULL); if (rv == SANLK_AIO_TIMEOUT) { iobuf = NULL; /* areabuf will be freed when aio completes */ task->paxos_areabuf = NULL; task->paxos_areabuf_size = 0; } if (rv < 0) continue; num_reads++; for (q = 0; q < num_hosts; q++) { bk_end = (struct paxos_dblock *)(iobuf + ((2 + q)*sector_size)); checksum = dblock_checksum(bk_end); paxos_dblock_in(bk_end, &bk_in); bk = &bk_in; if (bk_in.mbal && ((flags & PAXOS_ACQUIRE_DEBUG_ALL) || (bk_in.lver >= dblock.lver))) { if (bk_debug_count >= BK_DEBUG_COUNT) { log_token(token, "ballot %llu phase1 read %s", (unsigned long long)next_lver, bk_debug); memset(bk_debug, 0, sizeof(bk_debug)); bk_debug_count = 0; } memset(bk_str, 0, sizeof(bk_str)); snprintf(bk_str, BK_STR_SIZE, "%d:%llu:%llu:%llu:%llu:%llu:%llu:%x,", q, (unsigned long long)bk_in.mbal, (unsigned long long)bk_in.bal, (unsigned long long)bk_in.inp, (unsigned long long)bk_in.inp2, (unsigned long long)bk_in.inp3, (unsigned long long)bk_in.lver, bk_in.flags); bk_str[BK_STR_SIZE-1] = '\0'; bk_debug_append(bk_debug, bk_str); bk_debug_count++; } rv = verify_dblock(token, bk, checksum); if (rv < 0) continue; check_mode_block(token, next_lver, q, (char *)bk_end); if (bk->lver < dblock.lver) continue; if (bk->lver > dblock.lver) { log_warnt(token, "ballot %llu abort1 larger lver in bk[%d] %llu:%llu:%llu:%llu:%llu:%llu " "our dblock %llu:%llu:%llu:%llu:%llu:%llu", (unsigned long long)next_lver, q, (unsigned long long)bk->mbal, (unsigned long long)bk->bal, (unsigned long long)bk->inp, (unsigned long long)bk->inp2, (unsigned long long)bk->inp3, (unsigned long long)bk->lver, (unsigned long long)dblock.mbal, (unsigned long long)dblock.bal, (unsigned long long)dblock.inp, (unsigned long long)dblock.inp2, (unsigned long long)dblock.inp3, (unsigned long long)dblock.lver); log_token(token, "ballot %llu phase1 read %s", (unsigned long long)next_lver, bk_debug); *larger_lver = bk->lver; error = SANLK_DBLOCK_LVER; goto out; } /* see "It aborts the ballot" in comment above */ if (bk->mbal > dblock.mbal) { log_warnt(token, "ballot %llu abort1 larger mbal in bk[%d] %llu:%llu:%llu:%llu:%llu:%llu " "our dblock %llu:%llu:%llu:%llu:%llu:%llu", (unsigned long long)next_lver, q, (unsigned long long)bk->mbal, (unsigned long long)bk->bal, (unsigned long long)bk->inp, (unsigned long long)bk->inp2, (unsigned long long)bk->inp3, (unsigned long long)bk->lver, (unsigned long long)dblock.mbal, (unsigned long long)dblock.bal, (unsigned long long)dblock.inp, (unsigned long long)dblock.inp2, (unsigned long long)dblock.inp3, (unsigned long long)dblock.lver); log_token(token, "ballot %llu phase1 read %s", (unsigned long long)next_lver, bk_debug); *other_max_mbal = bk->mbal; error = SANLK_DBLOCK_MBAL; goto out; } /* see choosing inp for phase 2 in comment below */ if (!bk->inp) continue; if (!bk->bal) { log_errot(token, "ballot %llu zero bal inp[%d] %llu", (unsigned long long)next_lver, q, (unsigned long long)bk->inp); continue; } if (bk->bal > bk_max.bal) { bk_max = *bk; q_max = q; } } } log_token(token, "ballot %llu phase1 read %s", (unsigned long long)next_lver, bk_debug); if (!majority_disks(num_disks, num_reads)) { log_errot(token, "ballot %llu dblock read error %d", (unsigned long long)next_lver, rv); error = SANLK_DBLOCK_READ; goto out; } /* * "When it completes phase 1, p chooses a new value of dblock[p].inp, * sets dblock[p].bal to dblock[p].mbal (its current ballot number), * and begins phase 2." * * "We now describe how processor p chooses the value of dblock[p].inp * that it tries to commit in phase 2. Let blocksSeen be the set * consisting of dblock[p] and all the records disk[d][q] read by p in * phase 1. Let nonInitBlks be the subset of blocksSeen consisting of * those records whose inp field is not NotAnInput. If nonInitBlks is * empty, then p sets dblock[p].inp to its own input value input[p]. * Otherwise, it sets dblock[p].inp to bk.inp for some record bk in * nonInitBlks having the largest value of bk.bal." */ if (bk_max.inp) { /* lver and mbal are already set */ dblock.inp = bk_max.inp; dblock.inp2 = bk_max.inp2; dblock.inp3 = bk_max.inp3; } else { /* lver and mbal are already set */ dblock.inp = token->host_id; dblock.inp2 = token->host_generation; dblock.inp3 = monotime(); } dblock.bal = dblock.mbal; dblock.checksum = 0; /* set after paxos_dblock_out */ if (bk_max.inp) { log_token(token, "ballot %llu choose bk_max[%d] lver %llu mbal %llu bal %llu inp %llu %llu %llu", (unsigned long long)next_lver, q_max, (unsigned long long)bk_max.lver, (unsigned long long)bk_max.mbal, (unsigned long long)bk_max.bal, (unsigned long long)bk_max.inp, (unsigned long long)bk_max.inp2, (unsigned long long)bk_max.inp3); } /* * phase 2 * * Same description as phase 1, same sequence of writes/reads. */ phase2 = 1; log_token(token, "ballot %llu phase2 write bal %llu inp %llu %llu %llu q_max %d", (unsigned long long)dblock.lver, (unsigned long long)dblock.bal, (unsigned long long)dblock.inp, (unsigned long long)dblock.inp2, (unsigned long long)dblock.inp3, q_max); num_writes = 0; for (d = 0; d < num_disks; d++) { /* acquire io: write 2 (sector) */ rv = write_dblock(task, token, &token->disks[d], token->host_id, &dblock); if (rv < 0) continue; num_writes++; } if (!majority_disks(num_disks, num_writes)) { log_errot(token, "ballot %llu our dblock write2 error %d", (unsigned long long)next_lver, rv); error = SANLK_DBLOCK_WRITE; goto out; } memset(bk_debug, 0, sizeof(bk_debug)); bk_debug_count = 0; num_reads = 0; for (d = 0; d < num_disks; d++) { disk = &token->disks[d]; memset(iobuf, 0, read_len); /* acquire io: read 3 (lease area) */ rv = read_iobuf(disk->fd, disk->offset, iobuf, read_len, task, task->use_aio, token->io_timeout, NULL); if (rv == SANLK_AIO_TIMEOUT) { iobuf = NULL; /* areabuf will be freed when aio completes */ task->paxos_areabuf = NULL; task->paxos_areabuf_size = 0; } if (rv < 0) continue; num_reads++; for (q = 0; q < num_hosts; q++) { bk_end = (struct paxos_dblock *)(iobuf + ((2 + q)*sector_size)); checksum = dblock_checksum(bk_end); paxos_dblock_in(bk_end, &bk_in); bk = &bk_in; if (bk->mbal && ((flags & PAXOS_ACQUIRE_DEBUG_ALL) || (bk->lver >= dblock.lver))) { if (bk_debug_count >= BK_DEBUG_COUNT) { log_token(token, "ballot %llu phase2 read %s", (unsigned long long)next_lver, bk_debug); memset(bk_debug, 0, sizeof(bk_debug)); bk_debug_count = 0; } memset(bk_str, 0, sizeof(bk_str)); snprintf(bk_str, BK_STR_SIZE, "%d:%llu:%llu:%llu:%llu:%llu:%llu:%x,", q, (unsigned long long)bk->mbal, (unsigned long long)bk->bal, (unsigned long long)bk->inp, (unsigned long long)bk->inp2, (unsigned long long)bk->inp3, (unsigned long long)bk->lver, bk->flags); bk_str[BK_STR_SIZE-1] = '\0'; bk_debug_append(bk_debug, bk_str); bk_debug_count++; } rv = verify_dblock(token, bk, checksum); if (rv < 0) continue; if (bk->lver < dblock.lver) continue; if (bk->lver > dblock.lver) { /* * This happens when we choose another host's bk, that host * acquires the lease itself, releases it, and reacquires it * with a new lver, all before we get here, at which point * we see the larger lver. I believe case this would always * also be caught the the bk->mbal > dblock.mbal condition * below. */ log_warnt(token, "ballot %llu abort2 larger lver in bk[%d] %llu:%llu:%llu:%llu:%llu:%llu " "our dblock %llu:%llu:%llu:%llu:%llu:%llu", (unsigned long long)next_lver, q, (unsigned long long)bk->mbal, (unsigned long long)bk->bal, (unsigned long long)bk->inp, (unsigned long long)bk->inp2, (unsigned long long)bk->inp3, (unsigned long long)bk->lver, (unsigned long long)dblock.mbal, (unsigned long long)dblock.bal, (unsigned long long)dblock.inp, (unsigned long long)dblock.inp2, (unsigned long long)dblock.inp3, (unsigned long long)dblock.lver); log_token(token, "ballot %llu phase2 read %s", (unsigned long long)next_lver, bk_debug); *larger_lver = bk->lver; error = SANLK_DBLOCK_LVER; goto out; } /* see "It aborts the ballot" in comment above */ if (bk->mbal > dblock.mbal) { log_warnt(token, "ballot %llu abort2 larger mbal in bk[%d] %llu:%llu:%llu:%llu:%llu:%llu " "our dblock %llu:%llu:%llu:%llu:%llu:%llu", (unsigned long long)next_lver, q, (unsigned long long)bk->mbal, (unsigned long long)bk->bal, (unsigned long long)bk->inp, (unsigned long long)bk->inp2, (unsigned long long)bk->inp3, (unsigned long long)bk->lver, (unsigned long long)dblock.mbal, (unsigned long long)dblock.bal, (unsigned long long)dblock.inp, (unsigned long long)dblock.inp2, (unsigned long long)dblock.inp3, (unsigned long long)dblock.lver); log_token(token, "ballot %llu phase2 read %s", (unsigned long long)next_lver, bk_debug); *other_max_mbal = bk->mbal; error = SANLK_DBLOCK_MBAL; goto out; } } } log_token(token, "ballot %llu phase2 read %s", (unsigned long long)next_lver, bk_debug); if (!majority_disks(num_disks, num_reads)) { log_errot(token, "ballot %llu dblock read2 error %d", (unsigned long long)next_lver, rv); error = SANLK_DBLOCK_READ; goto out; } /* "When it completes phase 2, p has committed dblock[p].inp." */ memcpy(dblock_out, &dblock, sizeof(struct paxos_dblock)); error = SANLK_OK; out: if (phase2 && (error < 0) && ((error == SANLK_DBLOCK_READ) || (error == SANLK_DBLOCK_WRITE))) { /* * After phase2 we might "win" the ballot even if we don't complete it * because another host could could pick and commit our dblock values. * If we abort the acquire, but are granted the lease, this would leave * us owning the lease on disk. With this flag, the release path will * try to ensure we are not and do not become the lease owner. */ token->flags |= T_RETRACT_PAXOS; log_errot(token, "ballot %llu retract error %d", (unsigned long long)next_lver, error); } memcpy(dblock_out, &dblock, sizeof(struct paxos_dblock)); return error; } static void log_leader_error(int result, struct token *token, struct sync_disk *disk, struct leader_record *lr, const char *caller) { log_errot(token, "leader1 %s error %d sn %.48s rn %.48s", caller ? caller : "unknown", result, token->r.lockspace_name, token->r.name); log_errot(token, "leader2 path %s offset %llu fd %d", disk->path, (unsigned long long)disk->offset, disk->fd); log_errot(token, "leader3 m %x v %x ss %u nh %llu mh %llu oi %llu og %llu lv %llu", lr->magic, lr->version, lr->sector_size, (unsigned long long)lr->num_hosts, (unsigned long long)lr->max_hosts, (unsigned long long)lr->owner_id, (unsigned long long)lr->owner_generation, (unsigned long long)lr->lver); log_errot(token, "leader4 sn %.48s rn %.48s ts %llu cs %x", lr->space_name, lr->resource_name, (unsigned long long)lr->timestamp, lr->checksum); log_errot(token, "leader5 wi %llu wg %llu wt %llu", (unsigned long long)lr->write_id, (unsigned long long)lr->write_generation, (unsigned long long)lr->write_timestamp); } static int _verify_leader(struct token *token, struct sync_disk *disk, struct leader_record *lr, uint32_t checksum, const char *caller, int print_error) { int result; if (lr->magic == PAXOS_DISK_CLEAR) return SANLK_LEADER_MAGIC; if (lr->magic != PAXOS_DISK_MAGIC) { result = SANLK_LEADER_MAGIC; goto fail; } if ((lr->version & 0xFFFF0000) != PAXOS_DISK_VERSION_MAJOR) { result = SANLK_LEADER_VERSION; goto fail; } if (strncmp(lr->space_name, token->r.lockspace_name, NAME_ID_SIZE)) { result = SANLK_LEADER_LOCKSPACE; goto fail; } if (strncmp(lr->resource_name, token->r.name, NAME_ID_SIZE)) { result = SANLK_LEADER_RESOURCE; goto fail; } if (lr->num_hosts < token->host_id) { result = SANLK_LEADER_NUMHOSTS; goto fail; } if (lr->checksum != checksum) { result = SANLK_LEADER_CHECKSUM; goto fail; } return SANLK_OK; fail: if (!print_error) return result; switch (result) { case SANLK_LEADER_MAGIC: log_errot(token, "verify_leader wrong magic %x %s", lr->magic, disk->path); break; case SANLK_LEADER_VERSION: log_errot(token, "verify_leader wrong version %x %s", lr->version, disk->path); break; case SANLK_LEADER_LOCKSPACE: log_errot(token, "verify_leader wrong space name %.48s %.48s %s", lr->space_name, token->r.lockspace_name, disk->path); break; case SANLK_LEADER_RESOURCE: log_errot(token, "verify_leader wrong resource name %.48s %.48s %s", lr->resource_name, token->r.name, disk->path); break; case SANLK_LEADER_NUMHOSTS: log_errot(token, "verify_leader num_hosts too small %llu %llu %s", (unsigned long long)lr->num_hosts, (unsigned long long)token->host_id, disk->path); break; case SANLK_LEADER_CHECKSUM: log_errot(token, "verify_leader wrong checksum %x %x %s", lr->checksum, checksum, disk->path); break; }; log_leader_error(result, token, disk, lr, caller); /* memset(&leader_end, 0, sizeof(struct leader_record)); rv = read_sector(disk, token->sector_size, 0, (char *)&leader_end, sizeof(struct leader_record), NULL, NO_AIO, 1, "paxos_verify"); leader_record_in(&leader_end, &leader_rr); log_leader_error(rv, token, disk, &leader_rr, "paxos_verify"); */ return result; } static int verify_leader(struct token *token, struct sync_disk *disk, struct leader_record *lr, uint32_t checksum, const char *caller) { return _verify_leader(token, disk, lr, checksum, caller, 1); } static int verify_leader_no_error(struct token *token, struct sync_disk *disk, struct leader_record *lr, uint32_t checksum, const char *caller) { return _verify_leader(token, disk, lr, checksum, caller, 0); } int paxos_verify_leader(struct token *token, struct sync_disk *disk, struct leader_record *lr, uint32_t checksum, const char *caller) { return verify_leader(token, disk, lr, checksum, caller); } /* read the lockspace name and resource name given the disk location */ int paxos_read_resource(struct task *task, struct token *token, struct sanlk_resource *res) { struct leader_record leader; uint32_t checksum; int align_size; int tmp_sector_size = 0; int rv; memset(&leader, 0, sizeof(struct leader_record)); /* * We don't know the sector size, so we don't know if we should read * 512 or 4k, but it doesn't matter since the leader record is all that * we need. It's probably better to read 4k on a 512 disk than to read 512 * on a 4k disk, so always do a 4k read. */ if (!token->sector_size) { token->sector_size = 4096; token->align_size = sector_size_to_align_size_old(4096); tmp_sector_size = 1; } rv = read_leader(task, task->use_aio, token, &token->disks[0], &leader, &checksum); if (rv < 0) return rv; if (!res->lockspace_name[0]) memcpy(token->r.lockspace_name, leader.space_name, NAME_ID_SIZE); if (!res->name[0]) memcpy(token->r.name, leader.resource_name, NAME_ID_SIZE); if (token->flags & T_CHECK_EXISTS) { if (leader.magic != PAXOS_DISK_MAGIC) rv = SANLK_LEADER_MAGIC; else rv = SANLK_OK; } else { rv = verify_leader_no_error(token, &token->disks[0], &leader, checksum, "read_resource"); } if (rv == SANLK_OK) { memcpy(res->lockspace_name, leader.space_name, NAME_ID_SIZE); memcpy(res->name, leader.resource_name, NAME_ID_SIZE); res->lver = leader.lver; if (leader.flags & LFL_CAW) res->flags |= SANLK_RES_USING_CAW; if ((leader.sector_size == 512) || (leader.sector_size == 4096)) { align_size = leader_align_size_from_flag(leader.flags); if (!align_size) align_size = sector_size_to_align_size_old(leader.sector_size); token->sector_size = leader.sector_size; token->align_size = align_size; /* The flags set by the user may be wrong. */ sanlk_res_sector_flags_clear(&res->flags); sanlk_res_align_flags_clear(&res->flags); res->flags |= sanlk_res_sector_size_to_flag(leader.sector_size); res->flags |= sanlk_res_align_size_to_flag(align_size); } else if (tmp_sector_size) { /* we don't know the correct value, so don't set any */ /* FIXME: add a note about when this can happen */ token->sector_size = 0; token->align_size = 0; } } return rv; } int paxos_read_buf(struct task *task, struct token *token, char **buf_out) { char *iobuf, **p_iobuf; struct sync_disk *disk = &token->disks[0]; int rv, iobuf_len; if (!token->sector_size || !token->align_size) { log_errot(token, "paxos_read_buf with sector_size %d align_size %d", token->sector_size, token->align_size); return -EINVAL; } iobuf_len = token->align_size; if (iobuf_len < 0) return iobuf_len; p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), iobuf_len); if (rv) return rv; memset(iobuf, 0, iobuf_len); rv = read_iobuf(disk->fd, disk->offset, iobuf, iobuf_len, task, task->use_aio, token->io_timeout, NULL); *buf_out = iobuf; return rv; } static int _leader_read_one(struct task *task, int use_aio, struct token *token, struct leader_record *leader_ret, const char *caller) { struct leader_record leader; uint32_t checksum; int rv; memset(&leader, 0, sizeof(struct leader_record)); rv = read_leader(task, use_aio, token, &token->disks[0], &leader, &checksum); if (rv < 0) return rv; rv = verify_leader(token, &token->disks[0], &leader, checksum, caller); /* copy what we read even if verify finds a problem */ memcpy(leader_ret, &leader, sizeof(struct leader_record)); return rv; } int paxos_lease_leader_read(struct task *task, int use_aio, struct token *token, struct leader_record *leader_ret, const char *caller) { int rv; rv = _leader_read_one(task, use_aio, token, leader_ret, caller); if (rv == SANLK_OK) log_token(token, "%s leader %llu owner %llu %llu %llu", caller, (unsigned long long)leader_ret->lver, (unsigned long long)leader_ret->owner_id, (unsigned long long)leader_ret->owner_generation, (unsigned long long)leader_ret->timestamp); return rv; } int paxos_lease_leader_read_buf(struct task *task, int use_aio, struct token *token, struct leader_record *leader_ret, char *buf, const char *caller) { struct sync_disk *disk = &token->disks[0]; struct leader_record leader; struct leader_record leader_end; uint32_t checksum; int rv; rv = read_iobuf(disk->fd, disk->offset, buf, token->sector_size, task, task->use_aio, token->io_timeout, NULL); if (rv < 0) return rv; memcpy(&leader_end, buf, sizeof(struct leader_record)); checksum = leader_checksum(&leader_end); leader_record_in(&leader_end, &leader); rv = verify_leader(token, disk, &leader, checksum, caller); if (rv < 0) return rv; memcpy(leader_ret, &leader, sizeof(struct leader_record)); return SANLK_OK; } static int _lease_read_one(struct task *task, struct token *token, int use_aio, uint32_t flags, struct leader_record *leader_ret, struct paxos_dblock *our_dblock, uint64_t *max_mbal, int *max_q, const char *caller, int log_bk_vals, int *is_caw) { char bk_debug[BK_DEBUG_SIZE]; char bk_str[BK_STR_SIZE]; int bk_debug_count; struct sync_disk *disk = &token->disks[0]; struct leader_record leader_end; struct paxos_dblock our_dblock_end; struct paxos_dblock bk; char *iobuf, **p_iobuf; uint32_t host_id = token->host_id; uint32_t checksum; struct paxos_dblock *bk_end; uint64_t tmp_mbal = 0; int sector_size; int read_len; int iobuf_len; int use_huge = 0; int read_mem_align; int q, tmp_q = -1, rv; /* FIXME: what case is this? */ if (token->align_size < 0) { log_errot(token, "lease_read_one align_size %d", token->align_size); return token->align_size; } sector_size = token->sector_size; /* token->align_size: 1,2,4,8 MB depending on sector_size */ read_len = token->align_size; /* remains at 1MB with hugepage */ iobuf_len = token->align_size; /* increases to 2MB with hugepage, even with 1MB align_size */ read_mem_align = getpagesize(); /* increases to 2MB with hugepage, even with 1MB align_size */ /* Reuse a previously allocated buffer. */ if (task->paxos_areabuf && (task->paxos_areabuf_size >= iobuf_len)) { memset(task->paxos_areabuf, 0, task->paxos_areabuf_size); iobuf = task->paxos_areabuf; task->paxos_areabuf_used++; goto mem_done; } /* Replace previously allocated buffer with a larger one. */ if (task->paxos_areabuf && (task->paxos_areabuf_size < iobuf_len)) { log_taskw(task, "replacing areabuf with size %u smaller than iobuf_len %u", task->paxos_areabuf_size, iobuf_len); free(task->paxos_areabuf); task->paxos_areabuf = NULL; task->paxos_areabuf_size = 0; } /* Determine if hugepages is applicable. */ if (com.use_hugepages && ((iobuf_len == ONE_MB_IN_BYTES) || !(iobuf_len % TWO_MB_IN_BYTES))) { read_mem_align = TWO_MB_IN_BYTES; if (iobuf_len < TWO_MB_IN_BYTES) iobuf_len = TWO_MB_IN_BYTES; use_huge = 1; } /* Allocate a new buffer. */ p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, read_mem_align, iobuf_len); if (rv) return rv; /* Make the new buffer use hugepages. */ if (use_huge) { char *bp; madvise(iobuf, iobuf_len, MADV_HUGEPAGE); /* allocate the pages */ for (bp = iobuf; bp < iobuf+iobuf_len; bp += TWO_MB_IN_BYTES) memset(bp, 0, 1); } /* Save the new buffer to reuse if the task doesn't have one. */ if (!task->paxos_areabuf) { task->paxos_areabuf = iobuf; task->paxos_areabuf_size = iobuf_len; task->paxos_areabuf_created++; log_taskd(task, "created areabuf %p size %u %u", iobuf, iobuf_len, task->paxos_areabuf_created); } mem_done: rv = read_iobuf(disk->fd, disk->offset, iobuf, read_len, task, use_aio, token->io_timeout, NULL); if (rv < 0) goto out; memcpy(&leader_end, iobuf, sizeof(struct leader_record)); checksum = leader_checksum(&leader_end); leader_record_in(&leader_end, leader_ret); rv = verify_leader(token, disk, leader_ret, checksum, caller); if (rv < 0) goto out; if (leader_ret->flags & LFL_CAW) { if (is_caw) *is_caw = 1; goto out; } memcpy(&our_dblock_end, iobuf + ((host_id + 1) * sector_size), sizeof(struct paxos_dblock)); paxos_dblock_in(&our_dblock_end, our_dblock); memset(bk_debug, 0, sizeof(bk_debug)); bk_debug_count = 0; for (q = 0; q < leader_ret->num_hosts; q++) { bk_end = (struct paxos_dblock *)(iobuf + ((2 + q) * sector_size)); checksum = dblock_checksum(bk_end); paxos_dblock_in(bk_end, &bk); if (log_bk_vals && bk.mbal && ((flags & PAXOS_ACQUIRE_DEBUG_ALL) || (bk.lver >= leader_ret->lver))) { if (bk_debug_count >= BK_DEBUG_COUNT) { log_token(token, "leader %llu dblocks %s", (unsigned long long)leader_ret->lver, bk_debug); memset(bk_debug, 0, sizeof(bk_debug)); bk_debug_count = 0; } memset(bk_str, 0, sizeof(bk_str)); snprintf(bk_str, BK_STR_SIZE, "%d:%llu:%llu:%llu:%llu:%llu:%llu:%x,", q, (unsigned long long)bk.mbal, (unsigned long long)bk.bal, (unsigned long long)bk.inp, (unsigned long long)bk.inp2, (unsigned long long)bk.inp3, (unsigned long long)bk.lver, bk.flags); bk_str[BK_STR_SIZE-1] = '\0'; bk_debug_append(bk_debug, bk_str); bk_debug_count++; } rv = verify_dblock(token, &bk, checksum); if (rv < 0) goto out; if (!tmp_mbal || bk.mbal > tmp_mbal) { tmp_mbal = bk.mbal; tmp_q = q; } } *max_mbal = tmp_mbal; *max_q = tmp_q; if (log_bk_vals) log_token(token, "leader %llu owner %llu %llu %llu dblocks %s", (unsigned long long)leader_ret->lver, (unsigned long long)leader_ret->owner_id, (unsigned long long)leader_ret->owner_generation, (unsigned long long)leader_ret->timestamp, bk_debug); out: if (rv == SANLK_AIO_TIMEOUT) { /* areabuf will be freed when aio completes */ task->paxos_areabuf = NULL; task->paxos_areabuf_size = 0; } return rv; } /* * read all the initial values needed to start disk paxos: * - the leader record * - our own dblock * - the max mbal from all dblocks * * Read the entire lease area in one i/o and copy all those * values from it. */ int paxos_lease_read(struct task *task, int use_aio, struct token *token, uint32_t flags, struct leader_record *leader_ret, uint64_t *max_mbal, const char *caller, int log_bk_vals, int *is_caw) { struct paxos_dblock our_dblock; int rv, q = -1; rv = _lease_read_one(task, token, use_aio, flags, leader_ret, &our_dblock, max_mbal, &q, caller, log_bk_vals, is_caw); if (rv != SANLK_OK) return rv; if (is_caw && *is_caw) log_token(token, "%s leader %llu owner %llu %llu %llu caw", caller, (unsigned long long)leader_ret->lver, (unsigned long long)leader_ret->owner_id, (unsigned long long)leader_ret->owner_generation, (unsigned long long)leader_ret->timestamp); else log_token(token, "%s leader %llu owner %llu %llu %llu max mbal[%d] %llu " "our_dblock %llu %llu %llu %llu %llu %llu", caller, (unsigned long long)leader_ret->lver, (unsigned long long)leader_ret->owner_id, (unsigned long long)leader_ret->owner_generation, (unsigned long long)leader_ret->timestamp, q, (unsigned long long)*max_mbal, (unsigned long long)our_dblock.mbal, (unsigned long long)our_dblock.bal, (unsigned long long)our_dblock.inp, (unsigned long long)our_dblock.inp2, (unsigned long long)our_dblock.inp3, (unsigned long long)our_dblock.lver); return SANLK_OK; } static int write_new_leader(struct task *task, struct token *token, struct leader_record *nl, const char *caller) { int num_disks = token->r.num_disks; int num_writes = 0; int timeout = 0; int rv = 0; int d; for (d = 0; d < num_disks; d++) { rv = write_leader(task, token, &token->disks[d], nl); if (rv == SANLK_AIO_TIMEOUT) timeout = 1; if (rv < 0) continue; num_writes++; } if (!majority_disks(num_disks, num_writes)) { log_errot(token, "%s write_new_leader error %d timeout %d owner %llu %llu %llu", caller, rv, timeout, (unsigned long long)nl->owner_id, (unsigned long long)nl->owner_generation, (unsigned long long)nl->timestamp); if (timeout) return SANLK_AIO_TIMEOUT; if (rv < 0) return rv; return SANLK_LEADER_WRITE; } return SANLK_OK; } void close_token_caw(struct token *token) { if (token->caw_dev_fd) { close(token->caw_dev_fd); token->caw_dev_fd = 0; } } int open_token_caw(struct token *token) { char caw_path[PATH_MAX] = { 0 }; uint32_t disk_major = token->resource->disk_major; uint32_t disk_minor = token->resource->disk_minor; uint32_t caw_major; uint32_t caw_minor; uint64_t disk_offset_sect512; uint64_t caw_offset_sect512; int caw_fd; int rv; if (token->caw_dev_fd) { /* shouldn't happen */ log_errot(token, "open_token_caw already open %d", token->caw_dev_fd); return -1; } restart: if (token->resource->caw_state == CAW_STATE_USE_DISK) { if ((caw_fd = open(token->disks[0].path, O_RDWR | O_DIRECT | O_SYNC, 0)) < 0) { log_errot(token, "open_token_caw disk %u:%u %s errno %d", disk_major, disk_minor, token->disks[0].path, errno); return -1; } if (!caw_fd) { log_errot(token, "open_token_caw disk %u:%u %s fd zero", disk_major, disk_minor, token->disks[0].path); return -1; } token->caw_dev_fd = caw_fd; return 0; } if (token->resource->caw_state == CAW_STATE_FOUND_DEV) { caw_major = token->resource->caw_major; caw_minor = token->resource->caw_minor; if (devno_to_path(caw_major, caw_minor, caw_path) < 0) { log_errot(token, "open_token_caw: disk %u:%u devno_to_path %u:%u error", disk_major, disk_minor, caw_major, caw_minor); return -1; } if ((caw_fd = open(caw_path, O_RDWR | O_DIRECT | O_SYNC, 0)) < 0) { log_errot(token, "open_token_caw disk %u:%u caw %u:%u %s open errno %d", disk_major, disk_minor, caw_major, caw_minor, caw_path, errno); return -1; } if (!caw_fd) { log_errot(token, "open_token_caw disk %u:%u caw %u:%u %s open fd zero", disk_major, disk_minor, caw_major, caw_minor, caw_path); return -1; } token->caw_dev_fd = caw_fd; return 0; } if (token->resource->caw_state == CAW_STATE_UNKNOWN) { if (!dev_is_lv(disk_major, disk_minor)) { if (dev_is_scsi(disk_major, disk_minor) || dev_is_mpath(disk_major, disk_minor)) { token->resource->caw_major = token->resource->disk_major; token->resource->caw_minor = token->resource->disk_minor; token->resource->caw_offset = token->disks[0].offset; token->resource->caw_state = CAW_STATE_USE_DISK; goto restart; } /* we shouldn't get this far */ log_errot(token, "open_caw_dev disk %u:%u does not support caw", disk_major, disk_minor); return -1; } /* * SGIO compare-and-write must be submitted to a scsi LUN or mpath device. * dm rejects SGIO to an LV, so get the scsi or mpath dev used by the LV * (dm table) at the given offset. */ /* dm units are always 512 byte sectors */ disk_offset_sect512 = token->disks[0].offset / 512; rv = lv_to_target(disk_major, disk_minor, disk_offset_sect512, &caw_major, &caw_minor, &caw_offset_sect512); if (rv < 0) { log_errot(token, "open_token_caw: unsupported LV %u:%u error %d", disk_major, disk_minor, rv); return -1; } token->resource->caw_major = caw_major; token->resource->caw_minor = caw_minor; token->resource->caw_offset = caw_offset_sect512 * 512; token->resource->caw_state = CAW_STATE_FOUND_DEV; goto restart; } log_errot(token, "open_caw_dev disk %u:%u invalid state %u", disk_major, disk_minor, token->resource->caw_state); return -1; } static int token_acquire_compare_and_write(struct task *task, struct token *token, char *area_buf, struct leader_record *leader_end) { char *caw_data; char **p_caw_data; int sector_size = token->sector_size; int rv; p_caw_data = &caw_data; rv = posix_memalign((void *)p_caw_data, getpagesize(), sector_size * 2); if (rv) return -ENOMEM; memcpy(caw_data, area_buf, sector_size); memcpy(caw_data + sector_size, area_buf, sector_size); memcpy(caw_data + sector_size, leader_end, sizeof(struct leader_record)); /* fd will already be open when retried */ if (!token->caw_dev_fd) { rv = open_token_caw(token); if (rv < 0) { log_errot(token, "paxos_acquire open caw error %d", rv); rv = SANLK_CAW_OPEN; goto out; } } rv = compare_and_write(task, token->caw_dev_fd, token->io_timeout, token->resource->caw_major, token->resource->caw_minor, sector_size, token->resource->caw_offset, caw_data, NULL, NULL); out: /* on success, keep caw_data so it can be used for release without rereading disk */ if (!rv) token->caw_data = caw_data; else free(caw_data); return rv; } /* caw alternative for check_mode_block() */ static void check_mode_blocks(struct token *token, int num_hosts, char *area_buf) { struct mode_block *mb_end; struct mode_block mb; char *dblock; int i; for (i = 0; i < num_hosts; i++) { dblock = (char *)(area_buf + ((2 + i) * token->sector_size)); mb_end = (struct mode_block *)(dblock + MBLOCK_OFFSET); mode_block_in(mb_end, &mb); if (mb.flags & MBLOCK_SHARED) { set_id_bit(i + 1, token->shared_bitmap, NULL); token->shared_count++; log_token(token, "mode_block[%d] shared %d gen %llu", i, token->shared_count, (unsigned long long)mb.generation); } } } /* * If we hang or crash after completing a ballot successfully, but before * committing the leader_record, then the next host that runs a ballot (with the * same lver since we did not commit the new lver to the leader_record) will * commit the same inp values that we were about to commit. If the inp values * they commit indicate we (who crashed or hung) are the new owner, then the * other hosts will begin monitoring the liveness of our host_id. Once enough * time has passed, they assume we're dead, and go on with new versions. The * "enough time" ensures that if we hung before writing the leader, that we * won't wake up and finally write what will then be an old invalid leader. */ /* * acquire io: disk ios needed to acquire a free and uncontended lease * * paxos_lease_acquire() * R1 paxos_lease_read() 1 read 1 MB (lease area size) * [ delta_lease_leader_read() N reads of 512 bytes (if held) ] * run_ballot() * W1 write_dblock() 1 write 512 bytes (1 dblock sector) * R2 read_iobuf() 1 read 1 MB (lease area size) * W2 write_dblock() 1 write 512 bytes (1 dblock sector) * R3 read_iobuf() 1 read 1 MB (lease area size) * W3 write_new_leader() 1 write 512 bytes (1 leader sector) * * 6 i/os = 3 1MB reads, 3 512 byte writes * (assuming 512 sector size and 1MB area size) */ /* * io when using caw * * acquire ex lease * ---------------- * paxos_lease_acquire() * paxos_lease_read() 1 read 1 MB (lease area) * [ delta_lease_leader_read() N reads 512 bytes (delta leader block of owner) ] * compare_and_write() 1 ioctl 512 bytes (leader block) * * release ex lease * ---------------- * paxos_lease_release() * compare_and_write() 1 ioctl 512 bytes (leader block) * * * * acquire sh lease * ---------------- * paxos_lease_acquire() * paxos_lease_read() 1 read 1 MB (lease area) * [ delta_lease_leader_read() N reads 512 bytes (delta leader block of owner) ] * compare_and_write() 1 ioctl 512 bytes (leader block) * write_mblock_shared_dblock_release() 1 write 512 bytes (mode block) * paxos_lease_release() * compare_and_write() 1 ioctl 512 bytes (leader block) * * release sh lease * ---------------- * write_mblock_shared_dblock_release() 1 write 512 bytes (mode block) */ int paxos_lease_acquire(struct task *task, struct token *token, uint32_t flags, struct leader_record *leader_ret, struct paxos_dblock *dblock_ret, uint64_t acquire_lver, int new_num_hosts) { struct sync_disk host_id_disk; struct leader_record host_id_leader; struct leader_record cur_leader; struct leader_record tmp_leader; struct leader_record new_leader; struct leader_record leader_end; struct paxos_dblock dblock; struct paxos_dblock owner_dblock; struct host_status hs; uint32_t checksum; uint64_t wait_start, now; uint64_t last_timestamp; uint64_t next_lver; uint64_t larger_lver = 0; uint64_t max_mbal; uint64_t num_mbal; uint64_t our_mbal; uint64_t other_max_mbal; uint64_t set_dead_ext_gen = 0; int copy_cur_leader; int disk_open = 0; int error, rv, us; int align_size; int ls_sector_size; int is_caw = 0; int other_io_timeout, other_host_dead_seconds; memset(&dblock, 0, sizeof(dblock)); /* shut up compiler */ log_token(token, "paxos_acquire begin offset %llu 0x%x %d %d", (unsigned long long)token->disks[0].offset, flags, token->sector_size, token->align_size); if (!token->sector_size) { log_errot(token, "paxos_acquire with zero sector_size"); return -EINVAL; } restart: memset(&tmp_leader, 0, sizeof(tmp_leader)); copy_cur_leader = 0; /* * acquire io: read 1 (lease area) * * We don't know if this lease uses caw until we read the * leader record and look for LFL_CAW. * * paxos_lease_read checks for the LFL_CAW flag, and if found * sets is_caw = 1. From paxos_lease_read, task->paxos_areabuf * contains the original lease area data that was read. * The caw code below needs the lease area data to check for * mblock shared flags, and for prev_data in compare-and-write. */ error = paxos_lease_read(task, task->use_aio, token, flags, &cur_leader, &max_mbal, "paxos_acquire", 1, &is_caw); if (error < 0) goto out; align_size = leader_align_size_from_flag(cur_leader.flags); if (!align_size) align_size = sector_size_to_align_size_old(cur_leader.sector_size); /* * token sector_size/align_size are initially set from the lockspace values. * It's possible but unusual that the paxos lease leader record will have * different sector/align sizes than we used initially. */ if ((cur_leader.sector_size != token->sector_size) || (align_size != token->align_size)) { log_token(token, "paxos_acquire restart with different sizes was %d %d now %d %d", token->sector_size, token->align_size, cur_leader.sector_size, align_size); token->sector_size = cur_leader.sector_size; token->align_size = align_size; goto restart; } if (is_caw) { token->flags |= T_CAW; token->resource->flags |= R_CAW; if (com.use_compare_and_write == USE_CAW_NO) { error = SANLK_CAW_CONFIG; goto out; } } if (flags & PAXOS_ACQUIRE_FORCE) { copy_cur_leader = 1; goto run; } if (acquire_lver && cur_leader.lver != acquire_lver) { log_errot(token, "paxos_acquire acquire_lver %llu cur_leader %llu", (unsigned long long)acquire_lver, (unsigned long long)cur_leader.lver); error = SANLK_ACQUIRE_LVER; goto out; } if (cur_leader.timestamp == LEASE_FREE) { log_token(token, "paxos_acquire leader %llu free", (unsigned long long)cur_leader.lver); copy_cur_leader = 1; goto run; } if (cur_leader.owner_id == token->host_id && cur_leader.owner_generation == token->host_generation) { log_token(token, "paxos_acquire owner %llu %llu %llu is already local %llu %llu", (unsigned long long)cur_leader.owner_id, (unsigned long long)cur_leader.owner_generation, (unsigned long long)cur_leader.timestamp, (unsigned long long)token->host_id, (unsigned long long)token->host_generation); copy_cur_leader = 1; goto run; } /* * We were the last host to hold this lease, but in a previous * lockspace generation in which we didn't cleanly release the * paxos lease. */ if (cur_leader.owner_id == token->host_id && cur_leader.owner_generation < token->host_generation) { log_token(token, "paxos_acquire owner %llu %llu %llu was old local new is %llu", (unsigned long long)cur_leader.owner_id, (unsigned long long)cur_leader.owner_generation, (unsigned long long)cur_leader.timestamp, (unsigned long long)token->host_generation); copy_cur_leader = 1; goto run; } /* * Check if current owner is alive based on its host_id renewals. * If the current owner has been dead long enough we can assume that * its watchdog has triggered and we can go for the paxos lease. */ if (!disk_open) { memset(&host_id_disk, 0, sizeof(host_id_disk)); rv = lockspace_disk(cur_leader.space_name, &host_id_disk, &ls_sector_size); if (rv < 0) { log_errot(token, "paxos_acquire no lockspace info %.48s", cur_leader.space_name); error = SANLK_ACQUIRE_LOCKSPACE; goto out; } host_id_disk.fd = -1; rv = open_disks_fd(&host_id_disk, 1); if (rv < 0) { log_errot(token, "paxos_acquire open host_id_disk error %d", rv); error = SANLK_ACQUIRE_IDDISK; goto out; } disk_open = 1; } rv = host_info(cur_leader.space_name, cur_leader.owner_id, &hs); if (!rv && hs.last_check && hs.last_live && hs.owner_id == cur_leader.owner_id && hs.owner_generation == cur_leader.owner_generation) { wait_start = hs.last_live; last_timestamp = hs.timestamp; log_token(token, "paxos_acquire owner %llu %llu %llu host_info %llu %llu %llu dead_ext %d wait_start %llu", (unsigned long long)cur_leader.owner_id, (unsigned long long)cur_leader.owner_generation, (unsigned long long)cur_leader.timestamp, (unsigned long long)hs.owner_id, (unsigned long long)hs.owner_generation, (unsigned long long)hs.timestamp, hs.dead_ext, (unsigned long long)wait_start); } else { wait_start = monotime(); last_timestamp = 0; /* Happens if local lockspace has not been running long enough to collect info on other host. */ log_token(token, "paxos_acquire owner %llu %llu %llu host_info rv %d not ready %llu,%llu,%llu,%llu,%llu,%llu,%u wait_start %llu", (unsigned long long)cur_leader.owner_id, (unsigned long long)cur_leader.owner_generation, (unsigned long long)cur_leader.timestamp, rv, (unsigned long long)hs.owner_id, (unsigned long long)hs.owner_generation, (unsigned long long)hs.timestamp, (unsigned long long)hs.first_check, (unsigned long long)hs.last_check, (unsigned long long)hs.last_live, hs.dead_ext, (unsigned long long)wait_start); } /* * If an external entity has said that the owner is dead, * then we don't need to use the delta lease timeout of * the owner as the basis for deciding the owner is dead. * * Note: hs from host_info() was last updated from our * previous renewal, and may not reflect the latest * info from the owner's delta lease. */ if (hs.dead_ext && (hs.owner_generation == cur_leader.owner_generation)) { log_token(token, "paxos_acquire owner %llu %llu %llu " "delta %llu %llu %llu host_status is dead_ext", (unsigned long long)cur_leader.owner_id, (unsigned long long)cur_leader.owner_generation, (unsigned long long)cur_leader.timestamp, (unsigned long long)host_id_leader.owner_id, (unsigned long long)host_id_leader.owner_generation, (unsigned long long)host_id_leader.timestamp); goto run; } /* * If the paxos leader shows gen N and host_id leader shows * gen N-1 with DEAD_EXT, then the host's latest delta lease was * clobbered by another host setting DEAD_EXT with the prev gen, * causing the generation to go back one. Return an error specific * to this case and retry. Eventually we should see the owner delta * lease gen N with DEAD_EXT or gen N+1. */ if (hs.dead_ext && (hs.owner_generation < cur_leader.owner_generation)) { log_errot(token, "paxos_acquire owner %llu %llu %llu reverted delta %llu %llu %llu dead_ext", (unsigned long long)cur_leader.owner_id, (unsigned long long)cur_leader.owner_generation, (unsigned long long)cur_leader.timestamp, (unsigned long long)host_id_leader.owner_id, (unsigned long long)host_id_leader.owner_generation, (unsigned long long)host_id_leader.timestamp); memcpy(leader_ret, &cur_leader, sizeof(struct leader_record)); return SANLK_ACQUIRE_OWNER_GEN_REVERTED; } /* * These would have been set by a local entity calling * set_host with dead_ext on the failed host. If a local * entity has not does this, then eventually we would see * the hs.dead_ext set from another host. */ if (hs.found_dead_ext_gen) set_dead_ext_gen = hs.found_dead_ext_gen; else if (hs.wrote_dead_ext_gen) set_dead_ext_gen = hs.wrote_dead_ext_gen; /* * A local entity called set_host+dead_ext on this host, * so we know it's dead from that, even if our local * renewal thread has not yet seen it and set hs.dead_ext * in check_other_leases. */ if (set_dead_ext_gen && (set_dead_ext_gen == cur_leader.owner_generation)) { log_token(token, "paxos_acquire owner %llu %llu %llu " "delta %llu %llu %llu is dead_ext (found %llu wrote %llu)", (unsigned long long)cur_leader.owner_id, (unsigned long long)cur_leader.owner_generation, (unsigned long long)cur_leader.timestamp, (unsigned long long)host_id_leader.owner_id, (unsigned long long)host_id_leader.owner_generation, (unsigned long long)host_id_leader.timestamp, (unsigned long long)hs.found_dead_ext_gen, (unsigned long long)hs.wrote_dead_ext_gen); goto run; } /* * Loop waiting for the owner of this paxos lease to be dead by timeout. * The paxos lease owner will be dead when the owner's delta lease expires, * indicating that the owner's watchdog has fired. */ while (1) { error = delta_lease_leader_read(task, ls_sector_size, token->io_timeout, &host_id_disk, cur_leader.space_name, cur_leader.owner_id, &host_id_leader, "paxos_acquire"); if (error < 0) { log_errot(token, "paxos_acquire owner %llu %llu %llu " "delta read %d fd %d path %s off %llu", (unsigned long long)cur_leader.owner_id, (unsigned long long)cur_leader.owner_generation, (unsigned long long)cur_leader.timestamp, error, host_id_disk.fd, host_id_disk.path, (unsigned long long)host_id_disk.offset); goto out; } /* a host_id cannot become free in less than host_dead_seconds after the final renewal because a host_id must first be acquired before being freed, and acquiring cannot take less than host_dead_seconds */ if (host_id_leader.timestamp == LEASE_FREE) { log_token(token, "paxos_acquire owner %llu delta free", (unsigned long long)cur_leader.owner_id); goto run; } /* another host has acquired the host_id of the host that owned this paxos lease; acquiring a host_id also cannot be done in less than host_dead_seconds, or the host_id that owns this lease may be alive, but it owned the lease in a previous generation without freeing it, and no longer owns it */ if (host_id_leader.owner_id != cur_leader.owner_id || host_id_leader.owner_generation > cur_leader.owner_generation) { log_token(token, "paxos_acquire owner %llu %llu %llu " "delta %llu %llu %llu mismatch", (unsigned long long)cur_leader.owner_id, (unsigned long long)cur_leader.owner_generation, (unsigned long long)cur_leader.timestamp, (unsigned long long)host_id_leader.owner_id, (unsigned long long)host_id_leader.owner_generation, (unsigned long long)host_id_leader.timestamp); goto run; } if (!last_timestamp) { last_timestamp = host_id_leader.timestamp; goto skip_live_check; } /* * The local renewal thread has not yet read the owner's delta * lease since DEAD_EXT was set, so hs.dead_ext was not seen above. * But we've read the owner's delta lease here, and see it here. */ if ((host_id_leader.flags & LFL_DEAD_EXT) && (host_id_leader.owner_generation == cur_leader.owner_generation)) { log_token(token, "paxos_acquire owner %llu %llu %llu " "delta %llu %llu %llu read flag dead_ext", (unsigned long long)cur_leader.owner_id, (unsigned long long)cur_leader.owner_generation, (unsigned long long)cur_leader.timestamp, (unsigned long long)host_id_leader.owner_id, (unsigned long long)host_id_leader.owner_generation, (unsigned long long)host_id_leader.timestamp); goto run; } /* * Check if the owner is alive: * * 1. We just read the delta lease of the owner (host_id_leader). * If that has a newer timestamp than the timestamp last seen by * our own renewal thread (last_timestamp), then the owner is alive. * * 2. If our own renewal thread saw the owner's timestamp change * the last time it was checked, then consider the owner to be alive. * * first_check != last_live: avoid returning IDLIVE when this * checks the host state soon after add_lockspace, before more * than one check_other_leases() has happened. Before two * checks of the other lease, a dead host and live host look * the same (and first_check == last_live). If the lease owner * is dead before acquire, then acquire will wait for the lease * to expire. If the caller doesn't want acquire to wait for a * lease to expire, they include the NOWAIT flag. */ if ((host_id_leader.timestamp != last_timestamp) || (hs.last_live && (hs.last_check == hs.last_live) && (hs.first_check != hs.last_live))) { log_token(token, "paxos_acquire owner %llu delta %llu %llu %llu alive", (unsigned long long)cur_leader.owner_id, (unsigned long long)host_id_leader.owner_id, (unsigned long long)host_id_leader.owner_generation, (unsigned long long)host_id_leader.timestamp); log_token(token, "paxos_acquire alive host info %llu %llu timestamp %llu first_check %llu last_check %llu last_live %llu", (unsigned long long)hs.owner_id, (unsigned long long)hs.owner_generation, (unsigned long long)hs.timestamp, (unsigned long long)hs.first_check, (unsigned long long)hs.last_check, (unsigned long long)hs.last_live); memcpy(leader_ret, &cur_leader, sizeof(struct leader_record)); /* It's possible that the live owner has released the lease, but its release was clobbered by another host that was running the ballot with it and wrote it as the owner. If the leader writer was not the owner, check if the owner's dblock is cleared. If so, then the owner released the lease and we can run a ballot. Comparing the write_id and owner_id is not required; we could always read the owner dblock here, but comparing the writer and owner can eliminate many unnecessary dblock reads. */ if (!is_caw && (cur_leader.write_id != cur_leader.owner_id)) { rv = read_dblock(task, token, &token->disks[0], cur_leader.owner_id, &owner_dblock); if (!rv && (owner_dblock.flags & DBLOCK_FL_RELEASED)) { /* not an error, but interesting to see */ log_warnt(token, "paxos_acquire owner %llu %llu %llu writer %llu owner dblock released", (unsigned long long)cur_leader.owner_id, (unsigned long long)cur_leader.owner_generation, (unsigned long long)cur_leader.timestamp, (unsigned long long)cur_leader.write_id); goto run; } } error = SANLK_ACQUIRE_IDLIVE; goto out; } /* If the owner hasn't renewed its host_id lease for host_dead_seconds then its watchdog should have fired by now. */ now = monotime(); other_io_timeout = hs.io_timeout; other_host_dead_seconds = calc_host_dead_seconds(other_io_timeout); if (now - wait_start > other_host_dead_seconds) { /* * This host would be considered dead if it weren't using NO_TIMEOUT, * and no dead_ext has been set for it yet allowing us to procede. */ if (host_id_leader.flags & LFL_NO_TIMEOUT) { log_token(token, "paxos_acquire owner %llu %llu %llu " "delta %llu %llu %llu no_timeout_dead %llu-%llu>%d", (unsigned long long)cur_leader.owner_id, (unsigned long long)cur_leader.owner_generation, (unsigned long long)cur_leader.timestamp, (unsigned long long)host_id_leader.owner_id, (unsigned long long)host_id_leader.owner_generation, (unsigned long long)host_id_leader.timestamp, (unsigned long long)now, (unsigned long long)wait_start, other_host_dead_seconds); memcpy(leader_ret, &cur_leader, sizeof(struct leader_record)); error = SANLK_ACQUIRE_OWNED_NO_TIMEOUT; goto out; } log_token(token, "paxos_acquire owner %llu %llu %llu " "delta %llu %llu %llu dead %llu-%llu>%d", (unsigned long long)cur_leader.owner_id, (unsigned long long)cur_leader.owner_generation, (unsigned long long)cur_leader.timestamp, (unsigned long long)host_id_leader.owner_id, (unsigned long long)host_id_leader.owner_generation, (unsigned long long)host_id_leader.timestamp, (unsigned long long)now, (unsigned long long)wait_start, other_host_dead_seconds); goto run; } if (flags & PAXOS_ACQUIRE_OWNER_NOWAIT) { log_token(token, "paxos_acquire owner %llu %llu %llu no wait", (unsigned long long)cur_leader.owner_id, (unsigned long long)cur_leader.owner_generation, (unsigned long long)cur_leader.timestamp); memcpy(leader_ret, &cur_leader, sizeof(struct leader_record)); error = SANLK_ACQUIRE_OWNED_RETRY; goto out; } skip_live_check: /* TODO: test with sleep(2) here */ sleep(1); if (external_shutdown) { error = -1; goto out; } /* * In this while loop we are waiting for an indication that the * current owner is alive or dead, but if we see the leader * owner change in the meantime, we'll restart the entire * process. */ error = paxos_lease_leader_read(task, task->use_aio, token, &tmp_leader, "paxos_acquire"); if (error < 0) goto out; if (memcmp(&cur_leader, &tmp_leader, sizeof(struct leader_record))) { log_token(token, "paxos_acquire restart leader changed1 from " "%llu %llu %llu to %llu %llu %llu", (unsigned long long)cur_leader.owner_id, (unsigned long long)cur_leader.owner_generation, (unsigned long long)cur_leader.timestamp, (unsigned long long)tmp_leader.owner_id, (unsigned long long)tmp_leader.owner_generation, (unsigned long long)tmp_leader.timestamp); goto restart; } } /* end of loop waiting for the owner to be dead */ run: if (is_caw) { memcpy(&new_leader, &cur_leader, sizeof(struct leader_record)); new_leader.lver += 1; new_leader.owner_id = token->host_id; new_leader.owner_generation = token->host_generation; new_leader.timestamp = monotime(); new_leader.write_id = token->host_id; new_leader.write_generation = token->host_generation; new_leader.write_timestamp = new_leader.timestamp; if (new_num_hosts) new_leader.num_hosts = new_num_hosts; if (flags & PAXOS_ACQUIRE_SHARED) new_leader.flags |= LFL_SHORT_HOLD; else new_leader.flags &= ~LFL_SHORT_HOLD; new_leader.checksum = 0; /* set after leader_record_out */ leader_record_out(&new_leader, &leader_end); checksum = leader_checksum(&leader_end); new_leader.checksum = checksum; leader_end.checksum = cpu_to_le32(checksum); /* acquire io: compare-and-write 1 */ error = token_acquire_compare_and_write(task, token, task->paxos_areabuf, &leader_end); if (error == SANLK_CAW_MISCOMPARE) { log_token(token, "paxos_acquire %llu caw miscompare", (unsigned long long)new_leader.lver); goto restart; } if (error < 0) { log_errot(token, "paxos_acquire %llu caw error %d", (unsigned long long)new_leader.lver, error); goto out; } #if 0 /* verify compare and write worked */ usleep(100); rv = paxos_lease_leader_read(task, task->use_aio, token, &tmp_leader, "paxos_caw_acquire_verify"); if (rv < 0) { log_errot(token, "paxos_caw_acquire_verify failed to read leader %d", rv); goto out; } if ((tmp_leader.owner_id != token->host_id) || (tmp_leader.owner_generation != token->host_generation) || (tmp_leader.timestamp == LEASE_FREE)) { log_errot(token, "paxos_caw_acquire_verify wrong leader %llu %llu %llu expect %llu %llu %llu", (unsigned long long)tmp_leader.owner_id, (unsigned long long)tmp_leader.owner_generation, (unsigned long long)tmp_leader.timestamp, (unsigned long long)new_leader.owner_id, (unsigned long long)new_leader.owner_generation, (unsigned long long)new_leader.timestamp); } #endif /* caw alternative to run_ballot/check_mode_block */ check_mode_blocks(token, new_leader.num_hosts, task->paxos_areabuf); memcpy(leader_ret, &new_leader, sizeof(struct leader_record)); error = SANLK_OK; goto out; } /* * Use the disk paxos algorithm to attempt to commit a new leader. * * If we complete a ballot successfully, we can commit a leader record * with next_lver. If we find a higher mbal during a ballot, we increase * our own mbal and try the ballot again. * * next_lver is derived from cur_leader with a zero or timed out owner. * We need to monitor the leader record to see if another host commits * a new leader_record with next_lver. * * TODO: may not need to increase mbal if dblock.inp and inp2 match * current host_id and generation? */ /* This next_lver assignment is based on the original cur_leader, not a re-reading of the leader here, i.e. we cannot just re-read the leader here, and make next_lver one more than that. This is because another node may have made us the owner of next_lver as it is now. */ next_lver = cur_leader.lver + 1; if (!max_mbal) { our_mbal = token->host_id; } else { num_mbal = max_mbal - (max_mbal % cur_leader.max_hosts); our_mbal = num_mbal + cur_leader.max_hosts + token->host_id; } retry_ballot: if (copy_cur_leader) { /* reusing the initial read removes an iop in the common case */ copy_cur_leader = 0; memcpy(&tmp_leader, &cur_leader, sizeof(struct leader_record)); } else { /* acquire io: read 1 (for retry) */ error = paxos_lease_leader_read(task, task->use_aio, token, &tmp_leader, "paxos_acquire"); if (error < 0) goto out; } if (tmp_leader.lver == next_lver) { /* * another host has committed a leader_record for next_lver, * check which inp (owner_id) they committed (possibly us). */ if (tmp_leader.owner_id == token->host_id && tmp_leader.owner_generation == token->host_generation) { /* not a problem, but interesting to see */ log_warnt(token, "paxos_acquire %llu owner is our inp " "%llu %llu %llu committed by %llu", (unsigned long long)next_lver, (unsigned long long)tmp_leader.owner_id, (unsigned long long)tmp_leader.owner_generation, (unsigned long long)tmp_leader.timestamp, (unsigned long long)tmp_leader.write_id); memcpy(leader_ret, &tmp_leader, sizeof(struct leader_record)); memcpy(dblock_ret, &dblock, sizeof(struct paxos_dblock)); error = SANLK_OK; } else { /* not a problem, but interesting to see */ log_warnt(token, "paxos_acquire %llu owner is %llu %llu %llu", (unsigned long long)next_lver, (unsigned long long)tmp_leader.owner_id, (unsigned long long)tmp_leader.owner_generation, (unsigned long long)tmp_leader.timestamp); memcpy(leader_ret, &tmp_leader, sizeof(struct leader_record)); error = SANLK_ACQUIRE_OWNED; } goto out; } if (tmp_leader.lver > next_lver) { /* * A case where this was observed: for next_lver 65 we abort1, and delay. * While sleeping, the lease v65 (which was acquired during our abort1) is * released and then reacquired as v66. When we goto retry_ballot, our * next_lver is 65, but the current lver on disk is 66, causing us to * we fail in the larger1 check.) */ log_token(token, "paxos_acquire %llu restart new lver %llu from " "%llu %llu %llu to %llu %llu %llu", (unsigned long long)next_lver, (unsigned long long)tmp_leader.lver, (unsigned long long)cur_leader.owner_id, (unsigned long long)cur_leader.owner_generation, (unsigned long long)cur_leader.timestamp, (unsigned long long)tmp_leader.owner_id, (unsigned long long)tmp_leader.owner_generation, (unsigned long long)tmp_leader.timestamp); goto restart; } if (memcmp(&cur_leader, &tmp_leader, sizeof(struct leader_record))) { log_token(token, "paxos_acquire %llu restart leader changed2 from " "%llu %llu %llu to %llu %llu %llu", (unsigned long long)next_lver, (unsigned long long)cur_leader.owner_id, (unsigned long long)cur_leader.owner_generation, (unsigned long long)cur_leader.timestamp, (unsigned long long)tmp_leader.owner_id, (unsigned long long)tmp_leader.owner_generation, (unsigned long long)tmp_leader.timestamp); goto restart; } other_max_mbal = 0; /* * Another host wrote an lver into their dblock, and then failed before * that lver could be committed to the leader record. Advance the next * lver past the larger lver last seen in a dblock, otherwise ballots * will continue to be aborted. */ if (larger_lver > next_lver) { log_warnt(token, "paxos_acquire larger lver %llu found beyond next lver %llu", (unsigned long long)larger_lver, (unsigned long long)next_lver); next_lver = larger_lver + 1; } /* acquire io: write 1 (sector), read 2 (lease area), write 2 (sector), read 3 (lease area) */ error = run_ballot(task, token, flags, cur_leader.num_hosts, next_lver, our_mbal, &dblock, &other_max_mbal, &larger_lver); if ((error == SANLK_DBLOCK_MBAL) || (error == SANLK_DBLOCK_LVER)) { us = get_rand(0, 1000000); if (us < 0) us = token->host_id * 100; log_token(token, "paxos_acquire %llu retry delay %d us", (unsigned long long)next_lver, us); usleep(us); /* * If other hosts have advanced max_mbal far beyond where we're at, then * skip ahead to the base max_mbal seen in the last ballot, rather than * adding just adding max_hosts to our_mbal repeatedly until we catch up. * (Note: run_ballot returns the first larger mbal it finds, and does not * scan all dblocks for the largest mbal that exists. So this doesn't * always skip ahead as far as it could. All dblocks could be scanned * to find the largest mbal, but it would be a bigger code change.) */ if (other_max_mbal > (our_mbal + (2 * cur_leader.max_hosts))) { uint64_t other_host_id, other_base_mbal, our_next_mbal; other_host_id = other_max_mbal % cur_leader.max_hosts; other_base_mbal = other_max_mbal - other_host_id; our_next_mbal = other_base_mbal + token->host_id; log_warnt(token, "paxos_acquire %llu other_max_mbal %llu advance our_mbal from %llu to %llu", (unsigned long long)next_lver, (unsigned long long)other_max_mbal, (unsigned long long)our_mbal, (unsigned long long)our_next_mbal); our_mbal = our_next_mbal; } else { our_mbal += cur_leader.max_hosts; } goto retry_ballot; } if (error < 0) { log_errot(token, "paxos_acquire %llu ballot error %d", (unsigned long long)next_lver, error); goto out; } /* ballot success, commit next_lver with dblock values */ memcpy(&new_leader, &cur_leader, sizeof(struct leader_record)); new_leader.lver = dblock.lver; new_leader.owner_id = dblock.inp; new_leader.owner_generation = dblock.inp2; new_leader.timestamp = dblock.inp3; new_leader.write_id = token->host_id; new_leader.write_generation = token->host_generation; new_leader.write_timestamp = monotime(); if (new_num_hosts) new_leader.num_hosts = new_num_hosts; if (new_leader.owner_id == token->host_id) { /* * The LFL_SHORT_HOLD flag is just a "hint" to help * other nodes be more intelligent about retrying * due to transient failures when acquiring shared * leases. Only modify SHORT_HOLD if we're committing * ourself as the new owner. If we're committing another * host as owner, we don't know if they are acquiring * shared or not. * * When acquiring a shared lease, we acquire the paxos * lease (which is inherently ex), set the shared flag * in our mode_block, then release the paxos lease. * The SHORT flag indicates that this paxos lease will * be released as soon as our mode_block is written. * to hold the lock for a long period (e.g. for an LV.) */ if (flags & PAXOS_ACQUIRE_SHARED) new_leader.flags |= LFL_SHORT_HOLD; else new_leader.flags &= ~LFL_SHORT_HOLD; } new_leader.checksum = 0; /* set after leader_record_out */ /* acquire io: write 3 (sector) */ error = write_new_leader(task, token, &new_leader, "paxos_acquire"); if (error < 0) { /* See comment in run_ballot about this flag. */ token->flags |= T_RETRACT_PAXOS; memcpy(leader_ret, &new_leader, sizeof(struct leader_record)); goto out; } if (new_leader.owner_id != token->host_id) { /* not a problem, but interesting to see */ /* It's possible that we commit an outdated owner id/gen here. If we go back to the top and retry, we may find that the owner host_id is alive but with a newer generation, and we'd be able to get the lease by running the ballot again. */ log_warnt(token, "ballot %llu commit other owner %llu %llu %llu", (unsigned long long)new_leader.lver, (unsigned long long)new_leader.owner_id, (unsigned long long)new_leader.owner_generation, (unsigned long long)new_leader.timestamp); memcpy(leader_ret, &new_leader, sizeof(struct leader_record)); error = SANLK_ACQUIRE_OTHER; goto out; } log_token(token, "ballot %llu commit self owner %llu %llu %llu", (unsigned long long)next_lver, (unsigned long long)new_leader.owner_id, (unsigned long long)new_leader.owner_generation, (unsigned long long)new_leader.timestamp); memcpy(leader_ret, &new_leader, sizeof(struct leader_record)); memcpy(dblock_ret, &dblock, sizeof(struct paxos_dblock)); error = SANLK_OK; out: if (disk_open) close_disks(&host_id_disk, 1); return error; } #if 0 int paxos_lease_renew(struct task *task, struct token *token, struct leader_record *leader_last, struct leader_record *leader_ret) { struct leader_record new_leader; int rv, d; int error; for (d = 0; d < token->r.num_disks; d++) { memset(&new_leader, 0, sizeof(struct leader_record)); rv = read_leader(task, task->use_aio, token, &token->disks[d], &new_leader); if (rv < 0) continue; if (memcmp(&new_leader, leader_last, sizeof(struct leader_record))) { log_errot(token, "leader changed between renewals"); return SANLK_BAD_LEADER; } } new_leader.timestamp = monotime(); new_leader.checksum = 0; /* set after leader_record_out */ error = write_new_leader(task, token, &new_leader); if (error < 0) goto out; memcpy(leader_ret, &new_leader, sizeof(struct leader_record)); out: return error; } #endif int paxos_caw_lease_release(struct task *task, struct token *token, struct sanlk_resource *resrename, struct leader_record *leader_ret) { struct leader_record new_leader; struct leader_record leader_end; int sector_size = token->sector_size; char *iobuf = NULL, **p_iobuf = NULL; char *rdbuf = NULL, **p_rdbuf = NULL; char *caw_data; uint32_t checksum; int read_data = 0; int not_owned; int rv; /* * caw_data is allocated by token_acquire_compare_and_write(), * and if the acquire is successful, that caw_data buffer is * saved as token->caw_data to be reused here for release. * * When release is done by resource_thread_release(), then * the token arg is a fake/temp token, and token->caw_data * may be set when the caw_data buffer was saved and passed * from the original token->caw_data. Sometimes token->caw_data * may be NULL when this is called by resource_thread_release. * * The caw_data buffer is not passed to aio read/write functions * so that it does not need to be reallocated on AIO_TIMEOUT. */ if (!token->caw_data) { p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), sector_size * 2); if (rv) return -ENOMEM; memset(iobuf, 0, sector_size * 2); caw_data = iobuf; read_data = 1; } else { caw_data = token->caw_data; /* previously written data is in second half, copy to first half */ memmove(caw_data, caw_data + sector_size, sector_size); memcpy(&leader_end, caw_data, sizeof(struct leader_record)); checksum = leader_checksum(&leader_end); leader_record_in(&leader_end, &new_leader); rv = verify_leader(token, &token->disks[0], &new_leader, checksum, "paxos_caw_release"); if (rv < 0) { log_token(token, "paxos_caw_release prev caw_data invalid"); memset(&new_leader, 0, sizeof(new_leader)); read_data = 1; } } /* * CLEANUP_OWNER: uncertain if we are owner, but release if we are. * If we are not the owner, it's not an error. */ if (token->flags & T_CLEANUP_OWNER) read_data = 1; retry: if (read_data) { log_token(token, "paxos_caw_release reading leader sector"); if (!rdbuf) { p_rdbuf = &rdbuf; rv = posix_memalign((void *)p_rdbuf, getpagesize(), sector_size); if (rv) { rv = -ENOMEM; goto out; } } rv = paxos_lease_leader_read_buf(task, task->use_aio, token, &new_leader, rdbuf, "paxos_caw_release"); if (rv == SANLK_AIO_TIMEOUT) { log_errot(token, "paxos_caw_release failed to read leader sector %d", rv); rdbuf = NULL; goto out; } if (rv < 0) { log_errot(token, "paxos_caw_release failed to read leader sector %d", rv); goto out; } not_owned = ((new_leader.owner_id != token->host_id) || (new_leader.owner_generation != token->host_generation) || (new_leader.timestamp == LEASE_FREE)); if (not_owned && (token->flags & T_CLEANUP_OWNER)) { /* CLEANUP_OWNER: we expect that we might not own the lease, so not an error */ log_warnt(token, "paxos_caw_release cleanup found %llu %llu %llu %llu for token %llu %llu %llu", (unsigned long long)new_leader.owner_id, (unsigned long long)new_leader.owner_generation, (unsigned long long)new_leader.timestamp, (unsigned long long)new_leader.lver, (unsigned long long)token->host_id, (unsigned long long)token->host_generation, (unsigned long long)token->r.lver); rv = SANLK_OK; goto out; } if (not_owned) { log_errot(token, "paxos_caw_release reread wrong lease state %llu %llu %llu %llu token %llu %llu %llu", (unsigned long long)new_leader.owner_id, (unsigned long long)new_leader.owner_generation, (unsigned long long)new_leader.timestamp, (unsigned long long)new_leader.lver, (unsigned long long)token->host_id, (unsigned long long)token->host_generation, (unsigned long long)token->r.lver); rv = SANLK_RELEASE_OWNER; goto out; } /* * the current data has been read into rdbuf, * so update both prev/next parts of caw_data * with rdbuf before udpating the second half * with the new leader. */ memmove(caw_data, rdbuf, sector_size); memmove(caw_data + sector_size, rdbuf, sector_size); } if (resrename) memcpy(new_leader.resource_name, resrename->name, NAME_ID_SIZE); new_leader.timestamp = LEASE_FREE; new_leader.write_id = token->host_id; new_leader.write_generation = token->host_generation; new_leader.write_timestamp = monotime(); new_leader.flags &= ~LFL_SHORT_HOLD; new_leader.checksum = 0; /* set after leader_record_out */ leader_record_out(&new_leader, &leader_end); checksum = leader_checksum(&leader_end); new_leader.checksum = checksum; leader_end.checksum = cpu_to_le32(checksum); memcpy(caw_data + sector_size, &leader_end, sizeof(struct leader_record)); if (!token->caw_dev_fd) { rv = open_token_caw(token); if (rv < 0) { log_errot(token, "paxos_release open caw error %d", rv); rv = SANLK_CAW_OPEN; goto out; } } /* * This release can race with another host doing compare and write in acquire: * 1. other: acquire read * 2. local: acquire read * 3. local: acquire caw * 4. local: release caw * 5. other: acquire caw * * steps 4 and 5 can race. step 4 would succeed, step 5 would miscompare. */ rv = compare_and_write(task, token->caw_dev_fd, token->io_timeout, token->resource->caw_major, token->resource->caw_minor, token->sector_size, token->resource->caw_offset, caw_data, NULL, NULL); if (rv == SANLK_CAW_MISCOMPARE) { log_errot(token, "paxos_caw_release miscompare new_leader %llu %llu %llu", (unsigned long long)new_leader.owner_id, (unsigned long long)new_leader.owner_generation, (unsigned long long)new_leader.timestamp); /* * generally shouldn't happen... * a case where our token->caw_data was not up to date with what's on disk, * e.g. we're trying to release after a failed attempt to acquire? */ if (!read_data) { read_data = 1; goto retry; } if (!rdbuf) { rv = SANLK_RELEASE_OWNER; goto out; } /* read here so we can log info about what the state is on disk */ rv = paxos_lease_leader_read_buf(task, task->use_aio, token, &new_leader, rdbuf, "paxos_caw_release"); if (rv == SANLK_AIO_TIMEOUT) { log_errot(token, "paxos_caw_release miscompare read error %d", rv); rdbuf = NULL; } else if (rv < 0) { log_errot(token, "paxos_caw_release miscompare read error %d", rv); } else { log_errot(token, "paxos_caw_release miscompare read %llu %llu %llu writer %llu %llu %llu", (unsigned long long)new_leader.owner_id, (unsigned long long)new_leader.owner_generation, (unsigned long long)new_leader.timestamp, (unsigned long long)new_leader.write_id, (unsigned long long)new_leader.write_generation, (unsigned long long)new_leader.write_timestamp); } rv = SANLK_RELEASE_OWNER; goto out; } if (rv < 0) { log_errot(token, "paxos_caw_release compare_and_write error %d", rv); goto out; } log_token(token, "paxos_caw_release done"); memcpy(leader_ret, &new_leader, sizeof(struct leader_record)); rv = SANLK_OK; out: /* iobuf/caw_data is not used for aio which might time out */ /* set to NULL above for io timeout so we don't free it */ if (rdbuf) free(rdbuf); /* this function allocated caw_data because the token had none */ if (iobuf) free(iobuf); /* caw_data is also freed by free_token in case this function isn't called */ if (token->caw_data) { free(token->caw_data); token->caw_data = NULL; } return rv; } int paxos_lease_release(struct task *task, struct token *token, struct sanlk_resource *resrename, struct leader_record *leader_last, struct leader_record *leader_ret) { struct leader_record leader; struct leader_record *last; int not_owned; int error; if (token->flags & T_CAW) return paxos_caw_lease_release(task, token, resrename, leader_ret); error = paxos_lease_leader_read(task, task->use_aio, token, &leader, "paxos_release"); if (error < 0) { log_errot(token, "paxos_release leader_read error %d", error); goto out; } /* * Used when the caller does not know who the owner is, but * wants to ensure it is not the owner. */ if (!leader_last) last = &leader; else last = leader_last; /* * This will happen when two hosts finish the same ballot * successfully, the second committing the same inp values * that the first did, as it should. But the second will * write its own write_id/gen/timestamp, which will differ * from what the first host wrote. So when the first host * rereads here in the release, it will find different * write_id/gen/timestamp from what it wrote. This is * perfectly fine (use log warn since it's interesting * to see when this happens.) * * If another host was the writer and committed us as the * owner, then we don't zero the leader record when we release, * we just release our dblock (by setting the release flag, * already done prior to calling paxos_lease_release). This is * because other hosts will ignore our leader record if we were * not the writer once we release our dblock. Those other * hosts will then run a ballot and commit/write a new leader. * If we are also zeroing the leader, that can race with * another host writing a new leader, and we could clobber the * new leader. */ if (leader.write_id != token->host_id) { log_warnt(token, "paxos_release skip write " "last lver %llu owner %llu %llu %llu writer %llu %llu %llu " "disk lver %llu owner %llu %llu %llu writer %llu %llu %llu", (unsigned long long)last->lver, (unsigned long long)last->owner_id, (unsigned long long)last->owner_generation, (unsigned long long)last->timestamp, (unsigned long long)last->write_id, (unsigned long long)last->write_generation, (unsigned long long)last->write_timestamp, (unsigned long long)leader.lver, (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation, (unsigned long long)leader.timestamp, (unsigned long long)leader.write_id, (unsigned long long)leader.write_generation, (unsigned long long)leader.write_timestamp); return 0; } not_owned = ((leader.owner_id != token->host_id) || (leader.owner_generation != token->host_generation) || (leader.timestamp == LEASE_FREE)); /* * CLEANUP_OWNER indicates the caller isn't sure if we are the * owner (likely due to a previous io timeout/error), and wants * to release if we are. */ if (not_owned && (token->flags & T_CLEANUP_OWNER)) { log_warnt(token, "paxos_release cleanup found %llu %llu %llu %llu for token %llu %llu", (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation, (unsigned long long)leader.timestamp, (unsigned long long)leader.lver, (unsigned long long)token->host_id, (unsigned long long)token->host_generation); return 0; } /* * When we were the writer of our own leader record, then * releasing the lease includes both setting the RELEASED flag * in our dblock and clearing out timestamp in the leader. * When we reread the leader here in release, we should find * it the same as we last saw in acquire. */ if (leader.lver != last->lver) { log_errot(token, "paxos_release other lver " "last lver %llu owner %llu %llu %llu writer %llu %llu %llu " "disk lver %llu owner %llu %llu %llu writer %llu %llu %llu", (unsigned long long)last->lver, (unsigned long long)last->owner_id, (unsigned long long)last->owner_generation, (unsigned long long)last->timestamp, (unsigned long long)last->write_id, (unsigned long long)last->write_generation, (unsigned long long)last->write_timestamp, (unsigned long long)leader.lver, (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation, (unsigned long long)leader.timestamp, (unsigned long long)leader.write_id, (unsigned long long)leader.write_generation, (unsigned long long)leader.write_timestamp); return SANLK_RELEASE_LVER; } if (leader.timestamp == LEASE_FREE) { log_errot(token, "paxos_release already free " "last lver %llu owner %llu %llu %llu writer %llu %llu %llu " "disk lver %llu owner %llu %llu %llu writer %llu %llu %llu", (unsigned long long)last->lver, (unsigned long long)last->owner_id, (unsigned long long)last->owner_generation, (unsigned long long)last->timestamp, (unsigned long long)last->write_id, (unsigned long long)last->write_generation, (unsigned long long)last->write_timestamp, (unsigned long long)leader.lver, (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation, (unsigned long long)leader.timestamp, (unsigned long long)leader.write_id, (unsigned long long)leader.write_generation, (unsigned long long)leader.write_timestamp); return SANLK_RELEASE_OWNER; } if (leader.owner_id != token->host_id || leader.owner_generation != token->host_generation) { log_errot(token, "paxos_release other owner " "last lver %llu owner %llu %llu %llu writer %llu %llu %llu " "disk lver %llu owner %llu %llu %llu writer %llu %llu %llu", (unsigned long long)last->lver, (unsigned long long)last->owner_id, (unsigned long long)last->owner_generation, (unsigned long long)last->timestamp, (unsigned long long)last->write_id, (unsigned long long)last->write_generation, (unsigned long long)last->write_timestamp, (unsigned long long)leader.lver, (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation, (unsigned long long)leader.timestamp, (unsigned long long)leader.write_id, (unsigned long long)leader.write_generation, (unsigned long long)leader.write_timestamp); return SANLK_RELEASE_OWNER; } if (memcmp(&leader, last, sizeof(struct leader_record))) { log_errot(token, "paxos_release different vals " "last lver %llu owner %llu %llu %llu writer %llu %llu %llu " "disk lver %llu owner %llu %llu %llu writer %llu %llu %llu", (unsigned long long)last->lver, (unsigned long long)last->owner_id, (unsigned long long)last->owner_generation, (unsigned long long)last->timestamp, (unsigned long long)last->write_id, (unsigned long long)last->write_generation, (unsigned long long)last->write_timestamp, (unsigned long long)leader.lver, (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation, (unsigned long long)leader.timestamp, (unsigned long long)leader.write_id, (unsigned long long)leader.write_generation, (unsigned long long)leader.write_timestamp); return SANLK_RELEASE_OWNER; } if (resrename) memcpy(leader.resource_name, resrename->name, NAME_ID_SIZE); leader.timestamp = LEASE_FREE; leader.write_id = token->host_id; leader.write_generation = token->host_generation; leader.write_timestamp = monotime(); leader.flags &= ~LFL_SHORT_HOLD; leader.checksum = 0; /* set after leader_record_out */ error = write_new_leader(task, token, &leader, "paxos_release"); if (error < 0) goto out; memcpy(leader_ret, &leader, sizeof(struct leader_record)); out: return error; } int paxos_lease_init(struct task *task, struct token *token, int num_hosts, int write_clear, int try_caw) { char *iobuf, **p_iobuf; struct leader_record leader; struct leader_record leader_end; struct request_record rr; struct request_record rr_end; uint32_t checksum; int iobuf_len; int sector_size = 0; int align_size = 0; int max_hosts = 0; int aio_timeout = 0; int write_io_timeout = 0; int use_caw = try_caw; int rv, d; rv = sizes_from_flags(token->r.flags, §or_size, &align_size, &max_hosts, "RES"); if (rv) return rv; if (!sector_size) { /* sector/align flags were not set, use historical defaults */ sector_size = token->disks[0].sector_size; align_size = sector_size_to_align_size_old(sector_size); max_hosts = DEFAULT_MAX_HOSTS; } if (try_caw && (compare_and_write_test(task, token->disks[0].path, token->disks[0].offset, sector_size, com.paxos_caw_verify) < 0)) use_caw = 0; if (!num_hosts || (num_hosts > max_hosts)) num_hosts = max_hosts; token->sector_size = sector_size; token->align_size = align_size; iobuf_len = align_size; p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), iobuf_len); if (rv) return rv; memset(iobuf, 0, iobuf_len); memset(&leader, 0, sizeof(leader)); if (write_clear) { leader.magic = PAXOS_DISK_CLEAR; leader.write_timestamp = monotime(); } else { leader.magic = PAXOS_DISK_MAGIC; } if (com.create_paxos_64) leader.version = PAXOS_DISK_VERSION_64; else leader.version = PAXOS_DISK_VERSION_MAJOR | PAXOS_DISK_VERSION_MINOR; leader.flags = leader_align_flag_from_size(align_size); if (use_caw) leader.flags |= LFL_CAW; leader.sector_size = sector_size; leader.num_hosts = num_hosts; leader.max_hosts = max_hosts; leader.timestamp = LEASE_FREE; memcpy(leader.space_name, token->r.lockspace_name, NAME_ID_SIZE); memcpy(leader.resource_name, token->r.name, NAME_ID_SIZE); leader.checksum = 0; /* set after leader_record_out */ memset(&rr, 0, sizeof(rr)); rr.magic = REQ_DISK_MAGIC; rr.version = REQ_DISK_VERSION_MAJOR | REQ_DISK_VERSION_MINOR; leader_record_out(&leader, &leader_end); /* * N.B. must compute checksum after the data has been byte swapped. */ checksum = leader_checksum(&leader_end); leader.checksum = checksum; leader_end.checksum = cpu_to_le32(checksum); request_record_out(&rr, &rr_end); memcpy(iobuf, &leader_end, sizeof(struct leader_record)); memcpy(iobuf + sector_size, &rr_end, sizeof(struct request_record)); /* * The process of initializing the lease on disk can use a * longer timeout than the algorithm uses. */ if (com.write_init_io_timeout) write_io_timeout = com.write_init_io_timeout; for (d = 0; d < token->r.num_disks; d++) { if (!write_io_timeout) write_io_timeout = token->io_timeout; rv = write_iobuf(token->disks[d].fd, token->disks[d].offset, iobuf, iobuf_len, task, task->use_aio, write_io_timeout, NULL); if (rv == SANLK_AIO_TIMEOUT) aio_timeout = 1; if (rv < 0) { if (!aio_timeout) { free(iobuf); iobuf = NULL; } return rv; } } if (!aio_timeout) free(iobuf); return 0; } sanlock/src/paxos_lease.h000066400000000000000000000061611516326262600157700ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __PAXOS_LEASE_H__ #define __PAXOS_LEASE_H__ #define PAXOS_ACQUIRE_FORCE 0x00000001 #define PAXOS_ACQUIRE_QUIET_FAIL 0x00000002 #define PAXOS_ACQUIRE_SHARED 0x00000004 #define PAXOS_ACQUIRE_OWNER_NOWAIT 0x00000008 #define PAXOS_ACQUIRE_DEBUG_ALL 0x00000010 uint32_t leader_checksum(struct leader_record *lr); uint32_t dblock_checksum(struct paxos_dblock *pd); int paxos_lease_leader_read(struct task *task, int use_aio, struct token *token, struct leader_record *leader_ret, const char *caller); int paxos_lease_leader_read_buf(struct task *task, int use_aio, struct token *token, struct leader_record *leader_ret, char *buf, const char *caller); int paxos_lease_acquire(struct task *task, struct token *token, uint32_t flags, struct leader_record *leader_ret, struct paxos_dblock *dblock_ret, uint64_t acquire_lver, int new_num_hosts); int paxos_lease_release(struct task *task, struct token *token, struct sanlk_resource *resrename, struct leader_record *leader_last, struct leader_record *leader_ret); int paxos_lease_init(struct task *task, struct token *token, int num_hosts, int write_clear, int try_caw); int open_token_caw(struct token *token); void close_token_caw(struct token *token); int paxos_lease_request_read(struct task *task, struct token *token, struct request_record *rr); int paxos_lease_request_write(struct task *task, struct token *token, struct request_record *rr); int paxos_read_resource(struct task *task, struct token *token, struct sanlk_resource *res); int paxos_read_buf(struct task *task, struct token *token, char **buf_out); int paxos_lease_read(struct task *task, int use_aio, struct token *token, uint32_t flags, struct leader_record *leader_ret, uint64_t *max_mbal, const char *caller, int log_bk_vals, int *is_caw); int paxos_verify_leader(struct token *token, struct sync_disk *disk, struct leader_record *lr, uint32_t checksum, const char *caller); int paxos_erase_dblock(struct task *task, struct token *token, uint64_t host_id); int paxos_lease_leader_clobber(struct task *task, struct token *token, struct leader_record *leader, const char *caller); int paxos_caw_lease_release(struct task *task, struct token *token, struct sanlk_resource *resrename, struct leader_record *leader_ret); #endif sanlock/src/resource.c000066400000000000000000002601231516326262600153070ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock_internal.h" #include "sanlock_sock.h" #include "diskio.h" #include "ondisk.h" #include "log.h" #include "paxos_lease.h" #include "lockspace.h" #include "resource.h" #include "task.h" #include "timeouts.h" #include "helper.h" /* from cmd.c */ void send_state_resource(int fd, struct resource *r, const char *list_name, int pid, int client_id, uint32_t token_id); /* from main.c */ int get_rand(int a, int b); static struct worker_info rthread_info; static pthread_t resource_pt; static int resource_thread_stop; static int resource_thread_work; static int resource_thread_work_examine; static struct list_head resources_free; static struct list_head resources_held; static struct list_head resources_add; static struct list_head resources_rem; static struct list_head resources_orphan; static pthread_mutex_t resource_mutex; static pthread_cond_t resource_cond; static struct list_head host_events; static int resources_free_count; static uint32_t resource_id_counter = 2; /* id 1 used for internal rindex lease */ #define FREE_RES_COUNT 128 /* * There's not much advantage to saving resource structs and reusing them again * when they are requested again. One advantage can be that the res_id remains * unchanged for frequently requested resources, so a new resource description * isn't logged each time it's requested. There may be some other * optimizations that could be added. We may want per-lockspace lists of * resources, or purge free resources when lockspaces are removed. */ static void free_resource(struct resource *r) { struct resource *rtmp = NULL; struct resource *rmin = NULL; if (r->lvb) { free(r->lvb); r->lvb = NULL; } if (resources_free_count < FREE_RES_COUNT) { resources_free_count++; list_add(&r->list, &resources_free); return; } /* the max are being saved, free the least used before saving this one */ list_for_each_entry_reverse(rtmp, &resources_free, list) { if (!rtmp->reused) { list_del(&rtmp->list); free(rtmp); goto out; } if (!rmin || (rtmp->reused < rmin->reused)) rmin = rtmp; } if (rmin) { list_del(&rmin->list); free(rmin); } out: list_add(&r->list, &resources_free); } static struct resource *get_free_resource(struct token *token, int *token_matches) { struct resource *r; /* find a previous r that matches token */ list_for_each_entry(r, &resources_free, list) { if (memcmp(r->r.lockspace_name, token->r.lockspace_name, SANLK_NAME_LEN)) continue; if (memcmp(r->r.name, token->r.name, SANLK_NAME_LEN)) continue; if (r->r.num_disks != token->r.num_disks) continue; if (strcmp(r->r.disks[0].path, token->r.disks[0].path)) continue; *token_matches = 1; resources_free_count--; list_del(&r->list); r->reused++; return r; } return NULL; } /* N.B. the reporting function looks for the strings "add" and "rem", so if changed, they should be changed in both places. */ void send_state_resources(int fd) { struct resource *r; struct token *token; pthread_mutex_lock(&resource_mutex); list_for_each_entry(r, &resources_held, list) { list_for_each_entry(token, &r->tokens, list) send_state_resource(fd, r, "held", token->pid, token->client_id, token->token_id); } list_for_each_entry(r, &resources_add, list) { list_for_each_entry(token, &r->tokens, list) send_state_resource(fd, r, "add", token->pid, token->client_id, token->token_id); } list_for_each_entry(r, &resources_rem, list) send_state_resource(fd, r, "rem", r->pid, 0, 0); list_for_each_entry(r, &resources_orphan, list) send_state_resource(fd, r, "orphan", r->pid, 0, 0); pthread_mutex_unlock(&resource_mutex); } int read_resource_owners(struct task *task, struct token *token, struct sanlk_resource *res, char **send_buf, int *send_len, int *count) { struct leader_record leader; struct leader_record leader_end; struct mode_block mb; struct sync_disk *disk; struct sanlk_host *host; struct mode_block *mb_end; uint64_t host_id; uint32_t checksum; char *lease_buf_dblock; char *lease_buf = NULL; char *hosts_buf = NULL; const int sector_size_set = token->sector_size != 0; const int align_size_set = token->align_size != 0; int align_size; int host_count = 0; int i, rv; disk = &token->disks[0]; /* If sector size not set, start with the smaller one. */ if (!sector_size_set) token->sector_size = 512; /* If align size not set, default to the older align. */ if (!align_size_set) token->align_size = sector_size_to_align_size_old(token->sector_size); /* we could in-line paxos_read_buf here like we do in read_mode_block */ retry: rv = paxos_read_buf(task, token, &lease_buf); if (rv < 0) { log_errot(token, "read_resource_owners read_buf rv %d", rv); if (lease_buf && (rv != SANLK_AIO_TIMEOUT)) free(lease_buf); return rv; } memcpy(&leader_end, lease_buf, sizeof(struct leader_record)); checksum = leader_checksum(&leader_end); leader_record_in(&leader_end, &leader); align_size = leader_align_size_from_flag(leader.flags); if (!align_size) align_size = sector_size_to_align_size_old(leader.sector_size); /* If caller specified values are incorrect, fail. */ if (sector_size_set && token->sector_size != leader.sector_size) { log_errot(token, "read_resource_owners invalid sector_size: %d actual: %d", token->sector_size, leader.sector_size); rv = -EINVAL; goto out; } if (align_size_set && token->align_size != align_size) { log_errot(token, "read_resource_owners invalid align_size: %d actual: %d", token->align_size, align_size); rv = -EINVAL; goto out; } /* * If the caller did not specify sector size and our guess was wrong, * retry with the actual value */ if ((token->sector_size != leader.sector_size) || (token->align_size != align_size)) { log_debug("read_resource_owners rereading with correct sizes"); token->sector_size = leader.sector_size; token->align_size = align_size; free(lease_buf); lease_buf = NULL; goto retry; } token->sector_size = leader.sector_size; token->align_size = align_size; rv = paxos_verify_leader(token, disk, &leader, checksum, "read_resource_owners"); if (rv < 0) goto out; res->lver = leader.lver; if (leader.timestamp && leader.owner_id) host_count++; for (i = 0; i < leader.num_hosts; i++) { lease_buf_dblock = lease_buf + ((2 + i) * token->sector_size); mb_end = (struct mode_block *)(lease_buf_dblock + MBLOCK_OFFSET); mode_block_in(mb_end, &mb); host_id = i + 1; if (!(mb.flags & MBLOCK_SHARED)) continue; res->flags |= SANLK_RES_SHARED; /* the leader owner has already been counted above; in the ex case it won't have a mode block set */ if (leader.timestamp && leader.owner_id && (host_id == leader.owner_id)) continue; host_count++; } *count = host_count; if (!host_count) { rv = 0; goto out; } hosts_buf = malloc(host_count * sizeof(struct sanlk_host)); if (!hosts_buf) { host_count = 0; rv = -ENOMEM; goto out; } memset(hosts_buf, 0, host_count * sizeof(struct sanlk_host)); host = (struct sanlk_host *)hosts_buf; /* * Usually when leader owner is set, it's an exclusive lock and * we could skip to the end, but if we read while a new shared * owner is being added, we'll see the leader owner set, and * then may see other shared owners in the mode blocks. */ if (leader.timestamp && leader.owner_id) { host->host_id = leader.owner_id; host->generation = leader.owner_generation; host->timestamp = leader.timestamp; host++; } for (i = 0; i < leader.num_hosts; i++) { lease_buf_dblock = lease_buf + ((2 + i) * token->sector_size); mb_end = (struct mode_block *)(lease_buf_dblock + MBLOCK_OFFSET); mode_block_in(mb_end, &mb); host_id = i + 1; if (!(mb.flags & MBLOCK_SHARED)) continue; if (leader.timestamp && leader.owner_id && (host_id == leader.owner_id)) continue; host->host_id = host_id; host->generation = mb.generation; host++; } rv = 0; out: *send_len = host_count * sizeof(struct sanlk_host); *send_buf = hosts_buf; free(lease_buf); return rv; } /* return 1 (other host is alive) to force a failure if we don't have enough knowledge to know it's really not alive. Later we could have this sit and wait (like paxos_lease_acquire) until we have waited long enough or have enough knowledge to say it's safely dead (unless of course we find it is alive while waiting) */ /* check_other_leases is called once each time other leases are successfully read, and sets the host_info fields retrieved here. */ /* * Return 1 if host_id + generation are alive, or if not enough info to say they are dead. * Return 0 if host_id + generation are dead, in which case their lock will be cleared * (a shared lock in this case, by way of clearing the SHARED flag in their mode_block.) * * TODO: improve naming of "clear_dead_shared" and "sh_holder_live". */ static int sh_holder_live(struct token *token, uint64_t host_id, uint64_t mb_gen) { struct host_status hs; int other_io_timeout, other_host_dead_seconds; int rv; /* * host_info() returns what our own renewal thread has observed about * other host_id leases in the lockspace. Our renewal thread could be * delayed renewing our lease, or get io errors reading the lockspace * leases, so we need to be cautious when making judgements about other * host liveness based on this data. * * hs.first_check * Local time when we first looked at the host_id lease. * Will only be zero for a short period before our first call * to check_other_leases(). * While zero, just assume the other host is alive. * * hs.last_check * Local time that we last looked at the host_id lease. * Will only be zero for a short period before our first call * to check_other_leases() * While zero, just assume the other host is alive. * * hs.last_live * Local time that we last looked at the host_id lease, and * saw an updated timestamp (vs the prev check.) * Will be zero until three steps occur: * 1. we renew our lease, and read other lease, read timestamp=X * 2. other host updates its lease, writes timestamp=Y * 3. we renew our lease, and read other lease, read timestamp=Y * because X and Y differ, set last_live to local time. * If a new timestamp is never seen, last_live will remain zero, * in which case the difference between first_check and last_check * can be used to determine the host is dead. * * The local time difference between now and last_live cannot be used * to determine that another host is dead, because this time difference * can become long if we fail to read other leases, i.e. the other * lease may have a newer timestamp and we just haven't read it. * * If we read another host's lease, record the local time we read it * (time 'A'), then reread that same lease later (time 'B'). If the * lease is unchanged (no updated timestamp) between the two reads, * then the time difference between them (B-A) can be compared to * other_host_dead_seconds to determine if that host is dead. * It doesn't matter when our own times A and B occur, or if they * are extended because of our own failure to renew our lease. */ rv = host_info(token->r.lockspace_name, host_id, &hs); if (rv) { log_warnt(token, "acquire conflict sh from host %llu %llu uncertain/LIVE - host_info error %d", (unsigned long long)host_id, (unsigned long long)mb_gen, rv); return 1; } if (!hs.first_check) { log_warnt(token, "acquire conflict sh from host %llu %llu uncertain/LIVE - host_info not yet checked", (unsigned long long)host_id, (unsigned long long)mb_gen); return 1; } if (!hs.last_check) { /* not possible */ log_warnt(token, "acquire conflict sh from host %llu %llu error/LIVE - last_check zero impossible", (unsigned long long)host_id, (unsigned long long)mb_gen); return 1; } /* the host started and wrote its new gen before the local lockspace has had time to see that new gen */ if (hs.owner_generation && (hs.owner_generation < mb_gen)) { log_warnt(token, "acquire conflict sh from host %llu %llu LIVE - old host_info gen %llu timestamp %llu", (unsigned long long)host_id, (unsigned long long)mb_gen, (unsigned long long)hs.owner_generation, (unsigned long long)hs.timestamp); return 1; } if (hs.owner_generation > mb_gen) { log_warnt(token, "acquire conflict sh from host %llu %llu DEAD - new host_info gen %llu timestamp %llu", (unsigned long long)host_id, (unsigned long long)mb_gen, (unsigned long long)hs.owner_generation, (unsigned long long)hs.timestamp); return 0; } if (hs.dead_ext || (hs.found_dead_ext_gen && (hs.found_dead_ext_gen == hs.owner_generation)) || (hs.wrote_dead_ext_gen && (hs.wrote_dead_ext_gen == hs.owner_generation))) { log_warnt(token, "acquire conflict sh from host %llu %llu DEAD_EXT - %u,%llu,%llu", (unsigned long long)host_id, (unsigned long long)mb_gen, hs.dead_ext, (unsigned long long)hs.found_dead_ext_gen, (unsigned long long)hs.wrote_dead_ext_gen); return 0; } /* the host_id lease is free, not being used */ if (!hs.timestamp) { log_warnt(token, "acquire conflict sh from host %llu %llu DEAD - host_info gen %llu timestamp free", (unsigned long long)host_id, (unsigned long long)mb_gen, (unsigned long long)hs.owner_generation); return 0; } if (hs.first_check == hs.last_check) { /* not enough checks to know, assume live */ log_warnt(token, "acquire conflict sh from host %llu %llu uncertain/LIVE - first and last %llu", (unsigned long long)host_id, (unsigned long long)mb_gen, (unsigned long long)hs.first_check); return 1; } if (hs.last_check == hs.last_live) { log_token(token, "acquire conflict sh from host %llu %llu LIVE - last check live %llu", (unsigned long long)host_id, (unsigned long long)mb_gen, (unsigned long long)hs.last_live); return 1; } other_io_timeout = hs.io_timeout; other_host_dead_seconds = calc_host_dead_seconds(other_io_timeout); /* * last_live remains 0 until we see the host write a new timestamp, * which it may never do if it's dead. */ if (!hs.last_live) { if (hs.last_check - hs.first_check > other_host_dead_seconds) { log_warnt(token, "acquire conflict sh from host %llu %llu DEAD - last_check %llu first_check %llu host_dead %u", (unsigned long long)host_id, (unsigned long long)mb_gen, (unsigned long long)hs.last_check, (unsigned long long)hs.first_check, other_host_dead_seconds); return 0; } else { log_token(token, "acquire conflict sh from host %llu %llu uncertain/LIVE - last_check %llu first_check %llu host_dead %u", (unsigned long long)host_id, (unsigned long long)mb_gen, (unsigned long long)hs.last_check, (unsigned long long)hs.first_check, other_host_dead_seconds); return 1; } } /* * last_live is the local time when we last saw a new timestamp. * last_check is the local time when we last read the timestamp. * The host is dead if it has not updated its timestamp in * last_check-last_live. * * Note that 'now' could be much later than last_check if we * have failed to read the lockspace leases (and renew our own * lease.) So, now-last_live is not a correct way to measure * if the other host is dead. */ if (hs.last_check - hs.last_live > other_host_dead_seconds) { log_warnt(token, "acquire conflict sh from host %llu %llu DEAD - last_check %llu last_live %llu host_dead %u", (unsigned long long)host_id, (unsigned long long)mb_gen, (unsigned long long)hs.last_check, (unsigned long long)hs.last_live, other_host_dead_seconds); return 0; } log_token(token, "acquire conflict sh from host %llu %llu LIVE - last_check %llu last_live %llu", (unsigned long long)host_id, (unsigned long long)mb_gen, (unsigned long long)hs.last_check, (unsigned long long)hs.last_live); return 1; } void check_mode_block(struct token *token, uint64_t next_lver, int q, char *dblock_buf) { struct mode_block *mb_end; struct mode_block mb; mb_end = (struct mode_block *)(dblock_buf + MBLOCK_OFFSET); mode_block_in(mb_end, &mb); if (mb.flags & MBLOCK_SHARED) { set_id_bit(q + 1, token->shared_bitmap, NULL); token->shared_count++; log_token(token, "ballot %llu mode[%d] shared %d gen %llu", (unsigned long long)next_lver, q, token->shared_count, (unsigned long long)mb.generation); } } static int write_host_block(struct task *task, struct token *token, uint64_t host_id, uint64_t mb_gen, uint32_t mb_flags, struct paxos_dblock *pd) { struct sync_disk *disk; struct mode_block mb; struct mode_block mb_end; struct paxos_dblock pd_end; char *iobuf, **p_iobuf; uint64_t offset; uint32_t checksum; int iobuf_len, rv; disk = &token->disks[0]; iobuf_len = token->sector_size; if (!iobuf_len) return -EINVAL; p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), iobuf_len); if (rv) return -ENOMEM; memset(iobuf, 0, iobuf_len); /* * When writing our mode block, we need to keep our dblock * values intact because other hosts may be running the * paxos algorithm and these values need to remain intact * for them to reach the correct result. * * Be very careful that the latest/correct copy of our * dblock values are being used here. A paxos ballot * can get confused/stuck if we write the wrong dblock * values. */ if (pd) { if (!pd->mbal || !pd->bal || !pd->inp) { /* * In general this shouldn't happen, but this warning * has been observed in two cases so far: * a. RETRACT_PAXOS error handling (TODO: fix this by * rereading our dblock before writing it back with * zero mblock values ... now fixed?) * b. releasing lease when the last dblock that was * returned from paxos_lease_acquire has zero bal * and zero inp, because our last ballot was * was aborted, and then we detected another host * committed us as the owner. The other host committed * us as the owner using a previous dblock that we wrote * but had aborted. i.e. we did: * 1. run_ballot write dblock with non-zero mbal,bal,inp * 2. abort2 the ballot when we see a larger mbal * 3. run_ballot write dblock with increased non-zero mbal, * bal=0, inp=0 * 4. abort1 the ballot when we see a larger mbal * 5. read leader and see that another host committed * the non-zero mbal,bal,inp we wrote in step 1. * The "pd" we have here is the dblock we wrote in step 3, * that includes zero bal and inp. * I don't believe this is a problem. */ log_warnt(token, "write_host_block with zero dblock values %llu %llu %llu", (unsigned long long)pd->mbal, (unsigned long long)pd->bal, (unsigned long long)pd->inp); } if (pd->inp && (pd->inp != token->host_id)) { /* This should never happen, sanity check. */ log_errot(token, "Ignore bad dblock while writing mblock %llu:%llu:%llu:%llu", (unsigned long long)pd->inp, (unsigned long long)pd->inp2, (unsigned long long)pd->inp3, (unsigned long long)pd->lver); memset(pd, 0, sizeof(struct paxos_dblock)); } else { paxos_dblock_out(pd, &pd_end); } checksum = dblock_checksum(&pd_end); pd->checksum = checksum; pd_end.checksum = cpu_to_le32(checksum); memcpy(iobuf, (char *)&pd_end, sizeof(struct paxos_dblock)); } memset(&mb, 0, sizeof(mb)); mb.flags = mb_flags; mb.generation = mb_gen; mode_block_out(&mb, &mb_end); memcpy(iobuf + MBLOCK_OFFSET, &mb_end, sizeof(struct mode_block)); offset = disk->offset + ((2 + host_id - 1) * token->sector_size); rv = write_iobuf(disk->fd, offset, iobuf, iobuf_len, task, task->use_aio, token->io_timeout, NULL); if (rv < 0) { log_warnt(token, "write_host_block host_id %llu flags %x gen %llu error %d", (unsigned long long)host_id, mb_flags, (unsigned long long)mb_gen, rv); } else { if (pd) log_token(token, "write_host_block host_id %llu flags %x gen %llu dblock %llu:%llu:%llu:%llu:%llu:%llu%s", (unsigned long long)host_id, mb_flags, (unsigned long long)mb_gen, (unsigned long long)pd->mbal, (unsigned long long)pd->bal, (unsigned long long)pd->inp, (unsigned long long)pd->inp2, (unsigned long long)pd->inp3, (unsigned long long)pd->lver, (pd->flags & DBLOCK_FL_RELEASED) ? ":RELEASED." : "."); else log_token(token, "write_host_block host_id %llu flags %x gen %llu", (unsigned long long)host_id, mb_flags, (unsigned long long)mb_gen); } if (rv != SANLK_AIO_TIMEOUT) free(iobuf); return rv; } /* * normal release of sh: * write_host_block() * - set mode_block gen=0 flags=0 (main job, was previously set for shared lease) * - set dblock flags=DBLOCK_FL_RELEASED (should already be set when ex was released, keep it set) * - copy current dblock values so they are unchanged for ongoing ballot on other hosts (non-caw) */ static int write_host_block_release_sh(struct task *task, struct token *token) { struct paxos_dblock dblock; if (token->flags & T_CAW) return write_host_block(task, token, token->host_id, 0, 0, NULL); memcpy(&dblock, &token->resource->dblock, sizeof(dblock)); dblock.flags = DBLOCK_FL_RELEASED; return write_host_block(task, token, token->host_id, 0, 0, &dblock); } /* * normal release of ex: * write_host_block() * - keep mode_block gen=0 flags=0 (should already be 0) * - set dblock flags=DBLOCK_FL_RELEASED to indicate leader is released (main job) * - copy current dblock values so they are unchanged for ongoing ballot on other hosts (non-caw) */ static int write_host_block_release_ex(struct task *task, struct token *token) { struct paxos_dblock dblock; if (token->flags & T_CAW) return write_host_block(task, token, token->host_id, 0, 0, NULL); memcpy(&dblock, &token->resource->dblock, sizeof(dblock)); dblock.flags = DBLOCK_FL_RELEASED; return write_host_block(task, token, token->host_id, 0, 0, &dblock); } /* * normal acquire of sh, * normal convert of ex2sh: * write_host_block() * - set mode_block gen=token->host_generation flags=MBLOCK_SHARED (main job) * - set dblock flags=DBLOCK_FL_RELEASED * - copy current dblock values so they are unchanged for ongoing ballot on other hosts (non-caw) */ static int write_host_block_acquire_sh(struct task *task, struct token *token) { struct paxos_dblock dblock; if (token->flags & T_CAW) return write_host_block(task, token, token->host_id, token->host_generation, MBLOCK_SHARED, NULL); memcpy(&dblock, &token->resource->dblock, sizeof(dblock)); dblock.flags = DBLOCK_FL_RELEASED; return write_host_block(task, token, token->host_id, token->host_generation, MBLOCK_SHARED, &dblock); } static int read_mode_block(struct task *task, struct token *token, uint64_t host_id, struct mode_block *mb_out) { struct sync_disk *disk; struct mode_block *mb_end; struct mode_block mb; char *iobuf, **p_iobuf; uint64_t offset; int num_disks = token->r.num_disks; int iobuf_len, rv, d; disk = &token->disks[0]; iobuf_len = token->sector_size; if (!iobuf_len) return -EINVAL; p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), iobuf_len); if (rv) return -ENOMEM; for (d = 0; d < num_disks; d++) { disk = &token->disks[d]; offset = disk->offset + ((2 + host_id - 1) * token->sector_size); rv = read_iobuf(disk->fd, offset, iobuf, iobuf_len, task, task->use_aio, token->io_timeout, NULL); if (rv < 0) break; mb_end = (struct mode_block *)(iobuf + MBLOCK_OFFSET); mode_block_in(mb_end, &mb); memcpy(mb_out, &mb, sizeof(struct mode_block)); /* FIXME: combine results for multi-disk case */ break; } if (rv != SANLK_AIO_TIMEOUT) free(iobuf); return rv; } static int clear_dead_shared(struct task *task, struct token *token, int num_hosts, int *live_count, uint64_t *first_live_host_id, uint64_t *first_live_host_gen) { struct mode_block mb; uint64_t host_id; int i, rv = 0, live = 0; for (i = 0; i < num_hosts; i++) { host_id = i + 1; if (host_id == token->host_id) continue; if (!test_id_bit(host_id, token->shared_bitmap)) continue; memset(&mb, 0, sizeof(mb)); rv = read_mode_block(task, token, host_id, &mb); if (rv < 0) { log_errot(token, "clear_dead_shared read_mode_block %llu %d", (unsigned long long)host_id, rv); return rv; } log_token(token, "checking mode_block for host %llu %llu 0x%x", (unsigned long long)host_id, (unsigned long long)mb.generation, mb.flags); /* * We get to this function because we saw the shared flag during * paxos, but the holder of the shared lease may have dropped their * shared lease and cleared the mode_block since then. */ if (!(mb.flags & MBLOCK_SHARED)) continue; if (!mb.generation) { /* shouldn't happen; if the shared flag is set, the generation should also be set. */ log_errot(token, "clear_dead_shared host_id %llu mode_block: flags %x gen %llu", (unsigned long long)host_id, mb.flags, (unsigned long long)mb.generation); continue; } if (sh_holder_live(token, host_id, mb.generation)) { log_token(token, "cannot clear shared flag for live host %llu %llu", (unsigned long long)host_id, (unsigned long long)mb.generation); live++; if (live == 1 && first_live_host_id && first_live_host_gen) { *first_live_host_id = host_id; *first_live_host_gen = mb.generation; } continue; } rv = write_host_block(task, token, host_id, 0, 0, NULL); if (rv < 0) { log_errot(token, "failed to clear shared flag for dead host %llu %llu error %d", (unsigned long long)host_id, (unsigned long long)mb.generation, rv); return rv; } log_warnt(token, "cleared shared flag for dead host %llu %llu", (unsigned long long)host_id, (unsigned long long)mb.generation); } *live_count = live; return rv; } /* the lvb is the sector after the dblock for host_id 2000, i.e. 2002 */ #define LVB_SECTOR 2002 static int read_lvb_block(struct task *task, struct token *token) { struct sync_disk *disk; struct resource *r; char *iobuf; uint64_t offset; int iobuf_len, rv; r = token->resource; disk = &token->disks[0]; iobuf_len = token->sector_size; iobuf = r->lvb; offset = disk->offset + (LVB_SECTOR * token->sector_size); if (!r->lvb) return 0; rv = read_iobuf(disk->fd, offset, iobuf, iobuf_len, task, task->use_aio, token->io_timeout, NULL); /* for io timeout, lvb iobuf is freed when the aio completes, not by free_resource */ if (rv == SANLK_AIO_TIMEOUT) r->lvb = NULL; return rv; } static int write_lvb_block(struct task *task, struct resource *r, struct token *token) { struct sync_disk *disk; char *iobuf; uint64_t offset; int iobuf_len, rv; disk = &token->disks[0]; iobuf_len = token->sector_size; iobuf = r->lvb; offset = disk->offset + (LVB_SECTOR * token->sector_size); if (!r->lvb) return 0; rv = write_iobuf(disk->fd, offset, iobuf, iobuf_len, task, task->use_aio, token->io_timeout, NULL); /* for io timeout, lvb iobuf is freed when the aio completes, not by free_resource */ if (rv == SANLK_AIO_TIMEOUT) r->lvb = NULL; return rv; } int res_set_lvb(struct sanlk_resource *res, char *lvb, int lvblen) { struct resource *r; int rv = -ENOENT; pthread_mutex_lock(&resource_mutex); list_for_each_entry(r, &resources_held, list) { if (strncmp(r->r.lockspace_name, res->lockspace_name, NAME_ID_SIZE)) continue; if (strncmp(r->r.name, res->name, NAME_ID_SIZE)) continue; if (!r->lvb) { rv = -EINVAL; break; } if (lvblen > r->leader.sector_size) { rv = -E2BIG; break; } memcpy(r->lvb, lvb, lvblen); r->flags |= R_LVB_WRITE_RELEASE; rv = 0; break; } pthread_mutex_unlock(&resource_mutex); return rv; } int res_get_lvb(struct sanlk_resource *res, char **lvb_out, int *lvblen) { struct resource *r; char *lvb; int rv = -ENOENT; int len = *lvblen; pthread_mutex_lock(&resource_mutex); list_for_each_entry(r, &resources_held, list) { if (strncmp(r->r.lockspace_name, res->lockspace_name, NAME_ID_SIZE)) continue; if (strncmp(r->r.name, res->name, NAME_ID_SIZE)) continue; if (!r->lvb) { rv = -EINVAL; break; } if (!len) len = r->leader.sector_size; lvb = malloc(len); if (!lvb) { rv = -ENOMEM; break; } memcpy(lvb, r->lvb, len); *lvb_out = lvb; *lvblen = len; rv = 0; break; } pthread_mutex_unlock(&resource_mutex); return rv; } /* return < 0 on error, 1 on success */ static int acquire_disk(struct task *task, struct token *token, uint64_t acquire_lver, int new_num_hosts, int owner_nowait, struct leader_record *leader, struct paxos_dblock *dblock) { struct leader_record leader_tmp; int rv; uint32_t flags = 0; if (com.quiet_fail) flags |= PAXOS_ACQUIRE_QUIET_FAIL; if (com.paxos_debug_all) flags |= PAXOS_ACQUIRE_DEBUG_ALL; if (token->acquire_flags & SANLK_RES_SHARED) flags |= PAXOS_ACQUIRE_SHARED; if (owner_nowait) flags |= PAXOS_ACQUIRE_OWNER_NOWAIT; memset(&leader_tmp, 0, sizeof(leader_tmp)); rv = paxos_lease_acquire(task, token, flags, &leader_tmp, dblock, acquire_lver, new_num_hosts); log_token(token, "acquire_disk paxos result %d lver %llu ts %llu", rv, (unsigned long long)leader_tmp.lver, (unsigned long long)leader_tmp.timestamp); memcpy(leader, &leader_tmp, sizeof(struct leader_record)); return rv; /* SANLK_RV */ } /* return < 0 on error, 1 on success */ static int release_disk(struct task *task, struct token *token, struct sanlk_resource *resrename, struct leader_record *leader) { struct leader_record leader_tmp; int rv; if (token->flags & T_CAW) rv = paxos_caw_lease_release(task, token, resrename, &leader_tmp); else rv = paxos_lease_release(task, token, resrename, leader, &leader_tmp); /* log_token(token, "release_disk rv %d", rv); */ if (rv < 0) return rv; memcpy(leader, &leader_tmp, sizeof(struct leader_record)); return rv; /* SANLK_OK */ } /* * This function will: * 1. list_del token from the struct resource (caller frees struct token) * 2. perform on-disk operations to remove this host's ownership of the lease * 3. list_del and free the struct resource * * Normal cases: * * 1. release ex lease * * . zero our dblock values [see *] * (zeroing our mblock at the same time is ok because it's not used) * . Use paxos_lease_release to set LEASE_FREE in leader_record. * . (If r->leader is zero, it implies that the on-disk lease was never * acquired, so all on-disk operations are skipped.) * * 2. release sh lease (R_SHARED is set in r_flags) * * . As a shared lease holder we do not own the leader, so no * change to the leader is needed. * . zero our mblock values (our SHARED flag) * (zeroing our dblock at the same time is ok because it's not used) * * Unusual cases: * * 3. skip all disk operations * * . "nodisk" is used when the caller only needs to remove the token (step 1), * i.e. on an error path prior to any disk operations having been started. * * . the token is being released because the lockspace is failed/dead, * so disk operations are skipped since they'll fail. * * . the token is being released after acquiring the lease failed, * e.g. it was owned by another host. * * 4. try to unwind from failed acquire of a shared lease (R_UNDO_SHARED) * * . A disk operation failed while trying to acquire a shared lease, * so we want to back out and leave the lease unowned. This means * ensuring that our mblock does not have SHARED set and that we * don't own the leader. * . zero our mblock values * . zero our dblock values [see *] * . Use paxos_lease_release to set LEASE_FREE in leader_record. * * 5. try to unwind from failed acquire (R_ERASE_ALL) * * . A disk operation failed at some point while changing a lease, * and we want to clear all ownership/state we have in the lease. * . zero our mblock values * . zero our dblock values [see * and **] * . Use paxos_lease_release to set LEASE_FREE in leader_record. * * (4 and 5 are basically the same and should be combined) * * * Error handling: * * If any on-disk i/o operation times out in step 2, then the struct resource * is moved to the resource_thread for retrying and step 3 is deferred. * The resource_thread will retry the on-disk operations until they succeed, * then free the resource. * * [*] Reason for clearing our dblock when releasing an ex/owned lease: * If we are releasing this lease very quickly after acquiring it, * there's a chance that another host was running the same acquire * ballot that we were and also committed us as the owner of this * lease, writing our inp values to the leader after we did ourself. * That leader write from the other host may happen after the leader * write we will do here releasing ownership. So the release we do * here may be clobbered and lost. The result is that we own the lease * on disk, but don't know it, so it won't be released unless we happen * to acquire and release it again. The solution is that we clear our * dblock in addition to clearing the leader record. Other hosts can * then check our dblock to see if we really do own the lease. If the * leader says we own the lease, but our dblock is cleared, then our * leader write in release was clobbered, and other hosts will run a * ballot to set a new owner. * UPDATE to above: we no longer clear our dblock values because that * can interfere with other hosts running a paxos ballot at the same time, * instead we now set the DBLOCK_FL_RELEASED flag in our dblock, leaving our * other dblock values intact, and other hosts look for this flag to indicate * that we have released. * * [**] For ERASE_ALL we don't want another host running the ballot to select * our dblock values and commit them, making us the owner after we've aborted * the acquire. So, we clear our dblock values first to prevent that from * happening from this point forward. However, another host contending for the * lease at the same time we failed, could already have read our dblock values * from before we cleared them. In the worst case, that host could commit our * dblock values as the new leader, and that new leader write could appear on * disk up to host_dead_seconds later. So it seems that technically we would * need to monitor the leader for up to host_dead_seconds after clearing our * dblock to check if we become the on-disk owner of the lease. The chances * of all this happening seem so remote that we don't do this monitoring. * The best approach to dealing with the ERASE_ALL case is to run a full ballot * again, to ensure there's a known owner, and then release normally from that * state. We don't attempt to queue up an another async ballot in the error * path either because it would get fairly complicated. If the caller wants * to be extra sure that these obscure cases do not leave an orphaned lease * on disk, it can either: * - repeat the acquire call until it does not fail with a timeout, i.e. * rerun the ballot until there's a known owner * - leave and rejoin the lockspace after an acquire times out, which will * invalidate any on-disk lease state */ static int _release_token_cleanup(struct task *task, struct token *token, int *retry) { struct sync_disk *disk; struct leader_record leader_done; struct mode_block mb; struct mode_block mb_end; char *iobuf, **p_iobuf; uint64_t offset; int iobuf_len, rv; disk = &token->disks[0]; iobuf_len = token->sector_size; if (!iobuf_len) return -EINVAL; p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), iobuf_len); if (rv) return -ENOMEM; /* read mblock, if SHARED is set, then clear it */ offset = disk->offset + ((2 + token->host_id - 1) * token->sector_size); rv = read_iobuf(disk->fd, offset, iobuf, iobuf_len, task, task->use_aio, token->io_timeout, NULL); if (rv < 0) { log_warnt(token, "release_token_cleanup read mblock error %d", rv); if (rv != SANLK_AIO_TIMEOUT) free(iobuf); *retry = 1; return rv; } mode_block_in((struct mode_block *)(iobuf + MBLOCK_OFFSET), &mb); if (mb.generation || (mb.flags & MBLOCK_SHARED)) { log_warnt(token, "release_token_cleanup clearing mblock %llu 0x%x", (unsigned long long)mb.generation, mb.flags); mb.generation = 0; mb.flags &= ~MBLOCK_SHARED; mode_block_out(&mb, &mb_end); memcpy(iobuf + MBLOCK_OFFSET, &mb_end, sizeof(struct mode_block)); rv = write_iobuf(disk->fd, offset, iobuf, iobuf_len, task, task->use_aio, token->io_timeout, NULL); if (rv < 0) { log_warnt(token, "release_token_cleanup clear mblock error %d", rv); if (rv != SANLK_AIO_TIMEOUT) free(iobuf); *retry = 1; return rv; } } free(iobuf); /* read leader, if we are owner, then release it */ token->flags |= T_CLEANUP_OWNER; log_warnt(token, "release_token_cleanup releasing leader"); if (token->flags & T_CAW) rv = paxos_caw_lease_release(task, token, NULL, &leader_done); else rv = paxos_lease_release(task, token, NULL, NULL, &leader_done); if (rv == SANLK_AIO_TIMEOUT || rv == -EIO) { log_warnt(token, "release_token_cleanup release leader error %d", rv); *retry = 1; return rv; } if (rv < 0) log_errot(token, "release_token_cleanup release leader error %d", rv); else log_token(token, "release_token_cleanup done"); /* if a read or write times out or has io error, then repeat this function in resource_thread_release until they complete */ *retry = 0; return rv; } static int _release_token(struct task *task, struct token *token, struct sanlk_resource *resrename, int opened, int nodisk) { struct resource *r = token->resource; uint64_t lver; uint32_t r_flags = 0; int retry_async = 0; int last_token = 0; int cleanup_retry = 0; int ret = SANLK_OK; int rv; if (!opened && !nodisk) log_token(token, "release_token"); else log_token(token, "release_token opened=%d nodisk=%d", opened, nodisk); /* We keep r on the resources_rem list while doing the actual release on disk so another acquire for the same resource will see it on the list and fail. we can't have one thread releasing and another acquiring the same resource. While on the rem list, the resource can't be used by anyone. */ pthread_mutex_lock(&resource_mutex); list_del(&token->list); if (list_empty(&r->tokens)) { list_move(&r->list, &resources_rem); last_token = 1; } lver = r->leader.lver; r_flags = r->flags; pthread_mutex_unlock(&resource_mutex); if ((r_flags & R_SHARED) && !last_token) { /* will release when final sh token is released */ log_token(token, "release_token more shared"); close_disks(token->disks, token->r.num_disks); close_token_caw(token); return SANLK_OK; } if (!last_token) { /* should never happen */ log_errot(token, "release_token exclusive not last"); close_disks(token->disks, token->r.num_disks); close_token_caw(token); return SANLK_ERROR; } if (token->space_dead) { /* don't bother trying disk op which will probably timeout */ close_disks(token->disks, token->r.num_disks); goto out; } if (nodisk && opened) { log_token(token, "release_token close only"); close_disks(token->disks, token->r.num_disks); goto out; } if (nodisk) goto out; if (!opened) { rv = open_disks_fd(token->disks, token->r.num_disks); if (rv < 0) { log_errot(token, "release_token open error %d", rv); ret = rv; goto out; } } log_token(token, "release_token t 0x%x r 0x%x %s%s%s%s lver %llu", token->flags, r_flags, (token->flags & T_CLEANUP_CONVERT) ? "CLEANUP_CONVERT," : "", (r_flags & R_ERASE_ALL) ? "ERASE_ALL," : "", (r_flags & R_UNDO_SHARED) ? "UNDO_SHARED," : "", (r_flags & R_SHARED) ? "SHARED" : "", (unsigned long long)lver); /* * Note: CLEANUP_CONVERT can be set in one conversion, and then * any number of subsequent conversions may occur successfully * before release (here), where we see CLEANUP_CONVERT that was * set some time ago and may no longer be necessary. */ /* FIXME: use one flag */ if ((token->flags & T_CLEANUP_CONVERT) || (r_flags & R_ERASE_ALL) || (r_flags & R_UNDO_SHARED)) { rv = _release_token_cleanup(task, token, &cleanup_retry); if (rv < 0) ret = rv; if (cleanup_retry) { retry_async = 1; goto out_close; } } else if (r_flags & R_SHARED) { /* normal release of sh lease */ rv = write_host_block_release_sh(task, token); if (rv < 0) { log_warnt(token, "release_token sh write_host_block error %d", rv); ret = rv; } if (rv == SANLK_AIO_TIMEOUT || rv == -EIO) retry_async = 1; } else { /* normal release of ex lease */ if (!lver) { /* zero lver means acquire did not get to the point of writing a leader, so we don't need to release the lease on disk. */ close_disks(token->disks, token->r.num_disks); goto out; } if (r_flags & R_LVB_WRITE_RELEASE) { rv = write_lvb_block(task, r, token); if (!rv) r->flags &= ~R_LVB_WRITE_RELEASE; else log_warnt(token, "release_token ex write_lvb error %d", rv); /* do we want to give more effort to writing lvb? */ } if (!(token->flags & T_CAW)) { /* Failure here is not a big deal and can be ignored. */ rv = write_host_block_release_ex(task, token); if (rv < 0) log_warnt(token, "release_token ex write_host_block error %d", rv); } rv = release_disk(task, token, resrename, &r->leader); if (rv < 0) { log_warnt(token, "release_token ex lease_release error %d", rv); ret = rv; } if (rv == SANLK_AIO_TIMEOUT || rv == -EIO) retry_async = 1; } out_close: close_disks(token->disks, token->r.num_disks); out: /* * open/close of token->caw_dev_fd: * * acquire ex: * caw_dev_fd opened by open_token_caw() in paxos_lease_acquire() / token_acquire_compare_and_write() * caw_dev_fd closed by close_token_caw() at end of acquire_token() * * release ex: * caw_dev_fd opened by open_token_caw() in paxos_caw_lease_release(), because caw_dev_fd == 0 * caw_dev_fd closed by close_token_caw() at end of release_token() * * acquire sh: * caw_dev_fd opened by open_token_caw() in paxos_lease_acquire() / token_acquire_compare_and_write() * caw_dev_fd reused by paxos_caw_lease_release(), because caw_dev_fd != 0 * caw_dev_fd closed by close_token_caw() at end of release_token() * * release sh: * caw_dev_fd not opened because compare_and_write not used * * resource_thread_release (uses temp/fake token, and does not use release_token) * caw_dev_fd opened by open_token_caw() in paxos_caw_lease_release(), because caw_dev_fd == 0 * caw_dev_fd closed by close_token_caw() at end of resource_thread_release() */ if (token->caw_dev_fd) { close(token->caw_dev_fd); token->caw_dev_fd = 0; } if (!retry_async) { if (ret != SANLK_OK) log_token(token, "release_token error %d 0x%x", ret, r_flags); else log_token(token, "release_token done 0x%x", r_flags); pthread_mutex_lock(&resource_mutex); list_del(&r->list); free_resource(r); pthread_mutex_unlock(&resource_mutex); return ret; } /* * If a transient i/o error prevented the release on disk, * then handle this like an async release; set R_THREAD_RELEASE, * leave r on resources_rem, let resource_thread_release attempt * to release it. We don't want to leave the lease locked on * disk, preventing others from acquiring it. */ log_warnt(token, "release_token set retry for rthread 0x%x", r_flags); pthread_mutex_lock(&resource_mutex); r->flags |= R_THREAD_RELEASE; if (token->flags & T_CLEANUP_CONVERT) r->flags |= R_CLEANUP_CONVERT; pthread_mutex_unlock(&resource_mutex); return ret; } static int release_token_nodisk_opened(struct task *task, struct token *token) { return _release_token(task, token, NULL, 1, 1); } int release_token_nodisk(struct task *task, struct token *token) { return _release_token(task, token, NULL, 0, 1); } static int release_token_opened(struct task *task, struct token *token) { return _release_token(task, token, NULL, 1, 0); } int release_token(struct task *task, struct token *token, struct sanlk_resource *resrename) { return _release_token(task, token, resrename, 0, 0); } /* We're releasing a token from the main thread, in which we don't want to block, so we can't do a real release involving disk io. So, pass the release off to the resource_thread. */ void release_token_async(struct token *token) { struct resource *r = token->resource; pthread_mutex_lock(&resource_mutex); list_del(&token->list); if (list_empty(&r->tokens)) { if (token->space_dead || !r->leader.lver) { if (token->space_dead) log_token(token, "release_token_async space_dead"); if (!r->leader.lver) log_token(token, "release_token_async zero lver"); /* don't bother trying to release if the lockspace is dead (release will probably fail), or the lease was never acquired */ list_del(&r->list); free_resource(r); } else if (token->acquire_flags & SANLK_RES_PERSISTENT) { log_token(token, "release_token_async to orphan"); list_move(&r->list, &resources_orphan); } else { log_token(token, "release_token_async to thread_release"); if (!r->thread_release_caw_data) { r->thread_release_caw_data = token->caw_data; token->caw_data = NULL; } r->flags |= R_THREAD_RELEASE; resource_thread_work = 1; list_move(&r->list, &resources_rem); pthread_cond_signal(&resource_cond); } } pthread_mutex_unlock(&resource_mutex); } static struct resource *find_resource(struct token *token, struct list_head *head) { struct resource *r; list_for_each_entry(r, head, list) { if (strncmp(r->r.lockspace_name, token->r.lockspace_name, NAME_ID_SIZE)) continue; if (strncmp(r->r.name, token->r.name, NAME_ID_SIZE)) continue; return r; } return NULL; } /* * Determines if lockspace is "used" for the purpose of * rem_lockspace(REM_UNUSED). */ int lockspace_is_used(struct sanlk_lockspace *ls) { struct resource *r; pthread_mutex_lock(&resource_mutex); list_for_each_entry(r, &resources_held, list) { if (!strncmp(r->r.lockspace_name, ls->name, NAME_ID_SIZE)) goto yes; } list_for_each_entry(r, &resources_add, list) { if (!strncmp(r->r.lockspace_name, ls->name, NAME_ID_SIZE)) goto yes; } list_for_each_entry(r, &resources_rem, list) { if (!strncmp(r->r.lockspace_name, ls->name, NAME_ID_SIZE)) goto yes; } list_for_each_entry(r, &resources_orphan, list) { if (!strncmp(r->r.lockspace_name, ls->name, NAME_ID_SIZE)) goto yes; } pthread_mutex_unlock(&resource_mutex); return 0; yes: pthread_mutex_unlock(&resource_mutex); return 1; } int resource_orphan_count(char *space_name) { struct resource *r; int count = 0; pthread_mutex_lock(&resource_mutex); list_for_each_entry(r, &resources_orphan, list) { if (!strncmp(r->r.lockspace_name, space_name, NAME_ID_SIZE)) count++; } pthread_mutex_unlock(&resource_mutex); return count; } static void copy_disks(void *dst, void *src, int num_disks) { struct sync_disk *d, *s; int i; d = (struct sync_disk *)dst; s = (struct sync_disk *)src; for (i = 0; i < num_disks; i++) { memcpy(d->path, s->path, SANLK_PATH_LEN); d->offset = s->offset; d->sector_size = s->sector_size; /* fd's are private */ d->fd = -1; d++; s++; } } static struct resource *get_resource(struct token *token, int *new_id) { struct resource *r; int token_matches = 0; uint32_t res_id = 0; uint32_t reused = 0; int disks_len, r_len; disks_len = token->r.num_disks * sizeof(struct sync_disk); r_len = sizeof(struct resource) + disks_len; r = get_free_resource(token, &token_matches); if (r && token_matches) { res_id = r->res_id; reused = r->reused; *new_id = 0; } else if (r) { res_id = resource_id_counter++; *new_id = 1; } else { r = malloc(r_len); if (!r) return NULL; res_id = resource_id_counter++; *new_id = 1; } memset(r, 0, r_len); /* preserved from one use to the next */ r->res_id = res_id; r->reused = reused; memcpy(&r->r, &token->r, sizeof(struct sanlk_resource)); r->io_timeout = token->io_timeout; /* disks copied after open_disks because open_disks sets sector_size which we want copied */ INIT_LIST_HEAD(&r->tokens); r->host_id = token->host_id; r->host_generation = token->host_generation; if (token->acquire_flags & SANLK_RES_SHARED) { r->flags |= R_SHARED; } else { r->pid = token->pid; if (token->flags & T_RESTRICT_SIGKILL) r->flags |= R_RESTRICT_SIGKILL; if (token->flags & T_RESTRICT_SIGTERM) r->flags |= R_RESTRICT_SIGTERM; } return r; } static int convert_sh2ex_token(struct task *task, struct resource *r, struct token *token, uint32_t cmd_flags) { struct leader_record leader; struct paxos_dblock dblock = { 0 }; uint32_t flags = 0; int live_count = 0; int retries = 0; int ret_error; int rv; memset(&leader, 0, sizeof(leader)); if (cmd_flags & SANLK_CONVERT_OWNER_NOWAIT) flags |= PAXOS_ACQUIRE_OWNER_NOWAIT; if (com.quiet_fail) flags |= PAXOS_ACQUIRE_QUIET_FAIL; if (com.paxos_debug_all) flags |= PAXOS_ACQUIRE_DEBUG_ALL; log_token(token, "convert_sh2ex"); /* paxos_lease_acquire modifies these token values, and we check them after */ token->shared_count = 0; memset(token->shared_bitmap, 0, HOSTID_BITMAP_SIZE); /* Using a token flag like this to manipulate the write_dblock to preserve our mblock is ugly. The diskio/paxos/resource layer separations are not quite right, but would take some major effort to change. The flag is needed to prevent the ballot from clobbering our SHARED mblock. Rewriting our mblock after acquire isn't safe because if the paxos acquire doesn't succeed, then we don't hold any lease for a time. */ token->flags |= T_WRITE_DBLOCK_MBLOCK_SH; rv = paxos_lease_acquire(task, token, flags, &leader, &dblock, 0, 0); token->flags &= ~T_WRITE_DBLOCK_MBLOCK_SH; if (rv < 0) { /* If the acquire failed before anything important was written, then the RETRACT flag will not be set, and there is nothing to undo/cleanup; we can simply return an error. Otherwise, the acquire failed part way through, and we need to try to clean up our state on disk. Do on-disk release of owner. Keep token and SH mblock. */ if (!(token->flags & T_RETRACT_PAXOS)) { log_warnt(token, "convert_sh2ex acquire leader error %d", rv); return rv; } token->flags &= ~T_RETRACT_PAXOS; log_errot(token, "convert_sh2ex acquire error %d", rv); ret_error = rv; goto fail; } memcpy(&r->leader, &leader, sizeof(struct leader_record)); memcpy(&r->dblock, &dblock, sizeof(dblock)); token->r.lver = leader.lver; /* paxos_lease_acquire set token->shared_count to the number of SHARED mode blocks it found. It should find at least 1 for our own shared mode block. */ log_token(token, "convert_sh2ex shared_count %d", token->shared_count); if (token->shared_count == 1) goto do_mb; if (!token->shared_count) { /* should never happen */ log_errot(token, "convert_sh2ex zero shared_count"); goto do_mb; } rv = clear_dead_shared(task, token, leader.num_hosts, &live_count, NULL, NULL); if (rv < 0) { /* Do on-disk release of owner. Keep token and SH mblock. */ log_errot(token, "convert_sh2ex clear_dead error %d", rv); ret_error = rv; goto fail; } log_token(token, "convert_sh2ex live_count %d", live_count); if (live_count) { /* * The convert fails because a live host with a sh lock exists. * The token/lease is kept shared, the lease owner is released. * Our SHARED mblock bit is still set on disk because * T_WRITE_DBLOCK_MBLOCK_SH kept it set during acquire, * so we only need to release the lease owner. */ rv = release_disk(task, token, NULL, &leader); if (rv < 0) { /* Do on-disk release of owner. Keep token and SH mblock. */ log_errot(token, "convert_sh2ex release_disk error %d", rv); ret_error = rv; goto fail; } /* standard exit when convert fails due to other shared locks */ return -EAGAIN; } do_mb: if (token->flags & T_CAW) rv = write_host_block(task, token, token->host_id, 0, 0, NULL); else rv = write_host_block(task, token, token->host_id, 0, 0, &dblock); if (rv < 0) { log_errot(token, "convert_sh2ex write_host_block error %d", rv); token->flags |= T_CLEANUP_CONVERT; /* * We have the ex lease, so return success. We just failed to * clear our SH mblock. When we later release this lease, * the release will need to clear the mblock SHARED flag * since it failed here. */ } /* TODO: clean up the duplication of stuff among: t, t->r, r, r->r */ token->r.flags &= ~SANLK_RES_SHARED; token->acquire_flags &= ~SANLK_RES_SHARED; r->r.flags &= ~SANLK_RES_SHARED; r->flags &= ~R_SHARED; return SANLK_OK; /* * Our SHARED flag is still set, but we may or may not hold the * ex lease (are leader owner). If we can release the ex lease * here, then the convert would be a clean failure which leaves * the SHARED flag. If we cannot release the ex lease here, * then we'll have the SHARED flag set, and may or may not hold * the ex lease, which is not a clean failure, leaving the lease * in an improper state. T_CLEANUP_CONVERT is set so that when * the token is eventually released, release_token() will ensure * both mblock SHARED and leader owner are released. * * It's uncertain if we are the leader owner or not. If not then * another host may own the leader, and release may complain about * releasing a lease with a different owner. T_CLEANUP_OWNER * tells releases_lease this might be the case. */ fail: if (token->space_dead) return SANLK_ERROR; token->flags |= T_CLEANUP_OWNER; if (token->flags & T_CAW) rv = paxos_caw_lease_release(task, token, NULL, &leader); else rv = paxos_lease_release(task, token, NULL, leader.lver ? &leader : NULL, &leader); token->flags &= ~T_CLEANUP_OWNER; if ((rv == SANLK_AIO_TIMEOUT || rv == -EIO) && (retries < 1)) { log_errot(token, "convert_sh2ex cleanup release_lease timeout retry %d", retries); retries++; sleep(2); goto fail; } else if (rv < 0) { /* release_token() needs to clear both leader and mblock */ log_errot(token, "convert_sh2ex cleanup release_lease error %d ret_error %d", rv, ret_error); token->flags |= T_CLEANUP_CONVERT; return ret_error; } else { /* We released the owner, so the lease is in a standard state. */ log_errot(token, "convert_sh2ex cleanup done ret_error %d", ret_error); return ret_error; } } static int convert_ex2sh_token(struct task *task, struct resource *r, struct token *token) { struct leader_record leader; int retries = 0; int rv; memcpy(&leader, &r->leader, sizeof(leader)); log_token(token, "convert_ex2sh"); if (r->flags & R_LVB_WRITE_RELEASE) write_lvb_block(task, r, token); rv = write_host_block_acquire_sh(task, token); if (rv < 0) { log_errot(token, "convert_ex2sh write_host_block error %d", rv); token->flags |= T_CLEANUP_CONVERT; return rv; } retry: if (token->space_dead) return SANLK_ERROR; rv = release_disk(task, token, NULL, &leader); if ((rv == SANLK_AIO_TIMEOUT || rv == -EIO) && (retries < 1)) { log_errot(token, "convert_ex2sh release_disk timeout retry %d", retries); retries++; sleep(2); goto retry; } else if (rv < 0) { log_errot(token, "convert_ex2sh release_disk error %d", rv); token->flags |= T_CLEANUP_CONVERT; return rv; } token->r.flags |= SANLK_RES_SHARED; token->acquire_flags |= SANLK_RES_SHARED; r->r.flags |= SANLK_RES_SHARED; r->flags |= R_SHARED; return SANLK_OK; } int convert_token(struct task *task, struct sanlk_resource *res, struct token *cl_token, uint32_t cmd_flags) { struct resource *r; struct token *tk; struct token *token = NULL; int sh_count = 0; int rv; /* we could probably grab cl_token->r, but it's good to verify */ pthread_mutex_lock(&resource_mutex); r = find_resource(cl_token, &resources_held); if (!r) { pthread_mutex_unlock(&resource_mutex); log_error("convert_token resource not found %.48s:%.48s", cl_token->r.lockspace_name, cl_token->r.name); rv = -ENOENT; goto out; } /* find existing token */ list_for_each_entry(tk, &r->tokens, list) { if (tk == cl_token) token = tk; if (tk->acquire_flags & SANLK_RES_SHARED) sh_count++; } pthread_mutex_unlock(&resource_mutex); if (!token) { log_errot(cl_token, "convert_token token not found pid %d %.48s:%.48s", cl_token->pid, cl_token->r.lockspace_name, cl_token->r.name); rv = -ENOENT; goto out; } if (sh_count && !(r->flags & R_SHARED)) { /* should not be possible */ log_errot(token, "convert_token invalid sh_count %d flags %x", sh_count, r->flags); rv = -EINVAL; goto out; } if (!sh_count && (r->flags & R_SHARED)) { /* should not be possible */ log_errot(token, "convert_token invalid sh_count %d flags %x", sh_count, r->flags); rv = -EINVAL; goto out; } if (!(res->flags & SANLK_RES_SHARED) && !(r->flags & R_SHARED)) { rv = -EALREADY; goto out; } if ((res->flags & SANLK_RES_SHARED) && (r->flags & R_SHARED)) { rv = -EALREADY; goto out; } rv = open_disks_fd(token->disks, token->r.num_disks); if (rv < 0) { log_errot(token, "convert_token open error %d", rv); goto out; } if (!(res->flags & SANLK_RES_SHARED)) { rv = convert_sh2ex_token(task, r, token, cmd_flags); } else if (res->flags & SANLK_RES_SHARED) { rv = convert_ex2sh_token(task, r, token); } else { /* not possible */ rv = -EINVAL; } close_disks(token->disks, token->r.num_disks); out: return rv; } int acquire_token(struct task *task, struct token *token, uint32_t cmd_flags, char *killpath, char *killargs, struct sanlk_host *owner_host) { struct leader_record leader; struct paxos_dblock dblock; struct resource *r; struct stat st; uint64_t acquire_lver = 0; uint64_t first_live_host_id = 0; uint64_t first_live_host_gen = 0; uint32_t new_num_hosts = 0; int sh_retries = 0; int live_count = 0; int allow_orphan = 0; int only_orphan = 0; int owner_nowait = 0; int new_id = 0; int rv; memset(&dblock, 0, sizeof(dblock)); if (token->acquire_flags & SANLK_RES_LVER) acquire_lver = token->acquire_lver; if (token->acquire_flags & SANLK_RES_NUM_HOSTS) new_num_hosts = token->acquire_data32; if (cmd_flags & (SANLK_ACQUIRE_ORPHAN | SANLK_ACQUIRE_ORPHAN_ONLY)) allow_orphan = 1; if (cmd_flags & SANLK_ACQUIRE_ORPHAN_ONLY) only_orphan = 1; if (cmd_flags & SANLK_ACQUIRE_OWNER_NOWAIT) owner_nowait = 1; pthread_mutex_lock(&resource_mutex); /* * Check if this resource already exists on any of the resource lists. */ r = find_resource(token, &resources_rem); if (r) { token->res_id = r->res_id; if (!com.quiet_fail) log_errot(token, "acquire_token resource being removed"); pthread_mutex_unlock(&resource_mutex); return -EAGAIN; } r = find_resource(token, &resources_add); if (r) { token->res_id = r->res_id; if (!com.quiet_fail) log_errot(token, "acquire_token resource being added"); pthread_mutex_unlock(&resource_mutex); return -EBUSY; } r = find_resource(token, &resources_held); if (r && (token->acquire_flags & SANLK_RES_SHARED) && (r->flags & R_SHARED)) { /* multiple shared holders allowed */ token->res_id = r->res_id; log_token(token, "acquire_token add shared"); copy_disks(&token->r.disks, &r->r.disks, token->r.num_disks); token->resource = r; list_add(&token->list, &r->tokens); pthread_mutex_unlock(&resource_mutex); return SANLK_OK; } if (r) { token->res_id = r->res_id; if (!com.quiet_fail) log_errot(token, "acquire_token resource exists"); pthread_mutex_unlock(&resource_mutex); return -EEXIST; } /* caller did not ask for orphan, but an orphan exists */ r = find_resource(token, &resources_orphan); if (r && !allow_orphan) { token->res_id = r->res_id; log_errot(token, "acquire_token found orphan"); pthread_mutex_unlock(&resource_mutex); return -EUCLEAN; } /* caller asked for exclusive orphan, but a shared orphan exists */ if (r && allow_orphan && (r->flags & R_SHARED) && !(token->acquire_flags & SANLK_RES_SHARED)) { token->res_id = r->res_id; log_errot(token, "acquire_token orphan is shared"); pthread_mutex_unlock(&resource_mutex); return -EUCLEAN; } /* caller asked for a shared orphan, but an exclusive orphan exists */ if (r && allow_orphan && !(r->flags & R_SHARED) && (token->acquire_flags & SANLK_RES_SHARED)) { token->res_id = r->res_id; log_errot(token, "acquire_token orphan is exclusive"); pthread_mutex_unlock(&resource_mutex); return -EUCLEAN; } /* caller asked for shared orphan, and a shared orphan exists */ if (r && allow_orphan && (r->flags & R_SHARED) && (token->acquire_flags & SANLK_RES_SHARED)) { token->res_id = r->res_id; log_token(token, "acquire_token adopt shared orphan"); token->resource = r; list_add(&token->list, &r->tokens); list_move(&r->list, &resources_held); pthread_mutex_unlock(&resource_mutex); /* do this to initialize some token fields */ rv = open_disks(token->disks, token->r.num_disks); if (rv < 0) { /* TODO: what parts above need to be undone? */ log_errot(token, "acquire_token sh orphan open error %d", rv); release_token_nodisk(task, token); return rv; } close_disks(token->disks, token->r.num_disks); return SANLK_OK; } /* caller asked for exclusive orphan, and an exclusive orphan exists */ if (r && allow_orphan && !(r->flags & R_SHARED) && !(token->acquire_flags & SANLK_RES_SHARED)) { token->res_id = r->res_id; log_token(token, "acquire_token adopt orphan"); token->r.lver = r->leader.lver; r->pid = token->pid; token->resource = r; list_add(&token->list, &r->tokens); list_move(&r->list, &resources_held); pthread_mutex_unlock(&resource_mutex); /* do this to initialize some token fields */ rv = open_disks(token->disks, token->r.num_disks); if (rv < 0) { /* TODO: what parts above need to be undone? */ log_errot(token, "acquire_token orphan open error %d", rv); release_token_nodisk(task, token); return rv; } close_disks(token->disks, token->r.num_disks); return SANLK_OK; } /* caller only wants to acquire an orphan */ if (only_orphan) { log_debug("acquire_token no orphan found for only_orphan"); pthread_mutex_unlock(&resource_mutex); return -ENOENT; } /* * The resource does not exist, so create it. */ r = get_resource(token, &new_id); if (!r) { pthread_mutex_unlock(&resource_mutex); return -ENOMEM; } memcpy(r->killpath, killpath, SANLK_HELPER_PATH_LEN); memcpy(r->killargs, killargs, SANLK_HELPER_ARGS_LEN); list_add(&token->list, &r->tokens); list_add(&r->list, &resources_add); token->res_id = r->res_id; token->resource = r; pthread_mutex_unlock(&resource_mutex); if (new_id) { /* save a record of what this id is for later debugging */ log_level(token->space_id, token->res_id, NULL, com.names_log_priority, "resource %.48s:%.48s:%.256s:%llu", token->r.lockspace_name, token->r.name, token->r.disks[0].path, (unsigned long long)token->r.disks[0].offset); } if (!token->resource->disk_major && !token->resource->disk_minor) { if (stat(token->disks[0].path, &st) < 0) { log_errot(token, "acquire_token stat %s error %d", token->disks[0].path, errno); release_token_nodisk(task, token); return -errno; } token->resource->disk_major = (uint32_t)major(st.st_rdev); token->resource->disk_minor = (uint32_t)minor(st.st_rdev); } rv = open_disks(token->disks, token->r.num_disks); if (rv < 0) { log_errot(token, "acquire_token open error %d", rv); release_token_nodisk(task, token); return rv; } copy_disks(&r->r.disks, &token->r.disks, token->r.num_disks); retry: memset(&leader, 0, sizeof(struct leader_record)); rv = acquire_disk(task, token, acquire_lver, new_num_hosts, owner_nowait, &leader, &dblock); /* * token sector_size/align_size starts by using sector_size/align_size * from the ls, but can change in paxos acquire when we see what's in * the leader_record. */ r->sector_size = token->sector_size; r->align_size = token->align_size; if (rv == SANLK_ACQUIRE_IDLIVE || rv == SANLK_ACQUIRE_OWNED || rv == SANLK_ACQUIRE_OTHER || rv == SANLK_ACQUIRE_OWNED_RETRY || rv == SANLK_ACQUIRE_OWNED_NO_TIMEOUT) { /* * Another host owns the lease. They may be holding for * only a short time while getting a shared lease. * Multiple parallel sh requests can fail because * the lease is briefly held in ex mode. The ex * holder sets SHORT_HOLD in the leader record to * indicate that it's only held for a short time * while acquiring a shared lease. A retry will * probably succeed. */ if ((token->acquire_flags & SANLK_RES_SHARED) && (leader.flags & LFL_SHORT_HOLD)) { if (sh_retries++ < com.sh_retries) { int us = get_rand(0, 1000000); log_token(token, "acquire_token sh_retry %d %d", rv, us); usleep(us); goto retry; } /* We must not write zero dblock values from write_host_block()! */ release_token_nodisk_opened(task, token); if (com.quiet_fail) log_token(token, "acquire_token error: %d held retried", rv); else log_errot(token, "acquire_token error: %d held retried", rv); return SANLK_ACQUIRE_SHRETRY; } if (com.quiet_fail) log_token(token, "acquire_token error: %d held by %llu %llu", rv, (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation); else log_errot(token, "acquire_token error: %d held by %llu %llu", rv, (unsigned long long)leader.owner_id, (unsigned long long)leader.owner_generation); /* We must not write zero dblock values from write_host_block()! */ release_token_nodisk_opened(task, token); if (owner_host) { owner_host->host_id = leader.owner_id; owner_host->generation = leader.owner_generation; owner_host->timestamp = leader.timestamp; owner_host->io_timeout = leader.io_timeout; } return rv; } if (rv < 0 && !(token->flags & T_RETRACT_PAXOS)) { log_token(token, "acquire_token error: %d", rv); release_token_nodisk_opened(task, token); return rv; } if (rv < 0 && (token->flags & T_RETRACT_PAXOS)) { /* * We might own the lease, we don't know, so we need to try to * release on disk to avoid possibly having an orphan lease on disk. */ log_warnt(token, "acquire_token error: %d RETRACT_PAXOS", rv); r->flags &= ~R_SHARED; r->flags |= R_ERASE_ALL; memcpy(&r->leader, &leader, sizeof(struct leader_record)); release_token_opened(task, token); return rv; } memcpy(&r->leader, &leader, sizeof(struct leader_record)); memcpy(&r->dblock, &dblock, sizeof(dblock)); /* copy lver into token because inquire looks there for it */ if (!(token->acquire_flags & SANLK_RES_SHARED)) token->r.lver = leader.lver; /* * acquiring shared lease, so we set SHARED in our mode_block * and release the leader owner. */ if (token->acquire_flags & SANLK_RES_SHARED) { rv = write_host_block_acquire_sh(task, token); if (rv < 0) { log_warnt(token, "acquire_token sh write_host_block error %d", rv); r->flags &= ~R_SHARED; r->flags |= R_UNDO_SHARED; release_token_opened(task, token); return rv; } /* the token is kept, the paxos lease is released but with shared set */ rv = release_disk(task, token, NULL, &leader); if (rv < 0) { log_errot(token, "acquire_token sh release_disk error %d", rv); r->flags &= ~R_SHARED; r->flags |= R_UNDO_SHARED; release_token_opened(task, token); return rv; } /* normal exit case for successful acquire sh */ goto out; } if (!token->shared_count) { goto out; } /* * acquiring normal ex lease, other hosts have it shared. * check if those other hosts are alive or dead (clear any that are dead). */ /* * paxos_lease_acquire() counted some SHARED mode blocks. * Here we check if they are held by live hosts. If a host * with SHARED mb is dead, we clear it, otherwise it's alive * and we count it in live_count. */ rv = clear_dead_shared(task, token, leader.num_hosts, &live_count, &first_live_host_id, &first_live_host_gen); if (rv < 0) { log_errot(token, "acquire_token clear_dead_shared error %d", rv); release_token_opened(task, token); return rv; } /* * acquiring normal ex lease, other hosts have it shared and are alive. * normal exit case for acquire ex that failed due to existing sh lock. */ if (live_count) { rv = release_token_opened(task, token); if (rv < 0) { log_errot(token, "acquire_token live_count release error %d", rv); return rv; } if (owner_host) { owner_host->host_id = first_live_host_id; owner_host->generation = first_live_host_gen; owner_host->timestamp = 0; owner_host->io_timeout = 0; } return -EAGAIN; } out: if (cmd_flags & SANLK_ACQUIRE_LVB) { char *iobuf, **p_iobuf; p_iobuf = &iobuf; /* TODO: we should probably notify the caller somehow about lvb read/write independent of the lease results. */ rv = posix_memalign((void *)p_iobuf, getpagesize(), token->sector_size); if (rv) { log_errot(token, "acquire_token lvb size %d memalign error %d", token->sector_size, rv); } else { r->lvb = iobuf; rv = read_lvb_block(task, token); if (rv < 0) log_errot(token, "acquire_token read_lvb error %d", rv); } } close_disks(token->disks, token->r.num_disks); close_token_caw(token); pthread_mutex_lock(&resource_mutex); list_move(&r->list, &resources_held); pthread_mutex_unlock(&resource_mutex); return SANLK_OK; } int request_token(struct task *task, struct token *token, uint32_t force_mode, uint64_t *owner_id, int next_lver) { struct leader_record leader; struct request_record req; int align_size; int rv; memset(&req, 0, sizeof(req)); rv = open_disks(token->disks, token->r.num_disks); if (rv < 0) { log_errot(token, "request_token open error %d", rv); return rv; } if (!token->acquire_lver && !force_mode) goto req_read; /* * cmd_request() takes the sector_size and align_size from the * lockspace as a starting point, setting them in the token. If the * leader_record for the paxos lease on disk is different, then adjust * the values in the token. */ rv = paxos_lease_leader_read(task, task->use_aio, token, &leader, "request"); if (rv < 0) goto out; align_size = leader_align_size_from_flag(leader.flags); if (!align_size) align_size = sector_size_to_align_size_old(leader.sector_size); if ((leader.sector_size != token->sector_size) || (align_size != token->align_size)) { /* paxos lease has different size than we borrowed from the lockspace */ token->sector_size = leader.sector_size; token->align_size = align_size; if (!token->align_size) token->align_size = sector_size_to_align_size_old(leader.sector_size); } if (leader.timestamp == LEASE_FREE) { *owner_id = 0; rv = SANLK_OK; goto out; } *owner_id = leader.owner_id; if (!token->acquire_lver && next_lver) token->acquire_lver = leader.lver + 1; if (leader.lver >= token->acquire_lver) { rv = SANLK_REQUEST_OLD; goto out; } req_read: rv = paxos_lease_request_read(task, token, &req); if (rv < 0) goto out; if (req.magic != REQ_DISK_MAGIC) { rv = SANLK_REQUEST_MAGIC; goto out; } if ((req.version & 0xFFFF0000) != REQ_DISK_VERSION_MAJOR) { rv = SANLK_REQUEST_VERSION; goto out; } if (!token->acquire_lver && !force_mode) goto req_write; /* > instead of >= so multiple hosts can request the same version at once and all succeed */ if (req.lver > token->acquire_lver) { rv = SANLK_REQUEST_LVER; goto out; } req_write: req.version = REQ_DISK_VERSION_MAJOR | REQ_DISK_VERSION_MINOR; req.lver = token->acquire_lver; req.force_mode = force_mode; rv = paxos_lease_request_write(task, token, &req); out: close_disks(token->disks, token->r.num_disks); log_token(token, "request_token rv %d owner %llu lver %llu mode %u", rv, (unsigned long long)*owner_id, (unsigned long long)req.lver, req.force_mode); return rv; } static int examine_token(struct task *task, struct token *token, struct request_record *req_out) { struct request_record req; int rv; memset(&req, 0, sizeof(req)); rv = paxos_lease_request_read(task, token, &req); if (rv < 0) goto out; if (req.magic != REQ_DISK_MAGIC) { rv = SANLK_REQUEST_MAGIC; goto out; } if ((req.version & 0xFFFF0000) != REQ_DISK_VERSION_MAJOR) { rv = SANLK_REQUEST_VERSION; goto out; } memcpy(req_out, &req, sizeof(struct request_record)); out: log_token(token, "examine_token rv %d lver %llu mode %u", rv, (unsigned long long)req.lver, req.force_mode); return rv; } static void do_request(struct token *tt, int pid, uint32_t force_mode) { char killpath[SANLK_HELPER_PATH_LEN]; char killargs[SANLK_HELPER_ARGS_LEN]; struct helper_msg hm; struct resource *r; uint32_t flags; int rv, found = 0; pthread_mutex_lock(&resource_mutex); r = find_resource(tt, &resources_held); if (r && r->pid == pid) { found = 1; flags = r->flags; memcpy(killpath, r->killpath, SANLK_HELPER_PATH_LEN); memcpy(killargs, r->killargs, SANLK_HELPER_ARGS_LEN); } pthread_mutex_unlock(&resource_mutex); if (!found) { log_error("do_request pid %d %.48s:%.48s not found", pid, tt->r.lockspace_name, tt->r.name); return; } log_token(tt, "do_request %d flags %x %.48s:%.48s", pid, flags, tt->r.lockspace_name, tt->r.name); if (helper_kill_fd == -1) { log_error("do_request %d no helper fd", pid); return; } memset(&hm, 0, sizeof(hm)); if (force_mode == SANLK_REQ_FORCE) { hm.type = HELPER_MSG_KILLPID; hm.pid = pid; hm.sig = (flags & R_RESTRICT_SIGKILL) ? SIGTERM : SIGKILL; } else if (force_mode == SANLK_REQ_GRACEFUL) { if (killpath[0]) { hm.type = HELPER_MSG_RUNPATH; memcpy(hm.path, killpath, SANLK_HELPER_PATH_LEN-1); memcpy(hm.args, killargs, SANLK_HELPER_ARGS_LEN-1); } else { hm.type = HELPER_MSG_KILLPID; hm.pid = pid; hm.sig = (flags & R_RESTRICT_SIGTERM) ? SIGKILL : SIGTERM; } } else { log_error("do_request %d unknown force_mode %d", pid, force_mode); return; } retry: rv = write(helper_kill_fd, &hm, sizeof(hm)); if (rv == -1 && errno == EINTR) goto retry; if (rv == -1) log_error("do_request %d helper write error %d", pid, errno); } int set_resource_examine(char *space_name, char *res_name) { struct resource *r; int count = 0; pthread_mutex_lock(&resource_mutex); list_for_each_entry(r, &resources_held, list) { if (strncmp(r->r.lockspace_name, space_name, NAME_ID_SIZE)) continue; if (res_name && strncmp(r->r.name, res_name, NAME_ID_SIZE)) continue; r->flags |= R_THREAD_EXAMINE; resource_thread_work = 1; resource_thread_work_examine = 1; count++; } if (count) pthread_cond_signal(&resource_cond); pthread_mutex_unlock(&resource_mutex); return count; } /* * resource_thread * - on-disk lease release for pid's that exit without doing release * - on-disk lease release for which release_token had transient i/o error * - examines request blocks of resources */ static struct resource *find_resource_thread(struct list_head *head, uint32_t flag) { struct resource *r; uint64_t now = monotime(); list_for_each_entry(r, head, list) { if (!(r->flags & flag)) continue; if (flag & R_THREAD_EXAMINE) return r; if (now >= r->thread_release_retry) return r; } return NULL; } /* * When release_token is called from a context where it cannot block by doing * disk io, the token itself is released, but the struct resource is passed to * the resource_thread to do the on-disk operations. * * Also, if release_token gets an io timeout during the disk operations, it * removes the token, but passes the struct resource to the resource_thread * to retry the on-disk release operations. It doesn't want to leave a * potentially locked lease on disk simply due to a transient io error. * * This does this non-token related on-disk release operations. It uses * a fake token emulating the original because the paxos layer wants that. * * As long as the on-disk release fails due to io timeouts, the struct resource * is kept and the on-disk release retried. If another, non-timeout error occurs, * we give up and delete/free the struct resource. */ static void resource_thread_release(struct task *task, struct resource *r, struct token *token) { struct space_info spi; uint32_t r_flags; int retry_async = 0; int cleanup_retry = 0; int rv; r_flags = r->flags; rv = open_disks_fd(token->disks, token->r.num_disks); if (rv < 0) { log_errot(token, "rthread release_token open error %d", rv); goto out; } /* The lockspace may fail after the resource was transferred to the resource_thread, so we need to check here if if that's the case. */ rv = lockspace_info(token->r.lockspace_name, &spi); if (rv < 0 || spi.killing_pids) { log_token(token, "rthread release_token quit lockspace_info %d killing_pids %d", rv, spi.killing_pids); rv = -1; goto out_close; } /* * See comments in _release_token. * FIXME: avoid duplicating all this from _release_token. */ log_token(token, "rthread release begin r_flags %x", r_flags); if ((token->flags & T_CLEANUP_CONVERT) || (r_flags & R_ERASE_ALL) || (r_flags & R_UNDO_SHARED)) { rv = _release_token_cleanup(task, token, &cleanup_retry); if (rv < 0) log_warnt(token, "rthread release_token_cleanup error %d 0x%x", rv, r_flags); else log_warnt(token, "rthread release_token_cleanup done 0x%x", r_flags); if (cleanup_retry) { retry_async = 1; goto out_close; } } else if (r_flags & R_SHARED) { /* normal release of sh lease */ rv = write_host_block_release_sh(task, token); if (rv < 0) log_warnt(token, "rthread release_token sh write_host_block error %d", rv); /* TODO: log done message if there was a prev error message? */ if (rv == SANLK_AIO_TIMEOUT || rv == -EIO) retry_async = 1; } else { /* normal release of ex lease */ if (r_flags & R_LVB_WRITE_RELEASE) { rv = write_lvb_block(task, r, token); if (!rv) r->flags &= ~R_LVB_WRITE_RELEASE; else log_warnt(token, "rthread release_token write_lvb error %d", rv); /* do we want to give more effort to writing lvb? */ } /* For an ex caw lease, the mblock is not used, so there's nothing to clear. */ if (!(r->flags & R_CAW)) { /* Failure here is not a big deal and can be ignored. */ rv = write_host_block_release_ex(task, token); if (rv < 0) log_warnt(token, "rthread release_token ex write_host_block error %d", rv); } rv = release_disk(task, token, NULL, &r->leader); if (rv < 0) log_warnt(token, "rthread release_token ex lease_release error %d", rv); /* TODO: log done message if there was a prev error message? */ if (rv == SANLK_AIO_TIMEOUT || rv == -EIO) retry_async = 1; } out_close: close_token_caw(token); close_disks(token->disks, token->r.num_disks); out: /* * caw_data may have been set by resource_thread() (moving * it from the original token.) Otherwise, caw_data may * have been allocated in the call to release_disk above. */ if (token->caw_data) { free(token->caw_data); token->caw_data = NULL; } if (!retry_async) { log_token(token, "rthread release_token done 0x%x", r_flags); pthread_mutex_lock(&resource_mutex); list_del(&r->list); free_resource(r); pthread_mutex_unlock(&resource_mutex); return; } /* Keep the resource on the list to keep trying. */ log_token(token, "rthread release_token error %d will retry 0x%x", rv, r_flags); pthread_mutex_lock(&resource_mutex); r->flags |= R_THREAD_RELEASE; pthread_mutex_unlock(&resource_mutex); } static void resource_thread_examine(struct task *task, struct token *tt, int pid, uint64_t lver) { struct request_record req; int rv; rv = open_disks_fd(tt->disks, tt->r.num_disks); if (rv < 0) { log_errot(tt, "examine open error %d", rv); return; } rv = examine_token(task, tt, &req); close_disks(tt->disks, tt->r.num_disks); if (rv != SANLK_OK) return; if (!req.force_mode || !req.lver) return; if (req.lver <= lver) { log_token(tt, "examine req lver %llu our lver %llu", (unsigned long long)req.lver, (unsigned long long)lver); return; } if (req.force_mode) { do_request(tt, pid, req.force_mode); } else { log_error("req force_mode %u unknown", req.force_mode); } } struct recv_he { struct list_head list; uint32_t space_id; uint64_t from_host_id; uint64_t from_generation; struct sanlk_host_event he; }; void add_host_event(uint32_t space_id, struct sanlk_host_event *he, uint64_t from_host_id, uint64_t from_generation) { struct recv_he *rhe; rhe = malloc(sizeof(struct recv_he)); if (!rhe) { log_error("add_host_event no mem"); return; } memset(rhe, 0, sizeof(struct recv_he)); memcpy(&rhe->he, he, sizeof(struct sanlk_host_event)); rhe->space_id = space_id; rhe->from_host_id = from_host_id; rhe->from_generation = from_generation; pthread_mutex_lock(&resource_mutex); list_add_tail(&rhe->list, &host_events); resource_thread_work = 1; pthread_cond_signal(&resource_cond); pthread_mutex_unlock(&resource_mutex); } static struct recv_he *find_host_event(void) { if (list_empty(&host_events)) return NULL; return list_first_entry(&host_events, struct recv_he, list); } int print_state_rthread(char *str); int print_state_rthread(char *str) { memset(str, 0, SANLK_STATE_MAXSTR); snprintf(str, SANLK_STATE_MAXSTR-1, "busy=%u " "thread_id=%u " "work_count=%u " "io_count=%u " "to_count=%u " "miscompares=%u " "paxos_areabuf_created=%u " "paxos_areabuf_used=%u", rthread_info.busy, rthread_info.thread_id, rthread_info.work_count, rthread_info.io_count, rthread_info.to_count, rthread_info.miscompares, rthread_info.paxos_areabuf_created, rthread_info.paxos_areabuf_used); return strlen(str) + 1; } /* * The single resource_thread handles resources for all lockspaces. * Primarily for releasing resources which were dropped by a client * that exited while holding locks, or releasing resources after the * initial release_token had an io error/timeout. * Should there be multiple resource threads (one for each ls?) * in case one lockspace's storage blocks i/o for a long period? */ static void *resource_thread(void *arg GNUC_UNUSED) { struct task task; struct resource *r; struct token *tt = NULL; struct recv_he *rhe; uint64_t lver; int pid, tt_len; memset(&task, 0, sizeof(struct task)); setup_task_aio(&task, main_task.use_aio, RESOURCE_AIO_CB_SIZE); sprintf(task.name, "%s", "rthread"); rthread_info.thread_id = gettid(); /* a fake/tmp token struct we copy necessary res info into, because other functions take a token struct arg */ tt_len = sizeof(struct token) + (SANLK_MAX_DISKS * sizeof(struct sync_disk)); tt = malloc(tt_len); if (!tt) { log_error("resource_thread tt malloc error"); goto out; } while (1) { pthread_mutex_lock(&resource_mutex); while (!resource_thread_work) { rthread_info.busy = 0; if (resource_thread_stop) { pthread_mutex_unlock(&resource_mutex); goto out; } pthread_cond_wait(&resource_cond, &resource_mutex); } rthread_info.busy = 1; rthread_info.io_count = task.io_count; rthread_info.to_count = task.to_count; rthread_info.miscompares = task.miscompares; rthread_info.paxos_areabuf_created = task.paxos_areabuf_created; rthread_info.paxos_areabuf_used = task.paxos_areabuf_used; rhe = find_host_event(); if (rhe) { list_del(&rhe->list); pthread_mutex_unlock(&resource_mutex); send_event_callbacks(rhe->space_id, rhe->from_host_id, rhe->from_generation, &rhe->he); free(rhe); continue; } /* FIXME: it's not nice how we copy a bunch of stuff * from token to r so that we can later copy it back from * r into a temp token. The whole duplication of stuff * between token and r would be nice to clean up. */ memset(tt, 0, tt_len); tt->disks = (struct sync_disk *)&tt->r.disks[0]; r = find_resource_thread(&resources_rem, R_THREAD_RELEASE); if (r) { rthread_info.work_count++; /* only counts release work */ memcpy(&tt->r, &r->r, sizeof(struct sanlk_resource)); copy_disks(&tt->r.disks, &r->r.disks, r->r.num_disks); tt->host_id = r->host_id; tt->host_generation = r->host_generation; tt->res_id = r->res_id; tt->io_timeout = r->io_timeout; tt->sector_size = r->sector_size; tt->align_size = r->align_size; tt->resource = r; if (r->flags & R_CAW) { tt->flags |= T_CAW; /* resource_thread_release will always free tt->caw_data if set */ tt->caw_data = r->thread_release_caw_data; r->thread_release_caw_data = NULL; } if (r->flags & R_CLEANUP_CONVERT) tt->flags |= T_CLEANUP_CONVERT; /* * Set the time after which we should try to release this * resource again if this current attempt times out. */ if (!r->thread_release_retry) r->thread_release_retry = monotime() + r->io_timeout; else r->thread_release_retry = monotime() + (r->io_timeout * 2); r->flags &= ~R_THREAD_RELEASE; pthread_mutex_unlock(&resource_mutex); /* resource_thread_release() may free r, does not free tt, always frees tt->caw_data if set */ resource_thread_release(&task, r, tt); if (tt->caw_data) log_errot(tt, "resource_thread_release did not free caw_data"); continue; } /* * We don't want to search all of resource_held each time * we are woken unless we know there is something to examine. */ if (!resource_thread_work_examine) goto find_done; r = find_resource_thread(&resources_held, R_THREAD_EXAMINE); if (r) { /* make copies of things we need because we can't use r once we unlock the mutex since it could be released */ memcpy(&tt->r, &r->r, sizeof(struct sanlk_resource)); copy_disks(&tt->r.disks, &r->r.disks, r->r.num_disks); tt->host_id = r->host_id; tt->host_generation = r->host_generation; tt->res_id = r->res_id; tt->io_timeout = r->io_timeout; tt->sector_size = r->sector_size; tt->align_size = r->align_size; pid = r->pid; lver = r->leader.lver; r->flags &= ~R_THREAD_EXAMINE; pthread_mutex_unlock(&resource_mutex); resource_thread_examine(&task, tt, pid, lver); continue; } find_done: resource_thread_work = 0; resource_thread_work_examine = 0; pthread_mutex_unlock(&resource_mutex); } out: if (tt) free_token(tt); close_task_aio(&task); return NULL; } int release_orphan(struct sanlk_resource *res) { struct resource *r, *safe; int count = 0; pthread_mutex_lock(&resource_mutex); list_for_each_entry_safe(r, safe, &resources_orphan, list) { if (strncmp(r->r.lockspace_name, res->lockspace_name, NAME_ID_SIZE)) continue; if (!res->name[0] || !strncmp(r->r.name, res->name, NAME_ID_SIZE)) { log_debug("release orphan %.48s:%.48s", r->r.lockspace_name, r->r.name); r->flags |= R_THREAD_RELEASE; list_move(&r->list, &resources_rem); count++; } } if (count) { resource_thread_work = 1; pthread_cond_signal(&resource_cond); } pthread_mutex_unlock(&resource_mutex); return count; } static void purge_resource_list(struct list_head *head, char *space_name, const char *list_name) { struct resource *r, *safe; pthread_mutex_lock(&resource_mutex); list_for_each_entry_safe(r, safe, head, list) { if (strncmp(r->r.lockspace_name, space_name, NAME_ID_SIZE)) continue; if (list_name) log_debug("purge %s %.48s:%.48s", list_name, r->r.lockspace_name, r->r.name); list_del(&r->list); free(r); } pthread_mutex_unlock(&resource_mutex); } void purge_resource_orphans(char *space_name) { purge_resource_list(&resources_orphan, space_name, "orphan_list"); } void purge_resource_free(char *space_name) { purge_resource_list(&resources_free, space_name, "free_list"); } /* * This is called by the main_loop once a second during normal operation. * The resources_rem list should normally be empty, so this does nothing. * This is needed to wake up the resource_thread to retry release operations * that had timed out previously and need to be retried. */ void rem_resources(void) { pthread_mutex_lock(&resource_mutex); if (!list_empty(&resources_rem) && !resource_thread_work) { resource_thread_work = 1; pthread_cond_signal(&resource_cond); } pthread_mutex_unlock(&resource_mutex); } int setup_token_manager(void) { int rv; pthread_mutex_init(&resource_mutex, NULL); pthread_cond_init(&resource_cond, NULL); INIT_LIST_HEAD(&resources_add); INIT_LIST_HEAD(&resources_rem); INIT_LIST_HEAD(&resources_held); INIT_LIST_HEAD(&resources_free); INIT_LIST_HEAD(&resources_orphan); INIT_LIST_HEAD(&host_events); rv = pthread_create(&resource_pt, NULL, resource_thread, NULL); if (rv) return -1; return 0; } void close_token_manager(void) { pthread_mutex_lock(&resource_mutex); resource_thread_stop = 1; pthread_cond_signal(&resource_cond); pthread_mutex_unlock(&resource_mutex); pthread_join(resource_pt, NULL); } sanlock/src/resource.h000066400000000000000000000047001516326262600153110ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __RESOURCE_H__ #define __RESOURCE_H__ /* * We mostly avoid holding resource_mutex and spaces_mutex at once. When they * are held at once, the order is spaces_mutex, then resource_mutex. */ /* locks resource_mutex */ void send_state_resources(int fd); /* locks resource_mutex */ int lockspace_is_used(struct sanlk_lockspace *ls); /* locks resource_mutex */ int resource_orphan_count(char *space_name); /* no locks */ void check_mode_block(struct token *token, uint64_t next_lver, int q, char *dblock); /* locks resource_mutex */ int convert_token(struct task *task, struct sanlk_resource *res, struct token *cl_token, uint32_t cmd_flags); /* locks resource_mutex */ int acquire_token(struct task *task, struct token *token, uint32_t cmd_flags, char *killpath, char *killargs, struct sanlk_host *owner_host); /* locks resource_mutex */ int release_token(struct task *task, struct token *token, struct sanlk_resource *resrename); /* locks resource_mutex */ int release_token_nodisk(struct task *task, struct token *token); /* locks resource_mutex */ void release_token_async(struct token *token); /* no locks */ int request_token(struct task *task, struct token *token, uint32_t force_mode, uint64_t *owner_id, int next_lver); /* locks resource_mutex */ int set_resource_examine(char *space_name, char *res_name); /* locks resource_mutex */ int res_set_lvb(struct sanlk_resource *res, char *lvb, int lvblen); /* locks resource_mutex */ int res_get_lvb(struct sanlk_resource *res, char **lvb_out, int *lvblen); /* no locks */ int read_resource_owners(struct task *task, struct token *token, struct sanlk_resource *res, char **send_buf, int *send_len, int *count); /* locks resource_mutex */ void rem_resources(void); /* locks resource_mutex */ int release_orphan(struct sanlk_resource *res); /* locks resource_mutex */ void purge_resource_orphans(char *space_name); void purge_resource_free(char *space_name); /* locks resource_mutex */ void add_host_event(uint32_t space_id, struct sanlk_host_event *he, uint64_t from_host_id, uint64_t from_generation); int setup_token_manager(void); void close_token_manager(void); #endif sanlock/src/rindex.c000066400000000000000000000735001516326262600147520ustar00rootroot00000000000000/* * Copyright 2018 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock_internal.h" #include "sanlock_admin.h" #include "diskio.h" #include "ondisk.h" #include "log.h" #include "paxos_lease.h" #include "lockspace.h" #include "resource.h" #include "task.h" #include "timeouts.h" #include "rindex_disk.h" #include "rindex.h" #include "paxos_dblock.h" #include "leader.h" struct rindex_info { struct sanlk_rindex *ri; /* point to sanlk_rindex */ struct sync_disk *disk; /* points to sanlk_rindex.disk */ struct rindex_header header; }; /* this token is used for paxos_lease_acquire/release */ static struct token *setup_rindex_token(struct rindex_info *rx, int sector_size, int align_size, struct space_info *spi) { struct token *token; int token_len; token_len = sizeof(struct token) + sizeof(struct sync_disk); token = malloc(token_len); if (!token) return NULL; memset(token, 0, token_len); memcpy(token->r.lockspace_name, rx->ri->lockspace_name, SANLK_NAME_LEN); strcpy(token->r.name, "rindex_lease"); token->sector_size = sector_size; token->align_size = align_size; token->io_timeout = spi ? spi->io_timeout : com.io_timeout; token->r.num_disks = 1; token->r.flags |= sanlk_res_sector_size_to_flag(sector_size); token->r.flags |= sanlk_res_align_size_to_flag(align_size); token->disks = (struct sync_disk *)&token->r.disks[0]; /* shorthand */ memcpy(token->disks[0].path, rx->disk->path, SANLK_PATH_LEN); token->disks[0].offset = rx->disk->offset + align_size; token->disks[0].fd = rx->disk->fd; if (spi) { token->host_id = spi->host_id; token->host_generation = spi->host_generation; token->space_id = spi->space_id; token->res_id = 1; } return token; } /* this token is only used for paxos_lease_init */ static struct token *setup_resource_token(struct rindex_info *rx, char *res_name, int sector_size, int align_size, struct space_info *spi) { struct token *token; int token_len; token_len = sizeof(struct token) + sizeof(struct sync_disk); token = malloc(token_len); if (!token) return NULL; memset(token, 0, token_len); memcpy(token->r.lockspace_name, rx->ri->lockspace_name, SANLK_NAME_LEN); memcpy(token->r.name, res_name, SANLK_NAME_LEN); token->sector_size = sector_size; token->align_size = align_size; token->io_timeout = spi ? spi->io_timeout : com.io_timeout; token->r.num_disks = 1; token->r.flags |= sanlk_res_sector_size_to_flag(sector_size); token->r.flags |= sanlk_res_align_size_to_flag(align_size); token->disks = (struct sync_disk *)&token->r.disks[0]; /* shorthand */ memcpy(token->disks[0].path, rx->disk->path, SANLK_PATH_LEN); token->disks[0].fd = rx->disk->fd; /* there is no offset yet, it is found and set later */ return token; } /* max resource entries supported by each combination of sector/align size */ static uint32_t size_to_max_resources(int sector_size, int align_size) { if ((sector_size == 512) && (align_size == ALIGN_SIZE_1M)) return 16000; if ((sector_size == 4096) && (align_size == ALIGN_SIZE_1M)) return 16000; if ((sector_size == 4096) && (align_size == ALIGN_SIZE_2M)) return 32000; if ((sector_size == 4096) && (align_size == ALIGN_SIZE_4M)) return 64000; if ((sector_size == 4096) && (align_size == ALIGN_SIZE_8M)) return 128000; /* this shouldn't happen */ return 16000; } static int search_entries(struct rindex_info *rx, char *rindex_iobuf, uint64_t *ent_offset, uint64_t *res_offset, int find_free, char *find_name) { struct rindex_entry re; struct rindex_entry *re_end; uint64_t entry_offset_in_rindex; uint32_t max_resources = rx->header.max_resources; int sector_size = rx->header.sector_size; int align_size = rindex_header_align_size_from_flag(rx->header.flags); int i; if (!max_resources) max_resources = size_to_max_resources(sector_size, align_size); for (i = 0; i < max_resources; i++) { /* skip first sector which holds header */ entry_offset_in_rindex = sector_size + (i * sizeof(struct rindex_entry)); re_end = (struct rindex_entry *)(rindex_iobuf + entry_offset_in_rindex); rindex_entry_in(re_end, &re); if (find_free && (!re.res_offset && !re.name[0])) { *ent_offset = entry_offset_in_rindex; *res_offset = rx->disk->offset + (2 * align_size) + (i * align_size); return 0; } if (find_name && re.name[0] && !strncmp(re.name, find_name, SANLK_NAME_LEN)) { *ent_offset = entry_offset_in_rindex; *res_offset = rx->disk->offset + (2 * align_size) + (i * align_size); return 0; } } return -ENOENT; } static int update_rindex(struct task *task, struct space_info *spi, struct rindex_info *rx, char *rindex_iobuf, struct sanlk_rentry *re, uint64_t ent_offset, uint64_t res_offset, int delete) { struct rindex_entry re_new; struct rindex_entry re_end; char *sector_iobuf; char **p_iobuf; uint32_t sector_offset; uint32_t entry_offset_in_sector; int sector_size = rx->header.sector_size; int iobuf_len; int rv; /* * ent_offset is the offset (in bytes) from the start of the rindex to * the entry being updated. (This includes the size of the header * sector; no offsets are calculated from the end of the header * sector.) * * sector_offset is the offset (in bytes) from the start of the rindex * to the sector containing ent_offset. The entire sector is written. * * entry_offset_in_sector is the offset (in bytes) from the start of * the target sector to the entry being updated. */ sector_offset = (ent_offset / sector_size) * sector_size; entry_offset_in_sector = ent_offset % sector_size; iobuf_len = sector_size; p_iobuf = §or_iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), iobuf_len); if (rv) return rv; memset(sector_iobuf, 0, iobuf_len); memset(&re_new, 0, sizeof(struct rindex_entry)); if (!delete) { memcpy(re_new.name, re->name, NAME_ID_SIZE); re_new.res_offset = res_offset; } rindex_entry_out(&re_new, &re_end); /* initialize new sector with existing index content */ memcpy(sector_iobuf, rindex_iobuf + sector_offset, sector_size); /* replace the specific entry */ memcpy(sector_iobuf + entry_offset_in_sector, &re_end, sizeof(struct rindex_entry)); rv = write_iobuf(rx->disk->fd, rx->disk->offset + sector_offset, sector_iobuf, iobuf_len, task, task->use_aio, spi->io_timeout, NULL); if (rv != SANLK_AIO_TIMEOUT) free(sector_iobuf); return rv; } static int read_rindex(struct task *task, struct space_info *spi, struct rindex_info *rx, char **rindex_iobuf_ret) { char *iobuf; char **p_iobuf; int align_size = rindex_header_align_size_from_flag(rx->header.flags); int iobuf_len; int rv; iobuf_len = align_size; p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), iobuf_len); if (rv) { return rv; } memset(iobuf, 0, iobuf_len); rv = read_iobuf(rx->disk->fd, rx->disk->offset, iobuf, iobuf_len, task, task->use_aio, spi->io_timeout, NULL); if (rv < 0) { if (rv != SANLK_AIO_TIMEOUT) { free(iobuf); iobuf = NULL; } return rv; } *rindex_iobuf_ret = iobuf; return rv; } static int read_rindex_header(struct task *task, struct space_info *spi, struct rindex_info *rx) { struct rindex_header *rh_end; char *iobuf; char **p_iobuf; int sector_size = spi->sector_size; int io_timeout = spi->io_timeout; int iobuf_len; int rv; if (!sector_size) sector_size = 4096; if (!io_timeout) { io_timeout = com.io_timeout; spi->io_timeout = io_timeout; } /* * lockspace sector_size will usually be the same as rindex sector_size. * use the lockspace sector size for reading the rindex header which * officially gives us the rindex sector_size. */ iobuf_len = sector_size; p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), iobuf_len); if (rv) return -ENOMEM; rv = read_iobuf(rx->disk->fd, rx->disk->offset, iobuf, iobuf_len, task, task->use_aio, io_timeout, NULL); if (rv < 0) goto out; rh_end = (struct rindex_header *)iobuf; rindex_header_in(rh_end, &rx->header); if (rx->header.magic != RINDEX_DISK_MAGIC) { log_debug("rindex header bad magic %x vs %x on %s:%llu", rx->header.magic, RINDEX_DISK_MAGIC, rx->disk->path, (unsigned long long)rx->disk->offset); rv = SANLK_RINDEX_MAGIC; goto out; } if ((rx->header.version & 0xFFFF0000) != RINDEX_DISK_VERSION_MAJOR) { log_debug("rindex header bad version %x vs %x on %s:%llu", rx->header.version, RINDEX_DISK_VERSION_MAJOR, rx->disk->path, (unsigned long long)rx->disk->offset); rv = SANLK_RINDEX_VERSION; goto out; } if (strcmp(rx->header.lockspace_name, rx->ri->lockspace_name)) { log_debug("rindex header bad lockspace_name %.48s vs %.48s on %s:%llu", rx->header.lockspace_name, rx->ri->lockspace_name, rx->disk->path, (unsigned long long)rx->disk->offset); rv = SANLK_RINDEX_LOCKSPACE; goto out; } if (rx->header.rx_offset != rx->disk->offset) { log_debug("rindex header bad offset %llu on %s:%llu", (unsigned long long)rx->header.rx_offset, rx->disk->path, (unsigned long long)rx->disk->offset); rv = SANLK_RINDEX_OFFSET; goto out; } out: if (rv != SANLK_AIO_TIMEOUT) free(iobuf); return rv; } /* * format rindex: write new rindex header, and initialize internal paxos lease * for protecting the rindex. */ int rindex_format(struct task *task, struct sanlk_rindex *ri) { struct rindex_info rx; struct rindex_header rh; struct rindex_header rh_end; struct token *token; char *iobuf; char **p_iobuf; uint32_t max_resources; uint32_t max_resources_limit; int write_io_timeout; int sector_size = 0; int align_size = 0; int max_hosts = 0; int iobuf_len; int rv; memset(&rx, 0, sizeof(rx)); rx.ri = ri; rx.disk = (struct sync_disk *)&ri->disk; rv = open_disk(rx.disk); if (rv < 0) { log_error("rindex_format open failed %d %s", rv, rx.disk->path); return rv; } rv = sizes_from_flags(ri->flags, §or_size, &align_size, &max_hosts, "RIF"); if (rv) return rv; if (!sector_size) { /* sector/align flags were not set, use historical defaults */ sector_size = rx.disk->sector_size; align_size = sector_size_to_align_size_old(sector_size); max_hosts = DEFAULT_MAX_HOSTS; } /* * When unspecified, default to 4096 to limit the amount of searching. */ max_resources = rx.ri->max_resources; if (!max_resources) max_resources = 4096; max_resources_limit = size_to_max_resources(sector_size, align_size); if (max_resources > max_resources_limit) max_resources = max_resources_limit; log_debug("rindex_format %.48s:%s:%llu %d %d max_res %u", rx.ri->lockspace_name, rx.disk->path, (unsigned long long)rx.disk->offset, sector_size, align_size, max_resources); iobuf_len = align_size; p_iobuf = &iobuf; rv = posix_memalign((void *)p_iobuf, getpagesize(), iobuf_len); if (rv) goto out_close; memset(iobuf, 0, iobuf_len); memset(&rh, 0, sizeof(struct rindex_header)); rh.magic = RINDEX_DISK_MAGIC; rh.version = RINDEX_DISK_VERSION_MAJOR | RINDEX_DISK_VERSION_MINOR; rh.flags = rindex_header_align_flag_from_size(align_size); rh.sector_size = sector_size; rh.max_resources = max_resources; rh.rx_offset = rx.disk->offset; memcpy(rh.lockspace_name, rx.ri->lockspace_name, NAME_ID_SIZE); memset(&rh_end, 0, sizeof(struct rindex_header)); rindex_header_out(&rh, &rh_end); memcpy(iobuf, &rh_end, sizeof(struct rindex_header)); if (com.write_init_io_timeout) write_io_timeout = com.write_init_io_timeout; else write_io_timeout = com.io_timeout; rv = write_iobuf(rx.disk->fd, rx.disk->offset, iobuf, iobuf_len, task, task->use_aio, write_io_timeout, NULL); if (rv < 0) { log_error("rindex_format write failed %d %s", rv, rx.disk->path); goto out_iobuf; } token = setup_rindex_token(&rx, sector_size, align_size, NULL); if (!token) { rv = -ENOMEM; goto out_iobuf; } rv = paxos_lease_init(task, token, 0, 0, 0); if (rv < 0) { log_error("rindex_format lease init failed %d", rv); goto out_token; } rv = 0; out_token: free_token(token); out_iobuf: if (rv != SANLK_AIO_TIMEOUT) free(iobuf); out_close: close_disks(rx.disk, 1); return rv; } int rindex_create(struct task *task, struct sanlk_rindex *ri, struct sanlk_rentry *re, struct sanlk_rentry *re_ret, uint32_t max_hosts, uint32_t num_hosts) { struct rindex_info rx; struct space_info spi; struct leader_record leader; struct paxos_dblock dblock; struct token *rx_token; struct token *res_token; char *rindex_iobuf = NULL; uint64_t ent_offset, res_offset; int sector_size, align_size; int rv; memset(&rx, 0, sizeof(rx)); rx.ri = ri; rx.disk = (struct sync_disk *)&ri->disk; rv = open_disk(rx.disk); if (rv < 0) { log_error("rindex_create open failed %d %s", rv, rx.disk->path); return rv; } /* * Allows only one rindex op for a given lockspace at a time. * If there's already one in progress, this returns EBUSY. * Also collects lockspace info at the same time. */ memset(&spi, 0, sizeof(spi)); rv = lockspace_begin_rindex_op(ri->lockspace_name, RX_OP_CREATE, &spi); if (rv < 0) { log_error("rindex_create lockspace not available %d %.48s", rv, ri->lockspace_name); goto out_close; } rv = read_rindex_header(task, &spi, &rx); if (rv < 0) { log_error("rindex_create failed to read rindex header %d on %s:%llu", rv, rx.disk->path, (unsigned long long)rx.disk->offset); goto out_clear; } sector_size = rx.header.sector_size; align_size = rindex_header_align_size_from_flag(rx.header.flags); log_debug("rindex_create %.48s:%s:%llu %d %d max_res %u", rx.ri->lockspace_name, rx.disk->path, (unsigned long long)rx.disk->offset, sector_size, align_size, rx.header.max_resources); /* used to acquire the internal paxos lease protecting the rindex */ rx_token = setup_rindex_token(&rx, sector_size, align_size, &spi); if (!rx_token) { rv = -ENOMEM; goto out_clear; } /* used to initialize the new paxos lease for the resource */ res_token = setup_resource_token(&rx, re->name, sector_size, align_size, &spi); if (!res_token) { free(rx_token); rv = -ENOMEM; goto out_clear; } log_debug("rindex_create acquire offset %llu sector_size %d align_size %d", (unsigned long long)rx_token->disks[0].offset, rx_token->sector_size, rx_token->align_size); rv = paxos_lease_acquire(task, rx_token, PAXOS_ACQUIRE_OWNER_NOWAIT | PAXOS_ACQUIRE_QUIET_FAIL, &leader, &dblock, 0, 0); if (rv < 0) { /* TODO: sleep and retry if this fails because it's held by another host? */ log_error("rindex_create failed to acquire rindex lease %d", rv); goto out_token; } rv = read_rindex(task, &spi, &rx, &rindex_iobuf); if (rv < 0) { log_error("rindex_create failed to read rindex %d", rv); goto out_lease; } rv = search_entries(&rx, rindex_iobuf, &ent_offset, &res_offset, 1, NULL); if (rv < 0) { log_error("rindex_create failed to find free offset %d", rv); goto out_iobuf; } /* set the location of the new paxos lease */ log_debug("rindex_create found offset %llu for %.48s:%.48s", (unsigned long long)res_offset, rx.ri->lockspace_name, re->name); res_token->disks[0].offset = res_offset; /* write the new paxos lease */ rv = paxos_lease_init(task, res_token, num_hosts, 0, 0); if (rv < 0) { log_error("rindex_create failed to init new lease %d", rv); goto out_iobuf; } rv = update_rindex(task, &spi, &rx, rindex_iobuf, re, ent_offset, res_offset, 0); if (rv < 0) { log_error("rindex_create failed to update rindex %d", rv); goto out_iobuf; } log_debug("rindex_create updated rindex entry %llu for %.48s %llu", (unsigned long long)ent_offset, re->name, (unsigned long long)res_offset); re_ret->offset = res_offset; rv = 0; out_iobuf: free(rindex_iobuf); out_lease: paxos_lease_release(task, rx_token, NULL, &leader, &leader); out_token: free(rx_token); free(res_token); out_clear: lockspace_clear_rindex_op(ri->lockspace_name); out_close: close_disks(rx.disk, 1); return rv; } /* * clear the rindex entry for a given resource lease name and offset * first the rentry is cleared, then the resource lease is cleared */ int rindex_delete(struct task *task, struct sanlk_rindex *ri, struct sanlk_rentry *re, struct sanlk_rentry *re_ret) { struct rindex_info rx; struct space_info spi; struct leader_record leader; struct paxos_dblock dblock; struct token *rx_token; struct token *res_token; char *rindex_iobuf = NULL; uint64_t res_offset = re->offset; uint64_t ent_offset; int sector_size, align_size; int rv; memset(&rx, 0, sizeof(rx)); rx.ri = ri; rx.disk = (struct sync_disk *)&ri->disk; rv = open_disk(rx.disk); if (rv < 0) { log_error("rindex_create open failed %d %s", rv, rx.disk->path); return rv; } /* * Allows only one rindex op for a given lockspace at a time. * If there's already one in progress, this returns EBUSY. * Also collects lockspace info at the same time. */ memset(&spi, 0, sizeof(spi)); rv = lockspace_begin_rindex_op(ri->lockspace_name, RX_OP_DELETE, &spi); if (rv < 0) { log_error("rindex_delete lockspace not available %d %.48s", rv, ri->lockspace_name); goto out_close; } rv = read_rindex_header(task, &spi, &rx); if (rv < 0) { log_error("rindex_delete failed to read rindex header %d on %s:%llu", rv, rx.disk->path, (unsigned long long)rx.disk->offset); goto out_clear; } sector_size = rx.header.sector_size; align_size = rindex_header_align_size_from_flag(rx.header.flags); /* resource lease locations must use the same alignment as the rindex */ if (re->offset && (re->offset % align_size)) { rv = SANLK_RINDEX_OFFSET; goto out_clear; } /* used to acquire the internal paxos lease protecting the rindex */ rx_token = setup_rindex_token(&rx, sector_size, align_size, &spi); if (!rx_token) { rv = -ENOMEM; goto out_clear; } /* used to write the cleared paxos lease for the resource */ res_token = setup_resource_token(&rx, re->name, sector_size, align_size, &spi); if (!res_token) { free(rx_token); rv = -ENOMEM; goto out_clear; } rv = paxos_lease_acquire(task, rx_token, PAXOS_ACQUIRE_OWNER_NOWAIT | PAXOS_ACQUIRE_QUIET_FAIL, &leader, &dblock, 0, 0); if (rv < 0) { /* TODO: sleep and retry if this fails because it's held by another host? */ log_error("rindex_create failed to acquire rindex lease %d", rv); goto out_token; } rv = read_rindex(task, &spi, &rx, &rindex_iobuf); if (rv < 0) { log_error("rindex_delete failed to read rindex %d", rv); goto out_lease; } /* find the entry */ rv = search_entries(&rx, rindex_iobuf, &ent_offset, &res_offset, 0, re->name); if (rv < 0) { log_error("rindex_delete failed to find entry '%s': %d", re->name, rv); goto out_iobuf; } rv = update_rindex(task, &spi, &rx, rindex_iobuf, re, ent_offset, res_offset, 1); if (rv < 0) { log_error("rindex_delete failed to update rindex %d", rv); goto out_iobuf; } /* clear the paxos lease */ res_token->disks[0].offset = res_offset; rv = paxos_lease_init(task, res_token, 0, 1, 0); if (rv < 0) { log_error("rindex_delete failed to init new lease %d", rv); goto out_iobuf; } log_debug("rindex_delete updated rindex entry %llu for %.48s %llu", (unsigned long long)ent_offset, re->name, (unsigned long long)res_offset); re_ret->offset = 0; rv = 0; out_iobuf: free(rindex_iobuf); out_lease: paxos_lease_release(task, rx_token, NULL, &leader, &leader); out_token: free(rx_token); free(res_token); out_clear: lockspace_clear_rindex_op(ri->lockspace_name); out_close: close_disks(rx.disk, 1); return rv; } int rindex_lookup(struct task *task, struct sanlk_rindex *ri, struct sanlk_rentry *re, struct sanlk_rentry *re_ret, uint32_t cmd_flags) { struct rindex_info rx; struct space_info spi; struct rindex_entry re_in; struct rindex_entry *re_end; char *rindex_iobuf = NULL; uint64_t ent_offset, res_offset; int entry_num; int sector_size, align_size; int nolock = cmd_flags & SANLK_RX_NO_LOCKSPACE; int rv; memset(&rx, 0, sizeof(rx)); rx.ri = ri; rx.disk = (struct sync_disk *)&ri->disk; rv = open_disk(rx.disk); if (rv < 0) { return rv; } memset(&spi, 0, sizeof(spi)); if (!nolock) { rv = lockspace_begin_rindex_op(ri->lockspace_name, RX_OP_LOOKUP, &spi); if (rv < 0) { goto out_close; } } rv = read_rindex_header(task, &spi, &rx); if (rv < 0) { goto out_clear; } sector_size = rx.header.sector_size; align_size = rindex_header_align_size_from_flag(rx.header.flags); rv = read_rindex(task, &spi, &rx, &rindex_iobuf); if (rv < 0) { goto out_clear; } if (re->offset && (re->offset % align_size)) { rv = SANLK_RINDEX_OFFSET; goto out_clear; } if (!re->name[0] && !re->offset) { /* find the first free resource lease offset */ rv = search_entries(&rx, rindex_iobuf, &ent_offset, &res_offset, 1, NULL); if (rv < 0) { goto out_iobuf; } memset(re_ret->name, 0, SANLK_NAME_LEN); re_ret->offset = res_offset; rv = 0; } else if (!re->name[0] && re->offset) { /* find the name of the resource lease that the index has recorded for the given resource lease offset */ res_offset = re->offset; entry_num = (res_offset - rx.disk->offset - (2 * align_size)) / align_size; ent_offset = sector_size + (entry_num * sizeof(struct rindex_entry)); re_end = (struct rindex_entry *)(rindex_iobuf + ent_offset); rindex_entry_in(re_end, &re_in); memcpy(re_ret->name, re_in.name, SANLK_NAME_LEN); re_ret->offset = res_offset; rv = 0; } else if (re->name[0] && !re->offset) { /* search the rindex entries for a given resource lease name and if found return the offset of the resource lease */ rv = search_entries(&rx, rindex_iobuf, &ent_offset, &res_offset, 0, re->name); if (rv < 0) { goto out_iobuf; } memcpy(re_ret->name, re->name, SANLK_NAME_LEN); re_ret->offset = res_offset; rv = 0; } else if (re->name[0] && re->offset) { /* find the name of the resource lease that the index has recorded for the given resource lease offset, and if it doesn't match the specified name, then it's an error */ res_offset = re->offset; entry_num = (res_offset - rx.disk->offset - (2 * align_size)) / align_size; ent_offset = sector_size + (entry_num * sizeof(struct rindex_entry)); re_end = (struct rindex_entry *)(rindex_iobuf + ent_offset); rindex_entry_in(re_end, &re_in); if (strncmp(re->name, re_in.name, SANLK_NAME_LEN)) rv = SANLK_RINDEX_DIFF; else rv = 0; memcpy(re_ret->name, re_in.name, SANLK_NAME_LEN); re_ret->offset = res_offset; } out_iobuf: free(rindex_iobuf); out_clear: if (!nolock) lockspace_clear_rindex_op(ri->lockspace_name); out_close: close_disks(rx.disk, 1); return rv; } int rindex_update(struct task *task, struct sanlk_rindex *ri, struct sanlk_rentry *re, struct sanlk_rentry *re_ret, uint32_t cmd_flags) { struct rindex_info rx; struct space_info spi; char *rindex_iobuf = NULL; uint64_t ent_offset, res_offset; int entry_num; int sector_size, align_size; int op_remove = 0, op_add = 0; int nolock = cmd_flags & SANLK_RX_NO_LOCKSPACE; int rv; memset(&rx, 0, sizeof(rx)); rx.ri = ri; rx.disk = (struct sync_disk *)&ri->disk; rv = open_disk(rx.disk); if (rv < 0) { return rv; } memset(&spi, 0, sizeof(spi)); if (!nolock) { rv = lockspace_begin_rindex_op(ri->lockspace_name, RX_OP_UPDATE, &spi); if (rv < 0) { goto out_close; } } rv = read_rindex_header(task, &spi, &rx); if (rv < 0) { goto out_clear; } rv = read_rindex(task, &spi, &rx, &rindex_iobuf); if (rv < 0) { goto out_clear; } sector_size = rx.header.sector_size; align_size = rindex_header_align_size_from_flag(rx.header.flags); if (re->offset && (re->offset % align_size)) { rv = SANLK_RINDEX_OFFSET; goto out_clear; } res_offset = re->offset; entry_num = (res_offset - rx.disk->offset - (2 * align_size)) / align_size; ent_offset = sector_size + (entry_num * sizeof(struct rindex_entry)); if ((cmd_flags & SANLK_RXUP_REM) && re->offset) { op_remove = 1; } else if ((cmd_flags & SANLK_RXUP_ADD) && re->name[0] && re->offset) { op_add = 1; } else { rv = -EINVAL; goto out_iobuf; } rv = update_rindex(task, &spi, &rx, rindex_iobuf, re, ent_offset, res_offset, op_remove); if (rv < 0) { log_error("rindex_update failed to update rindex %d", rv); goto out_iobuf; } rv = 0; if (op_remove) { memset(re_ret->name, 0, SANLK_NAME_LEN); re_ret->offset = 0; } if (op_add) { memcpy(re_ret->name, re->name, SANLK_NAME_LEN); re_ret->offset = res_offset; } out_iobuf: free(rindex_iobuf); out_clear: if (!nolock) lockspace_clear_rindex_op(ri->lockspace_name); out_close: close_disks(rx.disk, 1); return rv; } int rindex_rebuild(struct task *task, struct sanlk_rindex *ri, uint32_t cmd_flags) { struct rindex_info rx; struct rindex_entry re_new; struct rindex_entry re_end; struct space_info spi; struct leader_record leader; struct paxos_dblock dblock; struct token *rx_token; struct token *res_token; struct sanlk_resource res; char off_str[16]; char *rindex_iobuf = NULL; uint64_t res_offset; uint64_t ent_offset; uint32_t max_resources; int sector_size, align_size; int nolock = cmd_flags & SANLK_RX_NO_LOCKSPACE; int i, rv; memset(&rx, 0, sizeof(rx)); rx.ri = ri; rx.disk = (struct sync_disk *)&ri->disk; rv = open_disk(rx.disk); if (rv < 0) { log_error("rindex_rebuild open failed %d %s", rv, rx.disk->path); return rv; } /* * Allows only one rindex op for a given lockspace at a time. * If there's already one in progress, this returns EBUSY. * Also collects lockspace info at the same time. */ memset(&spi, 0, sizeof(spi)); if (!nolock) { rv = lockspace_begin_rindex_op(ri->lockspace_name, RX_OP_REBUILD, &spi); if (rv < 0) { log_error("rindex_rebuild lockspace not available %d %.48s", rv, ri->lockspace_name); goto out_close; } } rv = read_rindex_header(task, &spi, &rx); if (rv < 0) { log_error("rindex_rebuild failed to read rindex header %d on %s:%llu", rv, rx.disk->path, (unsigned long long)rx.disk->offset); goto out_clear; } sector_size = rx.header.sector_size; align_size = rindex_header_align_size_from_flag(rx.header.flags); max_resources = rx.header.max_resources; if (!max_resources) max_resources = size_to_max_resources(sector_size, align_size); log_debug("rindex_rebuild %.48s:%s:%llu %d %d max_res %u", rx.ri->lockspace_name, rx.disk->path, (unsigned long long)rx.disk->offset, sector_size, align_size, max_resources); /* used to acquire the internal paxos lease protecting the rindex */ rx_token = setup_rindex_token(&rx, sector_size, align_size, &spi); if (!rx_token) { rv = -ENOMEM; goto out_clear; } memset(&res, 0, sizeof(res)); res_token = setup_resource_token(&rx, res.name, sector_size, align_size, &spi); if (!res_token) { free(rx_token); rv = -ENOMEM; goto out_clear; } if (!nolock) { rv = paxos_lease_acquire(task, rx_token, PAXOS_ACQUIRE_OWNER_NOWAIT | PAXOS_ACQUIRE_QUIET_FAIL, &leader, &dblock, 0, 0); if (rv < 0) { /* TODO: sleep and retry if this fails because it's held by another host? */ log_error("rindex_rebuild failed to acquire rindex lease %d", rv); goto out_token; } } rv = read_rindex(task, &spi, &rx, &rindex_iobuf); if (rv < 0) { log_error("rindex_rebuild failed to read rindex %d", rv); goto out_lease; } /* * Zero all the entries after the header sector. Entries will be * recreated in the zeroed space if corresponding resource leases are * found. */ memset(rindex_iobuf + sector_size, 0, align_size - sector_size); /* * We read each potential resource lease offset to check if a * lease exists there. It's ok if there is none, and we don't * want to log errors if none is found. */ res_token->flags |= T_CHECK_EXISTS; /* * Read each potential resource lease area and add an rindex entry * for each one that's found. Resource leases begin after * the rindex area and the rindex lease area. */ res_offset = rx.disk->offset + (2 * align_size); for (i = 0; i < max_resources; i++) { memset(&re_new, 0, sizeof(re_new)); memset(&re_end, 0, sizeof(re_end)); memset(&res, 0, sizeof(res)); memset(res_token->r.name, 0, SANLK_NAME_LEN); res_token->disks[0].offset = res_offset; rv = paxos_read_resource(task, res_token, &res); offset_to_str(res_offset, sizeof(off_str), off_str); /* end of device */ if (rv == -EMSGSIZE) { log_debug("rindex_rebuild reached end of device at %d %s", i, off_str); break; } if (rv == SANLK_OK) { log_debug("rindex_rebuild found %.48s at %d %s", res.name, i, off_str); re_new.res_offset = res_offset; memcpy(re_new.name, res.name, SANLK_NAME_LEN); rindex_entry_out(&re_new, &re_end); /* Within rindex, entries begin after the header sector */ ent_offset = sector_size + (i * sizeof(struct rindex_entry)); memcpy(rindex_iobuf + ent_offset, &re_end, sizeof(re_end)); } else if ((i + 1) == max_resources) { log_debug("rindex_rebuild found no resource at last %d %s %d", i, off_str, rv); } res_offset += align_size; } rv = write_iobuf(rx.disk->fd, rx.disk->offset, rindex_iobuf, align_size, task, task->use_aio, spi.io_timeout, NULL); if (rv < 0) { if (rv != SANLK_AIO_TIMEOUT) free(rindex_iobuf); log_error("rindex_rebuild write failed %d %s", rv, rx.disk->path); goto out_lease; } rv = 0; free(rindex_iobuf); out_lease: if (!nolock) paxos_lease_release(task, rx_token, NULL, &leader, &leader); out_token: free(rx_token); free(res_token); out_clear: if (!nolock) lockspace_clear_rindex_op(ri->lockspace_name); out_close: close_disks(rx.disk, 1); return rv; } sanlock/src/rindex.h000066400000000000000000000021021516326262600147450ustar00rootroot00000000000000/* * Copyright 2018 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __RINDEX_H__ #define __RINDEX_H__ int rindex_format(struct task *task, struct sanlk_rindex *ri); int rindex_rebuild(struct task *task, struct sanlk_rindex *ri, uint32_t cmd_flags); int rindex_lookup(struct task *task, struct sanlk_rindex *ri, struct sanlk_rentry *re, struct sanlk_rentry *re_ret, uint32_t cmd_flags); int rindex_update(struct task *task, struct sanlk_rindex *ri, struct sanlk_rentry *re, struct sanlk_rentry *re_ret, uint32_t cmd_flags); int rindex_create(struct task *task, struct sanlk_rindex *ri, struct sanlk_rentry *re, struct sanlk_rentry *re_ret, uint32_t num_hosts, uint32_t max_hosts); int rindex_delete(struct task *task, struct sanlk_rindex *ri, struct sanlk_rentry *re, struct sanlk_rentry *re_ret); #endif sanlock/src/rindex_disk.h000066400000000000000000000070661516326262600157750ustar00rootroot00000000000000/* * Copyright 2018 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __RINDEX_DISK_H__ #define __RINDEX_DISK_H__ /* * The resource index uses two align-size areas: * * 1. The first area (the rindex itself) holds a header and * entries. with each entry recording a resource lease name * and the offset of that lease (the resource lease disk areads * follow the two align-size disk areas used by the resource index.) * * 2. The second area holds an internal paxos lease that sanlock * uses to protect updates to the rindex in the first area. * * The rindex is one align-size area containing between 256 and * 2048 sectors, depending on the sector_size and align_size. * * sector 0 of the index holds the rindex_header. * After this, sectors 1 to 250/500/1000/2000 hold rindex_entry's. * The remaining sectors in the align-size area are unused. * * 512 byte sectors hold 8 entries per sector, * 4096 byte sectors hold 64 entries per sector. * * ALIGN1M / SECTOR512 = 2000 sectors used for rindex, 16000 max entries * ALIGN1M / SECTOR4K = 250 sectors used for rindex, 16000 max entries * ALIGN2M / SECTOR4K = 500 sectors used for rindex, 32000 max entries * ALIGN4M / SECTOR4K = 1000 sectors used for rindex, 64000 max entries * ALIGN8M / SECTOR4K = 2000 sectors used for rindex, 128000 max entries * * rindex_header.sector_size = 512 | 4096 * * area_size = 1M | 2M | 4M | 8M * * rindex_header.max_resources defaults to 4096 to limit searching. * The caller can specify max_resources up to the max supported by * the sector_size/align_size combination. * * rindex_header.rindex_offset: * location of rindex_header from start of device, set by caller, * must be multiple of area_size. (rindex_offset will often be * 1*area_size because rindex typically follows the lockspace area * which typically starts at offset 0 on the device.) * * entry_size = 64 bytes * * entry_index = N = 0 to (max_resources - 1) * * rindex_entry N offset = rindex_offset + sector_size + (N * entry_size) * (the sector_size contains the rindex_header) * * rindex_entry N holds information about the resource lease in * the N'th area following the two areas used by the resource index. * * resource_leases_start = rindex_offset + (2 * area_size) * resource leases begin after the two resource index areas. * (rindex_offset will often be area_size, so resource_leases_start * will often by 3*area_size) * * resource lease N offset = resource_leases_start + (N * area_size) * * rindex_entry[N].res_offset = resource lease N offset */ #define RINDEX_DISK_MAGIC 0x01042018 #define RINDEX_DISK_VERSION_MAJOR 0x00010000 #define RINDEX_DISK_VERSION_MINOR 0x00000002 /* MINOR 2: addition of align flags */ /* rindex_header flags */ #define RHF_ALIGN_1M 0x00000001 #define RHF_ALIGN_2M 0x00000002 #define RHF_ALIGN_4M 0x00000004 #define RHF_ALIGN_8M 0x00000008 struct rindex_header { uint32_t magic; uint32_t version; uint32_t flags; /* RHF_ */ uint32_t sector_size; uint32_t max_resources; uint32_t unused; uint64_t rx_offset; /* location of rindex_header from start of disk */ char lockspace_name[NAME_ID_SIZE]; }; #define MAX_RINDEX_ENTRIES_1M 16000 #define MAX_RINDEX_ENTRIES_8M 128000 /* The entry size is fixed */ struct rindex_entry { uint64_t res_offset; /* location of resource from start of disk */ uint32_t flags; uint32_t unused; char name[NAME_ID_SIZE]; }; #endif sanlock/src/sanlock.8000066400000000000000000000650621516326262600150440ustar00rootroot00000000000000.TH SANLOCK 8 2026-02-27 .SH NAME sanlock \- shared storage lock manager .SH SYNOPSIS .B sanlock [COMMAND] [ACTION] ... .SH DESCRIPTION sanlock is a lock manager for shared storage environments. It allows applications running on multiple hosts to coordinate access to shared resources, such as data objects on shared storage systems like a SAN, preventing data corruption and ensuring data integrity. .SH OPTIONS .P COMMAND can be one of three primary top level choices .P .BR "sanlock daemon" " start daemon" .br .BR "sanlock client" " send request to daemon (default command if none given)" .br .BR "sanlock direct" " access storage directly (no coordination with daemon)" .SS Daemon Command .BR "sanlock daemon" " [options]" .BR -D " " no fork and print all logging to stderr .BR -Q " 0|1" quiet error messages for common lock contention .BR -R " 0|1" renewal debugging, log debug info for each renewal .BI -L " pri" write logging at priority level and up to logfile (-1 none) .BI -S " pri" write logging at priority level and up to syslog (-1 none) .BI -U " uid" user id .BI -G " gid" group id .BI -H " num" renewal history size .BI -t " num" max worker threads .BI -g " sec" seconds for graceful recovery .BR -w " 0|1" use watchdog through wdmd .BI -o " sec" io timeout .BR -h " 0|1" use high priority (RR) scheduling .BI -l " num" use mlockall (0 none, 1 current, 2 current and future) .BI -b " sec" seconds a host id bit will remain set in delta lease bitmap .BI -e " str" unique local host name used in delta leases as host_id owner .\" non-aio is untested and may not work .\" .BR \-a " 0|1" .\" use async i/o .SS Client Command .B "sanlock client" .I action [options] .B sanlock client status Print processes, lockspaces, and resources being managed by the sanlock daemon. Add -D to show extra internal daemon status for debugging. Add -o p to show resources by pid, or -o s to show resources by lockspace. .B sanlock client host_status Print state of host_id leases read during the last renewal. State of all lockspaces is shown (use -s to select one). Add -D to show extra internal daemon status for debugging. .B sanlock client gets Print lockspaces being managed by the sanlock daemon. The LOCKSPACE string will be followed by ADD or REM if the lockspace is currently being added or removed. Add -h 1 to also show hosts in each lockspace. .BR "sanlock client renewal -s" " LOCKSPACE" Print a history of renewals with timing details. See the Renewal history section below. .B sanlock client log_dump Print the sanlock daemon internal debug log. .B sanlock client shutdown Ask the sanlock daemon to exit. Without the force option (-f 0), the command will be ignored if any lockspaces exist. With the force option (-f 1), any registered processes will be killed, their resource leases released, and lockspaces removed. With the wait option (-w 1), the command will wait for a result from the daemon indicating that it has shut down and is exiting, or cannot shut down because lockspaces exist (command fails). .BR "sanlock client init -s" " LOCKSPACE" Tell the sanlock daemon to initialize a lockspace on disk. The -o option can be used to specify the io timeout to be written in the host_id leases. The -Z and -A options can be used to specify the sector size and align size, and both should be set together. Use -N 1 to include the NO_TIMEOUT flag in the newly formatted leases. Use -C 1 to request the use of CAW leases if supported, or -C 0 to not use CAW leases. (Also see sanlock direct init.) .BR "sanlock client init -r" " RESOURCE" Tell the sanlock daemon to initialize a resource lease on disk. The -Z and -A options can be used to specify the sector size and align size, and both should be set together. Use -C 1 to request the use of CAW leases if supported, or -C 0 to not use CAW leases. (Also see sanlock direct init.) .BR "sanlock client read -s" " LOCKSPACE" Tell the sanlock daemon to read a lockspace from disk. Only the LOCKSPACE path and offset are required. If host_id is zero, the first record at offset (host_id 1) is used. The complete LOCKSPACE is printed. Add -D to print other details. (Also see sanlock direct read_leader.) .BR "sanlock client read -r" " RESOURCE" Tell the sanlock daemon to read a resource lease from disk. Only the RESOURCE path and offset are required. The complete RESOURCE is printed. Add -D to print other details. (Also see sanlock direct read_leader.) .BR "sanlock client init_host -s" " LOCKSPACE" Tell the sanlock daemon to initialize a single host_id lease on disk. The host_id specified in the -s arg will be used, and written as the lease owner. Optionally specify host name with -e, generation with -g, and timestamp with -t. Use -Z to specify sector size. Use -N 1 to include the NO_TIMEOUT flag in the reformatted lease. Use -C 1 to request the use of CAW leases if supported, or -C 0 to not use CAW leases. (Also see sanlock direct init_host for more information.) .BR "sanlock client add_lockspace -s" " LOCKSPACE" Tell the sanlock daemon to acquire the host_id lease for the host_id specified in LOCKSPACE. This is also referred to as "joining" the lockspace. With a host_id lease held for the lockspace, the host is then able to acquire resource locks in the lockspace. Use -o to specify the io timeout of the acquiring host, which will be written in the host_id lease. .BR "sanlock client inq_lockspace -s" " LOCKSPACE" Inquire about the state of the lockspace in the sanlock daemon, whether it is being added or removed, or is joined. .BR "sanlock client rem_lockspace -s" " LOCKSPACE" Tell the sanlock daemon to release the specified host_id in the lockspace. Any processes holding resource leases in this lockspace will be killed, and the resource leases not released. .BR "sanlock client command -r" " RESOURCE " \ \fB-c\fP " " \fIpath\fP " " \fIargs\fP Register with the sanlock daemon, acquire the specified resource lease, and exec the command at path with args. When the command exits, the sanlock daemon will release the lease. -c must be the final option. .BR "sanlock client spawn -r" " RESOURCE " \ \fB-c\fP " " \fICOUNT\fP " " \fICMD\fP " [" \fIARG\fP "...] " \ \fR[\fP\fB-c\fP " " \fICOUNT\fP " " \fICMD\fP " [" \fIARG\fP "...]\fR]\fP..." Register with the sanlock daemon, acquire the specified resource lease, fork and exec each command specified by .B -c sequentially as separate processes, checking the exit status of each process, and only moving to the next process on success. After all processes are successfully executed, or at the first failure, the lease is released explicitly before exiting. Use -P 1 for persistent locks that will not be dropped if the spawn process dies while a child process is running. Use -h 1 to report a conflicting lock owner. Use -O 1 to acquire an orphan lock. Use -d 1 to skip the on-disk resource lease update when releasing the resource lease after all commands complete successfully (useful when the commands have removed the lease storage.) .BR "sanlock client acquire -r" " RESOURCE " \ \fB-p\fP " " \fIpid\fP .br .BR "sanlock client release -r" " RESOURCE " \ \fB-p\fP " " \fIpid\fP Tell the sanlock daemon to acquire or release the specified resource lease for the given pid. The pid must be registered with the sanlock daemon. acquire can optionally take a versioned RESOURCE string RESOURCE:lver, where lver is the version of the lease that must be acquired, or fail. Use -C in place of -p to specify client_id. .BR "sanlock client convert -r" " RESOURCE " \ \fB-p\fP " " \fIpid\fP Tell the sanlock daemon to convert the mode of the specified resource lease for the given pid. If the existing mode is exclusive (default), the mode of the lease can be converted to shared with RESOURCE:SH. If the existing mode is shared, the mode of the lease can be converted to exclusive with RESOURCE (no :SH suffix). Use -C in place of -p to specify client_id. .BI "sanlock client inquire -p" " pid" Print the resource leases held the given pid. The format is a versioned RESOURCE string "RESOURCE:lver" where lver is the version of the lease held. Use -C in place of -p to specify client_id. .BR "sanlock client request -r" " RESOURCE " \ \fB-f\fP " " \fIforce_mode\fP Request the owner of a resource do something specified by force_mode. A versioned RESOURCE:lver string must be used with a greater version than is presently held. Zero lver and force_mode clears the request. .BR "sanlock client examine -r" " RESOURCE" Examine the request record for the currently held resource lease and carry out the action specified by the requested force_mode. .BR "sanlock client examine -s" " LOCKSPACE" Examine requests for all resource leases currently held in the named lockspace. Only lockspace_name is used from the LOCKSPACE argument. .BR "sanlock client set_event -s" " LOCKSPACE " \ \fB-i\fP " " \fIhost_id\fP " " \ \fB-g\fP " " \fIgen\fP " " \ \fB-e\fP " " \fInum\fP " " \ \fB-d\fP " " \fInum\fP Set an event for another host. When the sanlock daemon next renews its host_id lease for the lockspace it will: set the bit for the host_id in its bitmap, and set the generation, event and data values in its own host_id lease. An application that has registered for events from this lockspace on the destination host will get the event that has been set when the destination sees the event during its next host_id lease renewal. .BR "sanlock client set_config -s" " LOCKSPACE Set a configuration value for a lockspace. Only lockspace_name is used from the LOCKSPACE argument. The USED flag has the same effect on a lockspace as a process holding a resource lease that will not exit. The USED_BY_ORPHANS flag means that an orphan resource lease will have the same effect as the USED. The -o option can be used to update the lockspace's io timeout. .br \-u 0|1 Set (1) or clear (0) the USED flag. .br \-O 0|1 Set (1) or clear (0) the USED_BY_ORPHANS flag. .BR "sanlock client set_host -s" " LOCKSPACE " \ \fB-i\fP " " \fIhost_id\fP " " \ \fB-g\fP " " \fIgen\fP " " \ \fB-F\fP " " \fIflag_name\fP When flag_name is DEAD_EXT, the DEAD_EXT flag is set in the host_id lease for the specified host_id. If the current host_id lease generation does not match the specified generation, then the command will fail. With DEAD_EXT set, the host_id+generation will be considered dead, and resource locks held by the specified host_id+generation will be free for other hosts to acquire. DEAD_EXT should only be set for a host if that host can no longer modify the shared resources that were protected by the resource locks in the lockspace. .SS Direct Command .B "sanlock direct" .I action [options] .\" non-aio is untested and may not work .\" .BR \-a " 0|1" .\" use async i/o .BI -o " sec" io timeout in seconds .BR "sanlock direct init -s" " LOCKSPACE" .br .BR "sanlock direct init -r" " RESOURCE" Initialize storage for a lockspace or resource. Use the -Z and -A flags to specify the sector size and align size. The max hosts that can use the lockspace/resource (and the max possible host_id) is determined by the sector/align size combination. Possible combinations are: 512/1M, 4096/1M, 4096/2M, 4096/4M, 4096/8M. Lockspaces and resources both use the same amount of space (align_size) for each combination. When initializing a lockspace, sanlock initializes host_id leases (delta leases) for max_hosts in the given space. When initializing a resource, sanlock initializes a single resource lock (paxos lease) in the space. With -s, the -o option specifies the io timeout to be written in the host_id leases. With -r, the -z 1 option invalidates the resource lease on disk so it cannot be used until reinitialized normally. Use -N 1 to include NO_TIMEOUT in newly formatted lockspace host_id leases. Use -C 1 to request the use of COMPARE AND WRITE (CAW) leases if supported, or -C 0 to not use CAW leases. .BR "sanlock direct init_host -s" " LOCKSPACE" Initialize a single host_id lease. The host_id specified in the -s arg will be used, and written as the lease owner (leader.owner_id). Optionally specify host name (leader.resource_name) with -e, generation number (leader.owner_generation) with -g, and timestamp (leader.timestamp) with -t (timestamp value 1 is special, and causes the current time to be written in the timestamp field. A timestamp value of 0 is means the host_id lease is free, as usual.) The -Z and -o options apply as with direct init. Use -N 1 to include NO_TIMEOUT in the reformatted host_id lease. Use -C 1 to request the use of CAW leases if supported, or -C 0 to not use CAW leases. .BR "sanlock direct read_leader -s" " LOCKSPACE" .br .BR "sanlock direct read_leader -r" " RESOURCE" Read a leader record from disk and print the fields. The leader record is the single sector of a delta lease, or the first sector of a paxos lease. .BR "sanlock direct read -s" " LOCKSPACE" .br .BR "sanlock direct read -r" " RESOURCE" Read a complete lockspace or resource from disk and print it. .\" .P .\" .BR "sanlock direct acquire_id -s" " LOCKSPACE" .\" .br .\" .BR "sanlock direct renew_id -s" " LOCKSPACE" .\" .br .\" .BR "sanlock direct release_id -s" " LOCKSPACE" .\" .\" Acquire, renew, or release a host_id directly to disk, independent from .\" the sanlock daemon. Not for general use. This should only be used for .\" testing or for manual recovery in an emergency. .\" .\" .P .\" .BR "sanlock direct acquire -r" " RESOURCE " \ .\" \fB-i\fP " " \fInum\fP " " \fB-g\fP " " \fInum\fP .\" .br .\" .BR "sanlock direct release -r" " RESOURCE " \ .\" \fB-i\fP " " \fInum\fP " " \fB-g\fP " " \fInum\fP .\" .\" Not supported. Not for general use. .\" .BI "sanlock direct dump" " path" \ \fR[\fP\fB:\fP\fIoffset\fP\fR[\fP\fB:\fP\fIsize\fP\fR]]\fP Read disk sectors and print leader records for delta or paxos leases. Add -f 1 to print the request record values for paxos leases, host_ids set in delta lease bitmaps. .SS LOCKSPACE option string .BR \-s " " \fIlockspace_name\fP:\fIhost_id\fP:\fIpath\fP:\fIoffset\fP .P .IR lockspace_name " name of lockspace" .br .IR host_id " local host identifier in lockspace" .br .IR path " path to storage to use for leases" .br .IR offset " offset on path (bytes)" .br .SS RESOURCE option string .BR \-r " " \fIlockspace_name\fP:\fIresource_name\fP:\fIpath\fP:\fIoffset\fP .P .IR lockspace_name " name of lockspace" .br .IR resource_name " name of resource" .br .IR path " path to storage to use leases" .br .IR offset " offset on path (bytes)" .SS RESOURCE option string with suffix .BR \-r " " \fIlockspace_name\fP:\fIresource_name\fP:\fIpath\fP:\fIoffset\fP:\fIlver\fP .P .IR lver " leader version" .BR \-r " " \fIlockspace_name\fP:\fIresource_name\fP:\fIpath\fP:\fIoffset\fP:SH .P SH indicates shared mode .SS Defaults .B sanlock help shows the default values for the options above. .B sanlock version shows the build version. .SH OTHER .SS Request/Examine The first part of making a request for a resource is writing the request record of the resource (the sector following the leader record). To make a successful request: .IP \(bu 2 RESOURCE:lver must be greater than the lver presently held by the other host. This implies the leader record must be read to discover the lver, prior to making a request. .IP \(bu 2 RESOURCE:lver must be greater than or equal to the lver presently written to the request record. Two hosts may write a new request at the same time for the same lver, in which case both would succeed, but the force_mode from the last would win. .IP \(bu 2 The force_mode must be greater than zero. .IP \(bu 2 To unconditionally clear the request record (set both lver and force_mode to 0), make request with RESOURCE:0 and force_mode 0. .P The owner of the requested resource will not know of the request unless it is explicitly told to examine its resources via the "examine" api/command, or otherwise notfied. The second part of making a request is notifying the resource lease owner that it should examine the request records of its resource leases. The notification will cause the lease owner to automatically run the equivalent of "sanlock client examine -s LOCKSPACE" for the lockspace of the requested resource. The notification is made using a bitmap in each host_id lease. Each bit represents each of the possible host_ids (1-2000). If host A wants to notify host B to examine its resources, A sets the bit in its own bitmap that corresponds to the host_id of B. When B next renews its host_id lease, it reads the host_id leases for all hosts and checks each bitmap to see if its own host_id has been set. It finds the bit for its own host_id set in A's bitmap, and examines its resource request records. (The bit remains set in A's bitmap for set_bitmap_seconds.) .I force_mode determines the action the resource lease owner should take: .IP \[bu] 2 FORCE (1): kill the process holding the resource lease. When the process has exited, the resource lease will be released, and can then be acquired by anyone. The kill signal is SIGKILL (or SIGTERM if SIGKILL is restricted.) .IP \[bu] 2 GRACEFUL (2): run the program configured by sanlock_killpath against the process holding the resource lease. If no killpath is defined, then FORCE is used. .P .SS Persistent and orphan resource leases A resource lease can be acquired with the PERSISTENT flag (-P 1). If the process holding the lease exits, the lease will not be released, but kept on an orphan list. Another local process can acquire an orphan lease using the ORPHAN flag (-O 1), or release the orphan lease using the ORPHAN flag (-O 1). All orphan leases can be released by setting the lockspace name (-s lockspace_name) with no resource name. .P .SS Renewal history sanlock saves a limited history of lease renewal information in each lockspace. See sanlock.conf renewal_history_size to set the amount of history or to disable (set to 0). IO times are measured in delta lease renewal (each delta lease renewal includes one read and one write). For each successful renewal, a record is saved that includes: .IP \[bu] 2 the timestamp written in the delta lease by the renewal .IP \[bu] 2 the time in milliseconds taken by the delta lease read .IP \[bu] 2 the time in milliseconds taken by the delta lease write .P Also counted and recorded are the number io timeouts and other io errors that occur between successful renewals. Two consecutive successful renewals would be recorded as: .br .nf timestamp=5332 read_ms=482 write_ms=5525 next_timeouts=0 next_errors=0 timestamp=5353 read_ms=99 write_ms=3161 next_timeouts=0 next_errors=0 .fi Those fields are: .IP \[bu] 2 timestamp is the value written into the delta lease during that renewal. .IP \[bu] 2 read_ms/write_ms are the milliseconds taken for the renewal read/write ios. .IP \[bu] 2 next_timeouts are the number of io timeouts that occurred after the renewal recorded on that line, and before the next successful renewal on the following line. .IP \[bu] 2 next_errors are the number of io errors (not timeouts) that occurred after renewal recorded on that line, and before the next successful renewal on the following line. .P The command 'sanlock client renewal -s lockspace_name' reports the full history of renewals saved by sanlock, which by default is 180 records, about 1 hour of history when using a 20 second renewal interval for a 10 second io timeout. .P .SS Configurable watchdog timeout Watchdog devices usually have a 60 second timeout, but some devices have a configurable timeout. To use a different watchdog timeout, set sanlock.conf watchdog_fire_timeout (in seconds) to a value supported by the device. The same watchdog_fire_timeout must be configured on all hosts (so all hosts must have watchdog devices that support the same timeout). Unmatching values will invalidate the lease protection provided by the watchdog. watchdog_fire_timeout and io_timeout should usually be configured together. By default, sanlock uses watchdog_fire_timeout=60 with io_timeout=10. Other combinations to consider are: .br watchdog_fire_timeout=30 with io_timeout=5 .br watchdog_fire_timeout=10 with io_timeout=2 Smaller values make it more likely that a host will be reset by the watchdog while waiting for slow io to complete or for temporary io failures to be resolved. Spurious watchdog resets will also become more likely due to independent, overlapping lockspace outages, each of which would be inconsequential by itself. .P .SH FILES /etc/sanlock/sanlock.conf The current settings in use by the sanlock daemon can be seen in the output of 'sanlock status -D'. .IP \[bu] 2 quiet_fail = 1 .br See -Q .IP \[bu] 2 debug_renew = 0 .br See -R .IP \[bu] 2 logfile_priority = 4 .br See -L .IP \[bu] 2 logfile_use_utc = 0 .br Use UTC instead of local time in log messages. .IP \[bu] 2 syslog_priority = 3 .br See -S .IP \[bu] 2 names_log_priority = 6 .br Log resource names at this priority level (uses syslog priority numbers). If this number less than or equal to logfile_priority, each requested resource name and location is recorded in sanlock.log. .IP \[bu] 2 use_watchdog = 1 .br See -w .IP \[bu] 2 high_priority = 1 .br See -h .IP \[bu] 2 mlock_level = 1 .br See -l .IP \[bu] 2 sh_retries = 8 .br The number of times to try acquiring a paxos lease when acquiring a shared lease when the paxos lease is held by another host acquiring a shared lease. .IP \[bu] 2 uname = sanlock .br The sanlock daemon will attempt to use this user id. Ignored when use_compare_and_write is enabled, which requires running as root. .IP \[bu] 2 gname = sanlock .br The sanlock daemon will attempt to use this group id. Ignored when use_compare_and_write is enabled, which requires running as root. .IP \[bu] 2 our_host_name = .br A unique name that a host uses to ensure exclusive ownership of a lockspace host_id (delta lease owner.) The maximum length is 48 characters. If no value is provided in sanlock.conf or on the command line (-e), sanlock attempts to set our_host_name from /sys/devices/virtual/dmi/id/product_uuid. If that is not available, sanlock generates a random uuid to use as our_host_name. Using a fixed our_host_name value will reduce delays when using a lockspace. Using product_uuid will reduce delays further. .IP \[bu] 2 renewal_read_extend_sec = .br If a renewal read i/o times out, wait this many additional seconds for that read to complete at the start of the subsequent renewal attempt. When not configured, sanlock waits for an additional io_timeout seconds for a previous timed out read to complete. .IP \[bu] 2 renewal_history_size = 180 .br See -H .IP \[bu] 2 paxos_debug_all = 0 .br Include all details in the paxos debug logging. .IP \[bu] 2 debug_io = .br Add debug logging for each i/o. "submit" (no quotes) produces debug output at submission time, "complete" produces debug output at completion time, and "submit,complete" (no space) produces both. .IP \[bu] 2 max_sectors_kb = | .br Set to "ignore" (no quotes) to prevent sanlock from checking or changing max_sectors_kb for the lockspace disk when starting a lockspace. Set to "align" (no quotes) to set max_sectors_kb for the lockspace disk to the align size of the lockspace. Set to a number to set a specific number of KB for all lockspace disks. A larger existing max_sectors_kb value will not be reduced by this setting. .IP \[bu] 2 debug_clients = 0 .br Enable or disable debug logging for all client connections to the sanlock daemon. .IP \[bu] 2 debug_cmd = +|- .br Enable (+name) or disable (-name) debug logging at the command processing level for specifically named commands, e.g. "debug_cmd = +acquire", or "debug_cmd = -inq_lockspace". Repeat this line for each command name. Use a plus prefix before the name to enable and a minus prefix to disable. By default sanlock disables some command level debugging for commands that are often repetitive and fill the in memory debug buffer. This only affects debug logging, not errors or warnings, and disabling command level debugging for a command does not disable lower level debugging for that command. Special values +all and -all can be used to enable or disable all commands, and can be used before or after other debug_cmd lines. .IP \[bu] 2 debug_hosts = 1 .br Log information about other host_id lease renewals. When set to 1 (the default), messages are logged when a host_id lease is observed reaching the failed and dead states. When set to 2, messages are logged when any update (e.g. renewal) is observed for another host_id lease. When set to 0, neither are logged. .IP \[bu] 2 write_init_io_timeout = .br The io timeout to use when initializing ondisk lease structures for a lockspace or resource. This timeout is not used as a part of either lease algorithm (as the standard io_timeout is.) .IP \[bu] 2 max_worker_threads = .br See -t .IP \[bu] 2 io_timeout = .br The io timeout for disk operations, most notably delta lease renewals. This value is basis for calculating most other timeout values. (Some special cases may use a different io timeout.) Tune this value with caution, it can substantially alter the overall sanlock behavior. .IP \[bu] 2 watchdog_fire_timeout = .br The watchdog device timeout. The watchdog device must support the specified value. It is critical that all hosts use the same value. Not doing so will invalidate the lease protection provided by sanlock. The io_timeout should usually be tuned along with this value, e.g. watchdog_fire_timeout = 30 with io_timeout = 5. .IP \[bu] 2 use_hugepages = .br Set to "all" to use transparent hugepages (2MB via MADV_HUGEPAGE.) This should minimize, or prevent, splitting read io's on lease areas. 2MB is allocated for 1MB lease areas, causing some extra memory usage. Set to "none" to disable. .IP \[bu] 2 use_compare_and_write = .br Set to "yes" to use CAW leases when supported by storage, unless the user/API requests that non-CAW leases be used. (opt-out) Set to "no" to not use CAW leases, even if requested by the user. Set to "allow" to use CAW if supported by storage and requested by the user/API. (opt-in) .IP \[bu] 2 create_old_delta_disk_version = .br Format delta leases using the specified old version number, e.g. 3.4. This allows new versions to remain compatible with old versions. Some new features will not be usable with the old version. .IP \[bu] 2 caw_fua = 1 .br caw_dpo = 1 .br Set to 0 to disable the FUA and/or DPO flags for SCSI COMPARE AND WRITE commands. Using FUA (Force Unit Access) ensures that an update is persistent on disk when the CAW command completes. This ensures a lock does not "disappear" if the device is abruptly restarted immediately after a lock is written (e.g. by power less.) Using DPO (Disable Page Out) indicates to the device that CAW lock data is not useful to keep in the device cache. If a device does not support FUA/DPO flags, setting these to 0 may allow sanlock to use CAW. .SH SEE ALSO .BR wdmd (8) sanlock/src/sanlock.conf000066400000000000000000000023061516326262600156120ustar00rootroot00000000000000# sanlock config file # # Values set here can be overridden on the command line. # See 'man sanlock' and 'sanlock -h' for more information. # # Example settings: # # quiet_fail = 1 # command line: -Q 0|1 # # debug_renew = 0 # command line: -R 0|1 # # logfile_priority = 4 # command line: -L 4 # # logfile_use_utc = 0 # # syslog_priority = 3 # command line: -S 3 # # names_log_priority = 6 # # use_watchdog = 1 # command line: -w 1 # # high_priority = 1 # command line: -h 1 # # mlock_level = 1 # command line: -l 1 # # sh_retries = 8 # # uname = sanlock # command line: -U # # gname = sanlock # command line: -G # # our_host_name = # command line: -e # # renewal_read_extend_sec = # # paxos_debug_all = 0 # # debug_io = # # max_sectors_kb = # # debug_clients = 0 # # debug_cmd = + # debug_cmd = - # ... # # write_init_io_timeout = # # max_worker_threads = 8 # command line: -t 8 # # io_timeout = 10 # command line: -o # # watchdog_fire_timeout = 60 # # debug_hosts = 1 # # use_hugepages = all # # create_old_delta_disk_version = # create_old_paxos_disk_version = # # use_compare_and_write = # # caw_dpo = 1 # caw_fua = 1 sanlock/src/sanlock.h000066400000000000000000000135031516326262600151150ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. */ #ifndef __SANLOCK_H__ #define __SANLOCK_H__ /* an acquire or release call can specify this many explicit resources in a single call. */ #define SANLK_MAX_RESOURCES 8 /* max resource name length */ #define SANLK_NAME_LEN 48 /* max disk path length, includes terminating \0 byte, and escape chars, i.e. the strlen with esc chars inserted must still be less than 1024. */ #define SANLK_PATH_LEN 1024 /* max length of kill script path and args, includes terminate \0 byte */ #define SANLK_HELPER_PATH_LEN 128 #define SANLK_HELPER_ARGS_LEN 128 /* max disks in a single lease */ #define SANLK_MAX_DISKS 4 /* * max length of a sanlk_resource in string format * :::[::...]: * 48 SANLK_NAME_LEN * + 1 colon * + 48 SANLK_NAME_LEN * + 1 colon * + 4184 (4 MAX_DISKS * (1024 SANLK_PATH_LEN + 1 colon + 20 offset + 1 colon)) * + 20 lver * ------ * 4302 */ #define SANLK_MAX_RES_STR 4400 /* TODO: add more padding to sanlk_disk so we can extend sync_disk later without changing abi */ struct sanlk_disk { char path[SANLK_PATH_LEN]; /* must include terminating \0 */ uint64_t offset; uint32_t pad1; uint32_t pad2; }; /* * PERSISTENT: if the pid holding the resource lease exits, * the lease will not be released, but will be moved to the * orphans list. On disk and from the perspective of other * hosts, nothing changes when a lease is orphaned; it continues * to be held by the host. * * (If persistent shared locks are used on a resource, then * all the locks on that resource should be persistent.) * * A new process can acquire an orphan resource using * the ACQUIRE_ORPHAN flag. This implies that the lockspace * had continued running and the resource not released by the * host between the time the resource became an orphan and was * then transferred to a new process. * * Orphan impact on the lockspace: if the lockspace is stopping * because of rem, or lease failure, the ls config option * USED_BY_ORPHANS will block the release of the lockspace * (like the USED option), if orphans exist for the lockspace. * Without USED_BY_ORPHANS, the lockspace would exit and * leave the orphan resources unchanged (not released) on disk. * The unreleased orphan resources could be acquired by another * host if the lockspace lease is cleanly released. */ #define SANLK_RES_LVER 0x00000001 /* lver field is set */ #define SANLK_RES_NUM_HOSTS 0x00000002 /* data32 field is new num_hosts */ #define SANLK_RES_SHARED 0x00000004 #define SANLK_RES_PERSISTENT 0x00000008 #define SANLK_RES_ALIGN1M 0x00000010 #define SANLK_RES_ALIGN2M 0x00000020 #define SANLK_RES_ALIGN4M 0x00000040 #define SANLK_RES_ALIGN8M 0x00000080 #define SANLK_RES_SECTOR512 0x00000100 #define SANLK_RES_SECTOR4K 0x00000200 #define SANLK_RES_USING_CAW 0x00000400 /* CAW leader flag is set */ struct sanlk_resource { char lockspace_name[SANLK_NAME_LEN] __attribute__ ((nonstring)); /* terminating \0 not required */ char name[SANLK_NAME_LEN] __attribute__ ((nonstring)); /* terminating \0 not required */ uint64_t lver; /* use with SANLK_RES_LVER */ uint64_t data64; /* per-resource command-specific data */ uint32_t data32; /* per-resource command-specific data */ uint32_t unused; uint32_t flags; /* SANLK_RES_ */ uint32_t num_disks; /* followed by num_disks sanlk_disk structs */ struct sanlk_disk disks[]; }; /* make these values match the RES equivalent in case of typos */ #define SANLK_RIF_ALIGN1M 0x00000010 #define SANLK_RIF_ALIGN2M 0x00000020 #define SANLK_RIF_ALIGN4M 0x00000040 #define SANLK_RIF_ALIGN8M 0x00000080 #define SANLK_RIF_SECTOR512 0x00000100 #define SANLK_RIF_SECTOR4K 0x00000200 struct sanlk_rindex { uint32_t flags; /* SANLK_RIF_ */ uint32_t max_resources; /* the max res structs that will follow rindex */ uint64_t unused; char lockspace_name[SANLK_NAME_LEN] __attribute__ ((nonstring)); /* terminating \0 not required */ struct sanlk_disk disk; /* location of rindex */ }; struct sanlk_rentry { char name[SANLK_NAME_LEN] __attribute__ ((nonstring)); /* terminating \0 not required */ uint64_t offset; uint32_t flags; uint32_t unused; }; /* command-specific command options (can include per resource data, but that requires the extra work of segmenting it by resource name) */ struct sanlk_options { char owner_name[SANLK_NAME_LEN] __attribute__ ((nonstring)); /* optional user friendly name */ uint32_t flags; uint32_t len; /* unused and ignored */ /* followed by len bytes (unused and ignored) */ char str[]; }; #define SANLK_LSF_ADD 0x00000001 #define SANLK_LSF_REM 0x00000002 #define SANLK_LSF_NO_TIMEOUT 0x00000004 #define SANLK_LSF_USING_CAW 0x00000008 /* CAW leader flag is set */ /* make these values match the RES equivalent in case of typos */ #define SANLK_LSF_ALIGN1M 0x00000010 #define SANLK_LSF_ALIGN2M 0x00000020 #define SANLK_LSF_ALIGN4M 0x00000040 #define SANLK_LSF_ALIGN8M 0x00000080 #define SANLK_LSF_SECTOR512 0x00000100 #define SANLK_LSF_SECTOR4K 0x00000200 struct sanlk_lockspace { char name[SANLK_NAME_LEN] __attribute__ ((nonstring)); uint64_t host_id; uint32_t flags; /* SANLK_LSF_ */ struct sanlk_disk host_id_disk; }; struct sanlk_host { uint64_t host_id; uint64_t generation; uint64_t timestamp; uint32_t io_timeout; uint32_t flags; }; struct sanlk_host_event { uint64_t host_id; uint64_t generation; uint64_t event; uint64_t data; }; size_t sanlock_path_export(char *dst, const char *src, size_t dstlen); size_t sanlock_path_import(char *dst, const char *src, size_t dstlen); const char *sanlock_strerror(int rv); #endif sanlock/src/sanlock.sysusers000066400000000000000000000001661516326262600165670ustar00rootroot00000000000000# See sysusers.d(5) man page # /usr/lib/sysusers.d/sanlock.conf u sanlock 179 sanlock /var/run/sanlock m sanlock disk sanlock/src/sanlock_admin.h000066400000000000000000000624401516326262600162710ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. */ #ifndef __SANLOCK_ADMIN_H__ #define __SANLOCK_ADMIN_H__ /* add flags */ #define SANLK_ADD_ASYNC 0x00000001 #define SANLK_ADD_NODELAY 0x00000002 /* can be unsafe */ /* rem flags */ #define SANLK_REM_ASYNC 0x00000001 #define SANLK_REM_UNUSED 0x00000002 /* inq flags */ #define SANLK_INQ_WAIT 0x00000001 /* sanlock_write_resource */ #define SANLK_WRITE_CLEAR 0x00000001 /* subsequent read will return error */ #define SANLK_WRITE_USE_CAW 0x00000002 /* use compare-and-write if supported */ #define SANLK_WRITE_NO_CAW 0x00000004 /* don't use compare-and-write */ /* sanlock_write_lockspace */ #define SANLK_WRITE_LS_FLAGS 0x00000001 /* update leader flags in lockspace delta leases */ #define SANLK_WRITE_LS_USE_CAW 0x00000002 /* prefer compare-and-write */ #define SANLK_WRITE_LS_NO_CAW 0x00000004 /* don't use compare-and-write */ /* sanlock_init_lockspace_host */ #define SANLK_INIT_HOST_USE_CAW 0x00000001 #define SANLK_INIT_HOST_NO_CAW 0x00000002 /* get_hosts flags */ #define SANLK_GET_HOST_LOCAL 0x00000001 /* host status returned in low byte of sanlk_host.flags by get */ #define SANLK_HOST_UNKNOWN 0x00000001 #define SANLK_HOST_FREE 0x00000002 #define SANLK_HOST_LIVE 0x00000003 #define SANLK_HOST_FAIL 0x00000004 #define SANLK_HOST_DEAD 0x00000005 #define SANLK_HOST_MASK 0x0000000F /* select SANLK_HOST_ from flags */ #define SANLK_HOST_DEAD_EXT 0x00000010 /* DEAD_EXT delta lease flag */ #define SANLK_HOST_DEAD_SEC 0x00000020 /* host_dead_seconds timeout */ #define SANLK_HOST_UNKNOWN_OLD 0x00000040 /* data is too old */ /* * CAW (COMPARE AND WRITE) * * USE_CAW/NO_CAW flags are API flags the caller passes * to request that sanlock enable, or disable, the use * of CAW for lockspace or resource leases. With neither * flag set, sanlock config settings and device support * determine if CAW is enabled for a lease. When enabled, * the lease leader record has the CAW flag set, which * indicates to all hosts that they must use CAW commands * to update the lease leader record. * * CAW is enabled for a lease when it is first initialized. * At that point, the USE_CAW/NO_CAW flags are evaluated in * combination with sanlock config settings controlling * the use of CAW, and the device's support for CAW. * * USING_CAW is a flag returned by sanlock in the * sanlk_lockspace/sanlk_resource structs which * indicate if the CAW flag is set in the on disk * lease. * * CAW is enabled in delta or paxos leases when lockspaces * and resources are initialized by * sanlock_write_resource()/sanlock_write_lockspace(): * * . USE_CAW: enable caw for the lease if supported. * It won't be used if sanlock config settings have * disabled caw, or if the device doesn't support it. * It will be used if sanlock config settings allow * it, and the device supports it. * . NO_CAW: disable the use of caw for the lease, so it * won't be used even if allowed by sanlock config * settings and supported by the device. * * sanlock_init_lockspace_host() reinitializes a delta * lease in an existing lockspace. USE_CAW should be * set if the lease previously used caw, and NO_CAW * should be set if the lease did not use caw. If * neither is set, caw will be enabled if sanlock * config settings allow it (device support is not * checked.) * * sanlock config settings which effect USE_CAW: * sanlock.conf use_compare_and_write = allow|yes|no * yes: use caw if supported by device, and api does not disable (opt-out) * allow: use caw if supported by device, and api enables (opt-in) * no: never use caw (ignore api enable) * sanlock.conf create_old_delta_disk_version = 3.4 or 4.0 * sanlock.conf create_old_paxos_disk_version = 6.4 * these versions prevent use of caw for leases, * even when USE_CAW is set, and are meant to preserve * compatiblity with hosts running versions of sanlock * that do not support caw. * * To determine if a lease is using caw, call * sanlock_read_lockspace/sanlock_read_resource, and * check the USING_CAW flag in the ls/res struct. * * sanlock_direct_write_lockspace()/sanlock_direct_write_resource() * caw flags work somewhat differently from the other APIs. * If SANLK_WRITE_LS_USE_CAW/SANLK_WRITE_USE_CAW are set, * then caw will be enabled if the device supports it. * These APIs do not consider the sanlock.conf settings, * and the NO_CAW flag does not apply. */ /* * add_lockspace returns: * 0: the lockspace has been added successfully * -EEXIST: the lockspace already exists * -EINPROGRESS: the lockspace is already in the process of being added * (the in-progress add may or may not succeed) * -EAGAIN: the lockspace is being removed * * The _timeout version takes the io_timeout in seconds. * If 0, the global setting for the daemon will be used. */ int sanlock_add_lockspace(struct sanlk_lockspace *ls, uint32_t flags); int sanlock_add_lockspace_timeout(struct sanlk_lockspace *ls, uint32_t flags, uint32_t io_timeout); /* * inq_lockspace returns: * 0: the lockspace exists and is currently held * -ENOENT: lockspace not found */ int sanlock_inq_lockspace(struct sanlk_lockspace *ls, uint32_t flags); /* * rem_lockspace returns: * 0: the lockspace has been removed successfully * -EINPROGRESS: the lockspace is already in the process of being removed * -ENOENT: lockspace not found * -EBUSY: UNUSED was set and lockspace is being used * * The sanlock daemon will kill any pids using the lockspace when the * lockspace is removed (unless UNUSED is set). */ int sanlock_rem_lockspace(struct sanlk_lockspace *ls, uint32_t flags); /* * get_lockspace returns: * 0: all lockspaces copied out, lss_count set to number * -ENOSPC: sanlock internal buffer ran out of space * (lss_count set to number that would have been copied) * * sanlk_lockspace.flags set to SANLK_LSF_ */ int sanlock_get_lockspaces(struct sanlk_lockspace **lss, int *lss_count, uint32_t flags); /* * When host_id is > 0, returns the sanlk_host info about the * specified host_id. * * When host_id is 0, returns sanlk_host info about all hosts * that have been seen alive. * * host status returned by sanlk_host.flags & SANLK_HOST_MASK: * * UNKNOWN: after adding lockspace, there has not yet been * enough time monitoring other hosts to make an accurate * assessment. * * FREE: delta lease not held * the delta lease timestamp is zero * * LIVE: the host is alive * now - last < other_host_fail_seconds * * FAIL: the host is failing and may be in recovery (killing pids) * now - last > other_host_fail_seconds * * DEAD: the host is dead, its watchdog has fired * now - last > other_host_dead_seconds * * now: local monotonic time * * last: if we have never seen the host's timestamp change, then * last is the local monotime when we first checked it, otherwise * last is the local monotime when we last saw the timestamp change * (which would be some time after it was written by the host.) * * other_host_fail_seconds: based on the host's io_timeout, * the number of seconds after which it would begin recovery * (killing pids) if still alive and unable to renew its lease. * * other_host_dead_seconds: based on the host's io_timeout, * the number of seconds after which its watchdog has fired. */ int sanlock_get_hosts(const char *ls_name, uint64_t host_id, struct sanlk_host **hss, int *hss_count, uint32_t flags); /* * SANLK_SET_HOST_DEAD_EXT * ----------------------- * Set the DEAD_EXT flag in a host's delta_lease. * *host_data is *sanlk_host, which specifies the host_id to change. * The lockspace arg specifies the lockspace to operate on (if host_id * is specified in the lockspace arg, it should match the local host_id.) * The delta_lease leader record will be read, modified, * and written back with the DEAD_EXT flag set. * If a generation number is set in sanlk_host, that generation * number will be set in the leader record that is written, * regardless of the generation number found in the leader * record when read. */ #define SANLK_SET_HOST_DEAD_EXT 0x00000001 int sanlock_set_host(struct sanlk_lockspace *ls, uint32_t cmd, uint32_t flags, uint32_t io_timeout, void *host_data); /* * set_config cmd values * * USED: the lockspace is being used by an external entity. * This has the same effect on lockspace recovery as resources * that are not released by a pid that cannot be killed. * If the lockspace fails (the delta lease cannot be renewed), * the lockspace will not exit while this flag is set, which * will lead to the watchdog firing. * * UNUSED: clears the USED flag in the lockspace. * * USED_BY_ORPHANS: the lockspace should be considered USED * if there are orphan resources for the lockspace. * * UNUSED_BY_ORPHANS: clear the USED_BY_ORPHAN flag in the lockspace. */ #define SANLK_CONFIG_USED 0x00000001 #define SANLK_CONFIG_UNUSED 0x00000002 #define SANLK_CONFIG_USED_BY_ORPHANS 0x00000004 #define SANLK_CONFIG_UNUSED_BY_ORPHANS 0x00000008 #define SANLK_CONFIG_IO_TIMEOUT 0x00000010 int sanlock_set_config(const char *ls_name, uint32_t flags, uint32_t cmd, void *data); /* * Returns the alignment in bytes required by sanlock_init() * (1MB for disks with 512 sectors, 8MB for disks with 4096 sectors) */ int sanlock_align(struct sanlk_disk *disk); /* * Ask sanlock daemon to initialize disk space. * Use max_hosts = 0 for default value. * Use num_hosts = 0 for default value. * Provide either lockspace or resource, not both * * (Old api, see write_lockspace/resource) */ int sanlock_init(struct sanlk_lockspace *ls, struct sanlk_resource *res, int max_hosts, int num_hosts); /* * Alignment and sector size * * The ALIGN and SECTOR flags can be set in sanlk_lockspace | sanlk_resource * and passed to sanlock_write_lockspace() | sanlock_write_resource(). * These flags cause sanlock to create a lockspace|resource area with the * given align_size, using the given sector_size. The maximum hosts * that can use a lockspace|resource is determined by the combined effect * of ALIGN and SECTOR flags. The following combinations are allowed: * * ALIGN1M | SECTOR512: max_hosts 2000 * ALIGN1M | SECTOR4K: max_hosts 250 * ALIGN2M | SECTOR4K: max_hosts 500 * ALIGN4M | SECTOR4K: max_hosts 1000 * ALIGN8M | SECTOR4K: max_hosts 2000 * * ALIGN and SECTOR flags must both be set, or neither can be set. When * neither are set, sanlock will: * - detect the sector_size of the disk and use 1M align_size if 512, * and 8M align_size for 4K. * - use 512 sector_size and 8M align_size for files. * * sanlock_read_lockspace() | sanlock_read_resource() will return the * ALIGN and SECTOR flags reflecting the state of the lockspace|resource. * These flags are returned whether or not they were specified in * sanlock_write_lockspace() | sanlock_write_resource(). * (The ALIGN flag can be passed to sanlock_read_lockspace() to avoid * an extra read to discover the sector size.) * * Prior to the addition of ALIGN and SECTOR flags, sanlock will return * neither flag from read. The align_size of the lockspace | resource * can then be determined with sanlock_align(). After the addition of * these flags, sanlock_align() no longer correctly indicates the alignment * of the lockspace | resource. * * With the addition of ALIGN and SECTOR flags, sanlock_align() still * reports the *default* alignment that sanlock will use for disks or * files if ALIGN|SECTOR is not passed to write. * * (A lockspace and its associated resources will typically use the * same align and sector size, but it's conceivable they would not, e.g. * if the were placed on different storage with different sector sizes.) */ /* * write a lockspace to disk * * Set SANLK_LSF_ALIGN and SANLK_LSF_SECTOR in ls.flags to define * the sector size and align size of the lockspace on disk. max_hosts * is derived from these values (the max_hosts areg is not used.) * It is best for resources in the lockspace to use these same * sector/align sizes. * * the sanlock daemon writes max_hosts lockspace leader records to disk * * the lockspace will support up to max_hosts using the lockspace at once * * valid host_id's for this lockspace are 1 to max_hosts. * * the first host_id (1) (the first record at offset) is the last * leader record written, so read_lockspace of host_id 1 will fail * until the entire write_lockspace is complete. * * SANLK_WRITE_LS_FLAGS: * Update the leader flags of each delta lease in the lockspace. * LFL_NO_TIMEOUT is set if SANLK_LSF_NO_TIMEOUT is set in the * sanlk_lockspace arg, or cleared if not set. */ int sanlock_write_lockspace(struct sanlk_lockspace *ls, int max_hosts, uint32_t flags, uint32_t io_timeout); /* * initialize one host_id lease in the lockspace */ int sanlock_init_lockspace_host(struct sanlk_lockspace *ls, char *our_host_name, uint64_t generation, uint64_t timestamp, uint32_t flags, uint32_t io_timeout); /* * read one host's lockspace record from disk * * the sanlock daemon reads one lockspace leader record from disk * * the minimum input is path and offset * * if name is specified and does not match the leader record name, * SANLK_LEADER_LOCKSPACE is returned * * if name is not specified, it is filled it with the value from disk * * if host_id is zero, host_id 1 is used (the first record at offset) * * if there is no delta lease magic number found at the host_id location, * SANLK_LEADER_MAGIC is returned * * on success, zero is returned and * io_timeout and the entire sanlk_lockspace struct are written to * * For sanlock_read_lockspace_host(), the host fields are returned * except for flags. */ int sanlock_read_lockspace(struct sanlk_lockspace *ls, uint32_t flags, uint32_t *io_timeout); int sanlock_read_lockspace_host(struct sanlk_lockspace *ls, uint32_t flags, uint32_t *io_timeout, struct sanlk_host *hs); /* * format a resource lease area on disk * * the sanlock daemon writes a resource lease area to disk * * Set flag SANLK_WRITE_CLEAR to cause a subsequent read_resource * to return an error. * * Set SANLK_RES_ALIGN and SANLK_RES_SECTOR in res.flags to define * the sector size and align size of the resource on disk. max_hosts * is derived from these values (the max_hosts arg is not used.) * It is best for the ALIGN and SECTOR flags to match those used * for the resource's lockspace. */ int sanlock_write_resource(struct sanlk_resource *res, int max_hosts, int num_hosts, uint32_t flags); /* * read a resource lease from disk * * the sanlock daemon reads the lease's leader record from disk * * the minimum input is one disk with path and offset * * if lockspace name is specified and does not match the leader record * lockspace name, SANLK_LEADER_LOCKSPACE is returned * * if resource name is specified and does not match the leader record * resource name, SANLK_LEADER_RESOURCE is returned * * if there is no paxos lease magic number found in the leader record, * SANLK_LEADER_MAGIC is returned * * on success, zero is returned and * the entire sanlk_resource struct is written to (res->disks is not changed) */ int sanlock_read_resource(struct sanlk_resource *res, uint32_t flags); /* * read resource lease and its owners from disk * * the sanlock daemon reads the entire lease area from disk, * including the leader record and all per-host dblock/mode_block records * * res.lver is set (from leader record) * res.flags is set to SANLK_RES_SHARED if any shared owners exist (from mode blocks) * host.host_id and host.generation are set for each owner (from leader or mode blocks) * host.timestamp is set for an exclusive owner (from leader record) */ int sanlock_read_resource_owners(struct sanlk_resource *res, uint32_t flags, struct sanlk_host **hss, int *hss_count); /* * Check the condition of a resource based on the state of the * resource's owners. This can be used to check if a resource * is held by hosts that would cause an acquire to fail. * * owners is the list of hosts returned by sanlock_read_resource_owners() * hosts is the list of hosts returned by sanlock_get_hosts() * * (This is a client side operation as does not go to the daemon.) * * For each owner, check its state in hosts: * * - if not found in hosts, then the owner is not running and * would not prevent acquire * * - if found in hosts but the generation does not match, * then the owner host has been restarted since owning the * resource and would not prevent acquire * * - if found in hosts with matching generation, then check * host.flags & MASK: * * - FREE: would not prevent acquire * - DEAD: would not prevent acquire * - LIVE: prevents acquire, test fails * - FAIL: prevents acquire, test fails * - UNKNOWN: might prevent acquire, test fails * * * test_flags returned: * SANLK_TRF_FAIL: state of owners would prevent acquire, test fails */ #define SANLK_TRF_FAIL 0x00000001 int sanlock_test_resource_owners(struct sanlk_resource *res, uint32_t flags, struct sanlk_host *owners, int owners_count, struct sanlk_host *hosts, int hosts_count, uint32_t *test_flags); /* * A resource index stores the disk locations (offsets) of resource leases. * Using it is optional; an application can keep track of lease offsets * without using the index. * * On disk, a resource index uses two alignment-sized regions. * The first holds the records mapping resource names to offsets. * The second holds a paxos lease that sanlock uses internally * to protect updates to the index. The caller chooses the disk * location of the resource index (path and offset), and passes * this as a parameter to all functions that use the index with * struct sanlk_rindex. * * The resource index is followed on disk by the resource leases * that it references. So, using the index removes the ability of * the application to place resource leases at any disk location. * A caller would usually place the resource index after a lockspace * struct on disk (not required.) * * The resource index and the following resource leases must all use * the same align size/flag. * * The rindex specifies the lockspace name that the referenced resource * leases are associated with. This lockspace will also be used for * the internal rindex paxos lease. * sanlock must be a member of the lockspace to use the create/delete * resource functions. * * format * ------ * Initializes resource index at the specified offset and * initializes an internal paxos lease in the following area. * Set the ALIGN flag in sanlk_rindex corresponding to the desired * sector size; the align size used for the rindex must match the * align size used for resources. * * lookup * ------ * Looks up a value in the resource index. When a res name is set * and *offset is 0, this searches for an entry with the matching * name and if found sets the res lease offset. When res name is not * set and an *offset is not 0, this checks for an entry with the given * res lease offset and if found sets the res name. When name and * offset or both unset, the first free entry is returned in offset. * All resource lease offsets are relative to the start of the device. * sanlock does not acquire the internal rindex paxos lease. * (The offsets are the disk locations of the resource leases, not * the disk locations of the rindex entries for the resource leases.) * * update * ------ * Add or remove an rindex entry. When adding, the rentry * name and offset must both be set, and the index entry is * set to indicate the named resource lease exists at the * specified offset. WHen removing, the rentry offset needs * to be set, and the index entry for that offset is cleared. * This is not generally used; the create/delete interfaces are * the standard method for updating the index. * * create_resource * --------------- * Searches the index for a free resource lease area, initializes a new * resource lease at that offset, and updates the index for * the new lease. Returns the offset of the new resource lease. * sanlock holds the internal rindex paxos lease around the index * lookup, resource init and index update. The new lease is initialized * before the index is updated, so the index will not reference * an uninitialized area if the host fails during create_resource. * * delete_resource * --------------- * Updates the index to remove the entry for the named resource lease, * and clears the resource lease at that offset. * sanlock holds the internal rindex paxos lease around the * index update and lease reinitialization. If sanlock fails * after the index update but before clearing the resource, a * subsequent create will overwrite the uncleared resource. * * rebuild * ------- * Rebuilds the rindex based on resource leases that are found. * Reads each potential resource lease area to check if a * resource lease exists at that offset. If so, an rindex * entry is added with that resource name and offset. */ /* * generic rindex flags use lower 16 bits * specific rindex function function use upper 16 bits */ #define SANLK_RX_NO_LOCKSPACE 0x000000001 /* don't use the lockspace */ /* update_rindex flags */ #define SANLK_RXUP_ADD 0x00010000 #define SANLK_RXUP_REM 0x00020000 int sanlock_format_rindex(struct sanlk_rindex *rx, uint32_t flags); int sanlock_update_rindex(struct sanlk_rindex *rx, uint32_t flags, struct sanlk_rentry *re); int sanlock_lookup_rindex(struct sanlk_rindex *rx, uint32_t flags, struct sanlk_rentry *re); int sanlock_rebuild_rindex(struct sanlk_rindex *rx, uint32_t flags); int sanlock_create_resource(struct sanlk_rindex *rx, uint32_t flags, struct sanlk_rentry *re, int max_hosts, int num_hosts); int sanlock_delete_resource(struct sanlk_rindex *rx, uint32_t flags, struct sanlk_rentry *re); int sanlock_version(uint32_t flags, uint32_t *version, uint32_t *proto); /* * Lockspace host events * * reg: register with the sanlock daemon, returns a fd to use in poll(2). * end: unregister and close our fd in the sanlock daemon. * set: set/write an event for another host, in the next ls lease renewal. * get: get/read an event from another host from the registered fd. * * reg_event * . he arg is unused, can be NULL * . returns -ENOCSI if no more event fds for the ls are available * (MAX_EVENT_FDS 32) * * set_event * . CUR_GENERATION with zero generation in he means that sanlock * will fill in the he generation with the current generation. * . CLEAR_HOSTID will cause sanlock to clear the host_id in its * bitmap in the next renewal, even if the default time for clearing * it has not been reached. generation/event/data are ignored. * . CLEAR_EVENT will cause sanlock to zero the generation/event/data values * in the next renewal. host_id is ignored. * . REPLACE_EVENT will cause sanlock to replace the existing event/data * values when they would otherwise be rejected with -EBUSY due to a * previous set_event. * . ALL_HOSTS causes the bits for all host_ids to be set. * * Multiple set_event calls * . set_event replaces the last event/data values * . set_event replaces the last generation value * . set_event adds the host_id to the notification bitmap, * leaving any host_id bits that are already set. * * This allows the same event/data values to be passed to multiple * host_ids at once, but without using host_id generations; * generation should be set to 0 in the set_event calls. * * To send the same event/data values (A,B) to hosts 1,2,3: * T=10 set_event(1, A, B); * T=10 set_event(2, A, B); * T=10 set_event(3, A, B); * * The A,B values from each call replace those from the previous call, * but with no effect because they are the same. Bits for 1,2,3 will * all be set in the notification bitmap. * * To send different event/data values to different hosts, wait for * set_bitmap_seconds between the two set_event calls: * T=10 set_event(1, A, B); * T=70 set_event(2, C, D); * * The bit for 1 will be cleared from the bitmap by the time that the * bit for 2 is set. C,D replace A,B, but host 1 will have seen A,B * already, or won't be looking for it any longer. * * Sequential set_events with different event/data values, within a short * time span is likely to produce unwanted results, because the new * event/data values replace the previous values before the previous values * have been read: * T=10 set_event(1, A, B); * T=11 set_event(2, C, D); * * In this case, A,B are replaced by C,D, and both hosts 1 and 2 will be * notified of an event. host 1 will see values C,D, and will not get A,B. * * Unless the REPLACE_EVENT flag is used, sanlock will return -EBUSY from * set_event in this case. */ #define SANLK_SETEV_CUR_GENERATION 0x00000001 #define SANLK_SETEV_CLEAR_HOSTID 0x00000002 #define SANLK_SETEV_CLEAR_EVENT 0x00000004 #define SANLK_SETEV_REPLACE_EVENT 0x00000008 #define SANLK_SETEV_ALL_HOSTS 0x00000010 int sanlock_reg_event(const char *ls_name, struct sanlk_host_event *he, uint32_t flags); int sanlock_end_event(int fd, const char *ls_name, uint32_t flags); int sanlock_set_event(const char *ls_name, struct sanlk_host_event *he, uint32_t flags); int sanlock_get_event(int fd, uint32_t flags, struct sanlk_host_event *he, uint64_t *from_host_id, uint64_t *from_generation); #endif sanlock/src/sanlock_direct.h000066400000000000000000000023711516326262600164500ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __SANLOCK_DIRECT_H__ #define __SANLOCK_DIRECT_H__ /* * Use num_hosts = 0 for default value. * Provide either lockspace or resource, not both * * (Old api, see write_lockspace/resource) */ int sanlock_direct_init(struct sanlk_lockspace *ls, struct sanlk_resource *res, int max_hosts_unused, int num_hosts, int use_aio); /* * write a lockspace to disk * (also see sanlock_write_lockspace) */ int sanlock_direct_write_lockspace(struct sanlk_lockspace *ls, int max_hosts_unused, uint32_t flags, uint32_t io_timeout); /* * format a resource lease area on disk * (also see sanlock_write_resource) */ int sanlock_direct_write_resource(struct sanlk_resource *res, int max_hosts_unused, int num_hosts, uint32_t flags); /* * Returns the alignment in bytes required by sanlock_direct_init() * (1MB for disks with 512 sectors, 8MB for disks with 4096 sectors) */ int sanlock_direct_align(struct sanlk_disk *disk); #endif sanlock/src/sanlock_internal.c000066400000000000000000000011671516326262600170070ustar00rootroot00000000000000#include #include #include #include "sanlock_internal.h" #include "log.h" /* Moved from resource.c to prevent circular dependency */ void free_token(struct token *token) { if (token->caw_dev_fd) log_debug("free_token leak caw_dev_fd %d", token->caw_dev_fd); if (token->caw_data) free(token->caw_data); free(token); } /* Moved from lockspace.c to prevent circular dependency */ void set_id_bit(int host_id, char *bitmap, char *c) { char *byte = bitmap + ((host_id - 1) / 8); unsigned int bit = (host_id - 1) % 8; char mask; mask = 1 << bit; *byte |= mask; if (c) *c = *byte; } sanlock/src/sanlock_internal.h000066400000000000000000000366031516326262600170170ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __SANLOCK_INTERNAL_H__ #define __SANLOCK_INTERNAL_H__ #ifndef GNUC_UNUSED #define GNUC_UNUSED __attribute__((__unused__)) #endif #define BUILD_BUG_ON(condition) ((void)sizeof(char[1 - 2*!!(condition)])) #ifndef EXTERN #define EXTERN extern #else #undef EXTERN #define EXTERN #endif #include "sanlock.h" #include "sanlock_rv.h" #include "sanlock_resource.h" #include "leader.h" #include "paxos_dblock.h" #include "mode_block.h" #include "rindex_disk.h" #include "list.h" #include "monotime.h" #include "sizeflags.h" #include #define ONE_MB_IN_BYTES 1048576 #define TWO_MB_IN_BYTES 2097152 #define ONE_MB_IN_KB 1024 /* default max number of hosts supported */ #define DEFAULT_MAX_HOSTS 2000 #define LOG_DUMP_SIZE (1024*1024) /* this is just the path to the executable, not full command line */ #define COMMAND_MAX 4096 #define SANLOCK_RUN_DIR "SANLOCK_RUN_DIR" #define DEFAULT_RUN_DIR "/run/sanlock" #define SANLOCK_PRIVILEGED "SANLOCK_PRIVILEGED" #define SANLK_LOG_DIR "/var/log" #define SANLK_LOGFILE_NAME "sanlock.log" #define SANLK_LOCKFILE_NAME "sanlock.pid" #define SANLK_CONF_PATH "/etc/sanlock/sanlock.conf" #define DAEMON_NAME "sanlock" /* for paxos_lease sync_disk + offset: points to 1 leader_record + 1 request_record + MAX_HOSTS paxos_dblock's = 256 blocks = 128KB, ref: lease_item_record */ /* must mirror external sanlk_disk */ struct sync_disk { char path[SANLK_PATH_LEN]; uint64_t offset; uint32_t sector_size; /* sanlk_disk pad1 */ int fd; /* sanlk_disk pad2 */ }; struct delta_extra { uint64_t field1; uint64_t field2; uint64_t field3; }; /* * There are two different wrappers around a sanlk_resource: * 'struct token' keeps track of resources per-client, client.tokens[] * 'struct resource' keeps track of resources globally, resources list */ #define T_RESTRICT_SIGKILL 0x00000001 /* inherited from client->restricted */ #define T_RESTRICT_SIGTERM 0x00000002 /* inherited from client->restricted */ #define T_RETRACT_PAXOS 0x00000004 #define T_WRITE_DBLOCK_MBLOCK_SH 0x00000008 /* make paxos layer include mb SHARED with dblock */ #define T_CHECK_EXISTS 0x00000010 /* make paxos layer not error if reading lease finds none */ #define T_CAW 0x00000020 #define T_CLEANUP_CONVERT 0x00000040 #define T_CLEANUP_OWNER 0x00000080 struct token { /* values copied from acquire res arg */ uint64_t acquire_lver; uint64_t acquire_data64; uint32_t acquire_data32; uint32_t acquire_flags; /* copied from the sp with r.lockspace_name */ uint64_t host_id; uint64_t host_generation; uint32_t space_id; uint32_t io_timeout; /* internal */ struct list_head list; /* resource->tokens */ struct resource *resource; int pid; int client_id; uint32_t flags; /* be careful to avoid using this from different threads */ uint32_t token_id; uint32_t res_id; int sector_size; int align_size; int space_dead; /* copied from sp->space_dead, set by main thread */ int shared_count; /* set during ballot by paxos_lease_acquire */ char shared_bitmap[HOSTID_BITMAP_SIZE]; /* bit set for host_id with SH */ int caw_dev_fd; char *caw_data; struct sync_disk *disks; /* shorthand, points to r.disks[0] */ struct sanlk_resource r; /* * sanlk_resource must be the last element of token. * sanlk_resource ends with sanlk_disk disks[0], * and allocating a token allocates N sanlk_disk structs * after the token struct so they follow the sanlk_resource. */ }; #define R_SHARED 0x00000001 #define R_THREAD_EXAMINE 0x00000002 #define R_THREAD_RELEASE 0x00000004 #define R_RESTRICT_SIGKILL 0x00000008 /* inherited from token */ #define R_RESTRICT_SIGTERM 0x00000010 /* inherited from token */ #define R_LVB_WRITE_RELEASE 0x00000020 #define R_UNDO_SHARED 0x00000040 #define R_ERASE_ALL 0x00000080 #define R_CAW 0x00000100 #define R_CLEANUP_CONVERT 0x00000200 #define CAW_STATE_UNKNOWN 0 #define CAW_STATE_USE_DISK 1 #define CAW_STATE_FOUND_DEV 2 struct resource { struct list_head list; struct list_head tokens; /* only one token when ex, multiple sh */ uint64_t host_id; uint64_t host_generation; uint32_t io_timeout; int pid; /* copied from token when ex */ int sector_size; int align_size; uint32_t res_id; uint32_t reused; uint32_t flags; uint64_t thread_release_retry; uint32_t disk_major; /* major:minor of r.disks[0].path */ uint32_t disk_minor; uint32_t caw_state; uint32_t caw_major; uint32_t caw_minor; uint64_t caw_offset; char *lvb; char *thread_release_caw_data; char killpath[SANLK_HELPER_PATH_LEN]; /* copied from client */ char killargs[SANLK_HELPER_ARGS_LEN]; /* copied from client */ struct leader_record leader; /* copy of last leader_record we wrote */ struct paxos_dblock dblock; /* copy of last paxos_dblock we wrote */ struct sanlk_resource r; }; struct lease_status { int corrupt_result; int acquire_last_result; int renewal_last_result; uint64_t acquire_last_attempt; uint64_t acquire_last_success; uint64_t renewal_last_attempt; uint64_t renewal_last_success; uint32_t renewal_read_count; uint32_t renewal_read_check; char *renewal_read_buf; }; struct host_status { uint64_t first_check; /* local monotime */ uint64_t last_check; /* local monotime */ uint64_t last_live; /* local monotime */ uint64_t last_req; /* local monotime */ uint64_t owner_id; uint64_t owner_generation; uint64_t timestamp; /* remote monotime */ uint64_t set_bit_time; uint64_t found_dead_ext_gen; uint64_t wrote_dead_ext_gen; uint16_t io_timeout; uint16_t lease_bad; uint8_t dead_ext; /* LFL_DEAD_EXT */ uint8_t no_timeout; /* LFL_NO_TIMEOUT */ uint8_t warned_fail; uint8_t warned_dead; uint8_t warned_dead_ext; char owner_name[NAME_ID_SIZE] __attribute__ ((nonstring)); }; struct renewal_history { uint64_t timestamp; int read_ms; int write_ms; int next_timeouts; int next_errors; }; /* The max number of connections that can get events for a lockspace. */ #define MAX_EVENT_FDS 32 #define SP_EXTERNAL_USED 0x00000001 #define SP_USED_BY_ORPHANS 0x00000002 struct space { struct list_head list; char space_name[NAME_ID_SIZE] __attribute__ ((nonstring)); uint32_t version; uint32_t space_id; /* used to refer to this space instance in log messages */ uint64_t host_id; uint64_t host_generation; struct sync_disk host_id_disk; uint32_t disk_major; uint32_t disk_minor; uint32_t io_timeout; uint32_t set_bitmap_seconds; uint32_t flags; /* SP_ */ uint32_t used_retries; uint32_t renewal_read_extend_sec; /* defaults to io_timeout */ uint32_t rindex_op; unsigned int set_max_sectors_kb; int nodelay; int no_timeout; int sector_size; int align_size; int max_hosts; int renew_fail; int space_dead; int killing_pids; int external_remove; int thread_stop; int wd_fd; int event_fds[MAX_EVENT_FDS]; struct sanlk_host_event host_event; uint64_t set_event_time; pthread_t thread; pthread_mutex_t mutex; /* protects lease_status, thread_stop */ struct lease_status lease_status; struct host_status host_status[DEFAULT_MAX_HOSTS]; struct renewal_history *renewal_history; int renewal_history_size; int renewal_history_next; int renewal_history_prev; int caw; /* compare-and-write flag */ int caw_dev_use; /* submit caw ioctl to a dev beneath host_id_disk */ int caw_dev_fd; uint32_t caw_dev_major; uint32_t caw_dev_minor; uint64_t caw_dev_offset; char *caw_data; /* 2*sector_size buffer */ }; /* Update lockspace_info() to copy any fields from struct space to space_info */ struct space_info { uint32_t space_id; uint32_t io_timeout; uint64_t host_id; uint64_t host_generation; int sector_size; int align_size; int killing_pids; }; #define RX_OP_FORMAT 1 #define RX_OP_CREATE 2 #define RX_OP_DELETE 3 #define RX_OP_LOOKUP 4 #define RX_OP_UPDATE 5 #define RX_OP_REBUILD 6 #define USE_AIO 1 #define NO_AIO 0 #define HOSTID_AIO_CB_SIZE 4 #define WORKER_AIO_CB_SIZE 2 #define DIRECT_AIO_CB_SIZE 1 #define RESOURCE_AIO_CB_SIZE 2 #define LIB_AIO_CB_SIZE 1 struct aicb { int used; char *buf; struct iocb iocb; }; struct task { char name[NAME_ID_SIZE+1]; /* for log messages */ unsigned int io_count; /* stats */ unsigned int to_count; /* stats */ unsigned int miscompares; /* stats */ unsigned int paxos_areabuf_used; /* stats */ unsigned int paxos_areabuf_created; /* stats */ int use_aio; int cb_size; int paxos_areabuf_size; char *paxos_areabuf; /* paxos_lease_acquire reads of lease area */ char *iobuf; /* delta_lease_renew reads of lease area */ io_context_t aio_ctx; struct aicb *read_iobuf_timeout_aicb; struct aicb *callbacks; }; EXTERN struct task main_task; struct worker_info { unsigned int busy; unsigned int cmd_last; unsigned int thread_id; unsigned int work_count; unsigned int io_count; unsigned int to_count; unsigned int miscompares; unsigned int paxos_areabuf_used; unsigned int paxos_areabuf_created; unsigned int paxos_areabuf_size; }; /* TODO: change used, suspend, need_free, pid_dead to flags */ #define CL_KILLPATH_PID 0x00000001 /* include pid as killpath arg */ #define CL_RUNPATH_SENT 0x00000002 /* a RUNPATH msg has been sent to helper */ struct client { int used; int fd; /* unset is -1 */ int pid; /* unset is -1 */ int cmd_active; int cmd_last; int pid_dead; int suspend; int need_free; int kill_count; int tokens_slots; uint32_t flags; uint32_t restricted; uint64_t kill_last; char owner_name[SANLK_NAME_LEN+1]; char killpath[SANLK_HELPER_PATH_LEN]; char killargs[SANLK_HELPER_ARGS_LEN]; pthread_mutex_t mutex; void *workfn; void *deadfn; struct token **tokens; }; /* * client array is only touched by main_loop, there is no lock for it. * individual cl structs are accessed by worker threads using cl->mutex */ EXTERN struct client *client; /* * yes: use caw if supported by device, and api does not disable (opt-out) * no: never use caw (ignore api enable) * allow: use caw if supported by device, and api enables (opt-in) */ #define USE_CAW_YES 1 #define USE_CAW_NO 2 #define USE_CAW_ALLOW 3 #define DEFAULT_WATCHDOG_FIRE_TIMEOUT 60 #define DEFAULT_USE_AIO 1 #define DEFAULT_IO_TIMEOUT 10 #define DEFAULT_GRACE_SEC 40 #define DEFAULT_USE_WATCHDOG 1 #define DEFAULT_HIGH_PRIORITY 0 #define DEFAULT_MLOCK_LEVEL 1 /* 1=CURRENT, 2=CURRENT|FUTURE */ #define DEFAULT_SOCKET_UID 0 #define DEFAULT_SOCKET_GID 0 #define DEFAULT_SOCKET_MODE (S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP) #define DEFAULT_MIN_WORKER_THREADS 2 #define DEFAULT_MAX_WORKER_THREADS 8 #define DEFAULT_SH_RETRIES 8 #define DEFAULT_QUIET_FAIL 1 #define DEFAULT_RENEWAL_HISTORY_SIZE 180 /* about 1 hour with 20 sec renewal interval */ #define DEFAULT_WRITE_INIT_IO_TIMEOUT 60 #define DEFAULT_USE_COMPARE_AND_WRITE USE_CAW_YES #define DEFAULT_MAX_SECTORS_KB_IGNORE 0 /* don't change it */ #define DEFAULT_MAX_SECTORS_KB_ALIGN 0 /* set it to align size */ #define DEFAULT_MAX_SECTORS_KB_NUM 1024 /* set it to num KB for all lockspaces */ struct spawn_cmd { int argc; char **argv; /* argv[argc] is NULL */ }; struct command_line { int type; /* COM_ */ int action; /* ACT_ */ int debug; int debug_renew; int debug_hosts; int debug_clients; int debug_io_submit; int debug_io_complete; int paxos_debug_all; uint64_t debug_cmds; int max_sectors_kb_ignore; int max_sectors_kb_align; int max_sectors_kb_num; int quiet_fail; int wait; int use_watchdog; int watchdog_fire_timeout; int io_timeout; /* DEFAULT_IO_TIMEOUT or sanlock.conf io_timeout */ int io_timeout_set; int kill_grace_seconds; /* -g */ int kill_grace_set; int high_priority; /* -h */ int get_hosts; /* -h */ int names_log_priority; int mlock_level; int max_worker_threads; int aio_arg; int write_init_io_timeout; int no_timeout; int set_bitmap_seconds; int persistent; int orphan_set; int orphan; int used_set; int used; int all; int clear_arg; int sector_size; int align_size; int use_hugepages; char *uname; /* -U */ int uid; /* -U */ char *gname; /* -G */ int gid; /* -G */ int pid; /* -p */ int cid; /* -C */ char sort_arg; uint64_t host_id; /* -i */ uint64_t host_generation; /* -g */ uint64_t he_event; /* -e */ uint64_t he_data; /* -d */ uint64_t timestamp; /* -t */ int num_hosts; /* -n */ int max_hosts; /* -m */ int res_count; int sh_retries; int create_delta_34; int create_delta_40; int create_paxos_64; int use_compare_and_write; int compare_and_write_opt1; /* 1 if -C 1 */ int compare_and_write_opt0; /* 1 if -C 0 */ int delta_caw_verify; int paxos_caw_verify; int caw_dpo; int caw_fua; uint32_t force_mode; int renewal_history_size; int renewal_read_extend_sec_set; /* 1 if renewal_read_extend_sec is configured */ uint32_t renewal_read_extend_sec; char our_host_name_opt[256]; /* max SANLK_NAME_LEN will be used */ char *file_path; char *dump_path; char flag_name[SANLK_NAME_LEN+1]; int rindex_op; struct sanlk_rentry rentry; /* -e */ struct sanlk_rindex rindex; /* -x RINDEX */ struct sanlk_lockspace lockspace; /* -s LOCKSPACE */ struct sanlk_resource *res_args[SANLK_MAX_RESOURCES]; /* -r RESOURCE */ struct spawn_cmd *spawn_args; /* -c COUNT CMD [ARG...], for ACT_SPAWN */ int spawn_count; int no_disk; }; EXTERN struct command_line com; uint32_t cmd_str_to_num(const char *str); const char *cmd_num_to_str(int cmd); uint64_t cmd_num_to_debug_flag(uint32_t cmd); int is_cmd_debug(uint32_t cmd); void set_cmd_debug(uint32_t cmd); void clear_cmd_debug(uint32_t cmd); /* main.c */ const char *host_state_str(uint32_t flags); /* sanlock_internal.c */ void free_token(struct token *token); void set_id_bit(int host_id, char *bitmap, char *c); /* command line types and actions */ #define COM_DAEMON 1 #define COM_CLIENT 2 #define COM_DIRECT 3 enum { ACT_STATUS = 1, ACT_HOST_STATUS, ACT_LOG_DUMP, ACT_SHUTDOWN, ACT_ADD_LOCKSPACE, ACT_INQ_LOCKSPACE, ACT_REM_LOCKSPACE, ACT_COMMAND, ACT_ACQUIRE, ACT_RELEASE, ACT_INQUIRE, ACT_CONVERT, ACT_REQUEST, ACT_ACQUIRE_ID, ACT_RELEASE_ID, ACT_RENEW_ID, ACT_DIRECT_INIT, ACT_DUMP, ACT_NEXT_FREE, ACT_READ_LEADER, ACT_CLIENT_INIT, ACT_CLIENT_READ, ACT_CLIENT_ALIGN, ACT_EXAMINE, ACT_GETS, ACT_VERSION, ACT_SET_EVENT, ACT_SET_CONFIG, ACT_WRITE_LEADER, ACT_RENEWAL, ACT_FORMAT, ACT_CREATE, ACT_DELETE, ACT_LOOKUP, ACT_UPDATE, ACT_REBUILD, ACT_DIRECT_INIT_HOST, ACT_CLIENT_INIT_HOST, ACT_READ, ACT_SET_HOST, ACT_SPAWN, }; EXTERN int external_shutdown; EXTERN int our_host_name_matches_product_uuid; EXTERN char our_host_name_global[SANLK_NAME_LEN+1]; EXTERN char our_product_uuid[SANLK_NAME_LEN+1]; EXTERN char our_product_uuid_compact[SANLK_NAME_LEN+1]; /* dash chars omitted from uuid */ EXTERN int daemon_status_num_workers; /* replicate thread pool counter for status reporting */ EXTERN int kill_count_max; EXTERN int is_helper; EXTERN int helper_ci; EXTERN int helper_pid; EXTERN int helper_kill_fd; EXTERN int helper_status_fd; EXTERN uint64_t helper_last_status; EXTERN uint32_t helper_full_count; EXTERN int efd; EXTERN struct list_head spaces; EXTERN struct list_head spaces_rem; EXTERN struct list_head spaces_add; EXTERN pthread_mutex_t spaces_mutex; /* major.minor.patch-build (TODO: get build) */ EXTERN uint8_t sanlock_version_major; EXTERN uint8_t sanlock_version_minor; EXTERN uint8_t sanlock_version_patch; EXTERN uint8_t sanlock_version_build; EXTERN uint32_t sanlock_version_combined; #endif sanlock/src/sanlock_resource.h000066400000000000000000000150561516326262600170310ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. */ #ifndef __SANLOCK_RESOURCE_H__ #define __SANLOCK_RESOURCE_H__ /* * sock > -1, pid is ignored: * process creates registered connection and acquires/releases leases on * that connection for itself * * A threaded sanlock client must serialize libsanlock calls that are * made using a registered socket connection. * * sock == -1, pid is used: * process asks daemon to acquire/release leases for another separately * registered pid */ /* restrict flags */ #define SANLK_RESTRICT_ALL 0x00000001 #define SANLK_RESTRICT_SIGKILL 0x00000002 #define SANLK_RESTRICT_SIGTERM 0x00000004 /* killpath flags */ #define SANLK_KILLPATH_PID 0x00000001 #define SANLK_FOR_CLIENT_ID 0x00000010 /* * acquire flags * * SANLK_ACQUIRE_LVB * Enable the use of an LVB with the lock. * * SANLK_ACQUIRE_ORPHAN * If the lock already exists as an orphan, * then acquire it. Otherwise, acquire a * new lock as usual. * * SANLK_ACQUIRE_ORPHAN_ONLY * If the lock already exists as an orphan, * then acquire it. Otherwise, do not acquire * a lock at all and return -ENOENT. * * SANLK_ACQUIRE_OWNER_NOWAIT * If the lock cannot be granted immediately * because the owner's lease needs to time out, do * not wait, but return -SANLK_ACQUIRE_OWNED_RETRY. */ #define SANLK_ACQUIRE_LVB 0x00000001 #define SANLK_ACQUIRE_ORPHAN 0x00000002 #define SANLK_ACQUIRE_ORPHAN_ONLY 0x00000004 #define SANLK_ACQUIRE_OWNER_NOWAIT 0x00000008 /* SANLK_FOR_CLIENT_ID 0x00000010 defined above */ /* * release flags * * SANLK_REL_ALL * Release all resources held by the client. * The res args are ignored. * * SANLK_REL_RENAME * Rename the resource lease on disk when it * is released. The resource is freed and * renamed in a single disk operation (write * to the leader record.) The first res * arg is the resource to release, and the * second resource arg contains the new name * for the first resource. * * SANLK_REL_ORPHAN * Release orphan resources asynchronously. * Takes a single resource struct. If the * resource name is empty, then all orphans * for the specified lockspace are released. * If the resource name is set, then an * orphan with the matching resource name is * released. * * SANLK_REL_NO_DISK * Skip the on-disk portion of release. * * Introduced to be able to remove the * resource file or disk itself while holding * the lease, to avoid ENOENT on release. */ #define SANLK_REL_ALL 0x00000001 #define SANLK_REL_RENAME 0x00000002 #define SANLK_REL_ORPHAN 0x00000004 #define SANLK_REL_NO_DISK 0x00000008 /* SANLK_FOR_CLIENT_ID 0x00000010 defined above */ /* * convert flags * * SANLK_CONVERT_OWNER_NOWAIT * Same as SANLK_ACQUIRE_OWNER_NOWAIT. */ #define SANLK_CONVERT_OWNER_NOWAIT 0x00000008 /* NB: value must match SANLK_ACQUIRE_OWNER_NOWAIT */ /* SANLK_FOR_CLIENT_ID 0x00000010 defined above */ /* * inquire flags */ /* SANLK_FOR_CLIENT_ID 0x00000010 defined above */ /* * request flags * * SANLK_REQUEST_NEXT_LVER * The caller specifies 0 lver in res, and the daemon * automatically requests the current lver + 1. When * multiple hosts are making requests, this flag can * produce unexpected results, and it would be safer * to read the resource, check that the current owner * is the one being targeted, and use that owner's * lver + 1 as the specifically requested lver. */ #define SANLK_REQUEST_NEXT_LVER 0x00000001 /* * request force_mode * * SANLK_REQ_FORCE (SANLK_REQ_KILL_PID deprecated) * Send SIGKILL to the pid holding the resource * (or SIGTERM if SIGKILL is restricted.) * * SANLK_REQ_GRACEFUL * Run killpath against the pid if it is defined, otherwise * send SIGTERM to the pid (or SIGKILL if SIGTERM is restricted). */ #define SANLK_REQ_FORCE 0x00000001 #define SANLK_REQ_GRACEFUL 0x00000002 /* old name deprecated */ #define SANLK_REQ_KILL_PID SANLK_REQ_FORCE int sanlock_register(void); int sanlock_restrict(int sock, uint32_t flags); int sanlock_killpath(int sock, uint32_t flags, const char *path, char *args); int sanlock_acquire(int sock, int pid, uint32_t flags, int res_count, struct sanlk_resource *res_args[], struct sanlk_options *opt_in); int sanlock_acquire2(int sock, int pid, uint32_t flags, struct sanlk_resource *res, struct sanlk_options *opt_in, struct sanlk_host *owner_host, char **owner_name); int sanlock_release(int sock, int pid, uint32_t flags, int res_count, struct sanlk_resource *res_args[]); int sanlock_inquire(int sock, int pid, uint32_t flags, int *res_count, char **res_state); int sanlock_convert(int sock, int pid, uint32_t flags, struct sanlk_resource *res); int sanlock_request(uint32_t flags, uint32_t force_mode, struct sanlk_resource *res); int sanlock_examine(uint32_t flags, struct sanlk_lockspace *ls, struct sanlk_resource *res); int sanlock_set_lvb(uint32_t flags, struct sanlk_resource *res, char *lvb, int lvblen); int sanlock_get_lvb(uint32_t flags, struct sanlk_resource *res, char *lvb, int lvblen); /* * Functions to convert between string and struct resource formats. * All allocate space for returned data that the caller must free. */ /* * convert from struct sanlk_resource to string with format: * :::[::...]: */ int sanlock_res_to_str(struct sanlk_resource *res, char **str_ret); /* * convert to struct sanlk_resource from string with format: * :::[::...][:] */ int sanlock_str_to_res(char *str, struct sanlk_resource **res_ret); /* * convert from array of struct sanlk_resource * to state string with format: * "RESOURCE1 RESOURCE2 RESOURCE3 ..." * RESOURCE format in sanlock_res_to_str() comment */ int sanlock_args_to_state(int res_count, struct sanlk_resource *res_args[], char **res_state); /* * convert to array of struct sanlk_resource * from state string with format: * "RESOURCE1 RESOURCE2 RESOURCE3 ..." * RESOURCE format in sanlock_str_to_res() comment */ int sanlock_state_to_args(char *res_state, int *res_count, struct sanlk_resource ***res_args); /* * convert to struct sanlk_lockspace from string with format: * ::: */ int sanlock_str_to_lockspace(char *str, struct sanlk_lockspace *ls); #endif sanlock/src/sanlock_rv.h000066400000000000000000000050251516326262600156240ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. */ #ifndef __SANLOCK_RV_H__ #define __SANLOCK_RV_H__ #define SANLK_OK 1 #define SANLK_NONE 0 /* unused */ #define SANLK_ERROR -201 #define SANLK_AIO_TIMEOUT -202 #define SANLK_WD_ERROR -203 #define SANLK_VERSION_INCOMPAT -204 /* run_ballot */ #define SANLK_DBLOCK_READ -210 #define SANLK_DBLOCK_WRITE -211 #define SANLK_DBLOCK_LVER -212 #define SANLK_DBLOCK_MBAL -213 #define SANLK_DBLOCK_CHECKSUM -214 /* verify_leader, leader_read, leader_write (paxos or delta) (when adding to list, check if it should be a corrupt_result()) */ #define SANLK_LEADER_READ -220 #define SANLK_LEADER_WRITE -221 #define SANLK_LEADER_DIFF -222 #define SANLK_LEADER_MAGIC -223 #define SANLK_LEADER_VERSION -224 #define SANLK_LEADER_SECTORSIZE -225 #define SANLK_LEADER_LOCKSPACE -226 #define SANLK_LEADER_RESOURCE -227 #define SANLK_LEADER_NUMHOSTS -228 #define SANLK_LEADER_CHECKSUM -229 #define SANLK_ADDLS_SIZES -230 #define SANLK_ADDLS_INVALID_HOSTID -231 /* paxos_lease_acquire, paxos_lease_release */ #define SANLK_ACQUIRE_LVER -240 #define SANLK_ACQUIRE_LOCKSPACE -241 #define SANLK_ACQUIRE_IDDISK -242 #define SANLK_ACQUIRE_IDLIVE -243 #define SANLK_ACQUIRE_OWNED -244 #define SANLK_ACQUIRE_OTHER -245 #define SANLK_ACQUIRE_SHRETRY -246 #define SANLK_ACQUIRE_OWNED_RETRY -247 #define SANLK_ACQUIRE_OWNER_GEN_REVERTED -248 #define SANLK_ACQUIRE_OWNED_NO_TIMEOUT -249 #define SANLK_RELEASE_LVER -252 #define SANLK_RELEASE_OWNER -253 /* delta_lease_renew, delta_lease_acquire */ #define SANLK_RENEW_OWNER -260 #define SANLK_RENEW_DIFF -261 #define SANLK_HOSTID_BUSY -262 #define SANLK_RENEW_DEAD_EXT -263 /* request_token */ #define SANLK_REQUEST_MAGIC -270 #define SANLK_REQUEST_VERSION -271 #define SANLK_REQUEST_OLD -272 #define SANLK_REQUEST_LVER -273 /* rindex ops */ #define SANLK_RINDEX_MAGIC -274 #define SANLK_RINDEX_VERSION -275 #define SANLK_RINDEX_LOCKSPACE -276 #define SANLK_RINDEX_OFFSET -277 #define SANLK_RINDEX_DIFF -278 #define SANLK_WRONG_GENERATION -280 #define SANLK_WRONG_CAW -281 #define SANLK_CAW_CONFIG -282 #define SANLK_CAW_OPEN -283 #define SANLK_CAW_IOCTL -284 #define SANLK_CAW_MISCOMPARE -285 #define SANLK_CAW_ERROR -286 #define SANLK_IOTYPE -287 #define SANLK_CAW_ILLEGAL -288 #endif sanlock/src/sanlock_sock.c000066400000000000000000000013641516326262600161310ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. */ #include #include #include #include #include #include #include #include "sanlock.h" #include "sanlock_sock.h" int sanlock_socket_address(const char *dir, struct sockaddr_un *addr) { memset(addr, 0, sizeof(struct sockaddr_un)); addr->sun_family = AF_LOCAL; snprintf(addr->sun_path, sizeof(addr->sun_path) - 1, "%s/%s", dir, SANLK_SOCKET_NAME); return 0; } sanlock/src/sanlock_sock.h000066400000000000000000000053361516326262600161410ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. */ #ifndef __SANLOCK_SOCK_H__ #define __SANLOCK_SOCK_H__ #define SANLK_SOCKET_NAME "sanlock.sock" #define SM_MAGIC 0x04282010 #define SM_PROTO 0x00000001 #define SM_CB_PROTO 0x00000001 #define MAX_CLIENT_MSG (1024 * 1024) /* TODO: this is random */ enum { SM_CMD_REGISTER = 1, SM_CMD_ADD_LOCKSPACE = 2, SM_CMD_REM_LOCKSPACE = 3, SM_CMD_SHUTDOWN = 4, SM_CMD_STATUS = 5, SM_CMD_LOG_DUMP = 6, SM_CMD_ACQUIRE = 7, SM_CMD_RELEASE = 8, SM_CMD_INQUIRE = 9, SM_CMD_RESTRICT = 10, SM_CMD_REQUEST = 11, SM_CMD_ALIGN = 12, SM_CMD_EXAMINE_LOCKSPACE = 13, SM_CMD_EXAMINE_RESOURCE = 14, SM_CMD_HOST_STATUS = 15, SM_CMD_INQ_LOCKSPACE = 16, SM_CMD_KILLPATH = 17, SM_CMD_WRITE_LOCKSPACE = 18, SM_CMD_WRITE_RESOURCE = 19, SM_CMD_READ_LOCKSPACE = 20, SM_CMD_READ_RESOURCE = 21, SM_CMD_GET_LOCKSPACES = 22, SM_CMD_GET_HOSTS = 23, SM_CMD_READ_RESOURCE_OWNERS = 24, SM_CMD_SET_LVB = 25, SM_CMD_GET_LVB = 26, SM_CMD_CONVERT = 27, SM_CMD_VERSION = 28, SM_CMD_SHUTDOWN_WAIT = 29, SM_CMD_REG_EVENT = 30, SM_CMD_END_EVENT = 31, SM_CMD_SET_EVENT = 32, SM_CMD_SET_CONFIG = 33, SM_CMD_RENEWAL = 34, SM_CMD_FORMAT_RINDEX = 35, SM_CMD_UPDATE_RINDEX = 36, SM_CMD_LOOKUP_RINDEX = 37, SM_CMD_CREATE_RESOURCE = 38, SM_CMD_DELETE_RESOURCE = 39, SM_CMD_REBUILD_RINDEX = 40, SM_CMD_ACQUIRE2 = 41, SM_CMD_READ_LOCKSPACE_HOST = 42, SM_CMD_INIT_LOCKSPACE_HOST = 43, SM_CMD_SET_HOST = 44, }; #define SM_CB_GET_EVENT 1 struct sm_header { uint32_t magic; uint32_t version; uint32_t cmd; /* SM_CMD_ */ uint32_t cmd_flags; uint32_t length; uint32_t seq; uint32_t data; uint32_t data2; }; #define SANLK_STATE_MAXSTR 4096 #define SANLK_STATE_DAEMON 1 #define SANLK_STATE_CLIENT 2 #define SANLK_STATE_LOCKSPACE 3 #define SANLK_STATE_RESOURCE 4 #define SANLK_STATE_HOST 5 #define SANLK_STATE_RENEWAL 6 #define SANLK_STATE_WORKER 7 #define SANLK_STATE_RTHREAD 8 struct sanlk_state { uint32_t type; /* SANLK_STATE_ */ uint32_t flags; uint32_t data32; /* pid (for client) */ uint64_t data64; char name[SANLK_NAME_LEN]; /* client name or resource name */ uint32_t str_len; char str[]; /* string of internal state */ }; int sanlock_socket_address(const char *dir, struct sockaddr_un *addr); struct event_cb { struct sm_header h; struct sanlk_host_event he; uint64_t from_host_id; uint64_t from_generation; }; #endif sanlock/src/sizeflags.c000066400000000000000000000210061516326262600154420ustar00rootroot00000000000000/* * Copyright 2018 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock_internal.h" #include "sanlock.h" #include "sizeflags.h" #include "log.h" int size_to_max_hosts(int sector_size, int align_size) { if ((align_size == ALIGN_SIZE_1M) && (sector_size == 512)) { return 2000; } else if ((align_size == ALIGN_SIZE_1M) && (sector_size == 4096)) { return 250; } else if ((align_size == ALIGN_SIZE_2M) && (sector_size == 4096)) { return 500; } else if ((align_size == ALIGN_SIZE_4M) && (sector_size == 4096)) { return 1000; } else if ((align_size == ALIGN_SIZE_8M) && (sector_size == 4096)) { return 2000; } else { return 0; } } /* * In previous versions, 512 always implied 1M, and 4K implied 8M. * We fall back to this if an align size flag is not set in the * leader record. */ int sector_size_to_align_size_old(int sector_size) { if (sector_size == 512) return ALIGN_SIZE_1M; if (sector_size == 4096) return ALIGN_SIZE_8M; return -1; } const char *align_size_debug_str(int align_size) { if (align_size == ALIGN_SIZE_1M) return "1M"; if (align_size == ALIGN_SIZE_2M) return "2M"; if (align_size == ALIGN_SIZE_4M) return "4M"; if (align_size == ALIGN_SIZE_8M) return "8M"; return NULL; } /* * struct leader_record */ uint32_t leader_align_flag_from_size(int align_size) { if (align_size == ALIGN_SIZE_1M) return LFL_ALIGN_1M; if (align_size == ALIGN_SIZE_2M) return LFL_ALIGN_2M; if (align_size == ALIGN_SIZE_4M) return LFL_ALIGN_4M; if (align_size == ALIGN_SIZE_8M) return LFL_ALIGN_8M; log_error("leader_align_flag_from_num unknown %d", align_size); return 0; } int leader_align_size_from_flag(uint32_t flags) { if (flags & LFL_ALIGN_1M) return ALIGN_SIZE_1M; if (flags & LFL_ALIGN_2M) return ALIGN_SIZE_2M; if (flags & LFL_ALIGN_4M) return ALIGN_SIZE_4M; if (flags & LFL_ALIGN_8M) return ALIGN_SIZE_8M; return 0; } /* * struct rindex_header */ uint32_t rindex_header_align_flag_from_size(int align_size) { if (align_size == ALIGN_SIZE_1M) return RHF_ALIGN_1M; if (align_size == ALIGN_SIZE_2M) return RHF_ALIGN_2M; if (align_size == ALIGN_SIZE_4M) return RHF_ALIGN_4M; if (align_size == ALIGN_SIZE_8M) return RHF_ALIGN_8M; log_error("rindex_header_align_flag_from_size unknown %d", align_size); return 0; } int rindex_header_align_size_from_flag(uint32_t flags) { if (flags & RHF_ALIGN_1M) return ALIGN_SIZE_1M; if (flags & RHF_ALIGN_2M) return ALIGN_SIZE_2M; if (flags & RHF_ALIGN_4M) return ALIGN_SIZE_4M; if (flags & RHF_ALIGN_8M) return ALIGN_SIZE_8M; return 0; } /* * struct sanlk_lockspace */ int sanlk_lsf_sector_flag_to_size(uint32_t flags) { if (flags & SANLK_LSF_SECTOR512) return 512; if (flags & SANLK_LSF_SECTOR4K) return 4096; return 0; } uint32_t sanlk_lsf_sector_size_to_flag(int sector_size) { if (sector_size == 512) return SANLK_LSF_SECTOR512; if (sector_size == 4096) return SANLK_LSF_SECTOR4K; log_error("sanlk_lsf_sector_size_to_flag invalid sector size %d", sector_size); return 0; } void sanlk_lsf_sector_flags_clear(uint32_t *flags) { *flags &= ~SANLK_LSF_SECTOR512; *flags &= ~SANLK_LSF_SECTOR4K; } void sanlk_lsf_align_flags_clear(uint32_t *flags) { *flags &= ~SANLK_LSF_ALIGN1M; *flags &= ~SANLK_LSF_ALIGN2M; *flags &= ~SANLK_LSF_ALIGN4M; *flags &= ~SANLK_LSF_ALIGN8M; } int sanlk_lsf_align_flag_to_size(uint32_t flags) { if (flags & SANLK_LSF_ALIGN1M) return ALIGN_SIZE_1M; if (flags & SANLK_LSF_ALIGN2M) return ALIGN_SIZE_2M; if (flags & SANLK_LSF_ALIGN4M) return ALIGN_SIZE_4M; if (flags & SANLK_LSF_ALIGN8M) return ALIGN_SIZE_8M; return 0; } uint32_t sanlk_lsf_align_size_to_flag(int align_size) { if (align_size == ALIGN_SIZE_1M) return SANLK_LSF_ALIGN1M; if (align_size == ALIGN_SIZE_2M) return SANLK_LSF_ALIGN2M; if (align_size == ALIGN_SIZE_4M) return SANLK_LSF_ALIGN4M; if (align_size == ALIGN_SIZE_8M) return SANLK_LSF_ALIGN8M; log_error("sanlk_lsf_align_size_to_flag invalid align size %d", align_size); return 0; } /* * struct sanlk_resource */ int sanlk_res_sector_flag_to_size(uint32_t flags) { if (flags & SANLK_RES_SECTOR512) return 512; if (flags & SANLK_RES_SECTOR4K) return 4096; return 0; } uint32_t sanlk_res_sector_size_to_flag(int sector_size) { if (sector_size == 512) return SANLK_RES_SECTOR512; if (sector_size == 4096) return SANLK_RES_SECTOR4K; log_error("sanlk_res_sector_size_to_flag invalid sector size %d", sector_size); return 0; } void sanlk_res_sector_flags_clear(uint32_t *flags) { *flags &= ~SANLK_RES_SECTOR512; *flags &= ~SANLK_RES_SECTOR4K; } void sanlk_res_align_flags_clear(uint32_t *flags) { *flags &= ~SANLK_RES_ALIGN1M; *flags &= ~SANLK_RES_ALIGN2M; *flags &= ~SANLK_RES_ALIGN4M; *flags &= ~SANLK_RES_ALIGN8M; } int sanlk_res_align_flag_to_size(uint32_t flags) { if (flags & SANLK_RES_ALIGN1M) return ALIGN_SIZE_1M; if (flags & SANLK_RES_ALIGN2M) return ALIGN_SIZE_2M; if (flags & SANLK_RES_ALIGN4M) return ALIGN_SIZE_4M; if (flags & SANLK_RES_ALIGN8M) return ALIGN_SIZE_8M; return 0; } uint32_t sanlk_res_align_size_to_flag(int align_size) { if (align_size == ALIGN_SIZE_1M) return SANLK_RES_ALIGN1M; if (align_size == ALIGN_SIZE_2M) return SANLK_RES_ALIGN2M; if (align_size == ALIGN_SIZE_4M) return SANLK_RES_ALIGN4M; if (align_size == ALIGN_SIZE_8M) return SANLK_RES_ALIGN8M; log_error("sanlk_res_align_size_to_flag invalid align size %d", align_size); return 0; } /* * struct sanlk_rindex */ int sanlk_rif_sector_flag_to_size(uint32_t flags) { if (flags & SANLK_RIF_SECTOR512) return 512; if (flags & SANLK_RIF_SECTOR4K) return 4096; return 0; } uint32_t sanlk_rif_sector_size_to_flag(int sector_size) { if (sector_size == 512) return SANLK_RIF_SECTOR512; if (sector_size == 4096) return SANLK_RIF_SECTOR4K; log_error("sanlk_rif_sector_size_to_flag invalid sector size %d", sector_size); return 0; } int sanlk_rif_align_flag_to_size(uint32_t flags) { if (flags & SANLK_RIF_ALIGN1M) return ALIGN_SIZE_1M; if (flags & SANLK_RIF_ALIGN2M) return ALIGN_SIZE_2M; if (flags & SANLK_RIF_ALIGN4M) return ALIGN_SIZE_4M; if (flags & SANLK_RIF_ALIGN8M) return ALIGN_SIZE_8M; return 0; } uint32_t sanlk_rif_align_size_to_flag(int align_size) { if (align_size == ALIGN_SIZE_1M) return SANLK_RIF_ALIGN1M; if (align_size == ALIGN_SIZE_2M) return SANLK_RIF_ALIGN2M; if (align_size == ALIGN_SIZE_4M) return SANLK_RIF_ALIGN4M; if (align_size == ALIGN_SIZE_8M) return SANLK_RIF_ALIGN8M; log_error("sanlk_rif_align_size_to_flag invalid align size %d", align_size); return 0; } /* * Translate struct flags passed from libsanlock to numbers. */ int sizes_from_flags(uint32_t flags, int *sector_size, int *align_size, int *max_hosts, const char *kind) { int no_align_flag = 0; int no_sector_flag = 0; *sector_size = 0; *align_size = 0; *max_hosts = 0; /* SANLK_RES flags in sanlk_resource.flags */ if (!strcmp(kind, "RES")) { *align_size = sanlk_res_align_flag_to_size(flags); if (!*align_size) no_align_flag = 1; *sector_size = sanlk_res_sector_flag_to_size(flags); if (!*sector_size) no_sector_flag = 1; } /* SANLK_LSF flags in sanlk_lockspace.flags */ else if (!strcmp(kind, "LSF")) { *align_size = sanlk_lsf_align_flag_to_size(flags); if (!*align_size) no_align_flag = 1; *sector_size = sanlk_lsf_sector_flag_to_size(flags); if (!*sector_size) no_sector_flag = 1; } /* SANLK_RIF flags in sanlk_rindex.flags */ else if (!strcmp(kind, "RIF")) { *align_size = sanlk_rif_align_flag_to_size(flags); if (!*align_size) no_align_flag = 1; *sector_size = sanlk_rif_sector_flag_to_size(flags); if (!*sector_size) no_sector_flag = 1; } else { log_error("unknown kind %s of flags %x", kind, flags); return -1; } if (no_sector_flag != no_align_flag) { log_error("ALIGN and SECTOR flags %s %x must both be set", kind, flags); return -1; } if (!*sector_size) return 0; *max_hosts = size_to_max_hosts(*sector_size, *align_size); if (!*max_hosts) { log_error("Invalid combination of ALIGN and SECTOR flags %s %x", kind, flags); return -1; } return 0; } sanlock/src/sizeflags.h000066400000000000000000000034211516326262600154500ustar00rootroot00000000000000/* * Copyright 2018 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __SIZES_H__ #define __SIZES_H__ #define ALIGN_SIZE_1M 1048576 #define ALIGN_SIZE_2M (2 * ALIGN_SIZE_1M) #define ALIGN_SIZE_4M (4 * ALIGN_SIZE_1M) #define ALIGN_SIZE_8M (8 * ALIGN_SIZE_1M) int size_to_max_hosts(int sector_size, int align_size); int sector_size_to_align_size_old(int sector_size); const char *align_size_debug_str(int align_size); uint32_t leader_align_flag_from_size(int align_size); int leader_align_size_from_flag(uint32_t flags); uint32_t rindex_header_align_flag_from_size(int align_size); int rindex_header_align_size_from_flag(uint32_t flags); int sanlk_lsf_sector_flag_to_size(uint32_t flags); uint32_t sanlk_lsf_sector_size_to_flag(int sector_size); int sanlk_lsf_align_flag_to_size(uint32_t flags); uint32_t sanlk_lsf_align_size_to_flag(int align_size); void sanlk_lsf_sector_flags_clear(uint32_t *flags); void sanlk_lsf_align_flags_clear(uint32_t *flags); int sanlk_res_sector_flag_to_size(uint32_t flags); uint32_t sanlk_res_sector_size_to_flag(int sector_size); int sanlk_res_align_flag_to_size(uint32_t flags); uint32_t sanlk_res_align_size_to_flag(int align_size); void sanlk_res_sector_flags_clear(uint32_t *flags); void sanlk_res_align_flags_clear(uint32_t *flags); int sanlk_rif_sector_flag_to_size(uint32_t flags); uint32_t sanlk_rif_sector_size_to_flag(int sector_size); int sanlk_rif_align_flag_to_size(uint32_t flags); uint32_t sanlk_rif_align_size_to_flag(int align_size); int sizes_from_flags(uint32_t flags, int *sector_size, int *align_size, int *max_hosts, const char *kind); #endif sanlock/src/task.c000066400000000000000000000060651516326262600144250ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock_internal.h" #include "log.h" #include "task.h" void setup_task_aio(struct task *task, int use_aio, int cb_size) { int rv; task->use_aio = use_aio; memset(&task->aio_ctx, 0, sizeof(task->aio_ctx)); /* main task doesn't actually do disk io so it passes in, * cb_size 0, but it still wants use_aio set for other * tasks to copy */ if (!use_aio) return; if (!cb_size) return; rv = io_setup(cb_size, &task->aio_ctx); if (rv < 0) goto fail; task->cb_size = cb_size; task->callbacks = malloc(cb_size * sizeof(struct aicb)); if (!task->callbacks) { rv = -ENOMEM; goto fail_setup; } memset(task->callbacks, 0, cb_size * sizeof(struct aicb)); return; fail_setup: io_destroy(task->aio_ctx); fail: task->use_aio = NO_AIO; } void close_task_aio(struct task *task) { struct timespec ts; struct io_event event; uint64_t last_warn; uint64_t begin; uint64_t now; int rv, i, used, lvl; if (!task->use_aio) goto skip_aio; memset(&ts, 0, sizeof(struct timespec)); ts.tv_sec = com.io_timeout; last_warn = time(NULL); begin = last_warn; /* wait for all outstanding aio to complete before destroying aio context, freeing iocb and buffers */ while (1) { now = time(NULL); if (now - last_warn >= (com.io_timeout * 6)) { last_warn = now; lvl = LOG_ERR; } else { lvl = LOG_DEBUG; } used = 0; for (i = 0; i < task->cb_size; i++) { if (!task->callbacks[i].used) continue; used++; log_level(0, 0, task->name, lvl, "close_task_aio %d %p busy", i, &task->callbacks[i]); } if (!used) break; if (now - begin >= 120) break; memset(&event, 0, sizeof(event)); rv = io_getevents(task->aio_ctx, 1, 1, &event, &ts); if (rv == -EINTR) continue; if (rv < 0) break; if (rv == 1) { struct iocb *ev_iocb = event.obj; struct aicb *ev_aicb = container_of(ev_iocb, struct aicb, iocb); if (ev_aicb->buf == task->iobuf) task->iobuf = NULL; log_taske(task, "aio collect %p:%p:%p result %ld:%ld close free", ev_aicb, ev_iocb, ev_aicb->buf, event.res, event.res2); ev_aicb->used = 0; free(ev_aicb->buf); ev_aicb->buf = NULL; } } if (used) log_taskd(task, "close_task_aio destroy %d incomplete ops", used); io_destroy(task->aio_ctx); if (used) log_taske(task, "close_task_aio destroyed %d incomplete ops", used); if (task->iobuf) free(task->iobuf); if (task->paxos_areabuf) free(task->paxos_areabuf); skip_aio: if (task->callbacks) free(task->callbacks); task->callbacks = NULL; } sanlock/src/task.h000066400000000000000000000006511516326262600144250ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __TASK_H__ #define __TASK_H__ void setup_task_aio(struct task *task, int use_aio, int cb_size); void close_task_aio(struct task *task); #endif sanlock/src/timeouts.c000066400000000000000000000126001516326262600153240ustar00rootroot00000000000000/* * Copyright 2012 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock_internal.h" #include "log.h" #include "task.h" #include "timeouts.h" void setup_timeouts(void) { /* * graceful shutdown is client pids stopping their activity and * releasing their sanlock leases in response to a killpath program * they configured, or in response to sigterm from sanlock if they * did not set a killpath program. It's an opportunity for the client * pid to exit more gracefully than getting sigkill. If the client * pid does not release leases in response to the killpath/sigterm, * then eventually sanlock will escalate and send a sigkill. * * It's hard to know what portion of recovery time should be allocated * to graceful shutdown before escalating to sigkill. The smaller the * watchdog timeout, the less time between entering recovery mode and * the watchdog potentially firing. 10 seconds before the watchdog * will fire, the idea is to give up on graceful shutdown and resort * to sending sigkill to any client pids that have not released their * leases. This gives 10 sec for the pids to exit from sigkill, * sanlock to get the exit statuses, clear the expiring wdmd connection, * and hopefully have wdmd ping the watchdog again before it fires. * A graceful shutdown period of less than 10/15 sec seems pointless, * so if there is anything less than 10/15 sec available for a graceful * shutdown we don't bother and go directly to sigkill (this could * of course be changed if programs are indeed able to respond * quickly during graceful shutdown.) */ if (!com.kill_grace_set && (com.watchdog_fire_timeout < DEFAULT_WATCHDOG_FIRE_TIMEOUT)) { if (com.watchdog_fire_timeout < 60 && com.watchdog_fire_timeout >= 30) com.kill_grace_seconds = 15; else if (com.watchdog_fire_timeout < 30) com.kill_grace_seconds = 0; } } /* * Some of these timeouts depend on the the io_timeout used by *another* * host, passed as the arg, not the local io_timeout. */ /* All hosts are required to use the same watchdog_fire_timeout. */ int calc_host_dead_seconds(int io_timeout) { /* id_renewal_fail_seconds + com.watchdog_fire_timeout */ return (8 * io_timeout) + com.watchdog_fire_timeout; } int calc_id_renewal_seconds(int io_timeout) { return 2 * io_timeout; } int calc_id_renewal_fail_seconds(int io_timeout) { return 8 * io_timeout; } int calc_id_renewal_warn_seconds(int io_timeout) { return 6 * io_timeout; } int calc_set_bitmap_seconds(int io_timeout) { if (com.set_bitmap_seconds) return com.set_bitmap_seconds; /* 3 * id_renewal_seconds, somewhat random choice */ return 6 * io_timeout; } void log_timeouts(int io_timeout_arg) { int io_timeout_seconds = io_timeout_arg; int id_renewal_seconds = 2 * io_timeout_seconds; int id_renewal_fail_seconds = 8 * io_timeout_seconds; int id_renewal_warn_seconds = 6 * io_timeout_seconds; /* those above are chosen by us, the rest are based on them */ int host_dead_seconds = id_renewal_fail_seconds + com.watchdog_fire_timeout; int delta_large_delay = id_renewal_seconds + (6 * io_timeout_seconds); int delta_short_delay = 2 * io_timeout_seconds; int max = host_dead_seconds; if (delta_large_delay > max) max = delta_large_delay; int delta_acquire_held_max = max + delta_short_delay + (4 * io_timeout_seconds); int delta_acquire_held_min = max; int delta_acquire_free_max = delta_short_delay + (3 * io_timeout_seconds); int delta_acquire_free_min = delta_short_delay; int delta_renew_max = 2 * io_timeout_seconds; int delta_renew_min = 0; int paxos_acquire_held_max = host_dead_seconds + (7 * io_timeout_seconds); int paxos_acquire_held_min = host_dead_seconds; int paxos_acquire_free_max = 6 * io_timeout_seconds; int paxos_acquire_free_min = 0; int request_finish_seconds = 3 * id_renewal_seconds; /* random */ log_debug("io_timeout_seconds %d", io_timeout_seconds); log_debug("id_renewal_seconds %d", id_renewal_seconds); log_debug("id_renewal_fail_seconds %d", id_renewal_fail_seconds); log_debug("id_renewal_warn_seconds %d", id_renewal_warn_seconds); log_debug("host_dead_seconds %d", host_dead_seconds); log_debug("delta_large_delay %d", delta_large_delay); log_debug("delta_short_delay %d", delta_short_delay); log_debug("delta_acquire_held_max %d", delta_acquire_held_max); log_debug("delta_acquire_held_min %d", delta_acquire_held_min); log_debug("delta_acquire_free_max %d", delta_acquire_free_max); log_debug("delta_acquire_free_min %d", delta_acquire_free_min); log_debug("delta_renew_max %d", delta_renew_max); log_debug("delta_renew_min %d", delta_renew_min); log_debug("paxos_acquire_held_max %d", paxos_acquire_held_max); log_debug("paxos_acquire_held_min %d", paxos_acquire_held_min); log_debug("paxos_acquire_free_max %d", paxos_acquire_free_max); log_debug("paxos_acquire_free_min %d", paxos_acquire_free_min); log_debug("request_finish_seconds %d", request_finish_seconds); } sanlock/src/timeouts.h000066400000000000000000001035731516326262600153430ustar00rootroot00000000000000/* * Copyright 2010-2012 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ /* * Example of watchdog behavior when host_id renewals fail, assuming * that sanlock cannot successfully kill the pids it is supervising that * depend on the given host_id. * * * Using these values in the example * wdmd test interval = 10 (defined in wdmd/main.c) * watchdog_fire_timeout = 60 (constant) * io_timeout_seconds = 10 (defined by us) * id_renewal_seconds = 20 (= delta_renew_max = 2 * io_timeout_seconds) * id_renewal_fail_seconds = 80 (= 4 * delta_renew_max = 8 * io_timeout_seconds) * host_dead_seconds = 140 (id_renewal_fail_seconds + watchdog_fire_timeout) * * T time in seconds * * 0: sanlock renews host_id on disk * sanlock calls wdmd_test_live(0, 80) [0 + 80] * wdmd test_client sees now 0 < expire 80 ok -> keepalive * * 10: wdmd test_client sees now 10 < expire 80 ok -> keepalive * * 20: sanlock renews host_id on disk ok * sanlock calls wdmd_test_live(20, 100) [20 + 80] * wdmd test_client sees now 20 < expire 100 or 80 ok -> keepalive * * 30: wdmd test_client sees now 30 < expire 100 ok -> keepalive * * 40: sanlock renews host_id on disk ok * sanlock calls wdmd_test_live(40, 120) [40 + 80] * wdmd test_client sees now 40 < expire 120 or 100 ok -> keepalive * * 50: wdmd test_client sees now 50 < expire 120 ok -> keepalive * * all normal until 59 * --------------------------------------------------------- * problems begin at 60 * * 60: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 60 < expire 120 ok -> keepalive * * 70: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 70 < expire 120 ok -> keepalive * * 80: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 80 < expire 120 ok -> keepalive * * 90: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 90 < expire 120 ok -> keepalive * * 100: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 100 < expire 120 ok -> keepalive * messages: check_our_lease warning (sanlock) * * 110: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 110 < expire 120 ok -> keepalive (from dev close) * messages: watchdog closed unclean (wdmd), test warning (wdmd) * * 120: sanlock fails to renew host_id on disk -> no wdmd_test_live * sanlock enters recovery mode and starts killing pids because we have reached * now (120) is id_renewal_fail_seconds (80) after last renewal (40) * wdmd test_client sees now 120 >= expire 120 fail -> no keepalive * messages: check_our_lease failed (sanlock), test failed (wdmd) * * . /dev/watchdog will fire at last keepalive + watchdog_fire_timeout = * T110 + 60 = T170 * . host_id will expire at * last disk renewal ok + id_renewal_fail_seconds + watchdog_fire_timeout * T40 + 80 + 60 = T180 * (aka last disk renewal ok + host_dead_seconds, T40 + 140 = T180) * . the wdmd test at T110 could have been at T119, so wdmd would have * seen the client unexpired/ok and done keepalive at 119 just before the * expiry at 120, which would lead to /dev/watchdog firing at 119+60 = T179 * . so, the watchdog could fire as early as T170 or as late as T179, but * the host_id will not expire until T180 * * 130: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 130 > expire 120 fail -> no keepalive * * 140: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 140 > expire 120 fail -> no keepalive * * 150: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 150 > expire 120 fail -> no keepalive * * 160: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 160 > expire 120 fail -> no keepalive * * 170: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 170 > expire 120 fail -> no keepalive * /dev/watchdog fires because last keepalive was T110, 60 seconds ago * (earliest possible /dev/watchdog firing due to wdmd checking expiry just * after sanlock calls wdmd_test_live at T110 and just after the expiry at T120) * * 179: (latest possible /dev/watchdog firing due to wdmd checking expiry just * before the expiry at T119) * * 180: another host can acquire leases held by host_id. * This is host_dead_seconds (140) after the last successful renewal (T40) */ /* * Example of watchdog behavior when host_id renewals fail, assuming * that sanlock cannot successfully kill the pids it is supervising that * depend on the given host_id. * * This analyzes the sanlock and wdmd operations every 5 seconds, and * assumes that the sanlock and wdmd daemons are both performing their * steps right at each 5 second mark, but in reality they will likely be * offset from each other. * * * Using these values in the example * wdmd test interval = 5 * watchdog_fire_timeout = 30 * io_timeout_seconds = 5 * id_renewal_seconds = 10 * id_renewal_fail_seconds = 40 * host_dead_seconds = 70 * * wdmd_test_live(renewal_time [now], * expire_time [now + id_renewal_fail_seconds]) * * T time in seconds (now) * * 0: sanlock renews host_id on disk * sanlock calls wdmd_test_live(0, 40) [expire 40 from 0 + 40] * wdmd test_client sees now 0 < expire 40 ok -> keepalive * * 5: wdmd test_client sees now 5 < expire 40 ok -> keepalive * * 10: sanlock renews host_id on disk ok * sanlock calls wdmd_test_live(10, 50) [expire 50 from 10 + 40] * wdmd test_client sees now 10 < expire 50 or 40 ok -> keepalive * (50 if the wdmd check is right after this wdmd_test_live, or * (40 if the wdmd check is right before this wdmd_test_live) * * 15: wdmd test_client sees now 15 < expire 50 ok -> keepalive * * 20: sanlock renews host_id on disk ok * sanlock calls wdmd_test_live(20, 60) [expire 60 from 20 + 40] * wdmd test_client sees now 20 < expire 60 or 50 ok -> keepalive * * 25: wdmd test_client sees now 25 < expire 60 ok -> keepalive * * all normal until 29 * --------------------------------------------------------- * problems begin at 30 * * 30: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 30 < expire 60 ok -> keepalive * * 35: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 35 < expire 60 ok -> keepalive * * 40: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 40 < expire 60 ok -> keepalive * * 45: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 45 < expire 60 ok -> keepalive * * 50: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 50 < expire 60 ok -> keepalive * messages: check_our_lease warning (sanlock) * * 55: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 55 < expire 60 ok -> keepalive (from dev close) * (wdmd sees now >= expire - test_interval) * messages: watchdog closed unclean (wdmd), test warning (wdmd) * * 60: sanlock fails to renew host_id on disk -> no wdmd_test_live * sanlock enters recovery mode and starts killing pids because we have reached * now (60) is id_renewal_fail_seconds (40) after last renewal (20) * wdmd test_client sees now 60 >= expire 60 fail -> no keepalive * messages: check_our_lease failed (sanlock), test failed (wdmd) * * . /dev/watchdog will fire at last keepalive + watchdog_fire_timeout = * T55 + 30 = T85 * . host_id will expire at * last disk renewal ok + id_renewal_fail_seconds + watchdog_fire_timeout * T20 + 40 + 30 = T90 * (aka last disk renewal ok + host_dead_seconds, T20 + 70 = T90) * . the wdmd test at T55 could have been at T59, so wdmd would have * seen the client unexpired/ok and done keepalive at 59 just before the * expiry at 60, which would lead to /dev/watchdog firing at 59+30 = T89 * . so, the watchdog could fire as early as T85 or as late as T89, but * the host_id will not expire until T90 * * 65: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 65 > expire 60 fail -> no keepalive * * 70: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 70 > expire 60 fail -> no keepalive * * 75: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 75 > expire 60 fail -> no keepalive * * 80: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 80 > expire 60 fail -> no keepalive * * 85: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 85 > expire 60 fail -> no keepalive * /dev/watchdog fires because last keepalive was T55, 30 seconds ago * (earliest possible /dev/watchdog firing due to wdmd checking expiry just * after sanlock calls wdmd_test_live at T55 and just after the expiry at T60) * * 89: (latest possible /dev/watchdog firing due to wdmd checking expiry just * before the expiry at T59) * * 90: another host can acquire leases held by host_id. * This is host_dead_seconds (70) after the last successful renewal (T20) * -- * * entering recovery mode at 60 until watchdog firing at 85 is 25 seconds * to hopefully clear client usage of leases and avert the watchdog firing. * kill_grace_seconds is 15, leaving clients 15 seconds to do a graceful * shutdown using their killpath or respond to sigterm. If the client * hasn't dropped its leases in these 15 seconds, sanlock escalates to * using sigkill with 10 seconds remaining until the watchdog fires. * 10 seconds is hopefully long enough for client pids to exit, sanlock * to collect the exit status and clear the wdmd connection before the * watchdog really fires. */ /* * Example of watchdog behavior when host_id renewals fail, assuming * that sanlock cannot successfully kill the pids it is supervising that * depend on the given host_id. * * This analyzes the sanlock and wdmd operations every 2 seconds, and * assumes that the sanlock and wdmd daemons are both performing their * steps right at each 2 second mark, but in reality they will likely be * offset from each other. * * Using these values in the example * wdmd test interval = 2 * watchdog_fire_timeout = 10 * io_timeout_seconds = 2 * id_renewal_seconds = 4 * id_renewal_fail_seconds = 16 * host_dead_seconds = 26 * * wdmd_test_live(renewal_time [now], * expire_time [now + id_renewal_fail_seconds]) * * T time in seconds (now) * * 0: sanlock renews host_id on disk * sanlock calls wdmd_test_live(0, 16) [expire 16 from 0 + 16] * wdmd test_client sees now 0 < expire 16 ok -> keepalive * * 2: wdmd test_client sees now 2 < expire 16 ok -> keepalive * * 4: sanlock renews host_id on disk ok * sanlock calls wdmd_test_live(4, 20) [expire 20 from 4 + 16] * wdmd test_client sees now 4 < expire 16 or 20 ok -> keepalive * * 6: wdmd test_client sees now 6 < expire 20 ok -> keepalive * * 8: sanlock renews host_id on disk ok * sanlock calls wdmd_test_live(8, 24) [expire 24 from 8 + 16] * wdmd test_client sees now 8 < expire 20 or 24 ok -> keepalive * * 10: wdmd test_client sees now 10 < expire 24 ok -> keepalive * * all normal until 11 * --------------------------------------------------------- * problems begin at 12 * * 12: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 12 < expire 24 ok -> keepalive * * 14: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 14 < expire 24 ok -> keepalive * * 16: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 16 < expire 24 ok -> keepalive * * 18: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 18 < expire 24 ok -> keepalive * * 20: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 20 < expire 24 ok -> keepalive * messages: check_our_lease warning (sanlock) * (these warnings appear 6*io_timeout after last renewal) * * 22: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 22 < expire 24 ok -> keepalive (from dev close) * (wdmd sees now >= expire - test_interval) * messages: watchdog closed unclean (wdmd), test warning (wdmd) * * 24: sanlock fails to renew host_id on disk -> no wdmd_test_live * sanlock enters recovery mode and starts killing pids because we have reached * now (24) is id_renewal_fail_seconds (16) after last renewal (8) * wdmd test_client sees now 24 >= expire 24 fail -> no keepalive * messages: check_our_lease failed (sanlock), test failed (wdmd) * * . /dev/watchdog will fire at last keepalive + watchdog_fire_timeout = * T22 + 10 = T32 * . host_id will expire at * last disk renewal ok + id_renewal_fail_seconds + watchdog_fire_timeout * T8 + 16 + 10 = T34 * (aka last disk renewal ok + host_dead_seconds, T8 + 26 = T34) * . the wdmd test at T22 could have been at T23, so wdmd would have * seen the client unexpired/ok and done keepalive at 23 just before the * expiry at 24, which would lead to /dev/watchdog firing at 23+10 = T33 * . so, the watchdog could fire as early as T32 or as late as T33, but * the host_id will not expire until T34 * * 26: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 26 > expire 24 fail -> no keepalive * * 28: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 28 > expire 24 fail -> no keepalive * * 30: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 30 > expire 24 fail -> no keepalive * * 32: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 32 > expire 24 fail -> no keepalive * /dev/watchdog fires because last keepalive was T22, 10 seconds ago * (earliest possible /dev/watchdog firing due to wdmd checking expiry just * after sanlock calls wdmd_test_live at T22 and just after the expiry at T24) * * 33: (latest possible /dev/watchdog firing due to wdmd checking expiry just * before the expiry at T23) * * 34: another host can acquire leases held by host_id. * This is host_dead_seconds (26) after the last successful renewal (T8) * * -- * * entering recovery mode at 24 until watchdog firing at 32 is only 8 seconds, * so there is no time for graceful recovery, so kill_grace_seconds would be * set to 0 here. All 8 seconds would be used to hopefully complete sigkill, * collect client exit statuses, and clear the expiring wdmd connection * before the watchdog actually fires. */ /* * Example of watchdog behavior when host_id renewals fail, assuming * that sanlock cannot successfully kill the pids it is supervising that * depend on the given host_id. * * * Using these values in the example * wdmd test interval = 10 (defined in wdmd/main.c) * watchdog_fire_timeout = 60 (constant) * io_timeout_seconds = 20 (defined by us) * id_renewal_seconds = 40 (= delta_renew_max = 2 * io_timeout_seconds) * id_renewal_fail_seconds = 160 (= 4 * delta_renew_max = 8 * io_timeout_seconds) * host_dead_seconds = 220 (id_renewal_fail_seconds + watchdog_fire_timeout) * * T time in seconds * * 0: sanlock renews host_id on disk * sanlock calls wdmd_test_live(0, 160) [0 + 160] * wdmd test_client sees now 0 < expire 160 ok -> keepalive * * 10: wdmd test_client sees now < expire 160 ok -> keepalive * 20: wdmd test_client sees now < expire 160 ok -> keepalive * 30: wdmd test_client sees now < expire 160 ok -> keepalive * * 40: sanlock renews host_id on disk ok * sanlock calls wdmd_test_live(40, 200) [40 + 160] * wdmd test_client sees now 40 < expire 200 or 160 ok -> keepalive * * 50: wdmd test_client sees now < expire 200 ok -> keepalive * 60: wdmd test_client sees now < expire 200 ok -> keepalive * 70: wdmd test_client sees now < expire 200 ok -> keepalive * * 80: sanlock renews host_id on disk ok * sanlock calls wdmd_test_live(80, 240) [80 + 160] * wdmd test_client sees now 80 < expire 240 or 200 ok -> keepalive * * 90: wdmd test_client sees now < expire 240 ok -> keepalive * 100: wdmd test_client sees now < expire 240 ok -> keepalive * 110: wdmd test_client sees now < expire 240 ok -> keepalive * * all normal until 119 * --------------------------------------------------------- * problems begin at 120 * * 120: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now 120 < expire 240 ok -> keepalive * * 130: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now < expire 240 ok -> keepalive * 140: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now < expire 240 ok -> keepalive * 150: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now < expire 240 ok -> keepalive * 160: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now < expire 240 ok -> keepalive * 170: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now < expire 240 ok -> keepalive * 180: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now < expire 240 ok -> keepalive * 190: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now < expire 240 ok -> keepalive * 200: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now < expire 240 ok -> keepalive * 210: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now < expire 240 ok -> keepalive * 220: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now < expire 240 ok -> keepalive * 230: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now < expire 240 ok -> keepalive * * 240: sanlock fails to renew host_id on disk -> no wdmd_test_live * sanlock enters recovery mode and starts killing pids * wdmd test_client sees now 240 >= expire 240 fail -> no keepalive * wdmd starts logging error messages every 10 sec * * . /dev/watchdog will fire at last keepalive + watchdog_fire_timeout = * T230 + 60 = T290 * . host_id will expire at * last disk renewal ok + id_renewal_fail_seconds + watchdog_fire_timeout * T80 + 160 + 60 = T300 * (aka last disk renewal ok + host_dead_seconds, T80 + 220 = T300) * . the wdmd test at T230 could have been at T239, so wdmd would have * seen the client unexpired/ok and done keepalive at 239 just before the * expiry at 240, which would lead to /dev/watchdog firing at 239+60 = T299 * . so, the watchdog could fire as early as T290 or as late as T299, but * the host_id will not expire until T300 * * 250: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now > expire 240 fail -> no keepalive * 260: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now > expire 240 fail -> no keepalive * 270: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now > expire 240 fail -> no keepalive * 280: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now > expire 240 fail -> no keepalive * 290: sanlock fails to renew host_id on disk -> no wdmd_test_live * wdmd test_client sees now > expire 240 fail -> no keepalive * /dev/watchdog fires because last keepalive was T230, 60 seconds ago * (earliest possible /dev/watchdog firing due to wdmd checking expiry * just after sanlock calls wdmd_test_live at T230 and just after expiry at T240) * * 299: (latest possible /dev/watchdog firing due to wdmd checking expiry just * before the expiry at T239) * * 300: another host can acquire leases held by host_id * This is host_dead_seconds (220) after last successful renewal (T80) * -- * * entering recovery mode at 240 until watchdog firing at 290 is 50 seconds * to hopefully clear client usage of leases and avert the watchdog firing. * kill_grace_seconds is 40, leaving clients 40 seconds to do a graceful * shutdown using their killpath or respond to sigterm. If the client * hasn't dropped its leases in these 40 seconds, sanlock escalates to * using sigkill with 10 seconds remaining until the watchdog fires. * 10 seconds is hopefully long enough for client pids to exit, sanlock * to collect the exit status and clear the wdmd connection before the * watchdog really fires. */ /* * Example of how long it takes to acquire a lease that was held * by a failed host. * * wdmd test interval = 5 * watchdog_fire_timeout = 30 * io_timeout_seconds = 10 * id_renewal_seconds = 20 * id_renewal_fail_seconds = 80 * host_dead_seconds = 110 (id_renewal_fail_seconds + watchdog_fire_timeout) * * host that was killed * -------------------- * sanlock debug output: * 13:31:44 7630 [2157]: s1 delta_renew begin read * 13:31:44 7630 [2157]: s1 delta_renew begin write for new ts 7630 * 13:31:44 7630 [2157]: s1 renewed 7630 delta_length 0 interval 20 * * * wdmd debug output: * 1702323101 keepalive 0 * 1702323101 test_interval 5 sleep_seconds 5 poll_timeout 5000 * 1702323104 test_live ci 2 renewal 7630 expire 7710 * 1702323104 test_interval 5 sleep_seconds 3 poll_timeout 3000 * * * host_dead_seconds (110) after last successful renewal (13:31:44) * would be 13:33:34 as the earliest another host can take a lease. * * host that was trying to acquire lease held by dead host * ------------------------------------------------------- * running: while true; do date; lvchange -ay bb/lvol1; sleep 1; done * Mon Dec 11 13:33:50 CST 2023 * LV locked by other host: bb/lvol1 * Failed to lock logical volume bb/lvol1. * Mon Dec 11 13:33:51 CST 2023 * * * sanlock debug output: * 13:31:40 7584 [2342]: s1 renewed 7584 delta_length 0 interval 20 * ... * 13:32:00 7605 [2342]: s1 renewed 7605 delta_length 0 interval 21 * ... * in that renewal at 7605, we see the last renewal from the other host, * which is shown above from the other host's logs at "13:31:44 7630". * we record our own current time 7605 as the last live time for the dead host. * our initial attempts to get a lease held by the dead host will fail * as follows until host_dead_seconds has elapased: * paxos_acquire owner 2 2 7431 host_status 2 2 7630 wait_start 7605 * paxos_acquire owner 2 2 7431 no wait * acquire_disk rv -247 lver 0 at 0 * acquire_token disk error -247 * release_token r_flags 8 lver 0 * release_token done r_flags 8 * cmd_acquire 2,9,2124 acquire_token -247 lease owned by other host * * after retrying for host_dead_seconds, it succeeds: * 13:33:51 7716 [2282]: s1:r5 paxos_acquire begin offset 71303168 0xa 512 1048576 * 13:33:51 7716 [2282]: s1:r5 leader 2 owner 2 2 7431 dblocks 1:2002:2002:2:2:7431:2:0, * 13:33:51 7716 [2282]: s1:r5 paxos_acquire leader 2 owner 2 2 7431 max mbal[1] 2002 our_dblock 0 0 0 0 0 0 * 13:33:51 7716 [2282]: s1:r5 paxos_acquire owner 2 2 7431 host_status 2 2 7630 wait_start 7605 * 13:33:51 7716 [2282]: s1:r5 paxos_acquire owner 2 2 7431 delta 2 2 7630 dead 7716-7605>110 * 13:33:51 7716 [2282]: s1:r5 paxos_acquire leader 2 owner 2 2 7431 * * This shows that the difference between the current time (7716) and the * time it last saw the dead host alive (7605) is >= host_dead_seconds (110), * so it's allowed to get the lease. * * The host was killed at 13:31:44 and its lease was acquired at 13:33:51. * This is 127 seconds, which includes the 110 host_dead_seconds, plus * 17 extra seconds that elapsed between the time the host was killed * and the time that the other host ran its own renewal and recorded the * final renewal of the dead host. * The host that was killed last renewed at 13:31:44, and the other host's * next renewal after that was at 13:32:00 which is a 16 second difference. */ /* * killing pids * * From the time sanlock enters recovery mode and starts killing pids at T120, * until /dev/watchdog fires between T170 and T179, we need to attempt to * gracefully kill pids for some time, and then leave around 10 seconds to * escalate to SIGKILL and clean up leases from the exited pids. * * Working backward from the earlier watchdog firing at T170, leaving 10 seconds * for SIGKILL to succeed, we need to begin SIGKILL at T160. This means we * have from T120 to T160 to allow graceful kill to complete. So, kill_grace_seconds * should be set to 40 by default (T120 to T160). * * T40: last successful disk renewal * T120 - T159: graceful pid shutdown (40 sec) * T160 - T169: SIGKILL once per second (10 sec) * T170 - T179: watchdog fires sometime (SIGKILL continues) * T180: other hosts acquire our leases */ /* * "delta" refers to timed based leases described in Chockler/Malkhi that * we use for host_id ownership. * * "paxos" refers to disk paxos based leases described in Lamport that * we use for resource (vm) ownership. * * "free" refers to a lease (either type) that is not owned by anyone * * "held" refers to a lease (either type) that was owned by a host that * failed, so it was not released/freed. . (if a renewal fails we always attempt another renewal immediately) * * "max" refers to the maximum time that a successful acquire/renew can * take, assuming that every io operation takes the max allowable time * (io_timeout_seconds) * * "min" refers to the minimum time that a successful acquire/renew can * take, assuming that every io operation completes immediately, in * effectively zero time * * * io_timeout_seconds: defined by us * * id_renewal_seconds: defined by us * * id_renewal_fail_seconds: defined by us * * watchdog_fire_timeout: /dev/watchdog will fire without being petted this long * = 60 constant * * host_dead_seconds: the length of time from the last successful host_id * renewal until that host is killed by its watchdog. * = id_renewal_fail_seconds + watchdog_fire_timeout * * delta_large_delay: from the algorithm * = id_renewal_seconds + (6 * io_timeout_seconds) * * delta_short_delay: from the algorithm * = 2 * io_timeout_seconds * * delta_acquire_held_max: max time it can take to successfully * acquire a non-free delta lease * = io_timeout_seconds (read) + * max(delta_large_delay, host_dead_seconds) + * io_timeout_seconds (read) + * io_timeout_seconds (write) + * delta_short_delay + * io_timeout_seconds (read) * * delta_acquire_held_min: min time it can take to successfully * acquire a non-free delta lease * = max(delta_large_delay, host_dead_seconds) * * delta_acquire_free_max: max time it can take to successfully * acquire a free delta lease. * = io_timeout_seconds (read) + * io_timeout_seconds (write) + * delta_short_delay + * io_timeout_seconds (read) * * delta_acquire_free_min: min time it can take to successfully * acquire a free delta lease. * = delta_short_delay * * delta_renew_max: max time it can take to successfully * renew a delta lease. * = io_timeout_seconds (read) + * io_timeout_seconds (write) * * delta_renew_min: min time it can take to successfully * renew a delta lease. * = 0 * * paxos_acquire_held_max: max time it can take to successfully * acquire a non-free paxos lease, uncontended. * = io_timeout_seconds (read leader) + * host_dead_seconds + * io_timeout_seconds (read leader) + * io_timeout_seconds (write dblock) + * io_timeout_seconds (read dblocks) + * io_timeout_seconds (write dblock) + * io_timeout_seconds (read dblocks) + * io_timeout_seconds (write leader) * * paxos_acquire_held_min: min time it can take to successfully * acquire a non-free paxos lease, uncontended. * = host_dead_seconds * * paxos_acquire_free_max: max time it can take to successfully * acquire a free paxos lease, uncontended. * = io_timeout_seconds (read leader) + * io_timeout_seconds (write dblock) + * io_timeout_seconds (read dblocks) + * io_timeout_seconds (write dblock) + * io_timeout_seconds (read dblocks) + * io_timeout_seconds (write leader) * * paxos_acquire_free_min: min time it can take to successfully * acquire a free paxos lease, uncontended. * = 0 * * * How to configure the combination of related timeouts defined by us: * io_timeout_seconds * id_renewal_seconds * id_renewal_fail_seconds * * Here's one approach that seems to produce sensible sets of numbers: * * io_timeout_seconds = N * . max time one io can take * * delta_renew_max = 2N * . max time one renewal can take * * id_renewal_seconds = delta_renew_max (2N) * . delay this long after renewal success before next renew attempt begins * . this will be the difference between two successive renewal timestamps * when io times are effectively 0 * . there's no particular reason for it to be 2N exactly * . if a successful renewal takes the max possible time (delta_renew_max), * then the next renewal attempt will begin right away * . (if a renewal fails we always attempt another renewal immediately) * * id_renewal_fail_seconds = 4 * delta_renew_max (8N) * . time from last successful renewal until recovery begins * . allows for three consecutive max len renewal failures, i.e. * id_renewal_seconds + (3 * delta_renew_max) * * id_renewal_warn_seconds = 3 * delta_renew_max (6N) * . time from last successful renewal until warning about renewal length * . allows for two consecutive max len renewal failures * * T time in seconds * 0 renewal ok * 2N renewal attempt begin * 4N renewal attempt fail1 (each io takes max time) * 4N renewal attempt begin * 6N renewal attempt fail2 (each io takes max time) * 6N renewal attempt begin * 8N renewal attempt fail3 (each io takes max time) * 8N recovery begins (pids killed) * * If ios don't take the max len (delta_renew_max), this just * gives us more attempts to renew before recovery begins. * * io_timeout_seconds N 5 10 20 * id_renewal_seconds 2N 10 20 40 * id_renewal_fail_seconds 8N 40 80 160 * * 5 sec io timeout: fast storage io perf * 10 sec io timeout: normal storage io perf * 20 sec io timeout: slow storage io perf * * [We could break down these computations further by adding a variable * F = number of full len renewal failures allowed before recovery * begins. Above F is fixed at 3, but we may want to vary it to be * 2 or 4.] * * fast norm slow * watchdog_fire_timeout 60 60 60 * * io_timeout_seconds 5 10 20 * id_renewal_seconds 10 20 40 * id_renewal_fail_seconds 40 80 160 * id_renewal_warn_seconds 30 60 120 * * host_dead_seconds 100 140 220 * delta_large_delay 40 80 160 * delta_short_delay 10 20 40 * delta_acquire_held_max 130 200 340 * delta_acquire_held_min 100 140 220 * delta_acquire_free_max 25 50 100 * delta_acquire_free_min 10 20 40 * delta_renew_max 10 20 40 * delta_renew_min 0 0 0 * paxos_acquire_held_max 135 210 360 * paxos_acquire_held_min 100 140 220 * paxos_acquire_free_max 30 60 120 * paxos_acquire_free_min 0 0 0 */ /* * Why does delta_acquire use max(delta_large_delay, host_dead_seconds) instead * of just delta_large_delay as specified in the algorithm? * * 1. the time based lease algorithm uses delta_large_delay to determine that a * host is failed, but we want to be more certain the host is dead based on its * watchdog firing, and we know the watchdog has fired after host_dead_seconds. * * 2. if a delta lease can be acquired and released (freed) before * host_dead_seconds, that could allow the paxos leases of a failed host to be * acquired by someone else before host_dead_seconds (and before the failed * host is really dead), because acquiring a held paxos lease depends on the * delta lease of the failed owner not changing for host_dead_seconds. * We cannot allow a host to acquire another failed host's paxos lease before * host_dead_seconds. * * 3. ios can't be reliably canceled and never really time out; an io is only * really dead when the machine is dead/reset or storage access is cut off. * The delta lease algorithm expects real io timeouts. * * So, the delay is really meant to represent the time until we are certain a * host is safely gone and will no longer write, and for sanlock that means * until the watchdog has reset it. */ #ifndef __TIMEOUTS_H__ #define __TIMEOUTS_H__ void setup_timeouts(void); int calc_host_dead_seconds(int io_timeout); int calc_id_renewal_seconds(int io_timeout); int calc_id_renewal_fail_seconds(int io_timeout); int calc_id_renewal_warn_seconds(int io_timeout); int calc_set_bitmap_seconds(int io_timeout); void log_timeouts(int io_timeout_arg); #endif sanlock/src/watchdog.c000066400000000000000000000101071516326262600152530ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock_internal.h" #include "log.h" #include "watchdog.h" /* * Purpose of watchdog: to forcibly reset the host in the case where a * supervised pid is running but sanlock daemon does not renew its lease * and does not kill the pid (or it kills the pid but the pid does not * exit). So, just before the pid begins running with granted leases, * /dev/watchdog needs to be armed to reboot the host if things go bad right * after the pid goes ahead. */ #include "../wdmd/wdmd.h" /* tell wdmd to open the watchdog device, set the fire timeout and begin keepalives */ int open_watchdog(int con, int fire_timeout) { int rv; if (!com.use_watchdog) return 0; rv = wdmd_open_watchdog(con, fire_timeout); if (rv < 0) { log_error("wdmd_open_watchdog fire_timeout %d error", fire_timeout); return -1; } return 0; } /* tell wdmd that this connection is still good and watchdog pings can continue for it */ void update_watchdog(struct space *sp, uint64_t timestamp, int id_renewal_fail_seconds) { int rv; if (!com.use_watchdog) return; rv = wdmd_test_live(sp->wd_fd, timestamp, timestamp + id_renewal_fail_seconds); if (rv < 0) log_erros(sp, "wdmd_test_live %llu failed %d", (unsigned long long)timestamp, rv); } /* connects to the wdmd daemon */ int connect_watchdog(struct space *sp) { int con; if (!com.use_watchdog) return 0; con = wdmd_connect(); if (con < 0) { log_erros(sp, "wdmd_connect failed %d", con); return -1; } return con; } /* associate wdmd keepalives to the continued liveness of this lockspace */ int activate_watchdog(struct space *sp, uint64_t timestamp, int id_renewal_fail_seconds, int con) { char name[WDMD_NAME_SIZE]; int test_interval, fire_timeout; uint64_t last_keepalive; int rv; if (!com.use_watchdog) return 0; memset(name, 0, sizeof(name)); snprintf(name, WDMD_NAME_SIZE - 1, "sanlock_%s:%llu", sp->space_name, (unsigned long long)sp->host_id); rv = wdmd_register(con, name); if (rv < 0) { log_erros(sp, "wdmd_register failed %d", rv); goto fail_close; } /* the refcount tells wdmd that it should not cleanly exit */ rv = wdmd_refcount_set(con); if (rv < 0) { log_erros(sp, "wdmd_refcount_set failed %d", rv); goto fail_close; } rv = wdmd_status(con, &test_interval, &fire_timeout, &last_keepalive); if (rv < 0) { log_erros(sp, "wdmd_status failed %d", rv); goto fail_clear; } if (fire_timeout != com.watchdog_fire_timeout) { log_erros(sp, "wdmd invalid fire_timeout %d vs %d", fire_timeout, com.watchdog_fire_timeout); goto fail_clear; } rv = wdmd_test_live(con, timestamp, timestamp + id_renewal_fail_seconds); if (rv < 0) { log_erros(sp, "wdmd_test_live in create failed %d", rv); goto fail_clear; } sp->wd_fd = con; return 0; fail_clear: wdmd_refcount_clear(con); fail_close: close(con); return -1; } void deactivate_watchdog(struct space *sp) { int rv; if (!com.use_watchdog) return; log_space(sp, "wdmd_test_live 0 0 to disable"); rv = wdmd_test_live(sp->wd_fd, 0, 0); if (rv < 0) { log_erros(sp, "wdmd_test_live in deactivate failed %d", rv); /* We really want this to succeed to avoid a reset, so retry after a short delay in case the problem was transient... */ usleep(500000); rv = wdmd_test_live(sp->wd_fd, 0, 0); if (rv < 0) log_erros(sp, "wdmd_test_live in deactivate 2 failed %d", rv); } wdmd_refcount_clear(sp->wd_fd); } void disconnect_watchdog(struct space *sp) { if (!com.use_watchdog) return; close(sp->wd_fd); } sanlock/src/watchdog.h000066400000000000000000000017561516326262600152720ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #ifndef __WATCHDOG_H__ #define __WATCHDOG_H__ /* open/close socket connection to wdmd daemon */ int connect_watchdog(struct space *sp); void disconnect_watchdog(struct space *sp); /* tell wdmd to open the watchdog device which arms it and wdmd begins keepalive loop, but the watchdog keepalive is not yet influenced by lockspace renewals. */ int open_watchdog(int con, int fire_timeout); /* associate per-lockspace renewals in sanlock with watchdog petting in wdmd */ int activate_watchdog(struct space *sp, uint64_t timestamp, int id_renewal_fail_seconds, int con); void deactivate_watchdog(struct space *sp); void update_watchdog(struct space *sp, uint64_t timestamp, int id_renewal_fail_seconds); #endif sanlock/tests/000077500000000000000000000000001516326262600136635ustar00rootroot00000000000000sanlock/tests/Makefile000066400000000000000000000031621516326262600153250ustar00rootroot00000000000000TARGET1 = devcount TARGET2 = sanlk_load TARGET3 = sanlk_client TARGET4 = killpath TARGET5 = sanlk_path TARGET6 = sanlk_testr TARGET7 = sanlk_events TARGET8 = sanlk_mixmsg SOURCE1 = devcount.c SOURCE2 = sanlk_load.c SOURCE3 = sanlk_client.c SOURCE4 = killpath.c SOURCE5 = sanlk_path.c SOURCE6 = sanlk_testr.c SOURCE7 = sanlk_events.c SOURCE8 = sanlk_mixmsg.c CFLAGS += -D_GNU_SOURCE -g \ -Wall \ -Wformat \ -Wformat-security \ -Wnested-externs \ -Wpointer-arith \ -Wextra -Wshadow \ -Wcast-align \ -Wwrite-strings \ -Waggregate-return \ -Wstrict-prototypes \ -Winline \ -Wredundant-decls \ -Wno-sign-compare \ -Wp,-D_FORTIFY_SOURCE=2 \ -O2 \ -fexceptions \ -fasynchronous-unwind-tables \ -fdiagnostics-show-option LDFLAGS = -lrt -laio -lblkid -lsanlock all: $(TARGET1) $(TARGET2) $(TARGET3) $(TARGET4) $(TARGET5) $(TARGET6) $(TARGET7) $(TARGET8) $(TARGET1): $(SOURCE1) $(CC) $(CFLAGS) $(LDFLAGS) $< -o $@ -L. -I../src -L../src $(TARGET2): $(SOURCE2) $(CC) $(CFLAGS) $(LDFLAGS) $< -o $@ -L. -I../src -L../src $(TARGET3): $(SOURCE3) $(CC) $(CFLAGS) $(LDFLAGS) $< -o $@ -L. -I../src -L../src $(TARGET4): $(SOURCE4) $(CC) $(CFLAGS) $(LDFLAGS) $< -o $@ -L. -I../src -L../src $(TARGET5): $(SOURCE5) $(CC) $(CFLAGS) $(LDFLAGS) $< -o $@ -L. -I../src -L../src $(TARGET6): $(SOURCE6) $(CC) $(CFLAGS) $(LDFLAGS) $< -o $@ -L. -I../src -L../src $(TARGET7): $(SOURCE7) $(CC) $(CFLAGS) $(LDFLAGS) $< -o $@ -L. -I../src -L../src $(TARGET8): $(SOURCE8) $(CC) $(CFLAGS) $(LDFLAGS) $< -o $@ -L. -I../src -L../src clean: rm -f *.o *.so *.so.* $(TARGET) $(TARGET2) $(TARGET3) $(TARGET4) $(TARGET5) $(TARGET6) $(TARGET7) $(TARGET8) sanlock/tests/__init__.py000066400000000000000000000000001516326262600157620ustar00rootroot00000000000000sanlock/tests/clientn000077500000000000000000000067541516326262600152610ustar00rootroot00000000000000#!/bin/bash num=$1 cmd=$2 dev=$3 if [ "$cmd" == "init" ]; then echo sanlock direct init -s test:0:$dev:0 sanlock direct init -s test:0:$dev:0 for i in `seq 1 $num`; do off=`expr $i \* 1048576` echo sanlock direct init -r test:r$i:$dev:$off sanlock direct init -r test:r$i:$dev:$off done elif [ "$cmd" == "start" ]; then hostid=$4 killpath=$5 echo sanlock client add_lockspace -s test:$hostid:$dev:0 sanlock client add_lockspace -s test:$hostid:$dev:0 for i in `seq 1 $num`; do off=`expr $i \* 1048576` echo ./sanlk_client test r$i $dev $off $killpath & ./sanlk_client test r$i $dev $off $killpath & done elif [ "$cmd" == "delay" ]; then sec=$3 pid=`cat /run/sanlock/sanlock.pid` echo sync with daemon renewals kill -s SIGSTOP $pid sleep 20 kill -s SIGCONT $pid sleep 1 echo sigstop sanlock pid $pid kill -s SIGSTOP $pid echo sleep $sec sleep $sec echo sigcont sanlock pid $pid kill -s SIGCONT $pid elif [ "$cmd" == "iodelay" ]; then sec=$4 pid=`cat /run/sanlock/sanlock.pid` echo sync with daemon renewals kill -s SIGSTOP $pid sleep 20 kill -s SIGCONT $pid sleep 2 echo save linear rm -f /tmp/client-state.txt rm -f /tmp/client-linear.txt rm -f /tmp/client-error.txt dmsetup table $dev > /tmp/client-linear.txt sed "s/linear/error/" /tmp/client-linear.txt > /tmp/client-error.txt echo load error dmsetup suspend $dev dmsetup load $dev /tmp/client-error.txt dmsetup resume $dev echo sleep $sec sleep $sec echo load linear dmsetup suspend $dev dmsetup load $dev /tmp/client-linear.txt dmsetup resume $dev elif [ "$cmd" == "error" ]; then echo save linear rm -f /tmp/client-state.txt rm -f /tmp/client-linear.txt rm -f /tmp/client-error.txt dmsetup table $dev > /tmp/client-linear.txt sed "s/linear/error/" /tmp/client-linear.txt > /tmp/client-error.txt echo load error dmsetup suspend $dev dmsetup load $dev /tmp/client-error.txt dmsetup resume $dev elif [ "$cmd" == "linear" ]; then echo load linear dmsetup suspend $dev dmsetup load $dev /tmp/client-linear.txt dmsetup resume $dev elif [ "$cmd" == "resume" ]; then hostid=$4 echo load linear dmsetup suspend $dev dmsetup load $dev /tmp/client-linear.txt dmsetup resume $dev echo sanlock client add_lockspace -s test:$hostid:$dev:0 sanlock client add_lockspace -s test:$hostid:$dev:0 while read pid state; do echo sanlock client acquire -p $pid -r $state sanlock client acquire -p $pid -r $state ret=$? if [ $ret == 0 ]; then kill -s SIGCONT $pid else kill -s SIGKILL $pid fi done < /tmp/client-state.txt else echo "" echo "clientn N init DEV" echo " sanlock direct init -s test:0:DEV:0" echo " sanlock direct init -r test:rI:DEV:OFF" echo "" echo "clientn N start DEV HOSTID KILLPATH" echo " sanlock client add_lockspace -s test:HOSTID:DEV:0" echo " starts N ./sanlk_client processes" echo "" echo "clientn N delay SEC" echo " sigstop sanlock daemon" echo " sleep SEC" echo " sigcont sanlock daemon" echo "" echo "clientn N iodelay DEV SEC" echo " block i/o to DEV" echo " sleep SEC" echo " unblock i/o to DEV" echo "" echo "clientn N linear DEV" echo " unblock i/o to DEV" echo "" echo "clientn N error DEV" echo " blocks i/o to DEV" echo " causes KILLPATH to run" echo " causes lockspace to be removed" echo "" echo "clientn N resume DEV HOSTID" echo " sanlock client add_lockspace -s test:HOSTID:DEV:0" echo " reacquires leases for sanlk_client pids paused by" echo " killpath_pause, based on inquire state saved by killpath" fi sanlock/tests/conftest.py000066400000000000000000000027711516326262600160710ustar00rootroot00000000000000# Copyright (C) 2019 Red Hat, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. """ Fixtures for sanlock testing. """ from __future__ import absolute_import import pytest import userstorage from . import util # Mark tests with skip if userstorage is missing userstorage.missing_handler = pytest.skip # Requires relative path from tox basedir BACKENDS = userstorage.load_config("./tests/storage.py").BACKENDS class SanlockIsRunning(Exception): """ Raised if sanlock running when it should not """ @pytest.fixture def sanlock_daemon(): """ Run sanlock daemon during a test. """ p = util.start_daemon() try: util.wait_for_daemon(0.5) yield finally: # Killing sanlock allows terminating without removing the lockspace, # which takes about 3 seconds, slowing down the tests. p.kill() p.wait() @pytest.fixture(params=[ BACKENDS["block"], BACKENDS["file"], ]) def user_4k_path(request): """ A path to block device or file on file system on top of 4k block device, provided by the user. If storage is not available, skip the tests. """ backend = request.param with backend: yield backend.path @pytest.fixture def no_sanlock_daemon(): if util.sanlock_is_running(): raise SanlockIsRunning sanlock/tests/constants.py000066400000000000000000000012641516326262600162540ustar00rootroot00000000000000# Copyright (C) 2019 Red Hat, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. """ Constants copied from sanlock source. """ # src/leader.h PAXOS_DISK_MAGIC = 0x06152010 PAXOS_DISK_CLEAR = 0x11282016 DELTA_DISK_MAGIC = 0x12212010 # src/rindex_disk.h RINDEX_DISK_MAGIC = 0x01042018 # src/rindex_disk.h # Copied from the docs module comment. RINDEX_ENTRY_SIZE = 64 RINDEX_ENTRIES_SECTORS = 2000 # src/sanlock_rv.h SANLK_LEADER_MAGIC = -223 # src/sanlock.h SANLK_PATH_LEN = 1024 sanlock/tests/daemon_test.py000066400000000000000000000165741516326262600165540ustar00rootroot00000000000000# Copyright (C) 2019 Red Hat, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. """ Test sanlock client operations. """ from __future__ import absolute_import import io import signal import struct import pytest from . constants import ( DELTA_DISK_MAGIC, PAXOS_DISK_MAGIC, PAXOS_DISK_CLEAR, RINDEX_DISK_MAGIC, RINDEX_ENTRIES_SECTORS, RINDEX_ENTRY_SIZE ) from . import util from . units import MiB def test_single_instance(sanlock_daemon): # Starting another instance while the daemon must fail. p = util.start_daemon() try: util.wait_for_termination(p, 1.0) except util.TimeoutExpired: p.kill() p.wait() assert p.returncode == 1 def test_start_after_kill(): # After killing the daemon, next instance should be able to start. for i in range(5): p = util.start_daemon() try: util.wait_for_daemon(0.5) finally: p.kill() p.wait() assert p.returncode == -signal.SIGKILL def test_client_failure(): # No daemon is running, client must fail with pytest.raises(util.CommandError) as e: util.sanlock("client", "status") assert e.value.returncode == 1 def test_init_lockspace(tmpdir, sanlock_daemon): path = tmpdir.join("lockspace") size = MiB util.create_file(str(path), size) lockspace = "name:1:%s:0" % path util.sanlock("client", "init", "-s", lockspace) with io.open(str(path), "rb") as f: magic, = struct.unpack("< I", f.read(4)) assert magic == DELTA_DISK_MAGIC # TODO: check more stuff here... util.check_guard(str(path), size) def test_init_resource(tmpdir, sanlock_daemon): path = tmpdir.join("resources") size = MiB util.create_file(str(path), size) resource = "ls_name:res_name:%s:0" % path util.sanlock("client", "init", "-r", resource) with io.open(str(path), "rb") as f: magic, = struct.unpack("< I", f.read(4)) assert magic == PAXOS_DISK_MAGIC # TODO: check more stuff here... util.check_guard(str(path), size) def test_format(tmpdir, sanlock_daemon): path = tmpdir.join("rindex") size = 3 * MiB util.create_file(str(path), size) rindex = "ls_name:%s:1M" % path util.sanlock("client", "format", "-x", rindex) with io.open(str(path), "rb") as f: # The first slot should contain the rindex header sector. f.seek(MiB) magic, = struct.unpack("< I", f.read(4)) assert magic == RINDEX_DISK_MAGIC # The rindex entries starts at the second rindex slot sector. All # entries should be zeroed. f.seek(MiB + 512) entries_size = 512 * RINDEX_ENTRIES_SECTORS assert f.read(entries_size) == b"\0" * entries_size # The next slot should contain the internal lease. f.seek(2 * MiB) magic, = struct.unpack("< I", f.read(4)) assert magic == PAXOS_DISK_MAGIC util.check_guard(str(path), size) def test_create(tmpdir, sanlock_daemon): path = tmpdir.join("rindex") # Slots: lockspace rindex master-lease user-lease-1 size = 4 * MiB util.create_file(str(path), size) # Note: using 1 second io timeout (-o 1) for quicker tests. lockspace = "ls_name:1:%s:0" % path util.sanlock("client", "init", "-s", lockspace, "-o", "1") rindex = "ls_name:%s:1M" % path util.sanlock("client", "format", "-x", rindex) util.sanlock("client", "add_lockspace", "-s", lockspace, "-o", "1") util.sanlock("client", "create", "-x", rindex, "-e", "res") with io.open(str(path), "rb") as f: # New entry should be created at the first slot # The first rindex sector is used by the rindex header. f.seek(MiB + 512) util.check_rindex_entry(f.read(RINDEX_ENTRY_SIZE), b"res", 3 * MiB, 0) # The rest of the entries should not be modified. rest = 512 * RINDEX_ENTRIES_SECTORS - RINDEX_ENTRY_SIZE assert f.read(rest) == b"\0" * rest # The next slot should contain the internal lease. f.seek(3 * MiB) magic, = struct.unpack("< I", f.read(4)) assert magic == PAXOS_DISK_MAGIC util.check_guard(str(path), size) def test_delete(tmpdir, sanlock_daemon): path = tmpdir.join("rindex") # Slots: lockspace rindex master-lease user-lease-1 size = 4 * MiB util.create_file(str(path), size) # Note: using 1 second io timeout (-o 1) for quicker tests. lockspace = "ls_name:1:%s:0" % path util.sanlock("client", "init", "-s", lockspace, "-o", "1") rindex = "ls_name:%s:1M" % path util.sanlock("client", "format", "-x", rindex) util.sanlock("client", "add_lockspace", "-s", lockspace, "-o", "1") util.sanlock("client", "create", "-x", rindex, "-e", "res") util.sanlock("client", "delete", "-x", rindex, "-e", "res") with io.open(str(path), "rb") as f: # First entry should be cleared. f.seek(MiB + 512) util.check_rindex_entry(f.read(RINDEX_ENTRY_SIZE), b"", 0, 0) # Rest of entries should not be modified. rest = 512 * RINDEX_ENTRIES_SECTORS - RINDEX_ENTRY_SIZE assert f.read(rest) == b"\0" * rest # The next slot should contain a cleared lease. f.seek(3 * MiB) magic, = struct.unpack("< I", f.read(4)) assert magic == PAXOS_DISK_CLEAR util.check_guard(str(path), size) def test_lookup(tmpdir, sanlock_daemon): path = tmpdir.join("rindex") # Slots: lockspace rindex master-lease user-lease-1 ... user-lease-7 size = 10 * MiB util.create_file(str(path), size) # Note: using 1 second io timeout (-o 1) for quicker tests. lockspace = "ls_name:1:%s:0" % path util.sanlock("client", "init", "-s", lockspace, "-o", "1") rindex = "ls_name:%s:1M" % path util.sanlock("client", "format", "-x", rindex) util.sanlock("client", "add_lockspace", "-s", lockspace, "-o", "1") util.sanlock("client", "create", "-x", rindex, "-e", "res") lookup = util.sanlock("client", "lookup", "-x", rindex, "-e", "res") assert lookup == b"lookup done 0\nname res offset 3145728\n" def test_lookup_uninitialized(tmpdir, sanlock_daemon): path = tmpdir.join("rindex") util.create_file(str(path), MiB) rindex = "ls_name:%s:1M" % path with pytest.raises(util.CommandError) as e: util.sanlock("client", "lookup", "-x", rindex, "-e", "res") assert e.value.returncode == 1 assert e.value.stdout == b"lookup done -2\n" assert e.value.stderr == b"" def test_lookup_missing(tmpdir, sanlock_daemon): path = tmpdir.join("rindex") # Slots: lockspace rindex master-lease user-lease-1 ... user-lease-7 size = 10 * MiB util.create_file(str(path), size) # Note: using 1 second io timeout (-o 1) for quicker tests. lockspace = "ls_name:1:%s:0" % path util.sanlock("client", "init", "-s", lockspace, "-o", "1") rindex = "ls_name:%s:1M" % path util.sanlock("client", "format", "-x", rindex) util.sanlock("client", "add_lockspace", "-s", lockspace, "-o", "1") with pytest.raises(util.CommandError) as e: util.sanlock("client", "lookup", "-x", rindex, "-e", "res") assert e.value.returncode == 1 assert e.value.stdout == b"lookup done -2\n" assert e.value.stderr == b"" sanlock/tests/devcount-dmsetup000077500000000000000000000007731516326262600171260ustar00rootroot00000000000000#!/bin/bash if [ $# -le 1 ]; then echo "num $#" echo "" echo "devcount-dmsetup save " echo "" echo "devcount-dmsetup error " echo "" echo "devcount-dmsetup linear " echo "" fi cmd=$1 dev=$2 if [ "$cmd" == "save" ]; then rm -f /tmp/table-linear.txt rm -f /tmp/table-error.txt dmsetup table $dev > /tmp/table-linear.txt sed "s/linear/error/" /tmp/table-linear.txt > /tmp/table-error.txt exit 0 fi dmsetup suspend $dev dmsetup load $dev /tmp/table-$cmd.txt dmsetup resume $dev sanlock/tests/devcount.c000066400000000000000000001045401516326262600156620ustar00rootroot00000000000000#include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock.h" #include "sanlock_admin.h" #include "sanlock_resource.h" #include "sanlock_direct.h" #define ONEMB 1048576 #define LEASE_SIZE ONEMB FILE *turn_file; char count_path[PATH_MAX]; char lock_path[PATH_MAX]; int count_offset; int lock_offset; int our_hostid; int max_hostid; struct sanlk_lockspace lockspace; struct entry { uint32_t turn; uint32_t hostid; uint64_t pid; uint64_t time; uint64_t count; uint32_t last_turn; uint32_t last_hostid; uint64_t last_pid; uint64_t last_time; uint64_t last_count; }; #define log_debug(fmt, args...) \ do { \ printf("%llu " fmt "\n", (unsigned long long)time(NULL), ##args); \ } while (0) #define log_error(fmt, args...) \ do { \ printf("ERROR %llu " fmt "\n", (unsigned long long)time(NULL), ##args); \ } while (0) static void sigterm_handler(int sig) { log_debug("sigterm_handler %d", sig); } static void setup_sigterm(void) { struct sigaction act; memset(&act, 0, sizeof(act)); act.sa_handler = sigterm_handler; sigaction(SIGTERM, &act, NULL); } static int kill_pid(int pid) { int rv, status; kill(pid, SIGKILL); while (1) { rv = waitpid(pid, &status, 0); if (rv < 0) return -1; if (rv != pid) return -2; if (WIFEXITED(status)) return 0; } } /* kill(pid, SIGSTOP) would be nice, but that won't guarantee the pid has finished all i/o when it returns. Instead, we send SIGUSR1, which child sees after it's done with a (synchronous) write, and calls SIGSTOP on itself */ static void pause_pid(int pid, int child_stderr) { char buf[64]; int rv; kill(pid, SIGUSR1); /* child prints "we_are_paused" to stderr before stopping */ memset(buf, 0, sizeof(buf)); rv = read(child_stderr, buf, sizeof(buf)); if (strstr(buf, "we_are_paused")) return; while (1) { log_error("pause_pid %d read %s", pid, buf); sleep(2); } } static void resume_pid(int pid) { kill(pid, SIGCONT); } static int check_pause(int fd) { struct signalfd_siginfo fdsi; ssize_t rv; rv = read(fd, &fdsi, sizeof(struct signalfd_siginfo)); if (rv != sizeof(struct signalfd_siginfo)) { return 0; } if (fdsi.ssi_signo == SIGUSR1) { return 1; } return 0; } static int setup_pause(void) { sigset_t mask; int fd, rv; sigemptyset(&mask); sigaddset(&mask, SIGUSR1); rv = sigprocmask(SIG_BLOCK, &mask, NULL); if (rv < 0) return rv; fd = signalfd(-1, &mask, SFD_NONBLOCK); if (fd < 0) return -errno; return fd; } static int rand_int(int a, int b) { return a + (int) (((float)(b - a + 1)) * random() / (RAND_MAX+1.0)); } /* 64 byte entry: can fit up to 8 nodes in a 512 byte block */ void print_entries(char *path, int pid, char *buf) { struct entry *e = (struct entry *)buf; int i; for (i = 0; i < (512 / sizeof(struct entry)); i++) { log_error("%s c %d index %d turn %u time %llu %u:%llu:%llu " "last %u %llu %u:%llu:%llu", path, pid, i, e->turn, (unsigned long long)e->time, e->hostid, (unsigned long long)e->pid, (unsigned long long)e->count, e->last_turn, (unsigned long long)e->last_time, e->last_hostid, (unsigned long long)e->last_pid, (unsigned long long)e->last_count); e++; } } void print_our_we(char *path, int pid, int writes, struct entry *our_we, const char *stage) { log_debug("%s c %d %s w %d index %d turn %u time %llu %u:%llu:%llu " "last %u %llu %u:%llu:%llu", path, pid, stage, writes, our_hostid - 1, our_we->turn, (unsigned long long)our_we->time, our_we->hostid, (unsigned long long)our_we->pid, (unsigned long long)our_we->count, our_we->last_turn, (unsigned long long)our_we->last_time, our_we->last_hostid, (unsigned long long)our_we->last_pid, (unsigned long long)our_we->last_count); } #define COUNT_ARGS 6 #define LOCK_ARGS 8 #define MIGRATE_ARGS 9 /* * devcount rw|wr */ static int do_count(int argc, char *argv[]) { char *rbuf, **p_rbuf, *wbuf, **p_wbuf, *vbuf, **p_vbuf; struct entry *re, *max_re, *our_we; int i, fd, rv, error, max_i; int pause_fd; time_t start; uint32_t our_pid = getpid(); uint32_t max_turn; int sec1, sec2; int read_seconds, write_seconds; uint32_t writes = 0; if (argc < COUNT_ARGS) return -1; pause_fd = setup_pause(); strcpy(count_path, argv[2]); sec1 = atoi(argv[3]); sec2 = atoi(argv[4]); our_hostid = atoi(argv[5]); if (!strcmp(argv[1], "rw")) { read_seconds = sec1; write_seconds = sec2; } else { write_seconds = sec1; read_seconds = sec2; } /* printf("%d %s count_disk %s sec1 %d sec2 %d our_hostid %d\n", our_pid, argv[1], count_path, sec1, sec2, our_hostid); */ fd = open(count_path, O_RDWR | O_DIRECT | O_SYNC, 0); if (fd < 0) { perror("open failed"); error = 1; goto fail; } rv = ioctl(fd, BLKFLSBUF); if (rv) { perror("BLKFLSBUF failed"); error = 2; goto fail; } p_rbuf = &rbuf; p_wbuf = &wbuf; p_vbuf = &vbuf; rv = posix_memalign((void *)p_rbuf, getpagesize(), 512); if (rv) { perror("posix_memalign failed"); error = 3; goto fail; } rv = posix_memalign((void *)p_wbuf, getpagesize(), 512); if (rv) { perror("posix_memalign failed"); error = 4; goto fail; } rv = posix_memalign((void *)p_vbuf, getpagesize(), 512); if (rv) { perror("posix_memalign failed"); error = 5; goto fail; } lseek(fd, count_offset, SEEK_SET); rv = read(fd, rbuf, 512); if (rv != 512) { perror("read failed"); error = 6; goto fail; } /* print_entries(our_pid, rbuf); */ /* * reading for "rw" */ if (!strcmp(argv[1], "rw")) { for (i = 0; i < read_seconds; i++) { sleep(1); lseek(fd, count_offset, SEEK_SET); rv = read(fd, vbuf, 512); if (rv != 512) { perror("read failed"); error = 7; goto fail; } if (memcmp(rbuf, vbuf, 512)) { log_error("%s c %d rbuf:", count_path, our_pid); print_entries(count_path, our_pid, rbuf); log_error("%s c %d vbuf:", count_path, our_pid); print_entries(count_path, our_pid, vbuf); error = 8; goto fail; } } } /* * writing */ re = (struct entry *)rbuf; max_re = NULL; max_i = 0; max_turn = 0; for (i = 0; i < (512 / sizeof(struct entry)); i++) { if (!max_re || re->count > max_re->count) { max_re = re; max_i = i; } if (!max_turn || re->turn > max_turn) max_turn = re->turn; re++; } if (max_turn != max_re->turn) { log_error("%s c %d max_turn %d max_re->turn %d\n", count_path, our_pid, max_turn, max_re->turn); error = 9; goto fail; } /* printf("%d max index %d turn %d count %llu\n", our_pid, max_i, max_turn, (unsigned long long)max_re->count); */ memcpy(wbuf, rbuf, 512); our_we = (struct entry *)(wbuf + ((our_hostid - 1) * sizeof(struct entry))); our_we->last_turn = max_re->turn; our_we->last_hostid = max_re->hostid; our_we->last_pid = max_re->pid; our_we->last_time = max_re->time; our_we->last_count = max_re->count; our_we->turn = max_re->turn + 1; our_we->hostid = our_hostid; our_we->pid = our_pid; our_we->time = time(NULL); our_we->count = max_re->count + 1; lseek(fd, count_offset, SEEK_SET); rv = write(fd, wbuf, 512); if (rv != 512) { perror("write failed"); error = 10; goto fail; } writes = 1; print_our_we(count_path, our_pid, writes, our_we, "begin"); start = time(NULL); while (1) { our_we->count++; our_we->time = time(NULL); lseek(fd, count_offset, SEEK_SET); rv = write(fd, wbuf, 512); if (rv != 512) { perror("write failed"); error = 11; goto fail; } writes++; if (write_seconds && (our_we->time - start >= write_seconds)) break; if (!(writes % 64) && check_pause(pause_fd)) { print_our_we(count_path, our_pid, writes, our_we, "pause"); fprintf(stderr, "we_are_paused\n"); raise(SIGSTOP); /* this shouldn't appear until parent does kill(SIGCONT) */ print_our_we(count_path, our_pid, writes, our_we, "resume"); } } print_our_we(count_path, our_pid, writes, our_we, "end"); if (turn_file) { fprintf(turn_file, "turn %03u start %llu end %llu host %u pid %u\n", our_we->turn, (unsigned long long)(max_re->count + 1), (unsigned long long)our_we->count, our_hostid, our_pid); fflush(turn_file); fclose(turn_file); } /* * reading for "wr" */ if (!strcmp(argv[1], "wr")) { memcpy(rbuf, wbuf, 512); for (i = 0; i < read_seconds; i++) { sleep(1); lseek(fd, count_offset, SEEK_SET); rv = read(fd, vbuf, 512); if (rv != 512) { perror("read failed"); error = 12; goto fail; } if (memcmp(rbuf, vbuf, 512)) { log_error("%s c %d rbuf:", count_path, our_pid); print_entries(count_path, our_pid, rbuf); log_error("%s c %d vbuf:", count_path, our_pid); print_entries(count_path, our_pid, vbuf); error = 13; goto fail; } } } return 0; fail: fprintf(stderr, "error %d\n", error); while (1) { log_error("%s c %d error %d", count_path, our_pid, error); print_entries(count_path, our_pid, rbuf); print_entries(count_path, our_pid, vbuf); sleep(2); } } static int add_lockspace(void) { int rv; strcpy(lockspace.name, "devcount"); strcpy(lockspace.host_id_disk.path, lock_path); lockspace.host_id_disk.offset = lock_offset; lockspace.host_id = our_hostid; rv = sanlock_add_lockspace(&lockspace, 0); log_debug("%s p %d sanlock_add_lockspace %d", lock_path, getpid(), rv); return rv; } /* * Test inquire and acquire with version * * lock: * acquire (no lver) * if fail * goto lock; * else * goto run; * * relock: * acquire with saved lver * if fail (others may acquire in lock:) * sigkill pid; * goto lock; * else * sigcont pid; * goto run; * * run: * run rw for a while * inquire pid * save lver * sigstop pid * release ALL * goto relock * */ static int do_relock(int argc, char *argv[]) { char *av[COUNT_ARGS+1]; struct sanlk_resource *res, *res_inq; int i, j, pid, rv, sock, len, status; int c2p[2]; /* child to parent */ int res_count; uint32_t parent_pid = getpid(); uint64_t lver; char *state; if (argc < LOCK_ARGS) return -1; count_offset = 0; strcpy(lock_path, argv[2]); strcpy(count_path, argv[4]); our_hostid = atoi(argv[7]); add_lockspace(); len = sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk); res = malloc(len); memset(res, 0, len); strcpy(res->lockspace_name, lockspace.name); snprintf(res->name, SANLK_NAME_LEN, "resource%s", count_path); res->name[SANLK_NAME_LEN-1] = '\0'; res->num_disks = 1; strncpy(res->disks[0].path, lock_path, SANLK_PATH_LEN); res->disks[0].path[SANLK_PATH_LEN-1] = '\0'; res->disks[0].offset = LEASE_SIZE; /* * argv[0] = devcount * argv[1] = relock * argv[2] = * argv[3] = rw * start copying at argv[3] */ j = 0; av[j++] = strdup(argv[0]); for (i = 3; i < LOCK_ARGS; i++) av[j++] = strdup(argv[i]); av[j] = NULL; while (1) { pipe(c2p); pid = fork(); if (!pid) { int child_pid = getpid(); sock = sanlock_register(); if (sock < 0) { log_error("%s c %d sanlock_register error %d", count_path, child_pid, sock); exit(-1); } res->flags = 0; res->lver = 0; rv = sanlock_acquire(sock, -1, 0, 1, &res, NULL); if (rv < 0) { log_debug("%s c %d sanlock_acquire error %d", count_path, child_pid, rv); /* all hosts are trying to acquire so we expect this to acquire only sometimes; TODO: exit with an error for some rv's */ exit(0); } log_debug("%s c %d sanlock_acquire done", count_path, child_pid); rv = sanlock_restrict(sock, SANLK_RESTRICT_ALL); if (rv < 0) { log_error("%s c %d sanlock_restrict error %d", count_path, child_pid, sock); exit(-1); } /* make child's stderr go to parent c2p[0] */ close(2); dup(c2p[1]); close(c2p[0]); close(c2p[1]); execv(av[0], av); perror("execv devcount problem"); exit(EXIT_FAILURE); } run_more: /* let the child run for 30 seconds before stopping it */ for (i = 0; i < 30; i++) { rv = waitpid(pid, &status, WNOHANG); if (rv == pid) break; sleep(1); } /* we expect child to exit when it fails to acquire the lock because it's held by someone else, or rw run time is up */ if (rv == pid) goto dead_child; rv = sanlock_inquire(-1, pid, 0, &res_count, &state); if (rv == -EBUSY) { /* pid probably still busy doing acquire */ goto run_more; } if (rv == -ESTALE || rv == -ESRCH) { /* pid has exited */ goto run_more; } if (rv < 0) { log_error("%s p %d sanlock_inquire c %d error %d", count_path, parent_pid, pid, rv); goto run_more; } rv = sanlock_str_to_res(state, &res_inq); if (rv < 0) { log_error("%s p %d sanlock_str_to_res error %d %s", count_path, parent_pid, rv, state); goto fail; } lver = res_inq->lver; log_debug("%s p %d sanlock_inquire c %d lver %llu done", count_path, parent_pid, pid, (unsigned long long)lver); free(res_inq); free(state); pause_pid(pid, c2p[0]); log_debug("%s p %d paused c %d", count_path, parent_pid, pid); rv = sanlock_release(-1, pid, SANLK_REL_ALL, 0, NULL); if (rv < 0) { /* pid may have exited */ log_error("%s p %d sanlock_release c %d error %d", count_path, parent_pid, pid, rv); goto kill_child; } log_debug("%s p %d sanlock_release c %d done", count_path, parent_pid, pid); /* give a chance to someone else to acquire the lock in here */ usleep(1000000); res->flags = SANLK_RES_LVER; res->lver = lver; rv = sanlock_acquire(-1, pid, 0, 1, &res, NULL); if (!rv) { /* we got the lock back in the same version */ log_debug("%s p %d sanlock_acquire c %d lver %llu done", count_path, parent_pid, pid, (unsigned long long)lver); resume_pid(pid); goto run_more; } /* someone got the lock between our release and reacquire */ log_debug("%s p %d sanlock_acquire c %d lver %llu error %d", count_path, parent_pid, pid, (unsigned long long)lver, rv); kill_child: kill_pid(pid); log_debug("%s p %d killed c %d", count_path, parent_pid, pid); dead_child: close(c2p[0]); close(c2p[1]); sleep(rand_int(0, 1)); } fail: printf("test failed...\n"); sleep(1000000); return -1; } /* * devcount lock rw * sanlock add_lockspace -s devcount:::0 * devcount rw */ static int do_lock(int argc, char *argv[]) { char *av[COUNT_ARGS+1]; struct sanlk_resource *res; int i, j, pid, rv, sock, len, status; if (argc < LOCK_ARGS) return -1; count_offset = 0; strcpy(lock_path, argv[2]); strcpy(count_path, argv[4]); our_hostid = atoi(argv[7]); add_lockspace(); len = sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk); res = malloc(len); memset(res, 0, len); strcpy(res->lockspace_name, lockspace.name); snprintf(res->name, SANLK_NAME_LEN, "resource%s", count_path); res->name[SANLK_NAME_LEN-1] = '\0'; res->num_disks = 1; strncpy(res->disks[0].path, lock_path, SANLK_PATH_LEN); res->disks[0].path[SANLK_PATH_LEN-1] = '\0'; res->disks[0].offset = LEASE_SIZE; /* * argv[0] = devcount * argv[1] = lock * argv[2] = * argv[3] = rw * start copying at argv[3] */ j = 0; av[j++] = strdup(argv[0]); for (i = 3; i < LOCK_ARGS; i++) av[j++] = strdup(argv[i]); av[j] = NULL; while (1) { pid = fork(); if (!pid) { int child_pid = getpid(); sock = sanlock_register(); if (sock < 0) { log_error("%s c %d sanlock_register error %d", count_path, child_pid, sock); exit(-1); } rv = sanlock_acquire(sock, -1, 0, 1, &res, NULL); if (rv < 0) { log_debug("%s c %d sanlock_acquire error %d", count_path, child_pid, rv); /* all hosts are trying to acquire so we expect this to acquire only sometimes; TODO: exit with an error for some rv's */ exit(0); } log_debug("%s c %d sanlock_acquire done", count_path, child_pid); execv(av[0], av); perror("execv devcount problem"); exit(EXIT_FAILURE); } waitpid(pid, &status, 0); /* TODO: goto fail if exit status is an error */ sleep(rand_int(0, 1)); } printf("test failed...\n"); sleep(1000000); return -1; } static int do_wrap(int argc, char *argv[]) { char *av[COUNT_ARGS+1]; struct sanlk_resource *res; int i, j, rv, sock, len; uint32_t pid = getpid(); if (argc < LOCK_ARGS) return -1; count_offset = 0; strcpy(lock_path, argv[2]); strcpy(count_path, argv[4]); our_hostid = atoi(argv[7]); add_lockspace(); len = sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk); res = malloc(len); memset(res, 0, len); strcpy(res->lockspace_name, lockspace.name); snprintf(res->name, SANLK_NAME_LEN, "resource%s", count_path); res->name[SANLK_NAME_LEN-1] = '\0'; res->num_disks = 1; strncpy(res->disks[0].path, lock_path, SANLK_PATH_LEN); res->disks[0].path[SANLK_PATH_LEN-1] = '\0'; res->disks[0].offset = LEASE_SIZE; /* * argv[0] = devcount * argv[1] = wrap * argv[2] = * argv[3] = rw * start copying at argv[3] */ j = 0; av[j++] = strdup(argv[0]); for (i = 3; i < LOCK_ARGS; i++) av[j++] = strdup(argv[i]); av[j] = NULL; sock = sanlock_register(); if (sock < 0) { log_error("%s c %d sanlock_register error %d", count_path, pid, sock); exit(-1); } rv = sanlock_restrict(sock, SANLK_RESTRICT_SIGKILL); if (rv < 0) { log_error("%s c %d sanlock_restrict error %d", count_path, pid, sock); exit(-1); } rv = sanlock_acquire(sock, -1, 0, 1, &res, NULL); if (rv < 0) { log_error("%s c %d sanlock_acquire error %d", count_path, pid, rv); /* all hosts are trying to acquire so we expect this to acquire only sometimes; TODO: exit with an error for some rv's */ exit(0); } log_debug("%s c %d sanlock_acquire done", count_path, pid); execv(av[0], av); perror("execv devcount problem"); exit(EXIT_FAILURE); } /* * Test migration sequence (source inquires/releases, dest acquires lver) * * dest forks (e.g. libvirtd creates qemu pid) * dest child does sanlock_register, waits for parent (e.g. qemu incoming paused) * source parent does sanlock_inquire * source parent sigstop child, sanlock_release, writes state to disk * dest parent reads state from disk, sanlock_acquire(child_pid, state.lver) * dest parent tells child to run (e.g. qemu incoming resumed) * dest child execs rw * source parent sigkill child */ static void write_migrate_incoming(char *state_in) { char target_str[32]; char state[1024]; char *wbuf, **p_wbuf; int fd, rv; int offset = 4096; int target; target = (our_hostid % max_hostid) + 1; memset(state, 0, sizeof(state)); memset(target_str, 0, sizeof(target_str)); sprintf(target_str, " target=%d", target); strcat(state, state_in); strcat(state, target_str); if (strlen(state) > 512) { printf("state string too long\n"); goto fail; } fd = open(count_path, O_RDWR | O_DIRECT | O_SYNC, 0); if (fd < 0) { perror("open failed"); goto fail; } rv = ioctl(fd, BLKFLSBUF); if (rv) { perror("BLKFLSBUF failed"); goto fail; } p_wbuf = &wbuf; rv = posix_memalign((void *)p_wbuf, getpagesize(), 512); if (rv) { perror("posix_memalign failed"); goto fail; } memset(wbuf, 0, 512); memcpy(wbuf, state, strlen(state)); lseek(fd, offset, SEEK_SET); rv = write(fd, wbuf, 512); if (rv != 512) { perror("write failed"); goto fail; } /* printf("write_migrate_incoming \"%s\"\n", wbuf); */ close(fd); return; fail: printf("write_migrate %d failed %s\n", offset, state); sleep(10000000); } /* read incoming block until it's set and our_hostid is next */ static int wait_migrate_incoming(uint64_t *lver) { struct sanlk_resource *res; char *rbuf, **p_rbuf, *wbuf, **p_wbuf; char *target_str, *val_str; int fd, rv, val; int offset = 4096; fd = open(count_path, O_RDWR | O_DIRECT | O_SYNC, 0); if (fd < 0) { perror("open failed"); goto fail; } rv = ioctl(fd, BLKFLSBUF); if (rv) { perror("BLKFLSBUF failed"); goto fail; } p_rbuf = &rbuf; p_wbuf = &wbuf; rv = posix_memalign((void *)p_rbuf, getpagesize(), 512); if (rv) { perror("posix_memalign failed"); goto fail; } rv = posix_memalign((void *)p_wbuf, getpagesize(), 512); if (rv) { perror("posix_memalign failed"); goto fail; } retry: lseek(fd, offset, SEEK_SET); rv = read(fd, rbuf, 512); if (rv != 512) { perror("read failed"); goto fail; } rbuf[511] = '\0'; /* init case to get things going */ if (!rbuf[0] && our_hostid == 1) { *lver = 0; return 1; } target_str = strstr(rbuf, " target="); if (!target_str) { goto retry; } val_str = strstr(target_str, "=") + 1; if (!val_str) { goto retry; } val = atoi(val_str); if (val != our_hostid) { goto retry; } /* printf("wait_migrate_incoming \"%s\"\n", rbuf); */ *target_str = '\0'; rv = sanlock_str_to_res(rbuf, &res); if (rv < 0) { printf("str_to_res error %d\n", rv); goto fail; } *lver = res->lver; free(res); /* strcpy(state_out, rbuf); */ memset(wbuf, 0, 512); sprintf(wbuf, "%s", "empty"); lseek(fd, offset, SEEK_SET); rv = write(fd, wbuf, 512); if (rv != 512) { perror("write failed"); goto fail; } close(fd); return 0; fail: printf("wait_migrate_incoming failed\n"); sleep(10000000); return -1; } #define MAX_MIGRATE_STATE 512 /* keep in one block for simplicity */ static int do_migrate(int argc, char *argv[]) { char *av[MIGRATE_ARGS+1]; struct sanlk_resource *res; int i, j, pid, rv, sock, len, init; int p2c[2]; /* parent to child */ int c2p[2]; /* child to parent */ int res_count; uint32_t parent_pid = getpid(); uint64_t lver; char *state; if (argc < MIGRATE_ARGS) return -1; count_offset = 0; strcpy(lock_path, argv[2]); strcpy(count_path, argv[4]); our_hostid = atoi(argv[7]); max_hostid = atoi(argv[8]); add_lockspace(); len = sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk); res = malloc(len); memset(res, 0, len); strcpy(res->lockspace_name, lockspace.name); snprintf(res->name, SANLK_NAME_LEN, "resource%s", count_path); res->name[SANLK_NAME_LEN-1] = '\0'; res->num_disks = 1; strncpy(res->disks[0].path, lock_path, SANLK_PATH_LEN); res->disks[0].path[SANLK_PATH_LEN-1] = '\0'; res->disks[0].offset = LEASE_SIZE; /* * argv[0] = devcount * argv[1] = migrate * argv[2] = * argv[3] = rw * start copying at argv[3] */ j = 0; av[j++] = strdup(argv[0]); for (i = 3; i < MIGRATE_ARGS; i++) av[j++] = strdup(argv[i]); av[j] = NULL; while (1) { pipe(p2c); pipe(c2p); pid = fork(); if (!pid) { int child_pid = getpid(); char junk; sock = sanlock_register(); if (sock < 0) { log_error("%s c %d sanlock_register error %d", count_path, child_pid, sock); exit(-1); } log_debug("%s c %d wait", count_path, child_pid); read(p2c[0], &junk, 1); close(p2c[0]); close(p2c[1]); log_debug("%s c %d begin", count_path, child_pid); /* make child's stderr go to parent c2p[0] */ close(2); dup(c2p[1]); close(c2p[0]); close(c2p[1]); execv(av[0], av); perror("execv devcount problem"); exit(EXIT_FAILURE); } init = wait_migrate_incoming(&lver); /* from source */ if (init) { res->flags = 0; res->lver = 0; } else { res->flags = SANLK_RES_LVER; res->lver = lver; } rv = sanlock_acquire(-1, pid, 0, 1, &res, NULL); if (rv < 0) { log_error("%s p %d sanlock_acquire c %d error %d", count_path, parent_pid, pid, rv); exit(0); } log_debug("%s p %d sanlock_acquire c %d init %d lver %llu done", count_path, parent_pid, pid, init, (unsigned long long)lver); /* tell child to resume */ write(p2c[1], "\n", 1); close(p2c[0]); close(p2c[1]); /* let the child run for 10 seconds before stopping it; if the child exits before the 10 seconds, the sanlock_inquire call should return an error */ sleep(10); rv = sanlock_inquire(-1, pid, 0, &res_count, &state); if (rv < 0) { log_error("%s p %d sanlock_inquire c %d error %d", count_path, parent_pid, pid, rv); goto fail; } log_debug("%s p %d sanlock_inquire c %d done", count_path, parent_pid, pid); pause_pid(pid, c2p[0]); log_debug("%s p %d paused c %d", count_path, parent_pid, pid); rv = sanlock_release(-1, pid, SANLK_REL_ALL, 0, NULL); if (rv < 0) { log_error("%s p %d sanlock_release c %d error %d", count_path, parent_pid, pid, rv); goto fail; } log_debug("%s p %d sanlock_release c %d done", count_path, parent_pid, pid); write_migrate_incoming(state); /* to dest */ kill_pid(pid); log_debug("%s p %d killed c %d", count_path, parent_pid, pid); close(c2p[0]); close(c2p[1]); free(state); } fail: printf("test failed...\n"); sleep(10000000); return -1; } /* * dmsetup table /dev/bull/lock1 > /tmp/table-linear.txt * sed "s/linear/error/" /tmp/table-linear.txt > /tmp/table-error.txt * * dmsetup suspend /dev/bull/lock1 * dmsetup load /dev/bull/lock1 /tmp/table-error.txt * dmsetup resume /dev/bull/lock1 * * dmsetup suspend /dev/bull/lock1 * dmsetup load /dev/bull/lock1 /tmp/table-linear.txt * dmsetup resume /dev/bull/lock1 */ static void dmsetup_save_lock_disk(void) { char cmd[128]; sprintf(cmd, "./devcount-dmsetup save %s", lock_path); system(cmd); } static void dmsetup_error_lock_disk(void) { char cmd[128]; sprintf(cmd, "./devcount-dmsetup error %s", lock_path); system(cmd); } static void dmsetup_linear_lock_disk(void) { char cmd[128]; sprintf(cmd, "./devcount-dmsetup linear %s", lock_path); system(cmd); } int do_expire(int argc, char *argv[]) { char *av[COUNT_ARGS+1]; struct sanlk_resource *res; uint32_t parent_pid = getpid(); int i, j, pid, rv, sock, len, status; int c2p[2]; char result[5]; if (argc < LOCK_ARGS) return -1; count_offset = 0; strcpy(lock_path, argv[2]); strcpy(count_path, argv[4]); our_hostid = atoi(argv[7]); dmsetup_save_lock_disk(); add_lockspace(); len = sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk); res = malloc(len); memset(res, 0, len); strcpy(res->lockspace_name, lockspace.name); snprintf(res->name, SANLK_NAME_LEN, "resource%s", count_path); res->name[SANLK_NAME_LEN-1] = '\0'; res->num_disks = 1; strncpy(res->disks[0].path, lock_path, SANLK_PATH_LEN); res->disks[0].path[SANLK_PATH_LEN-1] = '\0'; res->disks[0].offset = LEASE_SIZE; /* * argv[0] = devcount * argv[1] = expire * argv[2] = * argv[3] = rw * start copying at argv[3] */ j = 0; av[j++] = strdup(argv[0]); for (i = 3; i < LOCK_ARGS; i++) av[j++] = strdup(argv[i]); av[j] = NULL; while (1) { pipe(c2p); pid = fork(); if (!pid) { int child_pid = getpid(); sock = sanlock_register(); if (sock < 0) { log_error("%s c %d sanlock_register error %d", count_path, child_pid, sock); exit(-1); } /* this acquire can take up to 90 seconds waiting for the host_id of the owner to time out */ log_debug("%s c %d sanlock_acquire begin", count_path, child_pid); rv = sanlock_acquire(sock, -1, 0, 1, &res, NULL); if (rv < 0) { log_debug("%s c %d sanlock_acquire error %d", count_path, child_pid, rv); /* all hosts are trying to acquire so we expect this to acquire only sometimes; TODO: exit with an error for some rv's */ write(c2p[1], "fail", 4); close(c2p[0]); close(c2p[1]); exit(0); } log_debug("%s c %d sanlock_acquire done", count_path, child_pid); write(c2p[1], "good", 4); close(c2p[0]); close(c2p[1]); execv(av[0], av); perror("execv devcount problem"); exit(EXIT_FAILURE); } memset(&result, 0, sizeof(result)); read(c2p[0], &result, 4); close(c2p[0]); close(c2p[1]); if (strstr(result, "fail")) { /* we expect child to exit when it fails to acquire the lock because it's held by someone else */ waitpid(pid, &status, 0); goto dead_child; } /* this test should be run with sec2 set to some large value that won't run out before sanlock daemon kills rw */ sleep(rand_int(6, 100)); dmsetup_error_lock_disk(); log_debug("%s p %d disable %s", count_path, parent_pid, lock_path); /* sanlock daemon kills pid when the renewals fail; after the kill it will try to release the resource lease, which will also fail if the resource lease is on the same disk as the host_id lease. Other nodes trying to get pid's resource lease are watching our host_id for 90 seconds, after which they will take pid's resource lease. If the resource lease is on a different disk, the daemon will be able to release it after the kill, and another node will be able to take it immediately after that, without watching our host_id for 90 seconds */ /* other nodes can't rely on the daemon being able to kill rw, so they need to wait 90 seconds to ensure that the watchdog has killed the host before taking pid's resource lease. In a different test, have the daemon kill fail, causing rw to continue running until the watchdog fires, after which another host will take pid's resource lease */ waitpid(pid, &status, 0); log_debug("%s p %d waitpid c %d done", count_path, parent_pid, pid); sleep(rand_int(0, 3)); dmsetup_linear_lock_disk(); log_debug("%s p %d enable %s", count_path, parent_pid, lock_path); log_debug("%s p %d sanlock_add_lockspace begin", lock_path, parent_pid); while (1) { sleep(1); rv = add_lockspace(); if (!rv) break; } dead_child: sleep(rand_int(0, 1)); } printf("test failed...\n"); sleep(1000000); return -1; } /* * devcount init * sanlock direct init -n 8 -s devcount:0::0 * sanlock direct init -n 8 -r devcount:resource::LEASE_SIZE * dd if=/dev/zero of= bs=512 count=24 */ #define INIT_NUM_HOSTS 0 int do_init(int argc, char *argv[]) { char resbuf[sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk)]; struct sanlk_disk disk; struct sanlk_resource *res; struct sanlk_lockspace ls; char command[4096]; int rv, align_size; if (argc < 4) return -1; strcpy(count_path, argv[3]); #if 0 /* initialize host_id lease area at offset 0 */ memset(command, 0, sizeof(command)); snprintf(command, sizeof(command), "sanlock direct init -s devcount:0:%s:0", argv[2]); printf("%s\n", command); system(command); /* initialize first resource lease area at offset LEASE_SIZE */ memset(command, 0, sizeof(command)); snprintf(command, sizeof(command), "sanlock direct init -r devcount:resource%s:%s:%d", argv[3], argv[2], LEASE_SIZE); printf("%s\n", command); system(command); #else memset(&disk, 0, sizeof(disk)); strcpy(disk.path, argv[2]); align_size = sanlock_direct_align(&disk); if (align_size != LEASE_SIZE) { printf("sanlock_direct align %s error %d\n", disk.path, align_size); return -1; } memset(&ls, 0, sizeof(ls)); strcpy(ls.name, "devcount"); strcpy(ls.host_id_disk.path, argv[2]); printf("init sync\n"); rv = sanlock_direct_init(&ls, NULL, 0, INIT_NUM_HOSTS, 0); if (rv < 0) { printf("sanlock_direct_init lockspace error %d\n", rv); return -1; } printf("init async\n"); rv = sanlock_direct_init(&ls, NULL, 0, INIT_NUM_HOSTS, 1); if (rv < 0) { printf("sanlock_direct_init lockspace error %d\n", rv); return -1; } memset(resbuf, 0, sizeof(resbuf)); res = (struct sanlk_resource *)&resbuf; strcpy(res->lockspace_name, "devcount"); sprintf(res->name, "resource%s", argv[3]); res->num_disks = 1; strcpy(res->disks[0].path, argv[2]); res->disks[0].offset = LEASE_SIZE; rv = sanlock_direct_init(NULL, res, 0, INIT_NUM_HOSTS, 0); if (rv < 0) { printf("sanlock_direct_init resource error %d\n", rv); return -1; } #endif memset(command, 0, sizeof(command)); snprintf(command, sizeof(command), "dd if=/dev/zero of=%s bs=512 count=24", count_path); printf("%s\n", command); system(command); return 0; } int main(int argc, char *argv[]) { int rv; if (argc < 2) goto out; if (!strcmp(argv[1], "init")) rv = do_init(argc, argv); else if (!strcmp(argv[1], "rw") || !strcmp(argv[1], "wr")) rv = do_count(argc, argv); else if (!strcmp(argv[1], "rwsig")) { setup_sigterm(); argv[1] = "rw"; rv = do_count(argc, argv); } else if (!strcmp(argv[1], "lock")) rv = do_lock(argc, argv); else if (!strcmp(argv[1], "wrap")) rv = do_wrap(argc, argv); else if (!strcmp(argv[1], "relock")) rv = do_relock(argc, argv); else if (!strcmp(argv[1], "migrate")) rv = do_migrate(argc, argv); else if (!strcmp(argv[1], "expire")) rv = do_expire(argc, argv); if (!rv) return 0; out: /* * sanlock direct init -s devcount:0:/dev/bull/leases:0 * sanlock direct init -r devcount:resource/dev/bull/count:/dev/bull/leases:LEASE_SIZE * * host_id leases exists at offset 0 * first resource lease exists at offset LEASE_SIZE */ printf("devcount init \n"); printf(" sanlock direct init -s devcount:0::0\n"); printf(" sanlock direct init -r devcount:resource::LEASE_SIZE\n"); printf(" dd if=/dev/zero of= bs=512 count=24\n"); printf("\n"); printf("devcount rw \n"); printf(" rw: read count for sec1, looking for writes, then write for sec2\n"); printf(" wr: write count for sec1, then read for sec2, looking for writes\n"); printf("\n"); printf("devcount lock rw \n"); printf(" sanlock add_lockspace -s devcount:::0\n"); printf(" loop around fork, sanlock_acquire, exec devcount rw\n"); printf("\n"); printf("devcount relock rw \n"); printf(" sanlock add_lockspace -s devcount:::0\n"); printf(" loop around fork, sanlock_acquire, exec devcount rw\n"); printf(" sigstop child, inquire, release, re-acquire, sigcont|sigkill\n"); printf("\n"); printf("devcount wrap rw \n"); printf(" sanlock add_lockspace -s devcount:::0\n"); printf(" sanlock_acquire, exec devcount rw\n"); printf("\n"); printf("devcount migrate rw \n"); printf(" sanlock add_lockspace -s devcount:::0\n"); printf(" loop around fork, sanlock_acquire, exec devcount rw\n"); printf("\n"); printf("devcount expire rw \n"); printf("\n"); return -1; } sanlock/tests/devcountn000077500000000000000000000043721516326262600156240ustar00rootroot00000000000000#!/bin/bash if [ $# -le 3 ]; then echo "" echo "Start N devcount commands" echo "" echo "devcountn N init LOCKDEV_BASE COUNTDEV_BASE" echo "devcountn N rw COUNTDEV_BASE SEC1 SEC2 HOSTID" echo "devcountn N lock LOCKDEV_BASE rw COUNTDEV_BASE SEC1 SEC2 HOSTID" echo "devcountn N relock LOCKDEV_BASE rw COUNTDEV_BASE SEC1 SEC2 HOSTID" echo "devcountn N wrap LOCKDEV_BASE rw COUNTDEV_BASE SEC1 SEC2 HOSTID" echo "devcountn N migrate LOCKDEV_BASE rw COUNTDEV_BASE SEC1 SEC2 HOSTID MAXID" echo "devcountn N expire LOCKDEV_BASE rw COUNTDEV_BASE SEC1 SEC2 HOSTID" echo "" echo "devcount LOCKDEV1 rw COUNTDEV1 ..." echo "devcount LOCKDEV2 rw COUNTDEV2 ..." echo "devcount LOCKDEV3 rw COUNTDEV3 ..." echo ... echo "devcount LOCKDEVN rw COUNTDEVN ..." echo "" echo "Examples" echo "" echo "devcountn 3 init /dev/lock /dev/count" echo " devcount init /dev/lock1 /dev/count1" echo " devcount init /dev/lock2 /dev/count2" echo " devcount init /dev/lock3 /dev/count3" echo "" echo "devcountn 3 rw /dev/count 5 5 1" echo " devcount rw /dev/count1 5 5 1" echo " devcount rw /dev/count2 5 5 1" echo " devcount rw /dev/count3 5 5 1" echo "" echo "devcountn 3 lock /dev/lock rw /dev/count 5 5 1" echo " sanlock add_lockspace -s devcount:1:/dev/lock1:0" echo " (the add_lockspace command from each subsequent devcount will fail)" echo " devcount lock /dev/lock1 rw /dev/count1 5 5 1" echo " devcount lock /dev/lock2 rw /dev/count2 5 5 1" echo " devcount lock /dev/lock3 rw /dev/count3 5 5 1" echo "" exit 0 fi num=$1 cmd1=$2 if [ "$cmd1" != "init" ]; then deva=$3 cmd2=$4 devb=$5 sec1=$6 sec2=$7 hostid=$8 maxid=$9 i=1 echo sanlock add_lockspace -s devcount:$hostid:$deva$i:0 sanlock add_lockspace -s devcount:$hostid:$deva$i:0 fi end=`expr $num - 1` for i in `seq 0 $end`; do if [ "$cmd1" == "init" ]; then deva=$3 devb=$4 echo ./devcount init $deva$i $devb$i ./devcount init $deva$i $devb$i elif [ "$cmd1" == "rw" ] || [ "$cmd1" == "wr" ]; then echo ./devcount $cmd1 $deva$i $sec1 $sec2 $hostid ./devcount $cmd1 $deva$i $sec1 $sec2 $hostid & else echo ./devcount $cmd1 $deva$i $cmd2 $devb$i $sec1 $sec2 $hostid $maxid ./devcount $cmd1 $deva$i $cmd2 $devb$i $sec1 $sec2 $hostid $maxid & fi done sanlock/tests/direct_test.py000066400000000000000000000101351516326262600165460ustar00rootroot00000000000000# Copyright (C) 2019 Red Hat, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. """ Test sanlock direct options. """ from __future__ import absolute_import import io import os import struct from . import constants from . import util from . units import MiB def test_init_lockspace(tmpdir): path = tmpdir.join("lockspace") size = MiB util.create_file(str(path), size) lockspace = "name:1:%s:0" % path util.sanlock("direct", "init", "-s", lockspace) with io.open(str(path), "rb") as f: magic, = struct.unpack("< I", f.read(4)) assert magic == constants.DELTA_DISK_MAGIC # TODO: check more stuff here... util.check_guard(str(path), size) def test_dump_lockspace_empty(tmpdir): path = tmpdir.join("lockspace") size = MiB util.create_file(str(path), size) lockspace = "name:1:%s:0" % path util.sanlock("direct", "init", "-s", lockspace) dump = "%s:0:1M" % path out = util.sanlock("direct", "dump", dump) lines = out.decode("utf-8").splitlines() spaces = [line.split() for line in lines] # Empty lockspace has no hosts. assert spaces == [ ['offset', 'lockspace', 'resource', 'timestamp', 'own', 'gen', 'lver'] ] def test_init_resource(tmpdir): path = tmpdir.join("resources") size = MiB util.create_file(str(path), size) resource = "ls_name:res_name:%s:0" % path util.sanlock("direct", "init", "-r", resource) with io.open(str(path), "rb") as f: magic, = struct.unpack("< I", f.read(4)) assert magic == constants.PAXOS_DISK_MAGIC # TODO: check more stuff here... util.check_guard(str(path), size) def test_dump_resources(tmpdir): path = tmpdir.join("resources") size = 8 * MiB util.create_file(str(path), size) # Write 2 resources with a hole between them. for i in [0, 2]: res = "ls_name:res_%d:%s:%dM" % (i, path, i) util.sanlock("direct", "init", "-r", res) dump = "%s:0:8M" % path out = util.sanlock("direct", "dump", dump) lines = out.decode("utf-8").splitlines() resources = [line.split() for line in lines] assert resources == [ ['offset', 'lockspace', 'resource', 'timestamp', 'own', 'gen', 'lver'], ['00000000', 'ls_name', 'res_0', '0000000000', '0000', '0000', '0'], ['02097152', 'ls_name', 'res_2', '0000000000', '0000', '0000', '0'], ] def test_dump_resources_start_before(tmpdir): path = tmpdir.join("resources") size = 8 * MiB util.create_file(str(path), size) # Write 2 resources at middle. for i in [4, 5]: res = "ls_name:res_%d:%s:%dM" % (i, path, i) util.sanlock("direct", "init", "-r", res) dump = "%s:2M:8M" % path out = util.sanlock("direct", "dump", dump) lines = out.decode("utf-8").splitlines() resources = [line.split() for line in lines] assert resources == [ ['offset', 'lockspace', 'resource', 'timestamp', 'own', 'gen', 'lver'], ['04194304', 'ls_name', 'res_4', '0000000000', '0000', '0000', '0'], ['05242880', 'ls_name', 'res_5', '0000000000', '0000', '0000', '0'], ] def test_path_with_colon(tmpdir): path = str(tmpdir.mkdir("with:colon").join("resources")) size = 8 * MiB util.create_file(path, size) # sanlock direct init does not support escaped colons in path. dirname, filename = os.path.split(path) res = "ls_name:res_0:%s:0M" % filename util.sanlock("direct", "init", "-r", res, cwd=dirname) # sanlock direct dump supports escaped colons in path. escaped_path = path.replace(":", "\\:") dump = "%s:0:8M" % escaped_path out = util.sanlock("direct", "dump", dump) lines = out.decode("utf-8").splitlines() resources = [line.split() for line in lines] assert resources == [ ['offset', 'lockspace', 'resource', 'timestamp', 'own', 'gen', 'lver'], ['00000000', 'ls_name', 'res_0', '0000000000', '0000', '0000', '0'], ] sanlock/tests/killpath.c000066400000000000000000000030671516326262600156450ustar00rootroot00000000000000#include #include #include #include #include #include #include #include #include #include "sanlock.h" #include "sanlock_resource.h" int main(int argc, char *argv[]) { FILE *out; char *cmd = argv[0]; char args[1024]; char arg[1024]; char *state = NULL; int count = 0; int i, pid, rv; if (argc > 1 && !strcmp(argv[1], "-h")) { printf("%s_args - syslog args\n", cmd); printf("%s_term - kill SIGTERM\n", cmd); printf("%s_pause - sanlock_inquire, sanlock_release\n", cmd); } openlog(cmd, LOG_PID, LOG_DAEMON); memset(args, 0, sizeof(args)); for (i = 1; i < argc; i++) { memset(arg, 0, sizeof(arg)); sprintf(arg, "%s ", argv[i]); strcat(args, arg); } pid = atoi(argv[argc-1]); if (strstr(cmd, "args")) { syslog(LOG_ERR, "pid %d args %s\n", pid, args); } else if (strstr(cmd, "term")) { rv = kill(pid, SIGTERM); syslog(LOG_ERR, "sigterm pid %d errno %d\n", pid, errno); } else if (strstr(cmd, "pause")) { rv = kill(pid, SIGSTOP); if (rv < 0) syslog(LOG_ERR, "sigstop pid %d errno %d", pid, errno); rv = sanlock_inquire(-1, pid, 0, &count, &state); syslog(LOG_ERR, "inquire pid %d rv %d count %d state %s\n", pid, rv, count, state ? state : ""); rv = sanlock_release(-1, pid, SANLK_REL_ALL, 0, NULL); syslog(LOG_ERR, "release pid %d rv %d\n", pid, rv); out = fopen("/tmp/client-state.txt", "a"); if (out) { fprintf(out, "%d %s\n", pid, state); fclose(out); } if (state) free(state); } return 0; } sanlock/tests/sanlk_client.c000066400000000000000000000037431516326262600165040ustar00rootroot00000000000000#include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock.h" #include "sanlock_resource.h" int main(int argc, char *argv[]) { char rd[sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk)]; char path[SANLK_HELPER_PATH_LEN]; char args[SANLK_HELPER_PATH_LEN]; struct sanlk_resource *res; time_t now, last; int sock, rv, i; if (argc < 6) { printf("sanlk_client \n"); return -1; } memset(rd, 0, sizeof(rd)); memset(path, 0, sizeof(path)); memset(args, 0, sizeof(args)); res = (struct sanlk_resource *)&rd; strcpy(res->lockspace_name, argv[1]); strcpy(res->name, argv[2]); res->num_disks = 1; strcpy(res->disks[0].path, argv[3]); res->disks[0].offset = atoi(argv[4]); strcpy(path, argv[5]); if (argc > 6) { for (i = 6; i < argc; i++) { strcat(args, argv[i]); strcat(args, " "); } } sock = sanlock_register(); if (sock < 0) { fprintf(stderr, "register error %d\n", sock); return -1; } if (!strcmp(path, "none")) goto acquire; rv = sanlock_killpath(sock, SANLK_KILLPATH_PID, path, args); if (rv < 0) { fprintf(stderr, "killpath error %d\n", rv); return -1; } acquire: rv = sanlock_acquire(sock, -1, 0, 1, &res, NULL); if (rv < 0) { fprintf(stderr, "acquire error %d\n", rv); return -1; } rv = sanlock_restrict(sock, SANLK_RESTRICT_ALL); if (rv < 0) { fprintf(stderr, "restrict error %d\n", rv); return -1; } printf("%d running\n", getpid()); last = time(NULL); while (1) { now = time(NULL); if (now - last > 2) printf("%d running (paused %llu sec)\n", getpid(), (unsigned long long)(now - last)); last = now; sleep(1); } } sanlock/tests/sanlk_events.c000066400000000000000000000041201516326262600165200ustar00rootroot00000000000000#include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock.h" #include "sanlock_admin.h" #include "../src/sanlock_sock.h" static int prog_stop; static void sigterm_handler(int sig) { if (sig == SIGTERM) prog_stop = 1; } int main(int argc, char *argv[]) { struct sigaction act; struct sanlk_host_event he; struct pollfd pollfd; uint64_t from_host, from_gen; char *ls_name; int fd, rv; if (argc < 2) { printf("sanlk_events \n"); return -1; } memset(&act, 0, sizeof(act)); act.sa_handler = sigterm_handler; sigaction(SIGTERM, &act, NULL); ls_name = argv[1]; printf("reg_event %s\n", ls_name); fd = sanlock_reg_event(ls_name, &he, 0); if (fd < 0) { printf("reg error %d\n", fd); return -1; } memset(&pollfd, 0, sizeof(pollfd)); pollfd.fd = fd; pollfd.events = POLLIN; while (1) { rv = poll(&pollfd, 1, 1000); if (rv == -1 && errno == EINTR) continue; if (prog_stop) break; if (rv < 0) { printf("poll error %d\n", rv); break; } if (pollfd.revents & POLLIN) { while (1) { rv = sanlock_get_event(fd, 0, &he, &from_host, &from_gen); if (rv == -EAGAIN) { /* no more events */ break; } if (rv < 0) { printf("get_event error %d\n", rv); break; } printf("get_event host_id %llu generation %llu event 0x%llx data 0x%llx from %llu %llu\n", (unsigned long long)he.host_id, (unsigned long long)he.generation, (unsigned long long)he.event, (unsigned long long)he.data, (unsigned long long)from_host, (unsigned long long)from_gen); } } if (pollfd.revents & (POLLERR | POLLHUP | POLLNVAL)) { printf("poll revents %x\n", pollfd.revents); break; } } printf("end_event %s\n", ls_name); sanlock_end_event(fd, ls_name, 0); return 0; } sanlock/tests/sanlk_load.c000066400000000000000000000510141516326262600161370ustar00rootroot00000000000000#include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock.h" #include "sanlock_admin.h" #include "sanlock_resource.h" #include "sanlock_direct.h" #define ONEMB 1048576 #define LEASE_SIZE ONEMB #define MAX_LS_COUNT 64 #define MAX_RES_COUNT 512 #define MAX_PID_COUNT 256 #define DEFAULT_LS_COUNT 4 #define DEFAULT_RES_COUNT 4 #define DEFAULT_PID_COUNT 4 #define MAX_RV 300 #define IV -1 #define UN 0 #define SH 3 #define EX 5 int prog_stop; int debug = 0; int debug_verbose = 0; char error_buf[4096]; char lock_disk_base[PATH_MAX]; int lock_state[MAX_LS_COUNT][MAX_RES_COUNT]; int ls_count = DEFAULT_LS_COUNT; int res_count = DEFAULT_RES_COUNT; int pid_count = DEFAULT_PID_COUNT; int one_mode = 0; int our_hostid; int run_sec; int count_sec; int error_count; int error_range = 1; int acquire_rv[MAX_RV]; int release_rv[MAX_RV]; #define log_debug(fmt, args...) \ do { \ if (debug) printf("%lu " fmt "\n", time(NULL), ##args); \ } while (0) #define log_error(fmt, args...) \ do { \ memset(error_buf, 0, sizeof(error_buf)); \ snprintf(error_buf, 4095, "%ld " fmt "\n", time(NULL), ##args); \ printf("ERROR: %s\n", error_buf); \ syslog(LOG_ERR, "%s", error_buf); \ error_count++; \ } while (0) static void sigterm_handler(int sig) { if (sig == SIGTERM) prog_stop = 1; } static int get_rand(int a, int b) { return a + (int) (((float)(b - a + 1)) * random() / (RAND_MAX+1.0)); } static int get_rand_sh_ex(void) { unsigned int n; if (one_mode == SH) return SH; if (one_mode == EX) return EX; n = (unsigned int)random();; if (n % 2) return SH; return EX; } static void save_rv(int pid, int rv, int acquire) { if (rv > 0) goto fail; if (-rv > MAX_RV) goto fail; if (acquire) { if (!rv) acquire_rv[0]++; else acquire_rv[-rv]++; } else { if (!rv) release_rv[0]++; else release_rv[-rv]++; } if (error_range == 1) { switch (rv) { case 0: case -EBUSY: /* -16 */ case -EEXIST: /* -17 */ case -EAGAIN: /* -11 */ break; default: log_error("%d ERROR range %d save_rv %d %d", pid, error_range, rv, acquire); break; }; } else if (error_range == 2) { switch (rv) { case 0: case -EBUSY: /* -16 */ case -EEXIST: /* -17 */ case -EAGAIN: /* -11 */ break; case -243: case -244: case -245: break; default: log_error("%d ERROR range %d save_rv %d %d", pid, error_range, rv, acquire); break; }; } return; fail: log_error("%d save_rv %d %d", pid, rv, acquire); printf("%lu %d ERROR save_rv %d %d", time(NULL), pid, rv, acquire); } static void display_rv(int pid) { int i; printf("%lu %d results acquire ", time(NULL), pid); for (i = 0; i < MAX_RV; i++) { if (acquire_rv[i]) printf("%d:%d ", i, acquire_rv[i]); } printf("release "); for (i = 0; i < MAX_RV; i++) { if (release_rv[i]) printf("%d:%d ", i, release_rv[i]); } printf("\n"); } static void dump_lock_state(int pid) { int i, j; for (i = 0; i < ls_count; i++) { for (j = 0; j < res_count; j++) { if (!lock_state[i][j]) continue; log_error("%d lockspace%d:resource%d", pid, i, j); } } } static void dump_inquire_state(int pid, char *state) { char *p = state; int len = strlen(state); int i; if (!len) return; for (i = 0; i < len; i++) { if (state[i] == ' ') { state[i] = '\0'; if (!i) log_debug("%d leading space", pid); else log_debug("%d %s", pid, p); p = state + i + 1; } } log_debug("%d %s", pid, p); } static int check_lock_state(int pid, int result, int count, char *res_state) { char buf[128]; char *found = NULL; int found_count = 0; int none_count = 0; int bad_count = 0; int i, j; memset(buf, 0, sizeof(buf)); if (result < 0) goto fail; if (!count) { if (res_state) { log_error("%d check_lock_state zero count res_state %s", pid, res_state); } for (i = 0; i < ls_count; i++) { for (j = 0; j < res_count; j++) { if (lock_state[i][j]) { bad_count++; log_error("%d check_lock_state zero count %d %d lock", pid, i, j); } } } if (bad_count) goto fail; return 0; } for (i = 0; i < ls_count; i++) { for (j = 0; j < res_count; j++) { memset(buf, 0, sizeof(buf)); sprintf(buf, "lockspace%d:resource%d:", i, j); found = strstr(res_state, buf); if (found && lock_state[i][j]) { found_count++; } else if (!found && !lock_state[i][j]) { none_count++; } else { bad_count++; log_error("%d check_lock_state %s lock_state %d res_state %s", pid, buf, lock_state[i][j], res_state); } } } if ((found_count != count) || bad_count) goto fail; return 0; fail: log_error("%d check_lock_state result %d count %d res_state %s", pid, result, count, res_state); log_error("%d check_lock_state found %d none %d bad %d", pid, found_count, none_count, bad_count); dump_lock_state(pid); printf("%lu %d ERROR check_lock_state result %d count %d found %d bad %d res_state %s", time(NULL), pid, result, count, found_count, bad_count, res_state); return -1; } #if 0 static int remove_lockspace(int i) { struct sanlk_lockspace ls; int rv; memset(&ls, 0, sizeof(ls)); sprintf(ls.host_id_disk.path, "%s%d", lock_disk_base, i); sprintf(ls.name, "lockspace%d", i); ls.host_id = our_hostid; printf("rem lockspace%d...\n", i); rv = sanlock_rem_lockspace(&ls, 0); if (rv < 0) { log_error("sanlock_rem_lockspace error %d %s", rv, ls.host_id_disk.path); return -1; } printf("rem done\n"); return 0; } #endif static int add_lockspace(int i) { struct sanlk_lockspace ls; int rv; int async = !(i % 2); uint32_t flags = 0; memset(&ls, 0, sizeof(ls)); sprintf(ls.host_id_disk.path, "%s%d", lock_disk_base, i); sprintf(ls.name, "lockspace%d", i); ls.host_id = our_hostid; if (async) flags = SANLK_ADD_ASYNC; printf("add lockspace%d...\n", i); rv = sanlock_add_lockspace(&ls, flags); if (rv == -EEXIST) return 0; if (rv < 0) { log_error("sanlock_add_lockspace error %d %s", rv, ls.host_id_disk.path); return -1; } if (!async) goto out; while (1) { rv = sanlock_inq_lockspace(&ls, 0); if (!rv) goto out; if (rv == -EINPROGRESS) { sleep(2); continue; } log_error("sanlock_inq_lockspace error %d", rv); return -1; } out: printf("add done\n"); return 0; } static int add_lockspaces(void) { int i, rv; for (i = 0; i < ls_count; i++) { rv = add_lockspace(i); if (rv < 0) return rv; } return 0; } static const char *mode_str(int n) { if (n == SH) return "sh"; if (n == EX) return "ex"; if (n == UN) return "un"; if (n == IV) return "iv"; return "er"; } static int do_one(int pid, int fd, int _s1, int _r1, int _n1, int *full) { char buf1[sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk)]; struct sanlk_resource *r1; int acquire = (_n1 != UN); int rv; memset(buf1, 0, sizeof(buf1)); r1 = (struct sanlk_resource *)&buf1; sprintf(r1->lockspace_name, "lockspace%d", _s1); sprintf(r1->name, "resource%d", _r1); sprintf(r1->disks[0].path, "%s%d", lock_disk_base, _s1); r1->disks[0].offset = (_r1+1)*LEASE_SIZE; r1->num_disks = 1; if (_n1 == SH) r1->flags |= SANLK_RES_SHARED; if (acquire) { rv = sanlock_acquire(fd, -1, 0, 1, &r1, NULL); if (rv == -E2BIG || rv == -ENOENT) *full = 1; } else { rv = sanlock_release(fd, -1, 0, 1, &r1); } log_debug("%d %s %d,%d %s = %d", pid, acquire ? "acquire" : "release", _s1, _r1, mode_str(_n1), rv); save_rv(pid, rv, acquire); return rv; } static int do_two(int pid, int fd, int _s1, int _r1, int _n1, int _s2, int _r2, int _n2, int *full) { char buf1[sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk)]; char buf2[sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk)]; struct sanlk_resource *r1; struct sanlk_resource *r2; struct sanlk_resource **res_args; int acquire = (_n1 != UN); int rv; res_args = malloc(2 * sizeof(struct sanlk_resource *)); if (!res_args) return -ENOMEM; memset(buf1, 0, sizeof(buf1)); memset(buf2, 0, sizeof(buf2)); r1 = (struct sanlk_resource *)&buf1; r2 = (struct sanlk_resource *)&buf2; res_args[0] = r1; res_args[1] = r2; sprintf(r1->lockspace_name, "lockspace%d", _s1); sprintf(r1->name, "resource%d", _r1); sprintf(r1->disks[0].path, "%s%d", lock_disk_base, _s1); r1->disks[0].offset = (_r1+1)*LEASE_SIZE; r1->num_disks = 1; if (_n1 == SH) r1->flags |= SANLK_RES_SHARED; sprintf(r2->lockspace_name, "lockspace%d", _s2); sprintf(r2->name, "resource%d", _r2); sprintf(r2->disks[0].path, "%s%d", lock_disk_base, _s2); r2->disks[0].offset = (_r2+1)*LEASE_SIZE; r2->num_disks = 1; if (_n2 == SH) r2->flags |= SANLK_RES_SHARED; if (acquire) { rv = sanlock_acquire(fd, -1, 0, 2, res_args, NULL); if (rv == -E2BIG || rv == -ENOENT) *full = 1; } else { rv = sanlock_release(fd, -1, 0, 2, res_args); } log_debug("%d %s %d,%d %s %d,%d %s = %d", pid, acquire ? "acquire" : "release", _s1, _r1, mode_str(_n1), _s2, _r2, mode_str(_n2), rv); save_rv(pid, rv, acquire); free(res_args); return rv; } static int acquire_one(int pid, int fd, int s1, int r1, int n1, int *full) { return do_one(pid, fd, s1, r1, n1, full); } static int acquire_two(int pid, int fd, int s1, int r1, int n1, int s2, int r2, int n2, int *full) { return do_two(pid, fd, s1, r1, n1, s2, r2, n2, full); } static int release_one(int pid, int fd, int s1, int r1) { return do_one(pid, fd, s1, r1, UN, NULL); } static int release_two(int pid, int fd, int s1, int r1, int s2, int r2) { return do_two(pid, fd, s1, r1, UN, s2, r2, UN, NULL); } static int release_all(int pid, int fd) { int rv; rv = sanlock_release(fd, -1, SANLK_REL_ALL, 0, NULL); log_debug("%d release all = %d", pid, rv); save_rv(pid, rv, 0); return rv; } static void inquire_all(int pid, int fd) { int rv, count = 0; char *state = NULL; if (prog_stop) return; rv = sanlock_inquire(fd, -1, 0, &count, &state); log_debug("%d inquire all = %d %d", pid, rv, count); if (prog_stop) return; check_lock_state(pid, rv, count, state); if (count && debug_verbose) dump_inquire_state(pid, state); if (state) free(state); } int do_rand_child(void) { int s1, s2, r1, r2, m1, m2, n1, n2, full; int fd, rv; int iter = 1; int pid = getpid(); error_count = 0; srandom(pid); memset(lock_state, 0, sizeof(lock_state)); fd = sanlock_register(); if (fd < 0) { log_error("%d sanlock_register error %d", pid, fd); exit(-1); } while (!prog_stop) { s1 = get_rand(0, ls_count-1); r1 = get_rand(0, res_count-1); m1 = lock_state[s1][r1]; s2 = -1; r2 = -1; m2 = IV; if (get_rand(1, 3) == 2) { s2 = get_rand(0, ls_count-1); r2 = get_rand(0, res_count-1); m2 = lock_state[s2][r2]; if (s1 == s2 && r1 == r2) { s2 = -1; r2 = -1; m2 = IV; } } full = 0; if (m1 == UN && m2 == UN) { /* both picks are unlocked, lock both together */ n1 = get_rand_sh_ex(); n2 = get_rand_sh_ex(); rv = acquire_two(pid, fd, s1, r1, n1, s2, r2, n2, &full); if (!rv) { lock_state[s1][r1] = n1; lock_state[s2][r2] = n2; } m1 = IV; m2 = IV; } if (m1 > UN && m2 > UN) { /* both picks are locked, unlock both together */ release_two(pid, fd, s1, r1, s2, r2); lock_state[s1][r1] = UN; lock_state[s2][r2] = UN; m1 = IV; m2 = IV; } if (m1 == UN) { n1 = get_rand_sh_ex(); rv = acquire_one(pid, fd, s1, r1, n1, &full); if (!rv) lock_state[s1][r1] = n1; } if (m2 == UN) { n2 = get_rand_sh_ex(); rv = acquire_one(pid, fd, s2, r2, n2, &full); if (!rv) lock_state[s2][r2] = n2; } if (m1 > UN) { release_one(pid, fd, s1, r1); lock_state[s1][r1] = UN; } if (m2 > UN) { release_one(pid, fd, s2, r2); lock_state[s2][r2] = UN; } if (full) { release_all(pid, fd); memset(lock_state, 0, sizeof(lock_state)); } if ((iter % 10) == 0) { display_rv(pid); inquire_all(pid, fd); } iter++; } display_rv(pid); if (error_count) { printf("pid %d done error_count %d\n", pid, error_count); exit(EXIT_FAILURE); } printf("pid %d done\n", pid); exit(EXIT_SUCCESS); } int do_all_child(void) { int sx, rx, full; int fd, rv; int pid = getpid(); srandom(pid); memset(lock_state, 0, sizeof(lock_state)); fd = sanlock_register(); if (fd < 0) { log_error("%d sanlock_register error %d", pid, fd); exit(-1); } while (!prog_stop) { for (sx = 0; sx < ls_count; sx++) { for (rx = 0; rx < res_count; rx++) { rv = acquire_one(pid, fd, sx, rx, EX, &full); if (!rv) lock_state[sx][rx] = EX; } inquire_all(pid, fd); for (rx = 0; rx < res_count-1; rx++) { rv = release_one(pid, fd, sx, rx); lock_state[sx][rx] = UN; } inquire_all(pid, fd); } } display_rv(pid); return 0; } /* * sanlk_load rand -i [-D -s -r -p ] */ void get_options(int argc, char *argv[]) { char optchar; char *optionarg; char *p; int i = 3; for (; i < argc; ) { p = argv[i]; if ((p[0] != '-') || (strlen(p) != 2)) { log_error("unknown option %s", p); log_error("space required before option value"); exit(EXIT_FAILURE); } optchar = p[1]; i++; if (optchar == 'D') { debug = 1; continue; } if (optchar == 'V') { debug_verbose = 1; continue; } if (i >= argc) { log_error("option '%c' requires arg", optchar); exit(EXIT_FAILURE); } optionarg = argv[i]; switch (optchar) { case 'i': our_hostid = atoi(optionarg); break; case 'S': run_sec = atoi(optionarg); break; case 's': ls_count = atoi(optionarg); if (ls_count > MAX_LS_COUNT) { log_error("max ls_count %d", MAX_LS_COUNT); exit(-1); } break; case 'r': res_count = atoi(optionarg); if (res_count > MAX_RES_COUNT) { log_error("max res_count %d", MAX_RES_COUNT); exit(-1); } break; case 'p': pid_count = atoi(optionarg); if (pid_count > MAX_PID_COUNT) { log_error("max pid_count %d", MAX_PID_COUNT); exit(-1); } break; case 'm': one_mode = atoi(optionarg); break; case 'e': error_range = atoi(optionarg); break; default: log_error("unknown option: %c", optchar); exit(EXIT_FAILURE); } i++; } } int find_pid(int *kids, int pid) { int i; for (i = 0; i < pid_count; i++) { if (kids[i] == pid) return i; } return -1; } int do_rand(int argc, char *argv[]) { struct sigaction act; int children[MAX_PID_COUNT]; int run_count = 0; int i, rv, pid, status; if (argc < 5) return -1; memset(&act, 0, sizeof(act)); act.sa_handler = sigterm_handler; sigaction(SIGTERM, &act, NULL); strcpy(lock_disk_base, argv[2]); get_options(argc, argv); rv = add_lockspaces(); if (rv < 0) return rv; printf("forking %d pids\n", pid_count); for (i = 0; i < pid_count; i++) { pid = fork(); if (pid < 0) { log_error("fork %d failed %d run_count %d", i, errno, run_count); break; } if (!pid) { do_rand_child(); exit(-1); } children[i] = pid; run_count++; } printf("children running\n"); while (!prog_stop) { #if 0 /* * kill and replace a random pid */ sleep(get_rand(1, 60)); if (prog_stop) break; i = get_rand(0, pid_count); pid = children[i]; printf("kill pid %d\n", pid); kill(pid, SIGKILL); rv = waitpid(pid, &status, 0); if (rv <= 0) continue; pid = fork(); if (pid < 0) { log_error("fork failed %d", errno); break; } else if (!pid) { do_rand_child(); exit(-1); } else { children[i] = pid; } #endif #if 0 /* * remove a random lockspace, replace any pids that were using * it, replace the lockspace */ sleep(get_rand(1, 60)); if (prog_stop) break; lsi = get_rand(0, ls_count-1); remove_lockspace(lsi); while (1) { rv = waitpid(-1, &status, WNOHANG); if (rv <= 0) break; if (!WIFEXITED(status)) continue; printf("exit pid %d\n", pid); i = find_pid(children, rv); if (i < 0) continue; pid = fork(); if (pid < 0) { log_error("fork failed %d", errno); break; } else if (!pid) { do_rand_child(); exit(-1); } else { children[i] = pid; } } add_lockspace(lsi); #endif if (run_sec && (count_sec >= run_sec)) break; count_sec++; sleep(1); } printf("stopping pids "); for (i = 0; i < pid_count; i++) kill(children[i], SIGTERM); while (run_count) { status = 0; pid = wait(&status); if (pid > 0) { run_count--; if (!WIFEXITED(status)) { error_count++; printf("-"); } else if (WEXITSTATUS(status)) { error_count++; printf("x"); } else { printf("."); } } } printf("\n"); if (error_count) { printf("child errors %d\n", error_count); exit(EXIT_FAILURE); } return 0; } int do_all(int argc, char *argv[]) { struct sigaction act; int children[MAX_PID_COUNT]; int run_count = 0; int i, rv, pid, status; if (argc < 5) return -1; memset(&act, 0, sizeof(act)); act.sa_handler = sigterm_handler; sigaction(SIGTERM, &act, NULL); strcpy(lock_disk_base, argv[2]); get_options(argc, argv); rv = add_lockspaces(); if (rv < 0) return rv; printf("forking %d pids\n", pid_count); for (i = 0; i < pid_count; i++) { pid = fork(); if (pid < 0) { log_error("fork %d failed %d run_count %d", i, errno, run_count); break; } if (!pid) { do_all_child(); exit(-1); } children[i] = pid; run_count++; } printf("children running\n"); while (!prog_stop) { sleep(1); } printf("stopping pids"); for (i = 0; i < pid_count; i++) kill(children[i], SIGTERM); while (run_count) { pid = wait(&status); if (pid > 0) { run_count--; printf("."); } } printf("\n"); if (error_count) { printf("error_count %d\n", error_count); exit(EXIT_FAILURE); } return 0; } /* * sanlk_load init [ ] * lock_disk_base = /dev/vg/foo * * sanlock direct init -s lockspace0:0:/dev/vg/foo0:0 * sanlock direct init -r lockspace0:resource0:/dev/vg/foo0:1M * sanlock direct init -r lockspace0:resource1:/dev/vg/foo0:2M * ... * sanlock direct init -s lockspace1:0:/dev/vg/foo1:0 * sanlock direct init -r lockspace1:resource0:/dev/vg/foo1:1M * sanlock direct init -r lockspace1:resource1:/dev/vg/foo1:2M * ... */ #define INIT_NUM_HOSTS 64 int do_init(int argc, char *argv[]) { char resbuf[sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk)]; struct sanlk_resource *res; struct sanlk_lockspace ls; int i, j, rv; if (argc < 3) return -1; strcpy(lock_disk_base, argv[2]); if (argc > 3) ls_count = atoi(argv[3]); if (argc > 4) res_count = atoi(argv[4]); for (i = 0; i < ls_count; i++) { memset(&ls, 0, sizeof(ls)); sprintf(ls.host_id_disk.path, "%s%d", lock_disk_base, i); sprintf(ls.name, "lockspace%d", i); rv = sanlock_direct_init(&ls, NULL, 0, INIT_NUM_HOSTS, 1); if (rv < 0) { printf("sanlock_direct_init lockspace error %d %s\n", rv, ls.host_id_disk.path); return -1; } for (j = 0; j < res_count; j++) { memset(resbuf, 0, sizeof(resbuf)); res = (struct sanlk_resource *)&resbuf; strcpy(res->lockspace_name, ls.name); sprintf(res->name, "resource%d", j); res->num_disks = 1; strcpy(res->disks[0].path, ls.host_id_disk.path); res->disks[0].offset = (j+1)*LEASE_SIZE; rv = sanlock_direct_init(NULL, res, 0, INIT_NUM_HOSTS, 0); if (rv < 0) { printf("sanlock_direct_init resource error %d\n", rv); return -1; } } } return 0; } int main(int argc, char *argv[]) { int rv = -1; if (argc < 2) goto out; if (!strcmp(argv[1], "init")) rv = do_init(argc, argv); else if (!strcmp(argv[1], "rand")) rv = do_rand(argc, argv); else if (!strcmp(argv[1], "all")) rv = do_all(argc, argv); if (!rv) return 0; out: printf("sanlk_load init [ ]\n"); printf(" init ls_count lockspaces, each with res_count resources\n"); printf(" devices for lockspaces 0..N are disk_base0..disk_baseN\n"); printf(" e.g. /dev/lock0, /dev/lock1, ... /dev/lockN\n"); printf("\n"); printf("sanlk_load rand -i [options]\n"); printf(" -s number of lockspaces\n"); printf(" -r number of resources per lockspace\n"); printf(" -p number of processes\n"); printf(" -m use one mode for all locks, 3 = SH, 5 = EX\n"); printf(" -S seconds to run (0 unlimited)\n"); printf(" -e error range expected (1 single node, 2 multi node)\n"); printf(" -D debug output\n"); printf(" -V verbose debug output\n"); printf("\n"); return -1; } sanlock/tests/sanlk_lockr.c000066400000000000000000000026251516326262600163360ustar00rootroot00000000000000#include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock.h" #include "sanlock_resource.h" /* gcc with -lsanlock */ int main(int argc, char *argv[]) { char rd[sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk)]; struct sanlk_resource *res; int sleep_sec; int fd, rv; if (argc < 6) { printf("acquire, [sleep], release\n"); printf("sanlk_lockr \n"); return -1; } memset(rd, 0, sizeof(rd)); res = (struct sanlk_resource *)&rd; strcpy(res->lockspace_name, argv[1]); strcpy(res->name, argv[2]); res->num_disks = 1; strcpy(res->disks[0].path, argv[3]); res->disks[0].offset = atoi(argv[4]); sleep_sec = atoi(argv[5]); fd = sanlock_register(); if (fd < 0) { fprintf(stderr, "register error %d\n", fd); return -1; } rv = sanlock_acquire(fd, -1, 0, 1, &res, NULL); if (rv < 0) { fprintf(stderr, "acquire error %d\n", rv); return -1; } if (sleep_sec) sleep(sleep_sec); rv = sanlock_release(fd, -1, 0, 1, &res); if (rv < 0) { fprintf(stderr, "release error %d\n", rv); return -1; } return 0; } sanlock/tests/sanlk_lvb.c000066400000000000000000000045621516326262600160110ustar00rootroot00000000000000#include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock.h" #include "sanlock_resource.h" /* gcc with -lsanlock */ int main(int argc, char *argv[]) { char rd[sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk)]; struct sanlk_resource *res; char lvb[512]; char *filename; char *act; FILE *fp; int fd, rv; int set = 0, get = 0; memset(lvb, 0, sizeof(lvb)); if (argc < 7) { printf("read file, write it to lvb:\n"); printf("sanlk_lvb set \n"); printf("\n"); printf("read lvb, write it to file:\n"); printf("sanlk_lvb get \n"); return -1; } memset(rd, 0, sizeof(rd)); res = (struct sanlk_resource *)&rd; act = argv[1]; strcpy(res->lockspace_name, argv[2]); strcpy(res->name, argv[3]); res->num_disks = 1; strcpy(res->disks[0].path, argv[4]); res->disks[0].offset = atoi(argv[5]); filename = argv[6]; if (!strcmp(act, "set")) { set = 1; fp = fopen(filename, "r"); } else if (!strcmp(act, "get")) { get = 1; fp = fopen(filename, "w"); } else { printf("bad action %s\n", act); return -1; } if (!fp) { printf("fopen failed %s\n", strerror(errno)); return -1; } fd = sanlock_register(); if (fd < 0) { printf("register error %d\n", fd); return -1; } rv = sanlock_acquire(fd, -1, SANLK_ACQUIRE_LVB, 1, &res, NULL); if (rv < 0) { printf("acquire error %d\n", rv); return -1; } if (get) { rv = sanlock_get_lvb(0, res, lvb, sizeof(lvb)); if (rv < 0) { printf("get_lvb error %d\n", rv); return -1; } fwrite(lvb, sizeof(lvb), 1, fp); if (ferror(fp)) { printf("fwrite error\n"); return -1; } } if (set) { fread(lvb, sizeof(lvb), 1, fp); if (ferror(fp)) { printf("fread error\n"); return -1; } rv = sanlock_set_lvb(0, res, lvb, sizeof(lvb)); if (rv < 0) { printf("set_lvb error %d\n", rv); return -1; } } fclose(fp); rv = sanlock_release(fd, -1, 0, 1, &res); if (rv < 0) { fprintf(stderr, "release error %d\n", rv); return -1; } return 0; } sanlock/tests/sanlk_mixmsg.c000066400000000000000000000117461516326262600165340ustar00rootroot00000000000000#include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock.h" #include "sanlock_resource.h" #include "sanlock_admin.h" #include "sanlock_sock.h" /* gcc with -lsanlock */ /* * sanlock direct init -s 1271384c-24db-4c9b-bebf-61a1916b6cb1:0:/dev/test/main:0 * sanlock add_lockspace -s 1271384c-24db-4c9b-bebf-61a1916b6cb1:1:/dev/test/main:0 */ /* copied from client.c */ static int send_header(int sock, int cmd, uint32_t cmd_flags, int datalen, uint32_t data, uint32_t data2) { struct sm_header header; int rv; memset(&header, 0, sizeof(header)); header.magic = SM_MAGIC; header.version = SM_PROTO; header.cmd = cmd; header.cmd_flags = cmd_flags; header.length = sizeof(header) + datalen; header.data = data; header.data2 = data2; retry: rv = send(sock, (void *) &header, sizeof(header), 0); if (rv == -1 && errno == EINTR) goto retry; if (rv < 0) return -errno; return 0; } int main(int argc, char *argv[]) { char rd1[sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk)]; char rd2[sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk)]; struct sanlk_resource *res1; struct sanlk_resource *res2; const char *lsname; const char *resname1; const char *resname2; char *path; int fd, rv; if (argc < 2) { printf("%s \n", argv[0]); return -1; } path = argv[1]; lsname = "1271384c-24db-4c9b-bebf-61a1916b6cb1"; resname1 = "2e794e7a-5a9c-4617-8cd0-dc03c917d7a1"; resname2 = "2e794e7a-5a9c-4617-8cd0-dc03c917d7a2"; memset(rd1, 0, sizeof(rd1)); memset(rd2, 0, sizeof(rd2)); res1 = (struct sanlk_resource *)&rd1; res2 = (struct sanlk_resource *)&rd2; strcpy(res1->lockspace_name, lsname); sprintf(res1->name, "%s", resname1); res1->num_disks = 1; strcpy(res1->disks[0].path, path); res1->disks[0].offset = 1048576; strcpy(res2->lockspace_name, lsname); sprintf(res2->name, "%s", resname2); res2->num_disks = 1; strcpy(res2->disks[0].path, path); res2->disks[0].offset = 2 * 1048576; /* struct sanlk_lockspace ls = { 0 }; sprintf(ls.name, lsname); sprintf(ls.host_id_disk.path, path); rv = sanlock_write_lockspace(&ls, 0, 0, 0); if (rv < 0) { printf("write_lockspace error %d\n", rv); return -1; } */ rv = sanlock_write_resource(res1, 0, 0, 0); if (rv < 0) { printf("write_resource1 error %d\n", rv); return -1; } rv = sanlock_write_resource(res2, 0, 0, 0); if (rv < 0) { printf("write_resource2 error %d\n", rv); return -1; } fd = sanlock_register(); if (fd < 0) { printf("register error %d\n", fd); return -1; } printf("acquiring both leases for registered fd %d\n", fd); rv = sanlock_acquire(fd, -1, 0, 1, &res1, NULL); if (rv < 0) { printf("acquire res1 error %d\n", rv); return -1; } rv = sanlock_acquire(fd, -1, 0, 1, &res2, NULL); if (rv < 0) { printf("acquire res2 error %d\n", rv); return -1; } printf("sleeping... check that both leases are held\n"); sleep(20); printf("sending res1 release header only\n"); rv = send_header(fd, SM_CMD_RELEASE, 0, sizeof(struct sanlk_resource), 1, -1); if (rv < 0) printf("send bad header error %d\n", rv); else printf("send bad header ok\n"); printf("sending res2 release interleaved\n"); rv = sanlock_release(fd, -1, 0, 1, &res2); if (rv < 0) printf("odd release res2 error %d\n", rv); else printf("odd release res2 ok\n"); printf("sending res1 release body only\n"); rv = send(fd, res1, sizeof(struct sanlk_resource), 0); if (rv < 0) printf("send bad body error %d\n", rv); else printf("send bad body ok\n"); /* * This is not simulating the recv() that each sanlock_release * would do in libsanlock to get a result for each release. * These would likely just cause the client block indefinitely * waiting for a reply that won't come because the bad release * calls were ignored. */ printf("sleeping... check which leases are held\n"); sleep(20); printf("releasing both leases normally\n"); rv = sanlock_release(fd, -1, 0, 1, &res1); if (rv < 0) printf("release res1 error %d\n", rv); else printf("release res1 ok\n"); rv = sanlock_release(fd, -1, 0, 1, &res2); if (rv < 0) printf("release res2 error %d\n", rv); else printf("release res2 ok\n"); printf("sleeping... check that both leases are released\n"); sleep(20); printf("acquiring lease res1\n"); rv = sanlock_acquire(fd, -1, 0, 1, &res1, NULL); if (rv < 0) printf("acquire res1 error %d\n", rv); else printf("acquire res1 ok\n"); /* exit should close our registered connection and automatically release res1 */ printf("exiting... check if held lease is released after exit\n"); return 0; } sanlock/tests/sanlk_path.c000066400000000000000000000103241516326262600161530ustar00rootroot00000000000000#include #include #include #include #include "sanlock.h" #include "sanlock_resource.h" #define DSTMAXSIZE 1024 static int __test_passed = 0; #define check_perror(expression, fmt, args...) \ if (expression) { \ __test_passed++; \ } \ else { \ fprintf(stderr, "%s:%i " fmt "\n", __FILE__, __LINE__, ##args); \ exit(1); \ } void test_sanlock_path_export(void) { int rv, dst_len; char dst_str[DSTMAXSIZE]; const char *src_str, *dst_exp; /* regular behavior, no escapes */ src_str = "Hello World"; dst_exp = src_str; dst_len = strlen(dst_exp); memset(dst_str, 'X', DSTMAXSIZE); /* destination too short */ rv = sanlock_path_export(dst_str, src_str, dst_len); check_perror(rv == 0, "sanlock_path_export wrong return code: %u", rv); check_perror(dst_str[dst_len] == 'X', "sanlock_path_export buffer overflow"); /* destination long enough */ rv = sanlock_path_export(dst_str, src_str, dst_len + 1); check_perror(rv == dst_len, "sanlock_path_export wrong return code: %u", rv); check_perror(dst_str[dst_len] == '\0', "sanlock_path_import destination not terminated"); check_perror(!strncmp(dst_str, dst_exp, dst_len), "sanlock_path_export destination is different"); /* special behavior, escapes */ src_str = "Hello World:"; dst_exp = "Hello World\\:"; dst_len = strlen(dst_exp); memset(dst_str, 'X', DSTMAXSIZE); /* destination too short */ rv = sanlock_path_export(dst_str, src_str, dst_len); check_perror(rv == 0, "sanlock_path_export wrong return code: %u", rv); check_perror(dst_str[dst_len] == 'X', "sanlock_path_export buffer overflow"); /* destination long enough */ rv = sanlock_path_export(dst_str, src_str, dst_len + 1); check_perror(rv == dst_len, "sanlock_path_export wrong return code: %u", rv); check_perror(dst_str[dst_len] == '\0', "sanlock_path_import destination not terminated"); check_perror(!strncmp(dst_str, dst_exp, dst_len), "sanlock_path_export destination is different"); } void test_sanlock_path_import(void) { int rv, dst_len; char dst_str[DSTMAXSIZE]; const char *src_str, *dst_exp; /* regular behavior, no escapes */ src_str = "Hello World"; dst_exp = src_str; dst_len = strlen(dst_exp); memset(dst_str, 'X', DSTMAXSIZE); /* destination too short */ rv = sanlock_path_import(dst_str, src_str, dst_len); check_perror(rv == 0, "sanlock_path_import wrong return code: %u", rv); check_perror(dst_str[dst_len] == 'X', "sanlock_path_import buffer overflow"); /* destination long enough */ rv = sanlock_path_import(dst_str, src_str, dst_len + 1); check_perror(rv == dst_len, "sanlock_path_import wrong return code: %u", rv); check_perror(dst_str[dst_len] == '\0', "sanlock_path_import destination not terminated"); check_perror(!strncmp(dst_str, dst_exp, dst_len), "sanlock_path_import destination is different"); /* special behavior, escapes */ src_str = "Hello World\\:"; dst_exp = "Hello World:"; dst_len = strlen(dst_exp); memset(dst_str, 'X', DSTMAXSIZE); /* destination too short */ rv = sanlock_path_import(dst_str, src_str, dst_len); check_perror(rv == 0, "sanlock_path_import wrong return code: %u", rv); check_perror(dst_str[dst_len] == 'X', "sanlock_path_import buffer overflow"); /* destination long enough */ rv = sanlock_path_import(dst_str, src_str, dst_len + 1); check_perror(rv == dst_len, "sanlock_path_import wrong return code: %u", rv); check_perror(dst_str[dst_len] == '\0', "sanlock_path_import destination not terminated"); check_perror(!strncmp(dst_str, dst_exp, dst_len), "sanlock_path_import destination is different"); } int main(int argc __attribute__ ((unused)), char *argv[] __attribute__ ((unused))) { test_sanlock_path_export(); test_sanlock_path_import(); printf("OK, %i tests successfully passed.\n", __test_passed); return 0; } sanlock/tests/sanlk_rename.c000066400000000000000000000033041516326262600164660ustar00rootroot00000000000000#include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "sanlock.h" #include "sanlock_resource.h" /* gcc with -lsanlock */ int main(int argc, char *argv[]) { char rd[sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk)]; char rd2[sizeof(struct sanlk_resource) + sizeof(struct sanlk_disk)]; struct sanlk_resource *res; struct sanlk_resource *res2; struct sanlk_resource **res_args; int fd, rv; if (argc < 6) { printf("acquire with old name, release with new name\n"); printf("sanlk_rename \n"); return -1; } res_args = malloc(2 * sizeof(struct sanlk_resource *)); memset(rd, 0, sizeof(rd)); memset(rd2, 0, sizeof(rd2)); res = (struct sanlk_resource *)&rd; res2 = (struct sanlk_resource *)&rd2; res_args[0] = res; res_args[1] = res2; strcpy(res->lockspace_name, argv[1]); strcpy(res->name, argv[2]); strcpy(res2->name, argv[3]); res->num_disks = 1; strcpy(res->disks[0].path, argv[4]); res->disks[0].offset = atoi(argv[5]); fd = sanlock_register(); if (fd < 0) { fprintf(stderr, "register error %d\n", fd); return -1; } rv = sanlock_acquire(fd, -1, 0, 1, &res, NULL); if (rv < 0) { fprintf(stderr, "acquire error %d\n", rv); return -1; } rv = sanlock_release(fd, -1, SANLK_REL_RENAME, 2, res_args); if (rv < 0) { fprintf(stderr, "release error %d\n", rv); return -1; } return 0; } sanlock/tests/sanlk_string.c000066400000000000000000000063401516326262600165300ustar00rootroot00000000000000#include #include #include #include #include #include #include #include "sanlock.h" #include "sanlock_resource.h" void print_res(struct sanlk_resource *res) { int i; printf("struct fields: \"%s\" \"%s\"", res->lockspace_name, res->name); for (i = 0; i < res->num_disks; i++) { printf(" \"%s\" %llu", res->disks[i].path, (unsigned long long)res->disks[i].offset); } printf(" flags %x", res->flags); printf(" lver %llu\n", (unsigned long long)res->lver); } int main(int argc, char *argv[]) { struct sanlk_lockspace ls; struct sanlk_resource *res; struct sanlk_resource **res_args = NULL; char *state; int res_count; int rv, i; if (argc < 2) { printf("%s RESOURCE RESOURCE ...\n", argv[0]); printf("%s -s LOCKSPACE\n", argv[0]); return 0; } if (!strcmp(argv[1], "-s")) { memset(&ls, 0, sizeof(ls)); rv = sanlock_str_to_lockspace(argv[2], &ls); printf("struct fields: \"%s\" %llu %u \"%s\" %llu\n", ls.name, (unsigned long long)ls.host_id, ls.flags, ls.host_id_disk.path, (unsigned long long)ls.host_id_disk.offset); return rv; } state = malloc(1024 * 1024); memset(state, 0, 1024 * 1024); printf("\n"); printf("sanlock_str_to_res for each argv\n", rv); printf("--------------------------------------------------------------------------------\n"); for (i = 1; i < argc; i++) { rv = sanlock_str_to_res(argv[i], &res); print_res(res); free(res); res = NULL; if (i > 1) strcat(state, " "); strcat(state, argv[i]); } printf("\n"); printf("combined argv input for state_to_args\n"); printf("--------------------------------------------------------------------------------\n"); printf("\"%s\"\n", state); rv = sanlock_state_to_args(state, &res_count, &res_args); printf("\n"); printf("sanlock_state_to_args %d res_count %d\n", rv, res_count); printf("--------------------------------------------------------------------------------\n"); for (i = 0; i < res_count; i++) { res = res_args[i]; print_res(res); } free(state); state = NULL; rv = sanlock_args_to_state(res_count, res_args, &state); printf("\n"); printf("sanlock_args_to_state %d\n", rv); printf("--------------------------------------------------------------------------------\n"); printf("\"%s\"\n", state); return 0; } #if 0 [root@bull-02 tests]# ./res_string 'LA:R1:/dev/foo1\:xx:0:/dev/foo2\:yy:0' 'LB:R2:/dev/bar:11' sanlock_str_to_res for each argv -------------------------------------------------------------------------------- struct fields: "LA" "R1" "/dev/foo1:xx" 0 "/dev/foo2:yy" 0 0 struct fields: "LB" "R2" "/dev/bar" 11 0 combined argv input for state_to_args -------------------------------------------------------------------------------- "LA:R1:/dev/foo1\:xx:0:/dev/foo2\:yy:0 LB:R2:/dev/bar:11" sanlock_state_to_args 0 res_count 2 -------------------------------------------------------------------------------- struct fields: "LA" "R1" "/dev/foo1:xx" 0 "/dev/foo2:yy" 0 0 struct fields: "LB" "R2" "/dev/bar" 11 0 sanlock_args_to_state 0 -------------------------------------------------------------------------------- "LA:R1:/dev/foo1\:xx:0:/dev/foo2\:yy:0:0 LB:R2:/dev/bar:11:0" #endif sanlock/tests/sanlk_testr.c000066400000000000000000000041171516326262600163630ustar00rootroot00000000000000#include #include #include #include #include #include #include #include "sanlock.h" #include "sanlock_admin.h" #include "sanlock_resource.h" void print_res(struct sanlk_resource *res) { int i; printf("struct fields: \"%s\" \"%s\"", res->lockspace_name, res->name); for (i = 0; i < res->num_disks; i++) { printf(" \"%s\" %llu", res->disks[i].path, (unsigned long long)res->disks[i].offset); } printf(" flags %x", res->flags); printf(" lver %llu\n", (unsigned long long)res->lver); } int main(int argc, char *argv[]) { struct sanlk_resource *res = NULL; struct sanlk_host *hosts = NULL; struct sanlk_host *owners = NULL; struct sanlk_host *host, *owner; int hosts_count = 0; int owners_count = 0; uint32_t test_flags = 0; int i, rv; if (argc < 2) { printf("%s RESOURCE\n", argv[0]); return 0; } rv = sanlock_str_to_res(argv[1], &res); if (rv < 0) { printf("str_to_res %d\n", rv); goto out; } rv = sanlock_get_hosts(res->lockspace_name, 0, &hosts, &hosts_count, 0); if (rv < 0) { printf("get_hosts %d\n", rv); goto out; } rv = sanlock_read_resource_owners(res, 0, &owners, &owners_count); if (rv < 0) { printf("read_resource_owners %d\n", rv); goto out; } rv = sanlock_test_resource_owners(res, 0, owners, owners_count, hosts, hosts_count, &test_flags); if (rv < 0) { printf("test_resource_owners %d\n", rv); goto out; } printf("lockspace hosts:\n"); host = hosts; for (i = 0; i < hosts_count; i++) { printf("host %llu gen %llu state %u\n", (unsigned long long)host->host_id, (unsigned long long)host->generation, host->flags & SANLK_HOST_MASK); host++; } printf("resource owners:\n"); owner = owners; for (i = 0; i < owners_count; i++) { printf("owner %llu gen %llu\n", (unsigned long long)owner->host_id, (unsigned long long)owner->generation); owner++; } printf("test_flags %x\n", test_flags); out: if (res) free(res); if (hosts) free(hosts); if (owners) free(owners); return 0; } sanlock/tests/storage.py000066400000000000000000000014621516326262600157040ustar00rootroot00000000000000# Copyright (C) 2019 Red Hat, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. """ storage - provide storage for sanlock tests. """ from userstorage import File, Mount, LoopDevice GiB = 1024**3 BASE_DIR = "/var/tmp/sanlock-storage" BACKENDS = { "file": File( Mount( LoopDevice( base_dir=BASE_DIR, name="file", size=GiB, sector_size=4096))), "block": LoopDevice( base_dir=BASE_DIR, name="loop", size=GiB, sector_size=4096), } sanlock/tests/test-recovery.sh000077500000000000000000000126321516326262600170410ustar00rootroot00000000000000#!/bin/bash # # recovery tests based on 10 sec io timeout # dev=$1 echo test lockspace storage loss, recovery by lease release using killpath echo messages: sanlock check lease warn/fail, kill 100, all pids clear echo messages: wdmd warn, close, fail echo messages: killpath_pause date set -x ./clientn 4 start $dev 1 /root/killpath_pause sleep 5 ./clientn 4 error $dev sleep 150 ./clientn 4 resume $dev 1 sleep 5 killall -9 sanlk_client sleep 5 set +x echo test lockspace storage loss, recovery by escalation from killpath to sigkill echo messages: sanlock check lease warn/fail, kill 100, kill 9, dead, all pids clear echo messages: wdmd warn, close, fail echo messages: killpath_args date set -x ./clientn 4 start $dev 1 /root/killpath_args sleep 5 ./clientn 4 error $dev sleep 150 ./clientn 4 linear $dev 1 sleep 5 set +x echo test lockspace storage loss, recovery by pid exit using killpath echo messages: sanlock check lease warn/fail, kill 100, dead, all pids clear echo messages: wdmd warn, close, fail echo messages: killpath_term date set -x ./clientn 4 start $dev 1 /root/killpath_term sleep 5 ./clientn 4 error $dev sleep 150 ./clientn 4 linear $dev 1 sleep 5 set +x echo test lockspace storage loss, recovery by pid sigterm without killpath echo messages: sanlock check lease warn/fail, kill 15, dead, all pids clear echo messages: wdmd warn, close, fail date set -x ./clientn 4 start $dev 1 none sleep 5 ./clientn 4 error $dev sleep 150 ./clientn 4 linear $dev 1 sleep 5 set +x echo test lockspace storage delay, small enough to have no effect echo messages: none date set -x ./clientn 4 start $dev 1 none sleep 22 ./clientn 4 iodelay $dev 57 sleep 5 killall -9 sanlk_client sleep 5 set +x echo test lockspace storage delay, long enough to produce sanlock warning, echo but not failure, not long enough for wdmd warn or close echo messages: sanlock check lease warn date set -x ./clientn 4 start $dev 1 none sleep 22 ./clientn 4 iodelay $dev 67 sleep 5 killall -9 sanlk_client sleep 5 set +x echo test lockspace storage delay, long enough to produce sanlock warning, echo but not failure/recovery, long enough for wdmd warn and close echo messages: sanlock check lease warn echo messages: wdmd warn, close date set -x ./clientn 4 start $dev 1 none sleep 22 ./clientn 4 iodelay $dev 77 sleep 5 killall -9 sanlk_client sleep 5 set +x echo test lockspace storage delay, long enough to produce sanlock warning, echo failure/recovery, recovery by lease release using killpath echo messages: sanlock check lease warn/fail, kill 100, all pids clear echo messages: killpath_pause echo messages: wdmd warn, close, fail date set -x ./clientn 4 start $dev 1 /root/killpath_pause sleep 22 ./clientn 4 iodelay $dev 87 sleep 5 set +x echo test lockspace storage delay, long enough to produce sanlock warning, echo failure/recovery, recovery by pid sigterm without killpath echo messages: sanlock check lease warn/fail, kill 15, dead, all pids clear echo messages: wdmd warn, close, fail date set -x ./clientn 4 start $dev 1 none sleep 22 ./clientn 4 iodelay $dev 87 sleep 5 set +x echo test daemon run delay, small enough to have no effect echo messages: none date set -x ./clientn 4 start $dev 1 none sleep 22 ./clientn 4 delay 58 sleep 5 killall -9 sanlk_client sleep 5 set +x echo test daemon run delay, long enough to produce sanlock warning, echo but not failure, not long enough for wdmd warn or close echo messages: sanlock check lease warn date set -x ./clientn 4 start $dev 1 none sleep 22 ./clientn 4 delay 68 sleep 5 killall -9 sanlk_client sleep 5 set +x echo test daemon run delay, long enough to produce sanlock warning, echo but not failure, long enough for wdmd warn and close echo messages: sanlock check lease warn echo messages: wdmd warn, close date set -x ./clientn 4 start $dev 1 none sleep 22 ./clientn 4 delay 78 sleep 5 killall -9 sanlk_client sleep 5 set +x echo test daemon run delay, long enough to produce sanlock echo failure/recovery, recovery by lease release using killpath echo messages: sanlock check lease fail, kill 100, all pids clear echo messages: wdmd warn, close, fail echo messages: killpath_pause date set -x ./clientn 4 start $dev 1 /root/killpath_pause sleep 22 ./clientn 4 delay 88 sleep 5 ./clientn 4 resume $dev 1 sleep 5 killall -9 sanlk_client sleep 5 set +x echo test daemon run delay, long enough to produce sanlock echo failure/recovery, recovery by pid sigterm without killpath echo messages: sanlock check lease fail, kill 15, dead, all pids clear echo messages: wdmd warn, close, fail date set -x ./clientn 4 start $dev 1 none sleep 22 ./clientn 4 delay 88 sleep 5 set +x echo test daemon run delay, long enough to produce sanlock echo failure/recovery, recovery by pid sigkill after skipping killpath echo messages: sanlock check lease fail, kill 9, dead, all pids clear echo messages: wdmd warn, close, fail date set -x ./clientn 4 start $dev 1 /root/killpath_pause sleep 22 ./clientn 4 delay 130 sleep 5 set +x echo test daemon run delay, long enough to produce sanlock echo failure/recovery, recovery by pid sigkill without killpath echo messages: sanlock check lease fail, kill 9, dead, all pids clear echo messages: wdmd warn, close, fail date set -x ./clientn 4 start $dev 1 none sleep 22 ./clientn 4 delay 130 sleep 5 set +x echo test daemon run delay, long enough to produce watchdog firing echo messages: wdmd warn, close, fail date set -x ./clientn 4 start $dev 1 none sleep 22 ./clientn 4 delay 140 echo should not get here sanlock/tests/units.py000066400000000000000000000006101516326262600153740ustar00rootroot00000000000000# Copyright (C) 2019 Red Hat, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. """ Constants for file/disk sizes. """ KiB = 1024 MiB = 1024**2 GiB = 1024**3 TiB = 1024**4 PiB = 1024**5 sanlock/tests/util.py000066400000000000000000000122241516326262600152130ustar00rootroot00000000000000# Copyright (C) 2019 Red Hat, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. """ Testing utilities """ from __future__ import absolute_import import errno import io import os import socket import struct import subprocess import time from . units import KiB TESTDIR = os.path.dirname(__file__) SANLOCK = os.path.join(TESTDIR, os.pardir, "src", "sanlock") GUARD = b"X" GUARD_SIZE = 4 * KiB class TimeoutExpired(Exception): """ Raised when timeout expired """ class CommandError(Exception): msg = ("Command {self.cmd} failed with returncode={self.returncode}, " "stdout={self.stdout!r}, stderr={self.stderr!r}") def __init__(self, cmd, returncode, stdout, stderr): self.cmd = cmd self.returncode = returncode self.stdout = stdout self.stderr = stderr def __str__(self): return self.msg.format(self=self) def start_daemon(): cmd = [SANLOCK, "daemon", # no fork and print all logging to stderr "-D", # don't use watchdog through wdmd "-w", "0", # don't use mlockall "-l", "0", # don't use high priority (RR) scheduling "-h", "0", # run as current user instead of "sanlock" "-U", os.environ["USER"], "-G", os.environ["USER"]] return subprocess.Popen(cmd) def wait_for_daemon(timeout): """ Wait until daemon is accepting connections """ deadline = time.time() + timeout path = os.path.join(os.environ["SANLOCK_RUN_DIR"], "sanlock.sock") s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) try: while True: try: s.connect(path) return except EnvironmentError as e: if e.errno not in (errno.ECONNREFUSED, errno.ENOENT): raise # Unexpected error if time.time() > deadline: raise TimeoutExpired time.sleep(0.05) finally: s.close() def sanlock(*args, cwd=None): """ Run sanlock returning the process stdout, or raising util.CommandError on failures. """ cmd = [SANLOCK] cmd.extend(args) p = subprocess.Popen( cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=cwd) out, err = p.communicate() if p.returncode: raise CommandError(cmd, p.returncode, out, err) return out def wait_for_termination(p, timeout): """ Wait until process terminates, or timeout expires. """ deadline = time.time() + timeout while True: if p.poll() is not None: return if time.time() > deadline: raise TimeoutExpired time.sleep(0.05) def create_file(path, size, guard=True): """ Create sparse file of size bytes. If guard is True, add a guard area beyond the end of the file. """ with io.open(path, "wb") as f: f.truncate(size) if guard: write_guard(path, size) def write_guard(path, offset): """ Write guard areas at offset and fill with guard byte. Use check_guard() to verify that nothing was written to the guard area. """ with io.open(path, "rb+") as f: f.seek(offset) f.write(GUARD * GUARD_SIZE) def check_guard(path, offset): """ Assert that guard area at offset was not modified. """ with io.open(path, "rb") as f: f.seek(offset) assert f.read(GUARD_SIZE) == GUARD * GUARD_SIZE def check_rindex_entry(entry, name, offset=None, flags=None): # See src/ondisk.c rindex_entry_in() e_offset, e_flags, e_unused, e_name = struct.unpack(" #include #include #include #include #include #include #include #include #include #include #include #include "wdmd.h" #include "wdmd_sock.h" int wdmd_connect(void) { int rv, s; struct sockaddr_un addr; s = socket(AF_LOCAL, SOCK_STREAM, 0); if (s < 0) return -errno; rv = wdmd_socket_address(&addr); if (rv < 0) return rv; rv = connect(s, (struct sockaddr *) &addr, sizeof(struct sockaddr_un)); if (rv < 0) { rv = -errno; close(s); return rv; } return s; } int wdmd_register(int con, char *name) { struct wdmd_header h; int rv; if (strlen(name) > WDMD_NAME_SIZE) return -ENAMETOOLONG; memset(&h, 0, sizeof(h)); h.cmd = CMD_REGISTER; strncpy(h.name, name, WDMD_NAME_SIZE); rv = send(con, (void *)&h, sizeof(struct wdmd_header), 0); if (rv < 0) return -errno; return 0; } static int send_header(int con, int cmd) { struct wdmd_header h; int rv; memset(&h, 0, sizeof(h)); h.cmd = cmd; rv = send(con, (void *)&h, sizeof(struct wdmd_header), 0); if (rv < 0) return -errno; return 0; } int wdmd_open_watchdog(int con, int fire_timeout) { struct wdmd_header h; int rv; memset(&h, 0, sizeof(h)); h.cmd = CMD_OPEN_WATCHDOG; h.fire_timeout = fire_timeout; rv = send(con, (void *)&h, sizeof(struct wdmd_header), 0); if (rv < 0) return -errno; memset(&h, 0, sizeof(h)); rv = recv(con, &h, sizeof(h), MSG_WAITALL); if (rv < 0) return -errno; if (h.fire_timeout != fire_timeout) return -1; return 0; } int wdmd_refcount_set(int con) { return send_header(con, CMD_REFCOUNT_SET); } int wdmd_refcount_clear(int con) { return send_header(con, CMD_REFCOUNT_CLEAR); } int wdmd_test_live(int con, uint64_t renewal_time, uint64_t expire_time) { struct wdmd_header h; int rv; memset(&h, 0, sizeof(h)); h.cmd = CMD_TEST_LIVE; h.renewal_time = renewal_time; h.expire_time = expire_time; rv = send(con, (void *)&h, sizeof(struct wdmd_header), 0); if (rv < 0) return -errno; return 0; } int wdmd_status(int con, int *test_interval, int *fire_timeout, uint64_t *last_keepalive) { struct wdmd_header h; int rv; rv = send_header(con, CMD_STATUS); if (rv < 0) return rv; rv = recv(con, &h, sizeof(h), MSG_WAITALL); if (rv < 0) return -errno; *test_interval = h.test_interval; *fire_timeout = h.fire_timeout; *last_keepalive = h.last_keepalive; return 0; } sanlock/wdmd/main.c000066400000000000000000001437331516326262600145570ustar00rootroot00000000000000/* * Copyright 2011-2012 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "wdmd.h" #include "wdmd_sock.h" #ifndef GNUC_UNUSED #define GNUC_UNUSED __attribute__((__unused__)) #endif #define DEFAULT_TEST_INTERVAL 10 #define RECOVER_TEST_INTERVAL 1 #define DEFAULT_FIRE_TIMEOUT 60 #define DEFAULT_HIGH_PRIORITY 0 /* * If the group name specified here, or specified on the * command line is not found, then default to gid 0 (root). */ #define SOCKET_GNAME "sanlock" #define DEFAULT_SOCKET_GID 0 #define DEFAULT_SOCKET_MODE (S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP) #define WDPATH_SIZE 64 #define WD_ID_SIZE 64 static int standard_test_interval = DEFAULT_TEST_INTERVAL; static int test_interval= DEFAULT_TEST_INTERVAL; static int fire_timeout = DEFAULT_FIRE_TIMEOUT; static int high_priority = DEFAULT_HIGH_PRIORITY; static int daemon_quit; static int daemon_debug; static int try_timeout; static int forcefire; static int socket_gid; static char *socket_gname = (char *)SOCKET_GNAME; static time_t last_keepalive; static time_t last_closeunclean; static char lockfile_path[PATH_MAX]; static int test_loop_enable; static int dev_fd = -1; static int shm_fd; static int itco; /* watchdog_identity is "iTCO_wdt" */ static int daemon_setup_done; static int allow_scripts; static int kill_script_sec; static const char *scripts_dir = "/etc/wdmd.d"; static char watchdog_path[WDPATH_SIZE]; static char option_path[WDPATH_SIZE]; static char saved_path[WDPATH_SIZE]; static char watchdog_identity[WD_ID_SIZE]; struct script_status { uint64_t start; int pid; int last_result; unsigned int run_count; unsigned int fail_count; unsigned int good_count; unsigned int kill_count; unsigned int long_count; char name[PATH_MAX]; }; #define MAX_SCRIPTS 8 static struct script_status scripts[MAX_SCRIPTS]; struct client { int used; int fd; int pid; int pid_dead; int refcount; int internal; uint64_t renewal; uint64_t expire; void *workfn; void *deadfn; char name[WDMD_NAME_SIZE]; }; #define CLIENT_NALLOC 16 static int client_maxi; static int client_size = 0; static struct client *client = NULL; static struct pollfd *pollfd = NULL; #define log_debug(fmt, args...) \ do { \ if (daemon_debug) \ fprintf(stderr, "%llu %s " fmt "\n", (unsigned long long)time(NULL), time_str(), ##args); \ } while (0) #define log_error(fmt, args...) \ do { \ log_debug(fmt, ##args); \ syslog(LOG_ERR, fmt, ##args); \ } while (0) #define log_info(fmt, args...) \ do { \ log_debug(fmt, ##args); \ syslog(LOG_INFO, fmt, ##args); \ } while (0) #define log_script(i) \ log_error("script %.64s last_result %d start %llu run %u fail %u good %u kill %u long %u", \ scripts[i].name, scripts[i].last_result, \ (unsigned long long)scripts[i].start, \ scripts[i].run_count, scripts[i].fail_count, \ scripts[i].good_count, scripts[i].kill_count, \ scripts[i].long_count); static uint64_t monotime(void) { struct timespec ts; clock_gettime(CLOCK_MONOTONIC, &ts); return ts.tv_sec; } char time_str_buf[128]; static char *time_str(void) { struct timeval cur_time; struct tm time_info; gettimeofday(&cur_time, NULL); localtime_r(&cur_time.tv_sec, &time_info); strftime(time_str_buf, sizeof(time_str_buf), "%Y-%m-%d %H:%M:%S ", &time_info); return time_str_buf; } /* * test clients */ static void client_alloc(void) { int i; if (!client) { client = malloc(CLIENT_NALLOC * sizeof(struct client)); pollfd = malloc(CLIENT_NALLOC * sizeof(struct pollfd)); } else { client = realloc(client, (client_size + CLIENT_NALLOC) * sizeof(struct client)); pollfd = realloc(pollfd, (client_size + CLIENT_NALLOC) * sizeof(struct pollfd)); if (!pollfd) log_error("can't alloc for pollfd"); } if (!client || !pollfd) log_error("can't alloc for client array"); for (i = client_size; i < client_size + CLIENT_NALLOC; i++) { memset(&client[i], 0, sizeof(struct client)); client[i].fd = -1; pollfd[i].fd = -1; pollfd[i].revents = 0; } client_size += CLIENT_NALLOC; } static int client_add(int fd, void (*workfn)(int ci), void (*deadfn)(int ci)) { int i; if (!client) client_alloc(); again: for (i = 0; i < client_size; i++) { if (!client[i].used) { client[i].used = 1; client[i].workfn = workfn; client[i].deadfn = deadfn; client[i].fd = fd; pollfd[i].fd = fd; pollfd[i].events = POLLIN; if (i > client_maxi) client_maxi = i; return i; } } client_alloc(); goto again; } static void client_pid_dead(int ci) { if (!client[ci].expire) { log_debug("client_pid_dead ci %d", ci); close(client[ci].fd); /* refcount automatically dropped if a client with no expiration is closed */ client[ci].used = 0; memset(&client[ci], 0, sizeof(struct client)); client[ci].fd = -1; pollfd[ci].fd = -1; pollfd[ci].events = 0; } else { /* * Leave used and expire set so that test_clients will continue * monitoring this client and expire if necessary. * * Leave refcount set so that the daemon will not cleanly shut * down if it gets a sigterm. * * This case of a client con with an expire time being closed * is a fatal condition; there's no way to clear or extend the * expire time and no way to cleanly shut down the daemon. * This should never happen. * * (We don't enforce that a client with an expire time also has refcount * set, but I can't think of case where setting expire but not refcount * would be useful.) */ log_error("client dead ci %d fd %d pid %d renewal %llu expire %llu %s", ci, client[ci].fd, client[ci].pid, (unsigned long long)client[ci].renewal, (unsigned long long)client[ci].expire, client[ci].name); close(client[ci].fd); client[ci].pid_dead = 1; client[ci].fd = -1; pollfd[ci].fd = -1; pollfd[ci].events = 0; } } static int get_peer_pid(int fd, int *pid) { struct ucred cred; unsigned int cl = sizeof(cred); if (getsockopt(fd, SOL_SOCKET, SO_PEERCRED, &cred, &cl) != 0) return -1; *pid = cred.pid; return 0; } #define DEBUG_SIZE (1024 * 1024) #define LINE_SIZE 256 char debug_buf[DEBUG_SIZE]; static void dump_debug(int fd) { char line[LINE_SIZE]; uint64_t now; int line_len; int debug_len = 0; int i; memset(debug_buf, 0, DEBUG_SIZE); now = monotime(); memset(line, 0, sizeof(line)); snprintf(line, 255, "wdmd %d socket_gid %d high_priority %d now %llu last_keepalive %llu last_closeunclean %llu allow_scripts %d kill_script_sec %d fire_timeout %d identity \"%s\"\n", getpid(), socket_gid, high_priority, (unsigned long long)now, (unsigned long long)last_keepalive, (unsigned long long)last_closeunclean, allow_scripts, kill_script_sec, fire_timeout, watchdog_identity); line_len = strlen(line); strncat(debug_buf, line, LINE_SIZE); debug_len += line_len; for (i = 0; i < MAX_SCRIPTS; i++) { if (!scripts[i].name[0]) continue; memset(line, 0, sizeof(line)); snprintf(line, 255, "script %d name %.64s pid %d now %llu start %llu last_result %d run %u fail %u good %u kill %u long %u\n", i, scripts[i].name, scripts[i].pid, (unsigned long long)now, (unsigned long long)scripts[i].start, scripts[i].last_result, scripts[i].run_count, scripts[i].fail_count, scripts[i].good_count, scripts[i].kill_count, scripts[i].long_count); line_len = strlen(line); if (debug_len + line_len >= DEBUG_SIZE - 1) goto out; strncat(debug_buf, line, LINE_SIZE); debug_len += line_len; } for (i = 0; i < client_size; i++) { if (!client[i].used) continue; memset(line, 0, sizeof(line)); snprintf(line, 255, "client %d name %.64s pid %d fd %d dead %d ref %d now %llu renewal %llu expire %llu\n", i, client[i].name, client[i].pid, client[i].fd, client[i].pid_dead, client[i].refcount, (unsigned long long)now, (unsigned long long)client[i].renewal, (unsigned long long)client[i].expire); line_len = strlen(line); if (debug_len + line_len >= DEBUG_SIZE - 1) goto out; strncat(debug_buf, line, LINE_SIZE); debug_len += line_len; } out: send(fd, debug_buf, debug_len, MSG_NOSIGNAL); } static void _init_test_interval(void) { if (fire_timeout >= 60) { standard_test_interval = 10; test_interval = 10; } else if (fire_timeout >= 30 && fire_timeout < 60) { standard_test_interval = 5; test_interval = 5; } else if (fire_timeout >= 10 && fire_timeout < 30) { standard_test_interval = 2; test_interval = 2; } else { standard_test_interval = 1; test_interval = 1; } } static int open_dev(void) { int fd; if (dev_fd != -1) { log_error("watchdog already open fd %d", dev_fd); return -1; } fd = open(watchdog_path, O_WRONLY | O_CLOEXEC); if (fd < 0) { log_error("open %s error %d", watchdog_path, errno); return fd; } dev_fd = fd; return 0; } static void close_watchdog(void) { int rv; if (dev_fd == -1) { log_debug("close_watchdog already closed"); return; } rv = write(dev_fd, "V", 1); if (rv < 0) log_error("%s disarm write error %d", watchdog_path, errno); else if (daemon_setup_done) log_info("%s disarmed", watchdog_path); close(dev_fd); dev_fd = -1; } static void close_watchdog_unclean(void) { if (dev_fd == -1) return; log_error("%s closed unclean", watchdog_path); close(dev_fd); dev_fd = -1; last_closeunclean = monotime(); } static void pet_watchdog(void) { int rv, unused; rv = ioctl(dev_fd, WDIOC_KEEPALIVE, &unused); last_keepalive = monotime(); log_debug("keepalive %d", rv); } static int _open_watchdog_itco(struct wdmd_header *h) { int get_timeout_itco, get_timeout_real, set_timeout_itco, set_timeout_real; int rv; /* Don't check dev_fd for -1 because dev_fd will be closed and set to -1 prior to timeout in close_watchdog_unclean(). */ if (test_loop_enable) return 0; if (!h->fire_timeout) return -1; rv = open_dev(); if (rv < 0) return -1; get_timeout_real = 0; get_timeout_itco = 0; rv = ioctl(dev_fd, WDIOC_GETTIMEOUT, &get_timeout_itco); if (rv < 0) { log_error("open_watchdog gettimeout error %d", errno); close_watchdog(); return -1; } get_timeout_real = get_timeout_itco * 2; if (get_timeout_real == h->fire_timeout) { /* success, requested value matches the default value */ fire_timeout = get_timeout_real; _init_test_interval(); log_error("%s open with timeout %d", watchdog_path, get_timeout_real); pet_watchdog(); test_loop_enable = 1; return 0; } set_timeout_real = h->fire_timeout; set_timeout_itco = set_timeout_real / 2; rv = ioctl(dev_fd, WDIOC_SETTIMEOUT, &set_timeout_itco); if (rv < 0) { log_error("open_watchdog settimeout %d error %d", set_timeout_real, errno); close_watchdog(); return -1; } get_timeout_real = 0; get_timeout_itco = 0; rv = ioctl(dev_fd, WDIOC_GETTIMEOUT, &get_timeout_itco); if (rv < 0) { log_error("open_watchdog gettimeout check error %d", errno); close_watchdog(); return -1; } get_timeout_real = get_timeout_itco * 2; if (get_timeout_real == set_timeout_real) { /* success setting a custom timeout */ fire_timeout = get_timeout_real; _init_test_interval(); log_error("%s open with timeout %d", watchdog_path, get_timeout_real); pet_watchdog(); test_loop_enable = 1; return 0; } /* failed to set a custom timeout */ log_error("open_watchdog gettimeout value real %d itco %d expect real %d", get_timeout_real, get_timeout_itco, set_timeout_real); close_watchdog(); return -1; } static int _open_watchdog(struct wdmd_header *h) { int get_timeout, set_timeout; int rv; if (itco) return _open_watchdog_itco(h); /* Don't check dev_fd for -1 because dev_fd will be closed and set to -1 prior to timeout in close_watchdog_unclean(). */ if (test_loop_enable) return 0; if (!h->fire_timeout) return -1; rv = open_dev(); if (rv < 0) return -1; get_timeout = 0; rv = ioctl(dev_fd, WDIOC_GETTIMEOUT, &get_timeout); if (rv < 0) { log_error("open_watchdog gettimeout error %d", errno); close_watchdog(); return -1; } if (get_timeout == h->fire_timeout) { /* success, requested value matches the default value */ fire_timeout = get_timeout; _init_test_interval(); log_error("%s open with timeout %d", watchdog_path, get_timeout); pet_watchdog(); test_loop_enable = 1; return 0; } set_timeout = h->fire_timeout; rv = ioctl(dev_fd, WDIOC_SETTIMEOUT, &set_timeout); if (rv < 0) { log_error("open_watchdog settimeout %d error %d", set_timeout, errno); close_watchdog(); return -1; } get_timeout = 0; rv = ioctl(dev_fd, WDIOC_GETTIMEOUT, &get_timeout); if (rv < 0) { log_error("open_watchdog gettimeout check error %d", errno); close_watchdog(); return -1; } if (get_timeout == set_timeout) { /* success setting a custom timeout */ fire_timeout = get_timeout; _init_test_interval(); log_error("%s open with timeout %d", watchdog_path, get_timeout); pet_watchdog(); test_loop_enable = 1; return 0; } /* failed to set a custom timeout */ log_error("open_watchdog gettimeout value %d set %d", get_timeout, set_timeout); close_watchdog(); return -1; } static void process_connection(int ci) { struct wdmd_header h; struct wdmd_header h_ret; void (*deadfn)(int ci); int rv, pid; memset(&h, 0, sizeof(h)); rv = recv(client[ci].fd, &h, sizeof(h), MSG_WAITALL); if (!rv) return; if (rv < 0) { log_error("ci %d recv error %d", ci, errno); goto dead; } if (rv != sizeof(h)) { log_error("ci %d recv size %d", ci, rv); goto dead; } switch(h.cmd) { case CMD_REGISTER: /* TODO: allow client to reconnect, search clients for h.name and copy the renewal and expire times, then clear the old client entry */ rv = get_peer_pid(client[ci].fd, &pid); if (rv < 0) goto dead; client[ci].pid = pid; memcpy(client[ci].name, h.name, WDMD_NAME_SIZE); log_debug("register ci %d fd %d pid %d %s", ci, client[ci].fd, pid, client[ci].name); break; case CMD_REFCOUNT_SET: client[ci].refcount = 1; break; case CMD_REFCOUNT_CLEAR: client[ci].refcount = 0; break; case CMD_OPEN_WATCHDOG: memcpy(&h_ret, &h, sizeof(h)); rv = _open_watchdog(&h); if (rv < 0) h_ret.fire_timeout = 0; else h_ret.fire_timeout = fire_timeout; log_debug("open_watchdog fire_timeout %u result %u", h.fire_timeout, fire_timeout); send(client[ci].fd, &h_ret, sizeof(h_ret), MSG_NOSIGNAL); break; case CMD_TEST_LIVE: client[ci].renewal = h.renewal_time; client[ci].expire = h.expire_time; log_debug("test_live ci %d renewal %llu expire %llu", ci, (unsigned long long)client[ci].renewal, (unsigned long long)client[ci].expire); break; case CMD_STATUS: memcpy(&h_ret, &h, sizeof(h)); h_ret.test_interval = test_interval; h_ret.fire_timeout = fire_timeout; h_ret.last_keepalive = last_keepalive; send(client[ci].fd, &h_ret, sizeof(h_ret), MSG_NOSIGNAL); break; case CMD_DUMP_DEBUG: strncpy(client[ci].name, "dump", WDMD_NAME_SIZE); dump_debug(client[ci].fd); break; }; return; dead: deadfn = client[ci].deadfn; if (deadfn) deadfn(ci); } static void process_listener(int ci) { int fd; int on = 1; fd = accept(client[ci].fd, NULL, NULL); if (fd < 0) return; setsockopt(fd, SOL_SOCKET, SO_PASSCRED, &on, sizeof(on)); client_add(fd, process_connection, client_pid_dead); } static void close_clients(void) { } static int setup_listener_socket(int *listener_socket) { int rv, s; struct sockaddr_un addr; s = socket(AF_LOCAL, SOCK_STREAM, 0); if (s < 0) return -errno; rv = wdmd_socket_address(&addr); if (rv < 0) return rv; unlink(addr.sun_path); rv = bind(s, (struct sockaddr *) &addr, sizeof(struct sockaddr_un)); if (rv < 0) { rv = -errno; close(s); return rv; } rv = listen(s, 5); if (rv < 0) { rv = -errno; close(s); return rv; } rv = chmod(addr.sun_path, DEFAULT_SOCKET_MODE); if (rv < 0) { rv = -errno; close(s); return rv; } rv = chown(addr.sun_path, -1, socket_gid); if (rv < 0) { rv = -errno; close(s); return rv; } fcntl(s, F_SETFL, fcntl(s, F_GETFL, 0) | O_NONBLOCK); *listener_socket = s; return 0; } static int setup_clients(void) { int rv, fd = -1, ci; rv = setup_listener_socket(&fd); if (rv < 0) return rv; ci = client_add(fd, process_listener, client_pid_dead); strncpy(client[ci].name, "listen", WDMD_NAME_SIZE); client[ci].internal = 1; return 0; } static int test_clients(void) { uint64_t t; time_t last_ping; int fail_count = 0; int i; t = monotime(); for (i = 0; i < client_size; i++) { if (!client[i].used) continue; if (!client[i].expire) continue; if (last_keepalive > last_closeunclean) last_ping = last_keepalive; else last_ping = last_closeunclean; if (t >= client[i].expire) { log_error("test failed rem %d now %llu ping %llu close %llu renewal %llu expire %llu client %d %s", fire_timeout - (int)(t - last_ping), (unsigned long long)t, (unsigned long long)last_keepalive, (unsigned long long)last_closeunclean, (unsigned long long)client[i].renewal, (unsigned long long)client[i].expire, client[i].pid, client[i].name); fail_count++; continue; } /* * If we can patch the kernel to avoid a close-ping, * then we can remove this early/preemptive fail/close * of the device, but instead just not pet the device * when the expiration time is reached. Also see * close_watchdog_unclean() below. * * We do this fail/close (which generates a ping) * TEST_INTERVAL before the expire time because we want * the device to fire at most 60 seconds after the * expiration time. That means we need the last ping * (from close) to be TEST_INTERVAL before to the * expiration time. * * If we did the close at/after the expiration time, * then the ping from the close would mean that the * device would fire between 60 and 70 seconds after the * expiration time. */ if (t >= client[i].expire - standard_test_interval) { log_error("test warning now %llu ping %llu close %llu renewal %llu expire %llu client %d %s", (unsigned long long)t, (unsigned long long)last_keepalive, (unsigned long long)last_closeunclean, (unsigned long long)client[i].renewal, (unsigned long long)client[i].expire, client[i].pid, client[i].name); fail_count++; continue; } } return fail_count; } static int active_clients(void) { int i; for (i = 0; i < client_size; i++) { if (client[i].refcount) return 1; } return 0; } static void count_clients(int *active, int *external) { int act = 0, ext = 0; int i; for (i = 0; i < client_size; i++) { if (!client[i].used) continue; if (client[i].refcount) act++; if (!client[i].internal) ext++; } if (active) *active = act; if (external) *external = ext; } #ifdef TEST_FILES #define FILES_DIR "/run/wdmd/test_files" const char *files_built = " files"; static DIR *files_dir; static void close_files(void) { closedir(files_dir); } static int setup_files(void) { mode_t old_umask; int rv; old_umask = umask(0022); rv = mkdir(FILES_DIR, 0777); if (rv < 0 && errno != EEXIST) goto out; files_dir = opendir(FILES_DIR); if (!files_dir) rv = -errno; else rv = 0; out: umask(old_umask); return rv; } static int read_file(char *name, uint64_t *renewal, uint64_t *expire) { FILE *file; char path[PATH_MAX]; snprintf(path, PATH_MAX-1, "%s/%s", FILES_DIR, name); file = fopen(path, "r"); if (!file) return -1; fscanf(file, "renewal %llu expire %llu", renewal, expire); fclose(file); return 0; } static int test_files(void) { struct dirent *de; uint64_t t, renewal, expire; int fail_count = 0; int rv; while ((de = readdir(files_dir))) { if (de->d_name[0] == '.') continue; rv = read_file(de->d_name, &renewal, &expire); if (rv < 0) continue; t = monotime(); if (t >= expire) { log_error("test failed file %s renewal %llu expire %llu ", de->d_name, (unsigned long long)renewal, (unsigned long long)expire); fail_count++; } } return fail_count; } #else static void close_files(void) { } static int setup_files(void) { return 0; } static int test_files(void) { return 0; } #endif /* TEST_FILES */ static int find_script(char *name) { int i; for (i = 0; i < MAX_SCRIPTS; i++) { if (!strncmp(scripts[i].name, name, PATH_MAX)) return i; } return -1; } static int add_script(char *name) { int i; for (i = 0; i < MAX_SCRIPTS; i++) { if (scripts[i].name[0]) continue; log_debug("add_script %d %s", i, name); strncpy(scripts[i].name, name, PATH_MAX); return i; } log_debug("script %s no space", name); return -1; } static int check_path(char *path) { struct stat st; int rv; rv = stat(path, &st); if (rv < 0) return -errno; if (!(S_ISREG(st.st_mode))) return -1; if (!(st.st_mode & S_IXUSR)) return -1; return 0; } static int run_script(int i) { char path[PATH_MAX]; int pid, rv; memset(path, 0, sizeof(path)); snprintf(path, PATH_MAX-1, "%s/%s", scripts_dir, scripts[i].name); rv = check_path(path); if (rv < 0) return rv; pid = fork(); if (pid < 0) return -errno; if (pid) { log_debug("script %s pid %d", scripts[i].name, pid); return pid; } else { execlp(path, path, NULL); exit(EXIT_FAILURE); } } static void close_scripts(void) { } static int setup_scripts(void) { char path[PATH_MAX]; struct dirent **namelist; int i, s, rv, de_count; if (!allow_scripts) return 0; de_count = scandir(scripts_dir, &namelist, 0, alphasort); if (de_count < 0) return 0; for (i = 0; i < de_count; i++) { if (namelist[i]->d_name[0] == '.') goto next; memset(path, 0, sizeof(path)); snprintf(path, PATH_MAX-1, "%s/%s", scripts_dir, namelist[i]->d_name); rv = check_path(path); if (rv < 0) { log_debug("script %s ignore %d", namelist[i]->d_name, rv); goto next; } s = find_script(namelist[i]->d_name); if (s < 0) add_script(namelist[i]->d_name); next: free(namelist[i]); } free(namelist); return 0; } static int test_scripts(void) { int i, rv, pid, result, running, fail_count, status; uint64_t begin, now; if (!allow_scripts) return 0; fail_count = 0; begin = monotime(); for (i = 0; i < MAX_SCRIPTS; i++) { if (!scripts[i].name[0]) continue; /* pid didn't exit in previous cycle */ if (scripts[i].pid) continue; /* * after a script reports success, don't call it again before * the normal test interval; this is needed because the test * interval becomes shorter when failures occur */ if (!scripts[i].last_result && ((begin - scripts[i].start) < (standard_test_interval - 1))) continue; pid = run_script(i); if (pid <= 0) { log_error("script %s removed %d", scripts[i].name, pid); memset(&scripts[i], 0, sizeof(struct script_status)); } else { scripts[i].pid = pid; scripts[i].start = begin; scripts[i].run_count++; } } /* wait up to standard_test_interval-1 for the pids to finish */ while (1) { running = 0; for (i = 0; i < MAX_SCRIPTS; i++) { if (!scripts[i].name[0]) continue; if (!scripts[i].pid) continue; rv = waitpid(scripts[i].pid, &status, WNOHANG); if (rv < 0) { /* shouldn't happen */ log_error("script %s pid %d waitpid error %d %d", scripts[i].name, scripts[i].pid, rv, errno); log_script(i); running++; } else if (!rv) { /* pid still running, has not changed state */ running++; } else if (rv == scripts[i].pid) { /* pid state has changed */ if (WIFEXITED(status)) { /* pid exited with an exit code */ result = WEXITSTATUS(status); if (result) { log_error("script %s pid %d exit status %d", scripts[i].name, scripts[i].pid, result); scripts[i].fail_count++; scripts[i].last_result = result; scripts[i].pid = 0; fail_count++; log_script(i); } else { scripts[i].good_count++; scripts[i].last_result = 0; scripts[i].pid = 0; } } else if (WIFSIGNALED(status)) { /* pid terminated due to a signal */ log_error("script %s pid %d term signal %d", scripts[i].name, scripts[i].pid, WTERMSIG(status)); scripts[i].kill_count++; scripts[i].last_result = EINTR; scripts[i].pid = 0; fail_count++; log_script(i); } else { /* pid state changed but still running */ running++; } } else { /* shouldn't happen */ log_error("script %s pid %d waitpid rv %d", scripts[i].name, scripts[i].pid, rv); log_script(i); running++; } /* option to kill script after it's run for kill_script_sec */ if (scripts[i].pid && kill_script_sec && (monotime() - scripts[i].start >= kill_script_sec)) { kill(scripts[i].pid, SIGKILL); } } if (!running) break; if (monotime() - begin >= standard_test_interval - 1) break; sleep(1); } if (!running) goto out; /* any pids that have not exited count as a failed for this cycle */ now = monotime(); for (i = 0; i < MAX_SCRIPTS; i++) { if (!scripts[i].name[0]) continue; if (!scripts[i].pid) continue; scripts[i].long_count++; fail_count++; log_error("script %s pid %d start %llu now %llu taking too long", scripts[i].name, scripts[i].pid, (unsigned long long)scripts[i].start, (unsigned long long)now); log_script(i); } out: return fail_count; } static int setup_identity(char *wdpath) { char sysfs_path[PATH_MAX] = { 0 }; char *base, *p; int fd, rv; /* * This function will be called multiple times when probing * different watchdog paths for one that works. */ itco = 0; memset(watchdog_identity, 0, sizeof(watchdog_identity)); /* * $ cat /sys/class/watchdog/watchdog0/identity * iTCO_wdt */ if (!(base = basename(wdpath))) return -1; snprintf(sysfs_path, PATH_MAX-1, "/sys/class/watchdog/%s/identity", base); if ((fd = open(sysfs_path, O_RDONLY)) < 0) return -1; rv = read(fd, watchdog_identity, WD_ID_SIZE-1); close(fd); if (rv <= 0) return -1; if ((p = strchr(watchdog_identity, '\n'))) *p = '\0'; log_debug("%s %s %s", wdpath, sysfs_path, watchdog_identity); if (!strcmp(watchdog_identity, "iTCO_wdt")) itco = 1; return 0; } static int _setup_watchdog(char *path) { struct stat buf; int rv, timeout; strncpy(watchdog_path, path, WDPATH_SIZE); watchdog_path[WDPATH_SIZE - 1] = '\0'; rv = stat(watchdog_path, &buf); if (rv < 0) return -1; rv = open_dev(); if (rv < 0) return -1; timeout = 0; rv = ioctl(dev_fd, WDIOC_GETTIMEOUT, &timeout); if (rv < 0) { log_error("%s failed to report timeout", watchdog_path); close_watchdog(); return -1; } log_debug("%s gettimeout reported %u", watchdog_path, timeout); close_watchdog(); /* TODO: save watchdog_path in /run/wdmd/saved_path, * and in startup read that file, copying it to saved_path */ return 0; } /* * Success: returns 0 with watchdog_path set. * Order of preference: * . saved path (path used before daemon restart) * . command line option (-w) * . /dev/watchdog0 * . /dev/watchdog1 * . /dev/watchdog */ static int setup_watchdog(void) { int rv; if (!saved_path[0]) goto opt; rv = _setup_watchdog(saved_path); if (!rv) return 0; opt: if (!option_path[0] || !strcmp(saved_path, option_path)) goto zero; rv = _setup_watchdog(option_path); if (!rv) return 0; zero: if (!strcmp(saved_path, "/dev/watchdog0") || !strcmp(option_path, "/dev/watchdog0")) goto one; rv = _setup_watchdog((char *)"/dev/watchdog0"); if (!rv) return 0; one: if (!strcmp(saved_path, "/dev/watchdog1") || !strcmp(option_path, "/dev/watchdog1")) goto old; rv = _setup_watchdog((char *)"/dev/watchdog1"); if (!rv) return 0; old: if (!strcmp(saved_path, "/dev/watchdog") || !strcmp(option_path, "/dev/watchdog")) goto out; rv = _setup_watchdog((char *)"/dev/watchdog"); if (!rv) return 0; out: log_error("no watchdog device, load a watchdog driver"); return -1; } /* * iTCO_wdt actual firing timeout is double the value used in get/set! * https://bugzilla.kernel.org/show_bug.cgi?id=213809 */ static int _try_timeout_itco(const char *path) { struct stat buf; int try_timeout_real, try_timeout_itco, get_timeout_real, get_timeout_itco, set_timeout_real, set_timeout_itco; int unused, fd, err, rv, rv2; rv = stat(path, &buf); if (rv < 0) { fprintf(stderr, "%s stat error %d\n", path, errno); return -1; } fd = open(path, O_WRONLY | O_CLOEXEC); if (fd < 0) { fprintf(stderr, "%s open error %d\n", path, errno); return fd; } printf("%s %s open fd %d\n", time_str(), path, fd); get_timeout_real = 0; get_timeout_itco = 0; rv = ioctl(fd, WDIOC_GETTIMEOUT, &get_timeout_itco); if (rv < 0) { fprintf(stderr, "%s gettimeout error %d\n", path, errno); rv = -1; goto out; } get_timeout_real = get_timeout_itco * 2; printf("%s %s gettimeout real %d itco %d\n", time_str(), path, get_timeout_real, get_timeout_itco); if (get_timeout_real == try_timeout) goto keepalive; try_timeout_real = try_timeout; try_timeout_itco = try_timeout_real / 2; set_timeout_real = try_timeout; set_timeout_itco = set_timeout_real / 2; rv = ioctl(fd, WDIOC_SETTIMEOUT, &set_timeout_itco); if (rv < 0) { fprintf(stderr, "%s settimeout real %d itco %d error %d\n", path, set_timeout_real, set_timeout_itco, errno); rv = -1; goto out; } set_timeout_real = set_timeout_itco * 2; printf("%s %s settimeout real %d itco %d result real %d itco %d\n", time_str(), path, try_timeout_real, try_timeout_itco, set_timeout_real, set_timeout_itco); if (set_timeout_itco != try_timeout_itco) { fprintf(stderr, "%s settimeout real %d itco %d failed\n", path, try_timeout_real, try_timeout_itco); rv = -1; goto out; } get_timeout_real = 0; get_timeout_itco = 0; rv = ioctl(fd, WDIOC_GETTIMEOUT, &get_timeout_itco); if (rv < 0) { fprintf(stderr, "%s gettimeout error %d\n", path, errno); rv = -1; goto out; } get_timeout_real = get_timeout_itco * 2; printf("%s %s gettimeout real %d itco %d\n", time_str(), path, get_timeout_real, get_timeout_itco); keepalive: rv = ioctl(fd, WDIOC_KEEPALIVE, &unused); if (rv < 0) { fprintf(stderr, "%s keepalive error %d\n", path, errno); rv = -1; goto out; } printf("%s %s keepalive fd %d result %d\n", time_str(), path, fd, rv); if (forcefire) { int sleep_sec = 0; int i; setbuf(stdout, NULL); printf("%s waiting for watchdog to reset machine:\n", time_str()); for (i = 1; i < get_timeout_real + 5; i++) { sleep(1); sleep_sec++; if (sleep_sec >= get_timeout_real+1) printf("%s %d %s failed to fire after timeout %d seconds\n", time_str(), i, path, get_timeout_real); else printf("%s %d\n", time_str(), i); } } rv = 0; out: err = write(fd, "V", 1); if (err < 0) { fprintf(stderr, "trytimeout failed to disarm %s error %d %d\n", path, err, errno); openlog("wdmd", LOG_CONS | LOG_PID, LOG_DAEMON); syslog(LOG_ERR, "trytimeout failed to disarm %s error %d %d\n", path, err, errno); } printf("%s %s disarm write V fd %d result %d\n", time_str(), path, fd, rv); rv2 = close(fd); printf("%s %s close fd %d result %d\n", time_str(), path, fd, rv2); return rv; } static int _try_timeout(const char *path) { struct stat buf; int get_timeout, set_timeout; int unused, fd, err, rv, rv2; rv = stat(path, &buf); if (rv < 0) { fprintf(stderr, "%s stat error %d\n", path, errno); return -1; } fd = open(path, O_WRONLY | O_CLOEXEC); if (fd < 0) { fprintf(stderr, "%s open error %d\n", path, errno); return fd; } printf("%s %s open fd %d\n", time_str(), path, fd); get_timeout = 0; rv = ioctl(fd, WDIOC_GETTIMEOUT, &get_timeout); if (rv < 0) { fprintf(stderr, "%s gettimeout error %d\n", path, errno); rv = -1; goto out; } printf("%s %s gettimeout %d\n", time_str(), path, get_timeout); if (get_timeout == try_timeout) goto keepalive; set_timeout = try_timeout; rv = ioctl(fd, WDIOC_SETTIMEOUT, &set_timeout); if (rv < 0) { fprintf(stderr, "%s settimeout %d error %d\n", path, set_timeout, errno); rv = -1; goto out; } printf("%s %s settimeout %d result %d\n", time_str(), path, try_timeout, set_timeout); if (set_timeout != try_timeout) { fprintf(stderr, "%s settimeout %d failed\n", path, try_timeout); rv = -1; goto out; } get_timeout = 0; rv = ioctl(fd, WDIOC_GETTIMEOUT, &get_timeout); if (rv < 0) { fprintf(stderr, "%s gettimeout error %d\n", path, errno); rv = -1; goto out; } printf("%s %s gettimeout %d\n", time_str(), path, get_timeout); keepalive: rv = ioctl(fd, WDIOC_KEEPALIVE, &unused); if (rv < 0) { fprintf(stderr, "%s keepalive error %d\n", path, errno); rv = -1; goto out; } printf("%s %s keepalive fd %d result %d\n", time_str(), path, fd, rv); if (forcefire) { int sleep_sec = 0; int i; setbuf(stdout, NULL); printf("%s waiting for watchdog to reset machine:\n", time_str()); for (i = 1; i < get_timeout + 5; i++) { sleep(1); sleep_sec++; if (sleep_sec >= get_timeout+1) printf("%s %d %s failed to fire after timeout %d seconds\n", time_str(), i, path, get_timeout); else printf("%s %d\n", time_str(), i); } } rv = 0; out: err = write(fd, "V", 1); if (err < 0) { fprintf(stderr, "trytimeout failed to disarm %s error %d %d\n", path, err, errno); openlog("wdmd", LOG_CONS | LOG_PID, LOG_DAEMON); syslog(LOG_ERR, "trytimeout failed to disarm %s error %d %d\n", path, err, errno); } printf("%s %s disarm write V fd %d result %d\n", time_str(), path, fd, rv); rv2 = close(fd); printf("%s %s close fd %d result %d\n", time_str(), path, fd, rv2); return rv; } static int _probe_dev_itco(const char *path) { struct stat buf; int fd, err, rv, timeout_real, timeout_itco; rv = stat(path, &buf); if (rv < 0) { fprintf(stderr, "error %d stat %s\n", errno, path); return -1; } fd = open(path, O_WRONLY | O_CLOEXEC); if (fd < 0) { fprintf(stderr, "error %d open %s\n", errno, path); return fd; } timeout_real = 0; timeout_itco = 0; rv = ioctl(fd, WDIOC_GETTIMEOUT, &timeout_itco); if (rv < 0) { fprintf(stderr, "error %d ioctl gettimeout %s\n", errno, path); rv = -1; goto out; } timeout_real = timeout_itco * 2; if (timeout_real == fire_timeout) { printf("%s\n", path); rv = 0; goto out; } timeout_real = fire_timeout; timeout_itco = timeout_real / 2; rv = ioctl(fd, WDIOC_SETTIMEOUT, &timeout_itco); if (rv < 0) { fprintf(stderr, "error %d ioctl settimeout %s\n", errno, path); rv = -1; goto out; } timeout_real = timeout_itco * 2; if (timeout_real != fire_timeout) { fprintf(stderr, "error %d invalid timeout %s\n", errno, path); rv = -1; goto out; } printf("%s\n", path); rv = 0; out: err = write(fd, "V", 1); if (err < 0) { fprintf(stderr, "probe failed to disarm %s error %d %d\n", path, err, errno); openlog("wdmd", LOG_CONS | LOG_PID, LOG_DAEMON); syslog(LOG_ERR, "probe failed to disarm %s error %d %d\n", path, err, errno); } close(fd); return rv; } static int _probe_dev(const char *path) { struct stat buf; int fd, err, rv, timeout; rv = stat(path, &buf); if (rv < 0) { fprintf(stderr, "error %d stat %s\n", errno, path); return -1; } fd = open(path, O_WRONLY | O_CLOEXEC); if (fd < 0) { fprintf(stderr, "error %d open %s\n", errno, path); return fd; } timeout = 0; rv = ioctl(fd, WDIOC_GETTIMEOUT, &timeout); if (rv < 0) { fprintf(stderr, "error %d ioctl gettimeout %s\n", errno, path); rv = -1; goto out; } if (timeout == fire_timeout) { printf("%s\n", path); rv = 0; goto out; } timeout = fire_timeout; rv = ioctl(fd, WDIOC_SETTIMEOUT, &timeout); if (rv < 0) { fprintf(stderr, "error %d ioctl settimeout %s\n", errno, path); rv = -1; goto out; } if (timeout != fire_timeout) { fprintf(stderr, "error %d invalid timeout %s\n", errno, path); rv = -1; goto out; } printf("%s\n", path); rv = 0; out: err = write(fd, "V", 1); if (err < 0) { fprintf(stderr, "probe failed to disarm %s error %d %d\n", path, err, errno); openlog("wdmd", LOG_CONS | LOG_PID, LOG_DAEMON); syslog(LOG_ERR, "probe failed to disarm %s error %d %d\n", path, err, errno); } close(fd); return rv; } static int probe_dev(const char *wdpath) { char *path = (char *)wdpath; setup_identity(path); /* sets itco=1 if iTCO_wdt */ if (try_timeout) { /* * Used to test support for a given timeout with: wdmd -t * or to test firing for a given timeout with: wdmd -F -t */ if (itco) return _try_timeout_itco(path); else return _try_timeout(path); } else { /* * Used to print on stdout just the path of the watchdog device * that wdmd would use with: wdmd -p */ if (itco) return _probe_dev_itco(path); else return _probe_dev(path); } } /* * Confusingly, this is the top level function for both * wdmd -t (test timeout) and wdmd -p (print functional watchdog device). */ static int probe_watchdog(void) { int rv; if (!saved_path[0]) goto opt; rv = probe_dev(saved_path); if (!rv) return 0; opt: if (!option_path[0] || !strcmp(saved_path, option_path)) goto zero; rv = probe_dev(option_path); if (!rv) return 0; zero: if (!strcmp(saved_path, "/dev/watchdog0") || !strcmp(option_path, "/dev/watchdog0")) goto one; rv = probe_dev((char *)"/dev/watchdog0"); if (!rv) return 0; one: if (!strcmp(saved_path, "/dev/watchdog1") || !strcmp(option_path, "/dev/watchdog1")) goto old; rv = probe_dev((char *)"/dev/watchdog1"); if (!rv) return 0; old: if (!strcmp(saved_path, "/dev/watchdog") || !strcmp(option_path, "/dev/watchdog")) goto out; rv = probe_dev((char *)"/dev/watchdog"); if (!rv) return 0; out: fprintf(stderr, "no watchdog device, load a watchdog driver\n"); return -1; } static void process_signals(int ci) { struct signalfd_siginfo fdsi; ssize_t rv; int fd = client[ci].fd; rv = read(fd, &fdsi, sizeof(struct signalfd_siginfo)); if (rv != sizeof(struct signalfd_siginfo)) { return; } if ((fdsi.ssi_signo == SIGTERM) || (fdsi.ssi_signo == SIGINT)) { if (!active_clients()) daemon_quit = 1; } if (fdsi.ssi_signo == SIGHUP) { setup_scripts(); } } static int setup_signals(void) { sigset_t mask; int fd, rv, ci; sigemptyset(&mask); sigaddset(&mask, SIGTERM); sigaddset(&mask, SIGINT); sigaddset(&mask, SIGHUP); rv = sigprocmask(SIG_BLOCK, &mask, NULL); if (rv < 0) return rv; fd = signalfd(-1, &mask, 0); if (fd < 0) return -errno; ci = client_add(fd, process_signals, client_pid_dead); strncpy(client[ci].name, "signal", WDMD_NAME_SIZE); client[ci].internal = 1; return 0; } /* * We're trying to detect whether the last wdmd exited uncleanly and the * system has not been reset since. In that case we don't want to start * and open /dev/watchdog, because that will ping the wd which will extend * the pending reset, which needs to happen on schedule. * * To detect this, we want to do/set something on the system that will * not go away (be cleared) if we exit, but will go away if the system * is reset. If we were certain there was a tmpfs file system we could * use, then we could create a file there and just refuse to start if * the file exists. * * Until we are certain of tmpfs somewhere, create a shared mem object * on the system. */ static int setup_shm(void) { int rv; rv = shm_open("/wdmd", O_RDWR|O_CREAT|O_EXCL, S_IRUSR|S_IWUSR|S_IRGRP|S_IROTH); if (rv < 0) { log_error("other wdmd not cleanly stopped, shm_open error %d", errno); return rv; } shm_fd = rv; return 0; } static void close_shm(void) { shm_unlink("/wdmd"); close(shm_fd); } static int test_loop(void) { void (*workfn) (int ci); void (*deadfn) (int ci); uint64_t test_time; int resetting = 0; int active_usage, external_usage; int poll_timeout; int sleep_seconds; int fail_count; int rv, i; test_time = 0; poll_timeout = test_interval * 1000; while (1) { rv = poll(pollfd, client_maxi + 1, poll_timeout); if (rv == -1 && errno == EINTR) continue; if (rv < 0) { /* not sure */ } for (i = 0; i <= client_maxi; i++) { if (client[i].fd < 0) continue; if (pollfd[i].revents & POLLIN) { workfn = client[i].workfn; if (workfn) workfn(i); } if (pollfd[i].revents & (POLLERR | POLLHUP | POLLNVAL)) { deadfn = client[i].deadfn; if (deadfn) deadfn(i); } } count_clients(&active_usage, &external_usage); if (daemon_quit && !active_usage) break; /* * No client has called open_watchdog() so the wd device is not open yet. */ if (!test_loop_enable) continue; /* * active_usage are client connections with a refcount. * external_usage are any clients other than internal. * (open_watchdog happens with external but not active * connections.) * * checking resetting here is critical to avoiding * unnecessary resets: while in recovery mode we * have done close_watchdog_unclean, then all clients * are cleared, and we need the loop below to see * no further failures and reopen and pet the watchdog * again to avoid a reset. After it's been reopened, * and no longer used due to all clients being cleared, * then it's ok to get here and close cleanly. */ if (!active_usage && !external_usage && !resetting) { log_debug("close watchdog unused"); close_watchdog(); test_loop_enable = 0; test_interval = standard_test_interval; poll_timeout = test_interval * 1000; test_time = 0; continue; } if (monotime() - test_time >= test_interval) { test_time = monotime(); log_debug("test_time %llu", (unsigned long long)test_time); fail_count = 0; fail_count += test_files(); fail_count += test_scripts(); fail_count += test_clients(); if (!fail_count) { if (dev_fd == -1) { open_dev(); pet_watchdog(); log_error("%s reopen", watchdog_path); } else { pet_watchdog(); } test_interval = standard_test_interval; resetting = 0; } else { /* If we can patch the kernel so that close does not generate a ping, then we can skip this close, and just not pet the device in this case. Also see test_client above. */ close_watchdog_unclean(); test_interval = RECOVER_TEST_INTERVAL; resetting = 1; } } sleep_seconds = test_time + test_interval - monotime(); poll_timeout = (sleep_seconds > 0) ? sleep_seconds * 1000 : 500; log_debug("test_interval %d sleep_seconds %d poll_timeout %d", test_interval, sleep_seconds, poll_timeout); } return 0; } static int lockfile(void) { char buf[16]; struct flock lock; mode_t old_umask; int fd, rv; old_umask = umask(0022); rv = mkdir(WDMD_RUN_DIR, 0775); if (rv < 0 && errno != EEXIST) { umask(old_umask); return rv; } umask(old_umask); sprintf(lockfile_path, "%s/wdmd.pid", WDMD_RUN_DIR); fd = open(lockfile_path, O_CREAT|O_WRONLY|O_CLOEXEC, 0644); if (fd < 0) { log_error("lockfile open error %s: %s", lockfile_path, strerror(errno)); return -1; } lock.l_type = F_WRLCK; lock.l_start = 0; lock.l_whence = SEEK_SET; lock.l_len = 0; rv = fcntl(fd, F_SETLK, &lock); if (rv < 0) { log_error("daemon already running (lockfile %s %d)", lockfile_path, errno); goto fail; } rv = ftruncate(fd, 0); if (rv < 0) { log_error("lockfile truncate error %s: %s", lockfile_path, strerror(errno)); goto fail; } memset(buf, 0, sizeof(buf)); snprintf(buf, sizeof(buf), "%d\n", getpid()); rv = write(fd, buf, strlen(buf)); if (rv <= 0) { log_error("lockfile write error %s: %s", lockfile_path, strerror(errno)); goto fail; } return fd; fail: close(fd); return -1; } static void setup_priority(void) { struct sched_param sched_param; int rv; if (!high_priority) return; rv = mlockall(MCL_CURRENT | MCL_FUTURE); if (rv < 0) { log_info("mlockall failed"); } rv = sched_get_priority_max(SCHED_RR); if (rv < 0) { log_info("could not get max scheduler priority err %d", errno); return; } sched_param.sched_priority = rv; rv = sched_setscheduler(0, SCHED_RR|SCHED_RESET_ON_FORK, &sched_param); if (rv < 0) { log_info("could not set RR|RESET_ON_FORK priority %d err %d", sched_param.sched_priority, errno); } } static int group_to_gid(char *arg) { struct group *gr; gr = getgrnam(arg); if (gr == NULL) { log_info("group '%s' not found, using socket gid: %i", arg, DEFAULT_SOCKET_GID); return DEFAULT_SOCKET_GID; } return gr->gr_gid; } static void print_debug_and_exit(void) { struct sockaddr_un addr; struct wdmd_header h; int rv, s; s = socket(AF_LOCAL, SOCK_STREAM, 0); if (s < 0) exit(1); rv = wdmd_socket_address(&addr); if (rv < 0) exit(1); rv = connect(s, (struct sockaddr *) &addr, sizeof(struct sockaddr_un)); if (rv < 0) exit(1); memset(&h, 0, sizeof(h)); h.cmd = CMD_DUMP_DEBUG; rv = send(s, (void *)&h, sizeof(struct wdmd_header), 0); if (rv < 0) exit(1); rv = recv(s, &debug_buf, DEBUG_SIZE, 0); if (rv < 0) exit(1); rv = write(STDOUT_FILENO, debug_buf, strlen(debug_buf)); exit(0); } static void print_usage_and_exit(int status) { printf("Usage:\n"); printf("wdmd [options]\n\n"); printf("--version, -V print version\n"); printf("--help, -h print usage\n"); printf("--dump, -d print debug from daemon\n"); printf("--probe, -p print path of functional watchdog device\n"); printf("--trytimeout, -t set the timeout value for watchdog device\n"); printf("--forcefire, -F force watchdog to fire and reset machine, use with -t\n"); printf("-D debug: no fork and print all logging to stderr\n"); printf("-H 0|1 use high priority features (1 yes, 0 no, default %d)\n", DEFAULT_HIGH_PRIORITY); printf("-G group ownership for the socket\n"); printf("-S 0|1 allow script tests (default %d)\n", allow_scripts); printf("-s path to scripts dir (default %s)\n", scripts_dir); printf("-k kill unfinished scripts after num seconds (default %d)\n", kill_script_sec); printf("-w path to the watchdog device to try first\n"); exit(status); } static void print_version_and_exit(void) { printf("wdmd version %s\n", VERSION); exit(0); } /* If wdmd exits abnormally, /dev/watchdog will eventually fire, and clients can detect wdmd is gone and begin to shut down cleanly ahead of the reset. But what if wdmd is restarted before the wd fires? It will begin petting /dev/watchdog again, leaving the previous clients unprotected. I don't know if this situation is important enough to try to prevent. One way would be for wdmd to fail starting if it found a pid file left over from its previous run. */ int main(int argc, char *argv[]) { int do_probe = 0; int rv; while (1) { int c; int option_index = 0; static struct option long_options[] = { {"help", no_argument, 0, 'h' }, {"probe", no_argument, 0, 'p' }, {"dump", no_argument, 0, 'd' }, {"trytimeout", required_argument, 0, 't' }, {"forcefire", no_argument, 0, 'F' }, {"version", no_argument, 0, 'V' }, {0, 0, 0, 0 } }; c = getopt_long(argc, argv, "hpdVDH:G:S:s:k:w:t:F", long_options, &option_index); if (c == -1) break; switch (c) { case 'h': print_usage_and_exit(0); break; case 'p': do_probe = 1; break; case 't': do_probe = 1; try_timeout = atoi(optarg); break; case 'F': forcefire = 1; break; case 'd': print_debug_and_exit(); break; case 'V': print_version_and_exit(); break; case 'D': daemon_debug = 1; break; case 'G': socket_gname = strdup(optarg); break; case 'H': high_priority = atoi(optarg); break; case 'S': allow_scripts = atoi(optarg); break; case 's': scripts_dir = strdup(optarg); break; case 'k': kill_script_sec = atoi(optarg); break; case 'w': snprintf(option_path, WDPATH_SIZE, "%s", optarg); option_path[WDPATH_SIZE - 1] = '\0'; break; } } if (forcefire && !do_probe) { fprintf(stderr, "Use force fire (-F) with a timeout (-t).\n"); exit(EXIT_FAILURE); } if (do_probe) { rv = setup_shm(); if (rv < 0) { fprintf(stderr, "cannot probe watchdog devices while wdmd is in use.\n"); openlog("wdmd-probe", LOG_CONS | LOG_PID, LOG_DAEMON); syslog(LOG_ERR, "cannot probe watchdog devices while wdmd is in use.\n"); exit(EXIT_FAILURE); } rv = probe_watchdog(); close_shm(); if (rv < 0) exit(EXIT_FAILURE); else exit(EXIT_SUCCESS); } socket_gid = group_to_gid(socket_gname); if (!daemon_debug) { if (daemon(0, 0) < 0) { fprintf(stderr, "cannot fork daemon\n"); exit(EXIT_FAILURE); } } openlog("wdmd", LOG_CONS | LOG_PID, LOG_DAEMON); setup_priority(); rv = lockfile(); if (rv < 0) goto out; rv = setup_shm(); if (rv < 0) goto out_lockfile; rv = setup_signals(); if (rv < 0) goto out_shm; rv = setup_scripts(); if (rv < 0) goto out_lockfile; rv = setup_files(); if (rv < 0) goto out_scripts; rv = setup_clients(); if (rv < 0) goto out_files; /* Sets watchdog_path */ rv = setup_watchdog(); if (rv < 0) goto out_clients; /* Sets watchdog_identity and itco */ setup_identity(watchdog_path); log_info("wdmd started S%d H%d G%d using %s \"%s\"", allow_scripts, high_priority, socket_gid, watchdog_path, watchdog_identity[0] ? watchdog_identity : "unknown"); daemon_setup_done = 1; rv = test_loop(); close_watchdog(); out_clients: close_clients(); out_files: close_files(); out_scripts: close_scripts(); out_shm: close_shm(); out_lockfile: unlink(lockfile_path); out: return rv; } sanlock/wdmd/wdmd.8000066400000000000000000000065411516326262600145060ustar00rootroot00000000000000.TH WDMD 8 2011-08-01 .SH NAME wdmd \- watchdog multiplexing daemon .SH SYNOPSIS .B wdmd [OPTIONS] .SH DESCRIPTION This daemon opens /dev/watchdog and allows multiple independent sources to detmermine whether each KEEPALIVE is done. Every test interval (default 10 seconds), the daemon tests each source. If any test fails, the KEEPALIVE is not done. In the default configuration, the watchdog timer will reset the system if no KEEPALIVE is done for 60 seconds ("fire timeout"). This means that if a single test fails 5-6 times in row, the watchdog will fire and reset the system. With multiple test sources, fewer separate failures back to back can also cause a reset, e.g. T seconds, P pass, F fail .br T00: test1 P, test2 P, test3 P: KEEPALIVE done .br T10: test1 F, test2 F, test3 P: KEEPALIVE skipped .br T20: test1 F, test2 P, test3 P: KEEPALIVE skipped .br T30: test1 P, test2 F, test3 P: KEEPALIVE skipped .br T40: test1 P, test2 P, test3 F: KEEPALIVE skipped .br T50: test1 F, test2 F, test3 P: KEEPALIVE skipped .br T60: test1 P, test2 F, test3 P: KEEPALIVE skipped .br T60: watchdog fires, system resets (Depending on timings, the system may be reset sometime shortly before T60, and the tests at T60 would not be run.) A crucial aspect to the design and function of wdmd is that if any single source does not pass the test for the length of the fire timeout, the watchdog is guaranteed to fire, regardless of whether other sources on the system have passed or failed. A spurious reset due to the combined effects of multiple failing tests as shown above, is an accepted side effect. The wdmd init script will load the softdog module if no other watchdog module has been loaded. wdmd cannot be used on the system with any other program that needs to open /dev/watchdog, e.g. watchdog(8). .SS Test Source: clients Using libwdmd, programs connect to wdmd via a unix socket, and send regular messages to wdmd to update an expiry time for their connection. Every test interval, wdmd will check if the expiry time for a connection has been reached. If so, the test for that client fails. .SS Test Source: scripts wdmd will run scripts from a designated directory every test interval. If a script exits with 0, the test is considered a success, otherwise a failure. If a script does not exit by the end of the test interval, it is considered a failure. .SH OPTIONS .TP .B \-\-version, \-V Print version. .TP .B \-\-help, \-h Print usage. .TP .B \-\-dump, \-d Print debug information from the daemon. .TP .B \-\-probe, \-p Print path of functional watchdog device. Exit code 0 indicates a functional device was found. Exit code 1 indicates a functional device was not found. .TP .B \-D Enable debugging to stderr and don't fork. .TP .BI \-H " 0|1" Enable (1) or disable (0) high priority features such as realtime scheduling priority and mlockall. .TP .BI \-G " name" Group ownership for the socket. .TP .BI \-S " 0|1" Enable (1) or disable (0) script tests. .TP .BI \-s " path" Path to scripts dir. .TP .BI \-k " num" Kill unfinished scripts after num seconds. .TP .BI \-w " path" The path to the watchdog device to try first. .TP .BI "\-\-trytimeout, \-t" " seconds" Set the timeout for the watchdog device. Use this to check for supported timeout values. .TP .B \-\-forcefire, \-F Force the watchdog to fire and reset the machine. Use with -t. sanlock/wdmd/wdmd.h000066400000000000000000000013321516326262600145570ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. */ #ifndef __WDMD_H__ #define __WDMD_H__ #define WDMD_NAME_SIZE 128 int wdmd_connect(void); int wdmd_open_watchdog(int con, int fire_timeout); int wdmd_register(int con, char *name); int wdmd_refcount_set(int con); int wdmd_refcount_clear(int con); int wdmd_test_live(int con, uint64_t renewal_time, uint64_t expire_time); int wdmd_status(int con, int *test_interval, int *fire_timeout, uint64_t *last_keepalive); #endif sanlock/wdmd/wdmd_client.c000066400000000000000000000027741516326262600161230ustar00rootroot00000000000000/* * Copyright 2011 Red Hat, Inc. * * This copyrighted material is made available to anyone wishing to use, * modify, copy, or redistribute it subject to the terms and conditions * of the GNU General Public License v2 or (at your option) any later version. */ #include #include #include #include #include #include #include #include #include #include #include "wdmd.h" int main(int argc, char *argv[]) { char name[WDMD_NAME_SIZE]; uint64_t t, last_keepalive; int test_interval, fire_timeout; int con, rv; int i = 0; int iter = 10; if (argc > 1) iter = atoi(argv[1]); memset(name, 0, sizeof(name)); sprintf(name, "%s", "wdmd_client"); con = wdmd_connect(); printf("wdmd_connect %d\n", con); if (con < 0) return con; rv = wdmd_register(con, name); printf("wdmd_register %d\n", rv); if (rv < 0) return rv; rv = wdmd_status(con, &test_interval, &fire_timeout, &last_keepalive); printf("wdmd_status %d test_interval %d fire_timeout %d last_keepalive %llu\n", rv, test_interval, fire_timeout, (unsigned long long)last_keepalive); if (rv < 0) return rv; while (1) { sleep(10); t = time(NULL); rv = wdmd_test_live(con, t, t + 40); printf("wdmd_test_live %d %llu %llu\n", rv, (unsigned long long)t, (unsigned long long)(t + 40)); if (i++ > iter) break; } rv = wdmd_test_live(con, t, 0); printf("wdmd_test_live 0 %d\n", rv); return 0; } sanlock/wdmd/wdmd_sock.c000066400000000000000000000013421516326262600155720ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. */ #include #include #include #include #include #include #include #include "wdmd.h" #include "wdmd_sock.h" int wdmd_socket_address(struct sockaddr_un *addr) { memset(addr, 0, sizeof(struct sockaddr_un)); addr->sun_family = AF_LOCAL; snprintf(addr->sun_path, sizeof(addr->sun_path) - 1, "%s/%s", WDMD_RUN_DIR, WDMD_SOCKET_NAME); return 0; } sanlock/wdmd/wdmd_sock.h000066400000000000000000000015371516326262600156050ustar00rootroot00000000000000/* * Copyright 2010-2011 Red Hat, Inc. * * This library is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public * License as published by the Free Software Foundation; either * version 2.1 of the License, or (at your option) any later version. */ #ifndef __WDMD_SOCK_H__ #define __WDMD_SOCK_H__ #define WDMD_RUN_DIR "/run/wdmd" #define WDMD_SOCKET_NAME "wdmd.sock" enum { CMD_REGISTER = 1, CMD_REFCOUNT_SET, CMD_REFCOUNT_CLEAR, CMD_TEST_LIVE, CMD_STATUS, CMD_DUMP_DEBUG, CMD_OPEN_WATCHDOG, }; struct wdmd_header { uint32_t magic; uint32_t cmd; uint32_t len; uint32_t flags; uint32_t test_interval; uint32_t fire_timeout; uint64_t last_keepalive; uint64_t renewal_time; uint64_t expire_time; char name[WDMD_NAME_SIZE]; }; int wdmd_socket_address(struct sockaddr_un *addr); #endif