ZTS: Provide for nested cleanup routines

Shared test library functions lack a simple way to ensure proper
cleanup in the event of a failure.  The `log_onexit` cleanup pattern
cannot be used in library functions because it uses one global
variable to store the cleanup command.

An example of where this is a serious issue is when a tunable that
artifically stalls kernel progress gets activated and then some check
fails.  Unless the caller knows about the tunable and sets it back,
the system will be left in a bad state.

To solve this problem, turn the global cleanup variable into a stack.
Provide push and pop functions to add additional cleanup steps and
remove them after it is safe again.

The first use of this new functionality is in attempt_during_removal,
which sets REMOVAL_SUSPEND_PROGRESS.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: John Kennedy <john.kennedy@delphix.com>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #10080

tests/test-runner/include/logapi.shlib

@@ -281,7 +281,23 @@ function log_pos
 
 function log_onexit
 {
-	_CLEANUP="$@"
+	_CLEANUP=("$*")
+}
+
+# Push an exit handler on the cleanup stack
+#
+# $@ - function(s) to perform on exit
+
+function log_onexit_push
+{
+	_CLEANUP+=("$*")
+}
+
+# Pop an exit handler off the cleanup stack
+
+function log_onexit_pop
+{
+	_CLEANUP=("${_CLEANUP[@]:0:${#_CLEANUP[@]}-1}")
 }
 
 #
@@ -425,12 +441,14 @@ function _endlog
 		_execute_testfail_callbacks
 	fi
 
-	if [[ -n $_CLEANUP ]] ; then
+	typeset stack=("${_CLEANUP[@]}")
-		typeset cleanup=$_CLEANUP
 	log_onexit ""
+	typeset i=${#stack[@]}
+	while (( i-- )); do
+		typeset cleanup="${stack[i]}"
 		log_note "Performing local cleanup via log_onexit ($cleanup)"
 		$cleanup
-	fi
+	done
 
 	exit $exitcode
 }

tests/zfs-tests/tests/functional/removal/removal.kshlib

@@ -60,6 +60,7 @@ function attempt_during_removal # pool disk callback [args]
 	typeset callback=$3
 
 	shift 3
+	log_onexit_push set_tunable32 REMOVAL_SUSPEND_PROGRESS 0
 	set_tunable32 REMOVAL_SUSPEND_PROGRESS 1
 
 	log_must zpool remove $pool $disk
@@ -80,6 +81,7 @@ function attempt_during_removal # pool disk callback [args]
 	log_must is_pool_removing $pool
 
 	set_tunable32 REMOVAL_SUSPEND_PROGRESS 0
+	log_onexit_pop
 
 	log_must wait_for_removal $pool
 	log_mustnot vdevs_in_pool $pool $disk