libspl/assert: add lock around assertion output

If multiple threads trip an assertion at the same moment (quite common),
they can be printing at the same time, and their output gets messy.

This adds a simple lock around the whole thing, to prevent a second task
printing assert output before the first has finished.

Additionally, if libspl_assert_ok is not set, abort() is called without
dropping the lock, so that any other asserting tasks will be killed
before starting any output, rather than only getting part-way through.
This is a tradeoff; it's assumed that multiple threads asserting at the
same moment are likely the same fault in different instances of a
thread, and so there won't be any more useful information from the other
tasks anyway.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Rob Norris <robn@despairlabs.com>
Sponsored-by: https://despairlabs.com/sponsor/
Closes #16140

lib/libspl/assert.c

@@ -27,6 +27,7 @@
  */
 
 #include <assert.h>
+#include <pthread.h>
 
 #if defined(__linux__)
 #include <errno.h>
@@ -56,11 +57,15 @@ libspl_set_assert_ok(boolean_t val)
 	libspl_assert_ok = val;
 }
 
+static pthread_mutex_t assert_lock = PTHREAD_MUTEX_INITIALIZER;
+
 /* printf version of libspl_assert */
 void
 libspl_assertf(const char *file, const char *func, int line,
     const char *format, ...)
 {
+	pthread_mutex_lock(&assert_lock);
+
 	va_list args;
 	char tname[64];
 
@@ -80,6 +85,7 @@ libspl_assertf(const char *file, const char *func, int line,
 
 #if !__has_feature(attribute_analyzer_noreturn) && !defined(__COVERITY__)
 	if (libspl_assert_ok) {
+		pthread_mutex_unlock(&assert_lock);
 		return;
 	}
 #endif