Remove default GFP_NOFS allocations

As originally described in commit 82b8c8fa64
this was done to prevent certain deadlocks from occuring in the system.
However, as suspected the price for doing this proved to be too high.
The VM is having a hard time effectively reclaiming memory thus we are
reverting this change.

However, we still need to fundamentally handle the issue.  Under
Solaris the KM_PUSHPAGE mask is used commonly in I/O paths to ensure
a memory allocations will succeed.  We leverage this fact and redefine
KM_PUSHPAGE to include GFP_NOFS.  This ensures that in these common
I/O path we don't trigger additional reclaim.  This minimizes the
change to the Solaris code.

include/sys/kmem.h

@@ -41,13 +41,12 @@
 /*
  * Memory allocation interfaces
  */
-#define KM_SLEEP                        GFP_NOFS
-#define KM_NOSLEEP                      GFP_ATOMIC
-#undef  KM_PANIC                        /* No linux analog */
-#define KM_PUSHPAGE                     (KM_SLEEP | __GFP_HIGH)
-#define KM_VMFLAGS                      GFP_LEVEL_MASK
+#define KM_SLEEP	GFP_KERNEL	/* Can sleep, never fails */
+#define KM_NOSLEEP	GFP_ATOMIC	/* Can not sleep, may fail */
+#define KM_PUSHPAGE	(GFP_NOFS | __GFP_HIGH)	/* Use reserved memory */
+#define KM_NODEBUG	__GFP_NOWARN	/* Suppress warnings */
 #define KM_FLAGS	__GFP_BITS_MASK
-#define KM_NODEBUG                      __GFP_NOWARN
+#define KM_VMFLAGS	GFP_LEVEL_MASK
 
 /*
  * Used internally, the kernel does not need to support this flag