lua: Increase reserved stack space for FreeBSD in debug config

FreeBSD uses more stack space in debug configurations and can overflow
the stack while formatting the error message when the call depth limit
of 20 frames is reached.  This is readily reproduced by running the
gsub recursion test with increased kstack size.  I hit the panic with
16 pages per kstack, and noticed it go away when bumped to 17.

Reserve an additional 64 bytes on the stack when building for FreeBSD.
This is enough to avoid the panic with a deep stack while not wasting
too much space when the default stack size is used.

Reviewed-by: Matt Ahrens <[email protected]>
Reviewed-by: Brian Behlendorf <[email protected]>
Signed-off-by: Ryan Moeller <[email protected]>
Closes #10634 

1 files changed, 9 insertions, 0 deletions

diff --git a/module/lua/llimits.h b/module/lua/llimits.h
index 177092fbc..25466f14e 100644
--- a/module/lua/llimits.h
+++ b/module/lua/llimits.h
@@ -126,7 +126,16 @@ typedef LUAI_UACNUMBER l_uacNumber;
  * Minimum amount of available stack space (in bytes) to make a C call.  With
  * gsub() recursion, the stack space between each luaD_call() is 1256 bytes.
  */
+#if defined(__FreeBSD__)
+/*
+ * FreeBSD needs a few extra bytes in unoptimized debug builds to avoid a
+ * double-fault handling the error when the max call depth is exceeded just
+ * before the C stack runs out.  64 bytes seems to do the trick.
+ */
+#define LUAI_MINCSTACK		4160
+#else
 #define LUAI_MINCSTACK		4096
+#endif
 
 /*
 ** maximum number of upvalues in a closure (both C and Lua). (Value