glsl: rewrite glsl_type::record_key_hash() to avoid buffer overflow

This should be more efficient than the previous snprintf() solution.
But more importantly, it avoids a buffer overflow bug that could result
in crashes or unpredictable results when processing very large interface
blocks.

For the app in question, key->length = 103 for some interfaces.  The check
if size >= sizeof(hash_key) was insufficient to prevent overflows of the
hash_key[128] array because it didn't account for the terminating zero.
In this case, this caused the call to hash_table_string_hash() to return
different results for identical inputs, and then shader linking failed.

This new solution also takes all structure fields into account instead
of just the first 15 when sizeof(pointer)==8.

Cc: [email protected]
Reviewed-by: Ian Romanick <[email protected]>

1 files changed, 13 insertions, 10 deletions

diff --git a/src/glsl/glsl_types.cpp b/src/glsl/glsl_types.cpp
index 4aa36a79490..9c9b7efcbc7 100644
--- a/src/glsl/glsl_types.cpp
+++ b/src/glsl/glsl_types.cpp
@@ -738,24 +738,27 @@ glsl_type::record_key_compare(const void *a, const void *b)
 }
 
 
+/**
+ * Generate an integer hash value for a glsl_type structure type.
+ */
 unsigned
 glsl_type::record_key_hash(const void *a)
 {
    const glsl_type *const key = (glsl_type *) a;
-   char hash_key[128];
-   unsigned size = 0;
-
-   size = snprintf(hash_key, sizeof(hash_key), "%08x", key->length);
+   uintptr_t hash = key->length;
+   unsigned retval;
 
    for (unsigned i = 0; i < key->length; i++) {
-      if (size >= sizeof(hash_key))
-	 break;
-
-      size += snprintf(& hash_key[size], sizeof(hash_key) - size,
-		       "%p", (void *) key->fields.structure[i].type);
+      /* casting pointer to uintptr_t */
+      hash = (hash * 13 ) + (uintptr_t) key->fields.structure[i].type;
    }
 
-   return hash_table_string_hash(& hash_key);
+   if (sizeof(hash) == 8)
+      retval = (hash & 0xffffffff) ^ ((uint64_t) hash >> 32);
+   else
+      retval = hash;
+
+   return retval;
 }