diff options
| author | Adam Jackson <[email protected]> | 2011-05-20 18:21:15 -0400 |
|---|---|---|
| committer | Adam Jackson <[email protected]> | 2011-05-23 11:07:34 -0400 |
| commit | e8b1c6d6f55f5be3bef25084fdd8b6127517e137 (patch) | |
| tree | 0bbb1b20dd02eea0f360d037b554b5c8f38db37c /src/mesa/main/formats.h | |
| parent | d0c6d24a9ebe31b3a805e8f5845fc7e6fa552a98 (diff) | |
mesa: Fix return type of _mesa_get_format_bytes() (#37351)
Despite that negative values aren't sensible here, making this unsigned
is dangerous. Consider get_pointer_generic, which computes a value of
the form:
void *base + (int x * int stride + int y) * unsigned bpp
The usual arithmetic conversions will coerce the (x*stride + y)
subexpression to unsigned. Since stride can be negative, this is
disastrous.
Fixes at least the following piglit tests on Ironlake:
fbo/fbo-blit-d24s8
spec/ARB_depth_texture/fbo-clear-formats
spec/EXT_packed_depth_stencil/fbo-clear-formats
NOTE: This is a candidate for the 7.10 branch.
Reviewed-by: Chad Versace <[email protected]>
Signed-off-by: Adam Jackson <[email protected]>
Diffstat (limited to 'src/mesa/main/formats.h')
| -rw-r--r-- | src/mesa/main/formats.h | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/src/mesa/main/formats.h b/src/mesa/main/formats.h index df9ed70a3e0..0640bbc4af1 100644 --- a/src/mesa/main/formats.h +++ b/src/mesa/main/formats.h @@ -216,7 +216,7 @@ typedef enum extern const char * _mesa_get_format_name(gl_format format); -extern GLuint +extern GLint _mesa_get_format_bytes(gl_format format); extern GLint |