summaryrefslogtreecommitdiffstats
path: root/src/mesa/drivers
Commit message (Collapse)AuthorAgeFilesLines
* i965: Implement ARB_fragment_layer_viewport.Kenneth Graunke2015-10-284-1/+46
| | | | | | | | | | | | | | Normally, we could read gl_Layer from bits 26:16 of R0.0. However, the specification requires that bogus out-of-range 32-bit values written by previous stages need to appear in the fragment shader as-written. Instead, we pass in the full 32-bit value from the VUE header as an extra flat-shaded varying. We have the SF override the value to 0 when the previous stage didn't actually write a value (it's actually defined to return 0). Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Chris Forbes <[email protected]>
* i965: Make calculate_attr_overrides return the URB read offset.Kenneth Graunke2015-10-284-10/+17
| | | | | | | | | | | | Traditionally, we've hardcoded "URB Entry Read Offset" to 1 (which represents 2 vec4 varying slots) to skip over the 8 DWord VUE header. In order to support ARB_fragment_layer_viewport, we'll need to read from that header. This patch adds the basic plumbing necessary to calculate a value dynamically and hook it up in the SBE packets. Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Chris Forbes <[email protected]>
* i965/fs: Properly check for PAD in fragment shaders with > 16 varyings.Kenneth Graunke2015-10-281-4/+1
| | | | | | | | | | | | | | Commit 268008f98c3810b9f276df985dc93efc0c49f33e changed unused VUE map slots to be initialized with BRW_VARYING_SLOT_PAD, not COUNT. I missed updating this. It also means that commit message was wrong, as some code *did* rely slots being initialized to COUNT. This may fix a bug with SSO programs with > 16 FS input varyings. I think we probably just emitted extra pointless code, but probably didn't break anything. We might also just have no tests for that. Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Chris Forbes <[email protected]>
* i965: Update stale comment about unused VUE map slots.Kenneth Graunke2015-10-281-3/+1
| | | | | | | I changed this from COUNT to PAD in commit 268008f98c3810b9f276df985dc93ef. Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Chris Forbes <[email protected]>
* i965: Fix missing BRW_NEW_*_PROG_DATA flagging caused by cache reuse.Kenneth Graunke2015-10-282-4/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Consider the case of two nearly identical GLSL fragment shaders: out vec4 color; void main() { color = vec4(1); } and layout(early_fragment_tests) in; out vec4 color; void main() { color = vec4(1); } These shaders compile to the exact same assembly, but have distinct values for brw_wm_prog_data::early_fragment_tests. Since these are two independent GLSL shaders, they have different program keys - notably, brw_wm_prog_key::program_string_id differs. When uploading the second, brw_upload_cache will find an existing copy of the assembly in the cache BO, which means matching_data will be non-NULL. Although we create a second cache item (with the new key and prog_data), we set item->offset to the existing copy and avoid re-uploading duplicate assembly. However, brw_search_cache() would only flag BRW_NEW_*_PROG_DATA if item->offset differed from the supplied offset. With reuse, both programs have the same offset, but prog_data changed. We have to flag it, but failed to. To fix this, we simply need to check if the aux (prog_data) pointer changed. If either the assembly or the prog_data differs, flag it. This fixes a regression since 1bba29ed403e735ba0bf04ed8aa2e571884f, where Topi fixed brw_upload_cache() to actually reuse identical assembly. Prior to that, reuse basically never happened due to bugs. Unfortunately, this code apparently wasn't prepared to handle reuse! Fixes GPU hangs in Dolphin on Broadwell. Huge thanks to Pierre Bourdon and Ilia Mirkin for debugging this and helping track down the real issue. Cc: "11.0" <[email protected]> Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=92623 Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Topi Pohjolainen <[email protected]> Reviewed-by: Kristian Høgsberg <[email protected]> Tested-by: Pierre Bourdon <[email protected]>
* i965: remove unneeded src_reg copy in emit_shader_time_writeEmil Velikov2015-10-281-1/+1
| | | | | | | | The variable is already of type src_reg. creating a new instance only to destroy it seems unnecessary. Signed-off-by: Emil Velikov <[email protected]> Reviewed-by: Kenneth Graunke <[email protected]>
* i965: remove cache_aux_free_func arrayEmil Velikov2015-10-282-12/+5
| | | | | | | | | | | | There is only one function that can be called, which is well known at compilation time. The abstraction used here seems unnecessary, so let's use a direct call to brw_stage_prog_data_free() when appropriate, cut down the size of struct brw_cache. Signed-off-by: Emil Velikov <[email protected]> Reviewed-by: Kenneth Graunke <[email protected]>
* i965: Add missing close-parenthesis in error messagesIan Romanick2015-10-261-2/+2
| | | | | | Trivial. Signed-off-by: Ian Romanick <[email protected]>
* i965: Fix is-renderable check in intel_image_target_renderbuffer_storageIan Romanick2015-10-261-5/+1
| | | | | | | | | | | | | | | | | | | | | | | | Previously we could create a renderbuffer with format MESA_FORMAT_R8G8B8A8_UNORM, convert that renderbuffer to an EGLImage, then FAIL to convert the EGLImage back to a renderbuffer because reasons. Just use the same check in intel_image_target_renderbuffer_storage that brw_render_target_supported uses. There are more checks in brw_render_target_supported, but I don't think they are necessary here. A different approach would be to refactor brw_render_target_supported to take rb->Format and rb->NumSamples as parameters (instead of a gl_renderbuffer) and use the new function here. Fixes: ES2-CTS.gtf.GL2ExtensionTests.egl_image.egl_image Signed-off-by: Ian Romanick <[email protected]> Reviewed-by: Anuj Phogat <[email protected]> Tested-by: Tapani Pälli <[email protected]> Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=92476 Cc: "10.3 10.4 10.5 10.6 11.0" <[email protected]>
* glsl: keep track of intra-stage indices for atomicsTimothy Arceri2015-10-276-27/+30
| | | | | | | | | | | | | | | This is more optimal as it means we no longer have to upload the same set of ABO surfaces to all stages in the program. This also fixes a bug where since commit c0cd5b var->data.binding was being used as a replacement for atomic buffer index, but they don't have to be the same value they just happened to end up the same when binding is 0. Reviewed-by: Francisco Jerez <[email protected]> Reviewed-by: Samuel Iglesias Gonsálvez <[email protected]> Cc: Ilia Mirkin <[email protected]> Cc: Alejandro Piñeiro <[email protected]> Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=90175
* i965: Make brw_varying_to_offset take a const pointer to the VUE map.Kenneth Graunke2015-10-241-2/+2
| | | | | | | It doesn't modify it. Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Matt Turner <[email protected]>
* i965: Remove unused devinfo revisionBen Widawsky2015-10-243-5/+13
| | | | | | | | | | | | | | | | I left the function to obtain the revision because it is, and will continue to be useful in the future. I'd rather not have to dig it up every time we need it. Comments left at the implementation to say as much. This was accidentally left here when I moved the early platform support: commit 28ed1e08e8ba98ebd4ff0b56326372f0df9c73ad Author: Ben Widawsky <[email protected]> Date: Fri Aug 7 13:58:37 2015 -0700 i965/skl: Remove early platform support Signed-off-by: Ben Widawsky <[email protected]> Reviewed-by: Jason Ekstrand <[email protected]>
* i965/fs: Allow copy propagating into new surface access opcodesKristian Høgsberg Kristensen2015-10-231-0/+15
| | | | | Reviewed-by: Francisco Jerez <[email protected]> Signed-off-by: Kristian Høgsberg Kristensen <[email protected]>
* i965/fs: Optimize ssbo storesKristian Høgsberg Kristensen2015-10-231-30/+25
| | | | | | | Reviewed-by: Francisco Jerez <[email protected]> Write groups of enabled components together. Signed-off-by: Kristian Høgsberg Kristensen <[email protected]>
* i965/fs: Drop offset_reg temporary in ssbo loadKristian Høgsberg Kristensen2015-10-231-5/+3
| | | | | | | | | | Now that we don't read each component one-by-one, we don't need the temoprary vgrf for the offset. More importantly, this register was type UD while the nir source was type D. This broke copy propagation and left a redundant MOV in the generated code. Reviewed-by: Francisco Jerez <[email protected]> Signed-off-by: Kristian Høgsberg Kristensen <[email protected]>
* i965/fs: Avoid scalar destinations in emit_uniformize()Kristian Høgsberg Kristensen2015-10-231-4/+11
| | | | | | | | | The scalar destination registers break copy propagation. Instead compute the results to a regular register and then reference a component when we later use the result as a source. Reviewed-by: Francisco Jerez <[email protected]> Signed-off-by: Kristian Høgsberg Kristensen <[email protected]>
* i965/fs: Don't uniformize surface index twiceKristian Høgsberg Kristensen2015-10-231-2/+0
| | | | | | | | The emit_untyped_read and emit_untyped_write helpers already uniformize the surface index argument. No need to do it before calling them. Reviewed-by: Matt Turner <[email protected]> Signed-off-by: Kristian Høgsberg Kristensen <[email protected]>
* i965/fs: Use unsigned immediate 0 when eliminating ↵Kristian Høgsberg Kristensen2015-10-231-1/+1
| | | | | | | | | | | | SHADER_OPCODE_FIND_LIVE_CHANNEL The destination for SHADER_OPCODE_FIND_LIVE_CHANNEL is always a UD register. When we replace the opcode with a MOV, make sure we use a UD immediate 0 so copy propagation doesn't bail because of non-matching types. Reviewed-by: Matt Turner <[email protected]> Signed-off-by: Kristian Høgsberg Kristensen <[email protected]>
* i965/fs: Read all components of a SSBO field with one sendKristian Høgsberg Kristensen2015-10-231-18/+7
| | | | | | | | | Instead of looping through single-component reads, read all components in one go. Reviewed-by: Iago Toral Quiroga <[email protected]> Reviewed-by: Jordan Justen <[email protected]> Signed-off-by: Kristian Høgsberg Kristensen <[email protected]>
* i965: Don't use message headers for untyped readsKristian Høgsberg Kristensen2015-10-232-3/+2
| | | | | | | | | | | | We always set the mask to 0xffff, which is what it defaults to when no header is present. Let's drop the header instead. v2: Only remove header for untyped reads. Typed reads always need the header. Reviewed-by: Francisco Jerez <[email protected]> Reviewed-by: Jordan Justen <[email protected]> Signed-off-by: Kristian Høgsberg Kristensen <[email protected]>
* i965/vec4: check opcode on vec4_instruction::reads_flag(channel)Alejandro Piñeiro2015-10-231-2/+2
| | | | | | | | | | | | | | Commit f17b78 added an alternative reads_flag(channel) that returned if the instruction was reading a specific channel flag. By mistake it only took into account the predicate, but when the opcode is VS_OPCODE_UNPACK_FLAGS_SIMD4X2 there isn't any predicate, but the flag are used. That mistake caused some regressions on old hw. More information on this bug: https://bugs.freedesktop.org/show_bug.cgi?id=92621 Reviewed-by: Kenneth Graunke <[email protected]>
* drivers/common: use _mesa_RasterPos instead of _tnl_RasterPosBrian Paul2015-10-221-1/+2
| | | | Reviewed-by: Roland Scheidegger <[email protected]>
* i965/vec4: print predicate control at brw_vec4 dump_instructionAlejandro Piñeiro2015-10-223-3/+5
| | | | | | | v2: externalize pred_ctrl_align16 from brw_disasm.c instead of adding a copy on brw_vec4.c, as suggested by Matt Turner Reviewed-by: Matt Turner <[email protected]>
* i965/vec4: use an envvar to decide to print the assembly on cmod_propagation ↵Alejandro Piñeiro2015-10-222-2/+2
| | | | | | | | | | | | | | | | | tests The complete way to do this would be parse INTEL_DEBUG and print the output if DEBUG_VS (or a new one) is present (see intel_debug.c). But that seems like an overkill for the unit tests, that after all, the most common use case is being run when calling make check. v2: use the same idea for the fs counterpart too, as suggested by Matt Turner Reviewed-by: Matt Turner <[email protected]>
* i965/vec4: Add unit tests for cmod propagation passAlejandro Piñeiro2015-10-222-0/+829
| | | | | | | | | | | | | | | | | | This include the same tests coming from test_fs_cmod_propagation, (non vector glsl types included) plus some new with vec4 types, inspired on the regressions found while the optimization was a work in progress. Additionally, the check of number of instructions after the optimization was changed from EXPECT_EQ to ASSERT_EQ. This was done to avoid a crash on failing tests that expected no optimization, as after checking the number of instructions, there were some checks related to this last instruction opcode/conditional mod. v2: update tests after Matt Turner's review of the optimization pass v3: tweaks on the tests (mostly on the comments), after Matt Turner's review Reviewed-by: Matt Turner <[email protected]>
* i965/vec4: adding vec4_cmod_propagation optimizationAlejandro Piñeiro2015-10-224-0/+160
| | | | | | | | | | | | | | | | | | | | | vec4 port of fs_cmod_propagation. Shader-db results (no vec4 grepping): total instructions in shared programs: 6240413 -> 6235841 (-0.07%) instructions in affected programs: 401933 -> 397361 (-1.14%) total loops in shared programs: 1979 -> 1979 (0.00%) helped: 2265 HURT: 0 v2: remove extra space and combine two if blocks, as suggested by Matt Turner v3: add condition check to bail out if current inst and inst being scanned has different writemask, as pointed by Matt Turner v3: updated shader-db numbers v4: remove block from foreach_inst_in_block_*_starting_from after commit 801f151917fedb13c5c6e96281a18d833dd6901f Reviewed-by: Matt Turner <[email protected]>
* i965/vec4: track and use independently each flag channelAlejandro Piñeiro2015-10-223-14/+52
| | | | | | | | | | | | | | | vec4_live_variables tracks now each flag channel independently, so vec4_dead_code_eliminate can update the writemask of null registers, based on which component are alive at the moment. This would allow vec4_cmod_propagation to optimize out several movs involving null registers. v2: added support to track each flag channel independently at vec4 live_variables, as v1 assumed that it was already doing it, as pointed by Francisco Jerez v3: general cleaningn after Matt Turner's review Reviewed-by: Matt Turner <[email protected]>
* i965/vec4: nir_emit_if doesn't need to predicate based on all the channelsAlejandro Piñeiro2015-10-221-1/+3
| | | | | | | v2: changed comment, as suggested by Matt Turner Reviewed-by: Matt Turner <[email protected]> Reviewed-by: Jason Ekstrand <[email protected]>
* i965/vec4/gs: Fix signed/unsigned comparison warning.Matt Turner2015-10-221-1/+1
|
* i965/fs: Emit a single ADD instruction for SET_SAMPLE_ID on Gen8+.Matt Turner2015-10-221-1/+1
| | | | | | | | | Gen8+ lifted the register region restriction that an instruction whose destination spans two registers must have sources that also span two registers. Reviewed-by: Kenneth Graunke <[email protected]> Reviewed-by: Anuj Phogat <[email protected]>
* i965/fs: Drop unnecessary write-enable-all from SET_SAMPLE_ID.Matt Turner2015-10-221-5/+5
| | | | | Reviewed-by: Kenneth Graunke <[email protected]> Reviewed-by: Anuj Phogat <[email protected]>
* i965/fs: Trim unneeded channels in SampleID setup.Matt Turner2015-10-221-6/+6
| | | | | | | | | The AND and SHR produce a scalar value that we had been replicating across $dispatch_width channels. The immediate MOV produces only four useful channels of data. Reviewed-by: Kenneth Graunke <[email protected]> Reviewed-by: Anuj Phogat <[email protected]>
* i965/fs: Use type-W for immediate in SampleID setup.Matt Turner2015-10-222-3/+3
| | | | | | | | | | | Not a functional difference, but register is loaded with a signed immediate (V) and added to a signed type (D) producing a signed result (D). Also change the type of g0 to allow for compaction. Reviewed-by: Kenneth Graunke <[email protected]> Reviewed-by: Anuj Phogat <[email protected]>
* i965/vec4: Initialize LOD to 0.0f for textureQueryLevels() and texture().Matt Turner2015-10-221-0/+12
| | | | | | | | | | | | | We implement textureQueryLevels (which takes no arguments, save the sampler) using the resinfo message (which takes an argument of LOD). Without initializing it, we'd generate a MOV from the null register to load the LOD argument. Essentially the same logic applies to texture. A vertex shader cannot compute derivatives and so cannot produce an LOD, so TXL with an LOD of 0.0 is used. Reviewed-by: Iago Toral Quiroga <[email protected]>
* i965: Note that the UV immediate type is Gen6+.Matt Turner2015-10-221-1/+1
|
* i965/fs: Disable CSE optimization for untyped & typed surface readsJordan Justen2015-10-223-1/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | An untyped surface read is volatile because it might be affected by a write. In the ES31-CTS.compute_shader.resources-max test, two back to back read/modify/writes of an SSBO variable looked something like this: r1 = untyped_surface_read(ssbo_float) r2 = r1 + 1 untyped_surface_write(ssbo_float, r2) r3 = untyped_surface_read(ssbo_float) r4 = r3 + 1 untyped_surface_write(ssbo_float, r4) And after CSE, we had: r1 = untyped_surface_read(ssbo_float) r2 = r1 + 1 untyped_surface_write(ssbo_float, r2) r4 = r1 + 1 untyped_surface_write(ssbo_float, r4) Signed-off-by: Jordan Justen <[email protected]> Reviewed-by: Iago Toral Quiroga <[email protected]>
* i965: Advertise ARB_shader_stencil_export (gen9+)Ben Widawsky2015-10-211-0/+1
| | | | | Signed-off-by: Ben Widawsky <[email protected]> Reviewed-by: Kenneth Graunke <[email protected]>
* i965: Implement ARB_shader_stencil_export (gen9+)Ben Widawsky2015-10-219-3/+98
| | | | | | | | | | | | | v2: remove useless source_stencil_to_render_target (Ken) Squash in the actual packing function, which also got to v2: Move the definition of the OPCODE outside of FB_WRITE opcodes (Matt) Reorder the regioning to be in VWH order (Matt) Don't retype src in the backend, just assert instead (Matt) Rename the debug prints to something better (Matt) Signed-off-by: Ben Widawsky <[email protected]> Reviewed-by: Kenneth Graunke <[email protected]>
* i965/fs: Enumerate logical fb writes argumentsBen Widawsky2015-10-213-21/+29
| | | | | | | | | | | | | | | | | | | Gen9 adds the ability to write out a stencil value, so we need to expand the virtual payload by one. Abstracting this now makes that change easier to read. I was admittedly confused early on about some of the hardcoding. If people believe the resulting code is inferior, I am not super attached to the patch. v2: Remove explicit numbering from the enumeration (Matt). Use a real naming scheme, and reference it in the opcode definition (Curro) Add a missed hardcoded logical position in get_lowered_simd_width (Ben) Add an assertion to make sure the component numbering is correct (Ben) Cc: Matt Turner <[email protected]> Cc: Francisco Jerez <[email protected]> Signed-off-by: Ben Widawsky <[email protected]> Reviewed-by: Kenneth Graunke <[email protected]>
* i965: Implement gl_InvocationID.Kenneth Graunke2015-10-211-0/+13
| | | | | | | It's stored in bits 31:27 of g1 (along with the URB handles). Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Kristian Høgsberg <[email protected]>
* i965: Implement nir_intrinsic_load_primitive.Kenneth Graunke2015-10-211-0/+8
| | | | | Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Kristian Høgsberg <[email protected]>
* i965: Add a fs_visitor constructor that takes a brw_gs_compile.Kenneth Graunke2015-10-212-3/+39
| | | | | | | | | | | Unlike the vs/wm structs, brw_gs_compile is actually useful: it contains the input VUE map and information about the control data headers. Passing this in allows us to share that code in brw_gs.c, and calculate them before deciding on vec4 vs. scalar mode, as it's independent of that choice. Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Kristian Høgsberg <[email protected]>
* i965: Add a brw->scalar_gs flag controlled by INTEL_SCALAR_GS=1.Kenneth Graunke2015-10-213-1/+8
| | | | | | | | | | | This patch introduces a brw->scalar_gs flag, similar to brw->scalar_vs, which controls whether or not to use SIMD8 geometry shaders. For now, we control it via a new environment variable, INTEL_SCALAR_GS. This provides a convenient way to try it out. Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Kristian Høgsberg <[email protected]>
* i965: Make emit_urb_writes() reserve space for GS header information.Kenneth Graunke2015-10-211-2/+16
| | | | | | | | | | | Geometry shaders have additional header data at the beginning of their output URB entries. Shaders that use EndPrimitive() or multiple streams have a control data header; shaders with a dynamic vertex count have an additional vec4 slot to hold the 32-bit vertex count (and 96 bits of padding). Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Kristian Høgsberg <[email protected]>
* i965: Make emit_urb_writes() only set EOT for the VS.Kenneth Graunke2015-10-211-1/+1
| | | | | | | | | The GS will emit a bunch of vertices, and we don't want to do an EOT prematurely. We'll emit GS_OPCODE_THREAD_END when we want to terminate the thread. Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Kristian Høgsberg <[email protected]>
* i965: Make fs_visitor::emit_urb_writes reusable for scalar GS.Kenneth Graunke2015-10-211-7/+7
| | | | | | | | GS doesn't have ClampVertexColor, and we don't want to go through VS structures. Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Kristian Høgsberg <[email protected]>
* i965: Introduce a brw_vue_prog_data::include_vue_handles flag.Kenneth Graunke2015-10-212-0/+5
| | | | | | | | | Tessellation shaders and SIMD8 geometry shaders may need to resort to the pull model for inputs at times. When set, the state upload code will tell the hardware to provide URB handles for input data. Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Kristian Høgsberg <[email protected]>
* i965: Introduce a new SHADER_OPCODE_URB_READ_SIMD8 opcode.Kenneth Graunke2015-10-215-0/+40
| | | | | | | | | | | | In scalar mode, geometry shader inputs can easily take up hundreds of registers. This makes pushing VUE entries impractical; we'll need to resort to the pull model in some cases. To support this, we introduce a new opcode corresponding to the "URB Read SIMD8" message. Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Kristian Høgsberg <[email protected]>
* i965: Introduce new SHADER_OPCODE_URB_WRITE_SIMD8_MASKED/PER_SLOT opcodes.Kenneth Graunke2015-10-215-0/+33
| | | | | | | | | | | | | In the vec4 backend, we have a vec4_instruction::urb_write_flags field. There are many kinds of flags for SIMD4x2 messages. However, there are really only two (per-slot offset, use channel masks) for SIMD8 messages. Rather than adding a boolean flag for per-slot offsets (polluting all instructions), I decided to just make three new opcodes. Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Kristian Høgsberg <[email protected]>
* i965/gs: Do prog_data setup and other calculations in brw_compile_gsJason Ekstrand2015-10-214-220/+222
| | | | | | | | | | | | | | | | This commit moves the large pile of setup calculations we have to do for geometry shaders out of brw_gs_emit and into brw_compile_gs. This has a couple of nice implications. First, it's less work that the caller of brw_compile_gs has to do. Second, it's consistent with the vertex and fragment stages. Finally, it allows us to put brw_gs_compile back behind the API boundary where it belongs. v2 (Jason Ekstrand): - Pull the changes to use nir info into a separate patch - Put brw_gs_compile into brw_shader.h rather than brw_vec4_gs_visitor.h so that we can use it for scalar GS. Reviewed-by: Kenneth Graunke <[email protected]>