summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* i965: Modify signature to update_texture_surface functions.Paul Berry2013-08-313-22/+17
| | | | | | | | | | | | Previously these functions would accept a pointer to the binding table and an index indicating which entry in the binding table should be updated. Now they merely take a pointer to the binding table entry to be updated. This will make it easier to generalize brw_texture_surfaces to support geometry shaders. Reviewed-by: Kenneth Graunke <[email protected]>
* i965/vs: generalize gen6_vs_push_constants in preparation for GS.Paul Berry2013-08-312-16/+36
| | | | Reviewed-by: Kenneth Graunke <[email protected]>
* i965/gs: make the state atom for compiling Gen7 geometry shaders.Paul Berry2013-08-3112-6/+401
| | | | | | Reviewed-by: Kenneth Graunke <[email protected]> v2: Use "unsigned" rather than "GLuint".
* i965/gs: Implement support for geometry shader surfaces.Paul Berry2013-08-315-0/+131
| | | | | | | | | | | | | | This patch implements pull constant upload, binding table upload, and surface setup for geometry shaders, by re-using vertex shader code that was generalized in previous patches. Based on work by Eric Anholt <[email protected]>. v2: Update ditry bits for brw_gs_ubo_surfaces to account for commit 77d8fbc (mesa: add & use a new driver flag for UBO updates instead of _NEW_BUFFER_OBJECT). Reviewed-by: Kenneth Graunke <[email protected]>
* i965/vs: generalize brw_vs_binding_table in preparation for GS.Paul Berry2013-08-312-13/+35
| | | | | | | v2: Use GLbitfield instead of GLbitfield64 in brw_vec4_upload_binding_table. Reviewed-by: Kenneth Graunke <[email protected]>
* i965: generalize brw_vs_pull_constants in preparation for GS.Paul Berry2013-08-312-26/+51
| | | | | | | v2: Use GLbitfield instead of GLbitfield64 in brw_upload_vec4_pull_constants. Reviewed-by: Kenneth Graunke <[email protected]>
* i965: Make sure constants re-sent after constant buffer reallocation.Paul Berry2013-08-317-5/+24
| | | | | | | | | | | | | | | | | | | | The hardware requires that after constant buffers for a stage are allocated using a 3DSTATE_PUSH_CONSTANT_ALLOC_{VS,HS,DS,GS,PS} command, and prior to execution of a 3DPRIMITIVE, the corresponding stage's constant buffers must be reprogrammed using a 3DSTATE_CONSTANT_{VS,HS,DS,GS,PS} command. Previously we didn't need to worry about this, because we only programmed 3DSTATE_PUSH_CONSTANT_ALLOC_{VS,HS,DS,GS,PS} once on startup (or, previous to that, whenever BRW_NEW_CONTEXT was flagged). But now that we reallocate the constant buffers whenever geometry shaders are switched on and off, we need to make sure the constant buffers are reprogrammed. We do this by adding a new bit, BRW_NEW_PUSH_CONSTANT_ALLOCATION, to brw->state.dirty.brw. Reviewed-by: Kenneth Graunke <[email protected]>
* i965/gs: Allocate push constant space for use by GS.Paul Berry2013-08-316-22/+68
| | | | | | | | | | | | | | | | | | Previously, we would always use the same push constant allocation regardless of what shader programs were being run: the available push constant space was split into 2 equal size partitions, one for the vertex shader, and one for the fragment shader. Now that we are adding geometry shader support, we need to do something smarter. This patch adjusts things so that when a geometry shader is in use, we split the available push constant space into 3 nearly-equal size partitions instead of 2. Since the push constant allocation is now affected by GL state, it can no longer be set up by brw_upload_initial_gpu_state(); instead it must be set up by a state atom. Reviewed-by: Kenneth Graunke <[email protected]>
* i965/gen7: Emit CS stall after 3DSTATE_PUSH_CONSTANT_ALLOC_PS.Paul Berry2013-08-311-0/+30
| | | | | | | | | This is required by the internal hardware docs and the PRM. Probably the reason we were getting away with not doing it was because we only emitted 3DSTATE_PUSH_CONSTANT_ALLOC_PS during startup. However that's going to change with the introduction of geometry shaders. Reviewed-by: Kenneth Graunke <[email protected]>
* i965/gs: Allocate URB space for use by GS.Paul Berry2013-08-313-35/+143
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, we gave all of the URB space (other than the small amount that is used for push constants) to the vertex shader. However, when a geometry shader is active, we need to divide it up between the vertex and geometry shaders. The size of the URB entries for the vertex and geometry shaders can vary dramatically from one shader to the next. So it doesn't make sense to simply split the available space in two. In particular: - On Ivy Bridge GT1, this would not leave enough space for the worst case geometry shader, which requires 64k of URB space. - Due to hardware-imposed limits on the maximum number of URB entries, sometimes a given shader stage will only be capable of using a small amount of URB space. When this happens, it may make sense to allocate substantially less than half of the available space to that stage. Our algorithm for dividing space between the two stages is to first compute (a) the minimum amount of URB space that each stage needs in order to function properly, and (b) the amount of additional URB space that each stage "wants" (i.e. that it would be capable of making use of). If the total amount of space available is not enough to satisfy needs + wants, then each stage's "wants" amount is scaled back by the same factor in order to fit. When only a vertex shader is active, this algorithm produces equivalent results to the old algorithm (if the vertex shader stage can make use of all the available URB space, we assign all the space to it; if it can't, we let it use as much as it can). In the future, when we need to support tessellation control and tessellation evaluation pipeline stages, it should be straightforward to expand this algorithm to cover them. v2: Use "unsigned" rather than "GLuint". Reviewed-by: Kenneth Graunke <[email protected]>
* i965: Make CACHE_NEW_GS_PROG.Paul Berry2013-08-311-0/+2
| | | | | Reviewed-by: Kenneth Graunke <[email protected]> Reviewed-by: Chad Versace <[email protected]>
* i965/gs: Create brw_context::gs structure to track GS program state.Paul Berry2013-08-311-0/+5
| | | | | | | v2: Change name from "vec4_gs" to simply "gs". Reviewed-by: Kenneth Graunke <[email protected]> Reviewed-by: Chad Versace <[email protected]>
* i965: Move data from brw->vs into a base class if gs will also need it.Paul Berry2013-08-3112-85/+108
| | | | | | | | | This paves the way for sharing the code that will set up the vertex and geometry shader pipeline state. v2: Rename the base class to brw_stage_state. Reviewed-by: Chad Versace <[email protected]>
* i965/gs: Update defines related to GS surface organization.Paul Berry2013-08-318-29/+29
| | | | | | | | | | | | | | | | Defines that previously referred to VS now refer to VEC4, since they will be shared by the user-programmable vertex shader and geometry shader stages. Defines that previously referred to the Gen6 geometry shader stage (which is only used for transform feedback) are now renamed to explicitly refer to Gen6, to avoid confusion with the Gen7 user-programmable geometry shader stage. Based on work by Eric Anholt <[email protected]>. Reviewed-by: Kenneth Graunke <[email protected]> Reviewed-by: Chad Versace <[email protected]>
* i965: Move vec4 register allocation data structures to brw->vec4.Paul Berry2013-08-312-29/+33
| | | | | | | | This will avoid confusion when we add geometry shaders, since these data structures will be shared by vertex and geometry shaders. Reviewed-by: Kenneth Graunke <[email protected]> Reviewed-by: Chad Versace <[email protected]>
* i965: Rename user-defined gs structs from vec4_gs to gs.Paul Berry2013-08-313-9/+9
| | | | | | | | | Now that the name "gs" is no longer used to refer to the legacy fixed function geometry shaders, we can use it to refer to user-defined geometry shaders. Reviewed-by: Kenneth Graunke <[email protected]> Reviewed-by: Chad Versace <[email protected]>
* i965: rename legacy gs structs and functions to ff_gs.Paul Berry2013-08-3114-146/+156
| | | | | | | | "ff" is for "fixed function". This frees up the name "gs" to refer to user-defined geometry shaders. Reviewed-by: Kenneth Graunke <[email protected]> Reviewed-by: Chad Versace <[email protected]>
* radeonsi: simplify and improve flushingMarek Olšák2013-08-3112-140/+125
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This mimics r600g. The R600_CONTEXT_xxx flags are added to rctx->b.flags and si_emit_cache_flush emits the packets. That's it. The shared radeon code tells us when the streamout cache should be flushed, so we have to check the flags anyway. There is a new atom "cache_flush", because caches must be flushed *after* resource descriptors are changed in memory. Functional changes: * Write caches are flushed at the end of CS and read caches are flushed at its beginning. * Sampler view states are removed from si_state, they only held the flush flags. * Everytime a shader is changed, the I cache is flushed. Is this needed? Due to a hw bug, this also flushes the K cache. * The WRITE_DATA packet is changed to use TC, which fixes a rendering issue in openarena. I'm not sure how TC interacts with CP DMA, but for now it seems to work better than any other solution I tried. (BTW CIK allows us to use TC for CP DMA.) * Flush the K cache instead of the texture cache when updating resource descriptors (due to a hw bug, this also flushes the I cache). I think the K cache flush is correct here, but I'm not sure if the texture cache should be flushed too (probably not considering we use TC for WRITE_DATA, but we don't use TC for CP DMA). * The number of resource contexts is decreased to 16. With all of these cache changes, 4 doesn't work, but 8 works, which suggests I'm actually doing the right thing here and the pipeline isn't drained during flushes. Reviewed-by: Michel Dänzer <[email protected]> Reviewed-by: Christian König <[email protected]> Tested-by: Tom Stellard <[email protected]>
* radeonsi: convert constant buffers to si_descriptorsMarek Olšák2013-08-315-128/+162
| | | | | | | | | | | | | | | There is a new "class" si_buffer_resources, which should be good enough for implementing any kind of buffer bindings (constant buffers, vertex buffers, streamout buffers, shader storage buffers, etc.) I don't even keep a copy of pipe_constant_buffer - we don't need it. The main motivation behind this is to have a well-tested infrastrusture for setting up streamout buffers. Reviewed-by: Michel Dänzer <[email protected]> Reviewed-by: Christian König <[email protected]> Tested-by: Tom Stellard <[email protected]>
* radeonsi: use r600_common_context, r600_common_screen, r600_resourceMarek Olšák2013-08-3128-777/+338
| | | | | | | | | Also r600_hw_context_priv.h and si_state_streamout.c are removed, because they are no longer needed. Reviewed-by: Michel Dänzer <[email protected]> Reviewed-by: Christian König <[email protected]> Tested-by: Tom Stellard <[email protected]>
* r600g: move streamout state to drivers/radeonMarek Olšák2013-08-3127-1486/+1825
| | | | | | | | | | | | | | | | | | | | | | | | This streamout state code will be used by radeonsi. There are new structures r600_common_context and r600_common_screen. What is inherited by what is shown here: pipe_context -> r600_common_context -> r600_context pipe_screen -> r600_common_screen -> r600_screen The common structures reside in drivers/radeon. Currently they only contain enough functionality to be able to handle streamout. Eventually I'd like the whole pipe_screen implementation to be shared and some of the context stuff too. This is quite big, but most changes are because of the new structures and the fact r600_write_value is replaced by radeon_emit. Thanks to Tom Stellard for fixing the build for r600g/compute. Reviewed-by: Michel Dänzer <[email protected]> Reviewed-by: Christian König <[email protected]> Tested-by: Tom Stellard <[email protected]>
* radeonsi: cleanup initialization of SGPR shader parametersMarek Olšák2013-08-311-13/+19
| | | | | | Reviewed-by: Michel Dänzer <[email protected]> Reviewed-by: Christian König <[email protected]> Tested-by: Tom Stellard <[email protected]>
* r600g,radeonsi: remove unused variablesMarek Olšák2013-08-312-8/+0
| | | | | | Reviewed-by: Michel Dänzer <[email protected]> Reviewed-by: Christian König <[email protected]> Tested-by: Tom Stellard <[email protected]>
* draw: fix segfaults with aaline and aapoint stages disabledMarek Olšák2013-08-311-2/+4
| | | | | | | | | | There are drivers not using these optional stages. Broken by a3ae5dc7dd5c2f8893f86a920247e690e550ebd4. Cc: [email protected] Reviewed-by: Jose Fonseca <[email protected]>
* i965/fs: Detect GRF sources in split_virtual_grfs send-from-GRF code.Kenneth Graunke2013-08-301-1/+5
| | | | | | | | | | | | | | | | | It is incorrect to assume that src[0] of a SEND-from-GRF opcode is the GRF. For example, FS_OPCODE_UNIFORM_PULL_CONSTANT_LOAD uses src[1] for the GRF. To be safe, loop over all the source registers and mark any GRFs. We probably won't ever have more than one, but it's simpler to just check all three rather than attempting to bail early. Not observed to fix anything yet, but likely to. Parallels the bug fix in the previous commit, which actually does fix known failures. Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Eric Anholt <[email protected]> Cc: [email protected]
* i965/vs: Detect GRF sources in split_virtual_grfs send-from-GRF code.Kenneth Graunke2013-08-301-2/+6
| | | | | | | | | | | | | | | | | | | | It is incorrect to assume that src[0] of a SEND-from-GRF opcode is the GRF. VS_OPCODE_PULL_CONSTANT_LOAD_GEN7 uses an IMM as src[0], and stores the GRF as src[1]. To be safe, loop over all the source registers and mark any GRFs. We probably won't ever have more than one, but it's simpler to just check all three rather than attempting to bail early. Fixes assertion failures in Unigine Sanctuary since we started making register allocation rely on split_virtual_grfs working. (The register classes were actually sufficient, we were just interpreting an IMM as a virtual GRF number.) Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=68637 Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Eric Anholt <[email protected]> Cc: [email protected]
* radeonsi: Do not suspend timer queriesNiels Ole Salscheider2013-08-306-14/+30
| | | | | Signed-off-by: Niels Ole Salscheider <[email protected]> Signed-off-by: Marek Olšák <[email protected]>
* draw: fix PIPE_MAX_SAMPLER/PIPE_MAX_SHADER_SAMPLER_VIEWS issuesRoland Scheidegger2013-08-302-6/+6
| | | | | | | | | | | | pstipple/aaline stages used PIPE_MAX_SAMPLER instead of PIPE_MAX_SHADER_SAMPLER_VIEWS when dealing with sampler views. Now these stages can't actually handle sampler_unit != texture_unit anyway (they cannot work with d3d10 shaders at all due to using tex not sample opcodes as "mixed mode" shaders are impossible) but this leads to crashes if a driver just installs these stages and then more than PIPE_MAX_SAMPLER views are set even if the stages aren't even used. Reviewed-by: Zack Rusin <[email protected]>
* gallivm: handle unbound textures in texture sampling / texture queriesRoland Scheidegger2013-08-301-0/+26
| | | | | | | | | | | | | | Turns out we don't need to do much extra work for detecting this case, since we are guaranteed to get a empty static texture state in this case, hence just rely on format being 0 and return all zero then. Previously needed dummy textures (would just have crashed on format being 0 otherwise) which cannot return the correct result for size queries and when sampling textures with wrap modes using border. As a bonus should hugely increase performance when sampling unbound textures - too bad it isn't a useful feature :-). Reviewed-by: Jose Fonseca <[email protected]> Reviewed-by: Zack Rusin <[email protected]>
* softpipe: handle NULL sampler views for texture sampling / queriesRoland Scheidegger2013-08-302-5/+26
| | | | | | | Instead of crashing just return all zero. Reviewed-by: Jose Fonseca <[email protected]> Reviewed-by: Zack Rusin <[email protected]>
* softpipe: check if so_target is NULL before accessing itRoland Scheidegger2013-08-301-2/+5
| | | | | | | | | | No idea if this is working right but copied straight from llvmpipe. (Not only does this check the so_target but also use buffer->data instead of buffer for the mapping.) Just trying to get rid of a segfault testing something else... Reviewed-by: Jose Fonseca <[email protected]> Reviewed-by: Zack Rusin <[email protected]>
* gallivm: (trivial) don't pass sampler_unit variable down to filtering funcsRoland Scheidegger2013-08-301-36/+21
| | | | | | | The only reason this was needed was because the fetch texel function had to get the (dynamic) border color, but this is now done much earlier. Reviewed-by: Jose Fonseca <[email protected]>
* gallivm: don't use AoS path if min/mag filter are different with multiple lodsRoland Scheidegger2013-08-301-1/+6
| | | | | | | | Instead of enhancing the AoS path so it can deal with it, just use SoA. Fixing AoS path wouldn't be all that difficult (use all the same logic as SoA) but considered not worth it for now. Reviewed-by: Jose Fonseca <[email protected]>
* mesa: Don't choose S3TC for generic compression if we can't compress.Eric Anholt2013-08-301-4/+8
| | | | | | | | | | | | | If the app is asking us to do GL_COMPRESSED_RGBA, then the app obviously doesn't have pre-compressed data to hand us. So don't choose a storage format that we won't actually be able to compress and store. Fixes black screen in warzone2100 when libtxc_dxtn is not present. Also 66 piglit tests. NOTE: This is a candidate for the 9.2 branch. Reported-by: Paul Wise <[email protected]> Reviewed-by: Kenneth Graunke <[email protected]>
* mesa: Rip out more extension checking from texformat.c.Eric Anholt2013-08-301-10/+4
| | | | | | | | You should only be flagging the formats as supported if you support them anyway. NOTE: This is a candidate for the 9.2 branch. (required for next commit) Reviewed-by: Kenneth Graunke <[email protected]>
* i965: Switch gen4-6 to using the sampler's base level for GL BASE_LEVEL.Eric Anholt2013-08-302-23/+12
| | | | | | | | Thanks to Ken for trawling through my neglected public branches and finding the bug in this change (inside a megacommit) that made me abandon this work. Reviewed-by: Kenneth Graunke <[email protected]>
* i965/gen7: Use the base_level field of the sampler to handle GL's BASE_LEVEL.Eric Anholt2013-08-302-21/+12
| | | | | | | This avoids the need to get the inter- and intra-tile offset and adjust our miptree info based on them. Reviewed-by: Kenneth Graunke <[email protected]>
* i965: Add missing state reset at the end of blorp.Eric Anholt2013-08-303-5/+8
| | | | | | | | | | | | | These are things that happen to be occurring because of the batch flush at the start of the blorp op (which exists to prevent batch space or aperture space overflow), but the intention was for this sequence of state resets at the end of blorp to be everything necessary for the next draw call. Found when debugging the next commit, by comparing brw_new_batch() and intel_batchbuffer_reset() to brw_blorp_exec(). Reviewed-by: Kenneth Graunke <[email protected]> Reviewed-by: Paul Berry <[email protected]>
* i965: Drop extra flush when calling intel_miptree_map_raw().Eric Anholt2013-08-301-1/+0
| | | | | | | | The code that got replaced with map_raw didn't do the flush, but now map_raw() is responsible for it and we don't have to worry about it. Reviewed-by: Kenneth Graunke <[email protected]> Reviewed-by: Paul Berry <[email protected]>
* i965: Make a slight distinction in perf debug for BOs versus miptrees.Eric Anholt2013-08-301-1/+1
| | | | | Reviewed-by: Kenneth Graunke <[email protected]> Reviewed-by: Paul Berry <[email protected]>
* intel: Reuse intel_glFlush().Eric Anholt2013-08-301-2/+1
| | | | | | | v2 (Kenneth Graunke): Rebase on latest master. Reviewed-by: Kenneth Graunke <[email protected]> Reviewed-by: Paul Berry <[email protected]>
* intel: Add support for the new flush_with_flags extension.Eric Anholt2013-08-301-10/+36
| | | | | | | | | | | | | | | | | This gives us more information about why we're flushing that we can use for handling our throttling. v2 (Kenneth Graunke): Rebase on latest master, add missing FLUSH_VERTICES and FLUSH_CURRENT, which fixes a regression in Glean's polygonOffset test. v3 (anholt): Drop FLUSH_CURRENT -- FLUSH_VERTICES is what we need, which is "get any queued prims out of VBO and into the driver", not "update ctx->Current so we can read it with the CPU." Also drop batch->used check, which intel_batchbuffer_flush() does anyway. Signed-off-by: Kenneth Graunke <[email protected]> Reviewed-by: Kenneth Graunke <[email protected]> Reviewed-by: Paul Berry <[email protected]>
* intel: Add a batch flush between front-buffer downsample and X protocol.Eric Anholt2013-08-301-0/+1
| | | | | | | | | | | | This was already happening because blorp happens to flush at the end of every call, but we have been talking about removing that at some point, and this would surely get overlooked. v2 (Kenneth Graunke): Rebase on latest master. Note that we did remove the other flush, and this change actually did get overlooked! Reviewed-by: Kenneth Graunke <[email protected]> Reviewed-by: Paul Berry <[email protected]>
* i965: Directly call intel_batchbuffer_flush() after i915 split.Eric Anholt2013-08-307-24/+10
| | | | | | | | intel_flush() now did nothing except call through (and intel_batchbuffer_flush() does the no-op check, too!) Reviewed-by: Kenneth Graunke <[email protected]> Reviewed-by: Paul Berry <[email protected]>
* i965/vs: Fix regression on pre-gen6 with no VS uniforms in use.Eric Anholt2013-08-301-0/+1
| | | | | | | | | | | | df06745c5adb524e15d157f976c08f1718f08efa made it so that we didn't allocate extra uniform space for unused clip planes, which also incidentally made us not allocate any space at all, which we were relying on for this no-uniforms case. Instead of putting the knowledge of this special HW exception into the thing that normally preallocates prog_data for us, just allocate it here. Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=68766 Reviewed-by: Kenneth Graunke <[email protected]>
* r600g: enable SB backend by defaultVadim Girlin2013-08-304-5/+6
| | | | | | | Signed-off-by: Vadim Girlin <[email protected]> Reviewed-by: Marek Olšák <[email protected]> Reviewed-by: Tom Stellard <[email protected]> Reviewed-by: Christian König <[email protected]>
* r600g: fix color exports when we have no CBsVadim Girlin2013-08-301-3/+4
| | | | | | | We need to export at least one color if the shader writes it, even when nr_cbufs==0. Signed-off-by: Vadim Girlin <[email protected]>
* nvc0/ir: Initialize NVC0LegalizePostRA member variables.Vinson Lee2013-08-291-1/+3
| | | | | | Fixes "Uninitialized pointer field" defects reported by Coverity. Signed-off-by: Vinson Lee <[email protected]>
* gallivm: support per-pixel min/mag filter in SoA pathRoland Scheidegger2013-08-301-43/+243
| | | | | | | | | | | | | | | | | | | | | | | Since we can have per-pixel lod we should also honor the filter per-pixel (in fact we didn't honor it per quad neither in the multiple quad case). Do this by running the linear path and simply beating the weights into shape (the sample with the higher weight is the one which should have been chosen with nearest filtering hence adjust filter weight to 1.0/0.0 based on that). If all pixels use nearest filter (either min and mag) then still run just a nearest filter as this is way cheaper (probably around 4 times faster for 2d, more for 3d case) and it should be relatively rare that pixels really need different filtering. OTOH if all pixels would require linear don't do anything special since the linear path with filter adjustments shouldn't really be all that much more expensive than ordinary linear, and we think it's rare that min/mag filters are configured differently so there doesn't seem much value in trying to optimize this further. This does not yet fix the AoS path (though currently AoS is only used for single quads hence it could be considered less broken, just never honoring per-pixel filter decision but doing it per quad). v2: simplify code a bit (unify min linear and min nearest cases) Reviewed-by: Jose Fonseca <[email protected]>
* gallivm: don't calculate square root of rho if we use accurate rho methodRoland Scheidegger2013-08-301-39/+74
| | | | | | | | | | | | | | | | | | | | While a sqrt here and there shouldn't hurt much (depending on the cpu) it is possible to completely omit it since rho is only used for calculating lod and there log2(x) == 0.5*log2(x^2). Depending on the exact path taken for calculating lod this means we get a simple mul instead of sqrt (in case of nearest mip filter in fact we don't need to replace the sqrt with something else at all), only in some not very useful path this doesn't work (combined brilinear calculation of int level and fractional lod, accurate rho calc but brilinear filtering seems odd). Apart from being faster as an added bonus this should increase our crappy fractional accuracy of lod, since fast_log2 is only good for ~3bits and this should increase accuracy by one bit (though not used if dimension is just one as we'd need an extra mul there as we never had the squared rho in the first place). v2: use separate ilog2_sqrt function if we have squared rho. Reviewed-by: Jose Fonseca <[email protected]>