intel/fs: Allow cmod propagation to instructions with saturate modifier

v2: Add unit tests.  Suggested by Matt.

All Intel GPUs had similar results. (Ice Lake shown)
total instructions in shared programs: 17229441 -> 17228658 (<.01%)
instructions in affected programs: 159574 -> 158791 (-0.49%)
helped: 489
HURT: 0
helped stats (abs) min: 1 max: 5 x̄: 1.60 x̃: 1
helped stats (rel) min: 0.07% max: 2.70% x̄: 0.61% x̃: 0.59%
95% mean confidence interval for instructions value: -1.72 -1.48
95% mean confidence interval for instructions %-change: -0.64% -0.58%
Instructions are helped.

total cycles in shared programs: 360944149 -> 360937144 (<.01%)
cycles in affected programs: 1072195 -> 1065190 (-0.65%)
helped: 254
HURT: 27
helped stats (abs) min: 2 max: 234 x̄: 30.51 x̃: 9
helped stats (rel) min: 0.04% max: 8.99% x̄: 0.75% x̃: 0.24%
HURT stats (abs)   min: 2 max: 83 x̄: 27.56 x̃: 24
HURT stats (rel)   min: 0.09% max: 3.79% x̄: 1.28% x̃: 1.16%
95% mean confidence interval for cycles value: -30.11 -19.75
95% mean confidence interval for cycles %-change: -0.70% -0.41%
Cycles are helped.

Reviewed-by: Matt Turner <[email protected]> [v1]

1 files changed, 42 insertions, 9 deletions

diff --git a/src/intel/compiler/brw_fs_cmod_propagation.cpp b/src/intel/compiler/brw_fs_cmod_propagation.cpp
index c9725263929..b430d4b2446 100644
--- a/src/intel/compiler/brw_fs_cmod_propagation.cpp
+++ b/src/intel/compiler/brw_fs_cmod_propagation.cpp
@@ -313,14 +313,6 @@ opt_cmod_propagation_local(const gen_device_info *devinfo, bblock_t *block)
                 scan_inst->opcode == BRW_OPCODE_CMPN)
                break;
 
-            /* From the Sky Lake PRM Vol. 7 "Assigning Conditional Mods":
-             *
-             *    * Note that the [post condition signal] bits generated at
-             *      the output of a compute are before the .sat.
-             */
-            if (scan_inst->saturate)
-               break;
-
             /* From the Sky Lake PRM, Vol 2a, "Multiply":
              *
              *    "When multiplying integer data types, if one of the sources
@@ -338,11 +330,52 @@ opt_cmod_propagation_local(const gen_device_info *devinfo, bblock_t *block)
                 scan_inst->opcode == BRW_OPCODE_MUL)
                break;
 
-            /* Otherwise, try propagating the conditional. */
             enum brw_conditional_mod cond =
                inst->src[0].negate ? brw_swap_cmod(inst->conditional_mod)
                                    : inst->conditional_mod;
 
+            /* From the Sky Lake PRM Vol. 7 "Assigning Conditional Mods":
+             *
+             *    * Note that the [post condition signal] bits generated at
+             *      the output of a compute are before the .sat.
+             *
+             * This limits the cases where we can propagate the conditional
+             * modifier.  If scan_inst has a saturate modifier, then we can
+             * only propagate from inst if inst is 'scan_inst <= 0',
+             * 'scan_inst == 0', 'scan_inst != 0', or 'scan_inst > 0'.  If
+             * inst is 'scan_inst == 0', the conditional modifier must be
+             * replace with LE.  Likewise, if inst is 'scan_inst != 0', the
+             * conditional modifier must be replace with G.
+             *
+             * The only other cases are 'scan_inst < 0' (which is a
+             * contradiction) and 'scan_inst >= 0' (which is a tautology).
+             */
+            if (scan_inst->saturate) {
+               if (scan_inst->dst.type != BRW_REGISTER_TYPE_F)
+                  break;
+
+               if (cond != BRW_CONDITIONAL_Z &&
+                   cond != BRW_CONDITIONAL_NZ &&
+                   cond != BRW_CONDITIONAL_LE &&
+                   cond != BRW_CONDITIONAL_G)
+                  break;
+
+               if (inst->opcode != BRW_OPCODE_MOV &&
+                   inst->opcode != BRW_OPCODE_CMP)
+                  break;
+
+               /* inst->src[1].is_zero() was tested before, but be safe
+                * against possible future changes in this code.
+                */
+               assert(inst->opcode != BRW_OPCODE_CMP || inst->src[1].is_zero());
+
+               if (cond == BRW_CONDITIONAL_Z)
+                  cond = BRW_CONDITIONAL_LE;
+               else if (cond == BRW_CONDITIONAL_NZ)
+                  cond = BRW_CONDITIONAL_G;
+            }
+
+            /* Otherwise, try propagating the conditional. */
             if (scan_inst->can_do_cmod() &&
                 ((!read_flag && scan_inst->conditional_mod == BRW_CONDITIONAL_NONE) ||
                  scan_inst->conditional_mod == cond)) {