com.linkedin.util.degrader.DegraderImpl
A Degrader that uses completed call latencies, error rate and outstanding requests to determine the computed drop rate. The degrader has a high watermark and low watermark. At each sampling interval, the degrader obtains statistics from its CallTracker and checks the statistics to determine if the high or low watermark condition has been met or neither. If either high or low watermark condition has been met, it adjusts the computedDropRate. The computedDropRate is a value from 0 (inclusive) to 1.0 (exclusive) that is the fraction of requests that should be dropped to reduce load. If the high watermark condition has been met (@see DegraderImpl.isHigh()), then the degrader will increase the computedDropRate by upStep, i.e. increasing the fraction of requests that should be dropped. The maximum value of computedDropRate is capped by maxDropRate. If the low watermark condition has been met (@see DegraderImpl.()), then the degrader will decrease the computedDropRate by downStep, i.e. reducing the fraction of requests that should be dropped. The minimum value of computedDropRate is 0. The actual dropRate is either the computedDropRate or the overrideDropRate. The overrideDropRate provides for runtime override. If overrideDropRate is enabled (i.e. its value is >= 0.0), then the actual dropRate is always set to the overrideDropRate, else it is set to the computedDropRate. The overrideDropRate can be set to 0.0 can be used to use the degrader in a passive observe mode, i.e. it can calculate computedDropRate but does not actually suggest any requests be dropped. This may useful for tuning the configuration parameters for high and low watermarks during initial deployment and observing the interaction of the degrader with the live services. The highLatency, highErrorRate and highOutstanding configuration parameters are used to determine if the high watermark condition has been met. The high watermark condition is met if (there are sufficient samples in the interval AND (the completed call latency >= highLatency OR the error rate is >= highErrorRate)) OR (there are sufficient outstanding calls AND the average of the outstanding calls is >= highOutstanding) (@see DegraderImpl.isHigh()). The lowLatency, lowErrorRate and lowOutstanding configuration parameters are used to determine if the low watermark condition has been met. The low watermark condition is met if there are sufficient samples in the interval AND the completed call latency <= lowLatency AND the error rate is <= lowErrorRate AND (there are insufficient outstanding calls OR the average of the outstanding calls is <= lowOutstanding) (@see DegraderImpl.()). Sufficient number of calls is determined by the minCallCount configuration parameter and the dropRate. The minCallCount is the minimum number of samples that must be present in the interval when no load should be shed (i.e. dropRate== 0.0) for the degrader to use the latency and errorRate statistics provided by CallTracker to make computedDropRate adjustments. The adjustedMinCallCount is adjusted to reflect reduced calls when dropRate is non-zero using max(1, round((1.0 - dropRate) * minCallCount)). For example, if all calls should be dropped, then adjustedMinCallCount should be 1, if no calls should be dropped, adjustedMinCallCount should be the same as minCallCount. Sufficient number of outstanding calls is determined by the minOustandingCount configuration parameter. The number of outstanding calls should be>= minOutstandingCount for the degrader to use the average outstanding latency to determine if high and low watermark condition has been met. If call rate is extremely low and dropRate approaches 1.0, then it is possible that almost all calls will be dropped, creating the problem that there is no new signal to enable the degrader to detect that the service has improved. To account for this, there is a maxDropDuration configuration parameter that specifies that the maximum duration that is allowed when all requests are dropped. For example, if maxDropDuration is 1 min and the last request that should not be dropped is older than 1 min, then the next request should not be dropped. In some cases, the proportion of traffic sent to the degrader may be adjusted dynamically by some other entity such as a load balancer or some other entity managing the drop rate, in such a way that the degrader is not aware of these actions. When this occurs, this other entity may want to override the minCallCount to account for the reduced traffic sent to the degrader. The overrideMinCallCount enabled overriding the default minCallCount to account for the reduced traffic sent to the degrader. The overrideMinCallCount enabled overriding the default minCallCount dynamic adjustment logic. If overrideMinCallCount is >= 0.0, this value overrides the computed/ dynamically adjusting min call count logic and always use this value. The latency metric from CallTracker that compared against highLatency and lowLatency is determined by the latencyToUse configuration parameter. It can be the average, 50, 90, 95, 99th percentile latency.