Mercurial > dive4elements > river

annotate artifacts/src/main/java/org/dive4elements/river/artifacts/math/StdDevOutlier.java @ 9801:1d7a72a50183 3.2.x tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Assume Compose V2, consistently
author Tom Gottfried <tom@intevation.de>
date Thu, 23 Nov 2023 10:14:13 +0100
parents 0a5239a1e46e
children
rev   line source
5863
4897a58c8746 River artifacts: Added new copyright headers.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
1 /* Copyright (C) 2011, 2012, 2013 by Bundesanstalt für Gewässerkunde
4897a58c8746 River artifacts: Added new copyright headers.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
2 * Software engineering by Intevation GmbH
4897a58c8746 River artifacts: Added new copyright headers.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
3 *
5994
af13ceeba52a Removed trailing whitespace.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5863
diff changeset
4 * This file is Free Software under the GNU AGPL (>=v3)
5863
4897a58c8746 River artifacts: Added new copyright headers.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
5 * and comes with ABSOLUTELY NO WARRANTY! Check out the
5994
af13ceeba52a Removed trailing whitespace.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5863
diff changeset
6 * documentation coming with Dive4Elements River for details.
5863
4897a58c8746 River artifacts: Added new copyright headers.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
7 */
4897a58c8746 River artifacts: Added new copyright headers.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
8
5831
bd047b71ab37 Repaired internal references
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4816
diff changeset
9 package org.dive4elements.river.artifacts.math;
4794
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
10
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
11 import java.util.List;
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
12
9726
0a5239a1e46e Upgrade to Log4j 2
Tom Gottfried <tom@intevation.de>
parents: 8856
diff changeset
13 import org.apache.logging.log4j.Logger;
0a5239a1e46e Upgrade to Log4j 2
Tom Gottfried <tom@intevation.de>
parents: 8856
diff changeset
14 import org.apache.logging.log4j.LogManager;
4794
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
15
6714
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
16 /* XXX:
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
17 * Warning: This class is called StdDevOutlier because it caculates the
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
18 * Standard Deviation method for outlier removal as the BFG calls it.
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
19 * But the actual calculation used to remove the outliers calculates
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
20 * the Standard Error and not the Standard Deviation! */
4794
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
21
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
22 public class StdDevOutlier
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
23 {
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
24 public static final double DEFAULT_FACTOR = 3;
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
25
9726
0a5239a1e46e Upgrade to Log4j 2
Tom Gottfried <tom@intevation.de>
parents: 8856
diff changeset
26 private static Logger log = LogManager.getLogger(StdDevOutlier.class);
4794
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
27
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
28 protected StdDevOutlier() {
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
29 }
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
30
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
31 public static Integer findOutlier(List<Double> values) {
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
32 return findOutlier(values, DEFAULT_FACTOR, null);
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
33 }
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
34
4795
8ee270a3ef25 Small code cleanups in S/Q outlier tests.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4794
diff changeset
35 public static Integer findOutlier(
8ee270a3ef25 Small code cleanups in S/Q outlier tests.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4794
diff changeset
36 List<Double> values,
4816
846b0441f905 Removed trailing whitespace.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4795
diff changeset
37 double factor,
6714
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
38 double [] stdErrResult
4795
8ee270a3ef25 Small code cleanups in S/Q outlier tests.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4794
diff changeset
39 ) {
4794
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
40 boolean debug = log.isDebugEnabled();
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
41
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
42 if (debug) {
8856
5e38e2924c07 Fix code style.
Tom Gottfried <tom@intevation.de>
parents: 6714
diff changeset
43 log.debug("factor for std dev test (that calculates std err): "
5e38e2924c07 Fix code style.
Tom Gottfried <tom@intevation.de>
parents: 6714
diff changeset
44 + factor);
4794
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
45 }
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
46
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
47 int N = values.size();
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
48
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
49 if (debug) {
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
50 log.debug("Values to check: " + N);
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
51 }
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
52
4795
8ee270a3ef25 Small code cleanups in S/Q outlier tests.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4794
diff changeset
53 if (N < 3) {
4794
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
54 return null;
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
55 }
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
56
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
57 double maxValue = -Double.MAX_VALUE;
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
58 int maxIndex = -1;
6714
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
59
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
60 double squareSumResiduals = 0;
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
61 for (Double db: values) {
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
62 squareSumResiduals += Math.pow(db, 2);
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
63 }
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
64
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
65 double stdErr = Math.sqrt(squareSumResiduals / (N - 2));
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
66
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
67 double accepted = factor * stdErr;
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
68
4795
8ee270a3ef25 Small code cleanups in S/Q outlier tests.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4794
diff changeset
69 for (int i = N-1; i >= 0; --i) {
4794
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
70 double value = Math.abs(values.get(i));
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
71 if (value > maxValue) {
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
72 maxValue = value;
4795
8ee270a3ef25 Small code cleanups in S/Q outlier tests.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4794
diff changeset
73 maxIndex = i;
4794
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
74 }
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
75 }
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
76
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
77 if (debug) {
6714
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
78 log.debug("std err: " + stdErr);
4794
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
79 log.debug("accepted: " + accepted);
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
80 log.debug("max value: " + maxValue);
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
81 }
4795
8ee270a3ef25 Small code cleanups in S/Q outlier tests.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4794
diff changeset
82
6714
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
83 if (stdErrResult != null) {
b265cd6cfda5 issue748: Change StandardDeviation implmentation to what BFG calls Standard Deviation
Andre Heinecke <aheinecke@intevation.de>
parents: 5994
diff changeset
84 stdErrResult[0] = stdErr;
4794
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
85 }
4795
8ee270a3ef25 Small code cleanups in S/Q outlier tests.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4794
diff changeset
86
4794
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
87 return maxValue > accepted ? maxIndex : null;
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
88 }
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
89 }
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents:
diff changeset
90 // vim:set ts=4 sw=4 si et sta sts=4 fenc=utf8 :
http://dive4elements.wald.intevation.org