Mercurial > dive4elements > river

annotate artifacts/src/main/java/org/dive4elements/river/artifacts/model/fixings/Fitting.java @ 9415:9744ce3c3853

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets. The facets also put the valid station range into their xml-metadata
author gernotbelger
date Thu, 16 Aug 2018 16:27:53 +0200
parents ddcd52d239cd
children f51e23eb036a
rev   line source
5863
4897a58c8746 River artifacts: Added new copyright headers.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
1 /* Copyright (C) 2011, 2012, 2013 by Bundesanstalt für Gewässerkunde
4897a58c8746 River artifacts: Added new copyright headers.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
2 * Software engineering by Intevation GmbH
4897a58c8746 River artifacts: Added new copyright headers.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
3 *
5994
af13ceeba52a Removed trailing whitespace.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5863
diff changeset
4 * This file is Free Software under the GNU AGPL (>=v3)
5863
4897a58c8746 River artifacts: Added new copyright headers.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
5 * and comes with ABSOLUTELY NO WARRANTY! Check out the
5994
af13ceeba52a Removed trailing whitespace.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5863
diff changeset
6 * documentation coming with Dive4Elements River for details.
5863
4897a58c8746 River artifacts: Added new copyright headers.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
7 */
4897a58c8746 River artifacts: Added new copyright headers.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
8
5831
bd047b71ab37 Repaired internal references
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4794
diff changeset
9 package org.dive4elements.river.artifacts.model.fixings;
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10
3107
cbf308f5c41b FixA: Store the standard deviation of delta Ws of reference points into parameters, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3096
diff changeset
11 import java.util.ArrayList;
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
12 import java.util.Date;
3107
cbf308f5c41b FixA: Store the standard deviation of delta Ws of reference points into parameters, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3096
diff changeset
13 import java.util.List;
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
14
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
15 import org.apache.commons.math.MathException;
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
16 import org.apache.commons.math.optimization.fitting.CurveFitter;
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
17 import org.apache.commons.math.optimization.general.LevenbergMarquardtOptimizer;
3107
cbf308f5c41b FixA: Store the standard deviation of delta Ws of reference points into parameters, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3096
diff changeset
18 import org.apache.commons.math.stat.descriptive.moment.StandardDeviation;
4794
a7d080347ac3 MINFO: Allow two methods for outlier test in SQ relation.
Raimund Renkert <rrenkert@intevation.de>
parents: 3729
diff changeset
19 import org.apache.log4j.Logger;
5831
bd047b71ab37 Repaired internal references
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4794
diff changeset
20 import org.dive4elements.river.artifacts.math.GrubbsOutlier;
bd047b71ab37 Repaired internal references
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4794
diff changeset
21 import org.dive4elements.river.artifacts.math.fitting.Function;
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
22
9348
a3f318347707 Show wq outliers within same thems with different symbol: not ready yet
gernotbelger
parents: 7300
diff changeset
23 public class Fitting {
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
24 private static Logger log = Logger.getLogger(Fitting.class);
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
25
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
26 /** Use instance of this factory to find meta infos for outliers. */
3096
d7b0f52d6d04 FixA: Calculate Delta W(t) for reference points, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3073
diff changeset
27 public interface QWDFactory {
9360
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
28 QWD create(double q, double w, double deltaW, boolean isOutlier);
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
29 }
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
30
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
31 private static final class FittingData {
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
32 public final FixingColumnWithData event;
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
33 public final double w;
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
34 public final double q;
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
35 public final boolean isInterpolated;
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
36
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
37 public FittingData(final FixingColumnWithData event, final double w, final double q, final boolean isInterpolated) {
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
38 this.event = event;
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
39 this.w = w;
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
40 this.q = q;
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
41 this.isInterpolated = isInterpolated;
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
42 }
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
43 }
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
44
9360
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
45 private final double chiSqr;
3022
705d2058b682 FixA: Store the referenced QW for each km, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3011
diff changeset
46
9360
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
47 private final double[] parameters;
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
48
9360
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
49 private final double standardDeviation;
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
50
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
51 private final double maxQ;
9360
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
52
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
53 public Fitting(final double[] parameters, final double standardDeviation, final double chiSqr, final double maxQ) {
9360
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
54 this.parameters = parameters;
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
55 this.standardDeviation = standardDeviation;
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
56 this.chiSqr = chiSqr;
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
57 this.maxQ = maxQ;
3022
705d2058b682 FixA: Store the referenced QW for each km, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3011
diff changeset
58 }
705d2058b682 FixA: Store the referenced QW for each km, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3011
diff changeset
59
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
60 public double getChiSquare() {
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
61 return this.chiSqr;
3022
705d2058b682 FixA: Store the referenced QW for each km, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3011
diff changeset
62 }
705d2058b682 FixA: Store the referenced QW for each km, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3011
diff changeset
63
3065
ee653971fa84 FixA: Figure out max Q during fitting.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3030
diff changeset
64 public double getMaxQ() {
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
65 return this.maxQ;
3065
ee653971fa84 FixA: Figure out max Q during fitting.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3030
diff changeset
66 }
ee653971fa84 FixA: Figure out max Q during fitting.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3030
diff changeset
67
9348
a3f318347707 Show wq outliers within same thems with different symbol: not ready yet
gernotbelger
parents: 7300
diff changeset
68 public double[] getParameters() {
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
69 return this.parameters;
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
70 }
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
71
3107
cbf308f5c41b FixA: Store the standard deviation of delta Ws of reference points into parameters, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3096
diff changeset
72 public double getStandardDeviation() {
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
73 return this.standardDeviation;
3107
cbf308f5c41b FixA: Store the standard deviation of delta Ws of reference points into parameters, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3096
diff changeset
74 }
cbf308f5c41b FixA: Store the standard deviation of delta Ws of reference points into parameters, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3096
diff changeset
75
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
76 public static Fitting fit(final FixResultColumns resultColumns, final double km, final Function function, final boolean checkOutliers,
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
77 final List<FixingColumnWithData> eventColumns) {
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
78
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
79 final int numEvents = eventColumns.size();
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
80 final List<FittingData> data = new ArrayList<>(numEvents);
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
81
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
82 final double[] wTemp = new double[1];
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
83
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
84 for (final FixingColumnWithData event : eventColumns) {
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
85
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
86 final double q = event.getQ(km);
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
87 if (!Double.isNaN(q)) {
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
88 final boolean isInterpolated = !event.getW(km, wTemp, 0);
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
89 final double w = wTemp[0];
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
90
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
91 if (!Double.isNaN(w))
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
92 data.add(new FittingData(event, w, q, isInterpolated));
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
93 }
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
94 }
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
95
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
96 if (data.size() < 2) {
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
97 log.warn("Not enough data for fitting.");
9360
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
98 return null;
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
99 }
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
100
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
101 final List<FittingData> outliers = new ArrayList<>(data.size());
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
102
5831
bd047b71ab37 Repaired internal references
Sascha L. Teichmann <teichmann@intevation.de>
parents: 4794
diff changeset
103 org.dive4elements.river.artifacts.math.Function instance = null;
3202
307842cf8d9e FixA: Make fitting more tolerant.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3127
diff changeset
104 LevenbergMarquardtOptimizer lmo = null;
9360
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
105 double[] parameters = null;
3202
307842cf8d9e FixA: Make fitting more tolerant.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3127
diff changeset
106
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
107 for (;;) {
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
108
3202
307842cf8d9e FixA: Make fitting more tolerant.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3127
diff changeset
109 parameters = null;
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
110
7300
83bb52fa0c32 (issue1529) Be more tolerant in the fitting.
Andre Heinecke <aheinecke@intevation.de>
parents: 6868
diff changeset
111 for (double tolerance = 1e-10; tolerance < 1e-1; tolerance *= 10d) {
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
112
3202
307842cf8d9e FixA: Make fitting more tolerant.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3127
diff changeset
113 lmo = new LevenbergMarquardtOptimizer();
307842cf8d9e FixA: Make fitting more tolerant.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3127
diff changeset
114 lmo.setCostRelativeTolerance(tolerance);
307842cf8d9e FixA: Make fitting more tolerant.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3127
diff changeset
115 lmo.setOrthoTolerance(tolerance);
307842cf8d9e FixA: Make fitting more tolerant.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3127
diff changeset
116 lmo.setParRelativeTolerance(tolerance);
307842cf8d9e FixA: Make fitting more tolerant.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3127
diff changeset
117
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
118 try {
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
119 final CurveFitter cf = new CurveFitter(lmo);
3202
307842cf8d9e FixA: Make fitting more tolerant.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3127
diff changeset
120
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
121 for (final FittingData fittingData : data)
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
122 cf.addObservedPoint(fittingData.q, fittingData.w);
3202
307842cf8d9e FixA: Make fitting more tolerant.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3127
diff changeset
123
307842cf8d9e FixA: Make fitting more tolerant.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3127
diff changeset
124 parameters = cf.fit(function, function.getInitialGuess());
307842cf8d9e FixA: Make fitting more tolerant.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3127
diff changeset
125 break;
307842cf8d9e FixA: Make fitting more tolerant.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3127
diff changeset
126 }
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
127 catch (final MathException me) {
3202
307842cf8d9e FixA: Make fitting more tolerant.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3127
diff changeset
128 if (log.isDebugEnabled()) {
9360
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
129 log.debug("tolerance " + tolerance + " + failed.", me);
3202
307842cf8d9e FixA: Make fitting more tolerant.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3127
diff changeset
130 }
307842cf8d9e FixA: Make fitting more tolerant.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3127
diff changeset
131 }
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
132 }
9360
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
133
3202
307842cf8d9e FixA: Make fitting more tolerant.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3127
diff changeset
134 if (parameters == null) {
7300
83bb52fa0c32 (issue1529) Be more tolerant in the fitting.
Andre Heinecke <aheinecke@intevation.de>
parents: 6868
diff changeset
135 /*
9348
a3f318347707 Show wq outliers within same thems with different symbol: not ready yet
gernotbelger
parents: 7300
diff changeset
136 * log.debug("Parameters is null");
a3f318347707 Show wq outliers within same thems with different symbol: not ready yet
gernotbelger
parents: 7300
diff changeset
137 * for (int i = 0, N = xs.size(); i < N; ++i) {
a3f318347707 Show wq outliers within same thems with different symbol: not ready yet
gernotbelger
parents: 7300
diff changeset
138 * log.debug("DATA: " + xs.getQuick(i) + " " + ys.getQuick(i));
a3f318347707 Show wq outliers within same thems with different symbol: not ready yet
gernotbelger
parents: 7300
diff changeset
139 * }
a3f318347707 Show wq outliers within same thems with different symbol: not ready yet
gernotbelger
parents: 7300
diff changeset
140 */
9360
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
141 return null;
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
142 }
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
143
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
144 // This is the parameterized function for a given km.
3127
71484036b6ae FixA: Moved function instantiation to always have a valid function instance.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3107
diff changeset
145 instance = function.instantiate(parameters);
71484036b6ae FixA: Moved function instantiation to always have a valid function instance.
Raimund Renkert <raimund.renkert@intevation.de>
parents: 3107
diff changeset
146
9360
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
147 if (!checkOutliers)
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
148 break;
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
149
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
150 /* find the outlier */
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
151 final List<Double> inputs = new ArrayList<>(data.size());
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
152 for (final FittingData fittingData : data) {
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
153
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
154 double y = instance.value(fittingData.q);
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
155 if (Double.isNaN(y))
3565
b136113dad53 FixA: Only evict only one(!) data point as outlier before recalculating the function.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3203
diff changeset
156 y = Double.MAX_VALUE;
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
157
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
158 inputs.add(Double.valueOf(fittingData.w - y));
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
159 }
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
160
9360
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
161 final Integer outlier = GrubbsOutlier.findOutlier(inputs);
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
162 if (outlier == null)
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
163 break;
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
164
9360
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
165 final int idx = outlier.intValue();
3066
4c22194b733a Some minor bugfixes in fix analysis calculation and some more debug output.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3065
diff changeset
166
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
167 // outliers.add(qwdFactory.create(xs.getQuick(idx), ys.getQuick(idx), Double.NaN, true));
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
168 final FittingData removed = data.remove(idx);
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
169 outliers.add(removed);
3022
705d2058b682 FixA: Store the referenced QW for each km, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3011
diff changeset
170 }
705d2058b682 FixA: Store the referenced QW for each km, too.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 3011
diff changeset
171
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
172 /* now build result data */
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
173 final List<QWD> qwds = new ArrayList<>(data.size());
9360
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
174
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
175 /* calculate dW of outliers against the resulting function and add them to results */
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
176 for (final FittingData outlier : outliers) {
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
177 final QWD qwd = createQWD(outlier, instance, true);
9360
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
178 qwds.add(qwd);
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
179 resultColumns.addQWD(outlier.event, km, qwd);
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
180 }
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
181
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
182 /*
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
183 * calculate dW of used values against the resulting function and add them to results , also calculate standard *
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
184 * deviation
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
185 */
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
186 final StandardDeviation stdDev = new StandardDeviation();
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
187 double maxQ = -Double.MAX_VALUE;
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
188
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
189 for (final FittingData fittingData : data) {
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
190
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
191 final QWD qwd = createQWD(fittingData, instance, false);
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
192 qwds.add(qwd);
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
193 resultColumns.addQWD(fittingData.event, km, qwd);
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
194
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
195 stdDev.increment(qwd.getDeltaW());
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
196
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
197 final double q = qwd.getQ();
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
198 if (q > maxQ)
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
199 maxQ = q;
9360
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
200 }
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
201
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
202 final double standardDeviation = stdDev.getResult();
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
203
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
204 final double chiSqr = lmo.getChiSquare();
ddcd52d239cd Outliers in fixation calculation are now shown within the other 'B' event themes and get a separate symbol (triangle).
gernotbelger
parents: 9348
diff changeset
205
9415
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
206 return new Fitting(parameters, standardDeviation, chiSqr, maxQ);
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
207 }
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
208
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
209 private static QWD createQWD(final FittingData data, final org.dive4elements.river.artifacts.math.Function function, final boolean isOutlier) {
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
210
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
211 final FixingColumnWithData event = data.event;
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
212 final Date date = event.getDate();
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
213 final boolean isInterpolated = data.isInterpolated;
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
214
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
215 final double w = data.w;
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
216 final double q = data.q;
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
217 final double dw = (w - function.value(q)) * 100.0;
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
218
9744ce3c3853 Rework of fixanalysis computation and dWt and WQ facets. Got rid of strange remapping and bitshifting code by explicitely saving the column information and using it in the facets.
gernotbelger
parents: 9360
diff changeset
219 return new QWD(q, w, date, isInterpolated, dw, isOutlier);
3011
ab81ffd1343e FixA: Reactivated rewrite of the outlier checks.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
220 }
9348
a3f318347707 Show wq outliers within same thems with different symbol: not ready yet
gernotbelger
parents: 7300
diff changeset
221 }
http://dive4elements.wald.intevation.org