Mercurial > dive4elements > river

annotate backend/src/main/java/org/dive4elements/river/importer/parsers/WaterlevelParser.java @ 9650:a2a42a6bac6b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Importer (s/u-info) extensions: outer try/catch for parse and log of line no, catching parsing exception if not enough value fields, parsing error and warning log messages with line number, detecting and rejecting duplicate data series, better differentiation between error and warning log messages
author mschaefer
date Mon, 23 Mar 2020 14:57:03 +0100
parents c43d8c1a4455
children
rev   line source
5844
4dd33b86dc61 Added header to river backend.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
1 /* Copyright (C) 2011, 2012, 2013 by Bundesanstalt für Gewässerkunde
4dd33b86dc61 Added header to river backend.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
2 * Software engineering by Intevation GmbH
4dd33b86dc61 Added header to river backend.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
3 *
5992
4c3ccf2b0304 Removed trailing whitespace.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5844
diff changeset
4 * This file is Free Software under the GNU AGPL (>=v3)
5844
4dd33b86dc61 Added header to river backend.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
5 * and comes with ABSOLUTELY NO WARRANTY! Check out the
5992
4c3ccf2b0304 Removed trailing whitespace.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5844
diff changeset
6 * documentation coming with Dive4Elements River for details.
5844
4dd33b86dc61 Added header to river backend.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
7 */
4dd33b86dc61 Added header to river backend.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
8
5829
18619c1e7c2a Repaired internal references.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5828
diff changeset
9 package org.dive4elements.river.importer.parsers;
2844
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
10
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
11 import java.io.File;
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
12 import java.io.IOException;
5049
b5e3df1feeb4 WaterlevelParser: Add Wst-Converting functionality.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
13 import java.math.BigDecimal;
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
14 import java.text.NumberFormat;
8991
c43d8c1a4455 Parse via decimalformat instead of bigdecimal constructor (to cope with strings with chars after the actual number)
mschaefer
parents: 8989
diff changeset
15 import java.text.ParseException;
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
16 import java.util.ArrayList;
2844
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
17 import java.util.List;
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
18 import java.util.regex.Matcher;
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
19 import java.util.regex.Pattern;
2844
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
20
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
21 import org.apache.log4j.Logger;
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
22 import org.dive4elements.river.backend.utils.DateUtil;
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
23 import org.dive4elements.river.importer.ImportRange;
6319
550cb41d37d9 issue1354: TimeIntervals in Waterlevels from csv files.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 6282
diff changeset
24 import org.dive4elements.river.importer.ImportTimeInterval;
5829
18619c1e7c2a Repaired internal references.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5828
diff changeset
25 import org.dive4elements.river.importer.ImportUnit;
18619c1e7c2a Repaired internal references.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5828
diff changeset
26 import org.dive4elements.river.importer.ImportWst;
18619c1e7c2a Repaired internal references.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5828
diff changeset
27 import org.dive4elements.river.importer.ImportWstColumn;
18619c1e7c2a Repaired internal references.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5828
diff changeset
28 import org.dive4elements.river.importer.ImportWstColumnValue;
18619c1e7c2a Repaired internal references.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5828
diff changeset
29 import org.dive4elements.river.importer.ImportWstQRange;
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
30 import org.dive4elements.river.importer.common.AbstractParser;
2844
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
31
5049
b5e3df1feeb4 WaterlevelParser: Add Wst-Converting functionality.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
32
b5e3df1feeb4 WaterlevelParser: Add Wst-Converting functionality.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
33 /**
b5e3df1feeb4 WaterlevelParser: Add Wst-Converting functionality.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
34 * Parse CSV Waterlevel files.
b5e3df1feeb4 WaterlevelParser: Add Wst-Converting functionality.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
35 * As these waterlevels are probably used in fixation analysis
b5e3df1feeb4 WaterlevelParser: Add Wst-Converting functionality.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
36 * only, functionality to export them to "fixation"-wsts
b5e3df1feeb4 WaterlevelParser: Add Wst-Converting functionality.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
37 * has been added (the ImportWaterlevel*- stuff is actually
b5e3df1feeb4 WaterlevelParser: Add Wst-Converting functionality.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
38 * not needed to do so.)
b5e3df1feeb4 WaterlevelParser: Add Wst-Converting functionality.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
39 */
2844
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
40 public class WaterlevelParser extends LineParser {
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
41
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
42 private static final Logger log = Logger.getLogger(WaterlevelParser.class);
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
43
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
44 private static final NumberFormat nf =
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
45 NumberFormat.getInstance(DEFAULT_LOCALE);
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
46
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
47 private static final Pattern META_Q_RANGE =
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
48 Pattern.compile("Abfluss\\s\\[(.*)\\];(.*)");
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
49
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
50 public static final Pattern META_UNIT =
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
51 Pattern.compile("^Einheit: \\[(.*)\\].*");
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
52
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
53 public static final BigDecimal INTERVAL_GAP = new BigDecimal("0.00001");
7377
ad2fdc34910a Importer: Avoid 0-lenght Q-ranges in WST-CSV-parsers.
Tom Gottfried <tom@intevation.de>
parents: 6319
diff changeset
54
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
55 private final List<ImportWst> waterlevels;
2844
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
56
5240
9c024718bffb WaterlevelParser: Use Wst-structures directly (instead of Waterlevel*).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5231
diff changeset
57 private ImportWst current;
9c024718bffb WaterlevelParser: Use Wst-structures directly (instead of Waterlevel*).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5231
diff changeset
58
9c024718bffb WaterlevelParser: Use Wst-structures directly (instead of Waterlevel*).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5231
diff changeset
59 /** The Waterlevel-Wst s will always have but one column. */
9c024718bffb WaterlevelParser: Use Wst-structures directly (instead of Waterlevel*).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5231
diff changeset
60 private ImportWstColumn column;
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
61
5240
9c024718bffb WaterlevelParser: Use Wst-structures directly (instead of Waterlevel*).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5231
diff changeset
62 /** The current (incomplete) Q Range. */
9c024718bffb WaterlevelParser: Use Wst-structures directly (instead of Waterlevel*).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5231
diff changeset
63 private ImportWstQRange currentQRange;
9c024718bffb WaterlevelParser: Use Wst-structures directly (instead of Waterlevel*).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5231
diff changeset
64
9c024718bffb WaterlevelParser: Use Wst-structures directly (instead of Waterlevel*).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5231
diff changeset
65 /** The current (incomplete) km range for Q Range. */
9c024718bffb WaterlevelParser: Use Wst-structures directly (instead of Waterlevel*).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5231
diff changeset
66 private ImportRange currentRange;
9c024718bffb WaterlevelParser: Use Wst-structures directly (instead of Waterlevel*).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5231
diff changeset
67
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
68 private String currentDescription;
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
69
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
70
2844
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
71 public WaterlevelParser() {
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
72 this.waterlevels = new ArrayList<>();
2844
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
73 }
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
74
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
75
5240
9c024718bffb WaterlevelParser: Use Wst-structures directly (instead of Waterlevel*).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5231
diff changeset
76 public List<ImportWst> getWaterlevels() {
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
77 return this.waterlevels;
2844
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
78 }
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
79
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
80
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
81 @Override
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
82 public void parse(final File file) throws IOException {
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
83 this.currentDescription = file.getName();
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
84
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
85 super.parse(file);
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
86 }
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
87
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
88
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
89 @Override
2844
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
90 protected void reset() {
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
91 this.currentQRange = null;
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
92 this.current = new ImportWst(this.currentDescription);
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
93 this.current.setNumberColumns(1);
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
94 this.column = this.current.getColumn(0);
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
95 this.column.setName(this.currentDescription);
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
96 this.column.setDescription(this.currentDescription);
6319
550cb41d37d9 issue1354: TimeIntervals in Waterlevels from csv files.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 6282
diff changeset
97
550cb41d37d9 issue1354: TimeIntervals in Waterlevels from csv files.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 6282
diff changeset
98 // Try to extract and set the TimeInterval.
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
99 final Matcher m = WaterlevelDifferencesParser.YEARS_IN_COLUMN.matcher(
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
100 this.currentDescription);
6319
550cb41d37d9 issue1354: TimeIntervals in Waterlevels from csv files.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 6282
diff changeset
101
550cb41d37d9 issue1354: TimeIntervals in Waterlevels from csv files.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 6282
diff changeset
102 if (m.matches()) {
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
103 final int startYear = Integer.parseInt(m.group(1));
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
104 final int endYear = Integer.parseInt(m.group(2));
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
105 final ImportTimeInterval time = new ImportTimeInterval(
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
106 DateUtil.getStartDateFromYear(startYear),
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
107 DateUtil.getEndDateFromYear(endYear)
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
108 );
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
109 this.column.setTimeInterval(time);
6319
550cb41d37d9 issue1354: TimeIntervals in Waterlevels from csv files.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 6282
diff changeset
110 } else {
8856
5e38e2924c07 Fix code style.
Tom Gottfried <tom@intevation.de>
parents: 8187
diff changeset
111 log.debug("No time interval in column header found: "
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
112 + this.currentDescription);
6319
550cb41d37d9 issue1354: TimeIntervals in Waterlevels from csv files.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 6282
diff changeset
113 }
550cb41d37d9 issue1354: TimeIntervals in Waterlevels from csv files.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 6282
diff changeset
114
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
115 this.current.setKind(7);
2844
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
116 }
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
117
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
118
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
119 @Override
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
120 protected void finish() {
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
121 if (this.current != null) {
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
122 if (this.currentQRange != null) {
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
123 final List<ImportWstColumnValue> cValues = this.column.getColumnValues();
7377
ad2fdc34910a Importer: Avoid 0-lenght Q-ranges in WST-CSV-parsers.
Tom Gottfried <tom@intevation.de>
parents: 6319
diff changeset
124 // Set end of range to last station
ad2fdc34910a Importer: Avoid 0-lenght Q-ranges in WST-CSV-parsers.
Tom Gottfried <tom@intevation.de>
parents: 6319
diff changeset
125 // or expand range to minimal length in case it would be 0
7883
fc8d5b5f2c99 Removed trailing whitespace.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 7377
diff changeset
126 // TODO: should otherwise be extended to
fc8d5b5f2c99 Removed trailing whitespace.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 7377
diff changeset
127 // (first station of next range - INTERVAL_GAP),
7377
ad2fdc34910a Importer: Avoid 0-lenght Q-ranges in WST-CSV-parsers.
Tom Gottfried <tom@intevation.de>
parents: 6319
diff changeset
128 // assuming always ascending stations
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
129 final BigDecimal lastStation = cValues.get(cValues.size() -1)
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
130 .getPosition();
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
131 if (lastStation.compareTo(this.currentRange.getA()) == 0) {
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
132 this.currentRange.setB(lastStation.add(INTERVAL_GAP));
7377
ad2fdc34910a Importer: Avoid 0-lenght Q-ranges in WST-CSV-parsers.
Tom Gottfried <tom@intevation.de>
parents: 6319
diff changeset
133 }
ad2fdc34910a Importer: Avoid 0-lenght Q-ranges in WST-CSV-parsers.
Tom Gottfried <tom@intevation.de>
parents: 6319
diff changeset
134 else {
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
135 this.currentRange.setB(lastStation);
7377
ad2fdc34910a Importer: Avoid 0-lenght Q-ranges in WST-CSV-parsers.
Tom Gottfried <tom@intevation.de>
parents: 6319
diff changeset
136 }
ad2fdc34910a Importer: Avoid 0-lenght Q-ranges in WST-CSV-parsers.
Tom Gottfried <tom@intevation.de>
parents: 6319
diff changeset
137
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
138 this.currentQRange.setRange(this.currentRange);
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
139 this.column.addColumnQRange(this.currentQRange);
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
140 }
2844
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
141
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
142 this.waterlevels.add(this.current);
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
143 }
2844
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
144 }
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
145
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
146 @Override
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
147 protected void handleLine(final int lineNum, final String line) {
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
148 if (line.startsWith(START_META_CHAR)) {
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
149 handleMetaLine(stripMetaLine(line));
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
150 return;
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
151 }
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
152 else if (handleQRange(line)) {
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
153 return;
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
154 }
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
155 else {
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
156 handleDataLine(line);
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
157 return;
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
158 }
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
159 }
2844
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
160
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
161
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
162 private void handleMetaLine(final String meta) {
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
163 final Matcher m = META_UNIT.matcher(meta);
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
164
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
165 if (m.matches()) {
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
166 final String unit = m.group(1);
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
167 log.debug("Found unit: '" + unit + "'");
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
168
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
169 this.current.setUnit(new ImportUnit(unit));
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
170 }
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
171 }
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
172
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
173
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
174 private boolean handleQRange(final String line) {
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
175 final Matcher m = META_Q_RANGE.matcher(line);
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
176
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
177 if (m.matches()) {
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
178 final String unitStr = m.group(1);
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
179 final String valueStr = m.group(2);
5240
9c024718bffb WaterlevelParser: Use Wst-structures directly (instead of Waterlevel*).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5231
diff changeset
180 try {
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
181 if (this.currentQRange != null) {
5240
9c024718bffb WaterlevelParser: Use Wst-structures directly (instead of Waterlevel*).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5231
diff changeset
182 // Finish off the last one.
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
183 final List<ImportWstColumnValue> cValues = this.column
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
184 .getColumnValues();
5240
9c024718bffb WaterlevelParser: Use Wst-structures directly (instead of Waterlevel*).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5231
diff changeset
185 // Set end of range to last station.
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
186 this.currentRange.setB(cValues.get(cValues.size() -1)
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
187 .getPosition());
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
188 this.currentQRange.setRange(this.currentRange);
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
189 this.column.addColumnQRange(this.currentQRange);
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
190 }
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
191 this.currentQRange = new ImportWstQRange(null,
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
192 AbstractParser.parseDecimal(valueStr));
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
193 this.currentRange = new ImportRange();
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
194
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
195 log.debug("Found new Q range: Q=" + valueStr);
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
196
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
197 return true;
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
198 }
8991
c43d8c1a4455 Parse via decimalformat instead of bigdecimal constructor (to cope with strings with chars after the actual number)
mschaefer
parents: 8989
diff changeset
199 catch (final ParseException pe) {
5490
b3dd14fc13a6 Parsers: do not call warnings error
Tom Gottfried <tom.gottfried@intevation.de>
parents: 5264
diff changeset
200 log.warn("Unparseable Q range: '" + line + "'");
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
201 }
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
202 }
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
203
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
204 return false;
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
205 }
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
206
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
207
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
208 private void handleDataLine(final String line) {
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
209 final String[] cols = line.split(SEPERATOR_CHAR);
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
210
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
211 if (cols == null || cols.length < 2) {
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
212 log.warn("skip invalid waterlevel line: '" + line + "'");
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
213 return;
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
214 }
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
215
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
216 try {
5240
9c024718bffb WaterlevelParser: Use Wst-structures directly (instead of Waterlevel*).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5231
diff changeset
217 // Store the value and remember the position for QRange, if needed.
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
218 final BigDecimal station = AbstractParser.parseDecimal(cols[0]);
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
219 final BigDecimal value = AbstractParser.parseDecimal(cols[1]);
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
220
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
221 this.column.addColumnValue(station, value);
5240
9c024718bffb WaterlevelParser: Use Wst-structures directly (instead of Waterlevel*).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5231
diff changeset
222
8989
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
223 if (this.currentRange.getA() == null) {
2693bfaf503d Fixed several BigDecimal(double) creations by BigDecimal(String) parsing to avoid unnecessary decimal digits
mschaefer
parents: 8856
diff changeset
224 this.currentRange.setA(station);
5240
9c024718bffb WaterlevelParser: Use Wst-structures directly (instead of Waterlevel*).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5231
diff changeset
225 }
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
226 }
8991
c43d8c1a4455 Parse via decimalformat instead of bigdecimal constructor (to cope with strings with chars after the actual number)
mschaefer
parents: 8989
diff changeset
227 catch (final ParseException pe) {
5490
b3dd14fc13a6 Parsers: do not call warnings error
Tom Gottfried <tom.gottfried@intevation.de>
parents: 5264
diff changeset
228 log.warn("Unparseable number in data row: " + line);
2845
a65b5b0ade6f Completed the MINFO waterlevel import.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2844
diff changeset
229 }
2844
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
230 }
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
231 }
c4db0f75a94a Prepared the importer to import MINFO specific waterlevel values and added a stub for a parser.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
232 // vim:set ts=4 sw=4 si et sta sts=4 fenc=utf8 :
http://dive4elements.wald.intevation.org