Mercurial > dive4elements > river

annotate backend/src/main/java/org/dive4elements/river/importer/ImportSedimentLoad.java @ 9650:a2a42a6bac6b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Importer (s/u-info) extensions: outer try/catch for parse and log of line no, catching parsing exception if not enough value fields, parsing error and warning log messages with line number, detecting and rejecting duplicate data series, better differentiation between error and warning log messages
author mschaefer
date Mon, 23 Mar 2020 14:57:03 +0100
parents 392bbcd8a88b
children
rev   line source
8841
e8d2042c9639 Add missing copyright headers.
Tom Gottfried <tom@intevation.de>
parents: 8412
diff changeset
1 /* Copyright (C) 2014 by Bundesanstalt für Gewässerkunde
e8d2042c9639 Add missing copyright headers.
Tom Gottfried <tom@intevation.de>
parents: 8412
diff changeset
2 * Software engineering by Intevation GmbH
e8d2042c9639 Add missing copyright headers.
Tom Gottfried <tom@intevation.de>
parents: 8412
diff changeset
3 *
e8d2042c9639 Add missing copyright headers.
Tom Gottfried <tom@intevation.de>
parents: 8412
diff changeset
4 * This file is Free Software under the GNU AGPL (>=v3)
e8d2042c9639 Add missing copyright headers.
Tom Gottfried <tom@intevation.de>
parents: 8412
diff changeset
5 * and comes with ABSOLUTELY NO WARRANTY! Check out the
e8d2042c9639 Add missing copyright headers.
Tom Gottfried <tom@intevation.de>
parents: 8412
diff changeset
6 * documentation coming with Dive4Elements River for details.
e8d2042c9639 Add missing copyright headers.
Tom Gottfried <tom@intevation.de>
parents: 8412
diff changeset
7 */
e8d2042c9639 Add missing copyright headers.
Tom Gottfried <tom@intevation.de>
parents: 8412
diff changeset
8
8028
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
9 package org.dive4elements.river.importer;
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
10
8986
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
11 import java.util.ArrayList;
8028
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
12 import java.util.List;
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
13
8986
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
14 import org.apache.log4j.Logger;
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
15 import org.dive4elements.river.importer.common.StoreMode;
8028
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
16 import org.dive4elements.river.model.GrainFraction;
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
17 import org.dive4elements.river.model.SedimentLoad;
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
18 import org.dive4elements.river.model.TimeInterval;
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
19 import org.hibernate.Query;
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
20 import org.hibernate.Session;
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
21
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
22 public class ImportSedimentLoad
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
23 {
8986
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
24 private static Logger log = Logger.getLogger(ImportSedimentLoad.class);
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
25
8028
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
26 private SedimentLoad peer;
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
27
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
28 private ImportGrainFraction grainFraction;
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
29 private ImportTimeInterval timeInterval;
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
30 private ImportTimeInterval sqTimeInterval;
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
31 private String description;
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
32 private Integer kind;
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
33
8986
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
34 private final List<ImportSedimentLoadValue> values;
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
35
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
36 protected StoreMode storeMode;
8056
d86cc6a17b7a Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents: 8028
diff changeset
37
8028
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
38 public ImportSedimentLoad() {
8986
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
39 this.values = new ArrayList<>();
8056
d86cc6a17b7a Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents: 8028
diff changeset
40 }
d86cc6a17b7a Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents: 8028
diff changeset
41
8842
ff27548d078c Whitespace cosmetics.
Tom Gottfried <tom@intevation.de>
parents: 8841
diff changeset
42 public ImportSedimentLoad(
8986
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
43 final ImportGrainFraction grainFraction,
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
44 final ImportTimeInterval timeInterval,
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
45 final ImportTimeInterval sqTimeInterval,
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
46 final String description,
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
47 final Integer kind
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
48 ) {
8059
bde5f5ec7c72 SCHEMA CHANGE and Importer: get time intervals of SQ-relations from metalines in sediment load files.
Tom Gottfried <tom@intevation.de>
parents: 8056
diff changeset
49 this.grainFraction = grainFraction;
bde5f5ec7c72 SCHEMA CHANGE and Importer: get time intervals of SQ-relations from metalines in sediment load files.
Tom Gottfried <tom@intevation.de>
parents: 8056
diff changeset
50 this.timeInterval = timeInterval;
bde5f5ec7c72 SCHEMA CHANGE and Importer: get time intervals of SQ-relations from metalines in sediment load files.
Tom Gottfried <tom@intevation.de>
parents: 8056
diff changeset
51 this.sqTimeInterval = sqTimeInterval;
bde5f5ec7c72 SCHEMA CHANGE and Importer: get time intervals of SQ-relations from metalines in sediment load files.
Tom Gottfried <tom@intevation.de>
parents: 8056
diff changeset
52 this.description = description;
bde5f5ec7c72 SCHEMA CHANGE and Importer: get time intervals of SQ-relations from metalines in sediment load files.
Tom Gottfried <tom@intevation.de>
parents: 8056
diff changeset
53 this.kind = kind;
8056
d86cc6a17b7a Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents: 8028
diff changeset
54
8986
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
55 this.values = new ArrayList<>();
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
56 this.storeMode = StoreMode.NONE;
8056
d86cc6a17b7a Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents: 8028
diff changeset
57 }
d86cc6a17b7a Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents: 8028
diff changeset
58
8986
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
59 public void addValue(final ImportSedimentLoadValue value) {
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
60 this.values.add(value);
8028
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
61 }
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
62
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
63 public void storeDependencies() {
8986
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
64 log.info("store dependencies for '" + this.description + "'");
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
65 this.grainFraction.getPeer();
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
66 this.timeInterval.getPeer();
8056
d86cc6a17b7a Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents: 8028
diff changeset
67
8986
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
68 if (this.sqTimeInterval != null) {
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
69 this.sqTimeInterval.getPeer();
8056
d86cc6a17b7a Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents: 8028
diff changeset
70 }
8028
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
71
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
72 getPeer();
8056
d86cc6a17b7a Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents: 8028
diff changeset
73
8986
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
74 for (final ImportSedimentLoadValue value : this.values) {
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
75 value.storeDependencies(this.peer, this.storeMode);
8056
d86cc6a17b7a Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents: 8028
diff changeset
76 }
8986
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
77 log.info("Values processed: " + this.values.size());
8028
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
78 }
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
79
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
80 public SedimentLoad getPeer() {
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
81
8986
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
82 if (this.peer == null) {
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
83 final Session session = ImporterSession.getInstance()
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
84 .getDatabaseSession();
8028
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
85
8986
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
86 final String sqtquery = this.sqTimeInterval == null
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
87 ? "sq_time_interval_id is null"
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
88 : "sqTimeInterval = :sqTimeInterval";
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
89 final Query query = session.createQuery(
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
90 "from SedimentLoad where " +
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
91 " grainFraction = :grainFraction and " +
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
92 " timeInterval = :timeInterval and " +
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
93 " description = :description and " +
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
94 " kind = :kind and " +
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
95 sqtquery);
8056
d86cc6a17b7a Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents: 8028
diff changeset
96
8986
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
97 final GrainFraction gf = this.grainFraction.getPeer();
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
98 final TimeInterval ti = this.timeInterval.getPeer();
8028
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
99
8986
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
100 final TimeInterval sqti = this.sqTimeInterval != null
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
101 ? this.sqTimeInterval.getPeer()
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
102 : null;
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
103
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
104 query.setParameter("grainFraction", gf);
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
105 query.setParameter("timeInterval", ti);
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
106
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
107 if (sqti != null) {
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
108 query.setParameter("sqTimeInterval", sqti);
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
109 }
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
110 query.setParameter("description", this.description);
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
111 query.setParameter("kind", this.kind);
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
112
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
113 final List<SedimentLoad> loads = query.list();
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
114 if (loads.isEmpty()) {
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
115 this.peer = new SedimentLoad(gf, ti, sqti, this.description, this.kind);
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
116 session.save(this.peer);
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
117 this.storeMode = StoreMode.INSERT;
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
118 }
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
119 else {
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
120 this.peer = loads.get(0);
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
121 this.storeMode = StoreMode.UPDATE;
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
122 }
8028
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
123 }
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
124
8986
392bbcd8a88b Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents: 8856
diff changeset
125 return this.peer;
8028
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
126 }
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
127 }
aa054f72e887 Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
128 // vim:set ts=4 sw=4 si et sta sts=4 fenc=utf8 :
http://dive4elements.wald.intevation.org