Mercurial > dive4elements > river
annotate backend/src/main/java/org/dive4elements/river/importer/ImportSedimentLoad.java @ 9650:a2a42a6bac6b
Importer (s/u-info) extensions:
outer try/catch for parse and log of line no,
catching parsing exception if not enough value fields,
parsing error and warning log messages with line number,
detecting and rejecting duplicate data series,
better differentiation between error and warning log messages
author | mschaefer |
---|---|
date | Mon, 23 Mar 2020 14:57:03 +0100 |
parents | 392bbcd8a88b |
children |
rev | line source |
---|---|
8841
e8d2042c9639
Add missing copyright headers.
Tom Gottfried <tom@intevation.de>
parents:
8412
diff
changeset
|
1 /* Copyright (C) 2014 by Bundesanstalt für Gewässerkunde |
e8d2042c9639
Add missing copyright headers.
Tom Gottfried <tom@intevation.de>
parents:
8412
diff
changeset
|
2 * Software engineering by Intevation GmbH |
e8d2042c9639
Add missing copyright headers.
Tom Gottfried <tom@intevation.de>
parents:
8412
diff
changeset
|
3 * |
e8d2042c9639
Add missing copyright headers.
Tom Gottfried <tom@intevation.de>
parents:
8412
diff
changeset
|
4 * This file is Free Software under the GNU AGPL (>=v3) |
e8d2042c9639
Add missing copyright headers.
Tom Gottfried <tom@intevation.de>
parents:
8412
diff
changeset
|
5 * and comes with ABSOLUTELY NO WARRANTY! Check out the |
e8d2042c9639
Add missing copyright headers.
Tom Gottfried <tom@intevation.de>
parents:
8412
diff
changeset
|
6 * documentation coming with Dive4Elements River for details. |
e8d2042c9639
Add missing copyright headers.
Tom Gottfried <tom@intevation.de>
parents:
8412
diff
changeset
|
7 */ |
e8d2042c9639
Add missing copyright headers.
Tom Gottfried <tom@intevation.de>
parents:
8412
diff
changeset
|
8 |
8028
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
9 package org.dive4elements.river.importer; |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
10 |
8986
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
11 import java.util.ArrayList; |
8028
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
12 import java.util.List; |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
13 |
8986
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
14 import org.apache.log4j.Logger; |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
15 import org.dive4elements.river.importer.common.StoreMode; |
8028
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
16 import org.dive4elements.river.model.GrainFraction; |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
17 import org.dive4elements.river.model.SedimentLoad; |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
18 import org.dive4elements.river.model.TimeInterval; |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
19 import org.hibernate.Query; |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
20 import org.hibernate.Session; |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
21 |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
22 public class ImportSedimentLoad |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
23 { |
8986
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
24 private static Logger log = Logger.getLogger(ImportSedimentLoad.class); |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
25 |
8028
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
26 private SedimentLoad peer; |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
27 |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
28 private ImportGrainFraction grainFraction; |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
29 private ImportTimeInterval timeInterval; |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
30 private ImportTimeInterval sqTimeInterval; |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
31 private String description; |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
32 private Integer kind; |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
33 |
8986
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
34 private final List<ImportSedimentLoadValue> values; |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
35 |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
36 protected StoreMode storeMode; |
8056
d86cc6a17b7a
Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents:
8028
diff
changeset
|
37 |
8028
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
38 public ImportSedimentLoad() { |
8986
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
39 this.values = new ArrayList<>(); |
8056
d86cc6a17b7a
Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents:
8028
diff
changeset
|
40 } |
d86cc6a17b7a
Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents:
8028
diff
changeset
|
41 |
8842 | 42 public ImportSedimentLoad( |
8986
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
43 final ImportGrainFraction grainFraction, |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
44 final ImportTimeInterval timeInterval, |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
45 final ImportTimeInterval sqTimeInterval, |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
46 final String description, |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
47 final Integer kind |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
48 ) { |
8059
bde5f5ec7c72
SCHEMA CHANGE and Importer: get time intervals of SQ-relations from metalines in sediment load files.
Tom Gottfried <tom@intevation.de>
parents:
8056
diff
changeset
|
49 this.grainFraction = grainFraction; |
bde5f5ec7c72
SCHEMA CHANGE and Importer: get time intervals of SQ-relations from metalines in sediment load files.
Tom Gottfried <tom@intevation.de>
parents:
8056
diff
changeset
|
50 this.timeInterval = timeInterval; |
bde5f5ec7c72
SCHEMA CHANGE and Importer: get time intervals of SQ-relations from metalines in sediment load files.
Tom Gottfried <tom@intevation.de>
parents:
8056
diff
changeset
|
51 this.sqTimeInterval = sqTimeInterval; |
bde5f5ec7c72
SCHEMA CHANGE and Importer: get time intervals of SQ-relations from metalines in sediment load files.
Tom Gottfried <tom@intevation.de>
parents:
8056
diff
changeset
|
52 this.description = description; |
bde5f5ec7c72
SCHEMA CHANGE and Importer: get time intervals of SQ-relations from metalines in sediment load files.
Tom Gottfried <tom@intevation.de>
parents:
8056
diff
changeset
|
53 this.kind = kind; |
8056
d86cc6a17b7a
Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents:
8028
diff
changeset
|
54 |
8986
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
55 this.values = new ArrayList<>(); |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
56 this.storeMode = StoreMode.NONE; |
8056
d86cc6a17b7a
Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents:
8028
diff
changeset
|
57 } |
d86cc6a17b7a
Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents:
8028
diff
changeset
|
58 |
8986
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
59 public void addValue(final ImportSedimentLoadValue value) { |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
60 this.values.add(value); |
8028
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
61 } |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
62 |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
63 public void storeDependencies() { |
8986
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
64 log.info("store dependencies for '" + this.description + "'"); |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
65 this.grainFraction.getPeer(); |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
66 this.timeInterval.getPeer(); |
8056
d86cc6a17b7a
Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents:
8028
diff
changeset
|
67 |
8986
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
68 if (this.sqTimeInterval != null) { |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
69 this.sqTimeInterval.getPeer(); |
8056
d86cc6a17b7a
Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents:
8028
diff
changeset
|
70 } |
8028
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
71 |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
72 getPeer(); |
8056
d86cc6a17b7a
Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents:
8028
diff
changeset
|
73 |
8986
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
74 for (final ImportSedimentLoadValue value : this.values) { |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
75 value.storeDependencies(this.peer, this.storeMode); |
8056
d86cc6a17b7a
Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents:
8028
diff
changeset
|
76 } |
8986
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
77 log.info("Values processed: " + this.values.size()); |
8028
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
78 } |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
79 |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
80 public SedimentLoad getPeer() { |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
81 |
8986
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
82 if (this.peer == null) { |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
83 final Session session = ImporterSession.getInstance() |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
84 .getDatabaseSession(); |
8028
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
85 |
8986
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
86 final String sqtquery = this.sqTimeInterval == null |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
87 ? "sq_time_interval_id is null" |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
88 : "sqTimeInterval = :sqTimeInterval"; |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
89 final Query query = session.createQuery( |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
90 "from SedimentLoad where " + |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
91 " grainFraction = :grainFraction and " + |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
92 " timeInterval = :timeInterval and " + |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
93 " description = :description and " + |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
94 " kind = :kind and " + |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
95 sqtquery); |
8056
d86cc6a17b7a
Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents:
8028
diff
changeset
|
96 |
8986
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
97 final GrainFraction gf = this.grainFraction.getPeer(); |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
98 final TimeInterval ti = this.timeInterval.getPeer(); |
8028
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
99 |
8986
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
100 final TimeInterval sqti = this.sqTimeInterval != null |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
101 ? this.sqTimeInterval.getPeer() |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
102 : null; |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
103 |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
104 query.setParameter("grainFraction", gf); |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
105 query.setParameter("timeInterval", ti); |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
106 |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
107 if (sqti != null) { |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
108 query.setParameter("sqTimeInterval", sqti); |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
109 } |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
110 query.setParameter("description", this.description); |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
111 query.setParameter("kind", this.kind); |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
112 |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
113 final List<SedimentLoad> loads = query.list(); |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
114 if (loads.isEmpty()) { |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
115 this.peer = new SedimentLoad(gf, ti, sqti, this.description, this.kind); |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
116 session.save(this.peer); |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
117 this.storeMode = StoreMode.INSERT; |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
118 } |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
119 else { |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
120 this.peer = loads.get(0); |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
121 this.storeMode = StoreMode.UPDATE; |
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
122 } |
8028
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
123 } |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
124 |
8986
392bbcd8a88b
Database inserts accelerated by suppressing unnecessary database queries for new data series
mschaefer
parents:
8856
diff
changeset
|
125 return this.peer; |
8028
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
126 } |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
127 } |
aa054f72e887
Added importer models for sediment load and sediment load values.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff
changeset
|
128 // vim:set ts=4 sw=4 si et sta sts=4 fenc=utf8 : |