Mercurial > dive4elements > river

annotate backend/src/main/java/org/dive4elements/river/importer/Importer.java @ 9729:7e767c0c9a6d 3.2.x

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Improve importer start _ Consider no given info gew files as error instead of starting to process an empty list. _ Reduce code duplication.
author Tom Gottfried <tom@intevation.de>
date Thu, 07 Jul 2022 11:19:08 +0200
parents 0a5239a1e46e
children
rev   line source
5844
4dd33b86dc61 Added header to river backend.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
1 /* Copyright (C) 2011, 2012, 2013 by Bundesanstalt für Gewässerkunde
4dd33b86dc61 Added header to river backend.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
2 * Software engineering by Intevation GmbH
4dd33b86dc61 Added header to river backend.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
3 *
5992
4c3ccf2b0304 Removed trailing whitespace.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5844
diff changeset
4 * This file is Free Software under the GNU AGPL (>=v3)
5844
4dd33b86dc61 Added header to river backend.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
5 * and comes with ABSOLUTELY NO WARRANTY! Check out the
5992
4c3ccf2b0304 Removed trailing whitespace.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5844
diff changeset
6 * documentation coming with Dive4Elements River for details.
5844
4dd33b86dc61 Added header to river backend.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
7 */
4dd33b86dc61 Added header to river backend.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5838
diff changeset
8
5829
18619c1e7c2a Repaired internal references.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5828
diff changeset
9 package org.dive4elements.river.importer;
177
31895d24387e Importer: Added info gew parser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
10
5829
18619c1e7c2a Repaired internal references.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5828
diff changeset
11 import org.dive4elements.artifacts.common.utils.XMLUtils;
765
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
12
5829
18619c1e7c2a Repaired internal references.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5828
diff changeset
13 import org.dive4elements.river.importer.parsers.AnnotationClassifier;
18619c1e7c2a Repaired internal references.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5828
diff changeset
14 import org.dive4elements.river.importer.parsers.BundesWasserStrassenParser;
18619c1e7c2a Repaired internal references.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 5828
diff changeset
15 import org.dive4elements.river.importer.parsers.InfoGewParser;
1211
f08fe480092c Moved file parsers to separate package.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 766
diff changeset
16
177
31895d24387e Importer: Added info gew parser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
17 import java.io.File;
31895d24387e Importer: Added info gew parser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
18 import java.io.IOException;
31895d24387e Importer: Added info gew parser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
19
9729
7e767c0c9a6d Improve importer start
Tom Gottfried <tom@intevation.de>
parents: 9726
diff changeset
20 import java.util.ArrayList;
7e767c0c9a6d Improve importer start
Tom Gottfried <tom@intevation.de>
parents: 9726
diff changeset
21 import java.util.Arrays;
5027
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
22 import java.util.HashMap;
180
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
23 import java.util.List;
5027
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
24 import java.util.Map;
180
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
25
188
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
26 import java.sql.SQLException;
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
27
9726
0a5239a1e46e Upgrade to Log4j 2
Tom Gottfried <tom@intevation.de>
parents: 8856
diff changeset
28 import org.apache.logging.log4j.Logger;
0a5239a1e46e Upgrade to Log4j 2
Tom Gottfried <tom@intevation.de>
parents: 8856
diff changeset
29 import org.apache.logging.log4j.LogManager;
177
31895d24387e Importer: Added info gew parser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
30
180
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
31 import org.hibernate.Transaction;
188
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
32 import org.hibernate.HibernateException;
180
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
33
765
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
34 import org.w3c.dom.Document;
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
35
8187
3bb1c62ad732 Moved package org.dive4elements.river.utils to org.dive4elements.river.backend.utils.
Sascha L. Teichmann <teichmann@intevation.de>
parents: 6890
diff changeset
36 import org.dive4elements.river.backend.utils.StringUtil;
4707
2b0426b79a92 Documentation.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 3659
diff changeset
37
5016
255898799ed9 Minimal whitespace and doc cosmetics.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 4708
diff changeset
38 /** Data Importer. Further processing happens per-river. */
177
31895d24387e Importer: Added info gew parser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
39 public class Importer
31895d24387e Importer: Added info gew parser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
40 {
8200
9d2e69f971f5 sed -i src/**/*.java 's/logger/log/g'
Sascha L. Teichmann <teichmann@intevation.de>
parents: 8187
diff changeset
41 /** Private log. */
9726
0a5239a1e46e Upgrade to Log4j 2
Tom Gottfried <tom@intevation.de>
parents: 8856
diff changeset
42 private static final Logger log = LogManager.getLogger(Importer.class);
177
31895d24387e Importer: Added info gew parser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
43
8843
3ecf1f76b2b8 Mark constants final.
Tom Gottfried <tom@intevation.de>
parents: 8821
diff changeset
44 private static final String BWASTR_ID_CSV_FILE = "BWASTR_ID.csv";
5027
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
45
180
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
46 protected List<ImportRiver> rivers;
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
47
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
48 public Importer() {
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
49 }
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
50
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
51 public Importer(List<ImportRiver> rivers) {
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
52 this.rivers = rivers;
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
53 }
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
54
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
55 public List<ImportRiver> getRivers() {
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
56 return rivers;
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
57 }
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
58
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
59 public void setRivers(List<ImportRiver> rivers) {
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
60 this.rivers = rivers;
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
61 }
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
62
5027
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
63 /** Write rivers and their dependencies/dependants to db. */
180
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
64 public void writeRivers() {
182
1c0afb01bc93 Importer: Fixed error in HQL statement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 180
diff changeset
65 log.debug("write rivers started");
188
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
66
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
67 for (ImportRiver river: rivers) {
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
68 log.debug("writing river '" + river.getName() + "'");
190
d40da430d2fe Added stub code to write gauges
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 189
diff changeset
69 river.storeDependencies();
497
67fd63e4ef66 Importer: centralized caching
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 199
diff changeset
70 ImporterSession.getInstance().getDatabaseSession().flush();
180
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
71 }
188
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
72
182
1c0afb01bc93 Importer: Fixed error in HQL statement.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 180
diff changeset
73 log.debug("write rivers finished");
180
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
74 }
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
75
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
76 public void writeToDatabase() {
183
222f4db3430a Importer: Used thread local pattern to make sharing of session easier.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 182
diff changeset
77
180
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
78 Transaction tx = null;
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
79
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
80 try {
497
67fd63e4ef66 Importer: centralized caching
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 199
diff changeset
81 tx = ImporterSession.getInstance()
67fd63e4ef66 Importer: centralized caching
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 199
diff changeset
82 .getDatabaseSession().beginTransaction();
180
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
83
188
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
84 try {
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
85 writeRivers();
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
86 }
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
87 catch (HibernateException he) {
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
88 Throwable t = he.getCause();
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
89 while (t instanceof SQLException) {
5238
2fceb5511dc0 Tiny whitespace cosmetic.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5106
diff changeset
90 SQLException sqle = (SQLException) t;
188
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
91 log.error("SQL exeception chain:", sqle);
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
92 t = sqle.getNextException();
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
93 }
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
94 throw he;
003ac16812dd Store annotations in backend.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 183
diff changeset
95 }
180
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
96
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
97 tx.commit();
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
98 }
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
99 catch (RuntimeException re) {
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
100 if (tx != null) {
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
101 tx.rollback();
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
102 }
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
103 throw re;
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
104 }
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
105 }
3908bfb2aa43 Store imported rivers into database. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 177
diff changeset
106
765
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
107 public static AnnotationClassifier getAnnotationClassifier() {
1223
268f8da412e3 Importer: Added a central configuration to allow skipping of parsing/storing individual sub systems.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1211
diff changeset
108 String annotationTypes = Config.INSTANCE.getAnnotationTypes();
765
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
109
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
110 if (annotationTypes == null) {
766
aa9e3da95c31 Importer: Fix import of annotation type classification.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 765
diff changeset
111 log.info("no annotation types file configured.");
765
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
112 return null;
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
113 }
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
114
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
115 File file = new File(annotationTypes);
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
116
766
aa9e3da95c31 Importer: Fix import of annotation type classification.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 765
diff changeset
117 log.info("use annotation types file '" + file + "'");
aa9e3da95c31 Importer: Fix import of annotation type classification.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 765
diff changeset
118
765
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
119 if (!(file.isFile() && file.canRead())) {
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
120 log.warn("annotation type file '" + file + "' is not readable.");
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
121 return null;
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
122 }
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
123
8821
a02204997546 Fix D4E framework API-call.
Tom Gottfried <tom@intevation.de>
parents: 8200
diff changeset
124 Document rules = XMLUtils.parseDocument(file, false, null);
765
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
125
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
126 if (rules == null) {
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
127 log.warn("cannot parse annotation types file.");
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
128 return null;
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
129 }
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
130
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
131 return new AnnotationClassifier(rules);
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
132 }
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
133
4707
2b0426b79a92 Documentation.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 3659
diff changeset
134
2b0426b79a92 Documentation.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 3659
diff changeset
135 /** Starting point for importing river data. */
177
31895d24387e Importer: Added info gew parser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
136 public static void main(String [] args) {
31895d24387e Importer: Added info gew parser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
137
9729
7e767c0c9a6d Improve importer start
Tom Gottfried <tom@intevation.de>
parents: 9726
diff changeset
138 // Take paths of GEW files from arguments and system property
7e767c0c9a6d Improve importer start
Tom Gottfried <tom@intevation.de>
parents: 9726
diff changeset
139 List<String> gews = new ArrayList<>(Arrays.asList(args));
7e767c0c9a6d Improve importer start
Tom Gottfried <tom@intevation.de>
parents: 9726
diff changeset
140 String gewProperty = Config.INSTANCE.getInfoGewFile();
7e767c0c9a6d Improve importer start
Tom Gottfried <tom@intevation.de>
parents: 9726
diff changeset
141 if (gewProperty != null && gewProperty.length() > 0) {
7e767c0c9a6d Improve importer start
Tom Gottfried <tom@intevation.de>
parents: 9726
diff changeset
142 gews.add(gewProperty);
7e767c0c9a6d Improve importer start
Tom Gottfried <tom@intevation.de>
parents: 9726
diff changeset
143 }
7e767c0c9a6d Improve importer start
Tom Gottfried <tom@intevation.de>
parents: 9726
diff changeset
144
7e767c0c9a6d Improve importer start
Tom Gottfried <tom@intevation.de>
parents: 9726
diff changeset
145 if (gews.isEmpty()) {
7e767c0c9a6d Improve importer start
Tom Gottfried <tom@intevation.de>
parents: 9726
diff changeset
146 log.error("No info gew files given");
7e767c0c9a6d Improve importer start
Tom Gottfried <tom@intevation.de>
parents: 9726
diff changeset
147 System.exit(1);
7e767c0c9a6d Improve importer start
Tom Gottfried <tom@intevation.de>
parents: 9726
diff changeset
148 }
7e767c0c9a6d Improve importer start
Tom Gottfried <tom@intevation.de>
parents: 9726
diff changeset
149
765
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
150 InfoGewParser infoGewParser = new InfoGewParser(
763c4137d6e1 Added classification of annotation types. Needs testing!
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 497
diff changeset
151 getAnnotationClassifier());
177
31895d24387e Importer: Added info gew parser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
152
2806
33f40b23edd8 Initial checkin for parsing MINFO bed heights.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2804
diff changeset
153 log.info("Start parsing rivers...");
33f40b23edd8 Initial checkin for parsing MINFO bed heights.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2804
diff changeset
154
5027
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
155 File bwastrFile = null;
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
156
9729
7e767c0c9a6d Improve importer start
Tom Gottfried <tom@intevation.de>
parents: 9726
diff changeset
157 for (String gew: gews) {
2804
68d1661a57cf Added a new config option to specify a INFO.gew file by system property.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 1223
diff changeset
158 log.info("parsing info gew file: " + gew);
5027
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
159 File gewFile = new File(gew);
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
160 if (bwastrFile == null) {
8856
5e38e2924c07 Fix code style.
Tom Gottfried <tom@intevation.de>
parents: 8843
diff changeset
161 bwastrFile = new File(
5e38e2924c07 Fix code style.
Tom Gottfried <tom@intevation.de>
parents: 8843
diff changeset
162 gewFile.getParentFile(), BWASTR_ID_CSV_FILE);
5027
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
163 }
2804
68d1661a57cf Added a new config option to specify a INFO.gew file by system property.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 1223
diff changeset
164 try {
5027
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
165 infoGewParser.parse(gewFile);
2804
68d1661a57cf Added a new config option to specify a INFO.gew file by system property.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 1223
diff changeset
166 }
68d1661a57cf Added a new config option to specify a INFO.gew file by system property.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 1223
diff changeset
167 catch (IOException ioe) {
5106
d16398d65a59 Importer: Print a decent error when failing to parse a gew and exit.
Andre Heinecke <aheinecke@intevation.de>
parents: 5080
diff changeset
168 log.error("error while parsing gew: " + gew, ioe);
d16398d65a59 Importer: Print a decent error when failing to parse a gew and exit.
Andre Heinecke <aheinecke@intevation.de>
parents: 5080
diff changeset
169 System.exit(1);
2804
68d1661a57cf Added a new config option to specify a INFO.gew file by system property.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 1223
diff changeset
170 }
68d1661a57cf Added a new config option to specify a INFO.gew file by system property.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 1223
diff changeset
171 }
68d1661a57cf Added a new config option to specify a INFO.gew file by system property.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 1223
diff changeset
172
5027
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
173 // Look for official numbers.
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
174 BundesWasserStrassenParser bwastrIdParser =
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
175 new BundesWasserStrassenParser();
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
176
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
177 // Read bwastFile (river-dir + BWASTR_ID_CSV_FILE).
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
178 if (!Config.INSTANCE.skipBWASTR()) {
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
179 try{
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
180 bwastrIdParser.parse(bwastrFile);
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
181 HashMap<String,Long> map = bwastrIdParser.getMap();
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
182
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
183 // Now link rivers with official numbers.
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
184 for(ImportRiver river: infoGewParser.getRivers()) {
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
185 for(Map.Entry<String, Long> entry: map.entrySet()) {
8856
5e38e2924c07 Fix code style.
Tom Gottfried <tom@intevation.de>
parents: 8843
diff changeset
186 if (StringUtil.containsIgnoreCase(
5e38e2924c07 Fix code style.
Tom Gottfried <tom@intevation.de>
parents: 8843
diff changeset
187 river.getName(), entry.getKey())) {
5027
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
188 river.setOfficialNumber(entry.getValue());
8856
5e38e2924c07 Fix code style.
Tom Gottfried <tom@intevation.de>
parents: 8843
diff changeset
189 log.debug(river.getName()
5e38e2924c07 Fix code style.
Tom Gottfried <tom@intevation.de>
parents: 8843
diff changeset
190 + " is mapped to bwastr " + entry.getValue());
5027
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
191 }
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
192 }
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
193 }
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
194 } catch (IOException ioe) {
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
195 log.warn("BWASTR-file could not be loaded.");
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
196 }
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
197 }
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
198 else {
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
199 log.debug("skip reading BWASTR_ID.csv");
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
200 }
25c180fa6162 Use BWASTRParser to set official numbers on rivers.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5016
diff changeset
201
1223
268f8da412e3 Importer: Added a central configuration to allow skipping of parsing/storing individual sub systems.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 1211
diff changeset
202 if (!Config.INSTANCE.dryRun()) {
199
ed38839a6b08 Ported over some WST parsing stuff from desktop flys
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 190
diff changeset
203 new Importer(infoGewParser.getRivers()).writeToDatabase();
ed38839a6b08 Ported over some WST parsing stuff from desktop flys
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents: 190
diff changeset
204 }
4708
45d05402b2a6 Importer: Emit log message when a dry run was run.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 4707
diff changeset
205 else {
45d05402b2a6 Importer: Emit log message when a dry run was run.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 4707
diff changeset
206 log.info("Dry run, not writing to database.");
45d05402b2a6 Importer: Emit log message when a dry run was run.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 4707
diff changeset
207 }
177
31895d24387e Importer: Added info gew parser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
208 }
31895d24387e Importer: Added info gew parser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
209 }
31895d24387e Importer: Added info gew parser.
Sascha L. Teichmann <sascha.teichmann@intevation.de>
parents:
diff changeset
210 // vim:set ts=4 sw=4 si et sta sts=4 fenc=utf8 :
http://dive4elements.wald.intevation.org