Mercurial > dive4elements > river

annotate backend/contrib/run_hydr_morph.sh @ 7481:bdb12632c5f5

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
WST-Parser and doc: reject files with wrong number of columns.
author Tom Gottfried <tom@intevation.de>
date Thu, 31 Oct 2013 15:25:31 +0100
parents 5e621410db05
children 4534659e200e
rev   line source
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
1 #!/bin/bash
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
2
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
3 ######################### CONFIG OPTIONS ############################
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
4 BACKEND_USER="flys28"
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
5 BACKEND_PASS="flys28"
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
6 BACKEND_HOST="czech-republic.atlas.intevation.de"
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
7 BACKEND_PORT="1521"
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
8 BACKEND_NAME="XE"
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
9 LOG4J_CONFIG="conf/log4j.properties"
5066
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 4766
diff changeset
10 JAR="hydr_morph/importer.jar"
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
11 #####################################################################
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
12
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
13
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
14 ########################## Oracle Settings ##########################
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
15 BACKEND_DB_PREFIX="jdbc:oracle:thin:@"
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
16 BACKEND_DB_DRIVER="oracle.jdbc.OracleDriver"
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
17 BACKEND_DB_DIALECT="org.hibernate.dialect.OracleDialect"
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
18 BACKEND_URL=$BACKEND_DB_PREFIX//$BACKEND_HOST:$BACKEND_PORT/$BACKEND_NAME
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
19 #####################################################################
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
20
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
21
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
22 ######################## Custom Importer Settings ###################
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
23 IMPORTER_DRY_RUN=false
6449
25912df0facf Import main values of type '-' by default
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6417
diff changeset
24 IMPORTER_MAINVALUE_TYPES=QWTD-
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
25 IMPORTER_ANNOTATION_TYPES="conf/annotation-types.xml"
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
26
5043
63ad79657453 importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5042
diff changeset
27 IMPORTER_SKIP_ANNOTATIONS=false
63ad79657453 importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5042
diff changeset
28 IMPORTER_SKIP_BWASTR=false
63ad79657453 importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5042
diff changeset
29 IMPORTER_SKIP_DA50S=false
63ad79657453 importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5042
diff changeset
30 IMPORTER_SKIP_DA66S=false
63ad79657453 importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5042
diff changeset
31 IMPORTER_SKIP_EXTRA_WST=false
63ad79657453 importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5042
diff changeset
32 IMPORTER_SKIP_FIXATIONS=false
63ad79657453 importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5042
diff changeset
33 IMPORTER_SKIP_FLOOD_WATER=false
63ad79657453 importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5042
diff changeset
34 IMPORTER_SKIP_FLOOD_PROTECTION=false
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
35 IMPORTER_SKIP_GAUGES=false
7326
5e621410db05 Importer: skip historical discharge tables per default as they come via ETL.
Tom Gottfried <tom@intevation.de>
parents: 6816
diff changeset
36 IMPORTER_SKIP_HISTORICAL_DISCHARGE_GAUGES=true
5043
63ad79657453 importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5042
diff changeset
37 IMPORTER_SKIP_HYKS=false
63ad79657453 importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5042
diff changeset
38 IMPORTER_SKIP_OFFICIAL_LINES=false
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
39 IMPORTER_SKIP_PRFS=false
4766
655e86f025c1 run_hydr_morph.sh: Add SKIP_DA50 and SKIP_W80 flags.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 4711
diff changeset
40 IMPORTER_SKIP_W80S=false
5807
b9ae676a9afe Config: Add new flag to skip w80-csv imports (which are not done yet anyways).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5086
diff changeset
41 IMPORTER_SKIP_W80_CSVS=false
5043
63ad79657453 importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5042
diff changeset
42 IMPORTER_SKIP_WST=false
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
43
6417
14b5674e57a4 Importer: add skip flag for measurement stations to scripts
Tom Gottfried <tom@intevation.de>
parents: 5838
diff changeset
44 IMPORTER_SKIP_MEASUREMENT_STATIONS=false
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
45 IMPORTER_SKIP_BED_HEIGHT_SINGLE=false
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
46 IMPORTER_SKIP_BED_HEIGHT_EPOCH=false
5043
63ad79657453 importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5042
diff changeset
47 IMPORTER_SKIP_FLOW_VELOCITY=false
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
48 IMPORTER_SKIP_MORPHOLOGICAL_WIDTH=false
5043
63ad79657453 importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5042
diff changeset
49 IMPORTER_SKIP_SEDIMENT_DENSITY=false
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
50 IMPORTER_SKIP_SEDIMENT_YIELD=false
5043
63ad79657453 importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5042
diff changeset
51 IMPORTER_SKIP_SQ_RELATION=false
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
52 IMPORTER_SKIP_WATERLEVELS=false
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
53 IMPORTER_SKIP_WATERLEVEL_DIFFERENCES=false
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
54 #####################################################################
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
55
6816
a3c4f094707e Importer: set memory size according to doc
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6811
diff changeset
56 MIN_MEMORY="8192m"
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
57
5066
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 4766
diff changeset
58 ######################### Run Importer ##############################
6811
6bf93fa1b744 Geo-importer and doc: use commandline-argument instead of configuration for river selection.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6651
diff changeset
59 INFO_GEW=$1
6bf93fa1b744 Geo-importer and doc: use commandline-argument instead of configuration for river selection.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6651
diff changeset
60
5066
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 4766
diff changeset
61 OPTIONAL_LIBS="${DIR}"/../opt
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 4766
diff changeset
62 if [ -d "$OPTIONAL_LIBS" ]; then
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 4766
diff changeset
63 export PATH="$OPTIONAL_LIBS/bin:$PATH"
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 4766
diff changeset
64 export LD_LIBRARY_PATH="$OPTIONAL_LIBS/lib:$LD_LIBRARY_PATH"
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 4766
diff changeset
65 export LD_LIBRARY_PATH="$OPTIONAL_LIBS/lib64:$LD_LIBRARY_PATH"
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 4766
diff changeset
66 fi
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
67
6651
8782ba5cb222 Importer scripts: add default encoding to correctly read gew-files
Tom Gottfried <tom@intevation.de>
parents: 6449
diff changeset
68 # Default encoding. Change here if necessary
8782ba5cb222 Importer scripts: add default encoding to correctly read gew-files
Tom Gottfried <tom@intevation.de>
parents: 6449
diff changeset
69 export LC_ALL=de_DE@euro
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
70
5066
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 4766
diff changeset
71 exec java -jar \
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
72 -Xmx$MIN_MEMORY \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
73 -server \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
74 -Dlog4j.configuration=file://`readlink -f $LOG4J_CONFIG` \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
75 -Dflys.backend.importer.infogew.file=$INFO_GEW \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
76 -Dflys.backend.main.value.types=$IMPORTER_MAINVALUE_TYPES \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
77 -Dflys.backend.importer.annotation.types=$IMPORTER_ANNOTATION_TYPES \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
78 -Dflys.backend.importer.dry.run=$IMPORTER_DRY_RUN \
5042
70650981a54d importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5018
diff changeset
79 -Dflys.backend.importer.skip.annotations=$IMPORTER_SKIP_ANNOTATIONS \
70650981a54d importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5018
diff changeset
80 -Dflys.backend.importer.skip.bed.height.single=$IMPORTER_SKIP_BED_HEIGHT_SINGLE \
70650981a54d importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5018
diff changeset
81 -Dflys.backend.importer.skip.bed.height.epoch=$IMPORTER_SKIP_BED_HEIGHT_EPOCH \
5018
e939098ee272 run_hydr_morph.sh,Config: Add flags for importer to skip bwastr_id.csv parsing.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 4766
diff changeset
82 -Dflys.backend.importer.skip.bwastr=$IMPORTER_SKIP_BWASTR \
4766
655e86f025c1 run_hydr_morph.sh: Add SKIP_DA50 and SKIP_W80 flags.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 4711
diff changeset
83 -Dflys.backend.importer.skip.da50s=$IMPORTER_SKIP_DA50S \
4711
35dd03e04e38 Added new parameter SKIP_DA66S to importer.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 4135
diff changeset
84 -Dflys.backend.importer.skip.da66s=$IMPORTER_SKIP_DA66S \
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
85 -Dflys.backend.importer.skip.extra.wsts=$IMPORTER_SKIP_EXTRA_WST \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
86 -Dflys.backend.importer.skip.fixations=$IMPORTER_SKIP_FIXATIONS \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
87 -Dflys.backend.importer.skip.flood.water=$IMPORTER_SKIP_FLOOD_WATER \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
88 -Dflys.backend.importer.skip.flood.protection=$IMPORTER_SKIP_FLOOD_PROTECTION \
6417
14b5674e57a4 Importer: add skip flag for measurement stations to scripts
Tom Gottfried <tom@intevation.de>
parents: 5838
diff changeset
89 -Dflys.backend.importer.skip.measurement.stations=$IMPORTER_SKIP_MEASUREMENT_STATIONS \
5042
70650981a54d importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5018
diff changeset
90 -Dflys.backend.importer.skip.flow.velocity=$IMPORTER_SKIP_FLOW_VELOCITY \
70650981a54d importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5018
diff changeset
91 -Dflys.backend.importer.skip.gauges=$IMPORTER_SKIP_GAUGES \
70650981a54d importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5018
diff changeset
92 -Dflys.backend.importer.skip.historical.discharge.tables=$IMPORTER_SKIP_HISTORICAL_DISCHARGE_GAUGES \
70650981a54d importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5018
diff changeset
93 -Dflys.backend.importer.skip.hyks=$IMPORTER_SKIP_HYKS \
70650981a54d importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5018
diff changeset
94 -Dflys.backend.importer.skip.morphological.width=$IMPORTER_SKIP_MORPHOLOGICAL_WIDTH \
70650981a54d importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5018
diff changeset
95 -Dflys.backend.importer.skip.official.lines=$IMPORTER_SKIP_OFFICIAL_LINES \
70650981a54d importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5018
diff changeset
96 -Dflys.backend.importer.skip.prfs=$IMPORTER_SKIP_PRFS \
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
97 -Dflys.backend.importer.skip.sediment.density=$IMPORTER_SKIP_SEDIMENT_DENSITY \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
98 -Dflys.backend.importer.skip.sediment.yield=$IMPORTER_SKIP_SEDIMENT_YIELD \
5042
70650981a54d importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5018
diff changeset
99 -Dflys.backend.importer.skip.sq.relation=$IMPORTER_SKIP_SQ_RELATION \
70650981a54d importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5018
diff changeset
100 -Dflys.backend.importer.skip.w80s=$IMPORTER_SKIP_W80S \
5807
b9ae676a9afe Config: Add new flag to skip w80-csv imports (which are not done yet anyways).
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5086
diff changeset
101 -Dflys.backend.importer.skip.w80.csvs=$IMPORTER_SKIP_W80_CSVS \
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
102 -Dflys.backend.importer.skip.waterlevels=$IMPORTER_SKIP_WATERLEVELS \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
103 -Dflys.backend.importer.skip.waterlevel.differences=$IMPORTER_SKIP_WATERLEVEL_DIFFERENCES \
5042
70650981a54d importer: Sort options alphabetically.
Felix Wolfsteller <felix.wolfsteller@intevation.de>
parents: 5018
diff changeset
104 -Dflys.backend.importer.skip.wst=$IMPORTER_SKIP_WST \
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
105 -Dflys.backend.user=$BACKEND_USER \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
106 -Dflys.backend.password=$BACKEND_PASS \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
107 -Dflys.backend.url=$BACKEND_URL \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
108 -Dflys.backend.driver=$BACKEND_DB_DRIVER \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
109 -Dflys.backend.dialect=$BACKEND_DB_DIALECT \
5066
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 4766
diff changeset
110 $JAR
http://dive4elements.wald.intevation.org