Mercurial > dive4elements > river

annotate backend/contrib/import_river.sh @ 8056:d86cc6a17b7a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Importer: Import sediment load at measurement stations.
author Tom Gottfried <tom@intevation.de>
date Fri, 18 Jul 2014 15:37:26 +0200
parents fd3a24336e6a
children 44ba71b0b75a
rev   line source
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
1 #!/bin/bash
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
2 # Import script for rivers
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
3 #
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
4 # Authors:
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
5 # Andre Heinecke <aheinecke@intevation.de>
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
6 #
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
7 # Copyright:
5168
650f94ec3122 Fix copyright on import_river.sh
Andre Heinecke <aheinecke@intevation.de>
parents: 5167
diff changeset
8 # Copyright (C) 2013 Intevation GmbH
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
9 #
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
10 # This program is free software; you can redistribute it and/or
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
11 # modify it under the terms of the GNU General Public License
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
12 # as published by the Free Software Foundation; either version 2
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
13 # of the License, or (at your option) any later version.
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
14 #
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
15 # This program is distributed in the hope that it will be useful,
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
16 # but WITHOUT ANY WARRANTY; without even the implied warranty of
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
17 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
18 # GNU General Public License for more details.
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
19 #
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
20 # You should have received a copy of the GNU General Public License
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
21 # along with this program; if not, write to the Free Software
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
22 # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
23
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
24 set -e
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
25
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
26 # Default settings
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
27 DEFAULT_HOST=localhost
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
28 DEFAULT_PORT=1521
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
29 DEFAULT_USER=flys_dami
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
30 DEFAULT_PASS=flys_dami
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
31 DEFAULT_LOG=$PWD/logs
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
32 DEFAULT_BACKEND_NAME="XE"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
33 JAR="hydr_morph/importer.jar"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
34 IMPORTER_DRY_RUN=false
6449
25912df0facf Import main values of type '-' by default
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6417
diff changeset
35 IMPORTER_MAINVALUE_TYPES=QWTD-
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
36 IMPORTER_ANNOTATION_TYPES="conf/annotation-types.xml"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
37
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
38
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
39 MIN_MEMORY="8024m"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
40
6651
8782ba5cb222 Importer scripts: add default encoding to correctly read gew-files
Tom Gottfried <tom@intevation.de>
parents: 6449
diff changeset
41 # Default encoding. Change here if necessary
8782ba5cb222 Importer scripts: add default encoding to correctly read gew-files
Tom Gottfried <tom@intevation.de>
parents: 6449
diff changeset
42 export LC_ALL=de_DE@euro
8782ba5cb222 Importer scripts: add default encoding to correctly read gew-files
Tom Gottfried <tom@intevation.de>
parents: 6449
diff changeset
43
5169
db1a000a21a9 Make OPTIONAL_LIBS overwritable and fix geo importer call
Andre Heinecke <aheinecke@intevation.de>
parents: 5168
diff changeset
44 if [ -z "$OPTIONAL_LIBS" ]; then
5212
14186e924534 Fix opt path if OPTIONAL_LIBS is not set
Andre Heinecke <aheinecke@intevation.de>
parents: 5192
diff changeset
45 OPTIONAL_LIBS="$(dirname $0)/opt"
5169
db1a000a21a9 Make OPTIONAL_LIBS overwritable and fix geo importer call
Andre Heinecke <aheinecke@intevation.de>
parents: 5168
diff changeset
46 fi
db1a000a21a9 Make OPTIONAL_LIBS overwritable and fix geo importer call
Andre Heinecke <aheinecke@intevation.de>
parents: 5168
diff changeset
47
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
48 if [ -d "$OPTIONAL_LIBS" ]; then
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
49 export PATH="$OPTIONAL_LIBS/bin:$PATH"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
50 export LD_LIBRARY_PATH="$OPTIONAL_LIBS/lib:$LD_LIBRARY_PATH"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
51 export LD_LIBRARY_PATH="$OPTIONAL_LIBS/lib64:$LD_LIBRARY_PATH"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
52 export PYTHONPATH="$OPTIONAL_LIBS/lib/python2.6/site-packages:$PYTHONPATH"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
53 export PYTHONPATH="$OPTIONAL_LIBS/lib64/python2.6/site-packages:$PYTHONPATH"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
54 export GDAL_DATA="$OPTIONAL_LIBS/share/gdal"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
55 fi
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
56
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
57 usage(){
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
58 cat << EOF
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
59
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
60 usage: $0 [options] gew_file
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
61
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
62 Import a river described by the gew_file
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
63
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
64 OPTIONS:
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
65 -?, --help Show this message
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
66 -u, --username=<username> Database username. Default: $DEFAULT_USER
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
67 -w, --password=<password> Database password. Default: $DEFAULT_PASS
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
68 -h, --host=<host> Connect to database on host <host>.
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
69 Default: $DEFAULT_HOST
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
70 -p, --port=<number> Use port number <number>. Default: $DEFAULT_PORT
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
71 -d, --db-name=<database_name> Name of the database / backend. Default: $DEFAULT_BACKEND_NAME
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
72 -l, --log-dir=<directory> Directory in which to create the log files.
7728
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
73 Default: $PWD/logs
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
74 --postgres Database is PostgreSQL
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
75 --skip-hydro Skip import of hydrological data
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
76 --skip-morpho Skip import of morphological data
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
77 --skip-geo Skip import of geographic data
6825
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
78 --skip-prf Skip import of cross section data
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
79 EOF
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
80 exit 0
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
81 }
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
82
7728
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
83 OPTS=`getopt -o ?u:w:h:p:d:l: \
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
84 -l help,username:,password:,host:,port:,db-name:,log-dir:,skip-hydro,skip-morpho,skip-geo,skip-prf,postgres \
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
85 -n $0 -- "$@"`
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
86 if [ $? != 0 ] ; then usage; fi
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
87 eval set -- "$OPTS"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
88 while true ; do
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
89 case "$1" in
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
90 "-?"|"--help")
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
91 usage;;
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
92 "--")
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
93 shift
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
94 break;;
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
95 "-u"|"--username")
5192
1b6d4edeeb4c Fix variable assignments after rename
Andre Heinecke <aheinecke@intevation.de>
parents: 5191
diff changeset
96 DBUSER=$2
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
97 shift 2;;
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
98 "-w"|"--password")
5192
1b6d4edeeb4c Fix variable assignments after rename
Andre Heinecke <aheinecke@intevation.de>
parents: 5191
diff changeset
99 DBPASS=$2
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
100 shift 2;;
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
101 "-h"|"--host")
5192
1b6d4edeeb4c Fix variable assignments after rename
Andre Heinecke <aheinecke@intevation.de>
parents: 5191
diff changeset
102 DBHOST=$2
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
103 shift 2;;
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
104 "-p"|"--port")
5192
1b6d4edeeb4c Fix variable assignments after rename
Andre Heinecke <aheinecke@intevation.de>
parents: 5191
diff changeset
105 DBPORT=$2
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
106 shift 2;;
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
107 "-l"|"--log-dir")
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
108 LOG=$2
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
109 shift 2;;
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
110 "-d"|"--db-name")
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
111 BACKEND_NAME=$2
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
112 shift 2;;
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
113 "--skip-hydro")
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
114 SKIP_HYDRO="TRUE"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
115 shift;;
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
116 "--skip-morpho")
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
117 SKIP_MORPHO="TRUE"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
118 shift;;
6825
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
119 "--skip-prf")
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
120 SKIP_PRF="TRUE"
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
121 shift;;
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
122 "--skip-geo")
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
123 SKIP_GEO="TRUE"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
124 shift;;
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
125 "--postgres")
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
126 POSTGRES="TRUE"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
127 shift;;
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
128 *)
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
129 echo "Unknown Option $1"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
130 usage;;
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
131 esac
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
132 done
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
133
5191
4aa748db8be3 Use less ambigious names for variables
Andre Heinecke <aheinecke@intevation.de>
parents: 5176
diff changeset
134 if [ -z $DBUSER ]; then
5192
1b6d4edeeb4c Fix variable assignments after rename
Andre Heinecke <aheinecke@intevation.de>
parents: 5191
diff changeset
135 DBUSER=$DEFAULT_USER
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
136 fi
5191
4aa748db8be3 Use less ambigious names for variables
Andre Heinecke <aheinecke@intevation.de>
parents: 5176
diff changeset
137 if [ -z $DBPASS ]; then
5192
1b6d4edeeb4c Fix variable assignments after rename
Andre Heinecke <aheinecke@intevation.de>
parents: 5191
diff changeset
138 DBPASS=$DEFAULT_PASS
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
139 fi
5191
4aa748db8be3 Use less ambigious names for variables
Andre Heinecke <aheinecke@intevation.de>
parents: 5176
diff changeset
140 if [ -z $DBPORT ]; then
5192
1b6d4edeeb4c Fix variable assignments after rename
Andre Heinecke <aheinecke@intevation.de>
parents: 5191
diff changeset
141 DBPORT=$DEFAULT_PORT
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
142 fi
5191
4aa748db8be3 Use less ambigious names for variables
Andre Heinecke <aheinecke@intevation.de>
parents: 5176
diff changeset
143 if [ -z $DBHOST ]; then
5192
1b6d4edeeb4c Fix variable assignments after rename
Andre Heinecke <aheinecke@intevation.de>
parents: 5191
diff changeset
144 DBHOST=$DEFAULT_HOST
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
145 fi
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
146 if [ -z $BACKEND_NAME ]; then
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
147 BACKEND_NAME=$DEFAULT_BACKEND_NAME
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
148 fi
7728
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
149 if [ -z $LOG ]; then
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
150 LOG=$DEFAULT_LOG
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
151 fi
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
152
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
153 if [ $# != 1 ]; then
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
154 usage
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
155 fi
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
156
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
157 if [ ! -r $1 ]; then
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
158 echo "Could not open $1 please ensure it exists and is readable"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
159 fi
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
160
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
161 GEW_FILE="$1"
5459
7c646d4d5103 be more flexible with river names (e.g. containing white space) in import script
Tom Gottfried <tom@intevation.de>
parents: 5355
diff changeset
162 RIVER_NAME=$(grep "Gew.sser" "$1" | sed 's/Gew.sser: //')
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
163 DATE=$(date +%Y.%m.%d_%H%M)
5459
7c646d4d5103 be more flexible with river names (e.g. containing white space) in import script
Tom Gottfried <tom@intevation.de>
parents: 5355
diff changeset
164 LOG_DIR=${LOG}/`basename $GEW_FILE .gew`-$DATE
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
165 mkdir -p ${LOG_DIR}
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
166
7728
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
167 cat > "$LOG_DIR/log4j.properties" << "EOF"
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
168 log4j.rootLogger=DEBUG, IMPORTER
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
169 log4j.appender.IMPORTER.layout=org.apache.log4j.PatternLayout
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
170 log4j.appender.IMPORTER.layout.ConversionPattern=%d [%t] %-5p %c - %m%n
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
171 log4j.appender.IMPORTER=org.apache.log4j.RollingFileAppender
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
172 log4j.appender.IMPORTER.File=./import.log
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
173 log4j.appender.IMPORTER.MaxFileSize=100000KB
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
174 log4j.appender.IMPORTER.MaxBackupIndex=10
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
175 EOF
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
176
5355
480d2f1f61cc Fix import_rivers script for postgres
Andre Heinecke <aheinecke@intevation.de>
parents: 5212
diff changeset
177 if [ "$POSTGRES" = "TRUE" ]; then
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
178 JAR=$(echo "$JAR" | sed 's/importer/importer_psql/')
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
179 if [ ! -r "$JAR" ]; then
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
180 echo "Could not find Postgres importer $JAR"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
181 exit 1
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
182 fi
5191
4aa748db8be3 Use less ambigious names for variables
Andre Heinecke <aheinecke@intevation.de>
parents: 5176
diff changeset
183 OGR_CONNECTION="PG:dbname=$BACKEND_NAME host=$DBHOST port=$DBPORT \
4aa748db8be3 Use less ambigious names for variables
Andre Heinecke <aheinecke@intevation.de>
parents: 5176
diff changeset
184 user=$DBUSER password=$DBPASS"
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
185 BACKEND_DB_PREFIX="jdbc:postgresql:"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
186 BACKEND_DB_DRIVER="org.postgresql.Driver"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
187 BACKEND_DB_DIALECT="org.hibernate.dialect.PostgreSQLDialect"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
188 else
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
189 BACKEND_DB_PREFIX="jdbc:oracle:thin:@"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
190 BACKEND_DB_DRIVER="oracle.jdbc.OracleDriver"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
191 BACKEND_DB_DIALECT="org.hibernate.dialect.OracleDialect"
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
192 fi
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
193
5191
4aa748db8be3 Use less ambigious names for variables
Andre Heinecke <aheinecke@intevation.de>
parents: 5176
diff changeset
194 BACKEND_URL=$BACKEND_DB_PREFIX//$DBHOST:$DBPORT/$BACKEND_NAME
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
195
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
196 echo "Importing $RIVER_NAME into $BACKEND_URL."
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
197
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
198 import_hydro(){
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
199 LOG_FILE=${LOG_DIR}/hydro.log
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
200 echo Importing Hydrological data.
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
201 echo Logging into: $LOG_FILE
7728
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
202 sed -i 's!\(log4j.appender.IMPORTER.File=\).*!\1'"$LOG_FILE"'!' \
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
203 $LOG_DIR/log4j.properties
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
204 java -jar \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
205 -Xmx$MIN_MEMORY \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
206 -server \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
207 -Dlog4j.configuration=file://$LOG_DIR/log4j.properties \
5191
4aa748db8be3 Use less ambigious names for variables
Andre Heinecke <aheinecke@intevation.de>
parents: 5176
diff changeset
208 -Dflys.backend.user=$DBUSER \
4aa748db8be3 Use less ambigious names for variables
Andre Heinecke <aheinecke@intevation.de>
parents: 5176
diff changeset
209 -Dflys.backend.password=$DBPASS \
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
210 -Dflys.backend.url=$BACKEND_URL \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
211 -Dflys.backend.driver=$BACKEND_DB_DRIVER \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
212 -Dflys.backend.dialect=$BACKEND_DB_DIALECT \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
213 -Dflys.backend.importer.infogew.file="$GEW_FILE" \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
214 -Dflys.backend.main.value.types=$IMPORTER_MAINVALUE_TYPES \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
215 -Dflys.backend.importer.annotation.types=$IMPORTER_ANNOTATION_TYPES \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
216 -Dflys.backend.importer.dry.run=$IMPORTER_DRY_RUN \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
217 -Dflys.backend.importer.skip.annotations=false \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
218 -Dflys.backend.importer.skip.bwastr=false \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
219 -Dflys.backend.importer.skip.extra.wsts=false \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
220 -Dflys.backend.importer.skip.fixations=false \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
221 -Dflys.backend.importer.skip.flood.water=false \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
222 -Dflys.backend.importer.skip.flood.protection=false \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
223 -Dflys.backend.importer.skip.gauges=false \
7326
5e621410db05 Importer: skip historical discharge tables per default as they come via ETL.
Tom Gottfried <tom@intevation.de>
parents: 6825
diff changeset
224 -Dflys.backend.importer.skip.historical.discharge.tables=true \
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
225 -Dflys.backend.importer.skip.hyks=false \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
226 -Dflys.backend.importer.skip.official.lines=false \
6825
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
227 -Dflys.backend.importer.skip.prfs=true \
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
228 -Dflys.backend.importer.skip.w80s=true \
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
229 -Dflys.backend.importer.skip.w80.csvs=true \
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
230 -Dflys.backend.importer.skip.da50s=true \
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
231 -Dflys.backend.importer.skip.da66s=true \
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
232 -Dflys.backend.importer.skip.wst=false \
6417
14b5674e57a4 Importer: add skip flag for measurement stations to scripts
Tom Gottfried <tom@intevation.de>
parents: 5838
diff changeset
233 -Dflys.backend.importer.skip.measurement.stations=true \
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
234 -Dflys.backend.importer.skip.waterlevel.differences=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
235 -Dflys.backend.importer.skip.waterlevels=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
236 -Dflys.backend.importer.skip.sq.relation=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
237 -Dflys.backend.importer.skip.sediment.density=true \
8056
d86cc6a17b7a Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents: 8032
diff changeset
238 -Dflys.backend.importer.skip.sediment.load=true \
8032
fd3a24336e6a SCHEMA CHANGE and Importer (only longitudinal section data so far): only grain fractions given now in schema are valid. We are handling sediment loads, not yields.
"Tom Gottfried <tom@intevation.de>"
parents: 7925
diff changeset
239 -Dflys.backend.importer.skip.sediment.load.ls=true \
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
240 -Dflys.backend.importer.skip.morphological.width=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
241 -Dflys.backend.importer.skip.flow.velocity=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
242 -Dflys.backend.importer.skip.bed.height.single=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
243 $JAR
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
244 }
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
245
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
246 import_morpho(){
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
247 LOG_FILE=${LOG_DIR}/morpho.log
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
248 echo Importing Morphological data.
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
249 echo Logging into: $LOG_FILE
7728
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
250 sed -i 's!\(log4j.appender.IMPORTER.File=\).*!\1'"$LOG_FILE"'!' \
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
251 $LOG_DIR/log4j.properties
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
252 java -jar \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
253 -Xmx$MIN_MEMORY \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
254 -server \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
255 -Dlog4j.configuration=file://$LOG_DIR/log4j.properties \
5191
4aa748db8be3 Use less ambigious names for variables
Andre Heinecke <aheinecke@intevation.de>
parents: 5176
diff changeset
256 -Dflys.backend.user=$DBUSER \
4aa748db8be3 Use less ambigious names for variables
Andre Heinecke <aheinecke@intevation.de>
parents: 5176
diff changeset
257 -Dflys.backend.password=$DBPASS \
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
258 -Dflys.backend.url=$BACKEND_URL \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
259 -Dflys.backend.driver=$BACKEND_DB_DRIVER \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
260 -Dflys.backend.dialect=$BACKEND_DB_DIALECT \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
261 -Dflys.backend.importer.infogew.file="$GEW_FILE" \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
262 -Dflys.backend.main.value.types=$IMPORTER_MAINVALUE_TYPES \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
263 -Dflys.backend.importer.annotation.types=$IMPORTER_ANNOTATION_TYPES \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
264 -Dflys.backend.importer.dry.run=$IMPORTER_DRY_RUN \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
265 -Dflys.backend.importer.skip.annotations=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
266 -Dflys.backend.importer.skip.bwastr=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
267 -Dflys.backend.importer.skip.extra.wsts=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
268 -Dflys.backend.importer.skip.fixations=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
269 -Dflys.backend.importer.skip.flood.water=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
270 -Dflys.backend.importer.skip.flood.protection=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
271 -Dflys.backend.importer.skip.gauges=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
272 -Dflys.backend.importer.skip.historical.discharge.tables=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
273 -Dflys.backend.importer.skip.hyks=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
274 -Dflys.backend.importer.skip.official.lines=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
275 -Dflys.backend.importer.skip.prfs=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
276 -Dflys.backend.importer.skip.w80s=true \
6706
c3e38ea87964 Importer scripts: add skip flag for W80-CSV to import_river.sh
Tom Gottfried <tom@intevation.de>
parents: 6658
diff changeset
277 -Dflys.backend.importer.skip.w80.csvs=true \
6825
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
278 -Dflys.backend.importer.skip.da50s=true \
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
279 -Dflys.backend.importer.skip.da66s=true \
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
280 -Dflys.backend.importer.skip.wst=true \
6417
14b5674e57a4 Importer: add skip flag for measurement stations to scripts
Tom Gottfried <tom@intevation.de>
parents: 5838
diff changeset
281 -Dflys.backend.importer.skip.measurement.stations=false \
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
282 -Dflys.backend.importer.skip.waterlevel.differences=false \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
283 -Dflys.backend.importer.skip.waterlevels=false \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
284 -Dflys.backend.importer.skip.sq.relation=false \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
285 -Dflys.backend.importer.skip.sediment.density=false \
8056
d86cc6a17b7a Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents: 8032
diff changeset
286 -Dflys.backend.importer.skip.sediment.load=false \
8032
fd3a24336e6a SCHEMA CHANGE and Importer (only longitudinal section data so far): only grain fractions given now in schema are valid. We are handling sediment loads, not yields.
"Tom Gottfried <tom@intevation.de>"
parents: 7925
diff changeset
287 -Dflys.backend.importer.skip.sediment.load.ls=false \
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
288 -Dflys.backend.importer.skip.morphological.width=false \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
289 -Dflys.backend.importer.skip.flow.velocity=false \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
290 -Dflys.backend.importer.skip.bed.height.single=false \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
291 $JAR
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
292 }
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
293
6825
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
294 import_prf(){
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
295 LOG_FILE=${LOG_DIR}/prf.log
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
296 echo Importing cross section data.
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
297 echo Logging into: $LOG_FILE
7728
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
298 sed -i 's!\(log4j.appender.IMPORTER.File=\).*!\1'"$LOG_FILE"'!' \
c842ebb0db86 Importer: fix logging in import_river.sh.
Tom Gottfried <tom@intevation.de>
parents: 7326
diff changeset
299 $LOG_DIR/log4j.properties
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
300 java -jar \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
301 -Xmx$MIN_MEMORY \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
302 -server \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
303 -Dlog4j.configuration=file://$LOG_DIR/log4j.properties \
5191
4aa748db8be3 Use less ambigious names for variables
Andre Heinecke <aheinecke@intevation.de>
parents: 5176
diff changeset
304 -Dflys.backend.user=$DBUSER \
4aa748db8be3 Use less ambigious names for variables
Andre Heinecke <aheinecke@intevation.de>
parents: 5176
diff changeset
305 -Dflys.backend.password=$DBPASS \
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
306 -Dflys.backend.url=$BACKEND_URL \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
307 -Dflys.backend.driver=$BACKEND_DB_DRIVER \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
308 -Dflys.backend.dialect=$BACKEND_DB_DIALECT \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
309 -Dflys.backend.importer.infogew.file="$GEW_FILE" \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
310 -Dflys.backend.main.value.types=$IMPORTER_MAINVALUE_TYPES \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
311 -Dflys.backend.importer.annotation.types=$IMPORTER_ANNOTATION_TYPES \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
312 -Dflys.backend.importer.dry.run=$IMPORTER_DRY_RUN \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
313 -Dflys.backend.importer.skip.annotations=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
314 -Dflys.backend.importer.skip.bwastr=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
315 -Dflys.backend.importer.skip.extra.wsts=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
316 -Dflys.backend.importer.skip.fixations=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
317 -Dflys.backend.importer.skip.flood.water=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
318 -Dflys.backend.importer.skip.flood.protection=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
319 -Dflys.backend.importer.skip.gauges=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
320 -Dflys.backend.importer.skip.historical.discharge.tables=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
321 -Dflys.backend.importer.skip.hyks=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
322 -Dflys.backend.importer.skip.official.lines=true \
6825
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
323 -Dflys.backend.importer.skip.prfs=false \
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
324 -Dflys.backend.importer.skip.w80s=false \
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
325 -Dflys.backend.importer.skip.w80.csvs=false \
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
326 -Dflys.backend.importer.skip.da50s=false \
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
327 -Dflys.backend.importer.skip.da66s=false \
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
328 -Dflys.backend.importer.skip.wst=true \
6417
14b5674e57a4 Importer: add skip flag for measurement stations to scripts
Tom Gottfried <tom@intevation.de>
parents: 5838
diff changeset
329 -Dflys.backend.importer.skip.measurement.stations=true \
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
330 -Dflys.backend.importer.skip.waterlevel.differences=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
331 -Dflys.backend.importer.skip.waterlevels=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
332 -Dflys.backend.importer.skip.sq.relation=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
333 -Dflys.backend.importer.skip.sediment.density=true \
8056
d86cc6a17b7a Importer: Import sediment load at measurement stations.
Tom Gottfried <tom@intevation.de>
parents: 8032
diff changeset
334 -Dflys.backend.importer.skip.sediment.load=true \
8032
fd3a24336e6a SCHEMA CHANGE and Importer (only longitudinal section data so far): only grain fractions given now in schema are valid. We are handling sediment loads, not yields.
"Tom Gottfried <tom@intevation.de>"
parents: 7925
diff changeset
335 -Dflys.backend.importer.skip.sediment.load.ls=true \
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
336 -Dflys.backend.importer.skip.morphological.width=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
337 -Dflys.backend.importer.skip.flow.velocity=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
338 -Dflys.backend.importer.skip.bed.height.single=true \
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
339 $JAR
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
340 }
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
341
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
342 import_geo(){
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
343 LOG_FILE=${LOG_DIR}/geo.log
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
344 echo Importing Geographic data.
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
345 echo Logging into: $LOG_FILE
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
346
5169
db1a000a21a9 Make OPTIONAL_LIBS overwritable and fix geo importer call
Andre Heinecke <aheinecke@intevation.de>
parents: 5168
diff changeset
347 RIVER_PATH=$(grep "WSTDatei:" "$GEW_FILE" | awk '{print $2}')
db1a000a21a9 Make OPTIONAL_LIBS overwritable and fix geo importer call
Andre Heinecke <aheinecke@intevation.de>
parents: 5168
diff changeset
348 RIVER_PATH=$(dirname "$RIVER_PATH")/../..
db1a000a21a9 Make OPTIONAL_LIBS overwritable and fix geo importer call
Andre Heinecke <aheinecke@intevation.de>
parents: 5168
diff changeset
349 RIVER_PATH=$(readlink -f "$RIVER_PATH")
db1a000a21a9 Make OPTIONAL_LIBS overwritable and fix geo importer call
Andre Heinecke <aheinecke@intevation.de>
parents: 5168
diff changeset
350
db1a000a21a9 Make OPTIONAL_LIBS overwritable and fix geo importer call
Andre Heinecke <aheinecke@intevation.de>
parents: 5168
diff changeset
351 exec python $(dirname $0)/geodaesie/shpimporter.py \
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
352 --directory $RIVER_PATH \
5563
673f68bb19a4 Geo-importer: accept river names with white space
Tom Gottfried <tom@intevation.de>
parents: 5459
diff changeset
353 --river_name "$RIVER_NAME" \
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
354 --ogr_connection "$OGR_CONNECTION" \
6658
89f856e24d27 Importer scripts: shpimporter needs backend name to play with Oracle.
Tom Gottfried <tom@intevation.de>
parents: 6651
diff changeset
355 --host $DBHOST/$BACKEND_NAME \
5191
4aa748db8be3 Use less ambigious names for variables
Andre Heinecke <aheinecke@intevation.de>
parents: 5176
diff changeset
356 --user $DBUSER \
4aa748db8be3 Use less ambigious names for variables
Andre Heinecke <aheinecke@intevation.de>
parents: 5176
diff changeset
357 --password $DBPASS \
5169
db1a000a21a9 Make OPTIONAL_LIBS overwritable and fix geo importer call
Andre Heinecke <aheinecke@intevation.de>
parents: 5168
diff changeset
358 --verbose 1 \
db1a000a21a9 Make OPTIONAL_LIBS overwritable and fix geo importer call
Andre Heinecke <aheinecke@intevation.de>
parents: 5168
diff changeset
359 > "$LOG_FILE" 2>&1
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
360 }
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
361
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
362
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
363 if [ "$SKIP_HYDRO" != "TRUE" ]; then
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
364 import_hydro
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
365 fi
6825
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
366 if [ "$SKIP_PRF" != "TRUE" ]; then
3aff1b00618a import_rivers.sh: skip cross sections instead of WST-file in first run.
Tom Gottfried <tom.gottfried@intevation.de>
parents: 6706
diff changeset
367 import_prf
5167
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
368 fi
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
369 if [ "$SKIP_MORPHO" != "TRUE" ]; then
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
370 import_morpho
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
371 fi
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
372 if [ "$SKIP_GEO" != "TRUE" ]; then
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
373 import_geo
a0abb6787ab1 Add first version of import_river script
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
374 fi
http://dive4elements.wald.intevation.org