Mercurial > dive4elements > river

annotate contrib/make_flys_release/bin/make-importer-package.sh @ 5779:ebec12def170

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Datacage: Add a pool of builders to make it multi threadable. XML DOM is not thread safe. Therefore the old implementation only allowed one thread to use the builder at a time. As the complexity of the configuration has increased over time this has become a bottleneck of the whole application because it took quiet some time to build a result. Furthermore the builder code path is visited very frequent. So many concurrent requests were piled up resulting in long waits for the users. To mitigate this problem a round robin pool of builders is used now. Each of the pooled builders has an independent copy of the XML template and can be run in parallel. The number of builders is determined by the system property 'flys.datacage.pool.size'. It defaults to 4.
author Sascha L. Teichmann <teichmann@intevation.de>
date Sun, 21 Apr 2013 12:48:09 +0200
parents d5e1e4d9559e
children cff1eea9854f
rev   line source
5097
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
1 #!/bin/bash
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
2 set -e
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
3
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
4 # See README for more information
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
5
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
6 # The working directory. Resulting tarball will be placed in the directory above.
5520
d5e1e4d9559e compile manual from within importer packaging script
Tom Gottfried <tom.gottfried@intevation.de>
parents: 5491
diff changeset
7 PKG_DIR=~/tmp/flys-importer
5097
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
8 # Path to the flys checkout
5520
d5e1e4d9559e compile manual from within importer packaging script
Tom Gottfried <tom.gottfried@intevation.de>
parents: 5491
diff changeset
9 FLYS_DIR=~/flys/flys/
5179
4355b4cecef8 Improve importer packaging by just expecting a tarball with
Andre Heinecke <aheinecke@intevation.de>
parents: 5172
diff changeset
10 # Tarball that will be extracted into flys-imprter/opt
4355b4cecef8 Improve importer packaging by just expecting a tarball with
Andre Heinecke <aheinecke@intevation.de>
parents: 5172
diff changeset
11 EXTRAS=$1
5097
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
12
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
13 rm -fr $PKG_DIR
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
14 mkdir -p $PKG_DIR/hydr_morph
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
15 mkdir -p $PKG_DIR/geodaesie
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
16 mkdir -p $PKG_DIR/opt/lib64
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
17 mkdir -p $PKG_DIR/schema
5179
4355b4cecef8 Improve importer packaging by just expecting a tarball with
Andre Heinecke <aheinecke@intevation.de>
parents: 5172
diff changeset
18 mkdir -p $PKG_DIR/conf
4355b4cecef8 Improve importer packaging by just expecting a tarball with
Andre Heinecke <aheinecke@intevation.de>
parents: 5172
diff changeset
19
4355b4cecef8 Improve importer packaging by just expecting a tarball with
Andre Heinecke <aheinecke@intevation.de>
parents: 5172
diff changeset
20 cat > "$PKG_DIR/conf/log4j.properties" << "EOF"
4355b4cecef8 Improve importer packaging by just expecting a tarball with
Andre Heinecke <aheinecke@intevation.de>
parents: 5172
diff changeset
21 log4j.rootLogger=DEBUG, IMPORTER
4355b4cecef8 Improve importer packaging by just expecting a tarball with
Andre Heinecke <aheinecke@intevation.de>
parents: 5172
diff changeset
22 log4j.appender.IMPORTER.layout=org.apache.log4j.PatternLayout
4355b4cecef8 Improve importer packaging by just expecting a tarball with
Andre Heinecke <aheinecke@intevation.de>
parents: 5172
diff changeset
23 log4j.appender.IMPORTER.layout.ConversionPattern=%d [%t] %-5p %c - %m%n
4355b4cecef8 Improve importer packaging by just expecting a tarball with
Andre Heinecke <aheinecke@intevation.de>
parents: 5172
diff changeset
24 log4j.appender.IMPORTER=org.apache.log4j.RollingFileAppender
4355b4cecef8 Improve importer packaging by just expecting a tarball with
Andre Heinecke <aheinecke@intevation.de>
parents: 5172
diff changeset
25 log4j.appender.IMPORTER.File=./import.log
4355b4cecef8 Improve importer packaging by just expecting a tarball with
Andre Heinecke <aheinecke@intevation.de>
parents: 5172
diff changeset
26 log4j.appender.IMPORTER.MaxFileSize=100000KB
4355b4cecef8 Improve importer packaging by just expecting a tarball with
Andre Heinecke <aheinecke@intevation.de>
parents: 5172
diff changeset
27 log4j.appender.IMPORTER.MaxBackupIndex=10
4355b4cecef8 Improve importer packaging by just expecting a tarball with
Andre Heinecke <aheinecke@intevation.de>
parents: 5172
diff changeset
28 EOF
5097
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
29
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
30 cd ${FLYS_DIR}/flys-backend
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
31 mvn -f pom-oracle.xml clean compile assembly:single
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
32 cp target/flys-backend-1.0-SNAPSHOT-jar-with-dependencies.jar \
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
33 $PKG_DIR/hydr_morph/importer.jar
5172
a51515056c45 Package postgresql importer and import_river script
Andre Heinecke <aheinecke@intevation.de>
parents: 5163
diff changeset
34 mvn -f pom.xml clean compile assembly:single
a51515056c45 Package postgresql importer and import_river script
Andre Heinecke <aheinecke@intevation.de>
parents: 5163
diff changeset
35 cp target/flys-backend-1.0-SNAPSHOT-jar-with-dependencies.jar \
a51515056c45 Package postgresql importer and import_river script
Andre Heinecke <aheinecke@intevation.de>
parents: 5163
diff changeset
36 $PKG_DIR/hydr_morph/importer_psql.jar
5097
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
37 cp ${FLYS_DIR}/flys-backend/contrib/shpimporter/*.py $PKG_DIR/geodaesie
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
38 cp ${FLYS_DIR}/flys-backend/contrib/run_geo.sh \
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
39 ${FLYS_DIR}/flys-backend/contrib/run_hydr_morph.sh \
5172
a51515056c45 Package postgresql importer and import_river script
Andre Heinecke <aheinecke@intevation.de>
parents: 5163
diff changeset
40 ${FLYS_DIR}/flys-backend/contrib/import_river.sh \
5097
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
41 $PKG_DIR
5163
9474e27e319a Use annotation-types.xml from flys-backend sources
Andre Heinecke <aheinecke@intevation.de>
parents: 5097
diff changeset
42 cp ${FLYS_DIR}/flys-backend/doc/annotation-types.xml $PKG_DIR/conf
5199
29d79741c2fe Fix quoting of EXTRAS in packaging script
Andre Heinecke <aheinecke@intevation.de>
parents: 5186
diff changeset
43 if [ -f "$EXTRAS" ]; then
5179
4355b4cecef8 Improve importer packaging by just expecting a tarball with
Andre Heinecke <aheinecke@intevation.de>
parents: 5172
diff changeset
44 cd $PKG_DIR
5199
29d79741c2fe Fix quoting of EXTRAS in packaging script
Andre Heinecke <aheinecke@intevation.de>
parents: 5186
diff changeset
45 tar -xzf "$EXTRAS"
5179
4355b4cecef8 Improve importer packaging by just expecting a tarball with
Andre Heinecke <aheinecke@intevation.de>
parents: 5172
diff changeset
46 fi
5097
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
47
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
48 cp ${FLYS_DIR}/flys-backend/doc/schema/*.sql $PKG_DIR/schema
5520
d5e1e4d9559e compile manual from within importer packaging script
Tom Gottfried <tom.gottfried@intevation.de>
parents: 5491
diff changeset
49
d5e1e4d9559e compile manual from within importer packaging script
Tom Gottfried <tom.gottfried@intevation.de>
parents: 5491
diff changeset
50 cd ${FLYS_DIR}/flys-backend/doc/documentation/de/
d5e1e4d9559e compile manual from within importer packaging script
Tom Gottfried <tom.gottfried@intevation.de>
parents: 5491
diff changeset
51 make importer-manual.pdf
d5e1e4d9559e compile manual from within importer packaging script
Tom Gottfried <tom.gottfried@intevation.de>
parents: 5491
diff changeset
52 cp importer-manual.pdf $PKG_DIR
5097
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
53
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
54 sed -i 's/shpimporter\/shp/geodaesie\/shp/' $PKG_DIR/run_geo.sh
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
55
415ac9bc5a56 Add importer packaging script and some doc about it
Andre Heinecke <aheinecke@intevation.de>
parents:
diff changeset
56 cd $PKG_DIR/..
5417
5422a798f105 Use minute instead of month as the last number of the date/timestamp
Andre Heinecke <aheinecke@intevation.de>
parents: 5199
diff changeset
57 DATE=$(date +%Y%m%d%H%M)
5179
4355b4cecef8 Improve importer packaging by just expecting a tarball with
Andre Heinecke <aheinecke@intevation.de>
parents: 5172
diff changeset
58 tar -czf flys-importer_${DATE}.tar.gz flys-importer
4355b4cecef8 Improve importer packaging by just expecting a tarball with
Andre Heinecke <aheinecke@intevation.de>
parents: 5172
diff changeset
59 sha1sum flys-importer_${DATE}.tar.gz > flys-importer_${DATE}.tar.gz.sha1
5186
801866a89ee0 Fix syntax error in importer packaging script
Andre Heinecke <aheinecke@intevation.de>
parents: 5179
diff changeset
60 echo Package is at: `readlink -f flys-importer_${DATE}.tar.gz`
http://dive4elements.wald.intevation.org