annotate flys-backend/contrib/run_geo.sh @ 5779:ebec12def170

Datacage: Add a pool of builders to make it multi threadable. XML DOM is not thread safe. Therefore the old implementation only allowed one thread to use the builder at a time. As the complexity of the configuration has increased over time this has become a bottleneck of the whole application because it took quiet some time to build a result. Furthermore the builder code path is visited very frequent. So many concurrent requests were piled up resulting in long waits for the users. To mitigate this problem a round robin pool of builders is used now. Each of the pooled builders has an independent copy of the XML template and can be run in parallel. The number of builders is determined by the system property 'flys.datacage.pool.size'. It defaults to 4.
author Sascha L. Teichmann <teichmann@intevation.de>
date Sun, 21 Apr 2013 12:48:09 +0200
parents 673f68bb19a4
children
rev   line source
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
1 #!/bin/sh
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
2
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
3 # Required
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
4 RIVER_PATH="/vol1/projects/Geospatial/flys-3.0/testdaten/Gewaesser/Saar"
4971
fc7ebf104779 Update documentation and example run script
Andre Heinecke <aheinecke@intevation.de>
parents: 4873
diff changeset
5 RIVER_NAME="Saar"
4873
2b371e42a9af Add HWS Lines parameters and ogr_connection example
Andre Heinecke <aheinecke@intevation.de>
parents: 3683
diff changeset
6
2b371e42a9af Add HWS Lines parameters and ogr_connection example
Andre Heinecke <aheinecke@intevation.de>
parents: 3683
diff changeset
7 # Set this to your target database for Oracle
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
8 HOST=localhost
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
9 USER=flys28
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
10 PASS=flys28
4873
2b371e42a9af Add HWS Lines parameters and ogr_connection example
Andre Heinecke <aheinecke@intevation.de>
parents: 3683
diff changeset
11 # Alternatively you can provide a direct connection string:
2b371e42a9af Add HWS Lines parameters and ogr_connection example
Andre Heinecke <aheinecke@intevation.de>
parents: 3683
diff changeset
12 # OGR_CONNECTION="PG:dbname=flys host=localhost port=5432 user=flys password=flys"
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
13
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
14 # Optional
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
15 VERBOSE=1
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
16 SKIP_AXIS=0
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
17 SKIP_KMS=0
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
18 SKIP_CROSSSECTIONS=0
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
19 SKIP_FIXPOINTS=0
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
20 SKIP_BUILDINGS=0
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
21 SKIP_FLOODPLAINS=0
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
22 SKIP_HYDR_BOUNDARIES=0
4873
2b371e42a9af Add HWS Lines parameters and ogr_connection example
Andre Heinecke <aheinecke@intevation.de>
parents: 3683
diff changeset
23 SKIP_HWS_LINES=0
2b371e42a9af Add HWS Lines parameters and ogr_connection example
Andre Heinecke <aheinecke@intevation.de>
parents: 3683
diff changeset
24 SKIP_HWS_POINTS=0
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
25 SKIP_GAUGE_LOCATION=0
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
26 SKIP_CATCHMENTS=0
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
27 SKIP_UESG=0
5035
1f33012cc02a Add skip-dgm option to example run script
Andre Heinecke <aheinecke@intevation.de>
parents: 4996
diff changeset
28 SKIP_DGM=0
5353
d086ce6c13d3 Add jetties importer
Andre Heinecke <aheinecke@intevation.de>
parents: 5074
diff changeset
29 SKIP_JETTIES=0
5545
ed149d5d7fb7 Add floodmarks importer
Andre Heinecke <aheinecke@intevation.de>
parents: 5353
diff changeset
30 SKIP_FLOODMARKS=0
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
31
5066
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 5035
diff changeset
32 # There should be no need to change anything below this line
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 5035
diff changeset
33
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
34 DIR=`dirname $0`
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
35 DIR=`readlink -f "$DIR"`
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
36
5074
90b9b80ffbbf Fix example run script to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 5066
diff changeset
37 OPTIONAL_LIBS="${DIR}"/opt
5066
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 5035
diff changeset
38 if [ -d "$OPTIONAL_LIBS" ]; then
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 5035
diff changeset
39 export PATH="$OPTIONAL_LIBS/bin:$PATH"
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 5035
diff changeset
40 export LD_LIBRARY_PATH="$OPTIONAL_LIBS/lib:$LD_LIBRARY_PATH"
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 5035
diff changeset
41 export LD_LIBRARY_PATH="$OPTIONAL_LIBS/lib64:$LD_LIBRARY_PATH"
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 5035
diff changeset
42 export PYTHONPATH="$OPTIONAL_LIBS/lib/python2.6/site-packages:$PYTHONPATH"
5074
90b9b80ffbbf Fix example run script to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 5066
diff changeset
43 export PYTHONPATH="$OPTIONAL_LIBS/lib64/python2.6/site-packages:$PYTHONPATH"
90b9b80ffbbf Fix example run script to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 5066
diff changeset
44 export GDAL_DATA="$OPTIONAL_LIBS/share/gdal"
5066
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 5035
diff changeset
45 fi
8208df10df79 Update run example scripts to work with the new packaging
Andre Heinecke <aheinecke@intevation.de>
parents: 5035
diff changeset
46
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
47 exec python $DIR/shpimporter/shpimporter.py \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
48 --directory $RIVER_PATH \
5563
673f68bb19a4 Geo-importer: accept river names with white space
Tom Gottfried <tom@intevation.de>
parents: 5545
diff changeset
49 --river_name "$RIVER_NAME" \
4873
2b371e42a9af Add HWS Lines parameters and ogr_connection example
Andre Heinecke <aheinecke@intevation.de>
parents: 3683
diff changeset
50 --ogr_connection "$OGR_CONNECTION" \
3683
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
51 --host $HOST \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
52 --user $USER \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
53 --password $PASS \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
54 --verbose $VERBOSE \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
55 --skip_axis $SKIP_AXIS \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
56 --skip_kms $SKIP_KMS \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
57 --skip_crosssections $SKIP_CROSSSECTIONS \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
58 --skip_fixpoints $SKIP_FIXPOINTS \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
59 --skip_buildings $SKIP_BUILDINGS \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
60 --skip_floodplains $SKIP_FLOODPLAINS \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
61 --skip_hydr_boundaries $SKIP_HYDR_BOUNDARIES \
4298ef9bd4a6 Added run scripts for both importers.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
62 --skip_gauge_locations $SKIP_GAUGE_LOCATION \
4873
2b371e42a9af Add HWS Lines parameters and ogr_connection example
Andre Heinecke <aheinecke@intevation.de>
parents: 3683
diff changeset
63 --skip_uesgs $SKIP_UESG \
2b371e42a9af Add HWS Lines parameters and ogr_connection example
Andre Heinecke <aheinecke@intevation.de>
parents: 3683
diff changeset
64 --skip_hws_lines $SKIP_HWS_LINES \
5035
1f33012cc02a Add skip-dgm option to example run script
Andre Heinecke <aheinecke@intevation.de>
parents: 4996
diff changeset
65 --skip_hws_points $SKIP_HWS_POINTS \
5353
d086ce6c13d3 Add jetties importer
Andre Heinecke <aheinecke@intevation.de>
parents: 5074
diff changeset
66 --skip_dgm $SKIP_DGM \
5545
ed149d5d7fb7 Add floodmarks importer
Andre Heinecke <aheinecke@intevation.de>
parents: 5353
diff changeset
67 --skip_jetties $SKIP_JETTIES \
ed149d5d7fb7 Add floodmarks importer
Andre Heinecke <aheinecke@intevation.de>
parents: 5353
diff changeset
68 --skip_floodmarks $SKIP_FLOODMARKS

http://dive4elements.wald.intevation.org