annotate flys-backend/contrib/shpimporter/run.sh @ 5779:ebec12def170

Datacage: Add a pool of builders to make it multi threadable. XML DOM is not thread safe. Therefore the old implementation only allowed one thread to use the builder at a time. As the complexity of the configuration has increased over time this has become a bottleneck of the whole application because it took quiet some time to build a result. Furthermore the builder code path is visited very frequent. So many concurrent requests were piled up resulting in long waits for the users. To mitigate this problem a round robin pool of builders is used now. Each of the pooled builders has an independent copy of the XML template and can be run in parallel. The number of builders is determined by the system property 'flys.datacage.pool.size'. It defaults to 4.
author Sascha L. Teichmann <teichmann@intevation.de>
date Sun, 21 Apr 2013 12:48:09 +0200
parents 94ce92e73603
children
rev   line source
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
1 #!/bin/sh
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
2
3656
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
3 # Required
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
4 RIVER_PATH="/path/to/rivers/river"
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
5 RIVER_ID=1
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
6 TARGET_SRS=31466
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
7 HOST=localhost
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
8 USER=the_user
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
9 PASS=the_pass
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
10
3656
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
11 # Optional
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
12 VERBOSE=1
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
13 SKIP_AXIS=0
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
14 SKIP_KMS=0
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
15 SKIP_CROSSSECTIONS=0
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
16 SKIP_LINES=0
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
17 SKIP_FIXPOINTS=0
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
18 SKIP_BUILDINGS=0
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
19 SKIP_FLOODPLAINS=0
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
20 SKIP_HYDR_BOUNDARIES=0
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
21 SKIP_HWS=0
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
22 SKIP_GAUGE_LOCATION=0
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
23 SKIP_UESG=0
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
24
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
25 exec python shpimporter.py \
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
26 --directory $RIVER_PATH \
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
27 --river_id $RIVER_ID \
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
28 --target_srs $TARGET_SRS \
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
29 --host $HOST \
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
30 --user $USER \
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
31 --password $PASS \
3656
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
32 --verbose $VERBOSE \
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
33 --skip_axis $SKIP_AXIS \
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
34 --skip_kms $SKIP_KMS \
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
35 --skip_crosssections $SKIP_CROSSSECTIONS \
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
36 --skip_lines $SKIP_LINES \
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
37 --skip_fixpoints $SKIP_FIXPOINTS \
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
38 --skip_buildings $SKIP_BUILDINGS \
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
39 --skip_floodplains $SKIP_FLOODPLAINS \
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
40 --skip_hydr_boundaries $SKIP_HYDR_BOUNDARIES \
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
41 --skip_hws $SKIP_HWS \
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
42 --skip_gauge_locations $SKIP_GAUGE_LOCATION \
6603aeeda6e3 Shape importer: modified structure of run.sh script.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
43 --skip_uesgs $SKIP_UESG
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
44

http://dive4elements.wald.intevation.org