Mercurial > dive4elements > river

annotate flys-backend/contrib/shpimporter/shpimporter.py @ 4970:174fbaa3d344 dami

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Add handling of River Names and remove target_src parameter This is the first step to make the shpimporter into a more generic geo importer that communicates directly (not only over ogr) with the database. Untested with Oracle
author Andre Heinecke <aheinecke@intevation.de>
date Wed, 13 Feb 2013 12:02:30 +0100
parents 1469066cc7d9
children 9eea3cd22ee7
rev   line source
4872
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
1 try:
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
2 from osgeo import ogr
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
3 except ImportErrror:
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
4 import ogr
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2798
diff changeset
5
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
6 import utils, optparse
4970
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
7 import sys
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
8 import os
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
9
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2798
diff changeset
10 from uesg import UESG
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2798
diff changeset
11 from axis import Axis
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2798
diff changeset
12 from km import KM
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2798
diff changeset
13 from lines import Line
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2798
diff changeset
14 from fixpoints import Fixpoint
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2798
diff changeset
15 from buildings import Building
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2798
diff changeset
16 from crosssectiontracks import CrosssectionTrack
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
17 from floodplains import Floodplain
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
18 from boundaries import HydrBoundary, HydrBoundaryPoly
4879
63f82bdd45f8 (importer) Enable hws_* importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4872
diff changeset
19 from hws import HWSLines, HWSPoints
2862
998f1a7dcfde New shape importers and schema modifications.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2861
diff changeset
20 from gauges import GaugeLocation
998f1a7dcfde New shape importers and schema modifications.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2861
diff changeset
21 from catchments import Catchment
2798
5a654f2e35bc Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
22
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
23
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
24 VERBOSE_DEBUG=2
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
25 VERBOSE_INFO=1
2798
5a654f2e35bc Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
26
5a654f2e35bc Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
27
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
28 def DEBUG(msg):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
29 config = getConfig()
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
30 if config.verbose >= VERBOSE_DEBUG:
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
31 print "DEBUG: %s" % msg
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
32
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
33 def INFO(msg):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
34 config = getConfig()
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
35 if config.verbose >= VERBOSE_INFO:
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
36 print "INFO: %s" % msg
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
37
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
38 def ERROR(msg):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
39 config = getConfig()
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
40 print "ERROR: %s" % msg
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
41
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
42
4970
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
43 def getImporters(river_id, dbconn, dry_run):
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2798
diff changeset
44 return [
4970
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
45 Axis(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
46 KM(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
47 CrosssectionTrack(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
48 Line(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
49 Fixpoint(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
50 Building(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
51 Floodplain(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
52 HydrBoundary(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
53 HydrBoundaryPoly(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
54 HWSLines(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
55 HWSPoints(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
56 GaugeLocation(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
57 Catchment(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
58 UESG(river_id, dbconn, dry_run)
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
59 ]
2798
5a654f2e35bc Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
60
5a654f2e35bc Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
61
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
62 def getConfig():
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
63 parser = optparse.OptionParser()
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
64 parser.add_option("--directory", type="string")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
65 parser.add_option("--target_srs", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
66 parser.add_option("--host", type="string")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
67 parser.add_option("--user", type="string")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
68 parser.add_option("--password", type="string")
4970
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
69 parser.add_option("--river_name", type="string")
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
70 parser.add_option("--verbose", type="int", default=1)
3655
8654e4480fc6 Shape importer: added command line option dry_run to supress database transactions.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
71 parser.add_option("--dry_run", type="int", default=0)
4871
890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr
Andre Heinecke <aheinecke@intevation.de>
parents: 3655
diff changeset
72 parser.add_option("--ogr_connection", type="string")
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
73 parser.add_option("--skip_axis", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
74 parser.add_option("--skip_hydr_boundaries", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
75 parser.add_option("--skip_buildings", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
76 parser.add_option("--skip_crosssections", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
77 parser.add_option("--skip_lines", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
78 parser.add_option("--skip_fixpoints", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
79 parser.add_option("--skip_floodplains", type="int")
4879
63f82bdd45f8 (importer) Enable hws_* importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4872
diff changeset
80 parser.add_option("--skip_hws_lines", type="int")
63f82bdd45f8 (importer) Enable hws_* importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4872
diff changeset
81 parser.add_option("--skip_hws_points", type="int")
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
82 parser.add_option("--skip_gauge_locations", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
83 parser.add_option("--skip_catchments", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
84 parser.add_option("--skip_kms", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
85 parser.add_option("--skip_uesgs", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
86 (config, args) = parser.parse_args()
2798
5a654f2e35bc Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
87
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
88 if config.directory == None:
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
89 ERROR("No river directory specified!")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
90 raise Exception("Invalid config")
4871
890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr
Andre Heinecke <aheinecke@intevation.de>
parents: 3655
diff changeset
91 if not config.ogr_connection:
890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr
Andre Heinecke <aheinecke@intevation.de>
parents: 3655
diff changeset
92 if not config.host:
890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr
Andre Heinecke <aheinecke@intevation.de>
parents: 3655
diff changeset
93 ERROR("No database host specified!")
890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr
Andre Heinecke <aheinecke@intevation.de>
parents: 3655
diff changeset
94 raise Exception("Invalid config")
890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr
Andre Heinecke <aheinecke@intevation.de>
parents: 3655
diff changeset
95 if not config.user:
890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr
Andre Heinecke <aheinecke@intevation.de>
parents: 3655
diff changeset
96 ERROR("No databaser user specified!")
890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr
Andre Heinecke <aheinecke@intevation.de>
parents: 3655
diff changeset
97 raise Exception("Invalid config")
890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr
Andre Heinecke <aheinecke@intevation.de>
parents: 3655
diff changeset
98 if not config.password:
890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr
Andre Heinecke <aheinecke@intevation.de>
parents: 3655
diff changeset
99 ERROR("No password specified!")
890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr
Andre Heinecke <aheinecke@intevation.de>
parents: 3655
diff changeset
100 raise Exception("Invalid config")
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
101
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
102 return config
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
103
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
104
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
105 def skip_importer(config, importer):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
106 if config.skip_axis == 1 and isinstance(importer, Axis):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
107 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
108 elif config.skip_hydr_boundaries == 1 and isinstance(importer, HydrBoundary):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
109 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
110 elif config.skip_hydr_boundaries == 1 and isinstance(importer, HydrBoundaryPoly):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
111 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
112 elif config.skip_buildings == 1 and isinstance(importer, Building):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
113 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
114 elif config.skip_crosssections == 1 and isinstance(importer, CrosssectionTrack):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
115 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
116 elif config.skip_lines == 1 and isinstance(importer, Line):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
117 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
118 elif config.skip_fixpoints == 1 and isinstance(importer, Fixpoint):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
119 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
120 elif config.skip_floodplains == 1 and isinstance(importer, Floodplain):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
121 return True
4963
1469066cc7d9 Importer: Fix the order of the config check
Andre Heinecke <aheinecke@intevation.de>
parents: 4955
diff changeset
122 elif config.skip_hws_lines == 1 and isinstance(importer, HWSLines):
1469066cc7d9 Importer: Fix the order of the config check
Andre Heinecke <aheinecke@intevation.de>
parents: 4955
diff changeset
123 return True
4879
63f82bdd45f8 (importer) Enable hws_* importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4872
diff changeset
124 elif config.skip_hws_points == 1 and isinstance(importer, HWSPoints):
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
125 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
126 elif config.skip_gauge_locations == 1 and isinstance(importer, GaugeLocation):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
127 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
128 elif config.skip_catchments == 1 and isinstance(importer, Catchment):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
129 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
130 elif config.skip_kms == 1 and isinstance(importer, KM):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
131 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
132 elif config.skip_uesgs == 1 and isinstance(importer, UESG):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
133 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
134
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
135 return False
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
136
4872
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
137 def main():
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
138 config=None
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
139 try:
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
140 config = getConfig()
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
141 except:
4872
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
142 return -1
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
143
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
144 if config == None:
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
145 ERROR("Unable to read config from command line!")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
146 return
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
147
3655
8654e4480fc6 Shape importer: added command line option dry_run to supress database transactions.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
148 if config.dry_run > 0:
8654e4480fc6 Shape importer: added command line option dry_run to supress database transactions.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
149 INFO("You enable 'dry_run'. No database transaction will take place!")
8654e4480fc6 Shape importer: added command line option dry_run to supress database transactions.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
150
4872
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
151 if config.ogr_connection:
4879
63f82bdd45f8 (importer) Enable hws_* importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4872
diff changeset
152 connstr = config.ogr_connection
4872
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
153 else:
4879
63f82bdd45f8 (importer) Enable hws_* importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4872
diff changeset
154 connstr = 'OCI:%s/%s@%s' % (config.user, config.password, config.host)
4872
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
155
4970
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
156 if 'OCI:' in connstr:
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
157 try:
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
158 import cx_Oracle as dbapi
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
159 raw_connstr=connstr.replace("OCI:", "")
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
160 except ImportErrror:
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
161 ERROR("Module cx_Oracle not found in: %s\n"
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
162 "Neccessary to connect to a Oracle Database.\n"
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
163 "Please refer to the installation "
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
164 "documentation." % sys.path)
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
165
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
166 else: # Currently only support for oracle and postgres
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
167 try:
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
168 import psycopg2 as dbapi
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
169 raw_connstr=connstr.replace("PG:", "")
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
170 except ImportErrror:
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
171 ERROR("Module psycopg2 not found in: %s\n"
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
172 "Neccessary to connect to a Posgresql Database.\n"
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
173 "Please refer to the installation "
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
174 "documentation." % sys.path)
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
175
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
176 dbconn_raw = dbapi.connect(raw_connstr)
4879
63f82bdd45f8 (importer) Enable hws_* importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4872
diff changeset
177 dbconn = ogr.Open(connstr)
63f82bdd45f8 (importer) Enable hws_* importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4872
diff changeset
178
63f82bdd45f8 (importer) Enable hws_* importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4872
diff changeset
179 if dbconn == None:
63f82bdd45f8 (importer) Enable hws_* importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4872
diff changeset
180 ERROR("Could not connect to database %s" % connstr)
4872
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
181 return -1
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
182
2798
5a654f2e35bc Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
183 types = {}
5a654f2e35bc Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
184
4970
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
185 directories = []
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
186 if not config.river_name:
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
187 for file in os.listdir(config.directory):
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
188 if os.path.isdir(file):
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
189 directories.append(os.path.join(file))
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
190 else:
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
191 directories.append(config.directory)
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
192
4970
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
193 for directory in directories:
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
194 if not config.river_name:
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
195 river_name = os.path.basename(os.path.normpath(directory))
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
196 else:
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
197 river_name = config.river_name
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
198 river_id = utils.getRiverId(dbconn_raw, river_name)
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2798
diff changeset
199
4970
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
200 if not river_id:
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
201 INFO("Could not find river in database. Skipping: %s"
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
202 % river_name)
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
203 continue
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
204 else:
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
205 INFO("Importing River: %s" % river_name)
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
206
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
207 for importer in getImporters(river_id, dbconn, config.dry_run):
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
208 if skip_importer(config, importer):
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
209 INFO("Skip import of '%s'" % importer.getName())
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
210 continue
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
211
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
212 INFO("Start import of '%s'" % importer.getName())
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
213
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
214 shapes = utils.findShapefiles(importer.getPath(config.directory))
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
215 DEBUG("Found %i Shapefiles" % len(shapes))
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
216
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
217 for shpTuple in shapes:
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
218 geomType = importer.walkOverShapes(shpTuple)
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
219 try:
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
220 if geomType is not None:
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
221 num = types[geomType]
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
222 types[geomType] = num+1
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
223 except:
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4963
diff changeset
224 types[geomType] = 1
2798
5a654f2e35bc Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
225
5a654f2e35bc Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
226 for key in types:
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
227 DEBUG("%i x geometry type %s" % (types[key], key))
2798
5a654f2e35bc Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
228
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
229
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
230 if __name__ == '__main__':
4872
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
231 main()
http://dive4elements.wald.intevation.org