dive4elements/river: backend/contrib/shpimporter/shpimporter.py annotate

annotate backend/contrib/shpimporter/shpimporter.py @ 6208:715e010d40de

Add * in the first line of AT Exports. According to the "spec" this is requried. Desktop flys does not care though as it just ignores the first line.

author	Andre Heinecke <aheinecke@intevation.de>
date	Wed, 05 Jun 2013 18:02:39 +0200
parents	5aa05a7a34b7
children	e68e414dceb2

rev	line source
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	1 try:
a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	2 from osgeo import ogr
4977 ee908e4946cd corrected Typo Tom Gottfried <tom.gottfried@intevation.de> parents: 4975 diff changeset	3 except ImportError:
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	4 import ogr
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	5
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	6 import utils, optparse
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	7 import sys
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	8 import os
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	9 import logging
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	10
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	11 from uesg import UESG
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	12 from axis import Axis
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	13 from km import KM
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	14 from fixpoints import Fixpoint
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	15 from buildings import Building
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	16 from crosssectiontracks import CrosssectionTrack
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	17 from floodplains import Floodplain
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	18 from boundaries import HydrBoundary, HydrBoundaryPoly
4879 63f82bdd45f8 (importer) Enable hws_* importers Andre Heinecke <aheinecke@intevation.de> parents: 4872 diff changeset	19 from hws import HWSLines, HWSPoints
2862 998f1a7dcfde New shape importers and schema modifications. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2861 diff changeset	20 from gauges import GaugeLocation
5353 d086ce6c13d3 Add jetties importer Andre Heinecke <aheinecke@intevation.de> parents: 5221 diff changeset	21 from jetties import Jetties
4975 9eea3cd22ee7 Add dgm file reader although an awk script should have been more appropiate Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	22 from dgm import insertRiverDgm
5545 ed149d5d7fb7 Add floodmarks importer Andre Heinecke <aheinecke@intevation.de> parents: 5353 diff changeset	23 from floodmarks import Floodmark
2798 5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	24
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	25 logger = logging.getLogger("shpimporter")
2798 5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	26
5219 40dc001594e4 Allow unicode river names Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	27 os.environ["NLS_LANG"] = ".AL32UTF8"
40dc001594e4 Allow unicode river names Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	28
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	29 def initialize_logging(level):
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	30 """Initializes the logging system"""
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	31 root = logging.getLogger()
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	32 root.setLevel(level)
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	33 hdlr = logging.StreamHandler()
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	34 fmt = logging.Formatter("%(levelname)s %(name)s: %(message)s")
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	35 hdlr.setFormatter(fmt)
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	36 root.addHandler(hdlr)
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	37
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	38 def getImporters(river_id, dbconn, dry_run):
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	39 return [
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	40 Axis(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	41 KM(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	42 CrosssectionTrack(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	43 Fixpoint(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	44 Building(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	45 Floodplain(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	46 HydrBoundary(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	47 HydrBoundaryPoly(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	48 HWSLines(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	49 HWSPoints(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	50 GaugeLocation(river_id, dbconn, dry_run),
5353 d086ce6c13d3 Add jetties importer Andre Heinecke <aheinecke@intevation.de> parents: 5221 diff changeset	51 Jetties(river_id, dbconn, dry_run),
5545 ed149d5d7fb7 Add floodmarks importer Andre Heinecke <aheinecke@intevation.de> parents: 5353 diff changeset	52 Floodmark(river_id, dbconn, dry_run),
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	53 UESG(river_id, dbconn, dry_run)
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	54 ]
2798 5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	55
5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	56
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	57 def getConfig():
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	58 parser = optparse.OptionParser()
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	59 parser.add_option("--directory", type="string")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	60 parser.add_option("--target_srs", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	61 parser.add_option("--host", type="string")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	62 parser.add_option("--user", type="string")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	63 parser.add_option("--password", type="string")
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	64 parser.add_option("--river_name", type="string")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	65 parser.add_option("--verbose", type="int", default=1)
3655 8654e4480fc6 Shape importer: added command line option dry_run to supress database transactions. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 3654 diff changeset	66 parser.add_option("--dry_run", type="int", default=0)
4871 890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr Andre Heinecke <aheinecke@intevation.de> parents: 3655 diff changeset	67 parser.add_option("--ogr_connection", type="string")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	68 parser.add_option("--skip_axis", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	69 parser.add_option("--skip_hydr_boundaries", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	70 parser.add_option("--skip_buildings", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	71 parser.add_option("--skip_crosssections", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	72 parser.add_option("--skip_fixpoints", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	73 parser.add_option("--skip_floodplains", type="int")
4879 63f82bdd45f8 (importer) Enable hws_* importers Andre Heinecke <aheinecke@intevation.de> parents: 4872 diff changeset	74 parser.add_option("--skip_hws_lines", type="int")
63f82bdd45f8 (importer) Enable hws_* importers Andre Heinecke <aheinecke@intevation.de> parents: 4872 diff changeset	75 parser.add_option("--skip_hws_points", type="int")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	76 parser.add_option("--skip_gauge_locations", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	77 parser.add_option("--skip_kms", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	78 parser.add_option("--skip_uesgs", type="int")
4975 9eea3cd22ee7 Add dgm file reader although an awk script should have been more appropiate Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	79 parser.add_option("--skip_dgm", type="int")
5353 d086ce6c13d3 Add jetties importer Andre Heinecke <aheinecke@intevation.de> parents: 5221 diff changeset	80 parser.add_option("--skip_jetties", type="int")
5545 ed149d5d7fb7 Add floodmarks importer Andre Heinecke <aheinecke@intevation.de> parents: 5353 diff changeset	81 parser.add_option("--skip_floodmarks", type="int")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	82 (config, args) = parser.parse_args()
2798 5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	83
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	84 if config.verbose > 1:
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	85 initialize_logging(logging.DEBUG)
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	86 elif config.verbose == 1:
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	87 initialize_logging(logging.INFO)
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	88 else:
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	89 initialize_logging(logging.WARN)
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	90
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	91 if config.directory == None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	92 logger.error("No river directory specified!")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	93 raise Exception("Invalid config")
4871 890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr Andre Heinecke <aheinecke@intevation.de> parents: 3655 diff changeset	94 if not config.ogr_connection:
890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr Andre Heinecke <aheinecke@intevation.de> parents: 3655 diff changeset	95 if not config.host:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	96 logger.error("No database host specified!")
4871 890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr Andre Heinecke <aheinecke@intevation.de> parents: 3655 diff changeset	97 raise Exception("Invalid config")
890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr Andre Heinecke <aheinecke@intevation.de> parents: 3655 diff changeset	98 if not config.user:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	99 logger.error("No databaser user specified!")
4871 890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr Andre Heinecke <aheinecke@intevation.de> parents: 3655 diff changeset	100 raise Exception("Invalid config")
890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr Andre Heinecke <aheinecke@intevation.de> parents: 3655 diff changeset	101 if not config.password:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	102 logger.error("No password specified!")
4871 890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr Andre Heinecke <aheinecke@intevation.de> parents: 3655 diff changeset	103 raise Exception("Invalid config")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	104
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	105 return config
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	106
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	107
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	108 def skip_importer(config, importer):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	109 if config.skip_axis == 1 and isinstance(importer, Axis):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	110 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	111 elif config.skip_hydr_boundaries == 1 and isinstance(importer, HydrBoundary):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	112 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	113 elif config.skip_hydr_boundaries == 1 and isinstance(importer, HydrBoundaryPoly):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	114 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	115 elif config.skip_buildings == 1 and isinstance(importer, Building):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	116 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	117 elif config.skip_crosssections == 1 and isinstance(importer, CrosssectionTrack):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	118 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	119 elif config.skip_fixpoints == 1 and isinstance(importer, Fixpoint):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	120 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	121 elif config.skip_floodplains == 1 and isinstance(importer, Floodplain):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	122 return True
4963 1469066cc7d9 Importer: Fix the order of the config check Andre Heinecke <aheinecke@intevation.de> parents: 4955 diff changeset	123 elif config.skip_hws_lines == 1 and isinstance(importer, HWSLines):
1469066cc7d9 Importer: Fix the order of the config check Andre Heinecke <aheinecke@intevation.de> parents: 4955 diff changeset	124 return True
5011 514948fdae54 Do not skip hws_lines when skip hws_points is true Andre Heinecke <aheinecke@intevation.de> parents: 5006 diff changeset	125 elif config.skip_hws_points == 1 and isinstance(importer, HWSPoints) and \
514948fdae54 Do not skip hws_lines when skip hws_points is true Andre Heinecke <aheinecke@intevation.de> parents: 5006 diff changeset	126 not isinstance(importer, HWSLines):
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	127 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	128 elif config.skip_gauge_locations == 1 and isinstance(importer, GaugeLocation):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	129 return True
5353 d086ce6c13d3 Add jetties importer Andre Heinecke <aheinecke@intevation.de> parents: 5221 diff changeset	130 elif config.skip_jetties == 1 and isinstance(importer, Jetties):
d086ce6c13d3 Add jetties importer Andre Heinecke <aheinecke@intevation.de> parents: 5221 diff changeset	131 return True
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	132 elif config.skip_kms == 1 and isinstance(importer, KM):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	133 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	134 elif config.skip_uesgs == 1 and isinstance(importer, UESG):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	135 return True
5545 ed149d5d7fb7 Add floodmarks importer Andre Heinecke <aheinecke@intevation.de> parents: 5353 diff changeset	136 elif config.skip_floodmarks == 1 and isinstance(importer, Floodmark):
ed149d5d7fb7 Add floodmarks importer Andre Heinecke <aheinecke@intevation.de> parents: 5353 diff changeset	137 return True
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	138
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	139 return False
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	140
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	141 def main():
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	142 config=None
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	143 try:
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	144 config = getConfig()
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	145 except:
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	146 return -1
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	147
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	148 if config == None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	149 logger.error("Unable to read config from command line!")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	150 return
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	151
3655 8654e4480fc6 Shape importer: added command line option dry_run to supress database transactions. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 3654 diff changeset	152 if config.dry_run > 0:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	153 logger.info("You enable 'dry_run'. No database transaction will take place!")
3655 8654e4480fc6 Shape importer: added command line option dry_run to supress database transactions. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 3654 diff changeset	154
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	155 if config.ogr_connection:
4879 63f82bdd45f8 (importer) Enable hws_* importers Andre Heinecke <aheinecke@intevation.de> parents: 4872 diff changeset	156 connstr = config.ogr_connection
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	157 else:
4879 63f82bdd45f8 (importer) Enable hws_* importers Andre Heinecke <aheinecke@intevation.de> parents: 4872 diff changeset	158 connstr = 'OCI:%s/%s@%s' % (config.user, config.password, config.host)
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	159
5036 d6860fca89e4 Pull information about the backend into dgm. Andre Heinecke <aheinecke@intevation.de> parents: 5028 diff changeset	160 oracle = False # Marker if oracle is used.
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	161 if 'OCI:' in connstr:
5036 d6860fca89e4 Pull information about the backend into dgm. Andre Heinecke <aheinecke@intevation.de> parents: 5028 diff changeset	162 oracle = True
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	163 try:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	164 import cx_Oracle as dbapi
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	165 raw_connstr=connstr.replace("OCI:", "")
5077 c5187ab9f571 Fix Syntax Errror Andre Heinecke <aheinecke@intevation.de> parents: 5036 diff changeset	166 except ImportError:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	167 logger.error("Module cx_Oracle not found in: %s\n"
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	168 "Neccessary to connect to a Oracle Database.\n"
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	169 "Please refer to the installation "
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	170 "documentation." % sys.path)
5036 d6860fca89e4 Pull information about the backend into dgm. Andre Heinecke <aheinecke@intevation.de> parents: 5028 diff changeset	171 return -1
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	172
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	173 else: # Currently only support for oracle and postgres
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	174 try:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	175 import psycopg2 as dbapi
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	176 raw_connstr=connstr.replace("PG:", "")
4977 ee908e4946cd corrected Typo Tom Gottfried <tom.gottfried@intevation.de> parents: 4975 diff changeset	177 except ImportError:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	178 logger.error("Module psycopg2 not found in: %s\n"
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	179 "Neccessary to connect to a Posgresql Database.\n"
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	180 "Please refer to the installation "
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	181 "documentation." % sys.path)
5036 d6860fca89e4 Pull information about the backend into dgm. Andre Heinecke <aheinecke@intevation.de> parents: 5028 diff changeset	182 return -1
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	183
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	184 dbconn_raw = dbapi.connect(raw_connstr)
4879 63f82bdd45f8 (importer) Enable hws_* importers Andre Heinecke <aheinecke@intevation.de> parents: 4872 diff changeset	185 dbconn = ogr.Open(connstr)
63f82bdd45f8 (importer) Enable hws_* importers Andre Heinecke <aheinecke@intevation.de> parents: 4872 diff changeset	186
63f82bdd45f8 (importer) Enable hws_* importers Andre Heinecke <aheinecke@intevation.de> parents: 4872 diff changeset	187 if dbconn == None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	188 logger.error("Could not connect to database %s" % connstr)
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	189 return -1
a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	190
2798 5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	191 types = {}
5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	192
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	193 directories = []
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	194 if not config.river_name:
5005 eb2d6609387c Importer: Fix directory paths and river name encoding Andre Heinecke <aheinecke@intevation.de> parents: 4998 diff changeset	195 for file in [os.path.join(config.directory, d) for d in \
eb2d6609387c Importer: Fix directory paths and river name encoding Andre Heinecke <aheinecke@intevation.de> parents: 4998 diff changeset	196 os.listdir(config.directory)]:
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	197 if os.path.isdir(file):
5005 eb2d6609387c Importer: Fix directory paths and river name encoding Andre Heinecke <aheinecke@intevation.de> parents: 4998 diff changeset	198 directories.append(file)
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	199 else:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	200 directories.append(config.directory)
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	201
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	202 for directory in directories:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	203 if not config.river_name:
5005 eb2d6609387c Importer: Fix directory paths and river name encoding Andre Heinecke <aheinecke@intevation.de> parents: 4998 diff changeset	204 river_name = utils.getUTF8Path(
eb2d6609387c Importer: Fix directory paths and river name encoding Andre Heinecke <aheinecke@intevation.de> parents: 4998 diff changeset	205 os.path.basename(os.path.normpath(directory)))
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	206 else:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	207 river_name = config.river_name
5036 d6860fca89e4 Pull information about the backend into dgm. Andre Heinecke <aheinecke@intevation.de> parents: 5028 diff changeset	208 river_id = utils.getRiverId(dbconn_raw, river_name, oracle)
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	209
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	210 if not river_id:
5221 aa043641d9c8 Fix syntax error when river id is not found (pylint could have told me this) Andre Heinecke <aheinecke@intevation.de> parents: 5219 diff changeset	211 logger.info(u"Could not find river in database. Skipping: %s"
aa043641d9c8 Fix syntax error when river id is not found (pylint could have told me this) Andre Heinecke <aheinecke@intevation.de> parents: 5219 diff changeset	212 % unicode(utils.getUTF8(river_name), "UTF-8"))
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	213 continue
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	214 else:
5219 40dc001594e4 Allow unicode river names Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	215 logger.info(u"Importing River: %s" % unicode(
40dc001594e4 Allow unicode river names Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	216 utils.getUTF8(river_name), "UTF-8"))
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	217
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	218 for importer in getImporters(river_id, dbconn, config.dry_run):
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	219 if skip_importer(config, importer):
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	220 logger.info("Skip import of '%s'" % importer.getName())
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	221 continue
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	222
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	223 logger.info("Start import of '%s'" % importer.getName())
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	224
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	225 shapes = utils.findShapefiles(importer.getPath(config.directory))
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	226 logger.debug("Found %i Shapefiles" % len(shapes))
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	227
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	228 for shpTuple in shapes:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	229 geomType = importer.walkOverShapes(shpTuple)
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	230 try:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	231 if geomType is not None:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	232 num = types[geomType]
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	233 types[geomType] = num+1
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	234 except:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	235 types[geomType] = 1
2798 5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	236
4975 9eea3cd22ee7 Add dgm file reader although an awk script should have been more appropiate Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	237 for key in types:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	238 logger.debug("%i x geometry type %s" % (types[key], key))
2798 5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	239
4975 9eea3cd22ee7 Add dgm file reader although an awk script should have been more appropiate Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	240 if not config.skip_dgm:
9eea3cd22ee7 Add dgm file reader although an awk script should have been more appropiate Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	241 dgmfilename = os.path.join(
9eea3cd22ee7 Add dgm file reader although an awk script should have been more appropiate Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	242 config.directory, "..", "DGMs.csv")
4989 f4d61ac84b76 If we bother to check if the file is readable we might also Andre Heinecke <aheinecke@intevation.de> parents: 4978 diff changeset	243 if not os.access(dgmfilename, os.R_OK) or not \
f4d61ac84b76 If we bother to check if the file is readable we might also Andre Heinecke <aheinecke@intevation.de> parents: 4978 diff changeset	244 os.path.isfile(dgmfilename):
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	245 logger.info("Could not find or access DGM file: %s \n"
4975 9eea3cd22ee7 Add dgm file reader although an awk script should have been more appropiate Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	246 "Skipping DGM import." % dgmfilename)
4978 b5ac7588cf4d Use os.access instead of try/except open Andre Heinecke <aheinecke@intevation.de> parents: 4977 diff changeset	247 else:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	248 logger.info("Inserting DGM meta information in 'dem' table.")
5036 d6860fca89e4 Pull information about the backend into dgm. Andre Heinecke <aheinecke@intevation.de> parents: 5028 diff changeset	249 insertRiverDgm(dbconn_raw, dgmfilename, river_name,
d6860fca89e4 Pull information about the backend into dgm. Andre Heinecke <aheinecke@intevation.de> parents: 5028 diff changeset	250 config.dry_run, oracle)
5012 a79f6a085ab6 Add message if DGM import is skipped Andre Heinecke <aheinecke@intevation.de> parents: 5011 diff changeset	251 else:
a79f6a085ab6 Add message if DGM import is skipped Andre Heinecke <aheinecke@intevation.de> parents: 5011 diff changeset	252 logger.info("Skip import of DGM.")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	253
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	254 if __name__ == '__main__':
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	255 main()

Mercurial > dive4elements > river

annotate backend/contrib/shpimporter/shpimporter.py @ 6208:715e010d40de