dive4elements/river: backend/contrib/shpimporter/shpimporter.py annotate

annotate backend/contrib/shpimporter/shpimporter.py @ 6061:e9a76ffa0f9a

Use maxOverlap to get the correct gauge for the MainValues Previously just the first matching gauge was taken even if it's range ended with the minimum value. This code is clearly intended to get one gauge for one range so the best match should be taken.

author	Andre Heinecke <aheinecke@intevation.de>
date	Wed, 22 May 2013 18:10:48 +0200
parents	5aa05a7a34b7
children	e68e414dceb2

rev	line source
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	1 try:
a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	2 from osgeo import ogr
4977 ee908e4946cd corrected Typo Tom Gottfried <tom.gottfried@intevation.de> parents: 4975 diff changeset	3 except ImportError:
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	4 import ogr
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	5
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	6 import utils, optparse
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	7 import sys
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	8 import os
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	9 import logging
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	10
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	11 from uesg import UESG
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	12 from axis import Axis
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	13 from km import KM
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	14 from fixpoints import Fixpoint
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	15 from buildings import Building
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	16 from crosssectiontracks import CrosssectionTrack
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	17 from floodplains import Floodplain
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	18 from boundaries import HydrBoundary, HydrBoundaryPoly
4879 63f82bdd45f8 (importer) Enable hws_* importers Andre Heinecke <aheinecke@intevation.de> parents: 4872 diff changeset	19 from hws import HWSLines, HWSPoints
2862 998f1a7dcfde New shape importers and schema modifications. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2861 diff changeset	20 from gauges import GaugeLocation
5353 d086ce6c13d3 Add jetties importer Andre Heinecke <aheinecke@intevation.de> parents: 5221 diff changeset	21 from jetties import Jetties
4975 9eea3cd22ee7 Add dgm file reader although an awk script should have been more appropiate Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	22 from dgm import insertRiverDgm
5545 ed149d5d7fb7 Add floodmarks importer Andre Heinecke <aheinecke@intevation.de> parents: 5353 diff changeset	23 from floodmarks import Floodmark
2798 5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	24
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	25 logger = logging.getLogger("shpimporter")
2798 5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	26
5219 40dc001594e4 Allow unicode river names Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	27 os.environ["NLS_LANG"] = ".AL32UTF8"
40dc001594e4 Allow unicode river names Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	28
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	29 def initialize_logging(level):
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	30 """Initializes the logging system"""
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	31 root = logging.getLogger()
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	32 root.setLevel(level)
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	33 hdlr = logging.StreamHandler()
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	34 fmt = logging.Formatter("%(levelname)s %(name)s: %(message)s")
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	35 hdlr.setFormatter(fmt)
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	36 root.addHandler(hdlr)
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	37
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	38 def getImporters(river_id, dbconn, dry_run):
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	39 return [
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	40 Axis(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	41 KM(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	42 CrosssectionTrack(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	43 Fixpoint(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	44 Building(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	45 Floodplain(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	46 HydrBoundary(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	47 HydrBoundaryPoly(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	48 HWSLines(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	49 HWSPoints(river_id, dbconn, dry_run),
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	50 GaugeLocation(river_id, dbconn, dry_run),
5353 d086ce6c13d3 Add jetties importer Andre Heinecke <aheinecke@intevation.de> parents: 5221 diff changeset	51 Jetties(river_id, dbconn, dry_run),
5545 ed149d5d7fb7 Add floodmarks importer Andre Heinecke <aheinecke@intevation.de> parents: 5353 diff changeset	52 Floodmark(river_id, dbconn, dry_run),
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	53 UESG(river_id, dbconn, dry_run)
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	54 ]
2798 5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	55
5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	56
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	57 def getConfig():
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	58 parser = optparse.OptionParser()
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	59 parser.add_option("--directory", type="string")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	60 parser.add_option("--target_srs", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	61 parser.add_option("--host", type="string")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	62 parser.add_option("--user", type="string")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	63 parser.add_option("--password", type="string")
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	64 parser.add_option("--river_name", type="string")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	65 parser.add_option("--verbose", type="int", default=1)
3655 8654e4480fc6 Shape importer: added command line option dry_run to supress database transactions. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 3654 diff changeset	66 parser.add_option("--dry_run", type="int", default=0)
4871 890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr Andre Heinecke <aheinecke@intevation.de> parents: 3655 diff changeset	67 parser.add_option("--ogr_connection", type="string")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	68 parser.add_option("--skip_axis", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	69 parser.add_option("--skip_hydr_boundaries", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	70 parser.add_option("--skip_buildings", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	71 parser.add_option("--skip_crosssections", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	72 parser.add_option("--skip_fixpoints", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	73 parser.add_option("--skip_floodplains", type="int")
4879 63f82bdd45f8 (importer) Enable hws_* importers Andre Heinecke <aheinecke@intevation.de> parents: 4872 diff changeset	74 parser.add_option("--skip_hws_lines", type="int")
63f82bdd45f8 (importer) Enable hws_* importers Andre Heinecke <aheinecke@intevation.de> parents: 4872 diff changeset	75 parser.add_option("--skip_hws_points", type="int")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	76 parser.add_option("--skip_gauge_locations", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	77 parser.add_option("--skip_kms", type="int")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	78 parser.add_option("--skip_uesgs", type="int")
4975 9eea3cd22ee7 Add dgm file reader although an awk script should have been more appropiate Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	79 parser.add_option("--skip_dgm", type="int")
5353 d086ce6c13d3 Add jetties importer Andre Heinecke <aheinecke@intevation.de> parents: 5221 diff changeset	80 parser.add_option("--skip_jetties", type="int")
5545 ed149d5d7fb7 Add floodmarks importer Andre Heinecke <aheinecke@intevation.de> parents: 5353 diff changeset	81 parser.add_option("--skip_floodmarks", type="int")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	82 (config, args) = parser.parse_args()
2798 5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	83
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	84 if config.verbose > 1:
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	85 initialize_logging(logging.DEBUG)
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	86 elif config.verbose == 1:
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	87 initialize_logging(logging.INFO)
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	88 else:
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	89 initialize_logging(logging.WARN)
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	90
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	91 if config.directory == None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	92 logger.error("No river directory specified!")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	93 raise Exception("Invalid config")
4871 890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr Andre Heinecke <aheinecke@intevation.de> parents: 3655 diff changeset	94 if not config.ogr_connection:
890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr Andre Heinecke <aheinecke@intevation.de> parents: 3655 diff changeset	95 if not config.host:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	96 logger.error("No database host specified!")
4871 890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr Andre Heinecke <aheinecke@intevation.de> parents: 3655 diff changeset	97 raise Exception("Invalid config")
890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr Andre Heinecke <aheinecke@intevation.de> parents: 3655 diff changeset	98 if not config.user:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	99 logger.error("No databaser user specified!")
4871 890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr Andre Heinecke <aheinecke@intevation.de> parents: 3655 diff changeset	100 raise Exception("Invalid config")
890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr Andre Heinecke <aheinecke@intevation.de> parents: 3655 diff changeset	101 if not config.password:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	102 logger.error("No password specified!")
4871 890eaa0a5162 Add ogr_connection parameter to allow directly setting the ogr Andre Heinecke <aheinecke@intevation.de> parents: 3655 diff changeset	103 raise Exception("Invalid config")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	104
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	105 return config
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	106
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	107
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	108 def skip_importer(config, importer):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	109 if config.skip_axis == 1 and isinstance(importer, Axis):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	110 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	111 elif config.skip_hydr_boundaries == 1 and isinstance(importer, HydrBoundary):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	112 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	113 elif config.skip_hydr_boundaries == 1 and isinstance(importer, HydrBoundaryPoly):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	114 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	115 elif config.skip_buildings == 1 and isinstance(importer, Building):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	116 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	117 elif config.skip_crosssections == 1 and isinstance(importer, CrosssectionTrack):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	118 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	119 elif config.skip_fixpoints == 1 and isinstance(importer, Fixpoint):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	120 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	121 elif config.skip_floodplains == 1 and isinstance(importer, Floodplain):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	122 return True
4963 1469066cc7d9 Importer: Fix the order of the config check Andre Heinecke <aheinecke@intevation.de> parents: 4955 diff changeset	123 elif config.skip_hws_lines == 1 and isinstance(importer, HWSLines):
1469066cc7d9 Importer: Fix the order of the config check Andre Heinecke <aheinecke@intevation.de> parents: 4955 diff changeset	124 return True
5011 514948fdae54 Do not skip hws_lines when skip hws_points is true Andre Heinecke <aheinecke@intevation.de> parents: 5006 diff changeset	125 elif config.skip_hws_points == 1 and isinstance(importer, HWSPoints) and \
514948fdae54 Do not skip hws_lines when skip hws_points is true Andre Heinecke <aheinecke@intevation.de> parents: 5006 diff changeset	126 not isinstance(importer, HWSLines):
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	127 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	128 elif config.skip_gauge_locations == 1 and isinstance(importer, GaugeLocation):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	129 return True
5353 d086ce6c13d3 Add jetties importer Andre Heinecke <aheinecke@intevation.de> parents: 5221 diff changeset	130 elif config.skip_jetties == 1 and isinstance(importer, Jetties):
d086ce6c13d3 Add jetties importer Andre Heinecke <aheinecke@intevation.de> parents: 5221 diff changeset	131 return True
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	132 elif config.skip_kms == 1 and isinstance(importer, KM):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	133 return True
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	134 elif config.skip_uesgs == 1 and isinstance(importer, UESG):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	135 return True
5545 ed149d5d7fb7 Add floodmarks importer Andre Heinecke <aheinecke@intevation.de> parents: 5353 diff changeset	136 elif config.skip_floodmarks == 1 and isinstance(importer, Floodmark):
ed149d5d7fb7 Add floodmarks importer Andre Heinecke <aheinecke@intevation.de> parents: 5353 diff changeset	137 return True
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	138
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	139 return False
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	140
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	141 def main():
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	142 config=None
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	143 try:
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	144 config = getConfig()
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	145 except:
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	146 return -1
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	147
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	148 if config == None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	149 logger.error("Unable to read config from command line!")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	150 return
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	151
3655 8654e4480fc6 Shape importer: added command line option dry_run to supress database transactions. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 3654 diff changeset	152 if config.dry_run > 0:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	153 logger.info("You enable 'dry_run'. No database transaction will take place!")
3655 8654e4480fc6 Shape importer: added command line option dry_run to supress database transactions. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 3654 diff changeset	154
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	155 if config.ogr_connection:
4879 63f82bdd45f8 (importer) Enable hws_* importers Andre Heinecke <aheinecke@intevation.de> parents: 4872 diff changeset	156 connstr = config.ogr_connection
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	157 else:
4879 63f82bdd45f8 (importer) Enable hws_* importers Andre Heinecke <aheinecke@intevation.de> parents: 4872 diff changeset	158 connstr = 'OCI:%s/%s@%s' % (config.user, config.password, config.host)
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	159
5036 d6860fca89e4 Pull information about the backend into dgm. Andre Heinecke <aheinecke@intevation.de> parents: 5028 diff changeset	160 oracle = False # Marker if oracle is used.
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	161 if 'OCI:' in connstr:
5036 d6860fca89e4 Pull information about the backend into dgm. Andre Heinecke <aheinecke@intevation.de> parents: 5028 diff changeset	162 oracle = True
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	163 try:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	164 import cx_Oracle as dbapi
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	165 raw_connstr=connstr.replace("OCI:", "")
5077 c5187ab9f571 Fix Syntax Errror Andre Heinecke <aheinecke@intevation.de> parents: 5036 diff changeset	166 except ImportError:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	167 logger.error("Module cx_Oracle not found in: %s\n"
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	168 "Neccessary to connect to a Oracle Database.\n"
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	169 "Please refer to the installation "
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	170 "documentation." % sys.path)
5036 d6860fca89e4 Pull information about the backend into dgm. Andre Heinecke <aheinecke@intevation.de> parents: 5028 diff changeset	171 return -1
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	172
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	173 else: # Currently only support for oracle and postgres
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	174 try:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	175 import psycopg2 as dbapi
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	176 raw_connstr=connstr.replace("PG:", "")
4977 ee908e4946cd corrected Typo Tom Gottfried <tom.gottfried@intevation.de> parents: 4975 diff changeset	177 except ImportError:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	178 logger.error("Module psycopg2 not found in: %s\n"
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	179 "Neccessary to connect to a Posgresql Database.\n"
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	180 "Please refer to the installation "
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	181 "documentation." % sys.path)
5036 d6860fca89e4 Pull information about the backend into dgm. Andre Heinecke <aheinecke@intevation.de> parents: 5028 diff changeset	182 return -1
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	183
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	184 dbconn_raw = dbapi.connect(raw_connstr)
4879 63f82bdd45f8 (importer) Enable hws_* importers Andre Heinecke <aheinecke@intevation.de> parents: 4872 diff changeset	185 dbconn = ogr.Open(connstr)
63f82bdd45f8 (importer) Enable hws_* importers Andre Heinecke <aheinecke@intevation.de> parents: 4872 diff changeset	186
63f82bdd45f8 (importer) Enable hws_* importers Andre Heinecke <aheinecke@intevation.de> parents: 4872 diff changeset	187 if dbconn == None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	188 logger.error("Could not connect to database %s" % connstr)
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	189 return -1
a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	190
2798 5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	191 types = {}
5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	192
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	193 directories = []
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	194 if not config.river_name:
5005 eb2d6609387c Importer: Fix directory paths and river name encoding Andre Heinecke <aheinecke@intevation.de> parents: 4998 diff changeset	195 for file in [os.path.join(config.directory, d) for d in \
eb2d6609387c Importer: Fix directory paths and river name encoding Andre Heinecke <aheinecke@intevation.de> parents: 4998 diff changeset	196 os.listdir(config.directory)]:
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	197 if os.path.isdir(file):
5005 eb2d6609387c Importer: Fix directory paths and river name encoding Andre Heinecke <aheinecke@intevation.de> parents: 4998 diff changeset	198 directories.append(file)
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	199 else:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	200 directories.append(config.directory)
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	201
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	202 for directory in directories:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	203 if not config.river_name:
5005 eb2d6609387c Importer: Fix directory paths and river name encoding Andre Heinecke <aheinecke@intevation.de> parents: 4998 diff changeset	204 river_name = utils.getUTF8Path(
eb2d6609387c Importer: Fix directory paths and river name encoding Andre Heinecke <aheinecke@intevation.de> parents: 4998 diff changeset	205 os.path.basename(os.path.normpath(directory)))
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	206 else:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	207 river_name = config.river_name
5036 d6860fca89e4 Pull information about the backend into dgm. Andre Heinecke <aheinecke@intevation.de> parents: 5028 diff changeset	208 river_id = utils.getRiverId(dbconn_raw, river_name, oracle)
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2798 diff changeset	209
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	210 if not river_id:
5221 aa043641d9c8 Fix syntax error when river id is not found (pylint could have told me this) Andre Heinecke <aheinecke@intevation.de> parents: 5219 diff changeset	211 logger.info(u"Could not find river in database. Skipping: %s"
aa043641d9c8 Fix syntax error when river id is not found (pylint could have told me this) Andre Heinecke <aheinecke@intevation.de> parents: 5219 diff changeset	212 % unicode(utils.getUTF8(river_name), "UTF-8"))
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	213 continue
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	214 else:
5219 40dc001594e4 Allow unicode river names Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	215 logger.info(u"Importing River: %s" % unicode(
40dc001594e4 Allow unicode river names Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	216 utils.getUTF8(river_name), "UTF-8"))
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	217
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	218 for importer in getImporters(river_id, dbconn, config.dry_run):
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	219 if skip_importer(config, importer):
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	220 logger.info("Skip import of '%s'" % importer.getName())
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	221 continue
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	222
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	223 logger.info("Start import of '%s'" % importer.getName())
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	224
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	225 shapes = utils.findShapefiles(importer.getPath(config.directory))
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	226 logger.debug("Found %i Shapefiles" % len(shapes))
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	227
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	228 for shpTuple in shapes:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	229 geomType = importer.walkOverShapes(shpTuple)
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	230 try:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	231 if geomType is not None:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	232 num = types[geomType]
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	233 types[geomType] = num+1
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	234 except:
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4963 diff changeset	235 types[geomType] = 1
2798 5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	236
4975 9eea3cd22ee7 Add dgm file reader although an awk script should have been more appropiate Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	237 for key in types:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	238 logger.debug("%i x geometry type %s" % (types[key], key))
2798 5a654f2e35bc Added a python tool to import shapefiles into database. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	239
4975 9eea3cd22ee7 Add dgm file reader although an awk script should have been more appropiate Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	240 if not config.skip_dgm:
9eea3cd22ee7 Add dgm file reader although an awk script should have been more appropiate Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	241 dgmfilename = os.path.join(
9eea3cd22ee7 Add dgm file reader although an awk script should have been more appropiate Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	242 config.directory, "..", "DGMs.csv")
4989 f4d61ac84b76 If we bother to check if the file is readable we might also Andre Heinecke <aheinecke@intevation.de> parents: 4978 diff changeset	243 if not os.access(dgmfilename, os.R_OK) or not \
f4d61ac84b76 If we bother to check if the file is readable we might also Andre Heinecke <aheinecke@intevation.de> parents: 4978 diff changeset	244 os.path.isfile(dgmfilename):
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	245 logger.info("Could not find or access DGM file: %s \n"
4975 9eea3cd22ee7 Add dgm file reader although an awk script should have been more appropiate Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	246 "Skipping DGM import." % dgmfilename)
4978 b5ac7588cf4d Use os.access instead of try/except open Andre Heinecke <aheinecke@intevation.de> parents: 4977 diff changeset	247 else:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5005 diff changeset	248 logger.info("Inserting DGM meta information in 'dem' table.")
5036 d6860fca89e4 Pull information about the backend into dgm. Andre Heinecke <aheinecke@intevation.de> parents: 5028 diff changeset	249 insertRiverDgm(dbconn_raw, dgmfilename, river_name,
d6860fca89e4 Pull information about the backend into dgm. Andre Heinecke <aheinecke@intevation.de> parents: 5028 diff changeset	250 config.dry_run, oracle)
5012 a79f6a085ab6 Add message if DGM import is skipped Andre Heinecke <aheinecke@intevation.de> parents: 5011 diff changeset	251 else:
a79f6a085ab6 Add message if DGM import is skipped Andre Heinecke <aheinecke@intevation.de> parents: 5011 diff changeset	252 logger.info("Skip import of DGM.")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	253
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	254 if __name__ == '__main__':
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	255 main()

Mercurial > dive4elements > river

annotate backend/contrib/shpimporter/shpimporter.py @ 6061:e9a76ffa0f9a