dive4elements/river: backend/contrib/shpimporter/importer.py annotate

annotate backend/contrib/shpimporter/importer.py @ 8803:080ee1c73bab

Entities will be resolved correctly if placeholder is given. Works since Rev f099f0b22e62 of framework. Theme config is thus found regardless of working dir of server.

author	Tom Gottfried <tom@intevation.de>
date	Thu, 24 Dec 2015 19:05:21 +0100
parents	5aa05a7a34b7
children

rev	line source
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	1 try:
5004 f1c01fecf194 Fix imports of ogr, this removes a warning in gdal 1.9 but keeps Andre Heinecke <aheinecke@intevation.de> parents: 5001 diff changeset	2 from osgeo import ogr, osr
5077 c5187ab9f571 Fix Syntax Errror Andre Heinecke <aheinecke@intevation.de> parents: 5006 diff changeset	3 except ImportError:
5004 f1c01fecf194 Fix imports of ogr, this removes a warning in gdal 1.9 but keeps Andre Heinecke <aheinecke@intevation.de> parents: 5001 diff changeset	4 import ogr, osr
4877 852d4c508f33 (importer) Try to convert paths to UTF-8 for DB insertion Andre Heinecke <aheinecke@intevation.de> parents: 4876 diff changeset	5 import utils
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	6 import re
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	7 import logging
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	8
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	9 logger = logging.getLogger("importer")
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	10
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	11 class Importer:
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	12
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4951 diff changeset	13 def __init__(self, river_id, dbconn, dry_run):
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4951 diff changeset	14 self.river_id = river_id
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	15 self.dbconn = dbconn
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4951 diff changeset	16 self.dry_run = dry_run
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	17 self.dest_srs = osr.SpatialReference()
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4951 diff changeset	18 self.dest_srs.ImportFromEPSG(31467)
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	19 self.handled_fields = []
4885 89a8764cabcc Importer: Only show imported fields if copy_fields is used Andre Heinecke <aheinecke@intevation.de> parents: 4884 diff changeset	20 self.tracking_import = False
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	21 self.srcLayer = None
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	22
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	23 def getKind(self, path):
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	24 raise NotImplementedError("Importer.getKind is abstract!")
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	25
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	26 def getPath(self, base):
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	27 raise NotImplementedError("Importer.getPath is abstract!")
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	28
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	29 def getTablename(self):
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	30 raise NotImplementedError("Importer.getTablename is abstract!")
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	31
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	32 def getName(self):
5160 0dae9e77c5da Importer fix issues reported by pylint Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	33 raise NotImplementedError("Importer.getName is abstract!")
0dae9e77c5da Importer fix issues reported by pylint Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	34
0dae9e77c5da Importer fix issues reported by pylint Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	35 def isGeometryValid(self, geomType):
0dae9e77c5da Importer fix issues reported by pylint Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	36 raise NotImplementedError("Importer.isGeometryValid is abstract!")
0dae9e77c5da Importer fix issues reported by pylint Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	37
0dae9e77c5da Importer fix issues reported by pylint Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	38 def createNewFeature(self, featureDef, feat, **args):
0dae9e77c5da Importer fix issues reported by pylint Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	39 raise NotImplementedError("Importer.createNewFeature is abstract!")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	40
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	41 def IsFieldSet(self, feat, name):
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	42 if not name:
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	43 return False
4878 82d00b0c7302 (importer) Avoid uneccessary error and improve debug output Andre Heinecke <aheinecke@intevation.de> parents: 4877 diff changeset	44 if feat.GetFieldIndex(name) == -1:
82d00b0c7302 (importer) Avoid uneccessary error and improve debug output Andre Heinecke <aheinecke@intevation.de> parents: 4877 diff changeset	45 return False # Avoids an Error in IsFieldSet
82d00b0c7302 (importer) Avoid uneccessary error and improve debug output Andre Heinecke <aheinecke@intevation.de> parents: 4877 diff changeset	46 return feat.IsFieldSet(feat.GetFieldIndex(name))
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	47
5386 6c2751c17869 Importer: Add function to search a value in all fields. Andre Heinecke <aheinecke@intevation.de> parents: 5365 diff changeset	48 def searchValue(self, feat, regex):
6c2751c17869 Importer: Add function to search a value in all fields. Andre Heinecke <aheinecke@intevation.de> parents: 5365 diff changeset	49 """
5401 ff11b178f152 Fix searchValue and mark a field where a string is found as handled Andre Heinecke <aheinecke@intevation.de> parents: 5386 diff changeset	50 Searches for a value that matches regex in all attribute
5386 6c2751c17869 Importer: Add function to search a value in all fields. Andre Heinecke <aheinecke@intevation.de> parents: 5365 diff changeset	51 fields of a feature.
5401 ff11b178f152 Fix searchValue and mark a field where a string is found as handled Andre Heinecke <aheinecke@intevation.de> parents: 5386 diff changeset	52
ff11b178f152 Fix searchValue and mark a field where a string is found as handled Andre Heinecke <aheinecke@intevation.de> parents: 5386 diff changeset	53 @returns the name of the field where a match was found or None
5386 6c2751c17869 Importer: Add function to search a value in all fields. Andre Heinecke <aheinecke@intevation.de> parents: 5365 diff changeset	54 """
6c2751c17869 Importer: Add function to search a value in all fields. Andre Heinecke <aheinecke@intevation.de> parents: 5365 diff changeset	55 for val in feat.items():
5407 db5def5456ce Only match strings vor field values Andre Heinecke <aheinecke@intevation.de> parents: 5401 diff changeset	56 if not isinstance(feat.items()[val], basestring):
db5def5456ce Only match strings vor field values Andre Heinecke <aheinecke@intevation.de> parents: 5401 diff changeset	57 continue
5401 ff11b178f152 Fix searchValue and mark a field where a string is found as handled Andre Heinecke <aheinecke@intevation.de> parents: 5386 diff changeset	58 match = re.match(regex, feat.items()[val], re.IGNORECASE)
ff11b178f152 Fix searchValue and mark a field where a string is found as handled Andre Heinecke <aheinecke@intevation.de> parents: 5386 diff changeset	59 if match:
ff11b178f152 Fix searchValue and mark a field where a string is found as handled Andre Heinecke <aheinecke@intevation.de> parents: 5386 diff changeset	60 return val
5386 6c2751c17869 Importer: Add function to search a value in all fields. Andre Heinecke <aheinecke@intevation.de> parents: 5365 diff changeset	61
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	62 def searchField(self, regex):
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	63 """
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	64 Searches for a field in the current src layer that matches
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	65 the expression regex.
5174 b3a15fa3d88e let import_river.sh parse gauge official numbers, typo Tom Gottfried <tom@intevation.de> parents: 5160 diff changeset	66 Throws an exception if more than one field matches
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	67 @param feat: The feature to search for attributes
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	68 @param regex: The regex to look for
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	69
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	70 @returns: The field name as a string
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	71 """
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	72
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	73 if not hasattr(self.srcLayer, "fieldnames"):
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	74 self.srcLayer.fieldnames = []
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	75 for i in range(0, self.srcLayer.GetLayerDefn().GetFieldCount()):
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	76 self.srcLayer.fieldnames.append(
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	77 self.srcLayer.GetLayerDefn().GetFieldDefn(i).GetNameRef())
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	78
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	79 result = None
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	80 for name in self.srcLayer.fieldnames:
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	81 match = re.match(regex, name, re.IGNORECASE)
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	82 if match:
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	83 if result:
5174 b3a15fa3d88e let import_river.sh parse gauge official numbers, typo Tom Gottfried <tom@intevation.de> parents: 5160 diff changeset	84 raise Exception("More than one field matches: %s" % regex)
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	85 else:
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	86 result = match.group(0)
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	87 return result
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	88
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	89 def IsDoubleFieldSet(self, feat, name):
5365 90f1c5f67698 Importer: Do not return true for IsDoubleField set if the field Andre Heinecke <aheinecke@intevation.de> parents: 5174 diff changeset	90 if not self.IsFieldSet(feat, name):
90f1c5f67698 Importer: Do not return true for IsDoubleField set if the field Andre Heinecke <aheinecke@intevation.de> parents: 5174 diff changeset	91 return False
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	92 try:
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	93 isset = feat.GetFieldAsDouble(name)
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	94 return isset is not None
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	95 except:
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	96 return False
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	97
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	98 def isShapeRelevant(self, name, path):
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	99 return True
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	100
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	101 def walkOverShapes(self, shape):
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	102 (name, path) = shape
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	103
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	104 shp = ogr.Open(shape[1])
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	105 if shp is None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	106 logger.error("Shapefile '%s' could not be opened!" % path)
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	107 return
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	108
4995 998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles Andre Heinecke <aheinecke@intevation.de> parents: 4974 diff changeset	109 if not self.isShapeRelevant(name, path):
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	110 logger.info("Skip shapefile: '%s' of Type: %s" % (path,
4995 998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles Andre Heinecke <aheinecke@intevation.de> parents: 4974 diff changeset	111 utils.getWkbString(shp.GetLayerByName(name).GetGeomType())))
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles Andre Heinecke <aheinecke@intevation.de> parents: 4974 diff changeset	112 return
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles Andre Heinecke <aheinecke@intevation.de> parents: 4974 diff changeset	113
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles Andre Heinecke <aheinecke@intevation.de> parents: 4974 diff changeset	114
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	115 logger.info("Processing shapefile '%s'" % path)
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	116 srcLayer = shp.GetLayerByName(name)
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	117
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	118 if srcLayer is None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	119 logger.error("Layer '%s' was not found!" % name)
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	120 return
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	121
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	122 return self.shape2Database(srcLayer, name, path)
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	123
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	124 def transform(self, feat):
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	125 geometry = feat.GetGeometryRef()
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	126 src_srs = geometry.GetSpatialReference()
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	127
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	128 if src_srs is None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	129 logger.error("No source SRS given! No transformation possible!")
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	130 return feat
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	131
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	132 transformer = osr.CoordinateTransformation(src_srs, self.dest_srs)
4974 a92241d40228 Handle transformation errors Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	133 if geometry.Transform(transformer):
a92241d40228 Handle transformation errors Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	134 return None
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	135
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	136 return feat
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	137
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	138 def handled(self, field):
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	139 """
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	140 Register a field or a map of as handled during the import.
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	141
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	142 There is a warning printed after the import for each unhandled field!
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	143 """
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	144 if not field in self.handled_fields:
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	145 self.handled_fields.append(field)
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	146
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	147 def copyFields(self, src, target, mapping):
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	148 """
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	149 Checks the mapping dictonary for key value pairs to
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	150 copy from the source to the destination feature.
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	151 The keys can be reguar expressions that are matched
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	152 agains the source fieldnames
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	153
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	154 The Key is the attribute of the source feature to be copied
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	155 into the target attribute named by the dict's value.
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	156 """
4885 89a8764cabcc Importer: Only show imported fields if copy_fields is used Andre Heinecke <aheinecke@intevation.de> parents: 4884 diff changeset	157 self.tracking_import = True
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	158 for key, value in mapping.items():
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	159 realname = self.searchField(key)
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	160 if realname == None:
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	161 continue
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	162 if not realname in self.handled_fields:
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	163 self.handled_fields.append(realname)
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	164 # 0 OFTInteger, Simple 32bit integer
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	165 # 1 OFTIntegerList, List of 32bit integers
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	166 # 2 OFTReal, Double Precision floating point
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	167 # 3 OFTRealList, List of doubles
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	168 # 4 OFTString, String of ASCII chars
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	169 # 5 OFTStringList, Array of strings
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	170 # 6 OFTWideString, deprecated
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	171 # 7 OFTWideStringList, deprecated
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	172 # 8 OFTBinary, Raw Binary data
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	173 # 9 OFTDate, Date
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	174 # 10 OFTTime, Time
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	175 # 11 OFTDateTime, Date and Time
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	176 if src.IsFieldSet(src.GetFieldIndex(realname)):
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	177 if src.GetFieldType(realname) == 2:
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	178 target.SetField(value, src.GetFieldAsDouble(realname))
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	179 else:
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	180 target.SetField(value, utils.getUTF8(src.GetField(realname)))
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	181
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	182 def shape2Database(self, srcLayer, name, path):
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	183 destLayer = self.dbconn.GetLayerByName(self.getTablename())
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	184
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	185 if srcLayer is None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	186 logger.error("Shapefile is None!")
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	187 return -1
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	188
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	189 if destLayer is None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	190 logger.error("No destination layer given!")
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	191 return -1
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	192
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	193 count = srcLayer.GetFeatureCount()
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	194 logger.debug("Try to add %i features to database." % count)
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	195
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	196 srcLayer.ResetReading()
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	197 self.srcLayer = srcLayer
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	198
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	199 geomType = -1
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	200 success = 0
4995 998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles Andre Heinecke <aheinecke@intevation.de> parents: 4974 diff changeset	201 unsupported = {}
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	202 creationFailed = 0
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	203 featureDef = destLayer.GetLayerDefn()
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	204
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	205 for feat in srcLayer:
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	206 geom = feat.GetGeometryRef()
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	207
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	208 if geom is None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	209 logger.debug("Unkown Geometry reference for feature")
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	210 continue
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	211
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	212 geomType = geom.GetGeometryType()
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	213
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	214 if self.isGeometryValid(geomType):
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	215 newFeat = self.createNewFeature(featureDef,
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	216 feat,
4951 23a090b41543 Importer: Check lowercase pathname and handle utf-8 conversion Andre Heinecke <aheinecke@intevation.de> parents: 4935 diff changeset	217 name=utils.getUTF8(name),
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	218 path=path)
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	219
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	220 if newFeat is not None:
4877 852d4c508f33 (importer) Try to convert paths to UTF-8 for DB insertion Andre Heinecke <aheinecke@intevation.de> parents: 4876 diff changeset	221 newFeat.SetField("path", utils.getUTF8Path(path))
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	222 newFeat = self.transform(newFeat)
4974 a92241d40228 Handle transformation errors Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	223 if newFeat:
a92241d40228 Handle transformation errors Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	224 res = destLayer.CreateFeature(newFeat)
a92241d40228 Handle transformation errors Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	225 if res is None or res > 0:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	226 logger.error("Unable to insert feature. Error: %r" % res)
4974 a92241d40228 Handle transformation errors Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	227 else:
a92241d40228 Handle transformation errors Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	228 success = success + 1
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	229 else:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	230 logger.error("Could not transform feature: %s " % feat.GetFID())
4974 a92241d40228 Handle transformation errors Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	231 creationFailed += 1
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	232 else:
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	233 creationFailed = creationFailed + 1
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	234 else:
4995 998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles Andre Heinecke <aheinecke@intevation.de> parents: 4974 diff changeset	235 unsupported[utils.getWkbString(geomType)] = \
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles Andre Heinecke <aheinecke@intevation.de> parents: 4974 diff changeset	236 unsupported.get(utils.getWkbString(geomType), 0) + 1
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	237
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	238 logger.info("Inserted %i features" % success)
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	239 logger.info("Failed to create %i features" % creationFailed)
5001 fed74592a2ba Fix syntax error Andre Heinecke <aheinecke@intevation.de> parents: 4995 diff changeset	240 for key, value in unsupported.items():
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	241 logger.info("Found %i unsupported features of type: %s" % (value, key))
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	242
4886 8e99d2d9364d Importer: When not tracking dont print info about unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4885 diff changeset	243 if self.tracking_import:
8e99d2d9364d Importer: When not tracking dont print info about unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4885 diff changeset	244 unhandled = []
8e99d2d9364d Importer: When not tracking dont print info about unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4885 diff changeset	245 for i in range(0, srcLayer.GetLayerDefn().GetFieldCount()):
8e99d2d9364d Importer: When not tracking dont print info about unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4885 diff changeset	246 act_field = srcLayer.GetLayerDefn().GetFieldDefn(i).GetNameRef()
8e99d2d9364d Importer: When not tracking dont print info about unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4885 diff changeset	247 if not act_field in self.handled_fields:
8e99d2d9364d Importer: When not tracking dont print info about unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4885 diff changeset	248 unhandled.append(act_field)
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	249
4886 8e99d2d9364d Importer: When not tracking dont print info about unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4885 diff changeset	250 if len(unhandled):
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	251 logger.info("Did not import values from fields: %s " % \
4886 8e99d2d9364d Importer: When not tracking dont print info about unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4885 diff changeset	252 " ".join(unhandled))
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	253
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	254 try:
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4951 diff changeset	255 if self.dry_run:
3655 8654e4480fc6 Shape importer: added command line option dry_run to supress database transactions. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 3654 diff changeset	256 return geomType
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	257 destLayer.CommitTransaction()
5160 0dae9e77c5da Importer fix issues reported by pylint Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	258 except:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	259 logger.error("Exception while committing transaction.")
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	260
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	261 return geomType

Mercurial > dive4elements > river

annotate backend/contrib/shpimporter/importer.py @ 8803:080ee1c73bab