dive4elements/river: backend/contrib/shpimporter/importer.py annotate

annotate backend/contrib/shpimporter/importer.py @ 7602:c50dbbe17950

issue1596: Store table (cell) data twice: Once as (formatted) string as coming from server, once transformed into float (or string). The benefit is that now we can sort table data numerically, while keeping the formatted and i18ned display of values.

author	Felix Wolfsteller <felix.wolfsteller@intevation.de>
date	Wed, 27 Nov 2013 14:55:25 +0100
parents	5aa05a7a34b7
children

rev	line source
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	1 try:
5004 f1c01fecf194 Fix imports of ogr, this removes a warning in gdal 1.9 but keeps Andre Heinecke <aheinecke@intevation.de> parents: 5001 diff changeset	2 from osgeo import ogr, osr
5077 c5187ab9f571 Fix Syntax Errror Andre Heinecke <aheinecke@intevation.de> parents: 5006 diff changeset	3 except ImportError:
5004 f1c01fecf194 Fix imports of ogr, this removes a warning in gdal 1.9 but keeps Andre Heinecke <aheinecke@intevation.de> parents: 5001 diff changeset	4 import ogr, osr
4877 852d4c508f33 (importer) Try to convert paths to UTF-8 for DB insertion Andre Heinecke <aheinecke@intevation.de> parents: 4876 diff changeset	5 import utils
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	6 import re
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	7 import logging
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	8
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	9 logger = logging.getLogger("importer")
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	10
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	11 class Importer:
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	12
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4951 diff changeset	13 def __init__(self, river_id, dbconn, dry_run):
174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4951 diff changeset	14 self.river_id = river_id
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	15 self.dbconn = dbconn
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4951 diff changeset	16 self.dry_run = dry_run
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	17 self.dest_srs = osr.SpatialReference()
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4951 diff changeset	18 self.dest_srs.ImportFromEPSG(31467)
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	19 self.handled_fields = []
4885 89a8764cabcc Importer: Only show imported fields if copy_fields is used Andre Heinecke <aheinecke@intevation.de> parents: 4884 diff changeset	20 self.tracking_import = False
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	21 self.srcLayer = None
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	22
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	23 def getKind(self, path):
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	24 raise NotImplementedError("Importer.getKind is abstract!")
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	25
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	26 def getPath(self, base):
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	27 raise NotImplementedError("Importer.getPath is abstract!")
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	28
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	29 def getTablename(self):
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	30 raise NotImplementedError("Importer.getTablename is abstract!")
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	31
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	32 def getName(self):
5160 0dae9e77c5da Importer fix issues reported by pylint Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	33 raise NotImplementedError("Importer.getName is abstract!")
0dae9e77c5da Importer fix issues reported by pylint Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	34
0dae9e77c5da Importer fix issues reported by pylint Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	35 def isGeometryValid(self, geomType):
0dae9e77c5da Importer fix issues reported by pylint Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	36 raise NotImplementedError("Importer.isGeometryValid is abstract!")
0dae9e77c5da Importer fix issues reported by pylint Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	37
0dae9e77c5da Importer fix issues reported by pylint Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	38 def createNewFeature(self, featureDef, feat, **args):
0dae9e77c5da Importer fix issues reported by pylint Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	39 raise NotImplementedError("Importer.createNewFeature is abstract!")
3654 59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2862 diff changeset	40
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	41 def IsFieldSet(self, feat, name):
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	42 if not name:
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	43 return False
4878 82d00b0c7302 (importer) Avoid uneccessary error and improve debug output Andre Heinecke <aheinecke@intevation.de> parents: 4877 diff changeset	44 if feat.GetFieldIndex(name) == -1:
82d00b0c7302 (importer) Avoid uneccessary error and improve debug output Andre Heinecke <aheinecke@intevation.de> parents: 4877 diff changeset	45 return False # Avoids an Error in IsFieldSet
82d00b0c7302 (importer) Avoid uneccessary error and improve debug output Andre Heinecke <aheinecke@intevation.de> parents: 4877 diff changeset	46 return feat.IsFieldSet(feat.GetFieldIndex(name))
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	47
5386 6c2751c17869 Importer: Add function to search a value in all fields. Andre Heinecke <aheinecke@intevation.de> parents: 5365 diff changeset	48 def searchValue(self, feat, regex):
6c2751c17869 Importer: Add function to search a value in all fields. Andre Heinecke <aheinecke@intevation.de> parents: 5365 diff changeset	49 """
5401 ff11b178f152 Fix searchValue and mark a field where a string is found as handled Andre Heinecke <aheinecke@intevation.de> parents: 5386 diff changeset	50 Searches for a value that matches regex in all attribute
5386 6c2751c17869 Importer: Add function to search a value in all fields. Andre Heinecke <aheinecke@intevation.de> parents: 5365 diff changeset	51 fields of a feature.
5401 ff11b178f152 Fix searchValue and mark a field where a string is found as handled Andre Heinecke <aheinecke@intevation.de> parents: 5386 diff changeset	52
ff11b178f152 Fix searchValue and mark a field where a string is found as handled Andre Heinecke <aheinecke@intevation.de> parents: 5386 diff changeset	53 @returns the name of the field where a match was found or None
5386 6c2751c17869 Importer: Add function to search a value in all fields. Andre Heinecke <aheinecke@intevation.de> parents: 5365 diff changeset	54 """
6c2751c17869 Importer: Add function to search a value in all fields. Andre Heinecke <aheinecke@intevation.de> parents: 5365 diff changeset	55 for val in feat.items():
5407 db5def5456ce Only match strings vor field values Andre Heinecke <aheinecke@intevation.de> parents: 5401 diff changeset	56 if not isinstance(feat.items()[val], basestring):
db5def5456ce Only match strings vor field values Andre Heinecke <aheinecke@intevation.de> parents: 5401 diff changeset	57 continue
5401 ff11b178f152 Fix searchValue and mark a field where a string is found as handled Andre Heinecke <aheinecke@intevation.de> parents: 5386 diff changeset	58 match = re.match(regex, feat.items()[val], re.IGNORECASE)
ff11b178f152 Fix searchValue and mark a field where a string is found as handled Andre Heinecke <aheinecke@intevation.de> parents: 5386 diff changeset	59 if match:
ff11b178f152 Fix searchValue and mark a field where a string is found as handled Andre Heinecke <aheinecke@intevation.de> parents: 5386 diff changeset	60 return val
5386 6c2751c17869 Importer: Add function to search a value in all fields. Andre Heinecke <aheinecke@intevation.de> parents: 5365 diff changeset	61
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	62 def searchField(self, regex):
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	63 """
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	64 Searches for a field in the current src layer that matches
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	65 the expression regex.
5174 b3a15fa3d88e let import_river.sh parse gauge official numbers, typo Tom Gottfried <tom@intevation.de> parents: 5160 diff changeset	66 Throws an exception if more than one field matches
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	67 @param feat: The feature to search for attributes
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	68 @param regex: The regex to look for
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	69
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	70 @returns: The field name as a string
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	71 """
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	72
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	73 if not hasattr(self.srcLayer, "fieldnames"):
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	74 self.srcLayer.fieldnames = []
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	75 for i in range(0, self.srcLayer.GetLayerDefn().GetFieldCount()):
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	76 self.srcLayer.fieldnames.append(
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	77 self.srcLayer.GetLayerDefn().GetFieldDefn(i).GetNameRef())
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	78
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	79 result = None
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	80 for name in self.srcLayer.fieldnames:
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	81 match = re.match(regex, name, re.IGNORECASE)
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	82 if match:
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	83 if result:
5174 b3a15fa3d88e let import_river.sh parse gauge official numbers, typo Tom Gottfried <tom@intevation.de> parents: 5160 diff changeset	84 raise Exception("More than one field matches: %s" % regex)
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	85 else:
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	86 result = match.group(0)
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	87 return result
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	88
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	89 def IsDoubleFieldSet(self, feat, name):
5365 90f1c5f67698 Importer: Do not return true for IsDoubleField set if the field Andre Heinecke <aheinecke@intevation.de> parents: 5174 diff changeset	90 if not self.IsFieldSet(feat, name):
90f1c5f67698 Importer: Do not return true for IsDoubleField set if the field Andre Heinecke <aheinecke@intevation.de> parents: 5174 diff changeset	91 return False
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	92 try:
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	93 isset = feat.GetFieldAsDouble(name)
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	94 return isset is not None
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	95 except:
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	96 return False
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	97
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	98 def isShapeRelevant(self, name, path):
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	99 return True
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	100
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	101 def walkOverShapes(self, shape):
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	102 (name, path) = shape
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	103
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	104 shp = ogr.Open(shape[1])
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	105 if shp is None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	106 logger.error("Shapefile '%s' could not be opened!" % path)
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	107 return
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	108
4995 998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles Andre Heinecke <aheinecke@intevation.de> parents: 4974 diff changeset	109 if not self.isShapeRelevant(name, path):
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	110 logger.info("Skip shapefile: '%s' of Type: %s" % (path,
4995 998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles Andre Heinecke <aheinecke@intevation.de> parents: 4974 diff changeset	111 utils.getWkbString(shp.GetLayerByName(name).GetGeomType())))
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles Andre Heinecke <aheinecke@intevation.de> parents: 4974 diff changeset	112 return
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles Andre Heinecke <aheinecke@intevation.de> parents: 4974 diff changeset	113
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles Andre Heinecke <aheinecke@intevation.de> parents: 4974 diff changeset	114
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	115 logger.info("Processing shapefile '%s'" % path)
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	116 srcLayer = shp.GetLayerByName(name)
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	117
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	118 if srcLayer is None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	119 logger.error("Layer '%s' was not found!" % name)
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	120 return
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	121
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	122 return self.shape2Database(srcLayer, name, path)
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	123
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	124 def transform(self, feat):
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	125 geometry = feat.GetGeometryRef()
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	126 src_srs = geometry.GetSpatialReference()
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	127
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	128 if src_srs is None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	129 logger.error("No source SRS given! No transformation possible!")
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	130 return feat
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	131
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	132 transformer = osr.CoordinateTransformation(src_srs, self.dest_srs)
4974 a92241d40228 Handle transformation errors Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	133 if geometry.Transform(transformer):
a92241d40228 Handle transformation errors Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	134 return None
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	135
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	136 return feat
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	137
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	138 def handled(self, field):
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	139 """
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	140 Register a field or a map of as handled during the import.
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	141
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	142 There is a warning printed after the import for each unhandled field!
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	143 """
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	144 if not field in self.handled_fields:
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	145 self.handled_fields.append(field)
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	146
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	147 def copyFields(self, src, target, mapping):
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	148 """
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	149 Checks the mapping dictonary for key value pairs to
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	150 copy from the source to the destination feature.
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	151 The keys can be reguar expressions that are matched
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	152 agains the source fieldnames
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	153
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	154 The Key is the attribute of the source feature to be copied
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	155 into the target attribute named by the dict's value.
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	156 """
4885 89a8764cabcc Importer: Only show imported fields if copy_fields is used Andre Heinecke <aheinecke@intevation.de> parents: 4884 diff changeset	157 self.tracking_import = True
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	158 for key, value in mapping.items():
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	159 realname = self.searchField(key)
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	160 if realname == None:
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	161 continue
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	162 if not realname in self.handled_fields:
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	163 self.handled_fields.append(realname)
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	164 # 0 OFTInteger, Simple 32bit integer
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	165 # 1 OFTIntegerList, List of 32bit integers
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	166 # 2 OFTReal, Double Precision floating point
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	167 # 3 OFTRealList, List of doubles
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	168 # 4 OFTString, String of ASCII chars
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	169 # 5 OFTStringList, Array of strings
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	170 # 6 OFTWideString, deprecated
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	171 # 7 OFTWideStringList, deprecated
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	172 # 8 OFTBinary, Raw Binary data
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	173 # 9 OFTDate, Date
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	174 # 10 OFTTime, Time
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	175 # 11 OFTDateTime, Date and Time
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	176 if src.IsFieldSet(src.GetFieldIndex(realname)):
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	177 if src.GetFieldType(realname) == 2:
c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	178 target.SetField(value, src.GetFieldAsDouble(realname))
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	179 else:
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	180 target.SetField(value, utils.getUTF8(src.GetField(realname)))
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	181
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	182 def shape2Database(self, srcLayer, name, path):
4872 a563e9f58f93 Improve error handling and unify dbconn for all importers Andre Heinecke <aheinecke@intevation.de> parents: 4871 diff changeset	183 destLayer = self.dbconn.GetLayerByName(self.getTablename())
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	184
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	185 if srcLayer is None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	186 logger.error("Shapefile is None!")
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	187 return -1
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	188
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	189 if destLayer is None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	190 logger.error("No destination layer given!")
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	191 return -1
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	192
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	193 count = srcLayer.GetFeatureCount()
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	194 logger.debug("Try to add %i features to database." % count)
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	195
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	196 srcLayer.ResetReading()
4935 c0a58558b817 Importer: - Handle regular expressions for attribute names Andre Heinecke <aheinecke@intevation.de> parents: 4886 diff changeset	197 self.srcLayer = srcLayer
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	198
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	199 geomType = -1
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	200 success = 0
4995 998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles Andre Heinecke <aheinecke@intevation.de> parents: 4974 diff changeset	201 unsupported = {}
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	202 creationFailed = 0
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	203 featureDef = destLayer.GetLayerDefn()
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	204
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	205 for feat in srcLayer:
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	206 geom = feat.GetGeometryRef()
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	207
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	208 if geom is None:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	209 logger.debug("Unkown Geometry reference for feature")
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	210 continue
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	211
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	212 geomType = geom.GetGeometryType()
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	213
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	214 if self.isGeometryValid(geomType):
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	215 newFeat = self.createNewFeature(featureDef,
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	216 feat,
4951 23a090b41543 Importer: Check lowercase pathname and handle utf-8 conversion Andre Heinecke <aheinecke@intevation.de> parents: 4935 diff changeset	217 name=utils.getUTF8(name),
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	218 path=path)
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	219
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	220 if newFeat is not None:
4877 852d4c508f33 (importer) Try to convert paths to UTF-8 for DB insertion Andre Heinecke <aheinecke@intevation.de> parents: 4876 diff changeset	221 newFeat.SetField("path", utils.getUTF8Path(path))
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	222 newFeat = self.transform(newFeat)
4974 a92241d40228 Handle transformation errors Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	223 if newFeat:
a92241d40228 Handle transformation errors Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	224 res = destLayer.CreateFeature(newFeat)
a92241d40228 Handle transformation errors Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	225 if res is None or res > 0:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	226 logger.error("Unable to insert feature. Error: %r" % res)
4974 a92241d40228 Handle transformation errors Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	227 else:
a92241d40228 Handle transformation errors Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	228 success = success + 1
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	229 else:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	230 logger.error("Could not transform feature: %s " % feat.GetFID())
4974 a92241d40228 Handle transformation errors Andre Heinecke <aheinecke@intevation.de> parents: 4970 diff changeset	231 creationFailed += 1
2861 b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	232 else:
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 2853 diff changeset	233 creationFailed = creationFailed + 1
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	234 else:
4995 998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles Andre Heinecke <aheinecke@intevation.de> parents: 4974 diff changeset	235 unsupported[utils.getWkbString(geomType)] = \
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles Andre Heinecke <aheinecke@intevation.de> parents: 4974 diff changeset	236 unsupported.get(utils.getWkbString(geomType), 0) + 1
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	237
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	238 logger.info("Inserted %i features" % success)
769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	239 logger.info("Failed to create %i features" % creationFailed)
5001 fed74592a2ba Fix syntax error Andre Heinecke <aheinecke@intevation.de> parents: 4995 diff changeset	240 for key, value in unsupported.items():
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	241 logger.info("Found %i unsupported features of type: %s" % (value, key))
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	242
4886 8e99d2d9364d Importer: When not tracking dont print info about unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4885 diff changeset	243 if self.tracking_import:
8e99d2d9364d Importer: When not tracking dont print info about unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4885 diff changeset	244 unhandled = []
8e99d2d9364d Importer: When not tracking dont print info about unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4885 diff changeset	245 for i in range(0, srcLayer.GetLayerDefn().GetFieldCount()):
8e99d2d9364d Importer: When not tracking dont print info about unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4885 diff changeset	246 act_field = srcLayer.GetLayerDefn().GetFieldDefn(i).GetNameRef()
8e99d2d9364d Importer: When not tracking dont print info about unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4885 diff changeset	247 if not act_field in self.handled_fields:
8e99d2d9364d Importer: When not tracking dont print info about unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4885 diff changeset	248 unhandled.append(act_field)
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	249
4886 8e99d2d9364d Importer: When not tracking dont print info about unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4885 diff changeset	250 if len(unhandled):
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	251 logger.info("Did not import values from fields: %s " % \
4886 8e99d2d9364d Importer: When not tracking dont print info about unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4885 diff changeset	252 " ".join(unhandled))
4884 b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields Andre Heinecke <aheinecke@intevation.de> parents: 4878 diff changeset	253
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	254 try:
4970 174fbaa3d344 Add handling of River Names and remove target_src parameter Andre Heinecke <aheinecke@intevation.de> parents: 4951 diff changeset	255 if self.dry_run:
3655 8654e4480fc6 Shape importer: added command line option dry_run to supress database transactions. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: 3654 diff changeset	256 return geomType
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	257 destLayer.CommitTransaction()
5160 0dae9e77c5da Importer fix issues reported by pylint Andre Heinecke <aheinecke@intevation.de> parents: 5077 diff changeset	258 except:
5006 769593a84606 Importer: Behold, Logging! Andre Heinecke <aheinecke@intevation.de> parents: 5004 diff changeset	259 logger.error("Exception while committing transaction.")
2853 bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	260
bd9e76e0b55d Improved the python shapefile importer. Ingo Weinzierl <ingo.weinzierl@intevation.de> parents: diff changeset	261 return geomType

Mercurial > dive4elements > river

annotate backend/contrib/shpimporter/importer.py @ 7602:c50dbbe17950