Mercurial > dive4elements > river

annotate flys-backend/contrib/shpimporter/importer.py @ 5386:6c2751c17869

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Importer: Add function to search a value in all fields. I've also discovered that you can access fields of feature by using it as a dict. \o/
author Andre Heinecke <aheinecke@intevation.de>
date Fri, 22 Mar 2013 17:31:48 +0100
parents 90f1c5f67698
children ff11b178f152
rev   line source
4872
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
1 try:
5004
f1c01fecf194 Fix imports of ogr, this removes a warning in gdal 1.9 but keeps
Andre Heinecke <aheinecke@intevation.de>
parents: 5001
diff changeset
2 from osgeo import ogr, osr
5077
c5187ab9f571 Fix Syntax Errror
Andre Heinecke <aheinecke@intevation.de>
parents: 5006
diff changeset
3 except ImportError:
5004
f1c01fecf194 Fix imports of ogr, this removes a warning in gdal 1.9 but keeps
Andre Heinecke <aheinecke@intevation.de>
parents: 5001
diff changeset
4 import ogr, osr
4877
852d4c508f33 (importer) Try to convert paths to UTF-8 for DB insertion
Andre Heinecke <aheinecke@intevation.de>
parents: 4876
diff changeset
5 import utils
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
6 import re
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
7 import logging
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
8
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
9 logger = logging.getLogger("importer")
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
10
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
11 class Importer:
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
12
4970
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4951
diff changeset
13 def __init__(self, river_id, dbconn, dry_run):
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4951
diff changeset
14 self.river_id = river_id
4872
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
15 self.dbconn = dbconn
4970
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4951
diff changeset
16 self.dry_run = dry_run
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
17 self.dest_srs = osr.SpatialReference()
4970
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4951
diff changeset
18 self.dest_srs.ImportFromEPSG(31467)
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
19 self.handled_fields = []
4885
89a8764cabcc Importer: Only show imported fields if copy_fields is used
Andre Heinecke <aheinecke@intevation.de>
parents: 4884
diff changeset
20 self.tracking_import = False
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
21 self.srcLayer = None
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
22
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
23 def getKind(self, path):
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
24 raise NotImplementedError("Importer.getKind is abstract!")
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
25
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
26 def getPath(self, base):
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
27 raise NotImplementedError("Importer.getPath is abstract!")
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
28
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
29 def getTablename(self):
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
30 raise NotImplementedError("Importer.getTablename is abstract!")
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
31
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
32 def getName(self):
5160
0dae9e77c5da Importer fix issues reported by pylint
Andre Heinecke <aheinecke@intevation.de>
parents: 5077
diff changeset
33 raise NotImplementedError("Importer.getName is abstract!")
0dae9e77c5da Importer fix issues reported by pylint
Andre Heinecke <aheinecke@intevation.de>
parents: 5077
diff changeset
34
0dae9e77c5da Importer fix issues reported by pylint
Andre Heinecke <aheinecke@intevation.de>
parents: 5077
diff changeset
35 def isGeometryValid(self, geomType):
0dae9e77c5da Importer fix issues reported by pylint
Andre Heinecke <aheinecke@intevation.de>
parents: 5077
diff changeset
36 raise NotImplementedError("Importer.isGeometryValid is abstract!")
0dae9e77c5da Importer fix issues reported by pylint
Andre Heinecke <aheinecke@intevation.de>
parents: 5077
diff changeset
37
0dae9e77c5da Importer fix issues reported by pylint
Andre Heinecke <aheinecke@intevation.de>
parents: 5077
diff changeset
38 def createNewFeature(self, featureDef, feat, **args):
0dae9e77c5da Importer fix issues reported by pylint
Andre Heinecke <aheinecke@intevation.de>
parents: 5077
diff changeset
39 raise NotImplementedError("Importer.createNewFeature is abstract!")
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
40
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
41 def IsFieldSet(self, feat, name):
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
42 if not name:
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
43 return False
4878
82d00b0c7302 (importer) Avoid uneccessary error and improve debug output
Andre Heinecke <aheinecke@intevation.de>
parents: 4877
diff changeset
44 if feat.GetFieldIndex(name) == -1:
82d00b0c7302 (importer) Avoid uneccessary error and improve debug output
Andre Heinecke <aheinecke@intevation.de>
parents: 4877
diff changeset
45 return False # Avoids an Error in IsFieldSet
82d00b0c7302 (importer) Avoid uneccessary error and improve debug output
Andre Heinecke <aheinecke@intevation.de>
parents: 4877
diff changeset
46 return feat.IsFieldSet(feat.GetFieldIndex(name))
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
47
5386
6c2751c17869 Importer: Add function to search a value in all fields.
Andre Heinecke <aheinecke@intevation.de>
parents: 5365
diff changeset
48 def searchValue(self, feat, regex):
6c2751c17869 Importer: Add function to search a value in all fields.
Andre Heinecke <aheinecke@intevation.de>
parents: 5365
diff changeset
49 """
6c2751c17869 Importer: Add function to search a value in all fields.
Andre Heinecke <aheinecke@intevation.de>
parents: 5365
diff changeset
50 Searches for a value that matches regx in all attribute
6c2751c17869 Importer: Add function to search a value in all fields.
Andre Heinecke <aheinecke@intevation.de>
parents: 5365
diff changeset
51 fields of a feature.
6c2751c17869 Importer: Add function to search a value in all fields.
Andre Heinecke <aheinecke@intevation.de>
parents: 5365
diff changeset
52 """
6c2751c17869 Importer: Add function to search a value in all fields.
Andre Heinecke <aheinecke@intevation.de>
parents: 5365
diff changeset
53 for val in feat.items():
6c2751c17869 Importer: Add function to search a value in all fields.
Andre Heinecke <aheinecke@intevation.de>
parents: 5365
diff changeset
54 match = re.match(regex, val, re.IGNORECASE)
6c2751c17869 Importer: Add function to search a value in all fields.
Andre Heinecke <aheinecke@intevation.de>
parents: 5365
diff changeset
55 return match != None
6c2751c17869 Importer: Add function to search a value in all fields.
Andre Heinecke <aheinecke@intevation.de>
parents: 5365
diff changeset
56
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
57 def searchField(self, regex):
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
58 """
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
59 Searches for a field in the current src layer that matches
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
60 the expression regex.
5174
b3a15fa3d88e let import_river.sh parse gauge official numbers, typo
Tom Gottfried <tom@intevation.de>
parents: 5160
diff changeset
61 Throws an exception if more than one field matches
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
62 @param feat: The feature to search for attributes
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
63 @param regex: The regex to look for
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
64
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
65 @returns: The field name as a string
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
66 """
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
67
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
68 if not hasattr(self.srcLayer, "fieldnames"):
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
69 self.srcLayer.fieldnames = []
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
70 for i in range(0, self.srcLayer.GetLayerDefn().GetFieldCount()):
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
71 self.srcLayer.fieldnames.append(
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
72 self.srcLayer.GetLayerDefn().GetFieldDefn(i).GetNameRef())
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
73
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
74 result = None
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
75 for name in self.srcLayer.fieldnames:
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
76 match = re.match(regex, name, re.IGNORECASE)
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
77 if match:
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
78 if result:
5174
b3a15fa3d88e let import_river.sh parse gauge official numbers, typo
Tom Gottfried <tom@intevation.de>
parents: 5160
diff changeset
79 raise Exception("More than one field matches: %s" % regex)
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
80 else:
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
81 result = match.group(0)
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
82 return result
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
83
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
84 def IsDoubleFieldSet(self, feat, name):
5365
90f1c5f67698 Importer: Do not return true for IsDoubleField set if the field
Andre Heinecke <aheinecke@intevation.de>
parents: 5174
diff changeset
85 if not self.IsFieldSet(feat, name):
90f1c5f67698 Importer: Do not return true for IsDoubleField set if the field
Andre Heinecke <aheinecke@intevation.de>
parents: 5174
diff changeset
86 return False
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
87 try:
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
88 isset = feat.GetFieldAsDouble(name)
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
89 return isset is not None
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
90 except:
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
91 return False
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
92
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
93 def isShapeRelevant(self, name, path):
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
94 return True
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
95
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
96 def walkOverShapes(self, shape):
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
97 (name, path) = shape
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
98
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
99 shp = ogr.Open(shape[1])
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
100 if shp is None:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
101 logger.error("Shapefile '%s' could not be opened!" % path)
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
102 return
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
103
4995
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents: 4974
diff changeset
104 if not self.isShapeRelevant(name, path):
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
105 logger.info("Skip shapefile: '%s' of Type: %s" % (path,
4995
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents: 4974
diff changeset
106 utils.getWkbString(shp.GetLayerByName(name).GetGeomType())))
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents: 4974
diff changeset
107 return
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents: 4974
diff changeset
108
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents: 4974
diff changeset
109
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
110 logger.info("Processing shapefile '%s'" % path)
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
111 srcLayer = shp.GetLayerByName(name)
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
112
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
113 if srcLayer is None:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
114 logger.error("Layer '%s' was not found!" % name)
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
115 return
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
116
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
117 return self.shape2Database(srcLayer, name, path)
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
118
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
119 def transform(self, feat):
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
120 geometry = feat.GetGeometryRef()
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
121 src_srs = geometry.GetSpatialReference()
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
122
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
123 if src_srs is None:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
124 logger.error("No source SRS given! No transformation possible!")
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
125 return feat
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
126
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
127 transformer = osr.CoordinateTransformation(src_srs, self.dest_srs)
4974
a92241d40228 Handle transformation errors
Andre Heinecke <aheinecke@intevation.de>
parents: 4970
diff changeset
128 if geometry.Transform(transformer):
a92241d40228 Handle transformation errors
Andre Heinecke <aheinecke@intevation.de>
parents: 4970
diff changeset
129 return None
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
130
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
131 return feat
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
132
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
133 def handled(self, field):
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
134 """
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
135 Register a field or a map of as handled during the import.
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
136
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
137 There is a warning printed after the import for each unhandled field!
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
138 """
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
139 if not field in self.handled_fields:
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
140 self.handled_fields.append(field)
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
141
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
142 def copyFields(self, src, target, mapping):
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
143 """
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
144 Checks the mapping dictonary for key value pairs to
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
145 copy from the source to the destination feature.
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
146 The keys can be reguar expressions that are matched
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
147 agains the source fieldnames
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
148
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
149 The Key is the attribute of the source feature to be copied
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
150 into the target attribute named by the dict's value.
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
151 """
4885
89a8764cabcc Importer: Only show imported fields if copy_fields is used
Andre Heinecke <aheinecke@intevation.de>
parents: 4884
diff changeset
152 self.tracking_import = True
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
153 for key, value in mapping.items():
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
154 realname = self.searchField(key)
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
155 if realname == None:
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
156 continue
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
157 if not realname in self.handled_fields:
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
158 self.handled_fields.append(realname)
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
159 # 0 OFTInteger, Simple 32bit integer
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
160 # 1 OFTIntegerList, List of 32bit integers
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
161 # 2 OFTReal, Double Precision floating point
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
162 # 3 OFTRealList, List of doubles
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
163 # 4 OFTString, String of ASCII chars
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
164 # 5 OFTStringList, Array of strings
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
165 # 6 OFTWideString, deprecated
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
166 # 7 OFTWideStringList, deprecated
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
167 # 8 OFTBinary, Raw Binary data
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
168 # 9 OFTDate, Date
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
169 # 10 OFTTime, Time
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
170 # 11 OFTDateTime, Date and Time
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
171 if src.IsFieldSet(src.GetFieldIndex(realname)):
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
172 if src.GetFieldType(realname) == 2:
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
173 target.SetField(value, src.GetFieldAsDouble(realname))
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
174 else:
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
175 target.SetField(value, utils.getUTF8(src.GetField(realname)))
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
176
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
177 def shape2Database(self, srcLayer, name, path):
4872
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
178 destLayer = self.dbconn.GetLayerByName(self.getTablename())
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
179
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
180 if srcLayer is None:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
181 logger.error("Shapefile is None!")
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
182 return -1
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
183
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
184 if destLayer is None:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
185 logger.error("No destination layer given!")
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
186 return -1
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
187
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
188 count = srcLayer.GetFeatureCount()
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
189 logger.debug("Try to add %i features to database." % count)
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
190
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
191 srcLayer.ResetReading()
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
192 self.srcLayer = srcLayer
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
193
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
194 geomType = -1
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
195 success = 0
4995
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents: 4974
diff changeset
196 unsupported = {}
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
197 creationFailed = 0
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
198 featureDef = destLayer.GetLayerDefn()
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
199
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
200 for feat in srcLayer:
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
201 geom = feat.GetGeometryRef()
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
202
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
203 if geom is None:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
204 logger.debug("Unkown Geometry reference for feature")
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
205 continue
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
206
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
207 geomType = geom.GetGeometryType()
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
208
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
209 if self.isGeometryValid(geomType):
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
210 newFeat = self.createNewFeature(featureDef,
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
211 feat,
4951
23a090b41543 Importer: Check lowercase pathname and handle utf-8 conversion
Andre Heinecke <aheinecke@intevation.de>
parents: 4935
diff changeset
212 name=utils.getUTF8(name),
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
213 path=path)
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
214
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
215 if newFeat is not None:
4877
852d4c508f33 (importer) Try to convert paths to UTF-8 for DB insertion
Andre Heinecke <aheinecke@intevation.de>
parents: 4876
diff changeset
216 newFeat.SetField("path", utils.getUTF8Path(path))
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
217 newFeat = self.transform(newFeat)
4974
a92241d40228 Handle transformation errors
Andre Heinecke <aheinecke@intevation.de>
parents: 4970
diff changeset
218 if newFeat:
a92241d40228 Handle transformation errors
Andre Heinecke <aheinecke@intevation.de>
parents: 4970
diff changeset
219 res = destLayer.CreateFeature(newFeat)
a92241d40228 Handle transformation errors
Andre Heinecke <aheinecke@intevation.de>
parents: 4970
diff changeset
220 if res is None or res > 0:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
221 logger.error("Unable to insert feature. Error: %r" % res)
4974
a92241d40228 Handle transformation errors
Andre Heinecke <aheinecke@intevation.de>
parents: 4970
diff changeset
222 else:
a92241d40228 Handle transformation errors
Andre Heinecke <aheinecke@intevation.de>
parents: 4970
diff changeset
223 success = success + 1
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
224 else:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
225 logger.error("Could not transform feature: %s " % feat.GetFID())
4974
a92241d40228 Handle transformation errors
Andre Heinecke <aheinecke@intevation.de>
parents: 4970
diff changeset
226 creationFailed += 1
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
227 else:
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
228 creationFailed = creationFailed + 1
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
229 else:
4995
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents: 4974
diff changeset
230 unsupported[utils.getWkbString(geomType)] = \
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents: 4974
diff changeset
231 unsupported.get(utils.getWkbString(geomType), 0) + 1
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
232
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
233 logger.info("Inserted %i features" % success)
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
234 logger.info("Failed to create %i features" % creationFailed)
5001
fed74592a2ba Fix syntax error
Andre Heinecke <aheinecke@intevation.de>
parents: 4995
diff changeset
235 for key, value in unsupported.items():
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
236 logger.info("Found %i unsupported features of type: %s" % (value, key))
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
237
4886
8e99d2d9364d Importer: When not tracking dont print info about unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4885
diff changeset
238 if self.tracking_import:
8e99d2d9364d Importer: When not tracking dont print info about unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4885
diff changeset
239 unhandled = []
8e99d2d9364d Importer: When not tracking dont print info about unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4885
diff changeset
240 for i in range(0, srcLayer.GetLayerDefn().GetFieldCount()):
8e99d2d9364d Importer: When not tracking dont print info about unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4885
diff changeset
241 act_field = srcLayer.GetLayerDefn().GetFieldDefn(i).GetNameRef()
8e99d2d9364d Importer: When not tracking dont print info about unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4885
diff changeset
242 if not act_field in self.handled_fields:
8e99d2d9364d Importer: When not tracking dont print info about unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4885
diff changeset
243 unhandled.append(act_field)
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
244
4886
8e99d2d9364d Importer: When not tracking dont print info about unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4885
diff changeset
245 if len(unhandled):
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
246 logger.info("Did not import values from fields: %s " % \
4886
8e99d2d9364d Importer: When not tracking dont print info about unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4885
diff changeset
247 " ".join(unhandled))
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
248
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
249 try:
4970
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4951
diff changeset
250 if self.dry_run:
3655
8654e4480fc6 Shape importer: added command line option dry_run to supress database transactions.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
251 return geomType
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
252 destLayer.CommitTransaction()
5160
0dae9e77c5da Importer fix issues reported by pylint
Andre Heinecke <aheinecke@intevation.de>
parents: 5077
diff changeset
253 except:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
254 logger.error("Exception while committing transaction.")
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
255
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
256 return geomType
http://dive4elements.wald.intevation.org