Mercurial > dive4elements > river

annotate flys-backend/contrib/shpimporter/importer.py @ 5084:ca45dd039b54 slt-simplify-cross-sections

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Log the number of points of cross sections to see the effect of the simplification.
author Sascha L. Teichmann <teichmann@intevation.de>
date Sun, 24 Feb 2013 21:38:09 +0100
parents c5187ab9f571
children 0dae9e77c5da
rev   line source
4872
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
1 try:
5004
f1c01fecf194 Fix imports of ogr, this removes a warning in gdal 1.9 but keeps
Andre Heinecke <aheinecke@intevation.de>
parents: 5001
diff changeset
2 from osgeo import ogr, osr
5077
c5187ab9f571 Fix Syntax Errror
Andre Heinecke <aheinecke@intevation.de>
parents: 5006
diff changeset
3 except ImportError:
5004
f1c01fecf194 Fix imports of ogr, this removes a warning in gdal 1.9 but keeps
Andre Heinecke <aheinecke@intevation.de>
parents: 5001
diff changeset
4 import ogr, osr
4877
852d4c508f33 (importer) Try to convert paths to UTF-8 for DB insertion
Andre Heinecke <aheinecke@intevation.de>
parents: 4876
diff changeset
5 import utils
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
6 import re
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
7 import logging
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
8
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
9 logger = logging.getLogger("importer")
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
10
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
11 class Importer:
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
12
4970
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4951
diff changeset
13 def __init__(self, river_id, dbconn, dry_run):
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4951
diff changeset
14 self.river_id = river_id
4872
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
15 self.dbconn = dbconn
4970
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4951
diff changeset
16 self.dry_run = dry_run
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
17 self.dest_srs = osr.SpatialReference()
4970
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4951
diff changeset
18 self.dest_srs.ImportFromEPSG(31467)
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
19 self.handled_fields = []
4885
89a8764cabcc Importer: Only show imported fields if copy_fields is used
Andre Heinecke <aheinecke@intevation.de>
parents: 4884
diff changeset
20 self.tracking_import = False
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
21 self.srcLayer = None
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
22
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
23 def getKind(self, path):
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
24 raise NotImplementedError("Importer.getKind is abstract!")
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
25
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
26 def getPath(self, base):
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
27 raise NotImplementedError("Importer.getPath is abstract!")
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
28
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
29 def getTablename(self):
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
30 raise NotImplementedError("Importer.getTablename is abstract!")
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
31
3654
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
32 def getName(self):
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
33 raise NotImplementedError("Importer.getTablename is abstract!")
59ca5dab2782 Shape importer: use python's OptionParse to read user specific configuration from command line.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2862
diff changeset
34
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
35 def IsFieldSet(self, feat, name):
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
36 if not name:
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
37 return False
4878
82d00b0c7302 (importer) Avoid uneccessary error and improve debug output
Andre Heinecke <aheinecke@intevation.de>
parents: 4877
diff changeset
38 if feat.GetFieldIndex(name) == -1:
82d00b0c7302 (importer) Avoid uneccessary error and improve debug output
Andre Heinecke <aheinecke@intevation.de>
parents: 4877
diff changeset
39 return False # Avoids an Error in IsFieldSet
82d00b0c7302 (importer) Avoid uneccessary error and improve debug output
Andre Heinecke <aheinecke@intevation.de>
parents: 4877
diff changeset
40 return feat.IsFieldSet(feat.GetFieldIndex(name))
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
41
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
42 def searchField(self, regex):
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
43 """
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
44 Searches for a field in the current src layer that matches
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
45 the expression regex.
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
46 Throws an exception if more then one field matches
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
47 @param feat: The feature to search for attributes
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
48 @param regex: The regex to look for
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
49
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
50 @returns: The field name as a string
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
51 """
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
52
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
53 if not hasattr(self.srcLayer, "fieldnames"):
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
54 self.srcLayer.fieldnames = []
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
55 for i in range(0, self.srcLayer.GetLayerDefn().GetFieldCount()):
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
56 self.srcLayer.fieldnames.append(
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
57 self.srcLayer.GetLayerDefn().GetFieldDefn(i).GetNameRef())
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
58
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
59 result = None
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
60 for name in self.srcLayer.fieldnames:
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
61 match = re.match(regex, name, re.IGNORECASE)
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
62 if match:
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
63 if result:
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
64 raise Exception("More then one field matches: %s" % regex)
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
65 else:
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
66 result = match.group(0)
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
67 return result
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
68
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
69 def IsDoubleFieldSet(self, feat, name):
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
70 try:
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
71 isset = feat.GetFieldAsDouble(name)
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
72 return isset is not None
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
73 except:
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
74 return False
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
75
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
76 def isShapeRelevant(self, name, path):
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
77 return True
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
78
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
79 def walkOverShapes(self, shape):
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
80 (name, path) = shape
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
81
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
82 shp = ogr.Open(shape[1])
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
83 if shp is None:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
84 logger.error("Shapefile '%s' could not be opened!" % path)
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
85 return
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
86
4995
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents: 4974
diff changeset
87 if not self.isShapeRelevant(name, path):
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
88 logger.info("Skip shapefile: '%s' of Type: %s" % (path,
4995
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents: 4974
diff changeset
89 utils.getWkbString(shp.GetLayerByName(name).GetGeomType())))
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents: 4974
diff changeset
90 return
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents: 4974
diff changeset
91
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents: 4974
diff changeset
92
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
93 logger.info("Processing shapefile '%s'" % path)
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
94 srcLayer = shp.GetLayerByName(name)
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
95
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
96 if srcLayer is None:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
97 logger.error("Layer '%s' was not found!" % name)
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
98 return
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
99
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
100 return self.shape2Database(srcLayer, name, path)
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
101
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
102 def transform(self, feat):
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
103 geometry = feat.GetGeometryRef()
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
104 src_srs = geometry.GetSpatialReference()
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
105
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
106 if src_srs is None:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
107 logger.error("No source SRS given! No transformation possible!")
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
108 return feat
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
109
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
110 transformer = osr.CoordinateTransformation(src_srs, self.dest_srs)
4974
a92241d40228 Handle transformation errors
Andre Heinecke <aheinecke@intevation.de>
parents: 4970
diff changeset
111 if geometry.Transform(transformer):
a92241d40228 Handle transformation errors
Andre Heinecke <aheinecke@intevation.de>
parents: 4970
diff changeset
112 return None
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
113
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
114 return feat
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
115
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
116 def handled(self, field):
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
117 """
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
118 Register a field or a map of as handled during the import.
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
119
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
120 There is a warning printed after the import for each unhandled field!
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
121 """
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
122 if not field in self.handled_fields:
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
123 self.handled_fields.append(field)
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
124
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
125 def copyFields(self, src, target, mapping):
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
126 """
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
127 Checks the mapping dictonary for key value pairs to
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
128 copy from the source to the destination feature.
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
129 The keys can be reguar expressions that are matched
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
130 agains the source fieldnames
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
131
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
132 The Key is the attribute of the source feature to be copied
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
133 into the target attribute named by the dict's value.
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
134 """
4885
89a8764cabcc Importer: Only show imported fields if copy_fields is used
Andre Heinecke <aheinecke@intevation.de>
parents: 4884
diff changeset
135 self.tracking_import = True
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
136 for key, value in mapping.items():
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
137 realname = self.searchField(key)
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
138 if realname == None:
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
139 continue
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
140 if not realname in self.handled_fields:
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
141 self.handled_fields.append(realname)
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
142 # 0 OFTInteger, Simple 32bit integer
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
143 # 1 OFTIntegerList, List of 32bit integers
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
144 # 2 OFTReal, Double Precision floating point
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
145 # 3 OFTRealList, List of doubles
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
146 # 4 OFTString, String of ASCII chars
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
147 # 5 OFTStringList, Array of strings
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
148 # 6 OFTWideString, deprecated
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
149 # 7 OFTWideStringList, deprecated
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
150 # 8 OFTBinary, Raw Binary data
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
151 # 9 OFTDate, Date
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
152 # 10 OFTTime, Time
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
153 # 11 OFTDateTime, Date and Time
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
154 if src.IsFieldSet(src.GetFieldIndex(realname)):
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
155 if src.GetFieldType(realname) == 2:
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
156 target.SetField(value, src.GetFieldAsDouble(realname))
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
157 else:
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
158 target.SetField(value, utils.getUTF8(src.GetField(realname)))
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
159
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
160 def shape2Database(self, srcLayer, name, path):
4872
a563e9f58f93 Improve error handling and unify dbconn for all importers
Andre Heinecke <aheinecke@intevation.de>
parents: 4871
diff changeset
161 destLayer = self.dbconn.GetLayerByName(self.getTablename())
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
162
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
163 if srcLayer is None:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
164 logger.error("Shapefile is None!")
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
165 return -1
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
166
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
167 if destLayer is None:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
168 logger.error("No destination layer given!")
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
169 return -1
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
170
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
171 count = srcLayer.GetFeatureCount()
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
172 logger.debug("Try to add %i features to database." % count)
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
173
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
174 srcLayer.ResetReading()
4935
c0a58558b817 Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents: 4886
diff changeset
175 self.srcLayer = srcLayer
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
176
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
177 geomType = -1
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
178 success = 0
4995
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents: 4974
diff changeset
179 unsupported = {}
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
180 creationFailed = 0
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
181 featureDef = destLayer.GetLayerDefn()
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
182
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
183 for feat in srcLayer:
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
184 geom = feat.GetGeometryRef()
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
185
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
186 if geom is None:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
187 logger.debug("Unkown Geometry reference for feature")
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
188 continue
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
189
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
190 geomType = geom.GetGeometryType()
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
191
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
192 if self.isGeometryValid(geomType):
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
193 newFeat = self.createNewFeature(featureDef,
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
194 feat,
4951
23a090b41543 Importer: Check lowercase pathname and handle utf-8 conversion
Andre Heinecke <aheinecke@intevation.de>
parents: 4935
diff changeset
195 name=utils.getUTF8(name),
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
196 path=path)
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
197
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
198 if newFeat is not None:
4877
852d4c508f33 (importer) Try to convert paths to UTF-8 for DB insertion
Andre Heinecke <aheinecke@intevation.de>
parents: 4876
diff changeset
199 newFeat.SetField("path", utils.getUTF8Path(path))
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
200 newFeat = self.transform(newFeat)
4974
a92241d40228 Handle transformation errors
Andre Heinecke <aheinecke@intevation.de>
parents: 4970
diff changeset
201 if newFeat:
a92241d40228 Handle transformation errors
Andre Heinecke <aheinecke@intevation.de>
parents: 4970
diff changeset
202 res = destLayer.CreateFeature(newFeat)
a92241d40228 Handle transformation errors
Andre Heinecke <aheinecke@intevation.de>
parents: 4970
diff changeset
203 if res is None or res > 0:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
204 logger.error("Unable to insert feature. Error: %r" % res)
4974
a92241d40228 Handle transformation errors
Andre Heinecke <aheinecke@intevation.de>
parents: 4970
diff changeset
205 else:
a92241d40228 Handle transformation errors
Andre Heinecke <aheinecke@intevation.de>
parents: 4970
diff changeset
206 success = success + 1
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
207 else:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
208 logger.error("Could not transform feature: %s " % feat.GetFID())
4974
a92241d40228 Handle transformation errors
Andre Heinecke <aheinecke@intevation.de>
parents: 4970
diff changeset
209 creationFailed += 1
2861
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
210 else:
b0132e1b9719 Added further shape importers and added the option to reproject shapes during the import process.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 2853
diff changeset
211 creationFailed = creationFailed + 1
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
212 else:
4995
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents: 4974
diff changeset
213 unsupported[utils.getWkbString(geomType)] = \
998b29c8d2fd Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents: 4974
diff changeset
214 unsupported.get(utils.getWkbString(geomType), 0) + 1
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
215
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
216 logger.info("Inserted %i features" % success)
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
217 logger.info("Failed to create %i features" % creationFailed)
5001
fed74592a2ba Fix syntax error
Andre Heinecke <aheinecke@intevation.de>
parents: 4995
diff changeset
218 for key, value in unsupported.items():
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
219 logger.info("Found %i unsupported features of type: %s" % (value, key))
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
220
4886
8e99d2d9364d Importer: When not tracking dont print info about unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4885
diff changeset
221 if self.tracking_import:
8e99d2d9364d Importer: When not tracking dont print info about unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4885
diff changeset
222 unhandled = []
8e99d2d9364d Importer: When not tracking dont print info about unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4885
diff changeset
223 for i in range(0, srcLayer.GetLayerDefn().GetFieldCount()):
8e99d2d9364d Importer: When not tracking dont print info about unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4885
diff changeset
224 act_field = srcLayer.GetLayerDefn().GetFieldDefn(i).GetNameRef()
8e99d2d9364d Importer: When not tracking dont print info about unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4885
diff changeset
225 if not act_field in self.handled_fields:
8e99d2d9364d Importer: When not tracking dont print info about unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4885
diff changeset
226 unhandled.append(act_field)
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
227
4886
8e99d2d9364d Importer: When not tracking dont print info about unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4885
diff changeset
228 if len(unhandled):
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
229 logger.info("Did not import values from fields: %s " % \
4886
8e99d2d9364d Importer: When not tracking dont print info about unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4885
diff changeset
230 " ".join(unhandled))
4884
b457532dae63 Importer: Move copy fields into importer class and track imported/unimported fields
Andre Heinecke <aheinecke@intevation.de>
parents: 4878
diff changeset
231
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
232 try:
4970
174fbaa3d344 Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents: 4951
diff changeset
233 if self.dry_run:
3655
8654e4480fc6 Shape importer: added command line option dry_run to supress database transactions.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents: 3654
diff changeset
234 return geomType
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
235 destLayer.CommitTransaction()
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
236 except e:
5006
769593a84606 Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents: 5004
diff changeset
237 logger.error("Exception while committing transaction.")
2853
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
238
bd9e76e0b55d Improved the python shapefile importer.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff changeset
239 return geomType
http://dive4elements.wald.intevation.org