Mercurial > dive4elements > river
annotate flys-backend/contrib/shpimporter/utils.py @ 5025:9ef49f3b0304 dami
Sync Posgres and Oracle spatial schema
author | Andre Heinecke <aheinecke@intevation.de> |
---|---|
date | Tue, 19 Feb 2013 14:58:33 +0100 |
parents | 769593a84606 |
children | d6860fca89e4 |
rev | line source |
---|---|
2798
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
1 import os |
4874
b1d7e600b43b
(importer) Add utility function to convert paths to utf-8
Andre Heinecke <aheinecke@intevation.de>
parents:
3654
diff
changeset
|
2 import sys |
5006
769593a84606
Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents:
4995
diff
changeset
|
3 import logging |
769593a84606
Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents:
4995
diff
changeset
|
4 |
4970
174fbaa3d344
Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents:
4935
diff
changeset
|
5 try: |
174fbaa3d344
Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents:
4935
diff
changeset
|
6 from osgeo import ogr |
174fbaa3d344
Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents:
4935
diff
changeset
|
7 except ImportErrror: |
174fbaa3d344
Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents:
4935
diff
changeset
|
8 import ogr |
2798
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
9 |
5006
769593a84606
Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents:
4995
diff
changeset
|
10 logger = logging.getLogger("utils") |
769593a84606
Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents:
4995
diff
changeset
|
11 |
2798
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
12 SHP='.shp' |
4976
a6ee62a070b0
I'm learning how to use cursors \o/
Andre Heinecke <aheinecke@intevation.de>
parents:
4970
diff
changeset
|
13 SQL_SELECT_RIVER_ID="SELECT id FROM rivers WHERE name = %s" |
2798
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
14 |
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
15 def findShapefiles(path): |
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
16 shapes = [] |
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
17 |
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
18 for root, dirs, files in os.walk(path): |
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
19 if len(files) == 0: |
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
20 continue |
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
21 |
5006
769593a84606
Importer: Behold, Logging!
Andre Heinecke <aheinecke@intevation.de>
parents:
4995
diff
changeset
|
22 logger.debug("Processing directory '%s' with %i files " % (root, len(files))) |
2798
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
23 |
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
24 for f in files: |
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
25 idx = f.find(SHP) |
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
26 if (idx+len(SHP)) == len(f): |
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
27 shapes.append((f.replace(SHP, ''), root + "/" + f)) |
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
28 |
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
29 return shapes |
5a654f2e35bc
Added a python tool to import shapefiles into database.
Ingo Weinzierl <ingo.weinzierl@intevation.de>
parents:
diff
changeset
|
30 |
4970
174fbaa3d344
Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents:
4935
diff
changeset
|
31 def getRiverId(dbconn, name): |
174fbaa3d344
Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents:
4935
diff
changeset
|
32 """ |
174fbaa3d344
Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents:
4935
diff
changeset
|
33 Returns the id of the river "name" |
174fbaa3d344
Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents:
4935
diff
changeset
|
34 Dbconn must be a python database connection api compliant object |
174fbaa3d344
Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents:
4935
diff
changeset
|
35 """ |
174fbaa3d344
Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents:
4935
diff
changeset
|
36 cur = dbconn.cursor() |
4976
a6ee62a070b0
I'm learning how to use cursors \o/
Andre Heinecke <aheinecke@intevation.de>
parents:
4970
diff
changeset
|
37 cur.execute(SQL_SELECT_RIVER_ID, (name,)) |
4970
174fbaa3d344
Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents:
4935
diff
changeset
|
38 row = cur.fetchone() |
174fbaa3d344
Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents:
4935
diff
changeset
|
39 if row: |
174fbaa3d344
Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents:
4935
diff
changeset
|
40 return row[0] |
174fbaa3d344
Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents:
4935
diff
changeset
|
41 else: |
174fbaa3d344
Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents:
4935
diff
changeset
|
42 return 0 |
174fbaa3d344
Add handling of River Names and remove target_src parameter
Andre Heinecke <aheinecke@intevation.de>
parents:
4935
diff
changeset
|
43 |
4935
c0a58558b817
Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents:
4887
diff
changeset
|
44 def getUTF8(string): |
c0a58558b817
Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents:
4887
diff
changeset
|
45 """ |
c0a58558b817
Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents:
4887
diff
changeset
|
46 Tries to convert the string to a UTF-8 encoding by first checking if it |
c0a58558b817
Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents:
4887
diff
changeset
|
47 is UTF-8 and then trying cp1252 |
c0a58558b817
Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents:
4887
diff
changeset
|
48 """ |
c0a58558b817
Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents:
4887
diff
changeset
|
49 try: |
c0a58558b817
Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents:
4887
diff
changeset
|
50 return unicode.encode(unicode(string, "UTF-8"), "UTF-8") |
c0a58558b817
Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents:
4887
diff
changeset
|
51 except UnicodeDecodeError: |
c0a58558b817
Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents:
4887
diff
changeset
|
52 # Probably European Windows names so lets try again |
c0a58558b817
Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents:
4887
diff
changeset
|
53 return unicode.encode(unicode(string, "cp1252"), "UTF-8") |
c0a58558b817
Importer: - Handle regular expressions for attribute names
Andre Heinecke <aheinecke@intevation.de>
parents:
4887
diff
changeset
|
54 |
4874
b1d7e600b43b
(importer) Add utility function to convert paths to utf-8
Andre Heinecke <aheinecke@intevation.de>
parents:
3654
diff
changeset
|
55 def getUTF8Path(path): |
b1d7e600b43b
(importer) Add utility function to convert paths to utf-8
Andre Heinecke <aheinecke@intevation.de>
parents:
3654
diff
changeset
|
56 """ |
b1d7e600b43b
(importer) Add utility function to convert paths to utf-8
Andre Heinecke <aheinecke@intevation.de>
parents:
3654
diff
changeset
|
57 Tries to convert path to utf-8 by first checking the filesystemencoding |
b1d7e600b43b
(importer) Add utility function to convert paths to utf-8
Andre Heinecke <aheinecke@intevation.de>
parents:
3654
diff
changeset
|
58 and trying the default windows encoding afterwards. |
b1d7e600b43b
(importer) Add utility function to convert paths to utf-8
Andre Heinecke <aheinecke@intevation.de>
parents:
3654
diff
changeset
|
59 Returns a valid UTF-8 encoded unicode object or throws a UnicodeDecodeError |
b1d7e600b43b
(importer) Add utility function to convert paths to utf-8
Andre Heinecke <aheinecke@intevation.de>
parents:
3654
diff
changeset
|
60 """ |
b1d7e600b43b
(importer) Add utility function to convert paths to utf-8
Andre Heinecke <aheinecke@intevation.de>
parents:
3654
diff
changeset
|
61 try: |
b1d7e600b43b
(importer) Add utility function to convert paths to utf-8
Andre Heinecke <aheinecke@intevation.de>
parents:
3654
diff
changeset
|
62 return unicode.encode(unicode(path, sys.getfilesystemencoding()), "UTF-8") |
b1d7e600b43b
(importer) Add utility function to convert paths to utf-8
Andre Heinecke <aheinecke@intevation.de>
parents:
3654
diff
changeset
|
63 except UnicodeDecodeError: |
4887
1f6e544f7a7f
Importer: Use cp1252 instead of latin-9 to guess filename encodings
Andre Heinecke <aheinecke@intevation.de>
parents:
4884
diff
changeset
|
64 # Probably European Windows names so lets try again |
1f6e544f7a7f
Importer: Use cp1252 instead of latin-9 to guess filename encodings
Andre Heinecke <aheinecke@intevation.de>
parents:
4884
diff
changeset
|
65 return unicode.encode(unicode(path, "cp1252"), "UTF-8") |
4995
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
66 |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
67 WKB_MAP = { |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
68 ogr.wkb25Bit : 'wkb25Bit', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
69 ogr.wkbGeometryCollection : 'wkbGeometryCollection', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
70 ogr.wkbGeometryCollection25D :'wkbGeometryCollection25D', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
71 ogr.wkbLineString : 'wkbLineString', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
72 ogr.wkbLineString25D : 'wkbLineString25D', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
73 ogr.wkbLinearRing : 'wkbLinearRing', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
74 ogr.wkbMultiLineString : 'wkbMultiLineString', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
75 ogr.wkbMultiLineString25D : 'wkbMultiLineString25D', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
76 ogr.wkbMultiPoint : 'wkbMultiPoint', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
77 ogr.wkbMultiPoint25D : 'wkbMultiPoint25D', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
78 ogr.wkbMultiPolygon : 'wkbMultiPolygon', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
79 ogr.wkbMultiPolygon25D : 'wkbMultiPolygon25D', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
80 ogr.wkbNDR : 'wkbNDR', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
81 ogr.wkbNone : 'wkbNone', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
82 ogr.wkbPoint : 'wkbPoint', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
83 ogr.wkbPoint25D : 'wkbPoint25D', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
84 ogr.wkbPolygon : 'wkbPolygon', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
85 ogr.wkbPolygon25D : 'wkbPolygon25D', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
86 ogr.wkbUnknown : 'wkbUnknown', |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
87 ogr.wkbXDR : 'wkbXDR' |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
88 } |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
89 |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
90 def getWkbString(type): |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
91 return WKB_MAP.get(type) or "Unknown" |
998b29c8d2fd
Improve debug output for unsupported features and skipped shapefiles
Andre Heinecke <aheinecke@intevation.de>
parents:
4976
diff
changeset
|
92 |