Mercurial > treepkg
comparison bin/publishdebianpackages.py @ 440:383d23c6bfde
Merged revisions 346-386 via svnmerge from
svn+ssh://svn.wald.intevation.org/treepkg/branches/treepkg-status
........
r346 | bricks | 2010-07-07 17:40:48 +0200 (Mi, 07 Jul 2010) | 2 lines
added classed for treepkg status xml generation
........
r347 | bricks | 2010-07-08 12:07:39 +0200 (Do, 08 Jul 2010) | 10 lines
renamed new status dir to info because of a naming conflict with status.py
let the user specify a treepkg name in the config
the name is propagated to PackagerGroup
[treepkg]
name: <treepkgname>
becomes:
pg = PackagerGroup(...)
pg.name
........
r348 | bricks | 2010-07-08 15:25:07 +0200 (Do, 08 Jul 2010) | 3 lines
moved test file to test main dir
fixed missing : after def statement
........
r349 | bricks | 2010-07-08 17:59:52 +0200 (Do, 08 Jul 2010) | 2 lines
added treepkg_dir andd tracks_dir attributes to PackageGroup
........
r350 | bricks | 2010-07-08 18:00:32 +0200 (Do, 08 Jul 2010) | 2 lines
changed <millpath> to <trackspath>
........
r351 | bricks | 2010-07-08 18:08:39 +0200 (Do, 08 Jul 2010) | 2 lines
check if config file exists to get usefull error output
........
r352 | bricks | 2010-07-08 18:11:13 +0200 (Do, 08 Jul 2010) | 2 lines
write first info about tracks
........
r353 | bricks | 2010-07-09 12:20:49 +0200 (Fr, 09 Jul 2010) | 2 lines
get all log files
........
r354 | bricks | 2010-07-09 14:42:42 +0200 (Fr, 09 Jul 2010) | 2 lines
added md5sum function
........
r355 | bricks | 2010-07-09 15:15:15 +0200 (Fr, 09 Jul 2010) | 4 lines
implemented nearly all info
only arch and os info are missing
........
r356 | bricks | 2010-07-09 16:19:17 +0200 (Fr, 09 Jul 2010) | 2 lines
Bugfix: source was not defined
........
r357 | bricks | 2010-07-09 16:44:52 +0200 (Fr, 09 Jul 2010) | 3 lines
fixed some typos
added missing add_checksum methond
........
r358 | bricks | 2010-07-09 17:06:15 +0200 (Fr, 09 Jul 2010) | 2 lines
fixed status line for revision
........
r359 | bricks | 2010-07-12 14:10:34 +0200 (Mo, 12 Jul 2010) | 2 lines
let the user specify num revisions that should be shown in the info
........
r360 | bricks | 2010-07-12 14:21:39 +0200 (Mo, 12 Jul 2010) | 2 lines
missed in last commit
........
r361 | bricks | 2010-07-12 17:06:22 +0200 (Mo, 12 Jul 2010) | 2 lines
add os config statement
........
r362 | bricks | 2010-07-12 17:47:42 +0200 (Mo, 12 Jul 2010) | 3 lines
moved packages and logs to revision tag
implemented os and arch info
........
r363 | bricks | 2010-07-13 14:23:28 +0200 (Di, 13 Jul 2010) | 2 lines
updated test readconfig for os config variable
........
r364 | bricks | 2010-07-13 14:24:14 +0200 (Di, 13 Jul 2010) | 2 lines
moved common publish functions to a seperate module
........
r365 | bricks | 2010-07-13 16:26:11 +0200 (Di, 13 Jul 2010) | 2 lines
moved arch info to each package info
........
r366 | bricks | 2010-07-13 18:46:17 +0200 (Di, 13 Jul 2010) | 2 lines
inital checkin for new publishpackages processing
........
r367 | bricks | 2010-07-21 17:52:15 +0200 (Mi, 21 Jul 2010) | 2 lines
implemented parsing from xml string
........
r368 | bricks | 2010-07-22 12:17:42 +0200 (Do, 22 Jul 2010) | 2 lines
copy binary-all packages in all binary-xyz dirs
........
r369 | bricks | 2010-07-22 12:26:40 +0200 (Do, 22 Jul 2010) | 2 lines
fixed small bugs
........
r370 | bricks | 2010-07-22 13:02:36 +0200 (Do, 22 Jul 2010) | 2 lines
added a new root element which is called TreepkgRoot for Treepkg Info classes
........
r371 | bricks | 2010-07-22 13:09:52 +0200 (Do, 22 Jul 2010) | 3 lines
fixed some small issues
remember: run pychecker BEFORE the commit
........
r372 | bricks | 2010-07-22 18:01:43 +0200 (Do, 22 Jul 2010) | 2 lines
improved publishdebianpackages and fixed a lot of bugs in the xml info parsing part
........
r373 | bricks | 2010-07-23 18:40:38 +0200 (Fr, 23 Jul 2010) | 2 lines
implemented CacheDb to store copied Packages from build host
........
r374 | bricks | 2010-07-26 10:28:48 +0200 (Mo, 26 Jul 2010) | 2 lines
moved data handling in publishdebianpackages into seperate module
........
r375 | bricks | 2010-07-26 11:42:07 +0200 (Mo, 26 Jul 2010) | 2 lines
fixed some typos
........
r376 | bricks | 2010-07-26 12:12:40 +0200 (Mo, 26 Jul 2010) | 2 lines
first version of incremental copying debian packages to cachedir
........
r377 | bricks | 2010-07-26 14:58:50 +0200 (Mo, 26 Jul 2010) | 2 lines
incremental copying only changed debian packages
........
r378 | bricks | 2010-07-26 15:39:10 +0200 (Mo, 26 Jul 2010) | 3 lines
use md5 instead of hashlib module
(md5 is deprecated in favour of hashlib in python 2.5)
........
r379 | bricks | 2010-07-26 15:46:19 +0200 (Mo, 26 Jul 2010) | 2 lines
make treepkg compatible to python 2.4
........
r380 | bricks | 2010-07-26 15:50:49 +0200 (Mo, 26 Jul 2010) | 2 lines
fixed last commit (db instead of sqlite3)
........
r381 | bricks | 2010-07-27 10:54:15 +0200 (Di, 27 Jul 2010) | 2 lines
fixed wrong import statement
........
r382 | bricks | 2010-07-27 10:54:34 +0200 (Di, 27 Jul 2010) | 2 lines
removed unnecessary debug output
........
r383 | bricks | 2010-07-27 18:28:22 +0200 (Di, 27 Jul 2010) | 2 lines
check if upload hook is empty
........
r384 | bricks | 2010-07-28 09:35:21 +0200 (Mi, 28 Jul 2010) | 2 lines
run rsync only an arch dirs that are present in the current treepkg
........
r385 | bricks | 2010-07-28 09:35:54 +0200 (Mi, 28 Jul 2010) | 2 lines
add helper tool to list content of a cache db
........
r386 | bricks | 2010-07-28 10:33:48 +0200 (Mi, 28 Jul 2010) | 3 lines
fixed a typo
regex fails if option is empty
........
| author | Bjoern Ricks <bricks@intevation.de> |
|---|---|
| date | Fri, 06 Aug 2010 13:28:47 +0000 |
| parents | 56f7da71d41e |
| children | 3d65b3176159 |
comparison
equal
deleted
inserted
replaced
| 427:25d702bb13f9 | 440:383d23c6bfde |
|---|---|
| 1 #! /usr/bin/python | |
| 2 # Copyright (C) 2007 - 2010 by Intevation GmbH | |
| 3 # Authors: | |
| 4 # Bernhard Herzog <bh@intevation.de> | |
| 5 # Bjoern Ricks <bjoern.ricks@intevation.de> | |
| 6 # | |
| 7 # This program is free software under the GPL (>=v2) | |
| 8 # Read the file COPYING coming with the software for details. | |
| 9 | |
| 10 """Publishes selected packages created by treepkg""" | |
| 11 | |
| 12 import os | |
| 13 import os.path | |
| 14 import re | |
| 15 import sys | |
| 16 import shlex | |
| 17 | |
| 18 from optparse import OptionParser | |
| 19 from ConfigParser import SafeConfigParser | |
| 20 | |
| 21 import treepkgcmd | |
| 22 from treepkg.readconfig import read_config_section, convert_bool | |
| 23 from treepkg.run import call, capture_output | |
| 24 from treepkg.cmdexpand import cmdexpand | |
| 25 from treepkg.publish import * | |
| 26 from treepkg.util import md5sum | |
| 27 from treepkg.info.status import TreepkgInfo | |
| 28 from treepkg.info.data import Package | |
| 29 from treepkg.info.data import CacheDb | |
| 30 | |
| 31 EMPTY = re.compile(r'\s*') | |
| 32 | |
| 33 config_desc = ["distribution", "section", "num_newest", | |
| 34 "build_user", "build_host", "build_listpackages", | |
| 35 "publish_user", "publish_host", | |
| 36 ("architectures", shlex.split, "armel i386 source"), | |
| 37 ("after_upload_hook", shlex.split), | |
| 38 ("publish_remove_old_packages", convert_bool), | |
| 39 ("publish_dir", remove_trailing_slashes), | |
| 40 ("cachedb", | |
| 41 lambda s: expand_filename(remove_trailing_slashes(s))), | |
| 42 ("cachedir", | |
| 43 lambda s: expand_filename(remove_trailing_slashes(s)))] | |
| 44 | |
| 45 | |
| 46 def read_config(filename): | |
| 47 if not os.path.exists(filename): | |
| 48 print >>sys.stderr, "Config file %s does not exist" % filename | |
| 49 sys.exit(1) | |
| 50 parser = SafeConfigParser() | |
| 51 parser.read([filename]) | |
| 52 return read_config_section(parser, "publishpackages", config_desc) | |
| 53 | |
| 54 def parse_commandline(): | |
| 55 parser = OptionParser() | |
| 56 parser.set_defaults(config_file=os.path.join(treepkgcmd.topdir, | |
| 57 "publishpackages.cfg"), | |
| 58 quiet=False) | |
| 59 parser.add_option("--config-file", | |
| 60 help=("The configuration file." | |
| 61 " Default is publishpackages.cfg")) | |
| 62 parser.add_option("--dist", | |
| 63 help=("The debian distribution name to use on" | |
| 64 " the publishing system")) | |
| 65 parser.add_option("--section", | |
| 66 help=("The debian distribution section name to use on" | |
| 67 " the publishing system")) | |
| 68 parser.add_option("--track", | |
| 69 help=("The package track whose files are to be" | |
| 70 " published. If not given, files of all tracks" | |
| 71 " will be published")) | |
| 72 parser.add_option("--quiet", action="store_true", | |
| 73 help=("Do not print progress meters or other" | |
| 74 " informational output")) | |
| 75 return parser.parse_args() | |
| 76 | |
| 77 def get_treepkg_info(variables): | |
| 78 runremote = prefix_for_remote_command(variables["build_user"], | |
| 79 variables["build_host"]) | |
| 80 xml = capture_output(cmdexpand("@runremote $build_listpackages" | |
| 81 " --newest=$num_newest" | |
| 82 " --only-successful", | |
| 83 runremote=runremote, | |
| 84 **variables)) | |
| 85 return TreepkgInfo.fromxml(xml) | |
| 86 | |
| 87 def get_binary_arch(arch): | |
| 88 if not arch is None and not arch.startswith("binary") and \ | |
| 89 not arch == "source": | |
| 90 arch = "binary-" + arch | |
| 91 return arch | |
| 92 | |
| 93 def check_package_is_new(packagename, destdir, packagemd5sum): | |
| 94 destpackage = os.path.join(destdir, packagename) | |
| 95 if not os.path.isfile(destpackage): | |
| 96 return True | |
| 97 destmd5sum = md5sum(destpackage) | |
| 98 return (destmd5sum != packagemd5sum) | |
| 99 | |
| 100 def get_md5sum(packageinfo): | |
| 101 md5sum = "" | |
| 102 if packageinfo: | |
| 103 for checksum in packageinfo.checksums: | |
| 104 if checksum.type == "md5": | |
| 105 md5sum = checksum.checksum | |
| 106 break | |
| 107 return md5sum | |
| 108 | |
| 109 def sort_trackname_arch(a, b): | |
| 110 if a.trackname < b.trackname: return -1 | |
| 111 if a.trackname > b.trackname: return +1 | |
| 112 return cmp(a.arch, b.arch) | |
| 113 | |
| 114 def copy_files_to_destdir(destdir, files, variables, quiet = False): | |
| 115 scp_flags = [] | |
| 116 if quiet: | |
| 117 scp_flags.append("-q") | |
| 118 | |
| 119 if not os.path.exists(destdir): | |
| 120 os.makedirs(destdir) | |
| 121 if files: | |
| 122 if variables["build_host"]: | |
| 123 userhost = "%(build_user)s@%(build_host)s:" % variables | |
| 124 files = [userhost + filename for filename in files] | |
| 125 # scp the packages to the cache dir | |
| 126 call(cmdexpand("scp -p @scp_flags @files $cachedir/", files=files, | |
| 127 scp_flags=scp_flags, cachedir=destdir)) | |
| 128 | |
| 129 def remove_old_packages(cachedb, newpackages, quiet): | |
| 130 newfiles = [package.filename for package in newpackages] | |
| 131 oldpackages = cachedb.get_old_packages(newfiles) | |
| 132 for package in oldpackages: | |
| 133 # better check if the file really exists | |
| 134 if os.path.isfile(package.filename): | |
| 135 if not quiet: | |
| 136 print "removing file %s" % package.filename | |
| 137 os.remove(package.filename) | |
| 138 cachedb.remove_packages(oldpackages) | |
| 139 | |
| 140 def copy_packages_to_destdir(cachedb, dir, packages, variables, quiet = False): | |
| 141 packages.sort(cmp=sort_trackname_arch) | |
| 142 package = packages[0] | |
| 143 trackname = package.trackname | |
| 144 arch = package.arch | |
| 145 destdir = os.path.join(dir, arch, trackname) | |
| 146 files = [] | |
| 147 for package in packages: | |
| 148 cachedb.add_package(package) | |
| 149 if package.trackname != trackname or \ | |
| 150 package.arch != arch: | |
| 151 copy_files_to_destdir(destdir, files, variables, quiet) | |
| 152 trackname = package.trackname | |
| 153 arch = package.arch | |
| 154 destdir = os.path.join(dir, arch, trackname) | |
| 155 files = [] | |
| 156 # add only to copy files list if the packages differ | |
| 157 if check_package_is_new(package.name, destdir, package.md5sum): | |
| 158 files.append(package.sourcepath) | |
| 159 if not quiet: | |
| 160 print "copy new file: %s" % package.name | |
| 161 copy_files_to_destdir(destdir, files, variables, quiet) | |
| 162 | |
| 163 def copy_to_cachedir(variables, track, revision, quiet = False, architectures=None): | |
| 164 cachedir = variables["cachedir"] | |
| 165 cachdebfilename = variables["cachedb"] | |
| 166 if not quiet: | |
| 167 print "using cachedb: %s" % cachdebfilename | |
| 168 cachedb = CacheDb(cachdebfilename) | |
| 169 newpackages = [] | |
| 170 treepkginfo = get_treepkg_info(variables) | |
| 171 #allowedarchs = set([]) # contains all wanted architectures (incl. source) | |
| 172 allarchs = set([]) # contains all present architectures (incl. source) | |
| 173 binaryallpackages = [] | |
| 174 # change e.g. armel in binary-armel | |
| 175 if not architectures is None: | |
| 176 allowedarchs = set([get_binary_arch(a) for a in architectures]) | |
| 177 else: | |
| 178 allowedarchs = set([]) | |
| 179 for track in treepkginfo.tracks: | |
| 180 for rev in track.revisions: | |
| 181 for packageinfo in rev.packages: | |
| 182 arch = get_binary_arch(packageinfo.arch) | |
| 183 if packageinfo.type == "binary": | |
| 184 # skip other files | |
| 185 if packageinfo.arch is None: | |
| 186 continue | |
| 187 # handle binary-all | |
| 188 if arch == "binary-all": | |
| 189 # add trackname for subdir name | |
| 190 packageinfo.trackname = track.name | |
| 191 binaryallpackages.append(packageinfo) | |
| 192 continue | |
| 193 allarchs.add(arch) | |
| 194 elif packageinfo.type == "source": | |
| 195 arch = packageinfo.type | |
| 196 # only copy requested archs | |
| 197 if len(allowedarchs) == 0 or \ | |
| 198 arch in allowedarchs: | |
| 199 filename = os.path.join(cachedir, arch, track.name, | |
| 200 packageinfo.name) | |
| 201 newpackage = Package(filename, track.name, packageinfo.name, | |
| 202 packageinfo.path, arch, | |
| 203 get_md5sum(packageinfo)) | |
| 204 newpackages.append(newpackage) | |
| 205 # copy binary-all packages | |
| 206 sourcearch = set(["source"]) | |
| 207 if len(allowedarchs) == 0: | |
| 208 binallarchs = allarchs - sourcearch | |
| 209 elif len(allarchs) == 0: | |
| 210 binallarchs = allowedarchs - sourcearch | |
| 211 else: | |
| 212 binallarchs = (allowedarchs & allarchs) - sourcearch | |
| 213 for packageinfo in binaryallpackages: | |
| 214 for arch in binallarchs: | |
| 215 filename = os.path.join(cachedir, arch, packageinfo.trackname, | |
| 216 packageinfo.name) | |
| 217 newpackage = Package(filename, packageinfo.trackname, packageinfo.name, | |
| 218 packageinfo.path, arch, get_md5sum(packageinfo)) | |
| 219 newpackages.append(newpackage) | |
| 220 copy_packages_to_destdir(cachedb, cachedir, newpackages, variables, quiet) | |
| 221 remove_old_packages(cachedb, newpackages, quiet) | |
| 222 return binallarchs | |
| 223 | |
| 224 def publish_packages(config_filename, track, revision, dist, section, quiet): | |
| 225 config = read_config(config_filename) | |
| 226 | |
| 227 if dist is None: | |
| 228 dist = config["distribution"] | |
| 229 if section is None: | |
| 230 section = config["section"] | |
| 231 | |
| 232 architectures = config["architectures"] | |
| 233 allarchs = copy_to_cachedir(config, track, revision, quiet, architectures) | |
| 234 for arch in allarchs: | |
| 235 copy_to_publishdir(config, dist, section, arch, quiet) | |
| 236 | |
| 237 # update apt archive | |
| 238 if not config["after_upload_hook"] or \ | |
| 239 not EMPTY.match(config["after_upload_hook"]): | |
| 240 if not quiet: | |
| 241 print "running after upload hook" | |
| 242 call(config["after_upload_hook"]) | |
| 243 | |
| 244 def main(): | |
| 245 options, args = parse_commandline() | |
| 246 revision = None # for future use cases | |
| 247 publish_packages(options.config_file, options.track, revision, | |
| 248 options.dist, options.section, options.quiet) | |
| 249 | |
| 250 if __name__ == "__main__": | |
| 251 main() |