treepkg/treepkg: contrib/bin/delete-old-debs.py annotate

annotate contrib/bin/delete-old-debs.py @ 538:6fb5e8b74414

contrib: Do a more conservative approach (full sorting) to figure out the youngest debs. This _much_ slower but it works correctly. TODO: make the heapq.nsmallest() work correct.

author	Sascha Teichmann <teichmann@intevation.de>
date	Mon, 10 Jan 2011 14:58:51 +0000
parents	aeccb5774939
children

rev	line source
535 fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	1 #!/usr/bin/env python
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	2 # -- coding: UTF-8 --
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	3 #
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	4 # Copyright (C) 2011 by Intevation GmbH
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	5 # Authors:
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	6 # Sascha L. Teichmann <sascha.teichmann@intevation.de>
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	7 #
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	8 # This program is free software under the GPL (>=v2)
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	9 # Read the file COPYING coming with the software for details.
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	10
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	11 import sys
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	12 import os
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	13 import re
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	14
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	15 import subprocess
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	16 import logging
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	17
538 6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	18 #from heapq import nsmallest
535 fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	19
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	20 from optparse import OptionParser
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	21
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	22 log = logging.getLogger(__name__)
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	23 log.setLevel(logging.WARNING)
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	24 log.addHandler(logging.StreamHandler(sys.stderr))
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	25
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	26 DEFAULT_KEEP = 3
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	27
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	28 FIELD = re.compile("([a-zA-Z]+):\s*(.+)")
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	29
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	30 class DebCmp(object):
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	31 """Helper class to make deb files comparable
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	32 by there versions.
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	33 """
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	34
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	35 def __init__(self, version, path):
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	36 self.version = version
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	37 self.path = path
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	38
538 6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	39 def __cmp__(self, other):
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	40 if self.version == other.version:
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	41 return 0
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	42 # switch lt and gt to reverse order in heap
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	43 if (subprocess.call([
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	44 "dpkg", "--compare-versions",
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	45 self.version, "gt", other.version]) == 0):
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	46 return -1
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	47 if (subprocess.call([
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	48 "dpkg", "--compare-versions",
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	49 self.version, "lt", other.version]) == 0):
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	50 return +1
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	51 return 0
535 fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	52
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	53
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	54 def deb_info(deb, fields=["Package", "Version"]):
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	55 """Extract some meta info from a deb file."""
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	56 po = subprocess.Popen(
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	57 ["dpkg-deb", "-f", deb] + fields,
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	58 stdout=subprocess.PIPE)
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	59 out = po.communicate()[0]
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	60 return dict([m.groups()
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	61 for m in map(FIELD.match, out.splitlines()) if m])
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	62
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	63
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	64 def oldest_debs(deb_dir, keep=DEFAULT_KEEP):
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	65 """Given directory containing deb files this function
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	66 returns the files that are older than the youngest
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	67 keep-th per package.
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	68 """
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	69
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	70 log.info("scanning dir '%s'" % deb_dir)
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	71
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	72 packages = {}
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	73
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	74 num = 1
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	75 for f in os.listdir(deb_dir):
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	76 if not f.endswith(".deb"): continue
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	77 deb = os.path.join(deb_dir, f)
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	78 if not os.path.isfile(deb): continue
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	79 info = deb_info(deb)
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	80 packages.setdefault(info['Package'], []).append(
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	81 DebCmp(info['Version'], deb))
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	82 if (num % 10) == 0:
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	83 log.info("%d debs found" % (num-1))
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	84 num += 1
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	85
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	86 if log.isEnabledFor(logging.INFO):
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	87 log.info("%d debs found" % (num-1))
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	88 log.info("number packages: %s" % len(packages))
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	89
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	90 for package, debs in packages.iteritems():
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	91 if len(debs) > keep:
538 6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	92 debs.sort()
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	93 for deb in debs[keep:]:
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	94 yield deb.path
535 fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	95
538 6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	96 ## full sorting is not required
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	97 #stay = frozenset([d.path for d in nsmallest(keep, debs)])
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	98
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	99 #for deb in debs:
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	100 # if deb.path not in stay:
6fb5e8b74414 contrib: Do a more conservative approach (full sorting) to figure out Sascha Teichmann <teichmann@intevation.de> parents: 537 diff changeset	101 # yield deb.path
535 fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	102
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	103
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	104 def main():
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	105 usage = "usage: %prog [options] dir ..."
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	106 parser = OptionParser(usage=usage)
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	107 parser.add_option(
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	108 "-v", "--verbose", action="store_true",
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	109 dest="verbose",
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	110 help="verbose output")
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	111 parser.add_option(
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	112 "-d", "--dry-run", action="store_true",
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	113 dest="dry_run",
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	114 help="don't remove the old deb files")
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	115 parser.add_option(
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	116 "-k", "--keep", action="store",
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	117 dest="keep", type="int", default=DEFAULT_KEEP,
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	118 help="number of files to keep. Default: %d" % DEFAULT_KEEP)
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	119
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	120 options, args = parser.parse_args()
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	121
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	122 remove = options.dry_run and (lambda x: None) or os.remove
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	123 keep = max(1, options.keep)
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	124 if options.verbose: log.setLevel(logging.INFO)
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	125
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	126 for deb_dir in args:
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	127
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	128 if not os.path.isdir(deb_dir):
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	129 log.warn("'%s' is not a directory" % deb_dir)
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	130 continue
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	131
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	132 for deb in oldest_debs(deb_dir, keep):
537 aeccb5774939 contrib: Small bug fixes in delete-old-debs.py. Sascha Teichmann <teichmann@intevation.de> parents: 536 diff changeset	133 log.info("remove '%s'" % deb)
535 fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	134 remove(deb)
537 aeccb5774939 contrib: Small bug fixes in delete-old-debs.py. Sascha Teichmann <teichmann@intevation.de> parents: 536 diff changeset	135 changes = deb[:-3] + "changes"
535 fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	136 if os.path.isfile(changes):
536 8a61185a3357 contrib: When running in verbose mode list files to be deleted. Sascha Teichmann <teichmann@intevation.de> parents: 535 diff changeset	137 log.info("remove '%s'" % changes)
535 fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	138 remove(changes)
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	139
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	140
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	141 if __name__ == "__main__":
fb7a900a649f contrib: Added script to remove old deb files from directories. Sascha Teichmann <teichmann@intevation.de> parents: diff changeset	142 main()

Mercurial > treepkg > treepkg

annotate contrib/bin/delete-old-debs.py @ 538:6fb5e8b74414