roundup-cc: collect_issues.py annotate

annotate collect_issues.py @ 17:adca5b3780d2

Add collecting no-prio issues. Restructure. * Move usage examples from collect_issues.py to examples/. Also add a section about this to the readme. This should make it easier to understand how it all works together. As we have two examples now, hardcode the database filenames. * Add a function to create the necessary database command from a list of columns. This makes it easier to work with tracker that have customized their priorities, * Add an example how to collect no-prio issues in a database. Note that the display part is not included. * Cleanup roundup_content_data by removing db file name suggestions and unused "reference db" command. This could be re-added for specific installations if they need it, however it probably should be separated as configuration.

author	Bernhard Reiter <bernhard@intevation.de>
date	Mon, 09 Jul 2018 12:28:28 +0200
parents	d0c439d1e833
children	3bb3d9a9f1b7

rev	line source
1 2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	1 #!/usr/bin/env python3
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	2 """ Fetch issues from a roundup-tracker and save them in a databse.
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	3
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	4 author: Sascha L. Teichmann <sascha.teichmann@intevation.de>
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	5 author: Bernhard Reiter <bernhard@intevation.de>
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	6 author: Sean Engelhardt <sean.engelhardt@intevation.de>
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	7
15 d0c439d1e833 Prepare for tracker entries without priority. Bernhard Reiter <bernhard@intevation.de> parents: 9 diff changeset	8 (c) 2010, 2015, 2018 by Intevation GmbH
1 2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	9
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	10 This is Free Software unter the terms of the
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	11 GNU GENERAL PUBLIC LICENSE Version 3 or later.
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	12 See http://www.gnu.org/licenses/gpl-3.0.txt for details
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	13
17 adca5b3780d2 Add collecting no-prio issues. Restructure. Bernhard Reiter <bernhard@intevation.de> parents: 15 diff changeset	14 For usage see examples/.
1 2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	15 """
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	16
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	17 import http.cookiejar
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	18 import urllib.parse
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	19 import urllib.request
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	20 import csv
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	21 import io
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	22 import sqlite3 as db
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	23 import os
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	24
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	25
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	26 CHECK_ROUNDUP_ORDER = "priority?@action=export_csv&@columns=id,order"
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	27 CHECK_ROUNDUP_SEARCH_VALUES = "status?@action=export_csv&@columns=id&@filter=open&open=1"
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	28 SEARCH_ROUNDUP = "issue?@action=export_csv&@columns=priority&@filter=status&@pagesize=500&@startwith=0&status=-1,{search_values}"
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	29
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	30
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	31 def connect_to_server(params, baseurl):
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	32 enc_data = urllib.parse.urlencode(params).encode()
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	33 cj = http.cookiejar.CookieJar()
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	34 opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	35 req = urllib.request.Request(url=baseurl, data=enc_data)
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	36 opener.open(req)
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	37 return opener
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	38
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	39
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	40 def get_csv_from_server(opener, roundup_url, sub_url):
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	41 csv_req = urllib.request.Request(url=roundup_url+sub_url)
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	42 f = opener.open(csv_req)
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	43 csv_reader = csv.DictReader(io.TextIOWrapper(f))
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	44 return csv_reader
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	45
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	46
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	47 def set_search_paramters_on_URL(url, search_param_csv):
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	48
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	49 id_list = []
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	50
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	51 for row in search_param_csv:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	52 id_list.append(row["id"])
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	53
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	54 new_url = url.format(search_values = ",".join(id_list))
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	55
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	56 return new_url
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	57
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	58
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	59 def check_create_database(database_file, sql_create_db):
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	60 if not os.path.isfile(database_file):
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	61 con = None
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	62 cur = None
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	63 try:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	64 con = db.connect(database_file)
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	65 cur = con.cursor()
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	66 try:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	67 cur.execute(sql_create_db)
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	68 con.commit()
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	69 os.chmod(database_file, 0o644)
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	70 except:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	71 con.rollback()
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	72 raise
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	73 finally:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	74 if cur:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	75 cur.close()
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	76 if con:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	77 con.close()
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	78
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	79
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	80 def issues_to_quantities(issue_csv, columns, orders_csv):
15 d0c439d1e833 Prepare for tracker entries without priority. Bernhard Reiter <bernhard@intevation.de> parents: 9 diff changeset	81 """Count issues per priority.
1 2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	82
15 d0c439d1e833 Prepare for tracker entries without priority. Bernhard Reiter <bernhard@intevation.de> parents: 9 diff changeset	83 Returns: a list of ints, containing how often a prio occurred [:-1]
d0c439d1e833 Prepare for tracker entries without priority. Bernhard Reiter <bernhard@intevation.de> parents: 9 diff changeset	84 in order of the priorities, with the last being the "None" prio
d0c439d1e833 Prepare for tracker entries without priority. Bernhard Reiter <bernhard@intevation.de> parents: 9 diff changeset	85 """
d0c439d1e833 Prepare for tracker entries without priority. Bernhard Reiter <bernhard@intevation.de> parents: 9 diff changeset	86
d0c439d1e833 Prepare for tracker entries without priority. Bernhard Reiter <bernhard@intevation.de> parents: 9 diff changeset	87 quantities = [0] * (len(columns) +1)
1 2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	88 order_dict = {}
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	89
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	90 #convert the csv-dict reader to real dict
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	91 for row in orders_csv:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	92 order_dict[row["id"]] = int(float(row["order"])) # int(float()) because the order-value is indeed "1.0, 2.0" etc
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	93
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	94 for issue in issue_csv:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	95 priority = issue["priority"]
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	96
2 3e9f4a6803d1 replaced funtion with string.isdigit. sean parents: 1 diff changeset	97 if priority.isdigit() == True :
1 2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	98 quantities[order_dict[priority] -1 ] += 1
15 d0c439d1e833 Prepare for tracker entries without priority. Bernhard Reiter <bernhard@intevation.de> parents: 9 diff changeset	99 else: # no priority set
d0c439d1e833 Prepare for tracker entries without priority. Bernhard Reiter <bernhard@intevation.de> parents: 9 diff changeset	100 quantities[-1] += 1
1 2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	101
15 d0c439d1e833 Prepare for tracker entries without priority. Bernhard Reiter <bernhard@intevation.de> parents: 9 diff changeset	102 # print("quantities : " + str(quantities))
1 2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	103
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	104 return quantities
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	105
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	106
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	107 def save_issues_to_db(quantities, database_file, sql_create_db, sql_insert_in_db):
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	108 check_create_database(database_file, sql_create_db)
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	109
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	110 cur = None
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	111 con = None
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	112
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	113 try:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	114 con = db.connect(database_file)
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	115 cur = con.cursor()
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	116 try:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	117 cur.execute(sql_insert_in_db, quantities)
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	118 con.commit()
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	119 except:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	120 con.rollback()
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	121 raise
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	122 finally:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	123 if cur:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	124 cur.close()
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	125 if con:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	126 con.close()
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	127
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	128
15 d0c439d1e833 Prepare for tracker entries without priority. Bernhard Reiter <bernhard@intevation.de> parents: 9 diff changeset	129 def save_stats_in_db(login_parmeters, baseurl, db_file, columns, sql_create_db, sql_insert_in_db, searchurl=False, include_no_prio=False):
1 2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	130 try:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	131
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	132 opener = connect_to_server(login_parmeters, baseurl)
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	133
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	134 search_operators_csv = get_csv_from_server(opener, baseurl, CHECK_ROUNDUP_SEARCH_VALUES)
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	135 order_csv = get_csv_from_server(opener, baseurl, CHECK_ROUNDUP_ORDER)
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	136
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	137 if searchurl == False:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	138 formated_search_url = set_search_paramters_on_URL(SEARCH_ROUNDUP, search_operators_csv)
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	139 else:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	140 formated_search_url = searchurl
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	141
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	142 current_issues_csv = get_csv_from_server(opener, baseurl, formated_search_url)
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	143
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	144 opener.close()
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	145
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	146 quantities = issues_to_quantities(current_issues_csv, columns, order_csv)
15 d0c439d1e833 Prepare for tracker entries without priority. Bernhard Reiter <bernhard@intevation.de> parents: 9 diff changeset	147 if not include_no_prio:
d0c439d1e833 Prepare for tracker entries without priority. Bernhard Reiter <bernhard@intevation.de> parents: 9 diff changeset	148 quantities = quantities[:-1]
1 2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	149
9 e95f7bee8643 Really safe data. Bernhard Reiter <bernhard@intevation.de> parents: 2 diff changeset	150 save_issues_to_db(quantities, db_file, sql_create_db, sql_insert_in_db)
1 2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	151
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	152 except urllib.error.URLError as e:
2df45f6ecd81 new appereance (solid and dotted lines), resonsive layout, new legend, sean parents: diff changeset	153 print("No Valid Connection to server : " + baseurl + "\nerror: " + str(e))

Mercurial > roundup-cc

annotate collect_issues.py @ 17:adca5b3780d2