annotate flys-artifacts/src/main/java/de/intevation/flys/artifacts/datacage/templating/BuilderPool.java @ 5779:ebec12def170

Datacage: Add a pool of builders to make it multi threadable. XML DOM is not thread safe. Therefore the old implementation only allowed one thread to use the builder at a time. As the complexity of the configuration has increased over time this has become a bottleneck of the whole application because it took quiet some time to build a result. Furthermore the builder code path is visited very frequent. So many concurrent requests were piled up resulting in long waits for the users. To mitigate this problem a round robin pool of builders is used now. Each of the pooled builders has an independent copy of the XML template and can be run in parallel. The number of builders is determined by the system property 'flys.datacage.pool.size'. It defaults to 4.
author Sascha L. Teichmann <teichmann@intevation.de>
date Sun, 21 Apr 2013 12:48:09 +0200
parents
children d38004f0c52f
rev   line source
5779
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
1 package de.intevation.flys.artifacts.datacage.templating;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
2
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
3 import java.util.ArrayDeque;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
4 import java.util.Deque;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
5 import java.util.List;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
6 import java.util.Map;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
7
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
8 import java.sql.SQLException;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
9
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
10 import javax.xml.transform.Transformer;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
11 import javax.xml.transform.TransformerConfigurationException;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
12 import javax.xml.transform.TransformerException;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
13 import javax.xml.transform.TransformerFactory;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
14
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
15 import javax.xml.transform.dom.DOMResult;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
16 import javax.xml.transform.dom.DOMSource;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
17
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
18 import org.apache.log4j.Logger;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
19
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
20 import org.w3c.dom.Document;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
21 import org.w3c.dom.Node;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
22
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
23 /** A little round robin pool of builders to mitigate
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
24 * the fact the XML DOM documents are not thread safe.
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
25 */
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
26 public class BuilderPool
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
27 {
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
28 private static Logger log = Logger.getLogger(BuilderPool.class);
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
29
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
30 private static final int DEFAULT_POOL_SIZE = 4;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
31
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
32 private static final int POOL_SIZE = Math.max(
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
33 Integer.getInteger("flys.datacage.pool.size", DEFAULT_POOL_SIZE), 1);
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
34
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
35 private Deque<Builder> pool;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
36
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
37 public BuilderPool(Document document) {
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
38 this(document, POOL_SIZE);
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
39 }
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
40
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
41 public BuilderPool(Document document, int poolSize) {
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
42
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
43 if (log.isDebugEnabled()) {
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
44 log.debug("Create build pool with " + poolSize + " elements.");
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
45 }
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
46
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
47 pool = new ArrayDeque<Builder>(poolSize);
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
48 for (int i = 0; i < poolSize; ++i) {
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
49 Document doc = i > 0 // Clone all but the first.
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
50 ? cloneDocument(document)
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
51 : document;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
52 pool.add(new Builder(doc));
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
53 }
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
54 }
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
55
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
56 private final static Document cloneDocument(Document document) {
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
57 try {
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
58 TransformerFactory tfactory = TransformerFactory.newInstance();
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
59 Transformer xformer = tfactory.newTransformer();
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
60 DOMSource src = new DOMSource(document);
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
61 DOMResult dst = new DOMResult();
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
62 xformer.transform(src, dst);
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
63 return (Document)dst.getNode();
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
64 }
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
65 catch (TransformerConfigurationException tce) {
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
66 log.error(tce);
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
67 }
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
68 catch (TransformerException te) {
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
69 log.error(te);
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
70 }
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
71
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
72 log.error(
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
73 "Returning original DOM document. " +
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
74 "This will result in threading errors!");
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
75
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
76 return document;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
77 }
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
78
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
79 public void build(
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
80 List<Builder.NamedConnection> connections,
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
81 Node output,
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
82 Map<String, Object> parameters
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
83 )
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
84 throws SQLException
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
85 {
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
86 Builder builder;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
87 synchronized (pool) {
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
88 try {
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
89 while ((builder = pool.poll()) == null) {
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
90 pool.wait();
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
91 }
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
92 }
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
93 catch (InterruptedException ie) {
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
94 log.debug("Waiting for builder interrupted. Build canceled.");
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
95 return;
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
96 }
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
97 }
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
98 try {
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
99 builder.build(connections, output, parameters);
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
100 }
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
101 finally {
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
102 synchronized (pool) {
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
103 pool.add(builder);
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
104 pool.notify();
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
105 }
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
106 }
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
107 }
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
108 }
ebec12def170 Datacage: Add a pool of builders to make it multi threadable.
Sascha L. Teichmann <teichmann@intevation.de>
parents:
diff changeset
109 // vim:set ts=4 sw=4 si et sta sts=4 fenc=utf-8 :

http://dive4elements.wald.intevation.org