view flys-backend/ChangeLog @ 202:29a408f80a89

Finished import of WSTs. flys-backend/trunk@1553 c6561f87-3c4e-4783-a992-168aeb5c3f6f
author Sascha L. Teichmann <sascha.teichmann@intevation.de>
date Wed, 23 Mar 2011 16:33:36 +0000
parents 3169b559ca3c
children bfee0e05b4e7
line wrap: on
line source
2011-03-22	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	Finished import of WSTs.

	TODO 1: Speed it up! It takes on a high end machine over 7(!)
	        minutes only for the data of the Saar.
	TODO 2: Double precision floating point representations produced
	        by the the parsers leed to unique constraint violations
			in the backend on a second run. So the import is currently
			only working on freshly initialized data bases.
			More consequent working with BigDecimal and some
			rounding may be of help here.

	* src/main/java/de/intevation/flys/model/WstColumnValue.java:
	  Added convinience constructors.

	* src/main/java/de/intevation/flys/importer/ImportWstColumnValue.java:
	  Added getPeer() method.

	* src/main/java/de/intevation/flys/importer/ImportWstColumn.java:
	  Add a list of the ImportWstColumnValues produced by the WST parser.

	* src/main/java/de/intevation/flys/importer/WstParser.java: Add
	  the (km, w) values to the ImportWstColumns.

2011-03-22	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/importer/WstParser.java:
	  Build models for wsts, wst columns and q ranges and
	  store them in the backend. TODO: store the w values.

	* src/main/java/de/intevation/flys/model/WstQRange.java
	  src/main/java/de/intevation/flys/model/Wst.java,
	  src/main/java/de/intevation/flys/model/Range.java,
	  src/main/java/de/intevation/flys/model/WstColumnQRange.java,
	  src/main/java/de/intevation/flys/model/WstColumn.java:
	  Added convinience constructors.

	* src/main/java/de/intevation/flys/importer/ImportWstQRange.java,
	  src/main/java/de/intevation/flys/importer/ImportWst.java,
	  src/main/java/de/intevation/flys/importer/ImportRiver.java,
	  src/main/java/de/intevation/flys/importer/ImportWstColumnQRange.java,
	  src/main/java/de/intevation/flys/importer/ImportWstColumn.java,
	  src/main/java/de/intevation/flys/importer/ImportRange.java:
	  Added getPeer() methods.

2011-03-22	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/importer/WstParser.java:
	  The unit extraction in the WST parser of desktop FLYS
	  is broken! Add a hack here to repair this for our
	  importer. Desktop FLYS needs a fix, too!

2011-03-22	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/importer/WstParser.java:
	  Ported some stuff ver from WSTSource.java of desktop flys to
	  parse WST files. TODO: create instances of the import models.

	* src/main/java/de/intevation/flys/utils/StringUtil.java:
	  Copied from desktop flys. Used for some string operations
	  in WST parser.

	* src/main/java/de/intevation/flys/importer/Importer.java:
	  Added system property 'flys.backend.importer.dry.run'.
	  Set to true only the parsing is done and no writing
	  to the backend. Default: false.

	*  src/main/java/de/intevation/flys/App.java,
	   src/main/java/de/intevation/flys/model/MainValueType.java:
	   Removed needless imports.

2011-03-22  Ingo Weinzierl <ingo@intevation.de>

	* src/main/java/de/intevation/flys/model/DischargeTableValue.java,
	  src/main/java/de/intevation/flys/model/DischargeTable.java: Added new
	  constructors.

	* src/main/java/de/intevation/flys/importer/AtFileParser.java: New. This
	  parser is used to '*.at' files.

	* src/main/java/de/intevation/flys/importer/ImportGauge.java: Added code to
	  import discharge tables.

	* src/main/java/de/intevation/flys/importer/ImportDischargeTableValue.java,
	  src/main/java/de/intevation/flys/importer/ImportDischargeTable.java: New.
	  Helper models for import discharge tables.

2011-03-22	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/importer/WstParser.java,
	  src/main/java/de/intevation/flys/importer/ImportRiver.java:
	  Added stub for WST parser.

2011-03-22	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/importer/ImportWstQRange.java,
	  src/main/java/de/intevation/flys/importer/ImportWst.java,
	  src/main/java/de/intevation/flys/importer/ImportWstColumnValue.java,
	  src/main/java/de/intevation/flys/importer/ImportWstColumnQRange.java,
	  src/main/java/de/intevation/flys/importer/ImportWstColumn.java:
	  Added importer helper model stubs for WST imports.
	
2011-03-21	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	Second part of parsing/storing main values. Should be finished now.

	* src/main/java/de/intevation/flys/importer/ImportNamedMainValue.java,
	  src/main/java/de/intevation/flys/importer/ImportMainValue.java:
	  New. Helper models for import main values,

	* src/main/java/de/intevation/flys/model/MainValue.java,
	  src/main/java/de/intevation/flys/model/NamedMainValue.java:
	  Added convinience constructors.

	* src/main/java/de/intevation/flys/importer/ImportGauge.java:
	  Write main values to backend, too.

	* src/main/java/de/intevation/flys/importer/StaFileParser.java:
	  Build importer models for main values.

	* src/main/java/de/intevation/flys/importer/ImportMainValueType.java:
	  Data was called 'value'. Now it is 'name' to fit the
	  schema.

2011-03-21	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* doc/schema/postgresql.sql: Fixed wrong unique constraint.

	* src/main/java/de/intevation/flys/importer/ImportRiver.java:
	  Added some logging when storing gauges.

2011-03-21	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/model/Gauge.java:
	  Add forgotten column river_id.

	* src/main/java/de/intevation/flys/importer/ImportGauge.java:
	  Small HQL fix.

2011-03-21	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/model/Gauge.java:
	  Added convinience contructor.

	* src/main/java/de/intevation/flys/importer/ImportGauge.java:
	  Fixed getPeer() method.

	* src/main/java/de/intevation/flys/importer/StaFileParser.java:
	  Fixed parsing of STA files.

2011-03-21	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/importer/ImportRiver.java,
	  src/main/java/de/intevation/flys/importer/ImportGauge.java:
	  Propagate river into storing of gauges.

2011-03-21	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/importer/Importer.java:
	  Added code to store rivers not only annotations.

	* src/main/java/de/intevation/flys/importer/ImportRiver.java:
	  Added stub code to write gauges.

2011-03-17	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	First part of parsing main values.

	* src/main/java/de/intevation/flys/App.java: Commented out
	  creation of dummy rivers.

	* src/main/java/de/intevation/flys/model/NamedMainValues.java:
	  Moved to NamedMainValue.

	* src/main/java/de/intevation/flys/model/NamedMainValue.java:
	  New. Formerly NamedMainValues.

	* src/main/java/de/intevation/flys/model/MainValue.java:
	  New. Forgotten part of the model.

	* src/main/java/de/intevation/flys/model/MainValueType.java:
	  Data is String not BigDecimal

	* src/main/java/de/intevation/flys/model/Range.java: Removed
	  contructor with double arguments. Using BigDecimal now.

	* src/main/java/de/intevation/flys/importer/PegelGltParser.java:
	  Propagate BigDecimal usage.

	* src/main/java/de/intevation/flys/importer/Importer.java:
	  Removed needless import. Added TODO

	* src/main/java/de/intevation/flys/importer/ImportRiver.java:
	  Parse the dependencies of the gauges, too.

	* src/main/java/de/intevation/flys/importer/StaFileParser.java:
	  New. Parser for STA files.

	* src/main/java/de/intevation/flys/importer/ImportGauge.java:
	  Call STA file parser.

	* src/main/java/de/intevation/flys/importer/AnnotationsParser.java,
	  src/main/java/de/intevation/flys/importer/ImportRange.java:
	  Uses BigDecimal now.

	* src/main/java/de/intevation/flys/importer/ImportAttribute.java:
	  Fixed wrong type cast in equals.

	* src/main/java/de/intevation/flys/importer/ImportMainValueType.java:
	  New. Helper model for importing main value types.

	* src/main/java/de/intevation/flys/backend/SessionFactoryProvider.java:
	  Register forgotten MainValue model.

2011-03-17	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	Store annotations in backend.

	* src/main/java/de/intevation/flys/model/Annotation.java:
	  New convinience constructor.

	* src/main/java/de/intevation/flys/model/River.java:
	  Added toString() method.

	* src/main/java/de/intevation/flys/model/Range.java:
	  Fixed nasty mistake in @OneToOne annotatation.
	  New convinience constructors.

	* src/main/java/de/intevation/flys/importer/ImportPosition.java
	  src/main/java/de/intevation/flys/importer/ImportAnnotation.java,
	  src/main/java/de/intevation/flys/importer/ImportRange.java
	  src/main/java/de/intevation/flys/importer/ImportAttribute.java:
	  Make storing to backend work. It's a bit too slow. :-/

	* src/main/java/de/intevation/flys/importer/ImportRiver.java:
	  Fetch peer from backend. Added method to store annotations.
	* src/main/java/de/intevation/flys/importer/Importer.java:
	  Stored annotations into backend. More eloquent SQL exception
	  handling.

2011-03-17	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/model/Attribute.java,
	  src/main/java/de/intevation/flys/model/Position.java:
	  Added convinience constructors.

	* src/main/java/de/intevation/flys/importer/ImportPosition.java,
	  src/main/java/de/intevation/flys/importer/ImportAttribute.java:
	  Bound them to there backend peers.

2011-03-17	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/importer/AnnotationsParser.java:
	  New. Added parser to read *.KM files.

	* src/main/java/de/intevation/flys/importer/ImportPosition.java,
	  src/main/java/de/intevation/flys/importer/PegelGltParser.java,
	  src/main/java/de/intevation/flys/importer/ImportRiver.java,
	  src/main/java/de/intevation/flys/importer/ImportAnnotation.java,
	  src/main/java/de/intevation/flys/importer/ImportRange.java,
	  src/main/java/de/intevation/flys/importer/InfoGewParser.java,
	  src/main/java/de/intevation/flys/importer/ImportAttribute.java:
	  Adjusted to load the annotations from *.KM files.

2011-03-17	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/importer/ImportPosition.java,
	  src/main/java/de/intevation/flys/importer/ImportRange.java,
	  src/main/java/de/intevation/flys/importer/ImportAttribute.java,
	  src/main/java/de/intevation/flys/importer/ImportAnnotation.java:
	  New helper models for import.

	* src/main/java/de/intevation/flys/importer/PegelGltParser.java,
	  src/main/java/de/intevation/flys/importer/ImportGauge.java:
	  Use new models.

2011-03-17	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/importer/PegelGltParser.java:
	  New. Parser for PEGEL.GLT files.

	* src/main/java/de/intevation/flys/importer/ImportGauge.java:
	  New. Import model for gauges.

	* src/main/java/de/intevation/flys/utils/DBCPConnectionProvider.java:
	  Removed needless imports.

	* src/main/java/de/intevation/flys/importer/ImportRiver.java:
	  Added method to parse the gauges.

	* src/main/java/de/intevation/flys/importer/InfoGewParser.java:
	  Trigger pegel glt file parsing.

2011-03-17	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/importer/Importer.java:
	  Used thread local pattern to make sharing of session easier.

2011-03-17	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/importer/Importer.java:
	  Fixed error in HQL statement.

2011-03-17	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/utils/DBCPConnectionProvider.java:
	  Commented out a debug block because it leaks the db password.

2011-03-16	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/importer/InfoGewParser.java:
	  Expose imported rivers.

	* src/main/java/de/intevation/flys/importer/InfoGewParser.java:
	  Store imported rivers into database. Needs testing!

2011-03-16	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/backend/SessionFactoryProvider.java:
	  Default connection parameters are now overwritable with
	  system properties (defaults in brackets):
	  - user name:     flys.backend.user     (flys)
	  - user password: flys.backend.password (flys)
	  - db dialect:    flys.backend.dialect  (org.hibernate.dialect.PostgreSQLDialect)
	  - db driver:     flys.backend.driver   (org.postgresql.Driver)
	  - db url:        flys.backend.url      (jdbc:postgresql://localhost:5432/flys)

2011-03-16	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/backend/SessionFactoryProvider.java:
	  Expose createSessionFactory() as public to be usable without
	  a artifact database running.

2011-03-16	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/utils/FileTools.java: 
	  Tools for handling with filenames. Currently there is
	  a repair(File) method with repairs letter case errors
	  which is useful when reading windows filenames on a
	  un*x platform.

	* src/main/java/de/intevation/flys/importer/Importer.java:
	  Standalone app to read data from the file system and
	  store it in a database. Currently it does not store 
	  anything. It only loads info gew files.

	* src/main/java/de/intevation/flys/importer/InfoGewParser.java:
	  Info gew parser.

	* src/main/java/de/intevation/flys/importer/ImportRiver.java:
	  Helper model of a river used produced by parsing the
	  info gew files.

2011-03-15	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/backend/SessionFactoryProvider.java:
	  New. SessionFactoryProvider.getSessionFactory() provides a
	  SessionFactory to use the Hibernate O/R mapper for the FLYS backend.

2011-03-15	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* pom.xml: Added dependency to artifacts-commons to
	  be able to use the global configuration of the artifact database.

2011-03-15	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/App.java: Wire all POJOs
	  to corresponding factory.

	* src/main/java/de/intevation/flys/model/*.java: Generate
	  all foreign key constraints. TODO: name them correctly
	  because the machine generated names are ugly and do
	  not fit the PostgreSQL names.

	* doc/schema/postgresql.sql: Small quantifier fix in descriptions
	  of wst columns.

2011-03-14	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* doc/schema/postgresql.sql: Fixed wrong spelled 
	  column references in foreign keys introduces with
	  last change.

2011-03-14	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* src/main/java/de/intevation/flys/model/*.java: Added
	  column annotations for simple fields. TODO: foreign keys.

2011-03-14	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* doc/schema/postgresql.sql, doc/schema/postgresql-cleanup.sql:
	  Fixed inconsistent table names.

	* src/main/java/de/intevation/flys/model/*.java: Added
	  entity and id annotations.

2011-03-14	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* doc/schema/postgresql.sql, doc/schema/postgresql-cleanup.sql:
	  Added missing sequences.

	* doc/schema/sqlite.sql: Deleted. Not longer supported.

2011-03-11	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* doc/schema/postgresql-cleanup.sql: Forgot to add.

	* src/main/java/de/intevation/flys/App.java: Use
	  Apache Commons DBCP as Hibernate connection provide.

	* src/main/java/de/intevation/flys/model/River.java:
	  Added a constructor with string argument. Set the
	  sequence increment to 1 (eat up 100 at a time before).

	* pom.xml: Added PostgreSQL 8.4 driver as runtime dependency.

2011-03-11	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* doc/schema/postgresql-cleanup.sql: New. Tear down schema
	  for a postgres database.

	* doc/schema/postgresql.sql: Added squence for 
	  auto generating ids in river table. Cleaned up schema.

	* src/main/java/de/intevation/flys/App.java: Simple
	  test app to interact with hibernate. Needs to be removed
	  because its only a toy.

	* src/main/java/de/intevation/flys/utils/DBCPConnectionProvider.java:
	  New. Binds Apache Commons to Hibernate.

	* pom.xml: Added dependencies to log4j, commons dbcp,
	  JPA of hibernate.

	* src/main/java/de/intevation/flys/model/River.java: Added
	  JPA annotations.

	* src/main/java/de/intevation/flys/model/*.java: Replaced
	  Long with Integer because column ids are only four bytes wide.

2011-03-11	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* doc/schema/sqlite.sql, doc/schema/postgresql.sql: Fixed
	  smaller issues in ddl.

	* src/main/java/de/intevation/flys/model/*.java: Added POJOs
	  of to be mapped. TODO: Map them!

	* pom.xml: Added plugin config for hibernate.

2011-03-09	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* pom.xml: Added dependency (and corresponding repository) to
	  Hibernate Core 3.6.1 Final

2011-03-09	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* pom.xml, src/**: Created a new empty maven project:
	  $ mvn archetype:create         \
	    -DgroupId=de.intevation.flys \
		-DartifactId=flys-backend

2011-03-09	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* README: New. Some setup instructions.

	* doc/schema/postgresql.sql: New. Schema converted to PostgreSQL

	* doc/schema/sqlite.sql: Fixed defect foreign key constraints.

2011-03-09	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* doc/schema/sqlite.sql: Factorized time intervals out into
	  a separated table.

2011-01-22	Sascha L. Teichmann	<sascha.teichmann@intevation.de>

	* contrib/import-kms.py, contrib/import-gew.py: Initial scripts
	to import data into SQLite database. They still need some work.

2011-02-10	Sascha L. Teichmann	<sascha.teichmann@intevation.de>:

	* doc/schema/sqlite.sql: Added initial schema for
	FLYS database.

http://dive4elements.wald.intevation.org