Configuring the Extract Settings

Symaptic Shadow

The Smallworld extract plugin used by Symaptic is called Symaptic Shadow. Symaptic Shadow is able to export data from Smallworld to Postgres, including geometry information.

The example scripts provided with the Symaptic Web and Mobile Package automatically configures Symaptic Shadow to extract visible fields only. It is recommended to use these example scripts to generate your first set of configuration files for Symaptic Shadow, and then configure those scripts as required.

The default configuration file is stored in the smallworld\symaptic_shadow folder with the file name being shadow_configuration.magik. The configuration consists of a rope of property lists, with each property list containing the configuration parameters for extracting a Smallworld table. For example:

symaptic_shadow.table_mapping.add(
	property_list.new_with(
		:sw_dataset, :gis,
		:enabled?, _true,
		:indexed?, _true,
		:index_universe, 0,
		:index_world, 0,
		:full_sync_only?, _false,
		:frequency, :daily,
		:shadow_schema, "",
		:sw_table, :min_road,
		:sw_table_external, "Road",
		:shadow_table, :min_road,
		:attribute_indices, {},
		:primary_key, :symaptic_id,
		:geometry_indices, {:centre_line},
		:field_mappings,
			{{:urn, :symaptic_id, "text PRIMARY KEY","ID"},
			 {:name, :name,"character varying(20)","Name"},
			 {:road_type, :road_type,"character varying(13)","Road Type"},
			 {:carriage_type, :carriage_type,"character varying(15)","Carriage Type"},
			 {:length, :length,"decimal","Length"},
			 {:centre_line, :centre_line,:shadow_geom_line_type,"Centreline"}}
	)
)

The purpose of each property in the property list is descibed as follows:

Property Name Explanation
 sw_dataset Source Smallworld dataset of the table
 enabled? Use in extract or skip
 indexed? A flag to allow extracts to be run at difference intervals – only extracts matching the frequency interval specified as an argument at run-time will run
new? OPTIONAL Automatically set/used by this module, but can be used to manually load new tables without resetting other tables. This flag should not be used in normal circumstances when adding new tables. When new tables are added, they will be automatically created and included in Postgres. However, should a new table need to be manually added to the shadow database, this flag can be used. This flag will allow this table to be created and a full extract run for this table.
 index_universe The universe to filter on when geometries are stored in the index database – as these need to be valid for the goto function when results are found in the index searches
 index_world The world to filter on when geometries are stored in the index database
– as these need to be valid for the goto function when results are found in the index searches
 full_sync_only?  OPTIONAL Full sync every time (no deltas)
 frequency A flag to allow extracts to be run at difference intervals – only extracts matching the frequency interval specified as an argument at run-time will run
sw_table  Smallworld table name (source)
sw_table_external External name to be used for the Smallworld table name
shadow_table Postgres table name (target)
attribute_indices OPTIONAL Attribute fields to index in Postgres
geometry_indices OPTIONAL Geometry fields to index in Postgres
primary_key Primary key to use for the Postgres table
sql_query OPTIONAL Use SQL to populate a table
additional_creation_sql OPTIONAL Additional SQL creation parameters – this is run only once after the table is created
predicate OPTIONAL Predicate to filter the table. No filtering used if not specified
field_mappings The fields of the Smallworld table to extract. Each field definition must contain:

  1. internal field name in smallworld*
  2. internal field name in postgres
  3. Postgres field type
  4. external table display name for the Symaptic Web GIS Client

Internal fields names can either be actual Smallworld fields or methods to be applied against the Smallworld record. Multiple methods can be applied by encapsulating the methods in the { } symbology. In such a case the methods will be applied on the result of the previous method applied. For example {a_rwo_record,zone_substation,sap_id} will be evaluated in Smallworld as rwo_record.zone_substation.sap_id

Other configuration parameters set via shared constants are further documented in the source code (symaptic_shadow.magik) and include:

  • Name of the alternative to use to store extract checkpoints (used for historical reference or to calculate difference/delta updates) – one for each Smallworld dataset
  • How often to commit the postgres database when performing the extract
  • How often to write to the log file (based on the number of records extracted, eg every 1000 records)
  • How many records to extract per table when running in test mode

Tips and Troubleshooting

Tips / Error Action
I have added a new extract table/definition, what do I need to do now The new table will be detected even if only doing diff load and create and a full load will be performed for the new table. However, for the new definition to be included in GeoServer and the Symaptic Web GIS Client, the current script requires a full extract of all tables to be run – it is possible to selectively run the updates for a single table, but this has yet to be automated in the script.
Check the log files and Postgres interface_status_log table Progress and success of the extraction will be logged in the log files and the Postgres interface_status_log. The final summary in the log file gives a good indication of the overall success. In particular look for the statement completed no errors or completed with errors to indicate if errors were encountered. A summary of the errors is provided in the Postgres table interface_status_log, as well as full details in the log file (search for **** Error).
There are very large delta changes occurring when the diff extract runs runs Is it likely that a Smallworld CASE change has triggered a rebuild of the datamodel and hence there are large differences between the Symaptic Shadow alternative/checkpoints and the parent alternative. If the changes are not required, remove the “Current” and “Previous” checkpoints in the Symaptic Shadow alternative. The Symaptic Shadow will automatically re-create the checkpoints then next time it runs and will not identify any changes.
Changes need to be made to the data / data model and I don’t want the Symaptic Shadow Extract to identify the changes Prior to making the changes, run the diff extract (eg. run_shadow_extract.sh diff false). Make the changes as required and then delete the “Current” and “Previous” checkpoints in the Symaptic Shadow alternative. The Symaptic Shadow extract will automatically re-create the checkpoints the next time it runs and will not identify any changes.
ACP or Connection Errors Check that the correct Java JRE is being used and is accessible by the application (see the system environment SHADOW_JAVA_HOME). Also ensure the Java JDBC library is accessible (see the system environment SHADOW_POSTGRES_JAR). Ensure the connection specification is correct (shared constant pg_connection_spec).
CASE Changes If CASE changes impact specific fields used by the Symaptic Shadow extract (eg. removing of a fields used by the extract), the extract configuration will also need to be updated accordingly.