Notes on Truncating Coordinates

Why shall I truncate the coordinates of all vertices?

The truncation of the coordinates can repair the following errors or prevent their occurrence:

Due to Differences in the Number Format between the different GIS data sources it may happen that nodes and lines that seem to be closed in ARC/INFO or AutoCAD are open and dangling after conversion into a shapefile (the deviations are so small that they are only visible at an extreme magnification). By truncating the coordinates these errors are mostly cleaned. Therefore you should always truncate the coordinates of shapefiles that were imported from an ARC/INFO Coverage or an AutoCad drawing.

Fuzzy and Incorrect Vertices: When intersecting two polygons whose vertices are slightly displaced (fuzzy vertices), it may occur that single vertices get the coordinate 0/0 due to calculation errors in ArcView (Division by Zero). Thus editing or intersecting of a theme may result in lines that are going from the themes extent far left to the bottom of the view. In that case you should have truncated the coordinates of both themes before. Thereby most of the fuzzy vertices are cleaned (i.e. will obtain the same coordinate) and the further occurrence of fuzzy or incorrect vertices is avoided to a great extent (see Effects of Truncation below).

Display Errors: If a polygon theme displays with gaps or overlaps depending on scale and extent of the view and single polygons are suddenly "turned inside out" when zooming the view, there are also problems with the precision of the coordinates (too many decimal places). It seems that at transforming map to display coordinates (pixels) also calculation errors occur (Division by Zero) which make single vertices fall out of the visible area. Editing or intersecting such themes causes numerous, mostly very obscure errors too. These problems can also be solved by truncating the coordinates. You can download such a faulty polygon theme from our Download Page.

Which effects has the truncation of the coordinates?

By rounding coordinates the occurrence of calculation errors and therefore the probability of fuzzy or incorrect vertices decreases: Intersections of lines are created by an arithmetical calculation with the coordinates of the neighboring vertices. Therefore the probability, that the number of available decimal places is exceeded and errors occur, increases by the number of decimal places of those coordinates (although few decimal places do not always mean few places in the binary notation of floating point numbers - so it would actually be optimal to truncate at values of the power of 2).

By removing decimal places the information quantity in a shapefile is reduced considerably (a lot of 0 bytes result in the floating point numbers) and the shapefile can be compressed much better afterwards. Hence you should truncate coordinates if you want to obtain small archive files.

How many decimal places shall be truncated?

For the reasons given above, the coordinates of a shapefile should always be truncated and all unneeded decimal places below the visible amount on your map should be removed. In order to avoid a visible data loss truncation should meet approximately 1/10 up to 1/100 mm on the map (this means according to the map scale 2 places at 1:1.000, 1 place at 1:10.000, 0 places at 1:100.000 etc.).

Because all identical vertices will get the same coordinates again, the topology of the theme will not be affected. Only the truncation of too many decimal places would lead to visible loss and stepped lines. In extreme cases, parallel running lines, narrow sections of a polygon or whole polygons may collapse (because opposite border lines can coincide to a single line or the whole polygon can shrink to one single point).

Why shall identical vertices be removed?

Due to truncation, closely situated vertices may get the same coordinate and thereby consecutive identical vertices arise. These redundant vertices do not directly mean topological errors. Nevertheless they should be removed, as they may lead to problems and errors in processing, especially at start or end of lines (nodes). Identical vertices that do not succeed one another are of course remaining. Additionally the removal of redundant vertices reduces the size of the shapefile too.

2003 WLM Klosterhuber & Partner OEG