forked from postgis/postgis
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathextras_address_standardizer.xml
424 lines (385 loc) · 18.8 KB
/
extras_address_standardizer.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
<?xml version="1.0" encoding="UTF-8"?>
<chapter id="Address_Standardizer">
<title>Address Standardizer</title>
<para>This is a fork of the <ulink url="http://www.pagcgeo.org/docs/html/pagc-11.html">PAGC standardizer</ulink> (original code for this portion was <ulink url="http://sourceforge.net/p/pagc/code/360/tree/branches/sew-refactor/postgresql">PAGC PostgreSQL Address Standardizer</ulink>). </para>
<para>The address standardizer is a single line address parser that takes an input address and normalizes it based on a set of rules stored in a table and helper lex and gaz tables.</para>
<para>The code is built into a single postgresql extension library called <code>address_standardizer</code> which can be installed with <code>CREATE EXTENSION address_standardizer;</code>.</para>
<para>The code for this extension can be found in the PostGIS <filename>extensions/address_standardizer</filename> and is currently self-contained.</para>
<para>For installation instructions refer to: <xref linkend="installing_pagc_address_standardizer" />.</para>
<sect1 id="Address_Standardizer_Basics"><title>How the Parser Works</title>
<para>The parser works from right to left looking first at the macro elements
for postcode, state/province, city, and then looks micro elements to determine
if we are dealing with a house number street or intersection or landmark.
It currently does not look for a country code or name, but that could be
introduced in the future.</para>
<variablelist>
<varlistentry>
<term>Country code</term>
<listitem><para>Assumed to be US or CA based on: postcode as US or Canada state/province as US or Canada else US</para></listitem>
</varlistentry>
<varlistentry>
<term>Postcode/zipcode</term>
<listitem><para>These are recognized using Perl compatible regular expressions.
These regexs are currently in the parseaddress-api.c and are relatively
simple to make changes to if needed.</para></listitem>
</varlistentry>
<varlistentry>
<term>State/province</term>
<listitem><para>These are recognized using Perl compatible regular expressions.
These regexs are currently in the parseaddress-api.c but could get moved
into includes in the future for easier maintenance.</para></listitem>
</varlistentry>
</variablelist>
</sect1>
<sect1 id="Address_Standardizer_Types">
<sect1info>
<abstract>
<para>This section lists the PostgreSQL data types installed by Address Standardizer extension. Note we describe the casting behavior of these which is very
important especially when designing your own functions.
</para>
</abstract>
</sect1info>
<title>Address Standardizer Types</title>
<refentry id="stdaddr">
<refnamediv>
<refname>stdaddr</refname>
<refpurpose>A composite type that consists of the elements of an address. This is the return type for <varname>standardize_address</varname> function.</refpurpose>
</refnamediv>
<refsection>
<title>Description</title>
<para>A composite type that consists of elements of an address. This is the return type for <xref linkend="standardize_address" /> function. Some descriptions for elements are borrowed from <ulink url="http://www.pagcgeo.org/docs/html/pagc-12.html#ss12.1">PAGC Postal Attributes</ulink>.</para>
<para>The token numbers denote the reference number in the <varname>rules</varname> table.</para>
<para>&address_standardizer_required;</para>
<variablelist>
<varlistentry>
<term>building</term>
<listitem>
<para> is text (token number <code>0</code>): Refers to building number or name. Unparsed building identifiers and types. Generally blank for most addresses.</para>
</listitem>
</varlistentry>
<varlistentry><term>house_num</term>
<listitem>
<para>is a text (token number <code>1</code>): This is the street number on a street. Example <emphasis>75</emphasis> in <code>75 State Street</code>.</para>
</listitem>
</varlistentry>
<varlistentry><term>predir</term><listitem>
<para> is text (token number <code>2</code>): STREET NAME PRE-DIRECTIONAL such as North, South, East, West etc.</para>
</listitem></varlistentry>
<varlistentry><term>qual</term>
<listitem>
<para>is text (token number <code>3</code>): STREET NAME PRE-MODIFIER Example <emphasis>OLD</emphasis> in <code>3715 OLD HIGHWAY 99</code>.</para>
</listitem>
</varlistentry>
<varlistentry><term>pretype</term>
<listitem>
<para> is text (token number <code>4</code>): STREET PREFIX TYPE</para>
</listitem>
</varlistentry>
<varlistentry><term>name</term>
<listitem>
<para>is text (token number <code>5</code>): STREET NAME</para>
</listitem>
</varlistentry>
<varlistentry><term>suftype</term>
<listitem>
<para>is text (token number <code>6</code>): STREET POST TYPE e.g. St, Ave, Cir. A street type following the root street name. Example <emphasis>STREET</emphasis> in <code>75 State Street</code>.</para>
</listitem>
</varlistentry>
<varlistentry><term>sufdir</term>
<listitem>
<para>is text (token number <code>7</code>): STREET POST-DIRECTIONAL A directional modifier that follows the street name.. Example <emphasis>WEST</emphasis> in <code>3715 TENTH AVENUE WEST</code>.</para>
</listitem>
</varlistentry>
<varlistentry><term>ruralroute</term>
<listitem>
<para>is text (token number <code>8</code>): RURAL ROUTE . Example <emphasis>8</emphasis> in <code>RR 7</code>.</para>
</listitem>
</varlistentry>
<varlistentry><term>extra</term>
<listitem>
<para>is text: Extra information like Floor number.</para>
</listitem>
</varlistentry>
<varlistentry><term>city</term>
<listitem>
<para>is text (token number <code>10</code>): Example Boston.</para>
</listitem>
</varlistentry>
<varlistentry><term>state</term>
<listitem>
<para>is text (token number <code>11</code>): Example <code>MASSACHUSETTS</code></para>
</listitem>
</varlistentry>
<varlistentry><term>country</term>
<listitem>
<para>is text (token number <code>12</code>): Example <code>USA</code></para>
</listitem>
</varlistentry>
<varlistentry><term>postcode</term>
<listitem>
<para>is text POSTAL CODE (ZIP CODE) (token number <code>13</code>): Example <code>02109</code></para>
</listitem>
</varlistentry>
<varlistentry><term>box</term>
<listitem>
<para>is text POSTAL BOX NUMBER (token number <code>14 and 15</code>): Example <code>02109</code></para>
</listitem>
</varlistentry>
<varlistentry><term>unit</term>
<listitem>
<para>is text Apartment number or Suite Number (token number <code>17</code>): Example <emphasis>3B</emphasis> in <code>APT 3B</code>.</para>
</listitem>
</varlistentry>
</variablelist>
</refsection>
</refentry>
</sect1>
<sect1 id="Address_Standardizer_Tables">
<sect1info>
<abstract>
<para>This section lists the PostgreSQL table formats used by the address_standardizer for normalizing addresses. Note that these tables do not need to be named the same as what is referenced here. You can have different lex, gaz, rules tables for each country for example or for your custom geocoder. The names of these tables get passed into the address standardizer functions.
</para>
</abstract>
</sect1info>
<title>Address Standardizer Tables</title>
<refentry id="rulestab">
<refnamediv>
<refname>rules table</refname>
<refpurpose>The rules table contains a set of rules that maps address input sequence tokens to standardized output sequence</refpurpose>
</refnamediv>
<refsection>
<title>Description</title>
<para>A rules table must have at least the following columns, though you are allowed to add more for your own uses. </para>
<variablelist>
<varlistentry>
<term>id</term>
<listitem>
<para>Primary key of table</para>
</listitem>
</varlistentry>
<varlistentry><term>rule</term>
<listitem>
<para>text field denoting the rule. Details at <ulink url="http://www.pagcgeo.org/docs/html/pagc-12.html#--r-rec--">PAGC Address Standardizer Rule records</ulink>.</para>
<para>A rule consists of a set of non-negative integers representing input tokens, terminated by a -1, followed by an equal number of non-negative integers representing postal attributes, terminated by a -1, followed by an integer representing a rule type, followed by an integer representing the rank of the rule. The rules are ranked from 0 (lowest) to 17 (highest).</para>
<para>So for example the rule <code>2 0 2 22 3 -1 5 5 6 7 3 -1 2 6</code> maps to sequence of tokens <emphasis>TYPE NUMBER TYPE DIRECT QUALIF</emphasis> to the output sequence <emphasis>STREET STREET SUFTYP SUFDIR QUALIF</emphasis>. The rule is an ARC_C rule of rank 6. </para>
</listitem>
</varlistentry>
</variablelist>
<para>Each rule has a rule type which is denoted by one of following:</para>
<variablelist>
<varlistentry>
<term>MACRO_C</term>
<listitem>
<para>(token number = "0"). The class of rules for parsing MACRO clauses.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>MICRO_C</term>
<listitem>
<para>(token number = "1"). The class of rules for parsing full MICRO clauses (ie ARC_C plus CIVIC_C). These rules are not used in the build phase.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>ARC_C</term>
<listitem>
<para>(token number = "2"). The class of rules for parsing MICRO clauses, excluding the HOUSE attribute.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>CIVIC_C</term>
<listitem>
<para>(token number = "3"). The class of rules for parsing the HOUSE attribute.</para>
</listitem>
</varlistentry>
<varlistentry>
<term>EXTRA_C</term>
<listitem>
<para>(token number = "4"). The class of rules for parsing EXTRA attributes - attributes excluded from geocoding. These rules are not used in the build phase.</para>
</listitem>
</varlistentry>
</variablelist>
</refsection>
</refentry>
<refentry id="lextab">
<refnamediv>
<refname>lex table</refname>
<refpurpose>A lex table is used to classify alphanumeric input and associate that input with (a) input tokens ( See Input Tokens) and (b) standardized representations.</refpurpose>
</refnamediv>
<refsection>
<title>Description</title>
<para>A lex (short for lexicon) table is used to classify alphanumeric input and associate that input with <ulink url="http://www.pagcgeo.org/docs/html/pagc-12.html#--i-tok--">(a) input tokens</ulink> and (b) standardized representations. Things you will find in these tables are <code>ONE</code> mapped to stdworkd: <code>1</code>.</para>
<para>A lex has at least the following columns in the table. You may add</para>
<variablelist>
<varlistentry>
<term>id</term>
<listitem>
<para>Primary key of table</para>
</listitem>
</varlistentry>
<varlistentry><term>seq</term>
<listitem>
<para>integer: definition number?</para>
</listitem>
</varlistentry>
<varlistentry><term>word</term>
<listitem>
<para>text: the input word</para>
</listitem>
</varlistentry>
<varlistentry><term>stdword</term>
<listitem>
<para>text: the standardized replacement word</para>
</listitem>
</varlistentry>
<varlistentry><term>token</term>
<listitem>
<para>integer: the kind of word it is. Only if it is used in this context will it be replaced. Refer to <ulink url="http://www.pagcgeo.org/docs/html/pagc-12.html#--i-tok--">PAGC Tokens</ulink>.</para>
</listitem>
</varlistentry>
</variablelist>
</refsection>
</refentry>
<refentry id="gaztab">
<refnamediv>
<refname>gaz table</refname>
<refpurpose>A gaz table is used to standardize place names and associate that input with (a) input tokens ( See Input Tokens) and (b) standardized representations.</refpurpose>
</refnamediv>
<refsection>
<title>Description</title>
<para>A gaz (short for gazeteer) table is used to classify place names and associate that input with <ulink url="http://www.pagcgeo.org/docs/html/pagc-12.html#--i-tok--">(a) input tokens</ulink> and (b) standardized representations. For example if you are in US, you may load these with State Names and associated abbreviations.</para>
<para>A gaz table has at least the following columns in the table. You may add more columns if you wish for your own purposes.</para>
<variablelist>
<varlistentry>
<term>id</term>
<listitem>
<para>Primary key of table</para>
</listitem>
</varlistentry>
<varlistentry><term>seq</term>
<listitem>
<para>integer: definition number?</para>
</listitem>
</varlistentry>
<varlistentry><term>word</term>
<listitem>
<para>text: the input word</para>
</listitem>
</varlistentry>
<varlistentry><term>stdword</term>
<listitem>
<para>text: the standardized replacement word</para>
</listitem>
</varlistentry>
<varlistentry><term>token</term>
<listitem>
<para>integer: the kind of word it is. Only if it is used in this context will it be replaced. Refer to <ulink url="http://www.pagcgeo.org/docs/html/pagc-12.html#--i-tok--">PAGC Tokens</ulink>.</para>
</listitem>
</varlistentry>
</variablelist>
</refsection>
</refentry>
</sect1>
<sect1 id="Address_Standardizer_Functions"><title>Address Standardizer Functions</title>
<refentry id="parse_address">
<refnamediv>
<refname>parse_address</refname>
<refpurpose>Takes a 1 line address and breaks into parts</refpurpose>
</refnamediv>
<refsynopsisdiv>
<funcsynopsis>
<funcprototype>
<funcdef>record <function>parse_address</function></funcdef>
<paramdef><type>text </type> <parameter>address</parameter></paramdef>
</funcprototype>
</funcsynopsis>
</refsynopsisdiv>
<refsection>
<title>Description</title>
<para>Returns takes as input an address, and returns a record output consisting of fields <emphasis>num</emphasis>, <emphasis>street</emphasis>, <emphasis>street2</emphasis>,
<emphasis>address1</emphasis>, <emphasis>city</emphasis>, <emphasis>state</emphasis>, <emphasis>zip</emphasis>, <emphasis>zipplus</emphasis>, <emphasis>country</emphasis>.</para>
<!-- use this format if new function -->
<para>Availability: 2.2.0</para>
<para>&address_standardizer_required;</para>
</refsection>
<refsection>
<title>Examples</title>
<programlisting>SELECT num, street, city, zip, zipplus FROM parse_address('1 Devonshire Place, Boston, MA 02109-1234');
num | street | city | zip | zipplus
-----+------------------+--------+-------+---------
1 | Devonshire Place | Boston | 02109 | 1234
</programlisting>
</refsection>
<!-- Optionally add a "See Also" section -->
<refsection>
<title>See Also</title>
<para></para>
</refsection>
</refentry>
<refentry id="standardize_address">
<refnamediv>
<refname>standardize_address</refname>
<refpurpose>Returns an stdaddr form of an input address utilizing lex, gaz, and rule tables.</refpurpose>
</refnamediv>
<refsynopsisdiv>
<funcsynopsis>
<funcprototype>
<funcdef>stdaddr <function>standardize_address</function></funcdef>
<paramdef><type>text </type> <parameter>lextab</parameter></paramdef>
<paramdef><type>text </type> <parameter>gaztab</parameter></paramdef>
<paramdef><type>text </type> <parameter>rultab</parameter></paramdef>
<paramdef><type>text </type> <parameter>address</parameter></paramdef>
</funcprototype>
<funcprototype>
<funcdef>stdaddr <function>standardize_address</function></funcdef>
<paramdef><type>text </type> <parameter>lextab</parameter></paramdef>
<paramdef><type>text </type> <parameter>gaztab</parameter></paramdef>
<paramdef><type>text </type> <parameter>rultab</parameter></paramdef>
<paramdef><type>text </type> <parameter>micro</parameter></paramdef>
<paramdef><type>text </type> <parameter>macro</parameter></paramdef>
</funcprototype>
</funcsynopsis>
</refsynopsisdiv>
<refsection>
<title>Description</title>
<para>Returns an <xref linkend="stdaddr" /> form of an input address utilizing <xref linkend="lextab" /> table name, <xref linkend="gaztab" />, and <xref linkend="rulestab" /> table names and an address.</para>
<!-- use this format if new function -->
<para>Availability: 2.2.0</para>
<para>&address_standardizer_required;</para>
</refsection>
<refsection>
<title>Examples</title>
<programlisting>SELECT * FROM standardize_address('tiger.pagc_lex',
'tiger.pagc_gaz', 'tiger.pagc_rules', 'One Devonshire Place, PH 301, Boston, MA 02109-1234');</programlisting>
<para>Make easier to read we'll dump output using hstore extension CREATE EXTENSION hstore; you need to install</para>
<programlisting>SELECT (each(hstore(p))).*
FROM standardize_address('tiger.pagc_lex', 'tiger.pagc_gaz',
'tiger.pagc_rules', 'One Devonshire Place, PH 301, Boston, MA 02109-1234') As p;</programlisting>
<screen> key | value
------------+-----------------
box |
city | BOSTON
name | DEVONSHIRE
qual |
unit | # PENTHOUSE 301
extra |
state | MA
predir |
sufdir |
country | USA
pretype |
suftype | PL
building |
postcode | 02109
house_num | 1
ruralroute |
(16 rows)
</screen>
</refsection>
<!-- Optionally add a "See Also" section -->
<refsection>
<title>See Also</title>
<para><xref linkend="stdaddr" />, <xref linkend="rulestab" />, <xref linkend="lextab" />, <xref linkend="gaztab" />, <xref linkend="Pagc_Normalize_Address" /></para>
</refsection>
</refentry>
</sect1>
</chapter>