Skip to content

Commit a8f0ada

Browse files
rvirdingrvirding
committed
Version 0.3.
1 parent e76be26 commit a8f0ada

File tree

6 files changed

+199
-64
lines changed

6 files changed

+199
-64
lines changed

‎README

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,20 @@ analyzed and processed by yecc. It is NOT a part of OTP at this point.
55
If you are looking at a good article on how to use leex and yecc please visit:
66
http://hopper.squarespace.com/blog/2008/5/29/leex-and-yecc.html
77

8+
v0.3
9+
----
10+
Added options to pushback characters into the input stream.
11+
12+
Can now handle full unicode character sets in both input file and
13+
generated scanner. Of course i/o system can't generate such an input
14+
file yet.
15+
16+
Enforce headings for all sections of .xrl file.
17+
18+
Added verbose flag to control printing of parsing information.
19+
20+
Fixed bug with completely empty erlang code section.
21+
822
v0.2
923
----
1024
Added includefile option. Search for include file in current directory

‎doc/leex.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,6 @@ <h3 class="function"><a name="format_error-1">format_error/1</a></h3>
3737
<hr>
3838

3939
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
40-
<p><i>Generated by EDoc, Aug 13 2008, 04:18:12.</i></p>
40+
<p><i>Generated by EDoc, Dec 29 2008, 02:29:16.</i></p>
4141
</body>
4242
</html>

‎doc/leex.txt

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,10 @@ file(FileName, Options) -> ok | error
3939
the DFA in a format which can be viewd with Graphviz,
4040
www.graphviz.com
4141

42+
verbose
43+
Output information from parsing the input file and
44+
generating the internal tables.
45+
4246
Generate a lexical analyzer from the definition in the input
4347
file. The input file has the extension .xrl. This is added to
4448
the filename if it is not given. The resulting module is the

‎doc/overview-summary.html

Lines changed: 115 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
99
<h1>leex documentation</h1>
1010
<p>Copyright © 2008 Robert Virding</p>
11-
<p><b>Version:</b> 0.1
11+
<p><b>Version:</b> 0.3
1212
</p>
1313
<p><b>Authors:</b> Robert Virding (<a href="mailto:rvirding@gmail.com"><tt>rvirding@gmail.com</tt></a>).</p>
1414

@@ -25,22 +25,34 @@ <h4><a name="DESCRIPTION">DESCRIPTION</a></h4>
2525

2626
<h4><a name="DATATYPES">DATATYPES</a></h4>
2727

28-
<pre> ScanRet = {ok,Tokens,EndLine} |
29-
{eof,EndLine} |
30-
{error,ErrorDescriptor,EndLine}
31-
Cont = continuation()
32-
ErrorDescriptor = {ErrorLine,Module,Error}</pre>
28+
<p>ScanRet = {ok,Tokens,EndLine} |
29+
{eof,EndLine} |
30+
{error,ErrorDescriptor,EndLine}</p>
31+
32+
<p>Cont = continuation()</p>
33+
34+
<p>ErrorDescriptor = {ErrorLine,Module,Error}</p>
35+
3336
<h4><a name="EXPORTS">EXPORTS</a></h4>
3437

35-
<p>file(FileName) -&gt; ok | error
36-
file(FileName, Options) -&gt; ok | error</p>
38+
<p>file(FileName) -&gt; ok | error</p>
39+
40+
<p>file(FileName, Options) -&gt; ok | error</p>
3741

3842
<p>The current options are:</p>
3943

44+
<p>{includefile,File}
45+
Use a specific or customised prologue file instead of
46+
default leex/include/leexinc.hrl which is otherwise
47+
included.</p>
48+
4049
<p>{outdir,Dir} - generate the scanner file in directory Dir
4150
dfa_graph - generate a .dot file which contains a
4251
desciption of the DFA in a format which can be
43-
viewd with Graphviz, www.graphviz.com</p>
52+
viewd with Graphviz, www.graphviz.com
53+
verbose
54+
Output information from parsing the input file and
55+
generating the internal tables.</p>
4456

4557
<p>Generate a lexical analyzer from the definition in the input
4658
file. The input file has the extension .xrl. This is added to
@@ -49,15 +61,17 @@ <h4><a name="EXPORTS">EXPORTS</a></h4>
4961

5062
<h4><a name="GENERATED_SCANNER_EXPORTS">GENERATED SCANNER EXPORTS</a></h4>
5163

52-
<p>string(String) -&gt; ScanRet
53-
string(String, StartLine) -&gt; ScanRet</p>
64+
<p>string(String) -&gt; ScanRet</p>
65+
66+
<p>string(String, StartLine) -&gt; ScanRet</p>
5467

5568
<p>Scan String and return all the tokens in it, or an error.
5669
N.B. it is an error if all of the characters in String are
5770
consumed.</p>
5871

59-
<p>token(Cont, Chars) -&gt; {more,Cont} | {done,ScanRet,RestChars}
60-
token(Cont, Chars, StartLine) -&gt; {more,Cont} | {done,ScanRet,RestChars}</p>
72+
<p>token(Cont, Chars) -&gt; {more,Cont} | {done,ScanRet,RestChars}</p>
73+
74+
<p>token(Cont, Chars, StartLine) -&gt; {more,Cont} | {done,ScanRet,RestChars}</p>
6175

6276
<p>This is a re-entrant call to try and scan one token from
6377
Chars. If there are enough characters in Chars to either scan
@@ -74,8 +88,9 @@ <h4><a name="GENERATED_SCANNER_EXPORTS">GENERATED SCANNER EXPORTS</a></h4>
7488
<p>io:request(InFile, {get_until,Prompt,Module,token,[Line]})
7589
-&gt; ScanRet</p>
7690

77-
<p>tokens(Cont, Chars) -&gt; {more,Cont} | {done,ScanRet,RestChars}
78-
tokens(Cont, Chars, StartLine) -&gt; {more,Cont} | {done,ScanRet,RestChars}</p>
91+
<p>tokens(Cont, Chars) -&gt; {more,Cont} | {done,ScanRet,RestChars}</p>
92+
93+
<p>tokens(Cont, Chars, StartLine) -&gt; {more,Cont} | {done,ScanRet,RestChars}</p>
7994

8095
<p>This is a re-entrant call to try and scan tokens from Chars.
8196
If there are enough characters in Chars to either scan tokens
@@ -169,16 +184,32 @@ <h5><a name="Input_File_Format">Input File Format</a></h5>
169184
{error,ErrString} - an error in the token, ErrString is a string
170185
describing the error</p>
171186

187+
<p>It is also possible to push back characters into the input
188+
characters with the following returns:</p>
189+
190+
<p>{token,Token,PushBackList}
191+
{end_token,Token,PushBackList}
192+
{skip_token,PushBackList}</p>
193+
194+
<p>These have the same meanings as the normal returns but the
195+
characters in PushBackList will be prepended to the input
196+
characters and scanned for the next token. Note that pushing
197+
back a newline will mean the line numbering will no longer be
198+
correct. N.B. Pushing back characters gives you unexpected
199+
possibilities to cause the scanner to loop!</p>
200+
172201
<p>The following example would match a simple Erlang integer or
173202
float and return a token which could be sent to the Erlang
174203
parser:</p>
175204

205+
176206
<p>D = [0-9]</p>
177207

178208
<p>{D}+ : {token,{integer,
179209
TokenLine,
180-
list_to_integer(TokenChars)}}.
181-
{D}+\.{D}+((E|e)(\+|\-)?{D}+)? :
210+
list_to_integer(TokenChars)}}.</p>
211+
212+
<p>{D}+\.{D}+((E|e)(\+|\-)?{D}+)? :
182213
{token,{float,TokenLine,list_to_float(TokenChars)}}.</p>
183214

184215
<p>The Erlang code in the "Erlang Code." section is written into
@@ -196,58 +227,80 @@ <h5><a name="Regular_Expressions">Regular Expressions</a></h5>
196227
following characters:</p>
197228

198229
<p>c
199-
matches the non-metacharacter c.
200-
\c
201-
matches the escape sequence or literal character c.
202-
.
203-
matches any character.
204-
^
205-
matches the beginning of a string.
206-
$
207-
matches the end of a string.
208-
[abc...]
230+
matches the non-metacharacter c.</p>
231+
232+
<p>\c
233+
matches the escape sequence or literal character c.</p>
234+
235+
<p>.
236+
matches any character.</p>
237+
238+
<p>^
239+
matches the beginning of a string.</p>
240+
241+
<p>$
242+
matches the end of a string.</p>
243+
244+
<p>[abc...]
209245
character class, which matches any of the characters
210246
abc... Character ranges are specified by a pair of
211-
characters separated by a -.
212-
[^abc...]
247+
characters separated by a -.</p>
248+
249+
<p>[^abc...]
213250
negated character class, which matches any character
214-
except abc....
215-
r1 | r2
216-
alternation. It matches either r1 or r2.
217-
r1r2
218-
concatenation. It matches r1 and then r2.
219-
r+
220-
matches one or more rs.
221-
r*
222-
matches zero or more rs.
223-
r?
224-
matches zero or one rs.
225-
(r)
251+
except abc....</p>
252+
253+
<p>r1 | r2
254+
alternation. It matches either r1 or r2.</p>
255+
256+
<p>r1r2
257+
concatenation. It matches r1 and then r2.</p>
258+
259+
<p>r+
260+
matches one or more rs.</p>
261+
262+
<p>r*
263+
matches zero or more rs.</p>
264+
265+
<p>r?
266+
matches zero or one rs.</p>
267+
268+
<p>(r)
226269
grouping. It matches r.</p>
227270

228271
<p>The escape sequences allowed are the same as for Erlang strings:</p>
229272

230273
<p>\b
231-
backspace
232-
\f
233-
form feed
234-
\n
235-
newline (line feed)
236-
\r
237-
carriage return
238-
\t
239-
tab
240-
\e
241-
escape
242-
\v
243-
vertical tab
244-
\s
245-
space
246-
\d
247-
delete
248-
\ddd
249-
the octal value ddd
250-
\c
274+
backspace</p>
275+
276+
<p>\f
277+
form feed</p>
278+
279+
<p>\n
280+
newline (line feed)</p>
281+
282+
<p>\r
283+
carriage return</p>
284+
285+
<p>\t
286+
tab</p>
287+
288+
<p>\e
289+
escape</p>
290+
291+
<p>\v
292+
vertical tab</p>
293+
294+
<p>\s
295+
space</p>
296+
297+
<p>\d
298+
delete</p>
299+
300+
<p>\ddd
301+
the octal value ddd</p>
302+
303+
<p>\c
251304
any other character literally, for example \\ for
252305
backslash, \" for ")</p>
253306

@@ -264,6 +317,6 @@ <h5><a name="Regular_Expressions">Regular Expressions</a></h5>
264317
a nasty error.
265318
<hr>
266319
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
267-
<p><i>Generated by EDoc, Aug 13 2008, 04:18:13.</i></p>
320+
<p><i>Generated by EDoc, Dec 29 2008, 02:29:17.</i></p>
268321
</body>
269322
</html>

0 commit comments

Comments
 (0)