flex.de/">JFlex 1.4.1; however, the generated file was modified for performance. Memory allocation needs to be almost completely removed to be competitive with the handwritten lexers (subclasses of
AbstractTokenMaker), so this class has been modified so that Strings are never allocated (via yytext()), and the scanner never has to worry about refilling its buffer (needlessly copying chars around). We can achieve this because RText always scans exactly 1 line of tokens at a time, and hands the scanner this line as an array of characters (a Segment really). Since tokens contain pointers to char arrays instead of Strings holding their contents, there is no need for allocating new memory for Strings.
The actual algorithm generated for scanning has, of course, not been modified.
If you wish to regenerate this file yourself, keep in mind the following:
- The generated
PlainTextTokenMaker.java file will contain two definitions of both zzRefill and yyreset. You should hand-delete the second of each definition (the ones generated by the lexer), as these generated methods modify the input buffer, which we'll never have to do. - You should also change the declaration/definition of zzBuffer to NOT be initialized. This is a needless memory allocation for us since we will be pointing the array somewhere else anyway.
- You should NOT call
yylex() on the generated scanner directly; rather, you should use getTokenList as you would with any other TokenMaker instance.
@author Robert Futrell
@version 0.5