A class that tokenizes the String representation of patterns for the {@link LayeredTokenPattern} class. A pattern consists of two types of tokens:layer/symbol pairs like {@code The_pos}, and meta-characters defined by {@link java.util.rexex.Pattern}. This class takes a string consisting of these and breaks them into tokens.
@author afader
All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.