Ariel Flesler: String Tokenizer for Javascript

This small class can easily parse a string, and generate different kind of tokens. It's very simple and straight-forward. It can perform as a base for other string parsing scripts, like templating engines, custom language interpreters, and many more.

jQuery plugin vs standalone

When called, the script will generate the class, and if jQuery is detected, it will be saved at $.tokenizer.
Otherwise, the class is saved at (window.)Tokenizer.
Note that this script doesn't need jQuery at all, this option is added to ease on jQuery developers.