An interesting and easy to read explanation of why you might like to keep lexing and parsing separate: Why Split Lexing and Parsing Into Two Separate Phases?