tokenizer

However , I believe that if one needed to build a tokenizer and wanted to avoid the complications of using another programming tool one could build an active pattern that fulfilled this requirement .

然而，我相信如果一个人需要构建一个标记器并想避免由于使用其他编程工具所带来的复杂性的话，那么他可以构建一个活动模式来满足这样的需求。

A good lexer example can help a lot with learning how to write a tokenizer .

一个好的lexer例子会非常有助于学习如何编写断词器（tokenizer）。

With the tokenizer , it is easy to decode the processing instructions .

有了tokenizer，就很容易译码处理指令。

If the tokenizer encounters neither token , it returns a lexical error .

如果记号赋予器未遇到这两个记号，则返回词法错误。

What the parser fills in depends on the path it takes through the tokenizer .

解析器填入的内容取决于它通过记号赋予器所采用的路径。

Analyzers can be composed as a tokenizer and a set of filters using the Apache Solr search library .

我们可以通过ApacheSolr搜索库将分析器分解为一个标识符和一套过滤器。

Not surprisingly , the part of the parser that 's responsible for that task is called the scanner or tokenizer .

解析器中负责该任务的部件称为扫描器（scanner）或记号赋予器（tokenizer）一点也不稀奇。

The tokenizer moves through the stream , inspecting characters one at a time until it encounters either an integer or the end of file .

记号赋予器穿过输入流，每次检查一个字符，直到它遇到一个整数或者直至文件结束。

The analysis screen displays the result of each term after it has been processed by the Tokenizer or TokenFilter named above the table results .

分析屏幕显示了每个条件在被上述表结果Tokenizer或TokenFilter处理后的结果。

If it 's an integer token , the expression is still potentially valid , and integerLiteral () calls the tokenizer once again to return the next token .

如果它是整数记号，表达式仍然可能是有效的，integerLiteral（）再次调用记号赋予器以返回下一个记号。

If the tokenizer returns either a plus or a minus token , the expression is still valid , and integerLiteral () calls the tokenizer one last time , looking for another integer .

如果记号赋予器返回加号或减号记号，则表达式仍然是有效的，integerLiteral（）将最后一次调用记号赋予器，以寻找另一个整数。

That 's because you didn 't save their values into the tree when the tokenizer encountered them in the input stream ; you need to modify your integerLiteral () production to do that .

那是因为当记号赋予器（tokenizer）在输入流中遇到它们时，您没有将它们的值保存到树中；您需要修改integerLiteral（）结果来执行该操作。

If the token returned by the tokenizer isn 't an integer token or an , integerLiteral () throws a ParseException and parsing is complete .

如果记号赋予器返回的记号不是整数记号或，那么integerLiteral（）抛出ParseException，同时解析完成。