public function Tokenizer::tokenize in Zircon Profile 8

Same name and namespace in other branches

1. 8.0 vendor/symfony/css-selector/Parser/Tokenizer/Tokenizer.php \Symfony\Component\CssSelector\Parser\Tokenizer\Tokenizer::tokenize()

Tokenize selector source code.

Parameters

Reader $reader:

Return value

TokenStream

File

vendor/symfony/css-selector/Parser/Tokenizer/Tokenizer.php, line 59

Class

Tokenizer

CSS selector tokenizer.

Namespace

Symfony\Component\CssSelector\Parser\Tokenizer

Code

public function tokenize(Reader $reader) {
  $stream = new TokenStream();
  while (!$reader
    ->isEOF()) {
    foreach ($this->handlers as $handler) {
      if ($handler
        ->handle($reader, $stream)) {
        continue 2;
      }
    }
    $stream
      ->push(new Token(Token::TYPE_DELIMITER, $reader
      ->getSubstring(1), $reader
      ->getPosition()));
    $reader
      ->moveForward(1);
  }
  return $stream
    ->push(new Token(Token::TYPE_FILE_END, null, $reader
    ->getPosition()))
    ->freeze();
}