You are here

class Tokenizer in Zircon Profile 8

Same name in this branch
  1. 8 vendor/symfony/css-selector/Parser/Tokenizer/Tokenizer.php \Symfony\Component\CssSelector\Parser\Tokenizer\Tokenizer
  2. 8 vendor/masterminds/html5/src/HTML5/Parser/Tokenizer.php \Masterminds\HTML5\Parser\Tokenizer
Same name and namespace in other branches
  1. 8.0 vendor/symfony/css-selector/Parser/Tokenizer/Tokenizer.php \Symfony\Component\CssSelector\Parser\Tokenizer\Tokenizer

CSS selector tokenizer.

This component is a port of the Python cssselect library, which is copyright Ian Bicking, @author Jean-François Simon <jeanfrancois.simon@sensiolabs.com>

Hierarchy

  • class \Symfony\Component\CssSelector\Parser\Tokenizer\Tokenizer

Expanded class hierarchy of Tokenizer

See also

https://github.com/SimonSapin/cssselect.

1 file declares its use of Tokenizer
Parser.php in vendor/symfony/css-selector/Parser/Parser.php

File

vendor/symfony/css-selector/Parser/Tokenizer/Tokenizer.php, line 27

Namespace

Symfony\Component\CssSelector\Parser\Tokenizer
View source
class Tokenizer {

  /**
   * @var Handler\HandlerInterface[]
   */
  private $handlers;

  /**
   * Constructor.
   */
  public function __construct() {
    $patterns = new TokenizerPatterns();
    $escaping = new TokenizerEscaping($patterns);
    $this->handlers = array(
      new Handler\WhitespaceHandler(),
      new Handler\IdentifierHandler($patterns, $escaping),
      new Handler\HashHandler($patterns, $escaping),
      new Handler\StringHandler($patterns, $escaping),
      new Handler\NumberHandler($patterns),
      new Handler\CommentHandler(),
    );
  }

  /**
   * Tokenize selector source code.
   *
   * @param Reader $reader
   *
   * @return TokenStream
   */
  public function tokenize(Reader $reader) {
    $stream = new TokenStream();
    while (!$reader
      ->isEOF()) {
      foreach ($this->handlers as $handler) {
        if ($handler
          ->handle($reader, $stream)) {
          continue 2;
        }
      }
      $stream
        ->push(new Token(Token::TYPE_DELIMITER, $reader
        ->getSubstring(1), $reader
        ->getPosition()));
      $reader
        ->moveForward(1);
    }
    return $stream
      ->push(new Token(Token::TYPE_FILE_END, null, $reader
      ->getPosition()))
      ->freeze();
  }

}

Members

Namesort descending Modifiers Type Description Overrides
Tokenizer::$handlers private property
Tokenizer::tokenize public function Tokenize selector source code.
Tokenizer::__construct public function Constructor.