html5lib-0.999999999/ 0000755 0001750 0000144 00000000000 12742037160 015151 5 ustar gsnedders users 0000000 0000000 html5lib-0.999999999/setup.cfg 0000644 0001750 0000144 00000000320 12742037160 016765 0 ustar gsnedders users 0000000 0000000 [bdist_wheel]
universal = 1
[pep8]
ignore = N
max-line-length = 139
exclude = .git,__pycache__,.tox,doc
[flake8]
ignore = N
max-line-length = 139
[egg_info]
tag_build =
tag_date = 0
tag_svn_revision = 0
html5lib-0.999999999/requirements-optional.txt 0000644 0001750 0000144 00000001041 12720211777 022260 0 ustar gsnedders users 0000000 0000000 -r requirements.txt
# We support a Genshi treewalker that can be used to serialize Genshi
# streams.
genshi
# chardet can be used as a fallback in case we are unable to determine
# the encoding of a document.
chardet>=2.2
# lxml is supported with its own treebuilder ("lxml") and otherwise
# uses the standard ElementTree support
lxml ; platform_python_implementation == 'CPython'
# DATrie can be used in place of our Python trie implementation for
# slightly better parsing performance.
datrie ; platform_python_implementation == 'CPython'
html5lib-0.999999999/LICENSE 0000644 0001750 0000144 00000002074 12145436112 016156 0 ustar gsnedders users 0000000 0000000 Copyright (c) 2006-2013 James Graham and other contributors
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
html5lib-0.999999999/pytest.ini 0000644 0001750 0000144 00000001146 12720203315 017175 0 ustar gsnedders users 0000000 0000000 [pytest]
# Output fails, errors, xpass, and warnings; ignore doctest; make warnings errors
addopts = -rfEXw -p no:doctest --strict
# Make xpass results be considered fail
xfail_strict = true
# Document our markers
markers =
DOM: mark a test as a DOM tree test
ElementTree: mark a test as a ElementTree tree test
cElementTree: mark a test as a cElementTree tree test
lxml: mark a test as a lxml tree test
genshi: mark a test as a genshi tree test
parser: mark a test as a parser test
namespaced: mark a test as a namespaced parser test
treewalker: mark a test as a treewalker test
html5lib-0.999999999/README.rst 0000644 0001750 0000144 00000010064 12741203344 016637 0 ustar gsnedders users 0000000 0000000 html5lib
========
.. image:: https://travis-ci.org/html5lib/html5lib-python.png?branch=master
:target: https://travis-ci.org/html5lib/html5lib-python
html5lib is a pure-python library for parsing HTML. It is designed to
conform to the WHATWG HTML specification, as is implemented by all major
web browsers.
Usage
-----
Simple usage follows this pattern:
.. code-block:: python
import html5lib
with open("mydocument.html", "rb") as f:
document = html5lib.parse(f)
or:
.. code-block:: python
import html5lib
document = html5lib.parse("Hello World!")
By default, the ``document`` will be an ``xml.etree`` element instance.
Whenever possible, html5lib chooses the accelerated ``ElementTree``
implementation (i.e. ``xml.etree.cElementTree`` on Python 2.x).
Two other tree types are supported: ``xml.dom.minidom`` and
``lxml.etree``. To use an alternative format, specify the name of
a treebuilder:
.. code-block:: python
import html5lib
with open("mydocument.html", "rb") as f:
lxml_etree_document = html5lib.parse(f, treebuilder="lxml")
When using with ``urllib2`` (Python 2), the charset from HTTP should be
pass into html5lib as follows:
.. code-block:: python
from contextlib import closing
from urllib2 import urlopen
import html5lib
with closing(urlopen("http://example.com/")) as f:
document = html5lib.parse(f, transport_encoding=f.info().getparam("charset"))
When using with ``urllib.request`` (Python 3), the charset from HTTP
should be pass into html5lib as follows:
.. code-block:: python
from urllib.request import urlopen
import html5lib
with urlopen("http://example.com/") as f:
document = html5lib.parse(f, transport_encoding=f.info().get_content_charset())
To have more control over the parser, create a parser object explicitly.
For instance, to make the parser raise exceptions on parse errors, use:
.. code-block:: python
import html5lib
with open("mydocument.html", "rb") as f:
parser = html5lib.HTMLParser(strict=True)
document = parser.parse(f)
When you're instantiating parser objects explicitly, pass a treebuilder
class as the ``tree`` keyword argument to use an alternative document
format:
.. code-block:: python
import html5lib
parser = html5lib.HTMLParser(tree=html5lib.getTreeBuilder("dom"))
minidom_document = parser.parse("
Hello World!")
More documentation is available at https://html5lib.readthedocs.io/.
Installation
------------
html5lib works on CPython 2.6+, CPython 3.3+ and PyPy. To install it,
use:
.. code-block:: bash
$ pip install html5lib
Optional Dependencies
---------------------
The following third-party libraries may be used for additional
functionality:
- ``datrie`` can be used under CPython to improve parsing performance
(though in almost all cases the improvement is marginal);
- ``lxml`` is supported as a tree format (for both building and
walking) under CPython (but *not* PyPy where it is known to cause
segfaults);
- ``genshi`` has a treewalker (but not builder); and
- ``chardet`` can be used as a fallback when character encoding cannot
be determined.
Bugs
----
Please report any bugs on the `issue tracker
`_.
Tests
-----
Unit tests require the ``pytest`` and ``mock`` libraries and can be
run using the ``py.test`` command in the root directory;
``ordereddict`` is required under Python 2.6. All should pass.
Test data are contained in a separate `html5lib-tests
`_ repository and included
as a submodule, thus for git checkouts they must be initialized::
$ git submodule init
$ git submodule update
If you have all compatible Python implementations available on your
system, you can run tests on all of them using the ``tox`` utility,
which can be found on PyPI.
Questions?
----------
There's a mailing list available for support on Google Groups,
`html5lib-discuss `_,
though you may get a quicker response asking on IRC in `#whatwg on
irc.freenode.net `_.
html5lib-0.999999999/tox.ini 0000644 0001750 0000144 00000000461 12717667252 016502 0 ustar gsnedders users 0000000 0000000 [tox]
envlist = {py26,py27,py33,py34,py35,pypy}-{base,optional}
[testenv]
deps =
flake8
pytest
pytest-expect>=1.1,<2.0
mock
base: six
base: webencodings
py26-base: ordereddict
optional: -r{toxinidir}/requirements-optional.txt
commands =
{envbindir}/py.test
{toxinidir}/flake8-run.sh
html5lib-0.999999999/PKG-INFO 0000644 0001750 0000144 00000040761 12742037160 016256 0 ustar gsnedders users 0000000 0000000 Metadata-Version: 1.1
Name: html5lib
Version: 0.999999999
Summary: HTML parser based on the WHATWG HTML specification
Home-page: https://github.com/html5lib/html5lib-python
Author: James Graham
Author-email: james@hoppipolla.co.uk
License: MIT License
Description: html5lib
========
.. image:: https://travis-ci.org/html5lib/html5lib-python.png?branch=master
:target: https://travis-ci.org/html5lib/html5lib-python
html5lib is a pure-python library for parsing HTML. It is designed to
conform to the WHATWG HTML specification, as is implemented by all major
web browsers.
Usage
-----
Simple usage follows this pattern:
.. code-block:: python
import html5lib
with open("mydocument.html", "rb") as f:
document = html5lib.parse(f)
or:
.. code-block:: python
import html5lib
document = html5lib.parse("Hello World!")
By default, the ``document`` will be an ``xml.etree`` element instance.
Whenever possible, html5lib chooses the accelerated ``ElementTree``
implementation (i.e. ``xml.etree.cElementTree`` on Python 2.x).
Two other tree types are supported: ``xml.dom.minidom`` and
``lxml.etree``. To use an alternative format, specify the name of
a treebuilder:
.. code-block:: python
import html5lib
with open("mydocument.html", "rb") as f:
lxml_etree_document = html5lib.parse(f, treebuilder="lxml")
When using with ``urllib2`` (Python 2), the charset from HTTP should be
pass into html5lib as follows:
.. code-block:: python
from contextlib import closing
from urllib2 import urlopen
import html5lib
with closing(urlopen("http://example.com/")) as f:
document = html5lib.parse(f, transport_encoding=f.info().getparam("charset"))
When using with ``urllib.request`` (Python 3), the charset from HTTP
should be pass into html5lib as follows:
.. code-block:: python
from urllib.request import urlopen
import html5lib
with urlopen("http://example.com/") as f:
document = html5lib.parse(f, transport_encoding=f.info().get_content_charset())
To have more control over the parser, create a parser object explicitly.
For instance, to make the parser raise exceptions on parse errors, use:
.. code-block:: python
import html5lib
with open("mydocument.html", "rb") as f:
parser = html5lib.HTMLParser(strict=True)
document = parser.parse(f)
When you're instantiating parser objects explicitly, pass a treebuilder
class as the ``tree`` keyword argument to use an alternative document
format:
.. code-block:: python
import html5lib
parser = html5lib.HTMLParser(tree=html5lib.getTreeBuilder("dom"))
minidom_document = parser.parse("
Hello World!")
More documentation is available at https://html5lib.readthedocs.io/.
Installation
------------
html5lib works on CPython 2.6+, CPython 3.3+ and PyPy. To install it,
use:
.. code-block:: bash
$ pip install html5lib
Optional Dependencies
---------------------
The following third-party libraries may be used for additional
functionality:
- ``datrie`` can be used under CPython to improve parsing performance
(though in almost all cases the improvement is marginal);
- ``lxml`` is supported as a tree format (for both building and
walking) under CPython (but *not* PyPy where it is known to cause
segfaults);
- ``genshi`` has a treewalker (but not builder); and
- ``chardet`` can be used as a fallback when character encoding cannot
be determined.
Bugs
----
Please report any bugs on the `issue tracker
`_.
Tests
-----
Unit tests require the ``pytest`` and ``mock`` libraries and can be
run using the ``py.test`` command in the root directory;
``ordereddict`` is required under Python 2.6. All should pass.
Test data are contained in a separate `html5lib-tests
`_ repository and included
as a submodule, thus for git checkouts they must be initialized::
$ git submodule init
$ git submodule update
If you have all compatible Python implementations available on your
system, you can run tests on all of them using the ``tox`` utility,
which can be found on PyPI.
Questions?
----------
There's a mailing list available for support on Google Groups,
`html5lib-discuss `_,
though you may get a quicker response asking on IRC in `#whatwg on
irc.freenode.net `_.
Change Log
----------
0.999999999/1.0b10
~~~~~~~~~~~~~~~~~~
Released on July 15, 2016
* Fix attribute order going to the tree builder to be document order
instead of reverse document order(!).
0.99999999/1.0b9
~~~~~~~~~~~~~~~~
Released on July 14, 2016
* **Added ordereddict as a mandatory dependency on Python 2.6.**
* Added ``lxml``, ``genshi``, ``datrie``, ``charade``, and ``all``
extras that will do the right thing based on the specific
interpreter implementation.
* Now requires the ``mock`` package for the testsuite.
* Cease supporting DATrie under PyPy.
* **Remove ``PullDOM`` support, as this hasn't ever been properly
tested, doesn't entirely work, and as far as I can tell is
completely unused by anyone.**
* Move testsuite to ``py.test``.
* **Fix #124: move to webencodings for decoding the input byte stream;
this makes html5lib compliant with the Encoding Standard, and
introduces a required dependency on webencodings.**
* **Cease supporting Python 3.2 (in both CPython and PyPy forms).**
* **Fix comments containing double-dash with lxml 3.5 and above.**
* **Use scripting disabled by default (as we don't implement
scripting).**
* **Fix #11, avoiding the XSS bug potentially caused by serializer
allowing attribute values to be escaped out of in old browser versions,
changing the quote_attr_values option on serializer to take one of
three values, "always" (the old True value), "legacy" (the new option,
and the new default), and "spec" (the old False value, and the old
default).**
* **Fix #72 by rewriting the sanitizer to apply only to treewalkers
(instead of the tokenizer); as such, this will require amending all
callers of it to use it via the treewalker API.**
* **Drop support of charade, now that chardet is supported once more.**
* **Replace the charset keyword argument on parse and related methods
with a set of keyword arguments: override_encoding, transport_encoding,
same_origin_parent_encoding, likely_encoding, and default_encoding.**
* **Move filters._base, treebuilder._base, and treewalkers._base to .base
to clarify their status as public.**
* **Get rid of the sanitizer package. Merge sanitizer.sanitize into the
sanitizer.htmlsanitizer module and move that to saniziter. This means
anyone who used sanitizer.sanitize or sanitizer.HTMLSanitizer needs no
code changes.**
* **Rename treewalkers.lxmletree to .etree_lxml and
treewalkers.genshistream to .genshi to have a consistent API.**
* Move a whole load of stuff (inputstream, ihatexml, trie, tokenizer,
utils) to be underscore prefixed to clarify their status as private.
0.9999999/1.0b8
~~~~~~~~~~~~~~~
Released on September 10, 2015
* Fix #195: fix the sanitizer to drop broken URLs (it threw an
exception between 0.9999 and 0.999999).
0.999999/1.0b7
~~~~~~~~~~~~~~
Released on July 7, 2015
* Fix #189: fix the sanitizer to allow relative URLs again (as it did
prior to 0.9999/1.0b5).
0.99999/1.0b6
~~~~~~~~~~~~~
Released on April 30, 2015
* Fix #188: fix the sanitizer to not throw an exception when sanitizing
bogus data URLs.
0.9999/1.0b5
~~~~~~~~~~~~
Released on April 29, 2015
* Fix #153: Sanitizer fails to treat some attributes as URLs. Despite how
this sounds, this has no known security implications. No known version
of IE (5.5 to current), Firefox (3 to current), Safari (6 to current),
Chrome (1 to current), or Opera (12 to current) will run any script
provided in these attributes.
* Pass error message to the ParseError exception in strict parsing mode.
* Allow data URIs in the sanitizer, with a whitelist of content-types.
* Add support for Python implementations that don't support lone
surrogates (read: Jython). Fixes #2.
* Remove localization of error messages. This functionality was totally
unused (and untested that everything was localizable), so we may as
well follow numerous browsers in not supporting translating technical
strings.
* Expose treewalkers.pprint as a public API.
* Add a documentEncoding property to HTML5Parser, fix #121.
0.999
~~~~~
Released on December 23, 2013
* Fix #127: add work-around for CPython issue #20007: .read(0) on
http.client.HTTPResponse drops the rest of the content.
* Fix #115: lxml treewalker can now deal with fragments containing, at
their root level, text nodes with non-ASCII characters on Python 2.
0.99
~~~~
Released on September 10, 2013
* No library changes from 1.0b3; released as 0.99 as pip has changed
behaviour from 1.4 to avoid installing pre-release versions per
PEP 440.
1.0b3
~~~~~
Released on July 24, 2013
* Removed ``RecursiveTreeWalker`` from ``treewalkers._base``. Any
implementation using it should be moved to
``NonRecursiveTreeWalker``, as everything bundled with html5lib has
for years.
* Fix #67 so that ``BufferedStream`` to correctly returns a bytes
object, thereby fixing any case where html5lib is passed a
non-seekable RawIOBase-like object.
1.0b2
~~~~~
Released on June 27, 2013
* Removed reordering of attributes within the serializer. There is now
an ``alphabetical_attributes`` option which preserves the previous
behaviour through a new filter. This allows attribute order to be
preserved through html5lib if the tree builder preserves order.
* Removed ``dom2sax`` from DOM treebuilders. It has been replaced by
``treeadapters.sax.to_sax`` which is generic and supports any
treewalker; it also resolves all known bugs with ``dom2sax``.
* Fix treewalker assertions on hitting bytes strings on
Python 2. Previous to 1.0b1, treewalkers coped with mixed
bytes/unicode data on Python 2; this reintroduces this prior
behaviour on Python 2. Behaviour is unchanged on Python 3.
1.0b1
~~~~~
Released on May 17, 2013
* Implementation updated to implement the `HTML specification
causes infinite loop;
* `Google Code 215
`_: Properly
detect seekable streams;
* `Google Code 206
`_: add
support for , ;
* `Google Code 205
`_: add
support for ;
* `Google Code 202
`_: Unicode
file breaks InputStream.
* Source code is now mostly PEP 8 compliant.
* Test harness has been improved and now depends on ``nose``.
* Documentation updated and moved to https://html5lib.readthedocs.io/.
0.95
~~~~
Released on February 11, 2012
0.90
~~~~
Released on January 17, 2010
0.11.1
~~~~~~
Released on June 12, 2008
0.11
~~~~
Released on June 10, 2008
0.10
~~~~
Released on October 7, 2007
0.9
~~~
Released on March 11, 2007
0.2
~~~
Released on January 8, 2007
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.6
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Markup :: HTML
html5lib-0.999999999/AUTHORS.rst 0000644 0001750 0000144 00000001232 12741025307 017025 0 ustar gsnedders users 0000000 0000000 Credits
=======
``html5lib`` is written and maintained by:
- James Graham
- Geoffrey Sneddon
- Łukasz Langa
Patches and suggestions
-----------------------
(In chronological order, by first commit:)
- Anne van Kesteren
- Lachlan Hunt
- lantis63
- Sam Ruby
- Thomas Broyer
- Tim Fletcher
- Mark Pilgrim
- Ryan King
- Philip Taylor
- Edward Z. Yang
- fantasai
- Mike West
- Philip Jägenstedt
- Ms2ger
- Mohammad Taha Jahangir
- Andy Wingo
- Juan Carlos Garcia Segovia
- Andreas Madsack
- Karim Valiev
- Marc DM
- Tony Lopes
- lilbludevil
- Simon Sapin
- Jon Dufresne
- Drew Hubl
- Austin Kumbera
- Jim Baker
- Michael[tm] Smith
- Marc Abramowitz
- Jon Dufresne
html5lib-0.999999999/MANIFEST.in 0000644 0001750 0000144 00000000343 12717622047 016715 0 ustar gsnedders users 0000000 0000000 include LICENSE
include AUTHORS.rst
include CHANGES.rst
include README.rst
include requirements*.txt
include .pytest.expect
include tox.ini
include pytest.ini
graft html5lib/tests/testdata
recursive-include html5lib/tests *.py
html5lib-0.999999999/CHANGES.rst 0000644 0001750 0000144 00000017765 12742036701 016773 0 ustar gsnedders users 0000000 0000000 Change Log
----------
0.999999999/1.0b10
~~~~~~~~~~~~~~~~~~
Released on July 15, 2016
* Fix attribute order going to the tree builder to be document order
instead of reverse document order(!).
0.99999999/1.0b9
~~~~~~~~~~~~~~~~
Released on July 14, 2016
* **Added ordereddict as a mandatory dependency on Python 2.6.**
* Added ``lxml``, ``genshi``, ``datrie``, ``charade``, and ``all``
extras that will do the right thing based on the specific
interpreter implementation.
* Now requires the ``mock`` package for the testsuite.
* Cease supporting DATrie under PyPy.
* **Remove ``PullDOM`` support, as this hasn't ever been properly
tested, doesn't entirely work, and as far as I can tell is
completely unused by anyone.**
* Move testsuite to ``py.test``.
* **Fix #124: move to webencodings for decoding the input byte stream;
this makes html5lib compliant with the Encoding Standard, and
introduces a required dependency on webencodings.**
* **Cease supporting Python 3.2 (in both CPython and PyPy forms).**
* **Fix comments containing double-dash with lxml 3.5 and above.**
* **Use scripting disabled by default (as we don't implement
scripting).**
* **Fix #11, avoiding the XSS bug potentially caused by serializer
allowing attribute values to be escaped out of in old browser versions,
changing the quote_attr_values option on serializer to take one of
three values, "always" (the old True value), "legacy" (the new option,
and the new default), and "spec" (the old False value, and the old
default).**
* **Fix #72 by rewriting the sanitizer to apply only to treewalkers
(instead of the tokenizer); as such, this will require amending all
callers of it to use it via the treewalker API.**
* **Drop support of charade, now that chardet is supported once more.**
* **Replace the charset keyword argument on parse and related methods
with a set of keyword arguments: override_encoding, transport_encoding,
same_origin_parent_encoding, likely_encoding, and default_encoding.**
* **Move filters._base, treebuilder._base, and treewalkers._base to .base
to clarify their status as public.**
* **Get rid of the sanitizer package. Merge sanitizer.sanitize into the
sanitizer.htmlsanitizer module and move that to saniziter. This means
anyone who used sanitizer.sanitize or sanitizer.HTMLSanitizer needs no
code changes.**
* **Rename treewalkers.lxmletree to .etree_lxml and
treewalkers.genshistream to .genshi to have a consistent API.**
* Move a whole load of stuff (inputstream, ihatexml, trie, tokenizer,
utils) to be underscore prefixed to clarify their status as private.
0.9999999/1.0b8
~~~~~~~~~~~~~~~
Released on September 10, 2015
* Fix #195: fix the sanitizer to drop broken URLs (it threw an
exception between 0.9999 and 0.999999).
0.999999/1.0b7
~~~~~~~~~~~~~~
Released on July 7, 2015
* Fix #189: fix the sanitizer to allow relative URLs again (as it did
prior to 0.9999/1.0b5).
0.99999/1.0b6
~~~~~~~~~~~~~
Released on April 30, 2015
* Fix #188: fix the sanitizer to not throw an exception when sanitizing
bogus data URLs.
0.9999/1.0b5
~~~~~~~~~~~~
Released on April 29, 2015
* Fix #153: Sanitizer fails to treat some attributes as URLs. Despite how
this sounds, this has no known security implications. No known version
of IE (5.5 to current), Firefox (3 to current), Safari (6 to current),
Chrome (1 to current), or Opera (12 to current) will run any script
provided in these attributes.
* Pass error message to the ParseError exception in strict parsing mode.
* Allow data URIs in the sanitizer, with a whitelist of content-types.
* Add support for Python implementations that don't support lone
surrogates (read: Jython). Fixes #2.
* Remove localization of error messages. This functionality was totally
unused (and untested that everything was localizable), so we may as
well follow numerous browsers in not supporting translating technical
strings.
* Expose treewalkers.pprint as a public API.
* Add a documentEncoding property to HTML5Parser, fix #121.
0.999
~~~~~
Released on December 23, 2013
* Fix #127: add work-around for CPython issue #20007: .read(0) on
http.client.HTTPResponse drops the rest of the content.
* Fix #115: lxml treewalker can now deal with fragments containing, at
their root level, text nodes with non-ASCII characters on Python 2.
0.99
~~~~
Released on September 10, 2013
* No library changes from 1.0b3; released as 0.99 as pip has changed
behaviour from 1.4 to avoid installing pre-release versions per
PEP 440.
1.0b3
~~~~~
Released on July 24, 2013
* Removed ``RecursiveTreeWalker`` from ``treewalkers._base``. Any
implementation using it should be moved to
``NonRecursiveTreeWalker``, as everything bundled with html5lib has
for years.
* Fix #67 so that ``BufferedStream`` to correctly returns a bytes
object, thereby fixing any case where html5lib is passed a
non-seekable RawIOBase-like object.
1.0b2
~~~~~
Released on June 27, 2013
* Removed reordering of attributes within the serializer. There is now
an ``alphabetical_attributes`` option which preserves the previous
behaviour through a new filter. This allows attribute order to be
preserved through html5lib if the tree builder preserves order.
* Removed ``dom2sax`` from DOM treebuilders. It has been replaced by
``treeadapters.sax.to_sax`` which is generic and supports any
treewalker; it also resolves all known bugs with ``dom2sax``.
* Fix treewalker assertions on hitting bytes strings on
Python 2. Previous to 1.0b1, treewalkers coped with mixed
bytes/unicode data on Python 2; this reintroduces this prior
behaviour on Python 2. Behaviour is unchanged on Python 3.
1.0b1
~~~~~
Released on May 17, 2013
* Implementation updated to implement the `HTML specification
causes infinite loop;
* `Google Code 215
`_: Properly
detect seekable streams;
* `Google Code 206
`_: add
support for , ;
* `Google Code 205
`_: add
support for ;
* `Google Code 202
`_: Unicode
file breaks InputStream.
* Source code is now mostly PEP 8 compliant.
* Test harness has been improved and now depends on ``nose``.
* Documentation updated and moved to https://html5lib.readthedocs.io/.
0.95
~~~~
Released on February 11, 2012
0.90
~~~~
Released on January 17, 2010
0.11.1
~~~~~~
Released on June 12, 2008
0.11
~~~~
Released on June 10, 2008
0.10
~~~~
Released on October 7, 2007
0.9
~~~
Released on March 11, 2007
0.2
~~~
Released on January 8, 2007
html5lib-0.999999999/html5lib.egg-info/ 0000755 0001750 0000144 00000000000 12742037160 020363 5 ustar gsnedders users 0000000 0000000 html5lib-0.999999999/html5lib.egg-info/requires.txt 0000644 0001750 0000144 00000000503 12742037156 022766 0 ustar gsnedders users 0000000 0000000 six
webencodings
setuptools>=18.5
[:python_version == '2.6']
ordereddict
[all]
genshi
chardet>=2.2
[all:platform_python_implementation == 'CPython']
datrie
lxml
[chardet]
chardet>=2.2
[datrie:platform_python_implementation == 'CPython']
datrie
[genshi]
genshi
[lxml:platform_python_implementation == 'CPython']
lxml
html5lib-0.999999999/html5lib.egg-info/PKG-INFO 0000644 0001750 0000144 00000040761 12742037156 021475 0 ustar gsnedders users 0000000 0000000 Metadata-Version: 1.1
Name: html5lib
Version: 0.999999999
Summary: HTML parser based on the WHATWG HTML specification
Home-page: https://github.com/html5lib/html5lib-python
Author: James Graham
Author-email: james@hoppipolla.co.uk
License: MIT License
Description: html5lib
========
.. image:: https://travis-ci.org/html5lib/html5lib-python.png?branch=master
:target: https://travis-ci.org/html5lib/html5lib-python
html5lib is a pure-python library for parsing HTML. It is designed to
conform to the WHATWG HTML specification, as is implemented by all major
web browsers.
Usage
-----
Simple usage follows this pattern:
.. code-block:: python
import html5lib
with open("mydocument.html", "rb") as f:
document = html5lib.parse(f)
or:
.. code-block:: python
import html5lib
document = html5lib.parse("Hello World!")
By default, the ``document`` will be an ``xml.etree`` element instance.
Whenever possible, html5lib chooses the accelerated ``ElementTree``
implementation (i.e. ``xml.etree.cElementTree`` on Python 2.x).
Two other tree types are supported: ``xml.dom.minidom`` and
``lxml.etree``. To use an alternative format, specify the name of
a treebuilder:
.. code-block:: python
import html5lib
with open("mydocument.html", "rb") as f:
lxml_etree_document = html5lib.parse(f, treebuilder="lxml")
When using with ``urllib2`` (Python 2), the charset from HTTP should be
pass into html5lib as follows:
.. code-block:: python
from contextlib import closing
from urllib2 import urlopen
import html5lib
with closing(urlopen("http://example.com/")) as f:
document = html5lib.parse(f, transport_encoding=f.info().getparam("charset"))
When using with ``urllib.request`` (Python 3), the charset from HTTP
should be pass into html5lib as follows:
.. code-block:: python
from urllib.request import urlopen
import html5lib
with urlopen("http://example.com/") as f:
document = html5lib.parse(f, transport_encoding=f.info().get_content_charset())
To have more control over the parser, create a parser object explicitly.
For instance, to make the parser raise exceptions on parse errors, use:
.. code-block:: python
import html5lib
with open("mydocument.html", "rb") as f:
parser = html5lib.HTMLParser(strict=True)
document = parser.parse(f)
When you're instantiating parser objects explicitly, pass a treebuilder
class as the ``tree`` keyword argument to use an alternative document
format:
.. code-block:: python
import html5lib
parser = html5lib.HTMLParser(tree=html5lib.getTreeBuilder("dom"))
minidom_document = parser.parse("
Hello World!")
More documentation is available at https://html5lib.readthedocs.io/.
Installation
------------
html5lib works on CPython 2.6+, CPython 3.3+ and PyPy. To install it,
use:
.. code-block:: bash
$ pip install html5lib
Optional Dependencies
---------------------
The following third-party libraries may be used for additional
functionality:
- ``datrie`` can be used under CPython to improve parsing performance
(though in almost all cases the improvement is marginal);
- ``lxml`` is supported as a tree format (for both building and
walking) under CPython (but *not* PyPy where it is known to cause
segfaults);
- ``genshi`` has a treewalker (but not builder); and
- ``chardet`` can be used as a fallback when character encoding cannot
be determined.
Bugs
----
Please report any bugs on the `issue tracker
`_.
Tests
-----
Unit tests require the ``pytest`` and ``mock`` libraries and can be
run using the ``py.test`` command in the root directory;
``ordereddict`` is required under Python 2.6. All should pass.
Test data are contained in a separate `html5lib-tests
`_ repository and included
as a submodule, thus for git checkouts they must be initialized::
$ git submodule init
$ git submodule update
If you have all compatible Python implementations available on your
system, you can run tests on all of them using the ``tox`` utility,
which can be found on PyPI.
Questions?
----------
There's a mailing list available for support on Google Groups,
`html5lib-discuss `_,
though you may get a quicker response asking on IRC in `#whatwg on
irc.freenode.net `_.
Change Log
----------
0.999999999/1.0b10
~~~~~~~~~~~~~~~~~~
Released on July 15, 2016
* Fix attribute order going to the tree builder to be document order
instead of reverse document order(!).
0.99999999/1.0b9
~~~~~~~~~~~~~~~~
Released on July 14, 2016
* **Added ordereddict as a mandatory dependency on Python 2.6.**
* Added ``lxml``, ``genshi``, ``datrie``, ``charade``, and ``all``
extras that will do the right thing based on the specific
interpreter implementation.
* Now requires the ``mock`` package for the testsuite.
* Cease supporting DATrie under PyPy.
* **Remove ``PullDOM`` support, as this hasn't ever been properly
tested, doesn't entirely work, and as far as I can tell is
completely unused by anyone.**
* Move testsuite to ``py.test``.
* **Fix #124: move to webencodings for decoding the input byte stream;
this makes html5lib compliant with the Encoding Standard, and
introduces a required dependency on webencodings.**
* **Cease supporting Python 3.2 (in both CPython and PyPy forms).**
* **Fix comments containing double-dash with lxml 3.5 and above.**
* **Use scripting disabled by default (as we don't implement
scripting).**
* **Fix #11, avoiding the XSS bug potentially caused by serializer
allowing attribute values to be escaped out of in old browser versions,
changing the quote_attr_values option on serializer to take one of
three values, "always" (the old True value), "legacy" (the new option,
and the new default), and "spec" (the old False value, and the old
default).**
* **Fix #72 by rewriting the sanitizer to apply only to treewalkers
(instead of the tokenizer); as such, this will require amending all
callers of it to use it via the treewalker API.**
* **Drop support of charade, now that chardet is supported once more.**
* **Replace the charset keyword argument on parse and related methods
with a set of keyword arguments: override_encoding, transport_encoding,
same_origin_parent_encoding, likely_encoding, and default_encoding.**
* **Move filters._base, treebuilder._base, and treewalkers._base to .base
to clarify their status as public.**
* **Get rid of the sanitizer package. Merge sanitizer.sanitize into the
sanitizer.htmlsanitizer module and move that to saniziter. This means
anyone who used sanitizer.sanitize or sanitizer.HTMLSanitizer needs no
code changes.**
* **Rename treewalkers.lxmletree to .etree_lxml and
treewalkers.genshistream to .genshi to have a consistent API.**
* Move a whole load of stuff (inputstream, ihatexml, trie, tokenizer,
utils) to be underscore prefixed to clarify their status as private.
0.9999999/1.0b8
~~~~~~~~~~~~~~~
Released on September 10, 2015
* Fix #195: fix the sanitizer to drop broken URLs (it threw an
exception between 0.9999 and 0.999999).
0.999999/1.0b7
~~~~~~~~~~~~~~
Released on July 7, 2015
* Fix #189: fix the sanitizer to allow relative URLs again (as it did
prior to 0.9999/1.0b5).
0.99999/1.0b6
~~~~~~~~~~~~~
Released on April 30, 2015
* Fix #188: fix the sanitizer to not throw an exception when sanitizing
bogus data URLs.
0.9999/1.0b5
~~~~~~~~~~~~
Released on April 29, 2015
* Fix #153: Sanitizer fails to treat some attributes as URLs. Despite how
this sounds, this has no known security implications. No known version
of IE (5.5 to current), Firefox (3 to current), Safari (6 to current),
Chrome (1 to current), or Opera (12 to current) will run any script
provided in these attributes.
* Pass error message to the ParseError exception in strict parsing mode.
* Allow data URIs in the sanitizer, with a whitelist of content-types.
* Add support for Python implementations that don't support lone
surrogates (read: Jython). Fixes #2.
* Remove localization of error messages. This functionality was totally
unused (and untested that everything was localizable), so we may as
well follow numerous browsers in not supporting translating technical
strings.
* Expose treewalkers.pprint as a public API.
* Add a documentEncoding property to HTML5Parser, fix #121.
0.999
~~~~~
Released on December 23, 2013
* Fix #127: add work-around for CPython issue #20007: .read(0) on
http.client.HTTPResponse drops the rest of the content.
* Fix #115: lxml treewalker can now deal with fragments containing, at
their root level, text nodes with non-ASCII characters on Python 2.
0.99
~~~~
Released on September 10, 2013
* No library changes from 1.0b3; released as 0.99 as pip has changed
behaviour from 1.4 to avoid installing pre-release versions per
PEP 440.
1.0b3
~~~~~
Released on July 24, 2013
* Removed ``RecursiveTreeWalker`` from ``treewalkers._base``. Any
implementation using it should be moved to
``NonRecursiveTreeWalker``, as everything bundled with html5lib has
for years.
* Fix #67 so that ``BufferedStream`` to correctly returns a bytes
object, thereby fixing any case where html5lib is passed a
non-seekable RawIOBase-like object.
1.0b2
~~~~~
Released on June 27, 2013
* Removed reordering of attributes within the serializer. There is now
an ``alphabetical_attributes`` option which preserves the previous
behaviour through a new filter. This allows attribute order to be
preserved through html5lib if the tree builder preserves order.
* Removed ``dom2sax`` from DOM treebuilders. It has been replaced by
``treeadapters.sax.to_sax`` which is generic and supports any
treewalker; it also resolves all known bugs with ``dom2sax``.
* Fix treewalker assertions on hitting bytes strings on
Python 2. Previous to 1.0b1, treewalkers coped with mixed
bytes/unicode data on Python 2; this reintroduces this prior
behaviour on Python 2. Behaviour is unchanged on Python 3.
1.0b1
~~~~~
Released on May 17, 2013
* Implementation updated to implement the `HTML specification
causes infinite loop;
* `Google Code 215
`_: Properly
detect seekable streams;
* `Google Code 206
`_: add
support for , ;
* `Google Code 205
`_: add
support for ;
* `Google Code 202
`_: Unicode
file breaks InputStream.
* Source code is now mostly PEP 8 compliant.
* Test harness has been improved and now depends on ``nose``.
* Documentation updated and moved to https://html5lib.readthedocs.io/.
0.95
~~~~
Released on February 11, 2012
0.90
~~~~
Released on January 17, 2010
0.11.1
~~~~~~
Released on June 12, 2008
0.11
~~~~
Released on June 10, 2008
0.10
~~~~
Released on October 7, 2007
0.9
~~~
Released on March 11, 2007
0.2
~~~
Released on January 8, 2007
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.6
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Markup :: HTML
html5lib-0.999999999/html5lib.egg-info/SOURCES.txt 0000644 0001750 0000144 00000014335 12742037160 022255 0 ustar gsnedders users 0000000 0000000 .pytest.expect
AUTHORS.rst
CHANGES.rst
LICENSE
MANIFEST.in
README.rst
pytest.ini
requirements-optional.txt
requirements-test.txt
requirements.txt
setup.cfg
setup.py
tox.ini
html5lib/__init__.py
html5lib/_ihatexml.py
html5lib/_inputstream.py
html5lib/_tokenizer.py
html5lib/_utils.py
html5lib/constants.py
html5lib/html5parser.py
html5lib/serializer.py
html5lib.egg-info/PKG-INFO
html5lib.egg-info/SOURCES.txt
html5lib.egg-info/dependency_links.txt
html5lib.egg-info/requires.txt
html5lib.egg-info/top_level.txt
html5lib/_trie/__init__.py
html5lib/_trie/_base.py
html5lib/_trie/datrie.py
html5lib/_trie/py.py
html5lib/filters/__init__.py
html5lib/filters/alphabeticalattributes.py
html5lib/filters/base.py
html5lib/filters/inject_meta_charset.py
html5lib/filters/lint.py
html5lib/filters/optionaltags.py
html5lib/filters/sanitizer.py
html5lib/filters/whitespace.py
html5lib/tests/__init__.py
html5lib/tests/conftest.py
html5lib/tests/sanitizer.py
html5lib/tests/support.py
html5lib/tests/test_encoding.py
html5lib/tests/test_meta.py
html5lib/tests/test_optionaltags_filter.py
html5lib/tests/test_parser2.py
html5lib/tests/test_sanitizer.py
html5lib/tests/test_serializer.py
html5lib/tests/test_stream.py
html5lib/tests/test_treeadapters.py
html5lib/tests/test_treewalkers.py
html5lib/tests/test_whitespace_filter.py
html5lib/tests/tokenizer.py
html5lib/tests/tokenizertotree.py
html5lib/tests/tree_construction.py
html5lib/tests/testdata/.git
html5lib/tests/testdata/.gitattributes
html5lib/tests/testdata/AUTHORS.rst
html5lib/tests/testdata/LICENSE
html5lib/tests/testdata/encoding/test-yahoo-jp.dat
html5lib/tests/testdata/encoding/tests1.dat
html5lib/tests/testdata/encoding/tests2.dat
html5lib/tests/testdata/encoding/chardet/test_big5.txt
html5lib/tests/testdata/serializer/core.test
html5lib/tests/testdata/serializer/injectmeta.test
html5lib/tests/testdata/serializer/optionaltags.test
html5lib/tests/testdata/serializer/options.test
html5lib/tests/testdata/serializer/whitespace.test
html5lib/tests/testdata/tokenizer/README.md
html5lib/tests/testdata/tokenizer/contentModelFlags.test
html5lib/tests/testdata/tokenizer/domjs.test
html5lib/tests/testdata/tokenizer/entities.test
html5lib/tests/testdata/tokenizer/escapeFlag.test
html5lib/tests/testdata/tokenizer/namedEntities.test
html5lib/tests/testdata/tokenizer/numericEntities.test
html5lib/tests/testdata/tokenizer/pendingSpecChanges.test
html5lib/tests/testdata/tokenizer/test1.test
html5lib/tests/testdata/tokenizer/test2.test
html5lib/tests/testdata/tokenizer/test3.test
html5lib/tests/testdata/tokenizer/test4.test
html5lib/tests/testdata/tokenizer/unicodeChars.test
html5lib/tests/testdata/tokenizer/unicodeCharsProblematic.test
html5lib/tests/testdata/tokenizer/xmlViolation.test
html5lib/tests/testdata/tree-construction/README.md
html5lib/tests/testdata/tree-construction/adoption01.dat
html5lib/tests/testdata/tree-construction/adoption02.dat
html5lib/tests/testdata/tree-construction/comments01.dat
html5lib/tests/testdata/tree-construction/doctype01.dat
html5lib/tests/testdata/tree-construction/domjs-unsafe.dat
html5lib/tests/testdata/tree-construction/entities01.dat
html5lib/tests/testdata/tree-construction/entities02.dat
html5lib/tests/testdata/tree-construction/foreign-fragment.dat
html5lib/tests/testdata/tree-construction/html5test-com.dat
html5lib/tests/testdata/tree-construction/inbody01.dat
html5lib/tests/testdata/tree-construction/isindex.dat
html5lib/tests/testdata/tree-construction/main-element.dat
html5lib/tests/testdata/tree-construction/math.dat
html5lib/tests/testdata/tree-construction/menuitem-element.dat
html5lib/tests/testdata/tree-construction/namespace-sensitivity.dat
html5lib/tests/testdata/tree-construction/noscript01.dat
html5lib/tests/testdata/tree-construction/pending-spec-changes-plain-text-unsafe.dat
html5lib/tests/testdata/tree-construction/pending-spec-changes.dat
html5lib/tests/testdata/tree-construction/plain-text-unsafe.dat
html5lib/tests/testdata/tree-construction/ruby.dat
html5lib/tests/testdata/tree-construction/scriptdata01.dat
html5lib/tests/testdata/tree-construction/tables01.dat
html5lib/tests/testdata/tree-construction/template.dat
html5lib/tests/testdata/tree-construction/tests1.dat
html5lib/tests/testdata/tree-construction/tests10.dat
html5lib/tests/testdata/tree-construction/tests11.dat
html5lib/tests/testdata/tree-construction/tests12.dat
html5lib/tests/testdata/tree-construction/tests14.dat
html5lib/tests/testdata/tree-construction/tests15.dat
html5lib/tests/testdata/tree-construction/tests16.dat
html5lib/tests/testdata/tree-construction/tests17.dat
html5lib/tests/testdata/tree-construction/tests18.dat
html5lib/tests/testdata/tree-construction/tests19.dat
html5lib/tests/testdata/tree-construction/tests2.dat
html5lib/tests/testdata/tree-construction/tests20.dat
html5lib/tests/testdata/tree-construction/tests21.dat
html5lib/tests/testdata/tree-construction/tests22.dat
html5lib/tests/testdata/tree-construction/tests23.dat
html5lib/tests/testdata/tree-construction/tests24.dat
html5lib/tests/testdata/tree-construction/tests25.dat
html5lib/tests/testdata/tree-construction/tests26.dat
html5lib/tests/testdata/tree-construction/tests3.dat
html5lib/tests/testdata/tree-construction/tests4.dat
html5lib/tests/testdata/tree-construction/tests5.dat
html5lib/tests/testdata/tree-construction/tests6.dat
html5lib/tests/testdata/tree-construction/tests7.dat
html5lib/tests/testdata/tree-construction/tests8.dat
html5lib/tests/testdata/tree-construction/tests9.dat
html5lib/tests/testdata/tree-construction/tests_innerHTML_1.dat
html5lib/tests/testdata/tree-construction/tricky01.dat
html5lib/tests/testdata/tree-construction/webkit01.dat
html5lib/tests/testdata/tree-construction/webkit02.dat
html5lib/tests/testdata/tree-construction/scripted/adoption01.dat
html5lib/tests/testdata/tree-construction/scripted/ark.dat
html5lib/tests/testdata/tree-construction/scripted/webkit01.dat
html5lib/treeadapters/__init__.py
html5lib/treeadapters/genshi.py
html5lib/treeadapters/sax.py
html5lib/treebuilders/__init__.py
html5lib/treebuilders/base.py
html5lib/treebuilders/dom.py
html5lib/treebuilders/etree.py
html5lib/treebuilders/etree_lxml.py
html5lib/treewalkers/__init__.py
html5lib/treewalkers/base.py
html5lib/treewalkers/dom.py
html5lib/treewalkers/etree.py
html5lib/treewalkers/etree_lxml.py
html5lib/treewalkers/genshi.py html5lib-0.999999999/html5lib.egg-info/top_level.txt 0000644 0001750 0000144 00000000011 12742037156 023112 0 ustar gsnedders users 0000000 0000000 html5lib
html5lib-0.999999999/html5lib.egg-info/dependency_links.txt 0000644 0001750 0000144 00000000001 12742037156 024436 0 ustar gsnedders users 0000000 0000000
html5lib-0.999999999/html5lib/ 0000755 0001750 0000144 00000000000 12742037160 016671 5 ustar gsnedders users 0000000 0000000 html5lib-0.999999999/html5lib/_trie/ 0000755 0001750 0000144 00000000000 12742037160 017773 5 ustar gsnedders users 0000000 0000000 html5lib-0.999999999/html5lib/_trie/datrie.py 0000644 0001750 0000144 00000002216 12741761364 021627 0 ustar gsnedders users 0000000 0000000 from __future__ import absolute_import, division, unicode_literals
from datrie import Trie as DATrie
from six import text_type
from ._base import Trie as ABCTrie
class Trie(ABCTrie):
def __init__(self, data):
chars = set()
for key in data.keys():
if not isinstance(key, text_type):
raise TypeError("All keys must be strings")
for char in key:
chars.add(char)
self._data = DATrie("".join(chars))
for key, value in data.items():
self._data[key] = value
def __contains__(self, key):
return key in self._data
def __len__(self):
return len(self._data)
def __iter__(self):
raise NotImplementedError()
def __getitem__(self, key):
return self._data[key]
def keys(self, prefix=None):
return self._data.keys(prefix)
def has_keys_with_prefix(self, prefix):
return self._data.has_keys_with_prefix(prefix)
def longest_prefix(self, prefix):
return self._data.longest_prefix(prefix)
def longest_prefix_item(self, prefix):
return self._data.longest_prefix_item(prefix)
html5lib-0.999999999/html5lib/_trie/py.py 0000644 0001750 0000144 00000003343 12741761364 021011 0 ustar gsnedders users 0000000 0000000 from __future__ import absolute_import, division, unicode_literals
from six import text_type
from bisect import bisect_left
from ._base import Trie as ABCTrie
class Trie(ABCTrie):
def __init__(self, data):
if not all(isinstance(x, text_type) for x in data.keys()):
raise TypeError("All keys must be strings")
self._data = data
self._keys = sorted(data.keys())
self._cachestr = ""
self._cachepoints = (0, len(data))
def __contains__(self, key):
return key in self._data
def __len__(self):
return len(self._data)
def __iter__(self):
return iter(self._data)
def __getitem__(self, key):
return self._data[key]
def keys(self, prefix=None):
if prefix is None or prefix == "" or not self._keys:
return set(self._keys)
if prefix.startswith(self._cachestr):
lo, hi = self._cachepoints
start = i = bisect_left(self._keys, prefix, lo, hi)
else:
start = i = bisect_left(self._keys, prefix)
keys = set()
if start == len(self._keys):
return keys
while self._keys[i].startswith(prefix):
keys.add(self._keys[i])
i += 1
self._cachestr = prefix
self._cachepoints = (start, i)
return keys
def has_keys_with_prefix(self, prefix):
if prefix in self._data:
return True
if prefix.startswith(self._cachestr):
lo, hi = self._cachepoints
i = bisect_left(self._keys, prefix, lo, hi)
else:
i = bisect_left(self._keys, prefix)
if i == len(self._keys):
return False
return self._keys[i].startswith(prefix)
html5lib-0.999999999/html5lib/_trie/__init__.py 0000644 0001750 0000144 00000000441 12741761364 022114 0 ustar gsnedders users 0000000 0000000 from __future__ import absolute_import, division, unicode_literals
from .py import Trie as PyTrie
Trie = PyTrie
# pylint:disable=wrong-import-position
try:
from .datrie import Trie as DATrie
except ImportError:
pass
else:
Trie = DATrie
# pylint:enable=wrong-import-position
html5lib-0.999999999/html5lib/_trie/_base.py 0000644 0001750 0000144 00000001723 12741761364 021432 0 ustar gsnedders users 0000000 0000000 from __future__ import absolute_import, division, unicode_literals
from collections import Mapping
class Trie(Mapping):
"""Abstract base class for tries"""
def keys(self, prefix=None):
# pylint:disable=arguments-differ
keys = super(Trie, self).keys()
if prefix is None:
return set(keys)
# Python 2.6: no set comprehensions
return set([x for x in keys if x.startswith(prefix)])
def has_keys_with_prefix(self, prefix):
for key in self.keys():
if key.startswith(prefix):
return True
return False
def longest_prefix(self, prefix):
if prefix in self:
return prefix
for i in range(1, len(prefix) + 1):
if prefix[:-i] in self:
return prefix[:-i]
raise KeyError(prefix)
def longest_prefix_item(self, prefix):
lprefix = self.longest_prefix(prefix)
return (lprefix, self[lprefix])
html5lib-0.999999999/html5lib/_inputstream.py 0000644 0001750 0000144 00000077353 12741761364 022005 0 ustar gsnedders users 0000000 0000000 from __future__ import absolute_import, division, unicode_literals
from six import text_type, binary_type
from six.moves import http_client, urllib
import codecs
import re
import webencodings
from .constants import EOF, spaceCharacters, asciiLetters, asciiUppercase
from .constants import ReparseException
from . import _utils
from io import StringIO
try:
from io import BytesIO
except ImportError:
BytesIO = StringIO
# Non-unicode versions of constants for use in the pre-parser
spaceCharactersBytes = frozenset([item.encode("ascii") for item in spaceCharacters])
asciiLettersBytes = frozenset([item.encode("ascii") for item in asciiLetters])
asciiUppercaseBytes = frozenset([item.encode("ascii") for item in asciiUppercase])
spacesAngleBrackets = spaceCharactersBytes | frozenset([b">", b"<"])
invalid_unicode_no_surrogate = "[\u0001-\u0008\u000B\u000E-\u001F\u007F-\u009F\uFDD0-\uFDEF\uFFFE\uFFFF\U0001FFFE\U0001FFFF\U0002FFFE\U0002FFFF\U0003FFFE\U0003FFFF\U0004FFFE\U0004FFFF\U0005FFFE\U0005FFFF\U0006FFFE\U0006FFFF\U0007FFFE\U0007FFFF\U0008FFFE\U0008FFFF\U0009FFFE\U0009FFFF\U000AFFFE\U000AFFFF\U000BFFFE\U000BFFFF\U000CFFFE\U000CFFFF\U000DFFFE\U000DFFFF\U000EFFFE\U000EFFFF\U000FFFFE\U000FFFFF\U0010FFFE\U0010FFFF]" # noqa
if _utils.supports_lone_surrogates:
# Use one extra step of indirection and create surrogates with
# eval. Not using this indirection would introduce an illegal
# unicode literal on platforms not supporting such lone
# surrogates.
assert invalid_unicode_no_surrogate[-1] == "]" and invalid_unicode_no_surrogate.count("]") == 1
invalid_unicode_re = re.compile(invalid_unicode_no_surrogate[:-1] +
eval('"\\uD800-\\uDFFF"') + # pylint:disable=eval-used
"]")
else:
invalid_unicode_re = re.compile(invalid_unicode_no_surrogate)
non_bmp_invalid_codepoints = set([0x1FFFE, 0x1FFFF, 0x2FFFE, 0x2FFFF, 0x3FFFE,
0x3FFFF, 0x4FFFE, 0x4FFFF, 0x5FFFE, 0x5FFFF,
0x6FFFE, 0x6FFFF, 0x7FFFE, 0x7FFFF, 0x8FFFE,
0x8FFFF, 0x9FFFE, 0x9FFFF, 0xAFFFE, 0xAFFFF,
0xBFFFE, 0xBFFFF, 0xCFFFE, 0xCFFFF, 0xDFFFE,
0xDFFFF, 0xEFFFE, 0xEFFFF, 0xFFFFE, 0xFFFFF,
0x10FFFE, 0x10FFFF])
ascii_punctuation_re = re.compile("[\u0009-\u000D\u0020-\u002F\u003A-\u0040\u005B-\u0060\u007B-\u007E]")
# Cache for charsUntil()
charsUntilRegEx = {}
class BufferedStream(object):
"""Buffering for streams that do not have buffering of their own
The buffer is implemented as a list of chunks on the assumption that
joining many strings will be slow since it is O(n**2)
"""
def __init__(self, stream):
self.stream = stream
self.buffer = []
self.position = [-1, 0] # chunk number, offset
def tell(self):
pos = 0
for chunk in self.buffer[:self.position[0]]:
pos += len(chunk)
pos += self.position[1]
return pos
def seek(self, pos):
assert pos <= self._bufferedBytes()
offset = pos
i = 0
while len(self.buffer[i]) < offset:
offset -= len(self.buffer[i])
i += 1
self.position = [i, offset]
def read(self, bytes):
if not self.buffer:
return self._readStream(bytes)
elif (self.position[0] == len(self.buffer) and
self.position[1] == len(self.buffer[-1])):
return self._readStream(bytes)
else:
return self._readFromBuffer(bytes)
def _bufferedBytes(self):
return sum([len(item) for item in self.buffer])
def _readStream(self, bytes):
data = self.stream.read(bytes)
self.buffer.append(data)
self.position[0] += 1
self.position[1] = len(data)
return data
def _readFromBuffer(self, bytes):
remainingBytes = bytes
rv = []
bufferIndex = self.position[0]
bufferOffset = self.position[1]
while bufferIndex < len(self.buffer) and remainingBytes != 0:
assert remainingBytes > 0
bufferedData = self.buffer[bufferIndex]
if remainingBytes <= len(bufferedData) - bufferOffset:
bytesToRead = remainingBytes
self.position = [bufferIndex, bufferOffset + bytesToRead]
else:
bytesToRead = len(bufferedData) - bufferOffset
self.position = [bufferIndex, len(bufferedData)]
bufferIndex += 1
rv.append(bufferedData[bufferOffset:bufferOffset + bytesToRead])
remainingBytes -= bytesToRead
bufferOffset = 0
if remainingBytes:
rv.append(self._readStream(remainingBytes))
return b"".join(rv)
def HTMLInputStream(source, **kwargs):
# Work around Python bug #20007: read(0) closes the connection.
# http://bugs.python.org/issue20007
if (isinstance(source, http_client.HTTPResponse) or
# Also check for addinfourl wrapping HTTPResponse
(isinstance(source, urllib.response.addbase) and
isinstance(source.fp, http_client.HTTPResponse))):
isUnicode = False
elif hasattr(source, "read"):
isUnicode = isinstance(source.read(0), text_type)
else:
isUnicode = isinstance(source, text_type)
if isUnicode:
encodings = [x for x in kwargs if x.endswith("_encoding")]
if encodings:
raise TypeError("Cannot set an encoding with a unicode input, set %r" % encodings)
return HTMLUnicodeInputStream(source, **kwargs)
else:
return HTMLBinaryInputStream(source, **kwargs)
class HTMLUnicodeInputStream(object):
"""Provides a unicode stream of characters to the HTMLTokenizer.
This class takes care of character encoding and removing or replacing
incorrect byte-sequences and also provides column and line tracking.
"""
_defaultChunkSize = 10240
def __init__(self, source):
"""Initialises the HTMLInputStream.
HTMLInputStream(source, [encoding]) -> Normalized stream from source
for use by html5lib.
source can be either a file-object, local filename or a string.
The optional encoding parameter must be a string that indicates
the encoding. If specified, that encoding will be used,
regardless of any BOM or later declaration (such as in a meta
element)
"""
if not _utils.supports_lone_surrogates:
# Such platforms will have already checked for such
# surrogate errors, so no need to do this checking.
self.reportCharacterErrors = None
elif len("\U0010FFFF") == 1:
self.reportCharacterErrors = self.characterErrorsUCS4
else:
self.reportCharacterErrors = self.characterErrorsUCS2
# List of where new lines occur
self.newLines = [0]
self.charEncoding = (lookupEncoding("utf-8"), "certain")
self.dataStream = self.openStream(source)
self.reset()
def reset(self):
self.chunk = ""
self.chunkSize = 0
self.chunkOffset = 0
self.errors = []
# number of (complete) lines in previous chunks
self.prevNumLines = 0
# number of columns in the last line of the previous chunk
self.prevNumCols = 0
# Deal with CR LF and surrogates split over chunk boundaries
self._bufferedCharacter = None
def openStream(self, source):
"""Produces a file object from source.
source can be either a file object, local filename or a string.
"""
# Already a file object
if hasattr(source, 'read'):
stream = source
else:
stream = StringIO(source)
return stream
def _position(self, offset):
chunk = self.chunk
nLines = chunk.count('\n', 0, offset)
positionLine = self.prevNumLines + nLines
lastLinePos = chunk.rfind('\n', 0, offset)
if lastLinePos == -1:
positionColumn = self.prevNumCols + offset
else:
positionColumn = offset - (lastLinePos + 1)
return (positionLine, positionColumn)
def position(self):
"""Returns (line, col) of the current position in the stream."""
line, col = self._position(self.chunkOffset)
return (line + 1, col)
def char(self):
""" Read one character from the stream or queue if available. Return
EOF when EOF is reached.
"""
# Read a new chunk from the input stream if necessary
if self.chunkOffset >= self.chunkSize:
if not self.readChunk():
return EOF
chunkOffset = self.chunkOffset
char = self.chunk[chunkOffset]
self.chunkOffset = chunkOffset + 1
return char
def readChunk(self, chunkSize=None):
if chunkSize is None:
chunkSize = self._defaultChunkSize
self.prevNumLines, self.prevNumCols = self._position(self.chunkSize)
self.chunk = ""
self.chunkSize = 0
self.chunkOffset = 0
data = self.dataStream.read(chunkSize)
# Deal with CR LF and surrogates broken across chunks
if self._bufferedCharacter:
data = self._bufferedCharacter + data
self._bufferedCharacter = None
elif not data:
# We have no more data, bye-bye stream
return False
if len(data) > 1:
lastv = ord(data[-1])
if lastv == 0x0D or 0xD800 <= lastv <= 0xDBFF:
self._bufferedCharacter = data[-1]
data = data[:-1]
if self.reportCharacterErrors:
self.reportCharacterErrors(data)
# Replace invalid characters
data = data.replace("\r\n", "\n")
data = data.replace("\r", "\n")
self.chunk = data
self.chunkSize = len(data)
return True
def characterErrorsUCS4(self, data):
for _ in range(len(invalid_unicode_re.findall(data))):
self.errors.append("invalid-codepoint")
def characterErrorsUCS2(self, data):
# Someone picked the wrong compile option
# You lose
skip = False
for match in invalid_unicode_re.finditer(data):
if skip:
continue
codepoint = ord(match.group())
pos = match.start()
# Pretty sure there should be endianness issues here
if _utils.isSurrogatePair(data[pos:pos + 2]):
# We have a surrogate pair!
char_val = _utils.surrogatePairToCodepoint(data[pos:pos + 2])
if char_val in non_bmp_invalid_codepoints:
self.errors.append("invalid-codepoint")
skip = True
elif (codepoint >= 0xD800 and codepoint <= 0xDFFF and
pos == len(data) - 1):
self.errors.append("invalid-codepoint")
else:
skip = False
self.errors.append("invalid-codepoint")
def charsUntil(self, characters, opposite=False):
""" Returns a string of characters from the stream up to but not
including any character in 'characters' or EOF. 'characters' must be
a container that supports the 'in' method and iteration over its
characters.
"""
# Use a cache of regexps to find the required characters
try:
chars = charsUntilRegEx[(characters, opposite)]
except KeyError:
if __debug__:
for c in characters:
assert(ord(c) < 128)
regex = "".join(["\\x%02x" % ord(c) for c in characters])
if not opposite:
regex = "^%s" % regex
chars = charsUntilRegEx[(characters, opposite)] = re.compile("[%s]+" % regex)
rv = []
while True:
# Find the longest matching prefix
m = chars.match(self.chunk, self.chunkOffset)
if m is None:
# If nothing matched, and it wasn't because we ran out of chunk,
# then stop
if self.chunkOffset != self.chunkSize:
break
else:
end = m.end()
# If not the whole chunk matched, return everything
# up to the part that didn't match
if end != self.chunkSize:
rv.append(self.chunk[self.chunkOffset:end])
self.chunkOffset = end
break
# If the whole remainder of the chunk matched,
# use it all and read the next chunk
rv.append(self.chunk[self.chunkOffset:])
if not self.readChunk():
# Reached EOF
break
r = "".join(rv)
return r
def unget(self, char):
# Only one character is allowed to be ungotten at once - it must
# be consumed again before any further call to unget
if char is not None:
if self.chunkOffset == 0:
# unget is called quite rarely, so it's a good idea to do
# more work here if it saves a bit of work in the frequently
# called char and charsUntil.
# So, just prepend the ungotten character onto the current
# chunk:
self.chunk = char + self.chunk
self.chunkSize += 1
else:
self.chunkOffset -= 1
assert self.chunk[self.chunkOffset] == char
class HTMLBinaryInputStream(HTMLUnicodeInputStream):
"""Provides a unicode stream of characters to the HTMLTokenizer.
This class takes care of character encoding and removing or replacing
incorrect byte-sequences and also provides column and line tracking.
"""
def __init__(self, source, override_encoding=None, transport_encoding=None,
same_origin_parent_encoding=None, likely_encoding=None,
default_encoding="windows-1252", useChardet=True):
"""Initialises the HTMLInputStream.
HTMLInputStream(source, [encoding]) -> Normalized stream from source
for use by html5lib.
source can be either a file-object, local filename or a string.
The optional encoding parameter must be a string that indicates
the encoding. If specified, that encoding will be used,
regardless of any BOM or later declaration (such as in a meta
element)
"""
# Raw Stream - for unicode objects this will encode to utf-8 and set
# self.charEncoding as appropriate
self.rawStream = self.openStream(source)
HTMLUnicodeInputStream.__init__(self, self.rawStream)
# Encoding Information
# Number of bytes to use when looking for a meta element with
# encoding information
self.numBytesMeta = 1024
# Number of bytes to use when using detecting encoding using chardet
self.numBytesChardet = 100
# Things from args
self.override_encoding = override_encoding
self.transport_encoding = transport_encoding
self.same_origin_parent_encoding = same_origin_parent_encoding
self.likely_encoding = likely_encoding
self.default_encoding = default_encoding
# Determine encoding
self.charEncoding = self.determineEncoding(useChardet)
assert self.charEncoding[0] is not None
# Call superclass
self.reset()
def reset(self):
self.dataStream = self.charEncoding[0].codec_info.streamreader(self.rawStream, 'replace')
HTMLUnicodeInputStream.reset(self)
def openStream(self, source):
"""Produces a file object from source.
source can be either a file object, local filename or a string.
"""
# Already a file object
if hasattr(source, 'read'):
stream = source
else:
stream = BytesIO(source)
try:
stream.seek(stream.tell())
except: # pylint:disable=bare-except
stream = BufferedStream(stream)
return stream
def determineEncoding(self, chardet=True):
# BOMs take precedence over everything
# This will also read past the BOM if present
charEncoding = self.detectBOM(), "certain"
if charEncoding[0] is not None:
return charEncoding
# If we've been overriden, we've been overriden
charEncoding = lookupEncoding(self.override_encoding), "certain"
if charEncoding[0] is not None:
return charEncoding
# Now check the transport layer
charEncoding = lookupEncoding(self.transport_encoding), "certain"
if charEncoding[0] is not None:
return charEncoding
# Look for meta elements with encoding information
charEncoding = self.detectEncodingMeta(), "tentative"
if charEncoding[0] is not None:
return charEncoding
# Parent document encoding
charEncoding = lookupEncoding(self.same_origin_parent_encoding), "tentative"
if charEncoding[0] is not None and not charEncoding[0].name.startswith("utf-16"):
return charEncoding
# "likely" encoding
charEncoding = lookupEncoding(self.likely_encoding), "tentative"
if charEncoding[0] is not None:
return charEncoding
# Guess with chardet, if available
if chardet:
try:
from chardet.universaldetector import UniversalDetector
except ImportError:
pass
else:
buffers = []
detector = UniversalDetector()
while not detector.done:
buffer = self.rawStream.read(self.numBytesChardet)
assert isinstance(buffer, bytes)
if not buffer:
break
buffers.append(buffer)
detector.feed(buffer)
detector.close()
encoding = lookupEncoding(detector.result['encoding'])
self.rawStream.seek(0)
if encoding is not None:
return encoding, "tentative"
# Try the default encoding
charEncoding = lookupEncoding(self.default_encoding), "tentative"
if charEncoding[0] is not None:
return charEncoding
# Fallback to html5lib's default if even that hasn't worked
return lookupEncoding("windows-1252"), "tentative"
def changeEncoding(self, newEncoding):
assert self.charEncoding[1] != "certain"
newEncoding = lookupEncoding(newEncoding)
if newEncoding is None:
return
if newEncoding.name in ("utf-16be", "utf-16le"):
newEncoding = lookupEncoding("utf-8")
assert newEncoding is not None
elif newEncoding == self.charEncoding[0]:
self.charEncoding = (self.charEncoding[0], "certain")
else:
self.rawStream.seek(0)
self.charEncoding = (newEncoding, "certain")
self.reset()
raise ReparseException("Encoding changed from %s to %s" % (self.charEncoding[0], newEncoding))
def detectBOM(self):
"""Attempts to detect at BOM at the start of the stream. If
an encoding can be determined from the BOM return the name of the
encoding otherwise return None"""
bomDict = {
codecs.BOM_UTF8: 'utf-8',
codecs.BOM_UTF16_LE: 'utf-16le', codecs.BOM_UTF16_BE: 'utf-16be',
codecs.BOM_UTF32_LE: 'utf-32le', codecs.BOM_UTF32_BE: 'utf-32be'
}
# Go to beginning of file and read in 4 bytes
string = self.rawStream.read(4)
assert isinstance(string, bytes)
# Try detecting the BOM using bytes from the string
encoding = bomDict.get(string[:3]) # UTF-8
seek = 3
if not encoding:
# Need to detect UTF-32 before UTF-16
encoding = bomDict.get(string) # UTF-32
seek = 4
if not encoding:
encoding = bomDict.get(string[:2]) # UTF-16
seek = 2
# Set the read position past the BOM if one was found, otherwise
# set it to the start of the stream
if encoding:
self.rawStream.seek(seek)
return lookupEncoding(encoding)
else:
self.rawStream.seek(0)
return None
def detectEncodingMeta(self):
"""Report the encoding declared by the meta element
"""
buffer = self.rawStream.read(self.numBytesMeta)
assert isinstance(buffer, bytes)
parser = EncodingParser(buffer)
self.rawStream.seek(0)
encoding = parser.getEncoding()
if encoding is not None and encoding.name in ("utf-16be", "utf-16le"):
encoding = lookupEncoding("utf-8")
return encoding
class EncodingBytes(bytes):
"""String-like object with an associated position and various extra methods
If the position is ever greater than the string length then an exception is
raised"""
def __new__(self, value):
assert isinstance(value, bytes)
return bytes.__new__(self, value.lower())
def __init__(self, value):
# pylint:disable=unused-argument
self._position = -1
def __iter__(self):
return self
def __next__(self):
p = self._position = self._position + 1
if p >= len(self):
raise StopIteration
elif p < 0:
raise TypeError
return self[p:p + 1]
def next(self):
# Py2 compat
return self.__next__()
def previous(self):
p = self._position
if p >= len(self):
raise StopIteration
elif p < 0:
raise TypeError
self._position = p = p - 1
return self[p:p + 1]
def setPosition(self, position):
if self._position >= len(self):
raise StopIteration
self._position = position
def getPosition(self):
if self._position >= len(self):
raise StopIteration
if self._position >= 0:
return self._position
else:
return None
position = property(getPosition, setPosition)
def getCurrentByte(self):
return self[self.position:self.position + 1]
currentByte = property(getCurrentByte)
def skip(self, chars=spaceCharactersBytes):
"""Skip past a list of characters"""
p = self.position # use property for the error-checking
while p < len(self):
c = self[p:p + 1]
if c not in chars:
self._position = p
return c
p += 1
self._position = p
return None
def skipUntil(self, chars):
p = self.position
while p < len(self):
c = self[p:p + 1]
if c in chars:
self._position = p
return c
p += 1
self._position = p
return None
def matchBytes(self, bytes):
"""Look for a sequence of bytes at the start of a string. If the bytes
are found return True and advance the position to the byte after the
match. Otherwise return False and leave the position alone"""
p = self.position
data = self[p:p + len(bytes)]
rv = data.startswith(bytes)
if rv:
self.position += len(bytes)
return rv
def jumpTo(self, bytes):
"""Look for the next sequence of bytes matching a given sequence. If
a match is found advance the position to the last byte of the match"""
newPosition = self[self.position:].find(bytes)
if newPosition > -1:
# XXX: This is ugly, but I can't see a nicer way to fix this.
if self._position == -1:
self._position = 0
self._position += (newPosition + len(bytes) - 1)
return True
else:
raise StopIteration
class EncodingParser(object):
"""Mini parser for detecting character encoding from meta elements"""
def __init__(self, data):
"""string - the data to work on for encoding detection"""
self.data = EncodingBytes(data)
self.encoding = None
def getEncoding(self):
methodDispatch = (
(b"")
def handleMeta(self):
if self.data.currentByte not in spaceCharactersBytes:
# if we have , , and
#errors
(1,46): unexpected-start-tag
#document
|
|
|
|
| -->
#errors
(1,64): unexpected-end-tag
#document
|
|
|
|
|
#errors
#document
|
|
|
|
| "
#data
#errors
#document
|
|
|
|
|
#errors
#document
|
|
|
|
| -->
#errors
(1,10): expected-doctype-but-got-start-tag
(1,49): unexpected-end-tag
#document
|
|
|
|
#errors
(1,10): expected-doctype-but-got-start-tag
#document
|
|
|
| "
#data
-->
#errors
(1,8): expected-doctype-but-got-start-tag
(1,41): unexpected-end-tag
#document
|
|
|
|
#errors
(1,8): expected-doctype-but-got-start-tag
#document
|
|
|
|
#errors
(1,9): expected-doctype-but-got-end-tag
(1,9): unexpected-end-tag-before-html
(1,13): unexpected-end-tag-before-html
(1,18): unexpected-end-tag-before-html
(1,22): unexpected-end-tag-before-html
(1,26): unexpected-end-tag-before-html
(1,35): unexpected-end-tag-before-html
(1,39): unexpected-end-tag-before-html
(1,47): unexpected-end-tag-before-html
(1,52): unexpected-end-tag-before-html
(1,58): unexpected-end-tag-before-html
(1,64): unexpected-end-tag-before-html
(1,72): unexpected-end-tag-before-html
(1,79): unexpected-end-tag-before-html
(1,88): unexpected-end-tag-before-html
(1,93): unexpected-end-tag-before-html
(1,98): unexpected-end-tag-before-html
(1,103): unexpected-end-tag-before-html
(1,108): unexpected-end-tag-before-html
(1,113): unexpected-end-tag-before-html
(1,118): unexpected-end-tag-before-html
(1,130): unexpected-end-tag-after-body
(1,130): unexpected-end-tag-treated-as
(1,134): unexpected-end-tag
(1,140): unexpected-end-tag
(1,148): unexpected-end-tag
(1,155): unexpected-end-tag
(1,163): unexpected-end-tag
(1,172): unexpected-end-tag
(1,180): unexpected-end-tag
(1,185): unexpected-end-tag
(1,190): unexpected-end-tag
(1,195): unexpected-end-tag
(1,203): unexpected-end-tag
(1,210): unexpected-end-tag
(1,217): unexpected-end-tag
(1,225): unexpected-end-tag
(1,230): unexpected-end-tag
(1,238): unexpected-end-tag
(1,244): unexpected-end-tag
(1,251): unexpected-end-tag
(1,258): unexpected-end-tag
(1,269): unexpected-end-tag
(1,279): unexpected-end-tag
(1,287): unexpected-end-tag
(1,296): unexpected-end-tag
(1,300): unexpected-end-tag
(1,305): unexpected-end-tag
(1,310): unexpected-end-tag
(1,320): unexpected-end-tag
(1,331): unexpected-end-tag
(1,339): unexpected-end-tag
(1,347): unexpected-end-tag
(1,355): unexpected-end-tag
(1,365): end-tag-too-early
(1,378): end-tag-too-early
(1,387): end-tag-too-early
(1,393): end-tag-too-early
(1,399): end-tag-too-early
(1,404): end-tag-too-early
(1,415): end-tag-too-early
(1,425): end-tag-too-early
(1,432): end-tag-too-early
(1,437): end-tag-too-early
(1,442): end-tag-too-early
(1,447): unexpected-end-tag
(1,454): unexpected-end-tag
(1,460): unexpected-end-tag
(1,467): unexpected-end-tag
(1,476): end-tag-too-early
(1,486): end-tag-too-early
(1,495): end-tag-too-early
(1,513): expected-eof-but-got-end-tag
(1,513): unexpected-end-tag
(1,520): unexpected-end-tag
(1,529): unexpected-end-tag
(1,537): unexpected-end-tag
(1,547): unexpected-end-tag
(1,557): unexpected-end-tag
(1,568): unexpected-end-tag
(1,579): unexpected-end-tag
(1,590): unexpected-end-tag
(1,599): unexpected-end-tag
(1,611): unexpected-end-tag
(1,622): unexpected-end-tag
#document
|
|
|
|
#data
#errors
(1,7): expected-doctype-but-got-start-tag
(1,20): unexpected-end-tag-implies-table-voodoo
(1,20): unexpected-end-tag
(1,24): unexpected-end-tag-implies-table-voodoo
(1,24): unexpected-end-tag
(1,29): unexpected-end-tag-implies-table-voodoo
(1,29): unexpected-end-tag
(1,33): unexpected-end-tag-implies-table-voodoo
(1,33): unexpected-end-tag
(1,37): unexpected-end-tag-implies-table-voodoo
(1,37): unexpected-end-tag
(1,46): unexpected-end-tag-implies-table-voodoo
(1,46): unexpected-end-tag
(1,50): unexpected-end-tag-implies-table-voodoo
(1,50): unexpected-end-tag
(1,58): unexpected-end-tag-implies-table-voodoo
(1,58): unexpected-end-tag
(1,63): unexpected-end-tag-implies-table-voodoo
(1,63): unexpected-end-tag
(1,69): unexpected-end-tag-implies-table-voodoo
(1,69): end-tag-too-early
(1,75): unexpected-end-tag-implies-table-voodoo
(1,75): unexpected-end-tag
(1,83): unexpected-end-tag-implies-table-voodoo
(1,83): unexpected-end-tag
(1,90): unexpected-end-tag-implies-table-voodoo
(1,90): unexpected-end-tag
(1,99): unexpected-end-tag-implies-table-voodoo
(1,99): unexpected-end-tag
(1,104): unexpected-end-tag-implies-table-voodoo
(1,104): end-tag-too-early
(1,109): unexpected-end-tag-implies-table-voodoo
(1,109): end-tag-too-early
(1,114): unexpected-end-tag-implies-table-voodoo
(1,114): end-tag-too-early
(1,119): unexpected-end-tag-implies-table-voodoo
(1,119): end-tag-too-early
(1,124): unexpected-end-tag-implies-table-voodoo
(1,124): end-tag-too-early
(1,129): unexpected-end-tag-implies-table-voodoo
(1,129): end-tag-too-early
(1,136): unexpected-end-tag-in-table-row
(1,141): unexpected-end-tag-implies-table-voodoo
(1,141): unexpected-end-tag-treated-as
(1,145): unexpected-end-tag-implies-table-voodoo
(1,145): unexpected-end-tag
(1,151): unexpected-end-tag-implies-table-voodoo
(1,151): unexpected-end-tag
(1,159): unexpected-end-tag-implies-table-voodoo
(1,159): unexpected-end-tag
(1,166): unexpected-end-tag-implies-table-voodoo
(1,166): unexpected-end-tag
(1,174): unexpected-end-tag-implies-table-voodoo
(1,174): unexpected-end-tag
(1,183): unexpected-end-tag-implies-table-voodoo
(1,183): unexpected-end-tag
(1,196): unexpected-end-tag
(1,201): unexpected-end-tag
(1,206): unexpected-end-tag
(1,214): unexpected-end-tag
(1,221): unexpected-end-tag
(1,228): unexpected-end-tag
(1,236): unexpected-end-tag
(1,241): unexpected-end-tag
(1,249): unexpected-end-tag
(1,255): unexpected-end-tag
(1,262): unexpected-end-tag
(1,269): unexpected-end-tag
(1,280): unexpected-end-tag
(1,290): unexpected-end-tag
(1,298): unexpected-end-tag
(1,307): unexpected-end-tag
(1,311): unexpected-end-tag
(1,316): unexpected-end-tag
(1,321): unexpected-end-tag
(1,331): unexpected-end-tag
(1,342): unexpected-end-tag
(1,350): unexpected-end-tag
(1,358): unexpected-end-tag
(1,366): unexpected-end-tag
(1,376): end-tag-too-early
(1,389): end-tag-too-early
(1,398): end-tag-too-early
(1,404): end-tag-too-early
(1,410): end-tag-too-early
(1,415): end-tag-too-early
(1,426): end-tag-too-early
(1,436): end-tag-too-early
(1,443): end-tag-too-early
(1,448): end-tag-too-early
(1,453): end-tag-too-early
(1,458): unexpected-end-tag
(1,465): unexpected-end-tag
(1,471): unexpected-end-tag
(1,478): unexpected-end-tag
(1,487): end-tag-too-early
(1,497): end-tag-too-early
(1,506): end-tag-too-early
(1,524): expected-eof-but-got-end-tag
(1,524): unexpected-end-tag
(1,531): unexpected-end-tag
(1,540): unexpected-end-tag
(1,548): unexpected-end-tag
(1,558): unexpected-end-tag
(1,568): unexpected-end-tag
(1,579): unexpected-end-tag
(1,590): unexpected-end-tag
(1,601): unexpected-end-tag
(1,610): unexpected-end-tag
(1,622): unexpected-end-tag
(1,633): unexpected-end-tag
#document
|
|
|
|
|
|
|
#data
#errors
(1,10): expected-doctype-but-got-start-tag
(1,10): eof-in-frameset
#document
|
|
|
html5lib-0.999999999/html5lib/tests/testdata/tree-construction/entities02.dat 0000644 0001750 0000144 00000010521 12632711126 030011 0 ustar gsnedders users 0000000 0000000 #data
#errors
(1,20): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ>YY"
#data
#errors
(1,15): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ&"
#data
#errors
(1,15): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ&"
#data
#errors
(1,13): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ&"
#data
#errors
(1,15): named-entity-without-semicolon
(1,20): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ>=YY"
#data
#errors
(1,20): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ>0YY"
#data
#errors
(1,20): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ>9YY"
#data
#errors
(1,20): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ>aYY"
#data
#errors
(1,20): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ>ZYY"
#data
#errors
(1,15): named-entity-without-semicolon
(1,20): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ> YY"
#data
#errors
(1,15): named-entity-without-semicolon
(1,17): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ>"
#data
#errors
(1,15): named-entity-without-semicolon
(1,17): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ>"
#data
#errors
(1,14): named-entity-without-semicolon
(1,15): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ>"
#data
#errors
(1,18): named-entity-without-semicolon
(1,26): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ£_id=23"
#data
#errors
(1,25): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ&prod_id=23"
#data
#errors
(1,27): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ£_id=23"
#data
#errors
(1,26): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ∏_id=23"
#data
#errors
(1,18): named-entity-without-semicolon
(1,23): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ£=23"
#data
#errors
(1,22): expected-doctype-but-got-start-tag
#document
|
|
|
|
| bar="ZZ&prod=23"
#data
ZZ£_id=23
#errors
(1,5): expected-doctype-but-got-start-tag
(1,13): named-entity-without-semicolon
#document
|
|
|
|
| "ZZ£_id=23"
#data
ZZ&prod_id=23
#errors
(1,5): expected-doctype-but-got-start-tag
#document
|
|
|
|
| "ZZ&prod_id=23"
#data
ZZ£_id=23
#errors
(1,5): expected-doctype-but-got-start-tag
#document
|
|
|
|
| "ZZ£_id=23"
#data
ZZ∏_id=23
#errors
(1,5): expected-doctype-but-got-start-tag
#document
|
|
|
|
| "ZZ∏_id=23"
#data
ZZ£=23
#errors
(1,5): expected-doctype-but-got-start-tag
(1,13): named-entity-without-semicolon
#document
|
|
|
|
| "ZZ£=23"
#data
ZZ&prod=23
#errors
(1,5): expected-doctype-but-got-start-tag
#document
|
|
|
|
| "ZZ&prod=23"
#data
ZZÆ=
#errors
#document
|
|
|
|
| "ZZÆ="
html5lib-0.999999999/html5lib/tests/testdata/tree-construction/tests11.dat 0000644 0001750 0000144 00000042417 12740556320 027343 0 ustar gsnedders users 0000000 0000000 #data
#errors
#document
|
|
|
|
|
| attributeName=""
| attributeType=""
| baseFrequency=""
| baseProfile=""
| calcMode=""
| clipPathUnits=""
| diffuseConstant=""
| edgeMode=""
| filterUnits=""
| glyphRef=""
| gradientTransform=""
| gradientUnits=""
| kernelMatrix=""
| kernelUnitLength=""
| keyPoints=""
| keySplines=""
| keyTimes=""
| lengthAdjust=""
| limitingConeAngle=""
| markerHeight=""
| markerUnits=""
| markerWidth=""
| maskContentUnits=""
| maskUnits=""
| numOctaves=""
| pathLength=""
| patternContentUnits=""
| patternTransform=""
| patternUnits=""
| pointsAtX=""
| pointsAtY=""
| pointsAtZ=""
| preserveAlpha=""
| preserveAspectRatio=""
| primitiveUnits=""
| refX=""
| refY=""
| repeatCount=""
| repeatDur=""
| requiredExtensions=""
| requiredFeatures=""
| specularConstant=""
| specularExponent=""
| spreadMethod=""
| startOffset=""
| stdDeviation=""
| stitchTiles=""
| surfaceScale=""
| systemLanguage=""
| tableValues=""
| targetX=""
| targetY=""
| textLength=""
| viewBox=""
| viewTarget=""
| xChannelSelector=""
| yChannelSelector=""
| zoomAndPan=""
#data
| attributeName=""
| attributeType=""
| baseFrequency=""
| baseProfile=""
| calcMode=""
| clipPathUnits=""
| diffuseConstant=""
| edgeMode=""
| filterUnits=""
| glyphRef=""
| gradientTransform=""
| gradientUnits=""
| kernelMatrix=""
| kernelUnitLength=""
| keyPoints=""
| keySplines=""
| keyTimes=""
| lengthAdjust=""
| limitingConeAngle=""
| markerHeight=""
| markerUnits=""
| markerWidth=""
| maskContentUnits=""
| maskUnits=""
| numOctaves=""
| pathLength=""
| patternContentUnits=""
| patternTransform=""
| patternUnits=""
| pointsAtX=""
| pointsAtY=""
| pointsAtZ=""
| preserveAlpha=""
| preserveAspectRatio=""
| primitiveUnits=""
| refX=""
| refY=""
| repeatCount=""
| repeatDur=""
| requiredExtensions=""
| requiredFeatures=""
| specularConstant=""
| specularExponent=""
| spreadMethod=""
| startOffset=""
| stdDeviation=""
| stitchTiles=""
| surfaceScale=""
| systemLanguage=""
| tableValues=""
| targetX=""
| targetY=""
| textLength=""
| viewBox=""
| viewTarget=""
| xChannelSelector=""
| yChannelSelector=""
| zoomAndPan=""
#data
| attributeName=""
| attributeType=""
| baseFrequency=""
| baseProfile=""
| calcMode=""
| clipPathUnits=""
| diffuseConstant=""
| edgeMode=""
| filterUnits=""
| filterres=""
| glyphRef=""
| gradientTransform=""
| gradientUnits=""
| kernelMatrix=""
| kernelUnitLength=""
| keyPoints=""
| keySplines=""
| keyTimes=""
| lengthAdjust=""
| limitingConeAngle=""
| markerHeight=""
| markerUnits=""
| markerWidth=""
| maskContentUnits=""
| maskUnits=""
| numOctaves=""
| pathLength=""
| patternContentUnits=""
| patternTransform=""
| patternUnits=""
| pointsAtX=""
| pointsAtY=""
| pointsAtZ=""
| preserveAlpha=""
| preserveAspectRatio=""
| primitiveUnits=""
| refX=""
| refY=""
| repeatCount=""
| repeatDur=""
| requiredExtensions=""
| requiredFeatures=""
| specularConstant=""
| specularExponent=""
| spreadMethod=""
| startOffset=""
| stdDeviation=""
| stitchTiles=""
| surfaceScale=""
| systemLanguage=""
| tableValues=""
| targetX=""
| targetY=""
| textLength=""
| viewBox=""
| viewTarget=""
| xChannelSelector=""
| yChannelSelector=""
| zoomAndPan=""
#data
| attributename=""
| attributetype=""
| basefrequency=""
| baseprofile=""
| calcmode=""
| clippathunits=""
| diffuseconstant=""
| edgemode=""
| filterunits=""
| glyphref=""
| gradienttransform=""
| gradientunits=""
| kernelmatrix=""
| kernelunitlength=""
| keypoints=""
| keysplines=""
| keytimes=""
| lengthadjust=""
| limitingconeangle=""
| markerheight=""
| markerunits=""
| markerwidth=""
| maskcontentunits=""
| maskunits=""
| numoctaves=""
| pathlength=""
| patterncontentunits=""
| patterntransform=""
| patternunits=""
| pointsatx=""
| pointsaty=""
| pointsatz=""
| preservealpha=""
| preserveaspectratio=""
| primitiveunits=""
| refx=""
| refy=""
| repeatcount=""
| repeatdur=""
| requiredextensions=""
| requiredfeatures=""
| specularconstant=""
| specularexponent=""
| spreadmethod=""
| startoffset=""
| stddeviation=""
| stitchtiles=""
| surfacescale=""
| systemlanguage=""
| tablevalues=""
| targetx=""
| targety=""
| textlength=""
| viewbox=""
| viewtarget=""
| xchannelselector=""
| ychannelselector=""
| zoomandpan=""
#data
| contentscripttype=""
| contentstyletype=""
| externalresourcesrequired=""
| filterres=""
#data
| contentscripttype=""
| contentstyletype=""
| externalresourcesrequired=""
| filterres=""
#data
| contentscripttype=""
| contentstyletype=""
| externalresourcesrequired=""
| filterres=""
#data
| contentscripttype=""
| contentstyletype=""
| externalresourcesrequired=""
| filterres=""
#data
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#data
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#data
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#data
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#data
|
html5lib-0.999999999/html5lib/tests/testdata/tree-construction/tests20.dat 0000644 0001750 0000144 00000022634 12740556320 027342 0 ustar gsnedders users 0000000 0000000 #data
#errors
(1,34): unexpected-start-tag-implies-end-tag
(1,34): expected-closing-tag-but-got-eof
#document
|
|
|
|
|
|
|
#data
#errors
(1,35): expected-closing-tag-but-got-eof
#document
|
|
|
|
|
|
|
#data
#errors
(1,38): expected-closing-tag-but-got-eof
#document
|
|
|
|
|
|
|
#data
#errors
(1,32): expected-closing-tag-but-got-eof
#document
|
|
|
|
|
|
|
#data
#errors
(1,29): expected-closing-tag-but-got-eof
#document
|
|
|
|
|
|
|
#data
#errors
(1,30): expected-closing-tag-but-got-eof
#document
|
|
|
|
|
|
|
#data
#errors
(1,30): expected-closing-tag-but-got-eof
#document
|
|
|
|
|
|
|
#data
#errors
(1,30): expected-closing-tag-but-got-eof
#document
|
|
|
|
|
|
|
#data
#errors
(1,35): expected-closing-tag-but-got-eof
#document
|
|
|
|
|