...
... '''
To remove the all superfluous content from this unparsed document, use the
``clean_html`` function:
.. sourcecode:: pycon
>>> from lxml_html_clean import clean_html
>>> print clean_html(html)
The ``Cleaner`` class supports several keyword arguments to control exactly
which content is removed:
.. sourcecode:: pycon
>>> from lxml_html_clean import Cleaner
>>> cleaner = Cleaner(page_structure=False, links=False)
>>> print cleaner.clean_html(html)
a link
another link
a paragraph
secret EVIL!
of EVIL!
Password:
annoying EVIL!
spam spam SPAM!
>>> cleaner = Cleaner(style=True, links=True, add_nofollow=True,
... page_structure=False, safe_attrs_only=False)
>>> print cleaner.clean_html(html)
a link
another link
a paragraph
secret EVIL!
of EVIL!
Password:
annoying EVIL!
spam spam SPAM!
To control the removal of CSS styles, set the ``style`` and/or ``inline_style``
keyword arguments to ``True`` when creating a ``Cleaner`` instance.
If neither option is enabled, only ``@import`` rules are automatically removed
from CSS content.
You can also whitelist some otherwise dangerous content with
``Cleaner(host_whitelist=['www.youtube.com'])``, which would allow
embedded media from YouTube, while still filtering out embedded media
from other sites.
See the docstring of ``Cleaner`` for the details of what can be
cleaned.
autolink
--------
In addition to cleaning up malicious HTML, ``lxml_html_clean``
contains functions to do other things to your HTML. This includes
autolinking::
autolink(doc, ...)
autolink_html(html, ...)
This finds anything that looks like a link (e.g.,
``http://example.com``) in the *text* of an HTML document, and
turns it into an anchor. It avoids making bad links.
Links in the elements ``'''))
A link in
>>> print(autolink_html('''
... '''))
>>> print(autolink_html('''
... A link in http://foo.com or
... http://bar.com
'''))
A link in http://foo.com or
http://bar.com
There's also a word wrapping function, that should probably be run
after autolink::
>>> from lxml_html_clean import word_break_html
>>> def pascii(s):
... print(s.encode('ascii', 'xmlcharrefreplace').decode('ascii'))
>>> pascii(word_break_html( u'''
... Hey you
... 12345678901234567890123456789012345678901234567890
'''))
Hey you
12345678901234567890123456789012345678901234567890
Not everything is broken:
>>> pascii(word_break_html('''
... Hey you
... 12345678901234567890123456789012345678901234567890
'''))
Hey you
12345678901234567890123456789012345678901234567890
>>> pascii(word_break_html('''
... text'''))
text
lxml_html_clean-0.4.4/tests/test_clean.py 0000664 0000000 0000000 00000060674 15150262265 0020537 0 ustar 00root root 0000000 0000000 import base64
import gzip
import io
import unittest
import warnings
import lxml.html
from lxml_html_clean import AmbiguousURLWarning, Cleaner, clean_html, LXMLHTMLCleanWarning
from .utils import peak_memory_usage
class CleanerTest(unittest.TestCase):
def test_allow_tags(self):
html = """
some text
![]()
"""
html_root = lxml.html.document_fromstring(html)
cleaner = Cleaner(
remove_unknown_tags = False,
allow_tags = ['table', 'tr', 'td'])
result = cleaner.clean_html(html_root)
self.assertEqual(12-5+1, len(list(result.iter())))
def test_allow_and_remove(self):
with self.assertRaises(ValueError):
Cleaner(allow_tags=['a'], remove_unknown_tags=True)
def test_remove_unknown_tags(self):
html = """lettuce, tomato, veggie patty
"""
clean_html = """lettuce, tomato, veggie patty
"""
cleaner = Cleaner(remove_unknown_tags=True)
result = cleaner.clean_html(html)
self.assertEqual(
result,
clean_html,
msg="Unknown tags not removed. Got: %s" % result,
)
def test_safe_attrs_included(self):
html = """Cyan
"""
safe_attrs=set(lxml.html.defs.safe_attrs)
safe_attrs.add('style')
cleaner = Cleaner(
safe_attrs_only=True,
safe_attrs=safe_attrs)
result = cleaner.clean_html(html)
self.assertEqual(html, result)
def test_safe_attrs_excluded(self):
html = """Cyan
"""
expected = """Cyan
"""
safe_attrs=set()
cleaner = Cleaner(
safe_attrs_only=True,
safe_attrs=safe_attrs)
result = cleaner.clean_html(html)
self.assertEqual(expected, result)
def test_clean_invalid_root_tag(self):
# only testing that cleaning with invalid root tags works at all
s = lxml.html.fromstring('parent child')
self.assertEqual('parent child', clean_html(s).text_content())
s = lxml.html.fromstring('child')
self.assertEqual('child', clean_html(s).text_content())
def test_clean_with_comments(self):
html = """Cyan
"""
s = lxml.html.fragment_fromstring(html)
self.assertEqual(
b'Cyan
',
lxml.html.tostring(clean_html(s)))
self.assertEqual(
'Cyan
',
clean_html(html))
cleaner = Cleaner(comments=False)
result = cleaner.clean_html(s)
self.assertEqual(
b'Cyan
',
lxml.html.tostring(result))
self.assertEqual(
'Cyan
',
cleaner.clean_html(html))
def test_sneaky_noscript_in_style(self):
# This gets parsed as through into the output.
html = '',
lxml.html.tostring(clean_html(s)))
def test_sneaky_js_in_math_style(self):
# This gets parsed as