HTML-TableExtract-2.13/0000755000175000017500000000000012527403163013272 5ustar sisksiskHTML-TableExtract-2.13/META.yml0000644000175000017500000000073312527403163014546 0ustar sisksisk--- abstract: unknown author: - unknown build_requires: ExtUtils::MakeMaker: 0 configure_requires: ExtUtils::MakeMaker: 0 dynamic_config: 1 generated_by: 'ExtUtils::MakeMaker version 6.66, CPAN::Meta::Converter version 2.120921' license: unknown meta-spec: url: http://module-build.sourceforge.net/META-spec-v1.4.html version: 1.4 name: HTML-TableExtract no_index: directory: - t - inc requires: HTML::ElementTable: 1.16 HTML::Parser: 0 version: 2.13 HTML-TableExtract-2.13/Changes0000644000175000017500000001437012527403061014567 0ustar sisksiskRevision history for HTML::TableExtract 2.14 Thu May 21 12:20:46 EDT 2015 - bundled examples html page 2.12 Fri Jan 9 11:29:08 EST 2015 - tightened up logic pertaining to tree mode and keep_html - documentation fixes 2.11 Tue Aug 23 16:01:04 EDT 2011 - added parsing context, override for eof() and parse() for memory clear on new docs or post-eof() - fixed some long standing test warnings 2.10 Sat Jul 15 20:50:41 EDT 2006 - minor bug fixed in HTML repair routines (thanks to Dave Gray) 2.09 Thu Jun 8 15:46:17 EDT 2006 - Tweaked rasterizer to handle some situations where the HTML is broken but tables can still be inferred. - Fixed TREE() definition for situations where import() is not invoked. (thanks to DDICK on cpan.org) 2.08 Wed May 3 17:17:33 EDT 2006 - Implemented new rasterizer for grid mapping. Thanks to Roland Schar for a tortuous example of span issues. - This also fixes a bug the old skew method had when it encountered ridiculously large spans (out of memory). Thanks to Andreas Gustafsson. - Regular extraction and TREE mode are using the same rasterizer now. - Fixed HTML stripping for a header matching bug on single word text in keep_html mode (thanks to Michael S. Muegel for pointing the bug out) 2.07 Sun Feb 19 13:40:44 EST 2006 - Fixed subtable slicing bug - Fixed hrow() attachment bug - Added tests 2.06 Tue Oct 18 13:13:52 EDT 2005 - Tightened up element interactions in TREE() mode when examining rows, columns, cells, etc. Was running into trouble with dereferencing scalars vs objects. - Documented space() H::TE::T method, added tests - Added POD tests - Documentation updates and fixes 2.05 Tue Oct 4 16:00:02 EDT 2005 - Fixed a TREE() definition bug and class method assignments - Fixed a 'row above header' bug, added tests 2.04 Wed Aug 3 14:42:23 EDT 2005 - Fixed some conditional optional dependency tests in order to avoid falure assertions on some test boxes. 2.03 Wed Jul 20 12:45:56 EDT 2005 - Fixed greedy attribute bug (non qualifying tables were being selected under certain circumstances) - Moved more completely to File::Spec operations in testload.pm in order to make windows boxes happy. 2.02 Thu Jun 23 12:42:44 EDT 2005 - squelched TREE() creation warnings for subclasses - fixed a rows() bug involving keep_headers 2.01 Tue Jun 21 22:05:53 EDT 2005 - fixed some test changes 2.00 Fri Jun 17 17:28:10 EDT 2005 - Can now return parsed tables as HTML::TableElement objects within an HTML::Element tree structure (via HTML::TreeBuilder) for such purposes as in-line editing of table content within documents. Invoked via 'use HTML::TableExtract qw(tree);'. - Added columns(), row(), column(), and cell() methods. - Added some handy reporting methods: tables_report() and tables_dump(). These are almost always handy while first analyzing a new HTML document for table content. - Debugging and error output can now be assigned to arbitrary file handles. ! Old 'table_state' methods are now merely 'table' methods, though the old table_state style is still supported. ! Chains have been dropped. Though interesting (think xpath), they needlessly complicated matters as they were nearly universally unused. 1.09 Fri Feb 25 17:49:00 EST 2005 - Tables can now be selected by table tag attributes - lineage() method now returns row and column information, as well as depth and count, for each ancestor (potential backwards incompatability, entries are now 4 element arrays now rather than 2) - header matching and column retention enhancements - header retention - old-style procedures deprecated in prepration for them to become methods - various bug fixes 1.08 Thu Apr 4 11:26:27 CST 2002 - Added some more crufty HTML tolerance -- not PC (puristicly correct) but HTML correctness is probably of no interest to those merely trying to extract information *out* of HTML. - Fixed a mapback problem with the legacy methods 1.07 Wed Aug 22 06:14:24 CDT 2001 - Added keep_html option for HTML retention - bug fix for depth/count targets 1.06 Thu Nov 2 15:29:49 CST 2000 - Added
translation to newlines (enabled by default) - cleaned up some warnings 1.05 Sun Aug 6 06:38:14 CDT 2000 - minor bug fix involving empty cells 1.04 Sat Jul 15 02:18:04 CDT 2000 - fixed gridmap bug involving skew calcs on unwanted columns - added example page reference in README 1.03 Tue Jul 7 03:43:30 CDT 2000 - gridmap option, columns are really columns regardless of cell span skew - Added chains for relative targeting * Terminus-matching by default * Elasticity option * Waypoint retention option * Lineage tracking (match record along chain) - Significant tests added to 'make test' - Documentation rewrite 0.05 Tue Mar 21 08:11:54 CST 2000 - Fixed -w init warnings for dangling columns in header mode - added 'decode' option to turn off text decoding when desired - internally stores real slices right now rather than sparse tables that later get massaged. 0.03 Thu Mar 9 13:10:03 CST 2000 - Fixed bug regarding incomplete defaults - Tables, rows, and cells that are either empty or contain no text are now properly noted - Header patterns now match across stripped tags - In some cases, mangled HTML tables are properly scanned by inferring missing tags. - Depth/Count votes are now properly honored. - Cleaned up some -w noise. 0.02 Thu Feb 10 13:43:04 CST 2000 - Fixed some problems tracking counts at revisited depths. - Minor doc fix, added mailing list 0.01 Wed Feb 2 18:24:07 CST 2000 - Initial version. HTML-TableExtract-2.13/LICENSE0000644000175000017500000004343612453522764014320 0ustar sisksiskThis is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. Terms of the Perl programming language system itself a) the GNU General Public License as published by the Free Software Foundation; either version 1, or (at your option) any later version, or b) the "Artistic License" --- The GNU General Public License, Version 1, February 1989 --- This is free software, licensed under: The GNU General Public License, Version 1, February 1989 GNU GENERAL PUBLIC LICENSE Version 1, February 1989 Copyright (C) 1989 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The license agreements of most software companies try to keep users at the mercy of those companies. By contrast, our General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. The General Public License applies to the Free Software Foundation's software and to any other program whose authors commit to using it. You can use it for your programs, too. When we speak of free software, we are referring to freedom, not price. Specifically, the General Public License is designed to make sure that you have the freedom to give away or sell copies of free software, that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of a such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must tell them their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License Agreement applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any work containing the Program or a portion of it, either verbatim or with modifications. Each licensee is addressed as "you". 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this General Public License and to the absence of any warranty; and give any other recipients of the Program a copy of this General Public License along with the Program. You may charge a fee for the physical act of transferring a copy. 2. You may modify your copy or copies of the Program or any portion of it, and copy and distribute such modifications under the terms of Paragraph 1 above, provided that you also do the following: a) cause the modified files to carry prominent notices stating that you changed the files and the date of any change; and b) cause the whole of any work that you distribute or publish, that in whole or in part contains the Program or any part thereof, either with or without modifications, to be licensed at no charge to all third parties under the terms of this General Public License (except that you may choose to grant warranty protection to some or all third parties, at your option). c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the simplest and most usual way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this General Public License. d) You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. Mere aggregation of another independent work with the Program (or its derivative) on a volume of a storage or distribution medium does not bring the other work under the scope of these terms. 3. You may copy and distribute the Program (or a portion or derivative of it, under Paragraph 2) in object code or executable form under the terms of Paragraphs 1 and 2 above provided that you also do one of the following: a) accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Paragraphs 1 and 2 above; or, b) accompany it with a written offer, valid for at least three years, to give any third party free (except for a nominal charge for the cost of distribution) a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Paragraphs 1 and 2 above; or, c) accompany it with the information you received as to where the corresponding source code may be obtained. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form alone.) Source code for a work means the preferred form of the work for making modifications to it. For an executable file, complete source code means all the source code for all modules it contains; but, as a special exception, it need not include source code for modules which are standard libraries that accompany the operating system on which the executable file runs, or for standard header files or definitions files that accompany that operating system. 4. You may not copy, modify, sublicense, distribute or transfer the Program except as expressly provided under this General Public License. Any attempt otherwise to copy, modify, sublicense, distribute or transfer the Program is void, and will automatically terminate your rights to use the Program under this License. However, parties who have received copies, or rights to use copies, from you under this General Public License will not have their licenses terminated so long as such parties remain in full compliance. 5. By copying, distributing or modifying the Program (or any work based on the Program) you indicate your acceptance of this license to do so, and all its terms and conditions. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. 7. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of the license which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the license, you may choose any version ever published by the Free Software Foundation. 8. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 9. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 10. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS Appendix: How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to humanity, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) 19yy This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 1, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston MA 02110-1301 USA Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) 19xx name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (a program to direct compilers to make passes at assemblers) written by James Hacker. , 1 April 1989 Ty Coon, President of Vice That's all there is to it! --- The Artistic License 1.0 --- This is free software, licensed under: The Artistic License 1.0 The Artistic License Preamble The intent of this document is to state the conditions under which a Package may be copied, such that the Copyright Holder maintains some semblance of artistic control over the development of the package, while giving the users of the package the right to use and distribute the Package in a more-or-less customary fashion, plus the right to make reasonable modifications. Definitions: - "Package" refers to the collection of files distributed by the Copyright Holder, and derivatives of that collection of files created through textual modification. - "Standard Version" refers to such a Package if it has not been modified, or has been modified in accordance with the wishes of the Copyright Holder. - "Copyright Holder" is whoever is named in the copyright or copyrights for the package. - "You" is you, if you're thinking about copying or distributing this Package. - "Reasonable copying fee" is whatever you can justify on the basis of media cost, duplication charges, time of people involved, and so on. (You will not be required to justify it to the Copyright Holder, but only to the computing community at large as a market that must bear the fee.) - "Freely Available" means that no fee is charged for the item itself, though there may be fees involved in handling the item. It also means that recipients of the item may redistribute it under the same conditions they received it. 1. You may make and give away verbatim copies of the source form of the Standard Version of this Package without restriction, provided that you duplicate all of the original copyright notices and associated disclaimers. 2. You may apply bug fixes, portability fixes and other modifications derived from the Public Domain or from the Copyright Holder. A Package modified in such a way shall still be considered the Standard Version. 3. You may otherwise modify your copy of this Package in any way, provided that you insert a prominent notice in each changed file stating how and when you changed that file, and provided that you do at least ONE of the following: a) place your modifications in the Public Domain or otherwise make them Freely Available, such as by posting said modifications to Usenet or an equivalent medium, or placing the modifications on a major archive site such as ftp.uu.net, or by allowing the Copyright Holder to include your modifications in the Standard Version of the Package. b) use the modified Package only within your corporation or organization. c) rename any non-standard executables so the names do not conflict with standard executables, which must also be provided, and provide a separate manual page for each non-standard executable that clearly documents how it differs from the Standard Version. d) make other distribution arrangements with the Copyright Holder. 4. You may distribute the programs of this Package in object code or executable form, provided that you do at least ONE of the following: a) distribute a Standard Version of the executables and library files, together with instructions (in the manual page or equivalent) on where to get the Standard Version. b) accompany the distribution with the machine-readable source of the Package with your modifications. c) accompany any non-standard executables with their corresponding Standard Version executables, giving the non-standard executables non-standard names, and clearly documenting the differences in manual pages (or equivalent), together with instructions on where to get the Standard Version. d) make other distribution arrangements with the Copyright Holder. 5. You may charge a reasonable copying fee for any distribution of this Package. You may charge any fee you choose for support of this Package. You may not charge a fee for this Package itself. However, you may distribute this Package in aggregate with other (possibly commercial) programs as part of a larger (possibly commercial) software distribution provided that you do not advertise this Package as a product of your own. 6. The scripts and library files supplied as input to or produced as output from the programs of this Package do not automatically fall under the copyright of this Package, but belong to whomever generated them, and may be sold commercially, and may be aggregated with this Package. 7. C or perl subroutines supplied by you and linked into this Package shall not be considered part of this Package. 8. The name of the Copyright Holder may not be used to endorse or promote products derived from this software without specific prior written permission. 9. THIS PACKAGE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTIBILITY AND FITNESS FOR A PARTICULAR PURPOSE. The End HTML-TableExtract-2.13/README0000644000175000017500000000327712527402637014170 0ustar sisksiskHTML-TableExtract ----------------- HTML::TableExtract is a module that simplifies the extraction of information contained in tables within HTML documents. Tables of note may be specified using Headers, Depth, Count, Attributes, or some combination of the three. See the module documentation for details. INSTALLATION You install HTML-TableExtract, as you would install any perl module library, by running these commands: perl Makefile.PL make make test make install DOCUMENTATION See HTML/TableExtract.pm for the code. See Changes for recent changes. POD style documentation is included in the module. This is normally converted to a manual page and installed as part of the "make install" process. You should also be able to use the 'perldoc' utility to extract and read documentation from the module directly. SUPPORT The project is tracked on GitHub: https://github.com/mojotoad/HTML-TableExtract Problems and patches can also be submitted via: https://rt.cpan.org/Ticket/Create.html?Queue=HTML-TableExtract Questions and comments may also be sent to sisk AT mojotoad.com ACKNOWLEDGEMENTS Thanks to the following people for their generous bug catching, fault analysis, and suggestions: Celeste Suliin Burris, Jeff Casey, David Finberg, Michael Fowler, Robert Goff, Klaus Gottschalk, Daniel Griscom, Jeremy Howard, Martin Joost, Jeff Lewwid, Nicholas R. Markham, Julian Mehnle, Michael S. Muegel, Patrick Naubert, Jani Ollikainen, Wilson Snyder, Volker Stuerzl, Steve Wong, and Matt Zip. COPYRIGHT Copyright (c) 1999-2015 Matthew P. Sisk. All rights reserved. All wrongs revenged. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. HTML-TableExtract-2.13/META.json0000644000175000017500000000156112527403163014716 0ustar sisksisk{ "abstract" : "unknown", "author" : [ "unknown" ], "dynamic_config" : 1, "generated_by" : "ExtUtils::MakeMaker version 6.66, CPAN::Meta::Converter version 2.120921", "license" : [ "unknown" ], "meta-spec" : { "url" : "http://search.cpan.org/perldoc?CPAN::Meta::Spec", "version" : "2" }, "name" : "HTML-TableExtract", "no_index" : { "directory" : [ "t", "inc" ] }, "prereqs" : { "build" : { "requires" : { "ExtUtils::MakeMaker" : "0" } }, "configure" : { "requires" : { "ExtUtils::MakeMaker" : "0" } }, "runtime" : { "requires" : { "HTML::ElementTable" : "1.16", "HTML::Parser" : "0" } } }, "release_status" : "stable", "version" : "2.13" } HTML-TableExtract-2.13/examples.html0000644000175000017500000003720007143723325016003 0ustar sisksisk HTML::TableExtract Examples

HTML::TableExtract Examples

Each table is labeled in the first row with coordinates in terms of depth and count, which both start at 0. Some of the tables have headers in the second row; although in this example these header cells are in fact <th> tags, header cells can be either <th> or <td>. The remaining cells in the table indicate row and column information from that cell, along with the table coordinates: depth,count:row,column. Rows and columns begin at 0 as well, so the table label and headers, if present, will affect these cell coordinates.

In the illustrations of what is extracted from these tables, content in italics is notational in nature; it was not actually extracted from the tables. In particular, whenever headers are used for extraction, the order in which the headers were provided is noted by listing the headers, but the header row is not actually extracted from the target table.

It might be helpful to open a new browser window with this table visible so that the table can be easily examined when scrolling through the examples.

Table (0,0)
0,0:1,0
Table (1,0)
EastCentralWest
1,0:2,01,0:2,11,0:2,2
1,0:3,01,0:3,2
1,0:4,01,0:4,2
1,0:5,01,0:5,11,0:5,2
0,0:1,1
Table (1,1)
LeftMiddleRight
1,1:2,01,1:2,11,1:2,2
1,1:3,01,1:3,11,1:3,2
1,1:4,01,1:4,11,1:4,2
1,1:5,01,1:5,11,1:5,2
0,0:2,0
Table (1,2)
LeftRight
1,2:2,0
Table (2,0)
PacificAtlantic
2,0:2,02,0:2,1
2,0:3,02,0:3,1
1,2:2,1
Table (2,1)
LeftyRighty
2,1:2,02,1:2,1
2,1:3,02,1:3,1
1,2:3,01,2:3,1
1,2:4,01,2:4,1
1,2:5,01,2:5,1
0,0:2,1
Table (1,3)
PacificPlainsAtlantic
1,3:2,01,3:2,11,3:2,2
1,3:3,11,3:3,2
1,3:4,01,3:4,2
1,3:5,01,3:5,2
Example 1 
$te = new HTML::TableExtract( headers => [qw(Right Left)] );
$te->parse($html_string);

Result:
Extracted from table (1,1)
Order: Right, Left
1,1:2,21,1:2,0
1,1:3,21,1:3,0
1,1:4,21,1:4,0
1,1:5,21,1:5,0
Extracted from table (2,1)
Order: Right, Left
2,1:2,12,1:2,0
2,1:3,12,1:3,0
Extracted from table (1,2)
Order: Right, Left
1,2:2,11,2:2,0
1,2:3,11,2:3,0
1,2:4,11,2:4,0
1,2:5,11,2:5,0


With headers, depth and count are irrelevant; all tables with columns matching those headers are extracted. Matches are accomplished as case-insensitive, non-anchored regular expressions. Columns are automatically rearranged in the same order as the headers were provided, so in this case we have reversed left and right. Rows above and including the rows where the headers were found are ignored; only the rows beneath the headers are extracted. Only the columns that line up with specific headers are retained.
Example 2 
$te = new HTML::TableExtract( headers => [qw(Lefty Righty)] );
$te->parse($html_string);

Result:
Extracted from table (2,1)
Order: Lefty, Righty
2,1:2,02,1:2,1
2,1:3,02,1:3,1


Using basic header extraction, tables can be reliably extracted from a document no matter how the HTML changes around them or deeply nested they are.
Example 3 
@tes = (
	new HTML::TableExtract( headers => [qw(Pacific Plains Atlantic)] ),
	new HTML::TableExtract( headers => [qw(Atlantic Pacific Plains)] ),
	new HTML::TableExtract( headers => [qw(Atlantic Plains)] ),
	new HTML::TableExtract( headers => [qw(Plains Pacific)] )
       );
grep($_->parse($html_string), @tes);

Result:
Extracted from table (1,3)
Order: Pacific, Plains, Atlantic
1,3:2,01,3:2,11,3:2,2
1,3:3,11,3:3,2
1,3:4,01,3:4,2
1,3:5,01,3:5,2
Extracted from table (1,3)
Order: Atlantic, Pacific, Plains
1,3:2,21,3:2,01,3:2,1
1,3:3,21,3:3,1
1,3:4,21,3:4,0
1,3:5,21,3:5,0
Extracted from table (1,3)
Order: Atlantic, Plains
1,3:2,21,3:2,1
1,3:3,21,3:3,1
1,3:4,2
1,3:5,2
Extracted from table (1,3)
Order: Plains, Pacific
1,3:2,11,3:2,0
1,3:3,1
1,3:4,0
1,3:5,0


The tables above represent different ways of extracting information from the same table using headers; notice how the column order is automatically adjusted to reflect the order in which the headers were provided. Gridmapping preserves the columns that you see in a browser. Tables are actually HTML tree structures, so when cell spans are involved, the "grid" is an illusion. Gridmapping superimposes a grid structure of 1x1 cells over the table, and reports columns intuitively. (note that the cell coordinates in this case represent these grid coordinates, rather than tree coordinates).
Example 4 
@tes = (
	new HTML::TableExtract( depth => 1, count => 3 ),
	new HTML::TableExtract( depth => 1, count => 3, gridmap => 0 )
       );
grep($_->parse($html_string), @tes);

Result:
Extracted from table (1,3)
Table (1,3)
PacificPlainsAtlantic
1,3:2,01,3:2,11,3:2,2
1,3:3,11,3:3,2
1,3:4,01,3:4,2
1,3:5,01,3:5,2
Extracted from table (1,3)
Table (1,3)
PacificPlainsAtlantic
1,3:2,01,3:2,11,3:2,2
1,3:3,11,3:3,2
1,3:4,01,3:4,2
1,3:5,01,3:5,2


Here we target the same table using depth and count. Taken together, depth and count uniquely specify at table in an HTML document, though it does introduce more context than using headers. Notice also that the entire table is retrieved, not just the columns beneath the headers. In the first example, gridmapping is enabled by default. In the second, it is explicity disabled in order to illustrate the tree ordering of cells.
Example 5 
$te = new HTML::TableExtract( depth => 2 );
$te->parse($html_string);

Result:
Extracted from table (2,0)
Table (2,0)
PacificAtlantic
2,0:2,02,0:2,1
2,0:3,02,0:3,1
Extracted from table (2,1)
Table (2,1)
LeftyRighty
2,1:2,02,1:2,1
2,1:3,02,1:3,1


When only a depth is specified, all tables at that depth are returned.
Example 6 
$te = new HTML::TableExtract( count => 1 );
$te->parse($html_string);

Result:
Extracted from table (1,1)
Table (1,1)
LeftMiddleRight
1,1:2,01,1:2,11,1:2,2
1,1:3,01,1:3,11,1:3,2
1,1:4,01,1:4,11,1:4,2
1,1:5,01,1:5,11,1:5,2
Extracted from table (2,1)
Table (2,1)
LeftyRighty
2,1:2,02,1:2,1
2,1:3,02,1:3,1


When only a count is specified, all tables at that count from each depth are returned. In this example, the second table within each depth is extracted (both depth and count begin with 0).
Example 7 
$te = new HTML::TableExtract( count => 1, headers => [qw(Left Middle Right)] );
$te->parse($html_string);

Result:
Extracted from table (1,1)
Order: Left, Middle, Right
1,1:2,01,1:2,11,1:2,2
1,1:3,01,1:3,11,1:3,2
1,1:4,01,1:4,11,1:4,2
1,1:5,01,1:5,11,1:5,2


When constraints are specified together, they each have a veto power on whether to extract the table. In this case, the same two tables in the prior example matched on this count, but the header constraint discarded the one without the proper headers.
HTML-TableExtract-2.13/Makefile.PL0000644000175000017500000000075012453522764015255 0ustar sisksiskuse ExtUtils::MakeMaker; my %prereq_pm = ( 'HTML::Parser' => 0, ); # The idea is to skip tests and dependencies on H::ET if it is not # installed at all. If it is presently installed, however, enforce the # version dependency. my $et_version = '1.16'; eval "use HTML::ElementTable"; unless ($@) { $prereq_pm{'HTML::ElementTable'} = $et_version; } WriteMakefile( NAME => 'HTML-TableExtract', VERSION_FROM => 'lib/HTML/TableExtract.pm', PREREQ_PM => \%prereq_pm, ); HTML-TableExtract-2.13/t/0000755000175000017500000000000012527403163013535 5ustar sisksiskHTML-TableExtract-2.13/t/20_skew.t0000755000175000017500000000310612453522764015206 0ustar sisksisk#!/usr/bin/perl use strict; use lib './lib'; use Test::More tests => 121; use FindBin; use lib $FindBin::RealBin; use testload; my $file = "$Dat_Dir/skew.html"; use HTML::TableExtract; # By count my $label = 'by header with span correction'; my $te = HTML::TableExtract->new( headers => [ qw(head0 head1 head2 head3) ], ); ok($te->parse_file($file), "$label (parse_file)"); my @tablestates = $te->tables; cmp_ok(@tablestates, '==', 1, "$label (extract count)"); good_skew_data($_, "$label (skew data)") foreach @tablestates; good_sticky_data($_, "$label (sticky data)") foreach @tablestates; # test aliasing directly -- this is tightly coupled with a cell in the # skew.html test file. my $str = "BIG\nJUNK"; $label = 'alias (no headers)'; alias_test($te->first_table_found, 2, 1, $str, $label); $label = 'alias (keep headers)'; $te = HTML::TableExtract->new( headers => [ qw(head0 head1 head2 head3) ], keep_headers => 1, ); ok($te->parse_file($file), "alias parse (keep headers)"); alias_test($te->first_table_found, 3, 1, $str, $label); sub alias_test { my($ts, $r, $c, $str, $label) = @_; my $item1 = $ts->row($r)->[$c]; my @rows = $ts->rows; my $item2 = $rows[$r][$c]; my $cell = $ts->cell($r,$c); my $cellno = $ts->cell($r+2,$c+1); my $space = $ts->space($r+2,$c+1); cmp_ok($str, 'eq', $item1, "$label (via row)"); cmp_ok($str, 'eq', $item2, "$label (via rows)"); cmp_ok($str, 'eq', $cell, "$label (via cell)"); cmp_ok($str, 'eq', $space, "$label (via space)"); cmp_ok(defined undef, '==', defined $cellno, "$label (undef via cell)"); } HTML-TableExtract-2.13/t/13_attrib.t0000755000175000017500000000211112453522764015517 0ustar sisksisk#!/usr/bin/perl use strict; use lib './lib'; use Test::More tests => 556; use FindBin; use lib $FindBin::RealBin; use testload; my $file = "$Dat_Dir/basic.html"; use HTML::TableExtract; my($label, $te, @tables); # By border $label = 'by attribute (regular)'; $te = HTML::TableExtract->new( attribs => { border => 1 } ); ok($te->parse_file($file), "$label (parse_file)"); @tables = $te->tables; cmp_ok(@tables, '==', 3, "$label (extract count)"); good_data($_, "$label (data)") foreach @tables; # By cellpadding $label = 'by attribute (subset)'; $te = HTML::TableExtract->new( attribs => { cellpadding => 1 } ); ok($te->parse_file($file), "$label (parse_file)"); @tables = $te->tables; cmp_ok(@tables, '==', 1, "$label (extract count)"); good_data($_, "$label (data)") foreach @tables; # By cellpadding existence $label = 'by attribute (undef)'; $te = HTML::TableExtract->new( attribs => { cellpadding => undef } ); ok($te->parse_file($file), "$label (parse_file)"); @tables = $te->tables; cmp_ok(@tables, '==', 1, "$label (extract count)"); good_data($_, "$label (data)") foreach @tables; HTML-TableExtract-2.13/t/subtable.html0000644000175000017500000001223512453522764016236 0ustar sisksisk TableExtract Test HTML

Here lies Table 3 with 4, 5, and 6 inside:

SubtableHead ZeroSubtableHead OneSubtableHead TwoSubtableHead ThreeSubtableHead FourSubtableHead FiveSubtableHead SixSubtableHead SevenSubtableHead EightSubtableHead Nine
(1,0) (1,1) (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) (1,8) (1,9)
(2,0) (2,1) (2,2) (2,3) (2,4) (2,5)
HubbaBubbaSandmanPiper
 (2,6) (2,7) (2,8) (2,9)
(3,0) (3,1) (3,2) (3,3) (3,4) (3,5) (3,6) (3,7) (3,8) (3,9)
(4,0) (4,1) (4,2) (4,3) (4,4) (4,5) (4,6) (4,7) (4,8) (4,9)
(5,0) (5,1) (5,2) (5,3) (5,4) (5,5)
Header ZeroHeader OneHeader TwoHeader ThreeHeader FourHeader FiveHeader SixHeader SevenHeader EightHeader Nine
(1,0) (1,1) (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) (1,8) (1,9)
(2,0) (2,1) (2,2) (2,3) (2,4) (2,5) (2,6) (2,7) (2,8) (2,9)
(3,0) (3,1) (3,2) (3,3) (3,4) (3,5) (3,6) (3,7) (3,8) (3,9)
(4,0) (4,1) (4,2) (4,3) (4,4) (4,5) (4,6) (4,7) (4,8) (4,9)
(5,0) (5,1) (5,2) (5,3) (5,4) (5,5) (5,6) (5,7) (5,8) (5,9)
(6,0) (6,1) (6,2) (6,3) (6,4) (6,5) (6,6) (6,7) (6,8) (6,9)
(7,0) (7,1) (7,2) (7,3) (7,4) (7,5) (7,6) (7,7) (7,8) (7,9)
(8,0) (8,1) (8,2) (8,3) (8,4) (8,5) (8,6) (8,7) (8,8) (8,9)
(9,0) (9,1) (9,2) (9,3) (9,4) (9,5) (9,6) (9,7) (9,8) (9,9)
 (5,6) (5,7) (5,8) (5,9)
(6,0) (6,1) (6,2) (6,3) (6,4) (6,5) (6,6) (6,7) (6,8) (6,9)
(7,0) (7,1) (7,2) (7,3) (7,4) (7,5) (7,6) (7,7)
Header ZeroHeader OneHeader TwoHeader ThreeHeader FourHeader FiveHeader SixHeader SevenHeader EightHeader Nine
(1,0) (1,1) (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) (1,8) (1,9)
(2,0) (2,1) (2,2) (2,3) (2,4) (2,5) (2,6) (2,7) (2,8) (2,9)
(3,0) (3,1) (3,2) (3,3) (3,4) (3,5) (3,6) (3,7) (3,8) (3,9)
(4,0) (4,1) (4,2) (4,3) (4,4) (4,5) (4,6) (4,7) (4,8) (4,9)
(5,0) (5,1) (5,2) (5,3) (5,4) (5,5) (5,6) (5,7) (5,8) (5,9)
(6,0) (6,1) (6,2) (6,3) (6,4) (6,5) (6,6) (6,7) (6,8) (6,9)
(7,0) (7,1) (7,2) (7,3) (7,4) (7,5) (7,6) (7,7) (7,8) (7,9)
(8,0) (8,1) (8,2) (8,3) (8,4) (8,5) (8,6) (8,7) (8,8) (8,9)
(9,0) (9,1) (9,2) (9,3) (9,4) (9,5) (9,6) (9,7) (9,8) (9,9)
 (7,8) (7,9)
(8,0) (8,1) (8,2) (8,3) (8,4) (8,5) (8,6) (8,7) (8,8) (8,9)
(9,0) (9,1) (9,2) (9,3) (9,4) (9,5) (9,6) (9,7) (9,8) (9,9)
HTML-TableExtract-2.13/t/01_pod.t0000755000175000017500000000035512453544541015016 0ustar sisksiskuse Test::More; my $msg; if (! $ENV{HTE_DEV_TESTS}) { $msg = "(dev only)"; } else { eval "use Test::Pod 1.00"; $msg = "Test::Pod 1.00 or greater required for testing POD" if $@; } plan skip_all => $msg if $msg; all_pod_files_ok(); HTML-TableExtract-2.13/t/10_bulk.t0000755000175000017500000000153612453522764015176 0ustar sisksisk#!/usr/bin/perl use strict; use lib './lib'; use Test::More tests => 52; use FindBin; use lib $FindBin::RealBin; use testload; my $file = "$Dat_Dir/ugly.html"; BEGIN { require_ok('HTML::TableExtract') } # by bulk, lineage integrity my $label = 'by bulk with lineage check'; my $te = HTML::TableExtract->new(); ok($te->parse_file($file), "$label (parse_file)"); my @tablestates = $te->tables; cmp_ok(@tablestates, '==', @LINEAGE_DATA, "$label (extract count)"); foreach my $tsc (0 .. $#tablestates) { my $ts = $tablestates[$tsc]; foreach (0 .. $#{$ts->{lineage}}) { cmp_ok(join(',', @{$ts->{lineage}[$_]}), 'eq', $LINEAGE_DATA[$tsc][$_], "$label (data)"); } my $mod = 1; $mod = 0 unless $ts->{headers} && !$ts->{keep_headers}; my @rows = $ts->rows; cmp_ok(@rows, '==', @{$ts->{grid}}-$mod, "rows() returns correct number of rows"); } HTML-TableExtract-2.13/t/skew.html0000644000175000017500000000131412453522764015402 0ustar sisksiskskew test
head0 head1 head2 head3
THIS IS A WHOLE ROW-CELL OF JUNK
JUNK Tasty tidbit (1,1) JUNK Tasty tidbit (1,3)
BIG
JUNK		 Tasty tidbit (2,3)
Tasty tidbit (3,0) Tasty tidbit (3,3)
Tasty tidbit (4,0) Tasty tidbit (4,3)
JUNK BUTTON Tasty tidbit (5,2) Tasty tidbit (5,3)
HTML-TableExtract-2.13/t/16_subtable.t0000755000175000017500000000334412453522764016047 0ustar sisksisk#!/usr/bin/perl use strict; use lib './lib'; use Test::More tests => 692; use FindBin; use lib $FindBin::RealBin; use testload; my $file = "$Dat_Dir/basic.html"; use HTML::TableExtract; # By count my $label = 'by subtable scoop'; my $te = HTML::TableExtract->new( depth => 0, count => 2, subtables => 1, ); ok($te->parse_file($file), "$label (parse_file)"); my @tablestates = $te->tables; cmp_ok(@tablestates, '==', 3, "$label (extract count)"); good_data($_, "$label (data)") foreach @tablestates; # Check subtable slice immunity $file = "$Dat_Dir/subtable.html"; $label = 'by subtable, slice immune'; $te = HTML::TableExtract->new( headers => [('SubtableHead Eight', 'SubtableHead Two')], subtables => 1, ); ok($te->parse_file($file), "$label (parse_file)"); @tablestates = $te->tables; cmp_ok(@tablestates, '==', 4, "$label (extract count)"); my $mule = splice(@tablestates, 0, 1); my @mrows = $mule->rows; cmp_ok(@mrows, '==', 1, "$label (mule row check)"); cmp_ok(@{$mrows[0]}, '==', 4, "$label (mule col check)"); good_slice_data($tablestates[-1], "$label (data)", 0, 3); good_data($_, "$label (data)") foreach @tablestates[0,1]; # Check subtable slice precedence $label = 'by subtable, slice precedence'; $te = HTML::TableExtract->new( headers => [('Head.*Eight', 'Head.*Two')], subtables => 1, ); ok($te->parse_file($file), "$label (parse_file)"); @tablestates = $te->tables; cmp_ok(@tablestates, '==', 4, "$label (extract count)"); $mule = splice(@tablestates, 2, 1); @mrows = $mule->rows; cmp_ok(@mrows, '==', 1, "$label (mule row check)"); cmp_ok(@{$mrows[0]}, '==', 4, "$label (mule col check)"); good_slice_data($tablestates[-1], "$label (data)", 0, 3); good_data($_, "$label (data)", 2, 8) foreach @tablestates[0,1]; HTML-TableExtract-2.13/t/15_depth_count.t0000755000175000017500000000076512453522764016565 0ustar sisksisk#!/usr/bin/perl use strict; use lib './lib'; use Test::More tests => 112; use FindBin; use lib $FindBin::RealBin; use testload; my $file = "$Dat_Dir/basic.html"; use HTML::TableExtract; # By count my $label = 'by depth and count'; my $te = HTML::TableExtract->new( depth => 0, count => 2, ); ok($te->parse_file($file), "$label (parse_file)"); my @tablestates = $te->tables; cmp_ok(@tablestates, '==', 1, "$label (extract count)"); good_data($_, "$label (data)") foreach @tablestates; HTML-TableExtract-2.13/t/14_headers.t0000755000175000017500000000140612453522764015654 0ustar sisksisk#!/usr/bin/perl use strict; use lib './lib'; use Test::More tests => 464; use FindBin; use lib $FindBin::RealBin; use testload; my $file = "$Dat_Dir/basic.html"; use HTML::TableExtract; # By headers my $label = 'by headers'; my $te = HTML::TableExtract->new( headers => [qw(Eight Six Four Two Zero)], ); ok($te->parse_file($file), "$label (parse_file)"); my @tablestates = $te->tables; cmp_ok(@tablestates, '==', 5, "$label (extract count)"); good_data($_, "$label (data)") foreach @tablestates; $te = HTML::TableExtract->new( headers => [qw(Eight Two)], ); ok($te->parse_file($file), "$label (parse_file)"); @tablestates = $te->tables; cmp_ok(@tablestates, '==', 5, "$label (extract count)"); good_slice_data($_, "$label (data)", 0, 3) foreach @tablestates; HTML-TableExtract-2.13/t/basic2.html0000644000175000017500000002166012453522764015602 0ustar sisksisk TableExtract Test HTML

Here lies Table 1:

not headernot headernot headernot headernot headernot headernot headernot headernot headernot header
not headernot headernot headernot headernot headernot headernot headernot headernot headernot header
not headernot headernot headernot headernot headernot headernot headernot headernot headernot header
Header ZeroHeader OneHeader TwoHeader ThreeHeader FourHeader FiveHeader SixHeader SevenHeader EightHeader Nine
(1,0) (1,1) (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) (1,8) (1,9)
(2,0) (2,1) (2,2) (2,3) (2,4) (2,5) (2,6) (2,7) (2,8) (2,9)
(3,0) (3,1) (3,2) (3,3) (3,4) (3,5) (3,6) (3,7) (3,8) (3,9)
(4,0) (4,1) (4,2) (4,3) (4,4) (4,5) (4,6) (4,7) (4,8) (4,9)
(5,0) (5,1) (5,2) (5,3) (5,4) (5,5) (5,6) (5,7) (5,8) (5,9)
(6,0) (6,1) (6,2) (6,3) (6,4) (6,5) (6,6) (6,7) (6,8) (6,9)
(7,0) (7,1) (7,2) (7,3) (7,4) (7,5) (7,6) (7,7) (7,8) (7,9)
(8,0) (8,1) (8,2) (8,3) (8,4) (8,5) (8,6) (8,7) (8,8) (8,9)
(9,0) (9,1) (9,2) (9,3) (9,4) (9,5) (9,6) (9,7) (9,8) (9,9)

Here lies Table 2:

Header ZeroHeader OneHeader TwoHeader ThreeHeader FourHeader FiveHeader SixHeader SevenHeader EightHeader Nine
(1,0) (1,1) (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) (1,8) (1,9)
(2,0) (2,1) (2,2) (2,3) (2,4) (2,5) (2,6) (2,7) (2,8) (2,9)
(3,0) (3,1) (3,2) (3,3) (3,4) (3,5) (3,6) (3,7) (3,8) (3,9)
(4,0) (4,1) (4,2) (4,3) (4,4) (4,5) (4,6) (4,7) (4,8) (4,9)
(5,0) (5,1) (5,2) (5,3) (5,4) (5,5) (5,6) (5,7) (5,8) (5,9)
(6,0) (6,1) (6,2) (6,3) (6,4) (6,5) (6,6) (6,7) (6,8) (6,9)
(7,0) (7,1) (7,2) (7,3) (7,4) (7,5) (7,6) (7,7) (7,8) (7,9)
(8,0) (8,1) (8,2) (8,3) (8,4) (8,5) (8,6) (8,7) (8,8) (8,9)
(9,0) (9,1) (9,2) (9,3) (9,4) (9,5) (9,6) (9,7) (9,8) (9,9)

Here lies Table 3 with 4 and 5 inside:

Header ZeroHeader OneHeader TwoHeader ThreeHeader FourHeader FiveHeader SixHeader SevenHeader EightHeader Nine
(1,0) (1,1) (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) (1,8) (1,9)
(2,0) (2,1) (2,2) (2,3) (2,4) (2,5) (2,6) (2,7) (2,8) (2,9)
(3,0) (3,1) (3,2) (3,3) (3,4) (3,5) (3,6) (3,7) (3,8) (3,9)
(4,0) (4,1) (4,2) (4,3) (4,4) (4,5) (4,6) (4,7) (4,8) (4,9)
(5,0) (5,1) (5,2) (5,3) (5,4) (5,5)
Header ZeroHeader OneHeader TwoHeader ThreeHeader FourHeader FiveHeader SixHeader SevenHeader EightHeader Nine
(1,0) (1,1) (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) (1,8) (1,9)
(2,0) (2,1) (2,2) (2,3) (2,4) (2,5) (2,6) (2,7) (2,8) (2,9)
(3,0) (3,1) (3,2) (3,3) (3,4) (3,5) (3,6) (3,7) (3,8) (3,9)
(4,0) (4,1) (4,2) (4,3) (4,4) (4,5) (4,6) (4,7) (4,8) (4,9)
(5,0) (5,1) (5,2) (5,3) (5,4) (5,5) (5,6) (5,7) (5,8) (5,9)
(6,0) (6,1) (6,2) (6,3) (6,4) (6,5) (6,6) (6,7) (6,8) (6,9)
(7,0) (7,1) (7,2) (7,3) (7,4) (7,5) (7,6) (7,7) (7,8) (7,9)
(8,0) (8,1) (8,2) (8,3) (8,4) (8,5) (8,6) (8,7) (8,8) (8,9)
(9,0) (9,1) (9,2) (9,3) (9,4) (9,5) (9,6) (9,7) (9,8) (9,9)
 (5,6) (5,7) (5,8) (5,9)
(6,0) (6,1) (6,2) (6,3) (6,4) (6,5) (6,6) (6,7) (6,8) (6,9)
(7,0) (7,1) (7,2) (7,3) (7,4) (7,5) (7,6) (7,7)
Header ZeroHeader OneHeader TwoHeader ThreeHeader FourHeader FiveHeader SixHeader SevenHeader EightHeader Nine
(1,0) (1,1) (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) (1,8) (1,9)
(2,0) (2,1) (2,2) (2,3) (2,4) (2,5) (2,6) (2,7) (2,8) (2,9)
(3,0) (3,1) (3,2) (3,3) (3,4) (3,5) (3,6) (3,7) (3,8) (3,9)
(4,0) (4,1) (4,2) (4,3) (4,4) (4,5) (4,6) (4,7) (4,8) (4,9)
(5,0) (5,1) (5,2) (5,3) (5,4) (5,5) (5,6) (5,7) (5,8) (5,9)
(6,0) (6,1) (6,2) (6,3) (6,4) (6,5) (6,6) (6,7) (6,8) (6,9)
(7,0) (7,1) (7,2) (7,3) (7,4) (7,5) (7,6) (7,7) (7,8) (7,9)
(8,0) (8,1) (8,2) (8,3) (8,4) (8,5) (8,6) (8,7) (8,8) (8,9)
(9,0) (9,1) (9,2) (9,3) (9,4) (9,5) (9,6) (9,7) (9,8) (9,9)
 (7,8) (7,9)
(8,0) (8,1) (8,2) (8,3) (8,4) (8,5) (8,6) (8,7) (8,8) (8,9)
(9,0) (9,1) (9,2) (9,3) (9,4) (9,5) (9,6) (9,7) (9,8) (9,9)
HTML-TableExtract-2.13/t/gnarly.html0000644000175000017500000000151312453522764015726 0ustar sisksiskgnarly table
(0,0) [1,4](0,1) [2,4]
(1,0) [2,1](1,1) [1,1](1,2) [1,2]
(2,0) [2,4](2,1) [2,2](2,2) [1,1]
(3,0) [1,1](3,1) [1,1]
(4,0) [3,2](4,1) [1,1](4,2) [3,1](4,3) [4,4]
(5,0) [1,1]
(6,0) [1,1]
(7,0) [1,4]
HTML-TableExtract-2.13/t/00_basic.t0000755000175000017500000000010212453522764015305 0ustar sisksiskuse Test::More tests => 1; BEGIN { use_ok('HTML::TableExtract') } HTML-TableExtract-2.13/t/22_header_row.t0000755000175000017500000000707712453522764016371 0ustar sisksisk#!/usr/bin/perl use strict; use lib './lib'; use Test::More tests => 19; use FindBin; use lib $FindBin::RealBin; use testload; my $file = "$Dat_Dir/basic.html"; my $file2 = "$Dat_Dir/basic2.html"; use HTML::TableExtract; # Check header row retention cases my($label, $te, @rows, $table); $label = 'header row (basic, default)'; $te = HTML::TableExtract->new(); ok($te->parse_file($file), "$label (parse)"); $table = $te->first_table_found; @rows = $table->rows; cmp_ok(@rows, '==', scalar @{$table->{grid}}, "$label (row count)"); $label = 'header row (basic, no keep)'; $te = HTML::TableExtract->new( keep_headers => 0 ); ok($te->parse_file($file), "$label (parse)"); $table = $te->first_table_found; @rows = $table->rows; cmp_ok(@rows, '==', scalar @{$table->{grid}}, "$label (row count)"); $label = 'header row (basic, keep)'; $te = HTML::TableExtract->new( keep_headers => 1 ); ok($te->parse_file($file), "$label (parse)"); $table = $te->first_table_found; @rows = $table->rows; cmp_ok(@rows, '==', scalar @{$table->{grid}}, "$label (row count)"); my(@hrow, $hindex); my @headers = qw(Eight Six Four Two Zero); my @hlabels = map("Header $_", @headers); $label = 'header row (header, default)'; $te = HTML::TableExtract->new( headers => [@headers] ); ok($te->parse_file($file), "$label (parse)"); $table = $te->first_table_found; @rows = $table->rows; cmp_ok(@rows, '==', scalar @{$table->{grid}} - 1, "$label (row count)"); $hindex = $table->hrow_index; @hrow = $table->hrow; cmp_ok(join(' ', @hrow), 'eq', join(' ', @hlabels), "$label (hrow)"); $label = 'header row (header, nokeep)'; $te = HTML::TableExtract->new( headers => [@headers], keep_headers => 0, ); ok($te->parse_file($file), "$label (parse)"); $table = $te->first_table_found; @rows = $table->rows; cmp_ok(@rows, '==', scalar @{$table->{grid}} - 1, "$label (row count)"); $hindex = $table->hrow_index; @hrow = $table->hrow; cmp_ok(join(' ', @hrow), 'eq', join(' ', @hlabels), "$label (hrow)"); $label = 'header row (header, keep)'; $te = HTML::TableExtract->new( headers => [@headers], keep_headers => 1, ); ok($te->parse_file($file), "$label (parse)"); $table = $te->first_table_found; @rows = $table->rows; cmp_ok(@rows, '==', scalar @{$table->{grid}}, "$label (row count)"); $hindex = $table->hrow_index; @hrow = $table->hrow; cmp_ok(join(' ', @hrow), 'eq', join(' ', @hlabels), "$label (hrow)"); ### # Traditionally we clip extraneous rows above our header rows. $label = 'pre-header row clip (header, nokeep)'; $te = HTML::TableExtract->new( headers => [@headers], keep_headers => 0, ); ok($te->parse_file($file2), "$label (parse)"); $table = $te->first_table_found; my $ghi = get_grid_header_index($table->{grid}); @rows = $table->rows; cmp_ok(@rows, '==', scalar @{$table->{grid}} - $ghi - 1, "$label (row count)"); $label = 'pre-header row clip (header, keep)'; $te = HTML::TableExtract->new( headers => [@headers], keep_headers => 1, ); ok($te->parse_file($file2), "$label (parse)"); $table = $te->first_table_found; $ghi = get_grid_header_index($table->{grid}); @rows = $table->rows; cmp_ok(@rows, '==', scalar @{$table->{grid}} - $ghi, "$label (row count)"); sub get_grid_header_index { my $grid = shift; my $ghi = 0; foreach (0 .. $#{$table->{grid}}) { my $item = $table->{grid}[$_][0]; $item = $$item if ref $item; next if $item =~ /not\s+header/i; $ghi = $_; last; } $ghi; } HTML-TableExtract-2.13/t/12_depth.t0000755000175000017500000000072212453522764015343 0ustar sisksisk#!/usr/bin/perl use strict; use lib './lib'; use Test::More tests => 222; use FindBin; use lib $FindBin::RealBin; use testload; my $file = "$Dat_Dir/basic.html"; use HTML::TableExtract; # By count my $label = 'by depth'; my $te = HTML::TableExtract->new( depth => 1 ); ok($te->parse_file($file), "$label (parse_file)"); my @tablestates = $te->tables; cmp_ok(@tablestates, '==', 2, "$label (extract count)"); good_data($_, "$label (data)") foreach @tablestates; HTML-TableExtract-2.13/t/testload.pm0000644000175000017500000001451112453522764015723 0ustar sisksiskpackage testload; use strict; use Test::More; use File::Spec; use vars qw( @ISA @EXPORT $Dat_Dir @LINEAGE_DATA @HEADERS @SKEW_DATA @GNARLY_DATA @TRANSLATION_DATA ); require Exporter; @ISA = qw(Exporter); @EXPORT = qw( $Dat_Dir @LINEAGE_DATA @HEADERS @SKEW_DATA i @TRANSLATION_DATA @GNARLY_DATA good_data good_slice_data good_skew_data good_gnarly_data good_sticky_data ); my $base_dir; BEGIN { my $pkg = __PACKAGE__; $pkg =~ s%::%/%g; $pkg .= '.pm'; my @parts = File::Spec->splitpath(File::Spec->canonpath($INC{$pkg})); $parts[-1] = ''; $base_dir = File::Spec->catpath(@parts); } $Dat_Dir = $base_dir; # For dataset 'chain' @LINEAGE_DATA = ( [ '0,0,1,0', '1,0,1,0', '2,0,1,0', '3,0' ], [ '0,0,1,0', '1,0,1,0', '2,0,2,1', '3,1' ], [ '0,0,1,0', '1,0,1,0', '2,0' ], [ '0,0,1,0', '1,0,2,1', '2,1,1,1', '3,2' ], [ '0,0,1,0', '1,0,2,1', '2,1,2,0', '3,3' ], [ '0,0,1,0', '1,0,2,1', '2,1' ], [ '0,0,1,0', '1,0' ], [ '0,0,2,1', '1,1,1,1', '2,2,1,0', '3,4' ], [ '0,0,2,1', '1,1,1,1', '2,2,2,1', '3,5' ], [ '0,0,2,1', '1,1,1,1', '2,2' ], [ '0,0,2,1', '1,1,2,0', '2,3,1,1', '3,6' ], [ '0,0,2,1', '1,1,2,0', '2,3,2,0', '3,7' ], [ '0,0,2,1', '1,1,2,0', '2,3' ], [ '0,0,2,1', '1,1' ], [ '0,0' ] ); # For data set 'basic' @HEADERS = ( 'Header Zero', 'Header One', 'Header Two', 'Header Three', 'Header Four', 'Header Five', 'Header Six', 'Header Seven', 'Header Eight', 'Header Nine', ); # For data set 'skew' @SKEW_DATA = ( [ 'head0','head1','head2','head3' ], [ 'THIS IS A WHOLE ROW-CELL OF JUNK','','','' ], [ 'JUNK','Tasty tidbit (1,1)','JUNK','Tasty tidbit (1,3)' ], [ '',"BIG\nJUNK",'','Tasty tidbit (2,3)' ], [ 'Tasty tidbit (3,0)','','','Tasty tidbit (3,3)' ], [ 'Tasty tidbit (4,0)','','','Tasty tidbit (4,3)' ], [ 'JUNK BUTTON','','Tasty tidbit (5,2)','Tasty tidbit (5,3)' ], ); @TRANSLATION_DATA = ( [ '0,0', '0,1', '0,2', '0,3' ], [ '1,0', '1,0', '1,0', '1,0' ], [ '2,0', '2,1', '2,2', '2,3' ], [ '2,0', '3,1', '3,1', '3,3' ], [ '4,0', '3,1', '3,1', '4,3' ], [ '5,0', '3,1', '3,1', '5,3' ], [ '6,0', '6,0', '6,2', '6,3' ] ); @GNARLY_DATA = ( [ '(0,0) [1,4]', '', '', '', '(0,1) [2,4]', '', '', '' ], [ '(1,0) [2,1]', '(1,1) [1,1]', '(1,2) [1,2]', '', '', '', '', '' ], [ '', '(2,0) [2,4]', '', '', '', '(2,1) [2,2]', '', '(2,2) [1,1]' ], [ '(3,0) [1,1]', '', '', '', '', '', '', '(3,1) [1,1]' ], [ '(4,0) [3,2]', '', '(4,1) [1,1]', '(4,2) [3,1]', '(4,3) [4,4]', '', '', '' ], [ '', '', '(5,0) [1,1]', '', '', '', '', '' ], [ '', '', '(6,0) [1,1]', '', '', '', '', '' ], [ '(7,0) [1,4]', '', '', '', '', '', '', '' ] ); sub good_data { my($ts, $label, @slice) = @_; ref $ts or die "Oops: Table state ref required\n"; my $t = $ts->{grid}; my $skew; my $txt = ref $t->[0][0] eq 'SCALAR' ? ${$t->[0][0]} : $t->[0][0]->as_text; $skew = $txt =~ /^Header/ ? 1 : 0; my $row = 0 + $skew; if (@slice) { my @rows = $ts->rows; cmp_ok(scalar @slice, '==', scalar @{$rows[0]}, "$label (col cnt)"); } # Must have rows ok(scalar @{$t}, "$label (rows)"); # See if we got the numbers. foreach my $r ($row .. $#$t) { # Must have columns ok(scalar @{$t->[$r]}, "$label (columns)"); my @indices = @slice ? @slice : 0 .. $#{$t->[$r]}; foreach my $c (@indices) { my $rc = $skew ? $r : $r + 1; next if $ts->{headers} && !$ts->{hits}{$c}; my $txt = ref $t->[$r][$c] eq 'SCALAR' ? ${$t->[$r][$c]} : $t->[$r][$c]->as_text; like($txt, qr/^ \($rc,$c\)/, "$label ($r,$c)"); } } # Header order check if ($skew) { foreach my $c (0 .. $#{$t->[0]}) { my $hs = $HEADERS[$c]; my $txt = ref $t->[0][$c] eq 'SCALAR' ? ${$t->[0][$c]} : $t->[0][$c]->as_text; like($txt, qr/^$hs$/, "$label (header order)"); } } 1; } sub good_slice_data { my($ts, $label, @slice) = @_; my $t = $ts->{grid}; my @rows = $ts->rows; my $txt = ref $t->[0][0] eq 'SCALAR' ? ${$t->[0][0]} : $t->[0][0]->as_text; my $skew = 1; foreach my $r (0 .. $#rows) { my $row = $rows[$r]; my $trow = $t->[$r+$skew]; ok(@$row == @slice, "$label (slice width)"); my @s = $ts->column_map; foreach my $c (0 .. $#$row) { my $sc = $s[$c]; my $cell = $trow->[$sc]; my $txt = ref $cell eq 'SCALAR' ? $$cell : $cell->as_text; ok($row->[$c] eq $txt, "$label ($r,$c)"); } } } sub good_skew_data { push(@_, 0) if @_ == 2; _good_span_data(@_, \@SKEW_DATA); } sub good_gnarly_data { push(@_, 0) if @_ == 2; _good_span_data(@_, \@GNARLY_DATA); } sub _good_span_data { my($ts, $label, $reverse, $REF_DATA) = @_; ref $ts or die "Oops: Table state ref required\n"; my $t = $ts->{grid}; foreach my $r (1 .. $#$t) { my $row = $t->[$r]; my @cols = 0 .. $#$row; @cols = reverse @cols if $reverse; foreach my $c (@cols) { my $txt = ref $row->[$c] eq 'SCALAR' ? ${$row->[$c]} : $row->[$c]->as_text; $txt = '' unless defined $txt; cmp_ok($txt, 'eq', $REF_DATA->[$r][$c], $label); } } 1; } sub good_sticky_data { # testing grid aliasing my($ts, $label, $reverse) = @_; ref $ts or die "Oops: Table state ref required\n"; my $t = $ts->_gridalias; foreach my $r (0 .. $#$t) { my $row = $t->[$r]; my @cols = 0 .. $#$row; @cols = reverse @cols if $reverse; foreach my $c (@cols) { my $txt = ref $row->[$c] eq 'SCALAR' ? ${$row->[$c]} : $row->[$c]->as_text; my($tr,$tc) = $ts->source_coords($r,$c); cmp_ok("$tr,$tc", 'eq', $TRANSLATION_DATA[$r][$c], "$label (coords)"); my $trow = $t->[$tr]; my $ttxt = ref $trow->[$tc] eq 'SCALAR' ? ${$trow->[$tc]} : $trow->[$tc]->as_text; cmp_ok($ttxt, 'eq', $txt, "$label (content)"); cmp_ok($ttxt, 'eq', $SKEW_DATA[$tr][$tc], "$label (abs)"); } } 1; } 1; HTML-TableExtract-2.13/t/21_skew_map.t0000755000175000017500000000101512453522764016041 0ustar sisksisk#!/usr/bin/perl use strict; use lib './lib'; use Test::More tests => 26; use FindBin; use lib $FindBin::RealBin; use testload; my $file = "$Dat_Dir/skew.html"; use HTML::TableExtract; # By count my $label = 'by header with column mapping'; my $te = HTML::TableExtract->new( headers => [ qw(head3 head2 head1 head0) ], ); ok($te->parse_file($file), "$label (parse_file)"); my @tablestates = $te->tables; cmp_ok(@tablestates, '==', 1, "$label (extract count)"); good_skew_data($_, "$label (data)") foreach @tablestates; HTML-TableExtract-2.13/t/30_tree.t0000755000175000017500000000362012453522764015176 0ustar sisksisk#!/usr/bin/perl my $test_count; BEGIN { $test_count = 126 } use strict; use lib './lib'; use Test::More tests => $test_count; use FindBin; use lib $FindBin::RealBin; use testload; my $et_version = '1.17'; my($tb_present, $et_present); eval "use HTML::TreeBuilder"; $tb_present = !$@; eval "use HTML::ElementTable $et_version"; $et_present = !$@; SKIP: { skip "HTML::TreeBuilder not installed", $test_count unless $tb_present; skip "HTML::ElementTable $et_version not installed", $test_count unless $et_present; use_ok("HTML::TableExtract", qw(tree)); my $file = "$Dat_Dir/gnarly.html"; my $label = 'element table'; my $te = HTML::TableExtract->new(); isa_ok($te, 'HTML::TreeBuilder', "$label - HTML::TableExtract"); ok($te->parse_file($file), "$label (parse_file)"); my @tablestates = $te->tables; cmp_ok(@tablestates, '==', 1, "$label (extract count)"); good_gnarly_data($_, "$label (data)") foreach @tablestates; my $tree = $te->tree; ok($tree, 'treetop'); isa_ok($tree, 'HTML::Element'); foreach my $ts ($te->tables) { my $tree = $ts->tree; ok($tree, 'tabletop'); isa_ok($tree, 'HTML::ElementTable'); } local *FH; open(FH, '<', $file) or die "Oops opening $file : $!\n"; my $hstr = join('', ); close(FH); $hstr =~ s/\n//gm; $te->_attribute_purge; my $estr = $te->elementify->as_HTML; $estr =~ s/\n//gm; $estr =~ s/\"//gm; cmp_ok($estr, 'eq', $hstr, 'mass html comp'); # TREE() gets called during header extractions, make sure it does $label .= ' (header)'; $te = HTML::TableExtract->new( headers => [qr|\(0,1\) \[2,4\]|], ); ok($te->parse_file($file), "$label (parse_file)"); $tree = $te->tree; ok($tree, 'treetop'); isa_ok($tree, 'HTML::Element'); my $table = $te->first_table_found; good_gnarly_data($table, "$label (data)"); $tree = $table->tree; ok($tree, 'tabletop'); isa_ok($tree, 'HTML::ElementTable'); } HTML-TableExtract-2.13/t/11_count.t0000755000175000017500000000072212453522764015366 0ustar sisksisk#!/usr/bin/perl use strict; use lib './lib'; use Test::More tests => 222; use FindBin; use lib $FindBin::RealBin; use testload; my $file = "$Dat_Dir/basic.html"; use HTML::TableExtract; # By count my $label = 'by count'; my $te = HTML::TableExtract->new( count => 1 ); ok($te->parse_file($file), "$label (parse_file)"); my @tablestates = $te->tables; cmp_ok(@tablestates, '==', 2, "$label (extract count)"); good_data($_, "$label (data)") foreach @tablestates; HTML-TableExtract-2.13/t/basic.html0000644000175000017500000002051512453522764015516 0ustar sisksisk TableExtract Test HTML

Here lies Table 1:

Header ZeroHeader OneHeader TwoHeader ThreeHeader FourHeader FiveHeader SixHeader SevenHeader EightHeader Nine
(1,0) (1,1) (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) (1,8) (1,9)
(2,0) (2,1) (2,2) (2,3) (2,4) (2,5) (2,6) (2,7) (2,8) (2,9)
(3,0) (3,1) (3,2) (3,3) (3,4) (3,5) (3,6) (3,7) (3,8) (3,9)
(4,0) (4,1) (4,2) (4,3) (4,4) (4,5) (4,6) (4,7) (4,8) (4,9)
(5,0) (5,1) (5,2) (5,3) (5,4) (5,5) (5,6) (5,7) (5,8) (5,9)
(6,0) (6,1) (6,2) (6,3) (6,4) (6,5) (6,6) (6,7) (6,8) (6,9)
(7,0) (7,1) (7,2) (7,3) (7,4) (7,5) (7,6) (7,7) (7,8) (7,9)
(8,0) (8,1) (8,2) (8,3) (8,4) (8,5) (8,6) (8,7) (8,8) (8,9)
(9,0) (9,1) (9,2) (9,3) (9,4) (9,5) (9,6) (9,7) (9,8) (9,9)

Here lies Table 2:

Header ZeroHeader OneHeader TwoHeader ThreeHeader FourHeader FiveHeader SixHeader SevenHeader EightHeader Nine
(1,0) (1,1) (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) (1,8) (1,9)
(2,0) (2,1) (2,2) (2,3) (2,4) (2,5) (2,6) (2,7) (2,8) (2,9)
(3,0) (3,1) (3,2) (3,3) (3,4) (3,5) (3,6) (3,7) (3,8) (3,9)
(4,0) (4,1) (4,2) (4,3) (4,4) (4,5) (4,6) (4,7) (4,8) (4,9)
(5,0) (5,1) (5,2) (5,3) (5,4) (5,5) (5,6) (5,7) (5,8) (5,9)
(6,0) (6,1) (6,2) (6,3) (6,4) (6,5) (6,6) (6,7) (6,8) (6,9)
(7,0) (7,1) (7,2) (7,3) (7,4) (7,5) (7,6) (7,7) (7,8) (7,9)
(8,0) (8,1) (8,2) (8,3) (8,4) (8,5) (8,6) (8,7) (8,8) (8,9)
(9,0) (9,1) (9,2) (9,3) (9,4) (9,5) (9,6) (9,7) (9,8) (9,9)

Here lies Table 3 with 4 and 5 inside:

Header ZeroHeader OneHeader TwoHeader ThreeHeader FourHeader FiveHeader SixHeader SevenHeader EightHeader Nine
(1,0) (1,1) (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) (1,8) (1,9)
(2,0) (2,1) (2,2) (2,3) (2,4) (2,5) (2,6) (2,7) (2,8) (2,9)
(3,0) (3,1) (3,2) (3,3) (3,4) (3,5) (3,6) (3,7) (3,8) (3,9)
(4,0) (4,1) (4,2) (4,3) (4,4) (4,5) (4,6) (4,7) (4,8) (4,9)
(5,0) (5,1) (5,2) (5,3) (5,4) (5,5)
Header ZeroHeader OneHeader TwoHeader ThreeHeader FourHeader FiveHeader SixHeader SevenHeader EightHeader Nine
(1,0) (1,1) (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) (1,8) (1,9)
(2,0) (2,1) (2,2) (2,3) (2,4) (2,5) (2,6) (2,7) (2,8) (2,9)
(3,0) (3,1) (3,2) (3,3) (3,4) (3,5) (3,6) (3,7) (3,8) (3,9)
(4,0) (4,1) (4,2) (4,3) (4,4) (4,5) (4,6) (4,7) (4,8) (4,9)
(5,0) (5,1) (5,2) (5,3) (5,4) (5,5) (5,6) (5,7) (5,8) (5,9)
(6,0) (6,1) (6,2) (6,3) (6,4) (6,5) (6,6) (6,7) (6,8) (6,9)
(7,0) (7,1) (7,2) (7,3) (7,4) (7,5) (7,6) (7,7) (7,8) (7,9)
(8,0) (8,1) (8,2) (8,3) (8,4) (8,5) (8,6) (8,7) (8,8) (8,9)
(9,0) (9,1) (9,2) (9,3) (9,4) (9,5) (9,6) (9,7) (9,8) (9,9)
 (5,6) (5,7) (5,8) (5,9)
(6,0) (6,1) (6,2) (6,3) (6,4) (6,5) (6,6) (6,7) (6,8) (6,9)
(7,0) (7,1) (7,2) (7,3) (7,4) (7,5) (7,6) (7,7)
Header ZeroHeader OneHeader TwoHeader ThreeHeader FourHeader FiveHeader SixHeader SevenHeader EightHeader Nine
(1,0) (1,1) (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) (1,8) (1,9)
(2,0) (2,1) (2,2) (2,3) (2,4) (2,5) (2,6) (2,7) (2,8) (2,9)
(3,0) (3,1) (3,2) (3,3) (3,4) (3,5) (3,6) (3,7) (3,8) (3,9)
(4,0) (4,1) (4,2) (4,3) (4,4) (4,5) (4,6) (4,7) (4,8) (4,9)
(5,0) (5,1) (5,2) (5,3) (5,4) (5,5) (5,6) (5,7) (5,8) (5,9)
(6,0) (6,1) (6,2) (6,3) (6,4) (6,5) (6,6) (6,7) (6,8) (6,9)
(7,0) (7,1) (7,2) (7,3) (7,4) (7,5) (7,6) (7,7) (7,8) (7,9)
(8,0) (8,1) (8,2) (8,3) (8,4) (8,5) (8,6) (8,7) (8,8) (8,9)
(9,0) (9,1) (9,2) (9,3) (9,4) (9,5) (9,6) (9,7) (9,8) (9,9)
 (7,8) (7,9)
(8,0) (8,1) (8,2) (8,3) (8,4) (8,5) (8,6) (8,7) (8,8) (8,9)
(9,0) (9,1) (9,2) (9,3) (9,4) (9,5) (9,6) (9,7) (9,8) (9,9)
HTML-TableExtract-2.13/t/ugly.html0000644000175000017500000001511112453522764015411 0ustar sisksisk Chain Test

You like ugly? I'll show you UGLY:

0,0: (0,0)0,0: (0,1)
0,0: (1,0)
1,0: (0,0)1,0: (0,1)
1,0: (1,0)
2,0: (0,0)2,0: (0,1)
2,0: (1,0)
3,0: (0,0)3,0: (0,1)
3,0: (1,0)3,0: (1,1)
3,0: (2,0)3,0: (2,1)
2,0: (1,1)
2,0: (2,0)2,0: (2,1)
3,1: (0,0)3,1: (0,1)
3,1: (1,0)3,1: (1,1)
3,1: (2,0)3,1: (2,1)
1,0: (1,1)
1,0: (2,0)1,0: (2,1)
2,1: (0,0)2,1: (0,1)
2,1: (1,0)2,1: (1,1)
3,2: (0,0)3,2: (0,1)
3,2: (1,0)3,2: (1,1)
3,2: (2,0)3,2: (2,1)
2,1: (2,0)
3,3: (0,0)3,3: (0,1)
3,3: (1,0)3,3: (1,1)
3,3: (2,0)3,3: (2,1)
2,1: (2,1)
0,0: (1,1)
0,0: (2,0)0,0: (2,1)
1,1: (0,0)1,1: (0,1)
1,1: (1,0)1,1: (1,1)
2,2: (0,0)2,2: (0,1)
2,2: (1,0)
3,4: (0,0)3,4: (0,1)
3,4: (1,0)3,4: (1,1)
3,4: (2,0)3,4: (2,1)
2,2: (1,1)
2,2: (2,0)2,2: (2,1)
3,5: (0,0)3,5: (0,1)
3,5: (1,0)3,5: (1,1)
3,5: (2,0)3,5: (2,1)
1,1: (2,0)
2,3: (0,0)2,3: (0,1)
2,3: (1,0)2,3: (1,1)
3,6: (0,0)3,6: (0,1)
3,6: (1,0)3,6: (1,1)
3,6: (2,0)3,6: (2,1)
2,3: (2,0)
3,7: (0,0)3,7: (0,1)
3,7: (1,0)3,7: (1,1)
3,7: (2,0)3,7: (2,1)
2,3: (2,1)
1,1: (2,1)
HTML-TableExtract-2.13/t/02_pod_coverage.t0000755000175000017500000000045012453544567016676 0ustar sisksiskuse Test::More; my $msg; if (! $ENV{HTE_DEV_TESTS}) { $msg = "(dev only)"; } else { eval "use Test::Pod::Coverage 1.00"; $msg = "Test::Pod::Coverage 1.00 required for testing POD coverage" if $@; } plan skip_all => $msg if $msg; all_pod_coverage_ok({also_private => [qw/TREE parse eof/]}); HTML-TableExtract-2.13/MANIFEST0000644000175000017500000000062112527402256014424 0ustar sisksiskREADME Changes LICENSE MANIFEST Makefile.PL examples.html META.json META.yml lib/HTML/TableExtract.pm t/00_basic.t t/01_pod.t t/02_pod_coverage.t t/10_bulk.t t/11_count.t t/12_depth.t t/13_attrib.t t/14_headers.t t/15_depth_count.t t/16_subtable.t t/20_skew.t t/21_skew_map.t t/22_header_row.t t/30_tree.t t/testload.pm t/basic.html t/basic2.html t/gnarly.html t/skew.html t/subtable.html t/ugly.html HTML-TableExtract-2.13/lib/0000755000175000017500000000000012527403163014040 5ustar sisksiskHTML-TableExtract-2.13/lib/HTML/0000755000175000017500000000000012527403163014604 5ustar sisksiskHTML-TableExtract-2.13/lib/HTML/TableExtract.pm0000644000175000017500000015365612527403075017546 0ustar sisksiskpackage HTML::TableExtract; # This package extracts tables from HTML. Tables of interest may be # specified using header information, depth, order in a depth, table tag # attributes, or some combination of the four. See the POD for more # information. # # Author: Matthew P. Sisk. See the POD for copyright information. use strict; use Carp; use vars qw($VERSION @ISA); $VERSION = '2.13'; use HTML::Parser; @ISA = qw(HTML::Parser); use HTML::Entities; # trickery for subclassing from HTML::TreeBuilder rather than the # default HTML::Parser. (use HTML::TableExtract qw(tree);) Also installs # a mode constant TREE(). BEGIN { *TREE = sub { 0 } } sub import { my $class = shift; no warnings; *TREE = @_ ? sub { 1 } : sub { 0 }; return unless @_; my $mode = shift; croak "Unknown mode '$mode'\n" unless $mode eq 'tree'; eval "use HTML::TreeBuilder"; croak "Problem loading HTML::TreeBuilder : $@\n" if $@; eval "use HTML::ElementTable 1.17"; croak "problem loading HTML::ElementTable : $@\n" if $@; @ISA = qw(HTML::TreeBuilder); $class; } # Backwards compatibility for deprecated methods *table_state = *table; *table_states = *tables; *first_table_state_found = *first_table_found; ### my %Defaults = ( headers => undef, depth => undef, count => undef, attribs => undef, subtables => undef, gridmap => 1, decode => 1, automap => 1, slice_columns => 1, keep_headers => 0, br_translate => 1, error_handle => \*STDOUT, debug => 0, keep_html => 0, strip_html_on_match => 1, ); my $Dpat = join('|', sort keys %Defaults); ### Constructor sub new { my $that = shift; my $class = ref($that) || $that; my(%pass, %parms, $k, $v); while (($k,$v) = splice(@_, 0, 2)) { if ($k eq 'headers') { ref $v eq 'ARRAY' or croak "Param '$k' must be passed in ref to array\n"; $parms{$k} = $v; } elsif ($k =~ /^$Dpat$/) { $parms{$k} = $v; } else { $pass{$k} = $v; } } my $self = $class->SUPER::new(%pass); bless $self, $class; foreach (keys %parms, keys %Defaults) { $self->{$_} = exists $parms{$_} && defined $parms{$_} ? $parms{$_} : $Defaults{$_}; } if ($self->{headers}) { $self->_emsg("TE here, headers: ", join(',', @{$self->{headers}}), "\n") if $self->{debug}; $self->{gridmap} = 1; } # Initialize counts and containers $self->_reset_state; $self; } ### HTML::Parser overrides sub start { my $self = shift; my @res; @res = $self->SUPER::start(@_) if TREE(); # Create a new table state if entering a table. if ($_[0] eq 'table') { my $ts = $self->_enter_table(@_); $ts->tree($res[0]) if @res; } elsif ($self->{_in_a_table}) { # Rows and cells are next. my $ts = $self->current_table; if ($_[0] eq 'tr') { $ts->_enter_row; } elsif ($_[0] eq 'td' || $_[0] eq 'th') { $ts->_enter_cell(@_); my %attrs = ref $_[1] ? %{$_[1]} : {}; my $rspan = $attrs{rowspan} || 1; my $cspan = $attrs{colspan} || 1; $ts->_rasterizer->($ts->row_count, $rspan, $cspan); $ts->_anchor_item(@res); } elsif (! TREE() && $ts->{in_cell}) { if ($self->{keep_html}) { # capture full text of tag $self->text($_[3]); } elsif ($_[0] eq 'br' && $self->{br_translate}) { # Replace
with newlines if requested $self->text("\n"); } } } @res; } # end start sub end { my $self = shift; my @res; @res = $self->SUPER::end(@_) if TREE(); if ($self->{_in_a_table}) { my $ts = $self->current_table; if ($_[0] eq 'td' || $_[0] eq 'th') { $ts->_exit_cell; } elsif ($_[0] eq 'tr') { $ts->_exit_row; } elsif ($_[0] eq 'table') { $self->_exit_table; } elsif (! TREE()) { if ($self->{keep_html} && $ts->{in_cell}) { # capture full text of tag $self->text($_[1]); } } } @res; } # end end sub text { my $self = shift; my @res; if (TREE()) { @res = $self->SUPER::text(@_); } elsif ($self->{_in_a_table}) { my $ts = $self->current_table; if ($ts->{in_cell}) { if ($self->{decode} && !$self->{keep_html}) { $ts->_add_text(decode_entities($_[0])); } else { $ts->_add_text($_[0]); } } } @res; } # end text sub parse { my $self = shift; $self->_reset_state unless $self->{_parsing}; $self->{_parsing} ||= 1; $self->SUPER::parse(@_); } sub eof { my $self = shift; $self->{_parsing} = 0; $self->SUPER::eof(@_); } ### End HTML::Parser overrides ### Report Methods sub depths { # Return all depths where valid tables were located. my $self = shift; return () unless ref $self->{_tables}; sort { $a <=> $b } keys %{$self->{_tables}}; } sub counts { # Given a depth, return the counts of all valid tables found therein. my($self, $depth) = @_; defined $depth or croak "Depth required\n"; return () unless exists $self->{_tables}{$depth}; sort { $a <=> $b } keys %{$self->{_tables}{$depth}}; } sub table { # Return the table state for a particular depth and count my($self, $depth, $count) = @_; defined $depth or croak "Depth required\n"; defined $count or croak "Count required\n"; if (! $self->{_tables}{$depth} || ! $self->{_tables}{$depth}{$count}) { return undef; } $self->{_tables}{$depth}{$count}; } sub first_table_found { my $self = shift; ref $self->{_ts_sequential}[0] ? $self->{_ts_sequential}[0] : undef; } sub rows { shift->first_table_found->rows(@_) } sub tables { # Return all valid table records found, in the order that they # were seen. my $self = shift; while ($self->{_in_a_table}) { my $ts = $self->current_table; $self->_emsg("Mangled HTML in table ($ts->{depth},$ts->{count}), inferring closing table tag.\n") if $self->{debug}; $self->_exit_table; } @{$self->{_ts_sequential}}; } # in tree mode, we already are an HTML::TreeBuilder, which is an # HTML::Element structure after parsing...but we provide this for # consistency with the table object method for accessing the tree # structures. sub tree { shift } sub tables_report { # Print out a summary of extracted tables, including depth/count my $self = shift; my $str; foreach my $ts ($self->tables) { $str .= $ts->report(@_); } $str; } sub tables_dump { my $self = shift; $self->_emsg($self->tables_report(@_)); } # for testing/debugging sub _attribute_purge { my $self = shift; foreach (keys %Defaults) { delete $self->{$_}; } } ### Runtime sub _enter_table { my($self, @args) = @_; ++$self->{_cdepth}; ++$self->{_in_a_table}; my $depth = $self->{_cdepth}; # Table tag attributes, if present my $attribs = $args[1] || {}; # Table states can come and go on the stack...here we retrieve the # table state for the table surrounding the current table tag (parent # table state). If the current table tag belongs to a top level table, # then this will be undef. my $pts = $self->current_table; # Counts are tracked for each depth. my $counts = $self->{_counts}; $counts->[$depth] = -1 unless defined $counts->[$depth]; ++$counts->[$depth]; my $count = $counts->[$depth]; $self->_emsg("TABLE: cdepth $depth, ccount $count, it: $self->{_in_a_table}\n") if $self->{debug} >= 2; # Umbrella status means that this current table and all of its # descendant tables will be harvested. my $umbrella = 0; if (! defined $self->{depth} && ! defined $self->{count} && ! $self->{attribs} && ! $self->{headers}) { ++$umbrella; } # Basic parameters for the soon-to-be-created table state. my %tsparms = ( depth => $depth, count => $count, attribs => $attribs, umbrella => $umbrella, automap => $self->{automap}, slice_columns => $self->{slice_columns}, keep_headers => $self->{keep_headers}, counts => $counts, error_handle => $self->{error_handle}, debug => $self->{debug}, keep_html => $self->{keep_html}, strip_html_on_match => $self->{strip_html_on_match}, parent_table => $pts, ); # Target constraints. There is no point in passing any of these along # if we are under an umbrella. Notice that with table states, "depth" # and "count" are absolute coordinates recording where this table was # created, whereas "tdepth" and "tcount" are the target constraints. # Headers have "absolute" meaning, therefore are passed by the # same name. if (!$umbrella) { $tsparms{tdepth} = $self->{depth}; $tsparms{tcount} = $self->{count}; $tsparms{tattribs} = $self->{attribs}; $tsparms{headers} = $self->{headers}; } # Abracadabra my $ts = HTML::TableExtract::Table->new(%tsparms); # Push the newly created and configured table state onto the stack. # This will now be the current_table(). push(@{$self->{_tablestack}}, $ts); $ts; } sub _exit_table { my $self = shift; my $ts = $self->current_table; # Last ditch fix for HTML mangle if ($ts->{in_cell}) { $self->_emsg("Mangled HTML in table ($self->{depth},$self->{count}), forcing exit of cell ($ts->{rc},$ts->{cc}) due to table exit\n") if $self->{debug}; $ts->_exit_cell; } if ($ts->{in_row}) { $self->_emsg("Mangled HTML in table ($self->{depth},$self->{count}), forcing exit of row $ts->{rc} due to table exit\n") if $self->{debug}; $ts->_exit_row; } # transform from tree to grid using our rasterized template $ts->_grid_map(); $self->_capture_table($ts) if $ts->_check_triggers; # Restore last table state pop(@{$self->{_tablestack}}); --$self->{_in_a_table}; my $lts = $self->current_table; if (ref $lts) { $self->{_cdepth} = $lts->{depth}; } else { # Back to the top level $self->{_cdepth} = -1; } $self->_emsg("LEAVE: cdepth: $self->{_cdepth}, ccount: $ts->{count}, it: $self->{_in_a_table}\n") if $self->{debug} >= 2; } sub _capture_table { my($self, $ts, $type) = @_; croak "Table state ref required\n" unless ref $ts; if ($self->{debug} >= 2) { my $msg = "Captured table (" . $ts->depth . ',' . $ts->count . ")"; $msg .= " ($type)" if $type; $msg .= "\n"; $self->_emsg($msg); } $ts->tree(HTML::ElementTable->new_from_tree($ts->tree)) if TREE(); if ($self->{subtables}) { foreach my $child (@{$ts->{children}}) { next if $child->{captured}; $self->_capture_table($child, 'subtable'); $child->{slice_columns} = 0; $child->{keep_headers} = 1; $child->{headers} = ''; } } $ts->{captured} = 1; $self->{_tables}{$ts->{depth}}{$ts->{count}} = $ts; push(@{$self->{_ts_sequential}}, $ts); } sub current_table { my $self = shift; $self->{_tablestack}[$#{$self->{_tablestack}}]; } sub _reset_state { my $self = shift; $self->{_cdepth} = -1; $self->{_tablestack} = []; $self->{_tables} = {}; $self->{_ts_sequential} = []; $self->{_counts} = []; $self->{_in_a_table} = 0; $self->{_parsing} = 0; } sub _emsg { my $self = shift; my $fh = $self->{error_handle}; return unless defined $_[0]; print $fh @_; } ########## { package HTML::TableExtract::Table; use strict; use Carp; *TREE = *HTML::TableExtract::TREE; sub new { my $that = shift; my $class = ref($that) || $that; # Note: # - 'depth' and 'count' are where this table were found. # - 'tdepth' and 'tcount' are target constraints on which to trigger. # - 'headers' represent a target constraint, location independent. # - 'attribs' represent target table tag constraints my $self = { umbrella => 0, in_row => 0, in_cell => 0, rc => -1, cc => -1, grid => [], translation => [], hrow => [], order => [], children => [], captured => 0, debug => 0, }; $self->{_rastamon} = HTML::TableExtract::Rasterize->make_rasterizer(); bless $self, $class; my %parms = @_; # Depth and Count -- this is the absolute address of the table. croak "Absolute depth required\n" unless defined $parms{depth}; croak "Count required\n" unless defined $parms{count}; croak "Counts required\n" unless defined $parms{counts}; foreach (keys %parms) { $self->{$_} = $parms{$_}; } # Register lineage my $pts = $self->{parent_table}; $self->lineage($pts || undef); push(@{$pts->{children}}, $self) if ($pts); delete $self->{parent_table}; $self; } sub _anchor_item { # anchor the reference to a cell in our grid -- in TREE mode this is # a reference to a data element, otherwise it's a reference to an # empty scalar in which we will collect our text. my($self, @res) = @_; my $row = $self->{grid}[-1]; my $item; if (@res && ref $res[0]) { $item = $res[0]; } else { my $scalar_ref; $item = \$scalar_ref; } push(@$row, $item); } sub _gridalias { my $self = shift; $self->{gridalias} ||= $self->_make_gridalias; } sub _grid_map { # using our rasterized template, flesh out our captured items which # are still in 'tree' format my $self = shift; my $template = $self->_rasterizer->(); my $grid = $self->{grid}; # drop empty rows if ($self->{debug}) { foreach (0 .. $#$grid) { next if @{$grid->[$_]}; $self->_emsg("Dropping empty row $_\n"); } } @$grid = grep(@$_, @$grid); foreach my $r (0 .. $#$template) { my $row = $grid->[$r]; my $trow = $template->[$r]; $self->_emsg("Flesh row $r ($#$row) to $#$trow\n") if $self->{debug} > 1; foreach my $c (0 .. $#$trow) { print STDERR $trow->[$c] ? '1' : '0' if $self->{debug} > 1; if ($trow->[$c]) { if (! defined $row->[$c]) { $row->[$c] = \undef; } next; } else { my $scalar; splice(@$row, $c, 0, \$scalar); } } print STDERR "\n" if $self->{debug} > 1; croak "row $r splice mismatch: $#$row vs $#$trow\n" unless $#$row == $#$trow; } $grid; } sub _make_gridalias { # our aliased grid will have references in masked cells to the same # cell that is covering it via spanning. my $self = shift; my $grid = $self->{grid}; my $template = $self->_rasterizer->(); my(@gridalias, @translation); $gridalias[$_] = [@{$grid->[$_]}] foreach 0 .. $#$grid; foreach my $r (0 .. $#gridalias) { my $row = $gridalias[$r]; foreach my $c (0 .. $#$row) { my $tcell = $template->[$r][$c] || next; my($rspan, $cspan) = @$tcell; foreach my $rs (0 .. $rspan-1) { foreach my $cs (0 .. $cspan-1) { $gridalias[$r + $rs][$c + $cs] = $grid->[$r][$c]; $translation[$r + $rs][$c + $cs] = "$r,$c"; } } } } $self->{translation} = \@translation; $self->{gridalias} = \@gridalias; } ### Constraint tests sub _check_dtrigger { # depth my $self = shift; return 1 unless defined $self->{tdepth}; $self->{tdepth} == $self->{depth} ? 1 : 0; } sub _check_ctrigger { # count my $self = shift; return 1 unless defined $self->{tcount}; return 1 if (exists $self->{counts}[$self->{depth}] && $self->{tcount} == $self->{counts}[$self->{depth}]); return 0; } sub _check_atrigger { # attributes my $self = shift; return 1 unless scalar keys %{$self->{tattribs}}; return 0 unless scalar keys %{$self->{attribs}}; my $a_hit = 1; foreach my $attrib (keys %{$self->{tattribs}}) { if (! defined $self->{attribs}{$attrib}) { $a_hit = 0; last; } if (! defined $self->{tattribs}{$attrib}) { # undefined, but existing, target attribs are wildcards next; } if ($self->{tattribs}{$attrib} ne $self->{attribs}{$attrib}) { $a_hit = 0; last; } } $self->_emsg("Matched attributes\n") if $self->{debug} > 3 && $a_hit; $a_hit; } sub _check_htrigger { # headers my $self = shift; return 1 if $self->{umbrella}; return 1 unless $self->{headers}; ROW: foreach my $r (0 .. $#{$self->{grid}}) { $self->_reset_hits; my $hpat = $self->_header_pattern; my @hits; foreach my $c (0 .. $#{$self->{grid}[$r]}) { my $ref = $self->{grid}[$r][$c]; my $target = ''; my $ref_type = ref $ref; if ($ref_type) { if ($ref_type eq 'SCALAR') { my $item = $$ref; if ($self->{keep_html} && $self->{strip_html_on_match}) { my $stripper = HTML::TableExtract::StripHTML->new; $target = $stripper->strip($item); } else { $target = $item; } } else { if (($self->{keep_html} || TREE()) && $self->{strip_html_on_match}) { $target = $ref->as_text; } else { $target = $ref->as_HTML; } } } $target = defined $target ? $target : ''; $self->_emsg("attempt match on $target ($hpat): ") if $self->{debug} >= 5; if ($target =~ $hpat) { my $hit = $1; $self->_emsg("($hit)\n") if $self->{debug} >= 5; # Get rid of the header segment that matched so we can tell # when we're through with all header patterns. my $real_hit; foreach (sort _header_string_sort keys %{$self->{hits_left}}) { if ($hit =~ /$_/im) { delete $self->{hits_left}{$_}; $real_hit = $_; $hpat = $self->_header_pattern; last; } } if (defined $real_hit) { if ($self->{debug} >= 4) { my $str = $ref_type eq 'SCALAR' ? $$ref : $ref->as_HTML; $self->_emsg("HIT on '$hit' ($real_hit) in $str ($r,$c)\n"); } push(@hits, $hit); # $self->{hits}{$c} = $real_hit; push(@{$self->{order}}, $c); if (!%{$self->{hits_left}}) { # Successful header row match ++$self->{head_found}; $self->{hrow_index} = $r; $self->{hrow} = $self->{grid}[$r]; last ROW; } } } elsif ($self->{debug} >= 5) { $self->_emsg("0\n"); } } if ($self->{debug} && @hits) { my $str = "Incomplete header match "; $str .= "(left: " . join(', ', sort keys %{$self->{hits_left}}) . ") "; $str .= "in row $r, resetting scan"; $str .= "\n"; $self->_emsg($str); } } $self->{head_found}; } sub _check_triggers { my $self = shift; return 1 if $self->{umbrella}; $self->_check_dtrigger && $self->_check_ctrigger && $self->_check_atrigger && $self->_check_htrigger; } ### Maintain table context sub _enter_row { my $self = shift; if ($self->{in_row}) { $self->_emsg("Mangled HTML in table ($self->{depth},$self->{count}), forcing exit of row $self->{rc} due to new row\n") if $self->{debug}; $self->_exit_row; } ++$self->{rc}; ++$self->{in_row}; push(@{$self->{grid}}, []) } sub _exit_row { my $self = shift; if ($self->{in_row}) { if ($self->{in_cell}) { $self->_emsg("Mangled HTML in table ($self->{depth},$self->{count}), forcing exit of cell ($self->{rc}, $self->{cc}) due to new row\n") if $self->{debug}; $self->_exit_cell; } $self->{in_row} = 0; $self->{cc} = -1; } else { $self->_emsg("Mangled HTML in table ($self->{depth},$self->{count}), extraneous ignored after row $self->{rc}\n") if $self->{debug}; } } sub _enter_cell { my $self = shift; if ($self->{in_cell}) { $self->_emsg("Mangled HTML in table ($self->{depth},$self->{count}), forcing exit of cell ($self->{rc},$self->{cc}) due to new cell\n") if $self->{debug}; $self->_exit_cell; } if (!$self->{in_row}) { # Go ahead and try to recover from mangled HTML, because we care. $self->_emsg("Mangled HTML in table ($self->{depth},$self->{count}), inferring as row $self->{rc}\n") if $self->{debug}; $self->_enter_row; } ++$self->{cc}; ++$self->{in_cell}; my %attrs = ref $_[1] ? %{$_[1]} : {}; my $rspan = $attrs{rowspan} || 1; my $cspan = $attrs{colspan} || 1; } sub _exit_cell { my $self = shift; if ($self->{in_cell}) { $self->{in_cell} = 0; } else { $self->_emsg("Mangled HTML in table ($self->{depth},$self->{count}), extraneous ignored in row $self->{rc}\n") if $self->{debug}; } } # Header stuff sub _header_pattern { my($self, @headers) = @_; my $str = join('|', map("($_)", sort _header_string_sort keys %{$self->{hits_left}} )); my $hpat = qr/($str)/im; $self->_emsg("HPAT: /$hpat/\n") if $self->{debug} >= 2; $self->{hpat} = $hpat; } sub _header_string_sort { # this ensures that supersets appear before subsets in our header # search pattern, eg, '10' appears before '1' and 'hubbabubba' # appears before 'hubba'. if ($a =~ /^$b/) { return -1; } elsif ($b =~ /^$a/) { return 1; } else { return $b cmp $a; } } # Report methods sub depth { shift->{depth} } sub count { shift->{count} } sub coords { my $self = shift; ($self->depth, $self->count); } sub row_count { shift->{rc} } sub col_count { shift->{cc} } sub tree { my $self = shift; @_ ? $self->{_tree_ref} = shift : $self->{_tree_ref}; } sub lineage { my $self = shift; $self->{lineage} ||= []; if (@_) { my $pts = shift; my(@lineage, $pcoords); if ($pts) { foreach my $pcoord ($pts->lineage) { push(@lineage, [@$pcoord]); } $pcoords = [$pts->depth, $pts->count, $pts->{rc}, $pts->{cc}]; push(@lineage, $pcoords); } $self->{lineage} = \@lineage; } @{$self->{lineage}}; } sub rows { shift->_rows(0) } sub space_rows { my $self = shift; $self->_rows(1); } sub _rows { my $self = shift; my $alias = shift; my @ri = $self->row_indices; my @rows; my $grid = $alias ? $self->_gridalias : $self->{grid}; foreach ($self->row_indices) { push(@rows, scalar $self->_slice_and_normalize_row($grid->[$_])); } wantarray ? @rows : \@rows; } sub columns { my $self = shift; my @cols; my @rows = $self->rows; foreach my $row (@rows) { foreach my $c (0 .. $#$row) { $cols[$c] ||= []; push(@{$cols[$c]}, $row->[$c]); } } @cols; } sub row_indices { my $self = shift; my $start_index = 0; if ($self->{headers}) { $start_index = $self->hrow_index; $start_index += 1 unless $self->{keep_headers}; } $start_index .. $#{$self->{grid}}; } sub col_indices { my $self = shift; my $row = $self->{grid}[0]; 0 .. $#$row; } sub row { my $self = shift; my $r = shift; $r <= $#{$self->{grid}} or croak "row $r out of range ($#{$self->{grid}})\n"; my @row = $self->_slice_and_normalize_row( $self->{grid}[($self->row_indices)[$r]] ); wantarray ? @row : \@row; } sub _slice_and_normalize_row { my $self = shift; my $rowref = shift; my @row; if ($self->{automap} && $self->_map_makes_a_difference) { @row = @{$rowref}[$self->column_map]; } else { @row = @$rowref; } @row = map($self->_cell_to_content($_), @row); wantarray ? @row : \@row; } sub column { my $self = shift; my $c = shift; my @column; foreach my $row ($self->rows) { push(@column, $self->cell($row, $c)); } wantarray ? @column : \@column; } sub cell { my $self = shift; my($r, $c) = @_; my $row = $self->row($r); $c <= $#$row or croak "Column $c out of range ($#$row)\n"; $self->_cell_to_content($row->[$c]); } sub _cell_to_content { my $self = shift; @_ or croak "cell item required\n"; my $cell = shift; return $cell unless ref $cell; return $cell if TREE(); return $$cell; } sub space { my $self = shift; my($r, $c) = @_; my $gridalias = $self->_gridalias; $r <= $#$gridalias or croak "row $r out of range ($#$gridalias)\n"; my $row = $gridalias->[$r]; $c <= $#$row or croak "Column $c out of range ($#$row)\n"; $self->_cell_to_content($row->[$c]); } sub source_coords { my $self = shift; my($r, $c) = @_; $r <= $#{$self->{translation}} or croak "row $r out of range ($#{$self->{translation}})\n"; my $row = $self->{translation}[$r]; $c <= $#$row or croak "Column $c out of range ($#$row)\n"; split(/,/, $self->{translation}[$r][$c]); } sub hrow_index { my $self = shift; $self->{hrow_index}; } sub hrow { my $self = shift; if ($self->{automap} && $self->_map_makes_a_difference) { return map(ref $_ ? $$_ : $_, @{$self->{hrow}}[$self->column_map]); } else { return map(ref $_ ? $$_ : $_, @{$self->{hrow}}); } } sub column_map { # Return the column numbers of this table in the same order as the # provided headers. my $self = shift; if ($self->{headers}) { # First we order the original column counts by taking a hash slice # based on the original header order. The resulting original # column numbers are mapped to the actual content indices since # we could have a sparse slice. my %order; foreach (keys %{$self->{hits}}) { $order{$self->{hits}{$_}} = $_; } return @order{@{$self->{headers}}}; } else { return 0 .. $#{$self->{grid}[0]}; } } sub _map_makes_a_difference { my $self = shift; return 0 unless $self->{slice_columns}; my $diff = 0; my @order = $self->column_map; my @sorder = sort { $a <=> $b } @order; ++$diff if $#order != $#sorder; ++$diff if $#sorder != $#{$self->{grid}[0]}; foreach (0 .. $#order) { if ($order[$_] != $sorder[$_]) { ++$diff; last; } } $diff; } sub _add_text { my($self, $txt) = @_; my $r = $self->{rc}; my $c = $self->{cc}; my $row = $self->{grid}[$r]; ${$row->[$c]} .= $txt; $txt; } sub _reset_hits { my $self = shift; return unless $self->{headers}; $self->{hits} = {}; $self->{order} = []; foreach (@{$self->{headers}}) { ++$self->{hits_left}{$_}; } 1; } sub _rasterizer { shift->{_rastamon} } sub report { # Print out a summary of this table, including depth/count my($self, $include_content, $col_sep) = @_; $col_sep ||= ':'; my $str; $str .= "TABLE(" . $self->depth . ", " . $self->count . ')'; if ($include_content) { $str .= ":\n"; foreach my $row ($self->rows) { $str .= join($col_sep, @$row) . "\n"; } } else { $str .= "\n"; } $str; } sub dump { my $self = shift; $self->_emsg($self->report(@_)); } sub _emsg { my $self = shift; my $fh = $self->{error_handle}; print $fh @_; } } ########## { package HTML::TableExtract::Rasterize; # Provide a closure that will rasterize (turn into a grid) a table # from a tree structure based on repeated data element calls with # rowspan and colspan information. Not as straight forward as it # seems...see test cases for an example bugaboo. my $DEBUG = 0; sub make_rasterizer { my $pkg = shift; my(@grid, @row_spinner, @col_spinner); my $empty_row_offset = 0; sub { return \@grid unless @_; my($row_num, $rspan, $cspan) = @_; $rspan = 1 unless $rspan > 1; $cspan = 1 unless $cspan > 1; my($rspin_propogate, $row_added); my $trigger = $#grid + $empty_row_offset; if ($row_num > $trigger) { # adjust for having been handed a row that skips a prior row, # otherwise the next cell will land in a wrong row. Hopefully # this doesn't happen too often but I've seen it in the wild! if ($row_num - $trigger > 1) { $empty_row_offset += $row_num - $trigger - 1; } # add new row $row_added = 1; my @new_row; # first add new row spinner if ($row_spinner[-1] && $col_spinner[-1]) { push(@row_spinner, $row_spinner[-1]); $rspin_propogate = 1; } else { push(@row_spinner, $cspan - 1); } # spin columns foreach (@col_spinner) { if ($_) { push(@new_row, 0); --$_; } else { push(@new_row, undef); } } @new_row = (undef) unless @new_row; push(@grid, \@new_row); } my $current_row = $grid[-1]; # locate next available cell in row my $col; foreach my $ci (0 .. $#$current_row) { if (! defined $current_row->[$ci]) { $col = $ci; last; } } if (! defined $col) { ADDCOL: while (! defined $col) { # if no cells were available, add a column foreach my $ri (0 .. $#grid) { my $row = $grid[$ri]; my $cspan_count = $row_spinner[$ri]; if (!$cspan_count) { push(@$row, undef); } else { push(@$row, 0); --$row_spinner[$ri]; } } push(@col_spinner, $col_spinner[-1]); foreach my $ci (0 .. $#$current_row) { if (! defined $current_row->[$ci]) { $col = $ci; last ADDCOL; } } } $col_spinner[-1] = $rspan - 1 if $col == $#$current_row; $row_spinner[$#grid] = $cspan - 1; } # we now have correct coordinates for this element $current_row->[$col] = [$rspan, $cspan]; $col_spinner[$col] = $rspan - 1; # if this is an embedded placement (not a trailing element), use up # the cspan if ($col < $#$current_row) { my $offset = 1; my $row_span = $col_spinner[$col]; if ($col + $row_spinner[-1] < $#$current_row && $row_added && !$rspin_propogate) { # cell is spun out -- clear spinner unless it inherited cspan # from a cell above $row_spinner[-1] = 0; } while ($offset < $cspan) { my $cursor = $col + $offset; $current_row->[$cursor] = 0; $col_spinner[$cursor] = $row_span; ++$offset; if ($col + $offset > $#$current_row) { $row_spinner[-1] = $cspan - $offset; last; } } } if ($DEBUG) { foreach my $r (0 .. $#grid) { my $row = $grid[$r]; foreach my $c (0 .. $#$row) { if (defined $row->[$c]) { print STDERR $row->[$c] ? 1 : 0; } else { print STDERR '?'; } } print STDERR " $row_spinner[$r]\n"; } print STDERR "\n"; foreach (@col_spinner) { print STDERR defined $_ ? $_ : '?'; } print STDERR "\n\n-----\n\n"; } return \@grid; } } } ########## { package HTML::TableExtract::StripHTML; use vars qw(@ISA); use HTML::Parser; @ISA = qw(HTML::Parser); sub tag { my($self, $tag, $num) = @_; $self->{_htes_inside}{$tag} += $num; } sub text { my $self = shift; return if $self->{_htes_inside}{script} || $self->{_htes_inside}{style}; $self->{_htes_tidbit} .= $_[0]; } sub new { my $class = shift; my $self = HTML::Parser->new( api_version => 3, handlers => [start => [\&tag, "self, tagname, '+1'"], end => [\&tag, "self, tagname, '-1'"], text => [\&text, "self, dtext"], ], marked_sections => 1, ); bless $self, $class; } sub strip { my $self = shift; $self->parse(shift); $self->eof; $self->{_htes_tidbit}; } } 1; __END__ =head1 NAME HTML::TableExtract - Perl module for extracting the content contained in tables within an HTML document, either as text or encoded element trees. =head1 SYNOPSIS # Matched tables are returned as table objects; tables can be matched # using column headers, depth, count within a depth, table tag # attributes, or some combination of the four. # Example: Using column header information. # Assume an HTML document with tables that have "Date", "Price", and # "Cost" somewhere in a row. The columns beneath those headings are # what you want to extract. They will be returned in the same order as # you specified the headers since 'automap' is enabled by default. use HTML::TableExtract; $te = HTML::TableExtract->new( headers => [qw(Date Price Cost)] ); $te->parse($html_string); # Examine all matching tables foreach $ts ($te->tables) { print "Table (", join(',', $ts->coords), "):\n"; foreach $row ($ts->rows) { print join(',', @$row), "\n"; } } # Shorthand...top level rows() method assumes the first table found in # the document if no arguments are supplied. foreach $row ($te->rows) { print join(',', @$row), "\n"; } # Example: Using depth and count information. # Every table in the document has a unique depth and count tuple, so # when both are specified it is a unique table. Depth and count both # begin with 0, so in this case we are looking for a table (depth 2) # within a table (depth 1) within a table (depth 0, which is the top # level HTML document). In addition, it must be the third (count 2) # such instance of a table at that depth. $te = HTML::TableExtract->new( depth => 2, count => 2 ); $te->parse_file($html_file); foreach $ts ($te->tables) { print "Table found at ", join(',', $ts->coords), ":\n"; foreach $row ($ts->rows) { print " ", join(',', @$row), "\n"; } } # Example: Using table tag attributes. # If multiple attributes are specified, all must be present and equal # for match to occur. $te = HTML::TableExtract->new( attribs => { border => 1 } ); $te->parse($html_string); foreach $ts ($te->tables) { print "Table with border=1 found at ", join(',', $ts->coords), ":\n"; foreach $row ($ts->rows) { print " ", join(',', @$row), "\n"; } } # Example: Extracting as an HTML::Element tree structure # Rather than extracting raw text, the html can be converted into a # tree of element objects. The HTML document is composed of # HTML::Element objects and the tables are HTML::ElementTable # structures. Using this, the contents of tables within a document can # be edited in-place. use HTML::TableExtract qw(tree); $te = HTML::TableExtract->new( headers => qw(Fee Fie Foe Fum) ); $te->parse_file($html_file); $table = $te->first_table_found; $table_tree = $table->tree; $table_tree->cell(4,4)->replace_content('Golden Goose'); $table_html = $table_tree->as_HTML; $table_text = $table_tree->as_text; $document_tree = $te->tree; $document_html = $document_tree->as_HTML; =head1 DESCRIPTION HTML::TableExtract is a subclass of HTML::Parser that serves to extract the information from tables of interest contained within an HTML document. The information from each extracted table is stored in table objects. Tables can be extracted as text, HTML, or HTML::ElementTable structures (for in-place editing or manipulation). There are currently four constraints available to specify which tables you would like to extract from a document: I, I, I, and I. I, the most flexible and adaptive of the techniques, involves specifying text in an array that you expect to appear above the data in the tables of interest. Once all headers have been located in a row of that table, all further cells beneath the columns that matched your headers are extracted. All other columns are ignored: think of it as vertical slices through a table. In addition, TableExtract automatically rearranges each row in the same order as the headers you provided. If you would like to disable this, set I to 0 during object creation, and instead rely on the column_map() method to find out the order in which the headers were found. Furthermore, TableExtract will automatically compensate for cell span issues so that columns are really the same columns as you would visually see in a browser. This behavior can be disabled by setting the I parameter to 0. HTML is stripped from the entire textual content of a cell before header matches are attempted -- unless the I parameter was enabled. I and I are more specific ways to specify tables in relation to one another. I represents how deeply a table resides in other tables. The depth of a top-level table in the document is 0. A table within a top-level table has a depth of 1, and so on. Each depth can be thought of as a layer; tables sharing the same depth are on the same layer. Within each of these layers, I represents the order in which a table was seen at that depth, starting with 0. Providing both a I and a I will uniquely specify a table within a document. I match based on the attributes of the html EtableE tag, for example, border widths or background color. Each of the I, I, I, and I specifications are cumulative in their effect on the overall extraction. For instance, if you specify only a I, then you get all tables at that depth (note that these could very well reside in separate higher- level tables throughout the document since depth extends across tables). If you specify only a I, then the tables at that I from all depths are returned (i.e., the Ith occurrence of a table at each depth). If you only specify I, then you get all tables in the document containing those column headers. If you have specified multiple constraints of I, I, I, and I, then each constraint has veto power over whether a particular table is extracted. If no I, I, I, or I are specified, then all tables match. When extracting only text from tables, the text is decoded with HTML::Entities by default; this can be disabled by setting the I parameter to 0. =head2 Extraction Modes The default mode of extraction for HTML::TableExtract is raw text or HTML. In this mode, embedded tables are completely decoupled from one another. In this case, HTML::TableExtract is a subclass of HTML::Parser: use HTML::TableExtract; Alternatively, tables can be extracted as HTML::ElementTable structures, which are in turn embedded in an HTML::Element tree representing the entire HTML document. Embedded tables are not decoupled from one another since this tree structure must be maintained. In this case, HTML::TableExtract is a subclass of HTML::TreeBuilder (itself a subclass of HTML:::Parser): use HTML::TableExtract qw(tree); In either case, the basic interface for HTML::TableExtract and the resulting table objects remains the same -- all that changes is what you can do with the resulting data. HTML::TableExtract is a subclass of HTML::Parser, and as such inherits all of its basic methods such as C and C. During scans, C, C, and C are utilized. Feel free to override them, but if you do not eventually invoke them in the SUPER class with some content, results are not guaranteed. =head2 Advice The main point of this module was to provide a flexible method of extracting tabular information from HTML documents without relying to heavily on the document layout. For that reason, I suggest using I whenever possible -- that way, you are anchoring your extraction on what the document is trying to communicate rather than some feature of the HTML comprising the document (other than the fact that the data is contained in a table). =head1 METHODS The following are the top-level methods of the HTML::TableExtract object. Tables that have matched a query are actually returned as separate objects of type HTML::TableExtract::Table. These table objects have their own methods, documented further below. =head2 CONSTRUCTOR =over =item new() Return a new HTML::TableExtract object. Valid attributes are: =over =item headers Passed as an array reference, headers specify strings of interest at the top of columns within targeted tables. They can be either strings or regular expressions (qr//). If they are strings, they will eventually be passed through a non-anchored, case-insensitive regular expression, so regexp special characters are allowed. The table row containing the headers is B returned, unless C was specified or you are extracting into an element tree. In either case the header row can be accessed via the hrow() method from within the table object. Columns that are not beneath one of the provided headers will be ignored unless C was set to 0. Columns will, by default, be rearranged into the same order as the headers you provide (see the I parameter for more information) I C is 0. Additionally, by default columns are considered what you would see visually beneath that header when the table is rendered in a browser. See the C parameter for more information. HTML within a header is stripped before the match is attempted, unless the C parameter was specified and C is false. =item depth Specify how embedded in other tables your tables of interest should be. Top-level tables in the HTML document have a depth of 0, tables within top-level tables have a depth of 1, and so on. =item count Specify which table within each depth you are interested in, beginning with 0. =item attribs Passed as a hash reference, attribs specify attributes of interest within the HTML EtableE tag itself. =item automap Automatically applies the ordering reported by column_map() to the rows returned by rows(). This only makes a difference if you have specified I and they turn out to be in a different order in the table than what you specified. Automap will rearrange the columns in the same order as the headers appear. To get the original ordering, you will need to take another slice of each row using column_map(). I is enabled by default. =item slice_columns Enabled by default, this option controls whether vertical slices are returned from under headers that match. When disabled, all columns of the matching table are retained, regardles of whether they had a matching header above them. Disabling this also disables C. =item keep_headers Disabled by default, and only applicable when header constraints have been specified, C will retain the matching header row as the first row of table data when enabled. This option has no effect if extracting into an element tree structure. In any case, the header row is accessible from the table method C. =item gridmap Controls whether the table contents are returned as a grid or a tree. ROWSPAN and COLSPAN issues are compensated for, and columns really are columns. Empty phantom cells are created where they would have been obscured by ROWSPAN or COLSPAN settings. This really becomes an issue when extracting columns beneath headers. Enabled by default. =item subtables Extract all tables embedded within matched tables. =item decode Automatically decode retrieved text with HTML::Entities::decode_entities(). Enabled by default. Has no effect if C was specified or if extracting into an element tree structure. =item br_translate Translate
tags into newlines. Sometimes the remaining text can be hard to parse if the
tag is simply dropped. Enabled by default. Has no effect if I is enabled or if extracting into an element tree structure. =item keep_html Return the raw HTML contained in the cell, rather than just the visible text. Embedded tables are B retained in the HTML extracted from a cell. Patterns for header matches must take into account HTML in the string if this option is enabled. This option has no effect if extracting into an elment tree structure. =item strip_html_on_match When C is enabled, HTML is stripped by default during attempts at matching header strings (so if C is not enabled and C is, you would have to include potential HTML tags in the regexp for header matches). Stripped header tags are replaced with an empty string, e.g. 'hot dEemEogE/emE' would become 'hot dog' before attempting a match. =item error_handle Filehandle where error messages are printed. STDERR by default. =item debug Prints some debugging information to STDERR, more for higher values. If C was provided, messages are printed there rather than STDERR. =back =back =head2 REGULAR METHODS The following methods are invoked directly from an HTML::TableExtract object. =over =item depths() Returns all depths that contained matched tables in the document. =item counts($depth) For a particular depth, returns all counts that contained matched tables. =item table($depth, $count) For a particular depth and count, return the table object for the table found, if any. =item tables() Return table objects for all tables that matched. Returns an empty list if no tables matched. =item first_table_found() Return the table state object for the first table matched in the document. Returns undef if no tables were matched. =item current_table() Returns the current table object while parsing the HTML. Only useful if you're messing around with overriding HTML::Parser methods. =item tree() If the module was invoked in tree extraction mode, returns a reference to the top node of the HTML::Element tree structure for the entire document (which includes, ultimately, all tables within the document). =item tables_report([$show_content, $col_sep]) Return a string summarizing extracted tables, along with their depth and count. Optionally takes a C<$show_content> flag which will dump the extracted contents of each table as well with columns separated by C<$col_sep>. Default C<$col_sep> is ':'. =item tables_dump([$show_content, $col_sep]) Same as C except dump the information to STDOUT. =item start =item end =item text These are the hooks into HTML::Parser. If you want to subclass this module and have things work, you must at some point call these with content. =back =head2 DEPRECATED METHODS Tables used to be called 'table states'. Accordingly, the following methods still work but have been deprecated: =over =item table_state() Is now table() =item table_states() Is now tables() =item first_table_state_found() Is now first_table_found() =back =head2 TABLE METHODS The following methods are invoked from an HTML::TableExtract::Table object, such as those returned from the C method. =over =item rows() Return all rows within a matched table. Each row returned is a reference to an array containing the text, HTML, or reference to the HTML::Element object of each cell depending the mode of extraction. Tables with rowspan or colspan attributes will have some cells containing undef. Returns a list or a reference to an array depending on context. =item columns() Return all columns within a matched table. Each column returned is a reference to an array containing the text, HTML, or reference to HTML::Element object of each cell depending on the mode of extraction. Tables with rowspan or colspan attributes will have some cells containing undef. =item row($row) Return a particular row from within a matched table either as a list or an array reference, depending on context. =item column($col) Return a particular column from within a matched table as a list or an array reference, depending on context. =item cell($row,$col) Return a particular item from within a matched table, whether it be the text, HTML, or reference to the HTML::Element object of that cell, depending on the mode of extraction. If the cell was covered due to rowspan or colspan effects, will return undef. =item space($row,$col) The same as cell(), except in cases where the given coordinates were covered due to rowspan or colspan issues, in which case the content of the covering cell is returned rather than undef. =item depth() Return the depth at which this table was found. =item count() Return the count for this table within the depth it was found. =item coords() Return depth and count in a list. =item tree() If the module was invoked in tree extraction mode, this accessor provides a reference to the HTML::ElementTable structure encompassing the table. =item hrow() Returns the header row as a list when headers were specified as a constraint. If C was specified initially, this is equivalent to the first row returned by the C method. =item column_map() Return the order (via indices) in which the provided headers were found. These indices can be used as slices on rows to either order the rows in the same order as headers or restore the rows to their natural order, depending on whether the rows have been pre-adjusted using the I parameter. =item lineage() Returns the path of matched tables that led to matching this table. The path is a list of array refs containing depth, count, row, and column values for each ancestor table involved. Note that corresponding table objects will not exist for ancestral tables that did not match specified constraints. =back =head1 NOTES ON TREE EXTRACTION MODE As mentioned above, HTML::TableExtract can be invoked in 'tree' mode where the resulting HTML and extracted tables are encoded in HTML::Element tree structures: use HTML::TableExtract 'tree'; There are a number of things to take note of while using this mode. The entire HTML document is encoded into an HTML::Element tree. Each table is part of this structure, but nevertheless is tracked separately via an HTML::ElementTable structure, which is a specialized form of HTML::Element tree. The HTML::ElementTable objects are accessible by invoking the tree() method from within each table object returned by HTML::TableExtract. The HTML::ElementTable objects have their own row(), col(), and cell() methods (among others). These are not to be confused with the row() and column() methods provided by the HTML::TableExtract::Table objects. For example, the row() method from HTML::ElementTable will provide a reference to a 'glob' of all the elements in that row. Actions (such as setting attributes) performed on that row reference will affect all elements within that row. On the other hand, the row() method from the HTML::TableExtract::Table object will return an array (either by reference or list, depending on context) of the contents of each cell within the row. In tree mode, the content is represented by individual references to each cell -- these are references to the same HTML::Element objects that reside in the HTML::Element tree. The cell() methods provided in both cases will therefore return references to the same object. The exception to this is when a 'cell' in the table grid was originally 'covered' due to rowspan or colspan issues -- in this case the cell content will be undef. Likewise, the row() or column() methods from HTML::TableExtract::Table objects will return arrays potentially containing a mixture of object references and undefs. If you're going to be doing lots of manipulation of the table elements, it might be more efficient to access them via the methods provided by the HTML::ElementTable object instead. See L for more information on how to manipulate those objects. An alternative to the cell() method in HTML::TableExtract::Table is the space() method. It is largely similar to cell(), except when given coordinates of a cell that was covered due to rowspan or colspan effects, it will return the contents of the cell that was covering that space rather than undef. So if, for example, cell (0,0) had a rowspan of 2 and colspan of 2, cell(1,1) would return undef and space(1,1) would return the same content as cell(0,0) or space(0,0). =head1 REQUIRES HTML::Parser(3), HTML::Entities(3) =head1 OPTIONALLY REQUIRES HTML::TreeBuilder(3), HTML::ElementTable(3) =head1 AUTHOR Matthew P. Sisk, EFE =head1 COPYRIGHT Copyright (c) 2000-2015 Matthew P. Sisk. All rights reserved. All wrongs revenged. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. =head1 SEE ALSO HTML::Parser(3), HTML::TreeBuilder(3), HTML::ElementTable(3), perl(1). =cut In honor of fragmented markup languages and sugar mining: The Good and The Bad Ted Hawkins (1936-1994) Living is good when you have someone to share it with Laughter is bad when there is no one there to share it with Talking is sad if you've got no one to talk to Dying is good when the one you love grows tired of you Sugar is no good once it's cast among the white sand What the point in pulling the gray hairs from among the black strands When you're old you shouldn't walk in the fast lane Oh ain't it useless to keep trying to draw true love from that man He'll hurt you, Yes just for the sake of hurting you and he'll hate you if you try to love him just the same He'll use you and everything you have to offer him On your way girl Get out and find you someone new