XML-TokeParser-0.05/ 0040777 0001040 0001001 00000000000 07671057452 014641 5 ustar Administrators None XML-TokeParser-0.05/Changes 0100777 0001040 0001001 00000002506 07671053015 016126 0 ustar Administrators None Revision history for Perl extension XML::TokeParser.
0.01 Tue Jan 16 17:59:21 2001
- original version; created by h2xs 1.19
0.02 Mon Jan 29 21:20:00 2001
- fixed bug in unget_token
0.03 Fri Jun 01 06:30:00 2001
- unget_token can now handle partial tokens returned by get_tag
- added begin_saving and restore_saved methods
- fixed bug causing warning when reaching the end of XML passed
in as string reference
- fixed bug preventing Latin conversion for tag and attribute names
0.04 Sat Jul 20 00:10:00 2001
- fixed bugs causing warnings under some circumstances
0.05 Sun Jun 8 08:08:25 2003
- PODMASTER takes over maintenance (with original authors "blessing")
and makes all tokens of type XML::TokeParser::Token
with the following methods:
is_text
is_comment
is_pi which is short for is_process_instruction
is_start_tag
is_end_tag
is_tag
target
data
raw
attr
attrseq
tag
text
** WARNING: The tokens will probably change
as per http://perlmonks.com/index.pl?node_id=264094
which shouldn't affect how you use them.
- more tests, more pod, added TODO
XML-TokeParser-0.05/Makefile.PL 0100777 0001040 0001001 00000000774 07671051246 016616 0 ustar Administrators None use ExtUtils::MakeMaker;
# See lib/ExtUtils/MakeMaker.pm for details of how to influence
# the contents of the Makefile that is written.
my $file = 'TokeParser.pm';
WriteMakefile(
'NAME' => 'XML::TokeParser',
'VERSION_FROM' => $file,
'PREREQ_PM' => {
'XML::Parser' => 2,
},
($] >= 5.005 ? ## Add these new keywords supported since 5.005
(ABSTRACT_FROM => $file, # retrieve abstract from module
AUTHOR => 'D.H. aka PodMaster'
) : ()),
);
XML-TokeParser-0.05/MANIFEST 0100777 0001040 0001001 00000000272 07671051363 015766 0 ustar Administrators None Changes
Makefile.PL
MANIFEST
TokeParser.pm
README
TODO
TokeParser.xml
t/1.normal.t
t/2.extended.t
META.yml Module meta-data (added by MakeMaker)
XML-TokeParser-0.05/META.yml 0100777 0001040 0001001 00000000455 07671057452 016116 0 ustar Administrators None #XXXXXXX This is a prototype!!! It will change in the future!!! XXXXX#
name: XML-TokeParser
version: 0.05
version_from: TokeParser.pm
installdirs: site
requires:
XML::Parser: 2
distribution_type: module
generated_by: ExtUtils::MakeMaker version 6.10_06
XML-TokeParser-0.05/README 0100777 0001040 0001001 00000031217 07671053236 015521 0 ustar Administrators None XML/TokeParser version 0.05
=======================
INSTALLATION
To install this module type the following:
perl Makefile.PL
make
make test
make install
NAME
XML::TokeParser - Simplified interface to XML::Parser
SYNOPSIS
use XML::TokeParser;
#
#parse from file
my $p = XML::TokeParser->new('file.xml')
#
#parse from open handle
open IN, 'file.xml' or die $!;
my $p = XML::TokeParser->new( \*IN, Noempty => 1 );
#
#parse literal text
my $text = 'text';
my $p = XML::TokeParser->new( \$text, Namespaces => 1 );
#
#read next token
my $token = $p->get_token();
#
#skip to
and read text
$p->get_tag('title');
$p->get_text();
#
#read text of next , ignoring any internal markup
$p->get_tag('para');
$p->get_trimmed_text('/para');
#
#process if interesting text
$t = $p->get_tag('para');
$p->begin_saving($t);
if ( $p->get_trimmed_text('/para') =~ /interesting stuff/ ) {
$p->restore_saved();
process_para($p);
}
DESCRIPTION
XML::TokeParser provides a procedural ("pull mode") interface to
XML::Parser in much the same way that Gisle Aas' HTML::TokeParser
provides a procedural interface to HTML::Parser. XML::TokeParser splits
its XML input up into "tokens," each corresponding to an XML::Parser
event.
A token is a bless'd reference to an array whose first element is an
event-type string and whose last element is the literal text of the XML
input that generated the event, with intermediate elements varying
according to the event type.
Each token is an *object* of type XML::TokeParser::Token. Read
"XML::TokeParser::Token" to learn what methods are available for
inspecting the token, and retrieving data from it.
METHODS
$p = XML::TokeParser->new($input, [options])
Creates a new parser, specifying the input source and any options.
If $input is a string, it is the name of the file to parse. If
$input is a reference to a string, that string is the actual text to
parse. If $input is a reference to a typeglob or an IO::Handle
object corresponding to an open file or socket, the text read from
the handle will be parsed.
Options are name=>value pairs and can be any of the following:
Namespaces
If set to a true value, namespace processing is enabled.
ParseParamEnt
This option is passed on to the underlying XML::Parser object;
see that module's documentation for details.
Noempty
If set to a true value, text tokens consisting of only
whitespace (such as those created by indentation and line breaks
in between tags) will be ignored.
Latin
If set to a true value, all text other than the literal text
elements of tokens will be translated into the ISO 8859-1
(Latin-1) character encoding rather than the normal UTF-8
encoding.
Catalog
The value is the URI of a catalog file used to resolve PUBLIC
and SYSTEM identifiers. See XML::Catalog for details.
$token = $p->get_token()
Returns the next token, as an array reference, from the input.
Returns undef if there are no remaining tokens.
$p->unget_token($token,...)
Pushes tokens back so they will be re-read. Useful if you've read
one or more tokens too far. Correctly handles "partial" tokens
returned by get_tag().
$token = $p->get_tag( [$token] )
If no argument given, skips tokens until the next start tag or end
tag token. If an argument is given, skips tokens until the start tag
or end tag (if the argument begins with '/') for the named element.
The returned token does not include an event type code; its first
element is the element name, prefixed by a '/' if the token is for
an end tag.
$text = $p->get_text( [$token] )
If no argument given, returns the text at the current position, or
an empty string if the next token is not a 'T' token. If an argument
is given, gathers up all text between the current position and the
specified start or end tag, stripping out any intervening tags (much
like the way a typical Web browser deals with unknown tags).
$text = $p->get_trimmed_text( [$token] )
Like get_text(), but deletes any leading or trailing whitespaces and
collapses multiple whitespace (including newlines) into single
spaces.
$p->begin_saving( [$token] )
Causes subsequent calls to get_token(), get_tag(), get_text(), and
get_trimmed_text() to save the returned tokens. In conjunction with
restore_saved(), allows you to "back up" within a token stream. If
an argument is supplied, it is placed at the beginning of the list
of saved tokens (useful because you often won't know you want to
begin saving until you've already read the first token you want
saved).
$p->restore_saved()
Pushes all the tokens saved by begin_saving() back onto the token
stream. Stops saving tokens. To cancel saving without backing up,
call begin_saving() and restore_saved() in succession.
XML::TokeParser::Token
A token is a blessed array reference, that you acquire using
"$p->get_token" or "$p->get_tag", and that might look like:
["S", $tag, $attr, $attrseq, $raw]
["E", $tag, $raw]
["T", $text, $raw]
["C", $text, $raw]
["PI", $target, $data, $raw]
If you don't like remembering array indices (you're a real programmer),
you may access the attributes of a token like:
"$t->tag", "$t->attr", "$t->attrseq", "$t->raw", "$t->text",
"$t->target", "$t->data".
****Please note that this may change in the future, where as there will
be 4 token types, XML::TokeParser::Token::StartTag ....
What kind of token is it?
To find out, inspect your token using any of these is_* methods (1 ==
true, 0 == false, d'oh):
is_text
is_comment
is_pi which is short for is_process_instruction
is_start_tag
is_end_tag
is_tag
What's that token made of? To retrieve data from your token, use any of
the following methods, depending on the kind of token you have:
target
only for process instructions
data
only for process instructions
raw for all tokens
attr
only for start tags, returns a hashref ( "print "#link ",
""$t->attr""->{href}" ).
my $attrseq = $t->attrseq
only for start tags, returns an array ref of the keys found in
"$t->attr" in the order they originally appeared in.
my $tagname = $t->tag
only for tags ( "print "opening ", ""$t->tag"" if
""$t->is_start_tag" ).
my $text = $token->text
only for tokens of type text and comment
Here's more detailed info about the tokens.
Start tag
The token has five elements: 'S', the element's name, a reference to
a hash of attribute values keyed by attribute names, a reference to
an array of attribute names in the order in which they appeared in
the tag, and the literal text.
End tag
The token has three elements: 'E', the element's name, and the
literal text.
Character data (text)
The token has three elements: 'T', the parsed text, and the literal
text. All contiguous runs of text are gathered into single tokens;
there will never be two 'T' tokens in a row.
Comment
The token has three elements: 'C', the parsed text of the comment,
and the literal text.
Processing instruction
The token has four elements: 'PI', the target, the data, and the
literal text.
The literal text includes any markup delimiters (pointy brackets,
some text ';
my $p = XML::TokeParser->new( \$text );
#
print $/;
#
while( defined( my $t = $p->get_token() ) ){
local $\="\n";
print ' raw = ', $t->raw;
#
if( $t->tag ){
print ' tag = ', $t->tag;
#
if( $t->is_start_tag ) {
print ' attr = ', join ',', %{$t->attr};
print ' attrseq = ', join ',', @{$t->attrseq};
}
#
print 'is_tag ', $t->is_tag;
print 'is_start_tag ', $t->is_start_tag;
print 'is_end_tag ', $t->is_end_tag;
}
elsif( $t->is_pi ){
print ' target = ', $t->target;
print ' data = ', $t->data;
print 'is_pi ', $t->is_pi;
}
else {
print ' text = ', $t->text;
print 'is_text ', $t->is_text;
print 'is_comment ', $t->is_comment;
}
#
print $/;
}
__END__
Output:
raw =
tag = tag
attr = foo,bar,foy,floy
attrseq = foo,foy
is_tag 1
is_start_tag 1
is_end_tag 0
raw = some text
text = some text
is_text 1
is_comment 0
raw =
text = comment
is_text 0
is_comment 1
raw =
tag = tag
is_tag 1
is_start_tag 0
is_end_tag 1
BUGS
To report bugs, go to
or send mail
to
AUTHOR
Copyright (c) 2003 D.H. aka PodMaster (current maintainer). Copyright
(c) 2001 Eric Bohlman (original author).
All rights reserved. This program is free software; you can redistribute
it and/or modify it under the same terms as Perl itself. If you don't
know what this means, visit or .
SEE ALSO
HTML::TokeParser, XML::Parser, XML::Catalog, XML::Smart, XML::Twig.
XML-TokeParser-0.05/t/ 0040777 0001040 0001001 00000000000 07671057452 015104 5 ustar Administrators None XML-TokeParser-0.05/t/1.normal.t 0100777 0001040 0001001 00000006560 07641035477 016727 0 ustar Administrators None # Before `make install' is performed this script should be runnable with
# `make test'. After `make install' it should work as `perl test.pl'
#print "#",q[],"\n";
#print "#",q[],"\n";
#print "#",q[],"\n";
use Test;
BEGIN { plan tests => 34, todo => [] }
use XML::TokeParser;
ok(1);
#parse from file
my $p = XML::TokeParser->new('TokeParser.xml');
ok($p);
my @tokens = (
[
'S',
qq[\Q]
],
[
'T',
'\s+'
],
[
'S',
qq[\Q]
],
[
'T',
'\s+'
],
[
'S',
qq[\Q]
],
[
'T',
qq[\QXML::TokeParser - Simplified interface to XML::Parser]
],
[
'E',
qq[\Q]
],
[
'T',
'\s+'
],
[
'E',
qq[\Q]
],
[
'T',
'\s+'
],
[
'S',
qq[\Q]
],
[
'T',
'\s+'
],
[
'S',
qq[\Q]
],
[
'T',
qq[\QSYNOPSIS]
],
[
'E',
qq[\Q]
],
[
'T',
'\s+'
],
[
'S',
qq[\Q]
],
[
'T',
qq[\Quse XML::TokeParser;]
],
[
'E',
qq[\Q]
],
[
'T',
'\s+'
]
);
for( 0.. $#tokens ) {
my $token = $p->get_token();
ok(2+$_) if $tokens[$_][0] eq $token->[0] and $token->[-1] =~ m{$tokens[$_][1]};
}
print "#",q[],"\n";
print "#",q[ Now testing get_tag, get_trimmed_text],"\n";
ok( $p->get_tag('title') );
print "#",q[$p->get_tag('title')],"\n";
ok( $p->get_trimmed_text('/title') eq 'DESCRIPTION' );
print "#",q[$p->get_trimmed_text('/title')],"\n";
ok( $p->get_tag('item') );
print "#",q[$p->get_tag('item')],"\n";
ok( $p->get_tag('itemtext') );
print "#",q[$p->get_tag('itemtext')],"\n";
ok( $p->get_trimmed_text('/itemtext') eq 'Start tag' );
print "#",q[$p->get_trimmed_text('/itemtext')],"\n";
print "#",q[],"\n";
print "#",q[ Now testing saving tokens so you can go return to this point in the stream],"\n";
ok( not $p->begin_saving() );
print "#",q[$p->begin_saving() ],"\n";
ok( $p->get_tag('para') );
print "#",q[$p->get_tag('para') 1],"\n";
ok( $p->get_tag('para') );
print "#",q[$p->get_tag('para') 2],"\n";
ok( $p->restore_saved() );
print "#",q[$p->restore_saved()],"\n";
print "#",q[],"\n";
print "#",q[ Now to see if we've backed up correctly (i think so)],"\n";
ok( $p->get_tag('para') );
print "#",q[$p->get_tag('para') 1],"\n";
ok( $p->get_tag('para') );
print "#",q[$p->get_tag('para') 2],"\n";
ok( $p->get_trimmed_text('/para') eq "The token has three elements: 'E', the element's name, and the literal text." );
#use Data::Dumper;die Dumper( );
#push @tokens, $p->get_token() for 1..10;use Data::Dumper;die Dumper\@tokens;
XML-TokeParser-0.05/t/2.extended.t 0100777 0001040 0001001 00000004720 07670620377 017235 0 ustar Administrators None # Before `make install' is performed this script should be runnable with
# `make test'. After `make install' it should work as `perl test.pl'
#print "#",q[],"\n";
#print "#",q[],"\n";
#print "#",q[],"\n";
use Test;
BEGIN { plan tests => 42, todo => [] }
use XML::TokeParser;
ok(1);
#parse from file
my $p = XML::TokeParser->new('TokeParser.xml');
ok($p);
my @tokens =(
'S',
'T',
'S',
'T',
'S',
'T',
'E',
'T',
'E',
'T',
'S',
'T',
'S',
'T',
'E',
'T',
'S',
'T',
'E',
'T',
'S',
'T',
'E',
'T',
'S',
'T',
'E',
'T',
'S',
'T',
'E',
'T',
'S',
'T',
'E',
'T',
'S',
'T',
'E',
'T'
);
my %Token2Method = (
S => 'is_start_tag',
E => 'is_end_tag',
T => 'is_text',
C => 'is_comment',
PI => 'is_pi',
);
print "#",q[],"\n";
#push @tokens, ( $p->get_token() )->[0] for 1..40;use Data::Dumper;die Dumper\@tokens;
for( 0.. $#tokens ) {
my $token = $p->get_token();
#use Data::Dumper;print Dumper $token;
my $method = $Token2Method{$token->[0]} || 'is_tag';
print '#$ ', $token->$method() ,$/;
print '#$ ', $token->is_start_tag(),$/;
ok( $token->$method() );
}
#$p->get_token()->toke;
__END__
sub is_start_tag {
if( $_[0]->[0] eq 'S' ){
if(defined $_[1]){
return 1 if $_[0]->[1] eq $_[1];
} else {
return 1;
}
}
}
sub is_end_tag {
if( $_[0]->[0] eq 'E' ){
if(defined $_[1]){
return 1 if $_[0]->[1] eq $_[1];
} else {
return 1;
}
}
}
sub is_tag {
if( $_[0]->[0] eq 'S' or $_[0]->[0] eq 'E' ){
if(defined $_[1]){
return 1 if $_[0]->[1] eq $_[1];
} else {
return 1;
}
}
}
## the old ones
sub is_start_tag { return $_[0]->_is( S => $_[1] ); }
sub is_end_tag { return $_[0]->_is( E => $_[1] ); }
sub is_tag { return $_[0]->_is( S => $_[1] )
|| $_[0]->_is( E => $_[1] ); }
sub _is {
if($_[0]->[0] eq $_[1]){
if(defined $_[2]){
return 1 if $_[0]->[1] eq $_[2];
}else{
return 1;
}
}
return 0;
} XML-TokeParser-0.05/TODO 0100777 0001040 0001001 00000000477 07671051657 015342 0 ustar Administrators None Here's the TODO list, in no particular order:
- cleanup code, maybe refactor (as per http://perlmonks.com/index.pl?node_id=264094 ).
- XML::Parser will die on bad xml, so I might wrap some calls in eval ...
- Improve test suite (add optional Catalog tests for one).
- Improve documentation (add more examples).
XML-TokeParser-0.05/TokeParser.pm 0100777 0001040 0001001 00000051760 07671053230 017255 0 ustar Administrators None =head1 NAME
XML::TokeParser - Simplified interface to XML::Parser
=head1 SYNOPSIS
use XML::TokeParser;
#
#parse from file
my $p = XML::TokeParser->new('file.xml')
#
#parse from open handle
open IN, 'file.xml' or die $!;
my $p = XML::TokeParser->new( \*IN, Noempty => 1 );
#
#parse literal text
my $text = 'text';
my $p = XML::TokeParser->new( \$text, Namespaces => 1 );
#
#read next token
my $token = $p->get_token();
#
#skip to and read text
$p->get_tag('title');
$p->get_text();
#
#read text of next , ignoring any internal markup
$p->get_tag('para');
$p->get_trimmed_text('/para');
#
#process if interesting text
$t = $p->get_tag('para');
$p->begin_saving($t);
if ( $p->get_trimmed_text('/para') =~ /interesting stuff/ ) {
$p->restore_saved();
process_para($p);
}
=head1 DESCRIPTION
XML::TokeParser provides a procedural ("pull mode") interface to XML::Parser
in much the same way that Gisle Aas' HTML::TokeParser provides a procedural
interface to HTML::Parser. XML::TokeParser splits its XML input up into
"tokens," each corresponding to an XML::Parser event.
A token is a B> reference to an array whose first element is an event-type
string and whose last element is the literal text of the XML input that
generated the event, with intermediate elements varying according to the
event type.
Each token is an I