Data-Stream-Bulk-0.11000755001750000144 011716351433 13547 5ustar00doyusers000000000000README100644001750000144 44611716351433 14474 0ustar00doyusers000000000000Data-Stream-Bulk-0.11 This archive contains the distribution Data-Stream-Bulk, version 0.11: N at a time iteration API This software is copyright (c) 2012 by Yuval Kogman. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. Changes100644001750000144 144111716351433 15123 0ustar00doyusers000000000000Data-Stream-Bulk-0.110.11 2012-02-13 - needs to actually depend on Path::Class 0.10 2012-02-12 - Fix ::FileHandle on pre-5.14 perls 0.09 2012-02-12 - oops, missed rafl's upload, rerelease with the correct version number 0.08 (DOY) 2012-02-12 - Convert to dzil - Add ::Chunked for combining incoming blocks into larger blocks - Add ::FileHandle for iterating over lines in a text file 0.08 (FLORA) - Fix new warnings in Moose 0.07 - Fix new warnings in Moose about overwriting methods 0.06 - Fix several pod errors (Closes RT#43490) (Ryan Niebur) - Add pod tests - Update version control notice 0.05 - Silence some new warnings from Moose 0.04 - add exclusions to role composition to avoid the new warning 0.03 - add Path::Class iterator t000755001750000144 011716351433 13733 5ustar00doyusers000000000000Data-Stream-Bulk-0.11dbi.t100644001750000144 253011716351433 15016 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/t#!/usr/bin/perl use strict; use warnings; use Test::More; use Test::Requires { 'DBI' => 0, 'DBD::Mock' => 1.37, }; our $dbh; BEGIN { plan skip_all => $@ unless eval { $dbh = DBI->connect( 'DBI:Mock:', '', '' ) || die "Cannot create handle: $DBI::errstr\n" }; } use Data::Stream::Bulk::DBI; my @data = ( [ qw(col1 col2 col3) ], [ qw(foo bar gorch) ], [ qw(zot oi lalala) ], [ qw(those meddling kids) ], ); { $dbh->{mock_add_resultset} = [ @data ]; my $sth = $dbh->prepare("SELECT * FROM foo;"); $sth->execute; my $d = Data::Stream::Bulk::DBI->new( sth => $sth, max_rows => 2, ); ok( !$d->is_done, "not yet done" ); is_deeply( $d->next, [ @data[1,2] ], "two rows" ); ok( !$d->is_done, "not yet done" ); is_deeply( [ $d->items ], [ $data[3] ], "one more" ); ok( !$d->is_done, "not yet done" ); is_deeply( [ $d->items ], [ ], "no more" ); ok( $d->is_done, "now we're done" ); } { $dbh->{mock_add_resultset} = [ @data ]; my $sth = $dbh->prepare("SELECT * FROM foo;"); $sth->execute; my $d = Data::Stream::Bulk::DBI->new( sth => $sth, max_rows => 1, ); ok( !$d->is_done, "not yet done" ); is_deeply( $d->next, [ $data[1] ], "one row" ); ok( !$d->is_done, "not yet done" ); is_deeply( [ $d->all ], [ @data[2,3] ], "all remaining rows" ); ok( $d->is_done, "now we're done" ); } done_testing; LICENSE100644001750000144 4370011716351433 14661 0ustar00doyusers000000000000Data-Stream-Bulk-0.11This software is copyright (c) 2012 by Yuval Kogman. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. Terms of the Perl programming language system itself a) the GNU General Public License as published by the Free Software Foundation; either version 1, or (at your option) any later version, or b) the "Artistic License" --- The GNU General Public License, Version 1, February 1989 --- This software is Copyright (c) 2012 by Yuval Kogman. This is free software, licensed under: The GNU General Public License, Version 1, February 1989 GNU GENERAL PUBLIC LICENSE Version 1, February 1989 Copyright (C) 1989 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The license agreements of most software companies try to keep users at the mercy of those companies. By contrast, our General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. The General Public License applies to the Free Software Foundation's software and to any other program whose authors commit to using it. You can use it for your programs, too. When we speak of free software, we are referring to freedom, not price. Specifically, the General Public License is designed to make sure that you have the freedom to give away or sell copies of free software, that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of a such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must tell them their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License Agreement applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any work containing the Program or a portion of it, either verbatim or with modifications. Each licensee is addressed as "you". 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this General Public License and to the absence of any warranty; and give any other recipients of the Program a copy of this General Public License along with the Program. You may charge a fee for the physical act of transferring a copy. 2. You may modify your copy or copies of the Program or any portion of it, and copy and distribute such modifications under the terms of Paragraph 1 above, provided that you also do the following: a) cause the modified files to carry prominent notices stating that you changed the files and the date of any change; and b) cause the whole of any work that you distribute or publish, that in whole or in part contains the Program or any part thereof, either with or without modifications, to be licensed at no charge to all third parties under the terms of this General Public License (except that you may choose to grant warranty protection to some or all third parties, at your option). c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the simplest and most usual way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this General Public License. d) You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. Mere aggregation of another independent work with the Program (or its derivative) on a volume of a storage or distribution medium does not bring the other work under the scope of these terms. 3. You may copy and distribute the Program (or a portion or derivative of it, under Paragraph 2) in object code or executable form under the terms of Paragraphs 1 and 2 above provided that you also do one of the following: a) accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Paragraphs 1 and 2 above; or, b) accompany it with a written offer, valid for at least three years, to give any third party free (except for a nominal charge for the cost of distribution) a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Paragraphs 1 and 2 above; or, c) accompany it with the information you received as to where the corresponding source code may be obtained. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form alone.) Source code for a work means the preferred form of the work for making modifications to it. For an executable file, complete source code means all the source code for all modules it contains; but, as a special exception, it need not include source code for modules which are standard libraries that accompany the operating system on which the executable file runs, or for standard header files or definitions files that accompany that operating system. 4. You may not copy, modify, sublicense, distribute or transfer the Program except as expressly provided under this General Public License. Any attempt otherwise to copy, modify, sublicense, distribute or transfer the Program is void, and will automatically terminate your rights to use the Program under this License. However, parties who have received copies, or rights to use copies, from you under this General Public License will not have their licenses terminated so long as such parties remain in full compliance. 5. By copying, distributing or modifying the Program (or any work based on the Program) you indicate your acceptance of this license to do so, and all its terms and conditions. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. 7. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of the license which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the license, you may choose any version ever published by the Free Software Foundation. 8. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 9. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 10. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS Appendix: How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to humanity, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) 19yy This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 1, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston MA 02110-1301 USA Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) 19xx name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (a program to direct compilers to make passes at assemblers) written by James Hacker. , 1 April 1989 Ty Coon, President of Vice That's all there is to it! --- The Artistic License 1.0 --- This software is Copyright (c) 2012 by Yuval Kogman. This is free software, licensed under: The Artistic License 1.0 The Artistic License Preamble The intent of this document is to state the conditions under which a Package may be copied, such that the Copyright Holder maintains some semblance of artistic control over the development of the package, while giving the users of the package the right to use and distribute the Package in a more-or-less customary fashion, plus the right to make reasonable modifications. Definitions: - "Package" refers to the collection of files distributed by the Copyright Holder, and derivatives of that collection of files created through textual modification. - "Standard Version" refers to such a Package if it has not been modified, or has been modified in accordance with the wishes of the Copyright Holder. - "Copyright Holder" is whoever is named in the copyright or copyrights for the package. - "You" is you, if you're thinking about copying or distributing this Package. - "Reasonable copying fee" is whatever you can justify on the basis of media cost, duplication charges, time of people involved, and so on. (You will not be required to justify it to the Copyright Holder, but only to the computing community at large as a market that must bear the fee.) - "Freely Available" means that no fee is charged for the item itself, though there may be fees involved in handling the item. It also means that recipients of the item may redistribute it under the same conditions they received it. 1. You may make and give away verbatim copies of the source form of the Standard Version of this Package without restriction, provided that you duplicate all of the original copyright notices and associated disclaimers. 2. You may apply bug fixes, portability fixes and other modifications derived from the Public Domain or from the Copyright Holder. A Package modified in such a way shall still be considered the Standard Version. 3. You may otherwise modify your copy of this Package in any way, provided that you insert a prominent notice in each changed file stating how and when you changed that file, and provided that you do at least ONE of the following: a) place your modifications in the Public Domain or otherwise make them Freely Available, such as by posting said modifications to Usenet or an equivalent medium, or placing the modifications on a major archive site such as ftp.uu.net, or by allowing the Copyright Holder to include your modifications in the Standard Version of the Package. b) use the modified Package only within your corporation or organization. c) rename any non-standard executables so the names do not conflict with standard executables, which must also be provided, and provide a separate manual page for each non-standard executable that clearly documents how it differs from the Standard Version. d) make other distribution arrangements with the Copyright Holder. 4. You may distribute the programs of this Package in object code or executable form, provided that you do at least ONE of the following: a) distribute a Standard Version of the executables and library files, together with instructions (in the manual page or equivalent) on where to get the Standard Version. b) accompany the distribution with the machine-readable source of the Package with your modifications. c) accompany any non-standard executables with their corresponding Standard Version executables, giving the non-standard executables non-standard names, and clearly documenting the differences in manual pages (or equivalent), together with instructions on where to get the Standard Version. d) make other distribution arrangements with the Copyright Holder. 5. You may charge a reasonable copying fee for any distribution of this Package. You may charge any fee you choose for support of this Package. You may not charge a fee for this Package itself. However, you may distribute this Package in aggregate with other (possibly commercial) programs as part of a larger (possibly commercial) software distribution provided that you do not advertise this Package as a product of your own. 6. The scripts and library files supplied as input to or produced as output from the programs of this Package do not automatically fall under the copyright of this Package, but belong to whomever generated them, and may be sold commercially, and may be aggregated with this Package. 7. C or perl subroutines supplied by you and linked into this Package shall not be considered part of this Package. 8. The name of the Copyright Holder may not be used to endorse or promote products derived from this software without specific prior written permission. 9. THIS PACKAGE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTIBILITY AND FITNESS FOR A PARTICULAR PURPOSE. The End dist.ini100644001750000144 73211716351433 15256 0ustar00doyusers000000000000Data-Stream-Bulk-0.11name = Data-Stream-Bulk author = Yuval Kogman license = Perl_5 copyright_holder = Yuval Kogman [@Filter] -bundle = @DOY -remove = NoTabsTests dist = Data-Stream-Bulk repository = github github_user = nothingmuch authority = cpan:NUFFIN Git::Tag_tag_format = %N-%v Git::NextVersion_version_regexp = ^Data-Stream-Bulk-(.+)$ [Prereqs] Moose = 0.90 namespace::clean = 0 Path::Class = 0 Sub::Exporter = 0 [Prereqs / TestRequires] Test::Requires = 0 dbic.t100644001750000144 277511716351433 15174 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/t#!/usr/bin/perl use strict; use warnings; use Test::More; use Test::Requires qw( DBIx::Class DBI DBD::SQLite Test::TempDir ); use Test::TempDir 'temp_root'; our $schema; BEGIN { plan skip_all => $@ unless eval { { package Schema::Foo; use base qw(DBIx::Class); __PACKAGE__->load_components(qw(Core)); __PACKAGE__->table("foo"); __PACKAGE__->add_columns(qw(id name)); __PACKAGE__->set_primary_key("id"); package Schema; use base qw(DBIx::Class::Schema); __PACKAGE__->load_classes(qw(Foo)); 1; } my $file = temp_root()->file("db"); $schema = Schema->connect("dbi:SQLite:dbname=$file", undef, undef, { RaiseError => 1 } ); $schema->storage->dbh->do("create table foo ( id integer primary key, name varchar )"); }; } use Data::Stream::Bulk::DBIC; { my $d = Data::Stream::Bulk::DBIC->new( resultset => $schema->resultset("Foo") ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [], "no items" ); ok( $d->is_done, "now done" ); } { $schema->resultset("Foo")->populate([ { name => "hello" }, { name => "goodbye" }, ]); my $d = Data::Stream::Bulk::DBIC->new( resultset => $schema->resultset("Foo") ); ok( !$d->is_done, "not done" ); is_deeply( [ map { $_->name } $d->items ], [ "hello" ], "one item" ); ok( !$d->is_done, "not done" ); is_deeply( [ map { $_->name } $d->items ], [ "goodbye" ], "one item" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [], "no items" ); ok( $d->is_done, "now done" ); } done_testing; META.yml100644001750000144 1253011716351433 15122 0ustar00doyusers000000000000Data-Stream-Bulk-0.11--- abstract: 'N at a time iteration API' author: - 'Yuval Kogman ' build_requires: Test::More: 0.88 Test::Requires: 0 configure_requires: ExtUtils::MakeMaker: 6.30 dynamic_config: 0 generated_by: 'Dist::Zilla version 4.300007, CPAN::Meta::Converter version 2.113640' license: perl meta-spec: url: http://module-build.sourceforge.net/META-spec-v1.4.html version: 1.4 name: Data-Stream-Bulk requires: Moose: 0.90 Path::Class: 0 Sub::Exporter: 0 namespace::clean: 0 resources: bugtracker: http://rt.cpan.org/Public/Dist/Display.html?Name=Data-Stream-Bulk homepage: http://metacpan.org/release/Data-Stream-Bulk repository: git://github.com/nothingmuch/data-stream-bulk.git version: 0.11 x_Dist_Zilla: plugins: - class: Dist::Zilla::Plugin::Prereqs config: Dist::Zilla::Plugin::Prereqs: phase: test type: requires name: '@Filter/TestMoreDoneTesting' version: 4.300007 - class: Dist::Zilla::Plugin::GatherDir name: '@Filter/GatherDir' version: 4.300007 - class: Dist::Zilla::Plugin::PruneCruft name: '@Filter/PruneCruft' version: 4.300007 - class: Dist::Zilla::Plugin::ManifestSkip name: '@Filter/ManifestSkip' version: 4.300007 - class: Dist::Zilla::Plugin::MetaYAML name: '@Filter/MetaYAML' version: 4.300007 - class: Dist::Zilla::Plugin::License name: '@Filter/License' version: 4.300007 - class: Dist::Zilla::Plugin::Readme name: '@Filter/Readme' version: 4.300007 - class: Dist::Zilla::Plugin::ExtraTests name: '@Filter/ExtraTests' version: 4.300007 - class: Dist::Zilla::Plugin::ExecDir name: '@Filter/ExecDir' version: 4.300007 - class: Dist::Zilla::Plugin::ShareDir name: '@Filter/ShareDir' version: 4.300007 - class: Dist::Zilla::Plugin::MakeMaker name: '@Filter/MakeMaker' version: 4.300007 - class: Dist::Zilla::Plugin::Manifest name: '@Filter/Manifest' version: 4.300007 - class: Dist::Zilla::Plugin::TestRelease name: '@Filter/TestRelease' version: 4.300007 - class: Dist::Zilla::Plugin::ConfirmRelease name: '@Filter/ConfirmRelease' version: 4.300007 - class: Dist::Zilla::Plugin::MetaConfig name: '@Filter/MetaConfig' version: 4.300007 - class: Dist::Zilla::Plugin::MetaJSON name: '@Filter/MetaJSON' version: 4.300007 - class: Dist::Zilla::Plugin::NextRelease name: '@Filter/NextRelease' version: 4.300007 - class: Dist::Zilla::Plugin::CheckChangesHasContent name: '@Filter/CheckChangesHasContent' version: 0.003 - class: Dist::Zilla::Plugin::PkgVersion name: '@Filter/PkgVersion' version: 4.300007 - class: Dist::Zilla::Plugin::Authority name: '@Filter/Authority' version: 1.006 - class: Dist::Zilla::Plugin::PodCoverageTests name: '@Filter/PodCoverageTests' version: 4.300007 - class: Dist::Zilla::Plugin::PodSyntaxTests name: '@Filter/PodSyntaxTests' version: 4.300007 - class: Dist::Zilla::Plugin::EOLTests name: '@Filter/EOLTests' version: 0.02 - class: Dist::Zilla::Plugin::Test::Compile name: '@Filter/Test::Compile' version: 1.112400 - class: Dist::Zilla::Plugin::Metadata name: '@Filter/Metadata' version: 3.03 - class: Dist::Zilla::Plugin::MetaResources name: '@Filter/MetaResources' version: 4.300007 - class: Dist::Zilla::Plugin::Git::Check name: '@Filter/Git::Check' version: 1.112440 - class: Dist::Zilla::Plugin::Git::Commit name: '@Filter/Git::Commit' version: 1.112440 - class: Dist::Zilla::Plugin::Git::Tag name: '@Filter/Git::Tag' version: 1.112440 - class: Dist::Zilla::Plugin::Git::NextVersion name: '@Filter/Git::NextVersion' version: 1.112440 - class: Dist::Zilla::Plugin::PodWeaver name: '@Filter/PodWeaver' version: 3.101641 - class: Dist::Zilla::Plugin::UploadToCPAN name: '@Filter/UploadToCPAN' version: 4.300007 - class: Dist::Zilla::Plugin::Prereqs config: Dist::Zilla::Plugin::Prereqs: phase: runtime type: requires name: Prereqs version: 4.300007 - class: Dist::Zilla::Plugin::Prereqs config: Dist::Zilla::Plugin::Prereqs: phase: test type: requires name: TestRequires version: 4.300007 - class: Dist::Zilla::Plugin::FinderCode name: ':InstallModules' version: 4.300007 - class: Dist::Zilla::Plugin::FinderCode name: ':IncModules' version: 4.300007 - class: Dist::Zilla::Plugin::FinderCode name: ':TestFiles' version: 4.300007 - class: Dist::Zilla::Plugin::FinderCode name: ':ExecFiles' version: 4.300007 - class: Dist::Zilla::Plugin::FinderCode name: ':ShareFiles' version: 4.300007 - class: Dist::Zilla::Plugin::FinderCode name: ':MainModule' version: 4.300007 zilla: class: Dist::Zilla::Dist::Builder config: is_trial: 0 version: 4.300007 x_authority: cpan:NUFFIN MANIFEST100644001750000144 111411716351433 14756 0ustar00doyusers000000000000Data-Stream-Bulk-0.11Changes LICENSE MANIFEST META.json META.yml Makefile.PL README dist.ini lib/Data/Stream/Bulk.pm lib/Data/Stream/Bulk/Array.pm lib/Data/Stream/Bulk/Callback.pm lib/Data/Stream/Bulk/Cat.pm lib/Data/Stream/Bulk/Chunked.pm lib/Data/Stream/Bulk/DBI.pm lib/Data/Stream/Bulk/DBIC.pm lib/Data/Stream/Bulk/DoneFlag.pm lib/Data/Stream/Bulk/FileHandle.pm lib/Data/Stream/Bulk/Filter.pm lib/Data/Stream/Bulk/Nil.pm lib/Data/Stream/Bulk/Path/Class.pm lib/Data/Stream/Bulk/Util.pm t/00-compile.t t/basic.t t/dbi.t t/dbic.t t/path_class.t t/release-eol.t t/release-pod-coverage.t t/release-pod-syntax.t basic.t100644001750000144 1675011716351433 15372 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/t#!/usr/bin/perl use strict; use warnings; use Test::More; use Data::Stream::Bulk::Nil; use Data::Stream::Bulk::Array; use Data::Stream::Bulk::Callback; use Data::Stream::Bulk::FileHandle; use Data::Stream::Bulk::Util qw(bulk nil cat filter unique); { my $d = Data::Stream::Bulk::Nil->new; ok( $d->is_done, "Nil is always done" ); ok( !$d->next, "no next block" ); isa_ok( nil, "Data::Stream::Bulk::Nil", "nil() helper" ); ok( nil->loaded, "nil is realized" ); isa_ok( bulk(), "Data::Stream::Bulk::Nil", "bulk() helper with no items" ); isa_ok( nil->cat(nil), "Data::Stream::Bulk::Nil", "cating nil with nil results in nil" ); isa_ok( cat(), "Data::Stream::Bulk::Nil", "cat with no args returns nil" ); isa_ok( cat(nil), "Data::Stream::Bulk::Nil", "cat of nil is nil" ); isa_ok( cat(nil, nil, nil, nil), "Data::Stream::Bulk::Nil", "cat of several nil is nil" ); is_deeply( [ nil->items ], [], "no items" ); is_deeply( [ nil->all ], [], "nothing at all" ); isa_ok( nil->filter(sub {[]}), "Data::Stream::Bulk::Nil" ); } { my @array = qw(foo bar gorch baz); my $d = Data::Stream::Bulk::Array->new( array => \@array ); ok( $d->loaded, "array is realized" ); ok( !$d->is_done, "not done" ); is_deeply( $d->next, \@array, "next" ); ok( $d->is_done, "now it's done" ); ok( !$d->next, "no next block" ); is_deeply( bulk(@array)->next, \@array, "bulk() helper" ); isa_ok( nil->cat(bulk(@array)), "Data::Stream::Bulk::Array", "nil cat Array results in Array" ); my $cat = bulk(qw(foo bar))->cat(bulk(qw(gorch baz))); isa_ok( $cat, "Data::Stream::Bulk::Array", "Array cat Array resuls in Array" ); is_deeply( $cat->next, \@array, "concatenated array into one block" ); is_deeply( [ cat(bulk(qw(foo bar)), bulk(qw(gorch baz)))->all ], \@array, "cat helper function" ); } { my @array = qw(foo bar gorch baz); my $d = Data::Stream::Bulk::Array->new( array => \@array ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], \@array, "items method" ); ok( $d->is_done, "now it's done" ); ok( !$d->next, "no next block" ); } { my @array = qw(foo bar); my $cb = sub { @array && [ shift @array ] }; my $d = Data::Stream::Bulk::Callback->new( callback => $cb ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ "foo" ], "items method" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ "bar" ], "items method" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ ], "items method" ); ok( $d->is_done, "now it's done" ); ok( !$d->next, "no next" ); } { my @copy = my @array = qw(foo bar gorch); my $cb = sub { @array && [ shift @array ] }; my $d = Data::Stream::Bulk::Callback->new( callback => $cb ); ok( !$d->loaded, "callback is not realized" ); is_deeply( [ $d->all ], \@copy, "all method" ); ok( $d->is_done, "done" ); } { my @array = qw(foo bar); my $cb = sub { @array && [ shift @array ] }; my $d = Data::Stream::Bulk::Callback->new( callback => $cb )->cat(bulk(qw(gorch baz))); isa_ok( $d, "Data::Stream::Bulk::Cat" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ "foo" ], "items method" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ "bar" ], "items method" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ qw(gorch baz) ], "reached array" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ ], "items method" ); ok( $d->is_done, "now it's done" ); ok( !$d->next, "no next" ); } { my @array = qw(foo bar); my $cb = sub { @array && [ shift @array ] }; my $d = nil->cat(bulk(qw(gorch baz))->cat(Data::Stream::Bulk::Callback->new( callback => $cb )->cat(bulk(qw(oi))->cat(nil->cat(bulk("vey"))))->cat(nil))->cat(nil))->cat(nil)->cat(Data::Stream::Bulk::Callback->new( callback => $cb )->cat(bulk(qw(last)))); isa_ok( $d, "Data::Stream::Bulk::Cat" ); ok( !$d->loaded, "concatenation is not realized" ); is_deeply( [ map { ref } @{ $d->streams } ], [ "Data::Stream::Bulk::Array", # qw(gorch baz) "Data::Stream::Bulk::Callback", # first cb "Data::Stream::Bulk::Array", # "oi" cat "vey" "Data::Stream::Bulk::Callback", # second CB "Data::Stream::Bulk::Array", # "last" ], "list_cat simplified concatenation", ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ qw(gorch baz) ], "array block" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ "foo" ], "items method" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ "bar" ], "items method" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ qw(oi vey) ], "items method" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ "last" ], "items method" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ ], "items method" ); ok( $d->is_done, "now it's done" ); ok( !$d->next, "no next" ); is_deeply( [ map { ref } @{ $d->streams } ], [ ], "no streams in concatenated", ); } { my $d = filter { [ grep { /o/ } @$_ ] } bulk(qw(foo bar gorch baz)); isa_ok( $d, "Data::Stream::Bulk::Array" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ qw(foo gorch) ], "items method" ); ok( $d->is_done, "done" ); ok( !$d->next, "no next" ); } { my @array = ( [qw(foo bar), "", ], [qw(gorch baz)] ); my $cb = sub { shift @array }; my $d = Data::Stream::Bulk::Callback->new( callback => $cb )->filter(sub { return [ grep { length } @$_ ]; })->filter(sub{ return [ grep { /o/ } @$_ ]; }); isa_ok( $d, "Data::Stream::Bulk::Filter" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ "foo" ], "items method" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ "gorch" ], "items method" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ ], "items method" ); ok( $d->is_done, "now it's done" ); ok( !$d->next, "no next" ); } { my @array = ( [qw(foo bar bar)], [qw(gorch foo baz)] ); my $cb = sub { shift @array }; my $d = unique(Data::Stream::Bulk::Callback->new( callback => $cb )); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ qw(foo bar) ], "items method" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ qw(gorch baz) ], "items method" ); ok( !$d->is_done, "not done" ); is_deeply( [ $d->items ], [ ], "items method" ); } { my ( $foo, $bar, $gorch ) = ( [], {}, "gorch" ); my $a = unique(bulk($foo, $bar, $foo, $foo, $gorch, $bar)); isa_ok( $a, "Data::Stream::Bulk::Array", "unique on array returns array" ); is_deeply([ $a->all ], [ $foo, $bar, $gorch ], "unique with refs" ); } { my $i = 0; my $cb = sub { $i++; return if $i > 6; return [ ($i) x $i ]; }; my $s = Data::Stream::Bulk::Callback->new(callback => $cb)->chunk(5); isa_ok( $s, 'Data::Stream::Bulk::Chunked' ); ok( !$s->is_done, "stream is not done"); is_deeply( $s->next, [ 1, 2, 2, 3, 3, 3 ] ); is_deeply( $s->next, [ 4, 4, 4, 4, 5, 5, 5, 5, 5 ] ); is_deeply( $s->next, [ 6, 6, 6, 6, 6, 6 ] ); ok( !defined($s->next), "stream is done" ); ok( $s->is_done, "stream is done" ); } { my $text = <<'TEXT'; foo bar baz quux TEXT chomp $text; open my $fh, '<', \$text or die "Couldn't open string: $!"; my $s = Data::Stream::Bulk::FileHandle->new(filehandle => $fh); isa_ok( $s, 'Data::Stream::Bulk::FileHandle' ); ok( !$s->is_done, "stream is not done"); is_deeply( $s->next, [ "foo\n" ] ); is_deeply( $s->next, [ "bar baz\n" ] ); is_deeply( $s->next, [ "\n" ] ); is_deeply( $s->next, [ "quux" ] ); ok( !defined($s->next), "stream is done"); ok( $s->is_done, "stream is done" ); } done_testing; META.json100644001750000144 2043111716351433 15271 0ustar00doyusers000000000000Data-Stream-Bulk-0.11{ "abstract" : "N at a time iteration API", "author" : [ "Yuval Kogman " ], "dynamic_config" : 0, "generated_by" : "Dist::Zilla version 4.300007, CPAN::Meta::Converter version 2.113640", "license" : [ "perl_5" ], "meta-spec" : { "url" : "http://search.cpan.org/perldoc?CPAN::Meta::Spec", "version" : "2" }, "name" : "Data-Stream-Bulk", "prereqs" : { "configure" : { "requires" : { "ExtUtils::MakeMaker" : "6.30" } }, "runtime" : { "requires" : { "Moose" : "0.90", "Path::Class" : 0, "Sub::Exporter" : 0, "namespace::clean" : 0 } }, "test" : { "requires" : { "Test::More" : "0.88", "Test::Requires" : 0 } } }, "release_status" : "stable", "resources" : { "bugtracker" : { "mailto" : "bug-data-stream-bulk@rt.cpan.org", "web" : "http://rt.cpan.org/Public/Dist/Display.html?Name=Data-Stream-Bulk" }, "homepage" : "http://metacpan.org/release/Data-Stream-Bulk", "repository" : { "type" : "git", "url" : "git://github.com/nothingmuch/data-stream-bulk.git", "web" : "https://github.com/nothingmuch/data-stream-bulk" } }, "version" : "0.11", "x_Dist_Zilla" : { "plugins" : [ { "class" : "Dist::Zilla::Plugin::Prereqs", "config" : { "Dist::Zilla::Plugin::Prereqs" : { "phase" : "test", "type" : "requires" } }, "name" : "@Filter/TestMoreDoneTesting", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::GatherDir", "name" : "@Filter/GatherDir", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::PruneCruft", "name" : "@Filter/PruneCruft", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::ManifestSkip", "name" : "@Filter/ManifestSkip", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::MetaYAML", "name" : "@Filter/MetaYAML", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::License", "name" : "@Filter/License", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::Readme", "name" : "@Filter/Readme", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::ExtraTests", "name" : "@Filter/ExtraTests", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::ExecDir", "name" : "@Filter/ExecDir", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::ShareDir", "name" : "@Filter/ShareDir", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::MakeMaker", "name" : "@Filter/MakeMaker", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::Manifest", "name" : "@Filter/Manifest", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::TestRelease", "name" : "@Filter/TestRelease", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::ConfirmRelease", "name" : "@Filter/ConfirmRelease", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::MetaConfig", "name" : "@Filter/MetaConfig", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::MetaJSON", "name" : "@Filter/MetaJSON", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::NextRelease", "name" : "@Filter/NextRelease", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::CheckChangesHasContent", "name" : "@Filter/CheckChangesHasContent", "version" : "0.003" }, { "class" : "Dist::Zilla::Plugin::PkgVersion", "name" : "@Filter/PkgVersion", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::Authority", "name" : "@Filter/Authority", "version" : "1.006" }, { "class" : "Dist::Zilla::Plugin::PodCoverageTests", "name" : "@Filter/PodCoverageTests", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::PodSyntaxTests", "name" : "@Filter/PodSyntaxTests", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::EOLTests", "name" : "@Filter/EOLTests", "version" : "0.02" }, { "class" : "Dist::Zilla::Plugin::Test::Compile", "name" : "@Filter/Test::Compile", "version" : "1.112400" }, { "class" : "Dist::Zilla::Plugin::Metadata", "name" : "@Filter/Metadata", "version" : "3.03" }, { "class" : "Dist::Zilla::Plugin::MetaResources", "name" : "@Filter/MetaResources", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::Git::Check", "name" : "@Filter/Git::Check", "version" : "1.112440" }, { "class" : "Dist::Zilla::Plugin::Git::Commit", "name" : "@Filter/Git::Commit", "version" : "1.112440" }, { "class" : "Dist::Zilla::Plugin::Git::Tag", "name" : "@Filter/Git::Tag", "version" : "1.112440" }, { "class" : "Dist::Zilla::Plugin::Git::NextVersion", "name" : "@Filter/Git::NextVersion", "version" : "1.112440" }, { "class" : "Dist::Zilla::Plugin::PodWeaver", "name" : "@Filter/PodWeaver", "version" : "3.101641" }, { "class" : "Dist::Zilla::Plugin::UploadToCPAN", "name" : "@Filter/UploadToCPAN", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::Prereqs", "config" : { "Dist::Zilla::Plugin::Prereqs" : { "phase" : "runtime", "type" : "requires" } }, "name" : "Prereqs", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::Prereqs", "config" : { "Dist::Zilla::Plugin::Prereqs" : { "phase" : "test", "type" : "requires" } }, "name" : "TestRequires", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::FinderCode", "name" : ":InstallModules", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::FinderCode", "name" : ":IncModules", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::FinderCode", "name" : ":TestFiles", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::FinderCode", "name" : ":ExecFiles", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::FinderCode", "name" : ":ShareFiles", "version" : "4.300007" }, { "class" : "Dist::Zilla::Plugin::FinderCode", "name" : ":MainModule", "version" : "4.300007" } ], "zilla" : { "class" : "Dist::Zilla::Dist::Builder", "config" : { "is_trial" : "0" }, "version" : "4.300007" } }, "x_authority" : "cpan:NUFFIN" } Makefile.PL100644001750000144 215311716351433 15603 0ustar00doyusers000000000000Data-Stream-Bulk-0.11 use strict; use warnings; use ExtUtils::MakeMaker 6.30; my %WriteMakefileArgs = ( "ABSTRACT" => "N at a time iteration API", "AUTHOR" => "Yuval Kogman ", "BUILD_REQUIRES" => { "Test::More" => "0.88", "Test::Requires" => 0 }, "CONFIGURE_REQUIRES" => { "ExtUtils::MakeMaker" => "6.30" }, "DISTNAME" => "Data-Stream-Bulk", "EXE_FILES" => [], "LICENSE" => "perl", "NAME" => "Data::Stream::Bulk", "PREREQ_PM" => { "Moose" => "0.90", "Path::Class" => 0, "Sub::Exporter" => 0, "namespace::clean" => 0 }, "VERSION" => "0.11", "test" => { "TESTS" => "t/*.t" } ); unless ( eval { ExtUtils::MakeMaker->VERSION(6.56) } ) { my $br = delete $WriteMakefileArgs{BUILD_REQUIRES}; my $pp = $WriteMakefileArgs{PREREQ_PM}; for my $mod ( keys %$br ) { if ( exists $pp->{$mod} ) { $pp->{$mod} = $br->{$mod} if $br->{$mod} > $pp->{$mod}; } else { $pp->{$mod} = $br->{$mod}; } } } delete $WriteMakefileArgs{CONFIGURE_REQUIRES} unless eval { ExtUtils::MakeMaker->VERSION(6.52) }; WriteMakefile(%WriteMakefileArgs); path_class.t100644001750000144 415211716351433 16403 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/t#!/usr/bin/perl use strict; use warnings; use Test::More; use Data::Stream::Bulk::Path::Class; use Path::Class; my $dist = file(__FILE__)->parent->parent; foreach my $dir ( $dist->subdir("t"), $dist->subdir("lib"), $dist ) { { my $paths = Data::Stream::Bulk::Path::Class->new( dir => $dir, chunk_size => 2, depth_first => 0, ); my $strings = $paths->filter(sub {[ grep { !/tmp/ } map { "$_" } @$_ ]}); my @rec; $dir->recurse( callback => sub { push @rec, "$_[0]" unless $_[0] =~ /tmp/ }, depthfirst => 0, preorder => 1 ); ok( !$_->is_done, "not done" ) for $paths, $strings; my @all = $strings->all; ok( $_->is_done, "done" ) for $paths, $strings; is_deeply( [ sort @all ], [ sort @rec ], "breadth first traversal path set", ); is_deeply( \@all, \@rec, "breadth first traversal order", ); } { my $paths = Data::Stream::Bulk::Path::Class->new( dir => $dir, chunk_size => 2, depth_first => 1, ); my $strings = $paths->filter(sub {[ grep { !/tmp/ } map { "$_" } @$_ ]}); my @rec; $dir->recurse( callback => sub { push @rec, "$_[0]" unless $_[0] =~ /tmp/ }, depthfirst => 1, preorder => 1 ); ok( !$_->is_done, "not done" ) for $paths, $strings; my @all = $strings->all; ok( $_->is_done, "done" ) for $paths, $strings; is_deeply( [ sort @all ], [ sort @rec ], "depth first traversal path set", ); is_deeply( \@all, \@rec, "depth first traversal order", ); } { my $paths = Data::Stream::Bulk::Path::Class->new( dir => $dir, chunk_size => 2, depth_first => 0, only_files => 1, ); my $strings = $paths->filter(sub {[ grep { !/tmp/ } map { "$_" } @$_ ]}); my @rec; $dir->recurse( callback => sub { push @rec, "$_[0]" if $_[0] !~ /tmp/ and -f $_[0] }, depthfirst => 0, preorder => 1 ); ok( !$_->is_done, "not done" ) for $paths, $strings; my @all = $strings->all; ok( $_->is_done, "done" ) for $paths, $strings; is_deeply( [ sort @all ], [ sort @rec ], "breadth first traversal path set", ); is_deeply( \@all, \@rec, "breadth first traversal order", ); } } done_testing; 00-compile.t100644001750000144 265111716351433 16131 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/t#!perl use strict; use warnings; use Test::More; use File::Find; use File::Temp qw{ tempdir }; my @modules; find( sub { return if $File::Find::name !~ /\.pm\z/; my $found = $File::Find::name; $found =~ s{^lib/}{}; $found =~ s{[/\\]}{::}g; $found =~ s/\.pm$//; # nothing to skip push @modules, $found; }, 'lib', ); my @scripts; if ( -d 'bin' ) { find( sub { return unless -f; my $found = $File::Find::name; # nothing to skip open my $FH, '<', $_ or do { note( "Unable to open $found in ( $! ), skipping" ); return; }; my $shebang = <$FH>; return unless $shebang =~ /^#!.*?\bperl\b\s*$/; push @scripts, $found; }, 'bin', ); } my $plan = scalar(@modules) + scalar(@scripts); $plan ? (plan tests => $plan) : (plan skip_all => "no tests to run"); { # fake home for cpan-testers # no fake requested ## local $ENV{HOME} = tempdir( CLEANUP => 1 ); like( qx{ $^X -Ilib -e "require $_; print '$_ ok'" }, qr/^\s*$_ ok/s, "$_ loaded ok" ) for sort @modules; SKIP: { eval "use Test::Script 1.05; 1;"; skip "Test::Script needed to test script compilation", scalar(@scripts) if $@; foreach my $file ( @scripts ) { my $script = $file; $script =~ s!.*/!!; script_compiles( $file, "$script script compiles" ); } } } release-eol.t100644001750000144 47611716351433 16444 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/t BEGIN { unless ($ENV{RELEASE_TESTING}) { require Test::More; Test::More::plan(skip_all => 'these tests are for release candidate testing'); } } use strict; use warnings; use Test::More; eval 'use Test::EOL'; plan skip_all => 'Test::EOL required' if $@; all_perl_files_ok({ trailing_whitespace => 1 }); release-pod-syntax.t100644001750000144 45011716351433 17763 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/t#!perl BEGIN { unless ($ENV{RELEASE_TESTING}) { require Test::More; Test::More::plan(skip_all => 'these tests are for release candidate testing'); } } use Test::More; eval "use Test::Pod 1.41"; plan skip_all => "Test::Pod 1.41 required for testing POD" if $@; all_pod_files_ok(); Stream000755001750000144 011716351433 16342 5ustar00doyusers000000000000Data-Stream-Bulk-0.11/lib/DataBulk.pm100644001750000144 1410611716351433 17757 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/lib/Data/Streampackage Data::Stream::Bulk; BEGIN { $Data::Stream::Bulk::AUTHORITY = 'cpan:NUFFIN'; } { $Data::Stream::Bulk::VERSION = '0.11'; } use Moose::Role; # ABSTRACT: N at a time iteration API use namespace::clean -except => 'meta'; requires qw(next is_done); sub items { my $self = shift; if ( my $a = $self->next ) { return @$a; } else { return (); } } sub all { my $self = shift; my @ret; while ( my $next = $self->next ) { push @ret, @$next; } return @ret; } sub cat { my ( $self, @streams ) = @_; return $self unless @streams; my @cat = $self->list_cat(@streams); unless ( @cat ) { return Data::Stream::Bulk::Nil->new; } elsif ( @cat == 1 ) { return $cat[0]; } else { return Data::Stream::Bulk::Cat->new( streams => \@cat, ); } } sub list_cat { my ( $self, $head, @tail ) = @_; return $self unless $head; return ( $self, $head->list_cat(@tail) ); } sub filter { my ( $self, $filter ) = @_; return Data::Stream::Bulk::Filter->new( filter => $filter, stream => $self, ); } sub chunk { my ( $self, $chunk_size ) = @_; return Data::Stream::Bulk::Chunked->new( chunk_size => $chunk_size, stream => $self, ); } sub loaded { 0 } # load it *after* the entire role is defined require Data::Stream::Bulk::Cat; require Data::Stream::Bulk::Chunked; require Data::Stream::Bulk::Nil; require Data::Stream::Bulk::Filter; __PACKAGE__; =pod =head1 NAME Data::Stream::Bulk - N at a time iteration API =head1 VERSION version 0.11 =head1 SYNOPSIS # get a bulk stream from somewere my $s = Data::Stream::Bulk::Foo->new( ... ); # can be used like this: until ( $s->is_done ) { foreach my $item ( $s->items ) { process($item); } } # or like this: while( my $block = $s->next ) { foreach my $item ( @$block ) { process($item); } } =head1 DESCRIPTION This module tries to find middle ground between one at a time and all at once processing of data sets. The purpose of this module is to avoid the overhead of implementing an iterative api when this isn't necessary, without breaking forward compatibility in case that becomes necessary later on. The API optimizes for when a data set typically fits in memory and is returned as an array, but the consumer cannot assume that the data set is bounded. The API is destructive in order to minimize the chance that resultsets are leaked due to improper usage. =head1 API =head2 Required Methods The API requires two methods to be implemented: =over 4 =item is_done Should return true if the stream is exhausted. As long as this method returns a false value (not done) C could potentially return another block. =item next Returns the next block. Note that C is not guaranteed to return an array reference, even if C returned false prior to calling it. =back =head2 Convenience Methods =over 4 =item items This method calls C and dereferences the result if there are pending items. =item all Force evaluation of the entire resultset. Note that for large data sets this might cause swap thrashing of various other undesired effects. Use with caution. =item cat @streams Concatenates this stream with @streams, returning a single stream. =item list_cat @tail Returns a possibly cleaned up list of streams. Used by C. Overridden by L, L and L to implement some simple short circuiting. =item filter $filter Applies a per-block block filter to the stream. Returns a possibly new stream with the filtering layered. C<$filter> is invoked once per block and should return an array reference to the filtered block. =item chunk $chunk_size Chunks the input stream so that each block returned by C will have at least C<$chunk_size> items. =item loaded Should be overridden to return true if all the items are already realized (e.g. in the case of L). Returns false by default. When true calling C is supposed to be safe (memory usage should be in the same order of magnitude as stream's own usage). This is typically useful when tranforming an array is easier than transorming a stream (e.g. optional duplicate filtering). =back =head1 CLASSES =over 4 =item L This class is not a stream at all, but just one block. When the data set easily fits in memory this class can be used, while retaining forward compatibility with larger data sets. =item L Callback driven iteration. =item L Wrapper to return larger blocks from an existing stream. =item L Bulk fetching of data from L statement handles. =item L L iteration. =item L Iterates over lines in a text file. =item L An empty result set. =item L A concatenation of several streams. =item L A filter wrapping a stream. =back =head1 SEE ALSO L, L, L etc for one by one iteration L, L L L L L, LISP, and all that other kool aid =head1 TODO =over 4 =item Sorted streams Add a hint for sorted streams (like C but as an attribute in the base role). Introduce a C operation for merging of sorted streams. Optimize C to make use of sorting hints for constant space uniquing. =item More utility functions To assist in proccessing and creating streams. =item Coercion tables L =back =head1 VERSION CONTROL This module is maintained using git. You can get the latest version from L. =head1 AUTHOR Yuval Kogman =head1 COPYRIGHT AND LICENSE This software is copyright (c) 2012 by Yuval Kogman. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. =cut __END__ release-pod-coverage.t100644001750000144 76511716351433 20241 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/t#!perl BEGIN { unless ($ENV{RELEASE_TESTING}) { require Test::More; Test::More::plan(skip_all => 'these tests are for release candidate testing'); } } use Test::More; eval "use Test::Pod::Coverage 1.08"; plan skip_all => "Test::Pod::Coverage 1.08 required for testing POD coverage" if $@; eval "use Pod::Coverage::TrustPod"; plan skip_all => "Pod::Coverage::TrustPod required for testing POD coverage" if $@; all_pod_coverage_ok({ coverage_class => 'Pod::Coverage::TrustPod' }); Bulk000755001750000144 011716351433 17237 5ustar00doyusers000000000000Data-Stream-Bulk-0.11/lib/Data/StreamCat.pm100644001750000144 344611716351433 20453 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/lib/Data/Stream/Bulkpackage Data::Stream::Bulk::Cat; BEGIN { $Data::Stream::Bulk::Cat::AUTHORITY = 'cpan:NUFFIN'; } { $Data::Stream::Bulk::Cat::VERSION = '0.11'; } use Moose; # ABSTRACT: Concatenated streams use namespace::clean -except => 'meta'; with qw(Data::Stream::Bulk) => { -excludes => 'list_cat' }; has streams => ( isa => "ArrayRef[Data::Stream::Bulk]", is => "ro", required => 1, ); sub is_done { my $self = shift; @{ $self->streams } == 0; } sub next { my $self = shift; my $s = $self->streams; return unless @$s; my $next; until ( $next = @$s && $s->[0]->next ) { shift @$s; return unless @$s; } return $next; } sub list_cat { my ( $self, @rest ) = @_; my ( $head, @tail ) = ( @{ $self->streams }, @rest ); return () unless $head; return $head->list_cat(@tail); } __PACKAGE__->meta->make_immutable; __PACKAGE__; =pod =head1 NAME Data::Stream::Bulk::Cat - Concatenated streams =head1 VERSION version 0.11 =head1 SYNOPSIS use Data::Stream::Bulk::Cat; Data::Stream::Bulk::Cat->new( streams => [ $s1, $s2, $s3 ], ); =head1 DESCRIPTION This stream is a concatenation of several other streams. =head1 METHODS =over 4 =item is_done Returns true if the list of streams is empty. =item next Returns the next block from the next ready stream. =item list_cat Breaks down the internal list of streams, and delegates C to the first one. Has the effect of inlining the nested streams into the total concatenation, allowing L to work better. =back =head1 AUTHOR Yuval Kogman =head1 COPYRIGHT AND LICENSE This software is copyright (c) 2012 by Yuval Kogman. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. =cut __END__ DBI.pm100644001750000144 447111716351433 20341 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/lib/Data/Stream/Bulkpackage Data::Stream::Bulk::DBI; BEGIN { $Data::Stream::Bulk::DBI::AUTHORITY = 'cpan:NUFFIN'; } { $Data::Stream::Bulk::DBI::VERSION = '0.11'; } use Moose; # ABSTRACT: N-at-a-time iteration of L statement results. use namespace::clean -except => 'meta'; with qw(Data::Stream::Bulk::DoneFlag) => { -excludes => [qw/is_done all finished/] }; has sth => ( isa => "Object", is => "ro", required => 1, handles => [qw(fetchall_arrayref)], clearer => "finished", ); has slice => ( is => "ro", ); has max_rows => ( isa => "Int", is => "rw", default => 500, ); sub get_more { my $self = shift; $self->fetchall_arrayref( $self->slice, $self->max_rows ); } sub all { my $self = shift; my $all = $self->fetchall_arrayref( $self->slice ); $self->_set_done; return @$all; } __PACKAGE__->meta->make_immutable; __PACKAGE__; =pod =head1 NAME Data::Stream::Bulk::DBI - N-at-a-time iteration of L statement results. =head1 VERSION version 0.11 =head1 SYNOPSIS use Data::Stream::Bulk::DBI; my $sth = $dbh->prepare("SELECT hate FROM sql"); # very big resultset $sth->execute; return Data::Stream::Bulk::DBI->new( sth => $sth, max_rows => $n, # how many at a time slice => [ ... ], # if you want to pass the first param to fetchall_arrayref ); =head1 DESCRIPTION This implementation of L api works with DBI statement handles, using L. It fetches C at a time (defaults to 500). =head1 ATTRIBUTES =over 4 =item sth The statement handle to call C on. =item slice Passed verbatim as the first param to C. Should usually be C, provided for completetness. =item max_rows The second param to C. Controls the size of each buffer. Defaults to 500. =back =head1 METHODS =over 4 =item get_more See L. Calls C to get the next chunk of rows. =item all Calls C to get the raminder of the data (without specifying C). =back =head1 AUTHOR Yuval Kogman =head1 COPYRIGHT AND LICENSE This software is copyright (c) 2012 by Yuval Kogman. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. =cut __END__ Nil.pm100644001750000144 264011716351433 20461 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/lib/Data/Stream/Bulkpackage Data::Stream::Bulk::Nil; BEGIN { $Data::Stream::Bulk::Nil::AUTHORITY = 'cpan:NUFFIN'; } { $Data::Stream::Bulk::Nil::VERSION = '0.11'; } use Moose; # ABSTRACT: An empty L iterator use namespace::clean -except => 'meta'; with qw(Data::Stream::Bulk) => { -excludes => [qw/loaded filter list_cat all items/] }; sub items { return () } sub all { return () } sub next { undef } sub is_done { 1 } sub list_cat { my ( $self, $head, @rest ) = @_; return () unless $head; return $head->list_cat(@rest); } sub filter { return $_[0] }; sub loaded { 1 } __PACKAGE__->meta->make_immutable; __PACKAGE__; =pod =head1 NAME Data::Stream::Bulk::Nil - An empty L iterator =head1 VERSION version 0.11 =head1 SYNOPSIS return Data::Stream::Bulk::Nil->new; # empty set =head1 DESCRIPTION This iterator can be used to return the empty resultset. =head1 METHODS =over 4 =item is_done Always returns true. =item next Always returns undef. =item items =item all Always returns the empty list. =item list_cat Skips $self =item filter Returns $self =item loaded Returns true. =back =head1 AUTHOR Yuval Kogman =head1 COPYRIGHT AND LICENSE This software is copyright (c) 2012 by Yuval Kogman. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. =cut __END__ Util.pm100644001750000144 471211716351433 20656 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/lib/Data/Stream/Bulkpackage Data::Stream::Bulk::Util; BEGIN { $Data::Stream::Bulk::Util::AUTHORITY = 'cpan:NUFFIN'; } { $Data::Stream::Bulk::Util::VERSION = '0.11'; } use strict; use warnings; # ABSTRACT: Utility functions for L use Data::Stream::Bulk::Nil; use Data::Stream::Bulk::Array; use Scalar::Util qw(refaddr); use namespace::clean; use Sub::Exporter -setup => { exports => [qw(nil bulk cat filter unique)], }; # use constant nil => Data::Stream::Bulk::Nil->new; sub nil () { Data::Stream::Bulk::Nil->new } sub bulk (@) { return @_ ? Data::Stream::Bulk::Array->new( array => [ @_ ] ) : nil } sub cat (@) { return @_ ? shift->cat(@_) : nil } sub filter (&$) { my ( $filter, $stream ) = @_; $stream->filter($filter); } sub unique ($) { my %seen; shift->filter(sub { [ grep { !$seen{ref($_) ? refaddr($_) : $_}++ } @$_ ] }); # FIXME Hash::Util::FieldHash::Compat::id()? } __PACKAGE__; =pod =head1 NAME Data::Stream::Bulk::Util - Utility functions for L =head1 VERSION version 0.11 =head1 SYNOPSIS use Data::Stream::Bulk::Util qw(array); use namespace::clean; # Wrap a list in L return bulk(qw(foo bar gorch baz)); # return an empty resultset return nil(); =head1 DESCRIPTION This module exports convenience functions for use with L. =head1 EXPORTS L is used to create the C routine, and all of its aliasing/currying goodness is of course supported. =over 4 =item nil Creates a new L object. Takes no arguments. =item bulk @items Creates a new L wrapping C<@items>. =item cat @streams Concatenate several streams together. Returns C if no arguments are provided. =item filter { ... } $stream Calls C on $stream with the provided filter. =item unique $stream Filter the stream to remove duplicates. Note that memory use may potentially scale to O(k) where k is the number of distinct items, because this is implemented in terms of a seen hash. In the future this will be optimized to be iterative for sorted streams. References are keyed by their refaddr (see L). =back =head1 AUTHOR Yuval Kogman =head1 COPYRIGHT AND LICENSE This software is copyright (c) 2012 by Yuval Kogman. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. =cut __END__ DBIC.pm100644001750000144 334111716351433 20437 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/lib/Data/Stream/Bulkpackage Data::Stream::Bulk::DBIC; BEGIN { $Data::Stream::Bulk::DBIC::AUTHORITY = 'cpan:NUFFIN'; } { $Data::Stream::Bulk::DBIC::VERSION = '0.11'; } use Moose; # ABSTRACT: Iterate DBIC resultsets with L use namespace::clean -except => 'meta'; with qw(Data::Stream::Bulk::DoneFlag) => { -excludes => [qw(is_done finished)] }; has resultset => ( isa => "Object", clearer => "finished", handles => { next_row => "next" }, required => 1, ); sub get_more { my $self = shift; if ( defined( my $next = $self->next_row ) ) { return [ $next ]; } else { return; } } __PACKAGE__->meta->make_immutable; __PACKAGE__; =pod =head1 NAME Data::Stream::Bulk::DBIC - Iterate DBIC resultsets with L =head1 VERSION version 0.11 =head1 SYNOPSIS Data::Stream::Bulk::DBIC->new( resultset => scalar($schema->rs("Foo")->search(...)) ); =head1 DESCRIPTION This is a wrapper for L that fits the L api. Due to the fact that DBIC inflation overhead is fairly negligiable to that of iteration though, I haven't actually bothered to make it bulk. If L will support n-at-a-time fetching as opposed to one-at-a-time or all-at-a-time at some point in the future this class will be updated to match. =head1 METHODS =over 4 =item get_more See L. Returns a single row. In the future this should return more than one row. =back =head1 AUTHOR Yuval Kogman =head1 COPYRIGHT AND LICENSE This software is copyright (c) 2012 by Yuval Kogman. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. =cut __END__ Array.pm100644001750000144 507711716351433 21024 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/lib/Data/Stream/Bulkpackage Data::Stream::Bulk::Array; BEGIN { $Data::Stream::Bulk::Array::AUTHORITY = 'cpan:NUFFIN'; } { $Data::Stream::Bulk::Array::VERSION = '0.11'; } use Moose; # ABSTRACT: L wrapper for simple arrays. use namespace::clean -except => 'meta'; with qw(Data::Stream::Bulk) => { -excludes => [qw/loaded filter list_cat/] }; has array => ( isa => "ArrayRef", reader => "_array", writer => "_set_array", clearer => "_clear_array", predicate => "_has_array", required => 1, ); sub is_done { my $self = shift; !$self->_has_array; } sub next { my $self = shift; if ( my $array = $self->_array ) { $self->_clear_array; return $array; } else { return; } } # squish several arrays into one sub list_cat { my ( $self, @rest ) = @_; return $self unless @rest; my @arrays = ( $self ); # fetch all adjacent arrays push @arrays, shift @rest while @rest and $rest[0]->isa(__PACKAGE__); if ( @arrays > 1 ) { my @cat; push @cat, @$_ for map { $_->_array } @arrays; return __PACKAGE__->new( array => \@cat, )->cat( @rest ); } else { my $head = shift @rest; return ( $self, $head->list_cat(@rest) ); } } sub filter { my ( $self, $filter ) = @_; local $_ = $self->next; $self->_set_array( $filter->($_) ); return $self; } sub loaded { 1 } __PACKAGE__->meta->make_immutable; __PACKAGE__; =pod =head1 NAME Data::Stream::Bulk::Array - L wrapper for simple arrays. =head1 VERSION version 0.11 =head1 SYNOPSIS return Data::Stream::Bulk::Array->new( array => \@results, ); =head1 DESCRIPTION This implementation of the L api wraps an array. The use case is to keep the consumer of the data set implementation agnostic so that it can deal with larger data sets if they are encountered, but still retain most of the simplicity when the current data set easily fits in memory. =head1 ATTRIBUTES =over 4 =item array The array reference to wrap. =back =head1 METHODS =over 4 =item next Returns the array reference on the first invocation, and nothing thereafter. =item is_done Returns true if C has been called. =item list_cat Squishes adjacent arrays into a new array. =item filter $filter Immediately applies C<$filter> to the internal array and returns C<$self>. =item loaded Returns true =back =head1 AUTHOR Yuval Kogman =head1 COPYRIGHT AND LICENSE This software is copyright (c) 2012 by Yuval Kogman. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. =cut __END__ Filter.pm100644001750000144 356711716351433 21175 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/lib/Data/Stream/Bulkpackage Data::Stream::Bulk::Filter; BEGIN { $Data::Stream::Bulk::Filter::AUTHORITY = 'cpan:NUFFIN'; } { $Data::Stream::Bulk::Filter::VERSION = '0.11'; } use Moose; # ABSTRACT: Streamed filtering (block oriented) use Data::Stream::Bulk; use namespace::clean -except => 'meta'; has filter => ( isa => "CodeRef", reader => "filter_body", required => 1, ); has stream => ( does => "Data::Stream::Bulk", is => "ro", required => 1, handles => [qw(is_done loaded)], ); with qw(Data::Stream::Bulk) => { -excludes => 'loaded' }; sub next { my $self = shift; local $_ = $self->stream->next; return $_ && ( $self->filter_body->($_) || [] ); } __PACKAGE__->meta->make_immutable; __PACKAGE__; =pod =head1 NAME Data::Stream::Bulk::Filter - Streamed filtering (block oriented) =head1 VERSION version 0.11 =head1 SYNOPSIS use Data::Stream::Bulk::Filter; Data::Stream::Bulk::Filter->new( filter => sub { ... }, stream => $stream, ); =head1 DESCRIPTION This class implements filtering of streams. =head1 ATTRIBUTES =over 4 =item filter The code reference to apply to each block. The block is passed to the filter both in C<$_> and as the first argument. The return value should be an array reference. If no true value is returned the output stream does B end, but instead an empty block is substituted (the parent stream controls when the stream is depleted). =item stream The stream to be filtered =back =head1 METHODS =over 4 =item is_done =item loaded Delegated to C =item next Calls C on C and applies C if a block was returned. =back =head1 AUTHOR Yuval Kogman =head1 COPYRIGHT AND LICENSE This software is copyright (c) 2012 by Yuval Kogman. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. =cut __END__ Chunked.pm100644001750000144 401711716351433 21320 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/lib/Data/Stream/Bulkpackage Data::Stream::Bulk::Chunked; BEGIN { $Data::Stream::Bulk::Chunked::AUTHORITY = 'cpan:NUFFIN'; } { $Data::Stream::Bulk::Chunked::VERSION = '0.11'; } use Moose; # ABSTRACT: combine streams into larger chunks use namespace::clean -except => 'meta'; with 'Data::Stream::Bulk::DoneFlag'; has stream => ( is => 'ro', does => 'Data::Stream::Bulk', required => 1, ); has chunk_size => ( is => 'ro', isa => 'Int', default => 1, ); sub get_more { my $self = shift; my $s = $self->stream; my $size = $self->chunk_size; my @data; push @data, $s->items until $s->is_done || @data >= $size; return unless @data; return \@data; } __PACKAGE__->meta->make_immutable; 1; __END__ =pod =head1 NAME Data::Stream::Bulk::Chunked - combine streams into larger chunks =head1 VERSION version 0.11 =head1 SYNOPSIS use Data::Stream::Bulk::Chunked; Data::Stream::Bulk::Chunked->new( stream => $s, chunk_size => 10000, ); =head1 DESCRIPTION This is a stream which wraps an existing stream to give more items in a single block. This can simplify application code which does its own processing one block at a time, and where processing larger blocks is more efficient. =head1 ATTRIBUTES =over 4 =item stream The stream to chunk. Required. =item chunk_size The minimum number of items to return in a block. Defaults to 1 (which does nothing). =back =head1 METHODS =over 4 =item get_more See L. Returns at least C items. Note that this isn't guaranteed to return exactly C items - it just returns multiple full blocks from the backend. Also, the final block returned may have less than C items. =back =head1 AUTHOR Yuval Kogman =head1 COPYRIGHT AND LICENSE This software is copyright (c) 2012 by Yuval Kogman. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. =cut Callback.pm100644001750000144 303711716351433 21434 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/lib/Data/Stream/Bulkpackage Data::Stream::Bulk::Callback; BEGIN { $Data::Stream::Bulk::Callback::AUTHORITY = 'cpan:NUFFIN'; } { $Data::Stream::Bulk::Callback::VERSION = '0.11'; } use Moose; # ABSTRACT: Callback based bulk iterator use namespace::clean -except => 'meta'; with qw(Data::Stream::Bulk::DoneFlag) => { -excludes => [qw(is_done finished)] }; has callback => ( isa => "CodeRef|Str", is => "ro", required => 1, clearer => "finished", ); sub get_more { my $self = shift; my $cb = $self->callback; $self->$cb; } __PACKAGE__->meta->make_immutable; __PACKAGE__; =pod =head1 NAME Data::Stream::Bulk::Callback - Callback based bulk iterator =head1 VERSION version 0.11 =head1 SYNOPSIS Data::Stream::Bulk::Callback->new( callback => sub { if ( @more_items = get_some() ) { return \@more_items; } else { return; # done } }, } =head1 DESCRIPTION This class provides a callback based implementation of L. =head1 ATTRIBUTES =over 4 =item callback The subroutine that is called when more items are needed. Should return an array reference for the next block, or a false value if there is nothing left. =back =head1 METHODS =over 4 =item get_more See L. Reinvokes C. =back =head1 AUTHOR Yuval Kogman =head1 COPYRIGHT AND LICENSE This software is copyright (c) 2012 by Yuval Kogman. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. =cut __END__ DoneFlag.pm100644001750000144 423411716351433 21417 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/lib/Data/Stream/Bulkpackage Data::Stream::Bulk::DoneFlag; BEGIN { $Data::Stream::Bulk::DoneFlag::AUTHORITY = 'cpan:NUFFIN'; } { $Data::Stream::Bulk::DoneFlag::VERSION = '0.11'; } use Moose::Role; # ABSTRACT: Implement the C method in terms of a flag use namespace::clean -except => 'meta'; with qw(Data::Stream::Bulk); requires "get_more"; sub is_done {} has done => ( isa => "Bool", init_arg => undef, reader => "is_done", writer => "_done", ); sub finished {} sub _set_done { my $self = shift; $self->_done(1); $self->finished; } sub next { my $self = shift; unless ( $self->is_done ) { if ( my $more = $self->get_more ) { return $more; } else { $self->_set_done; return; } } else { return; } } __PACKAGE__; =pod =head1 NAME Data::Stream::Bulk::DoneFlag - Implement the C method in terms of a flag =head1 VERSION version 0.11 =head1 SYNOPSIS package Data::Stream::Bulk::Blah; use Moose; with qw(Data::Stream::Bulk::DoneFlag); sub get_more { if ( my @more = more() ) { return \@more; } else { return; } } =head1 DESCRIPTION This role implements the C core API in terms of one method (C). As a convenience it calls C when the stream is exhausted, so that cleanup may be done. This is used by classes like L, L. =head1 METHODS =over 4 =item is_done Returns the state of the iterator. =item next As long as the iterator is not yet done, calls C. If C returned a false value instead of an array reference then C is set, C is called, and this C does nothing on subsequent calls. =item finished A noop by default. Can be overridden if so desired. =back =head1 REQUIRED_METHODS =over 4 =item get_more Returns the next block of data as an array ref, or a false value if no items are left. =back =head1 AUTHOR Yuval Kogman =head1 COPYRIGHT AND LICENSE This software is copyright (c) 2012 by Yuval Kogman. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. =cut __END__ FileHandle.pm100644001750000144 300711716351433 21730 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/lib/Data/Stream/Bulkpackage Data::Stream::Bulk::FileHandle; BEGIN { $Data::Stream::Bulk::FileHandle::AUTHORITY = 'cpan:NUFFIN'; } { $Data::Stream::Bulk::FileHandle::VERSION = '0.11'; } use Moose; # ABSTRACT: read lines from a filehandle use namespace::clean -except => 'meta'; use IO::Handle; with 'Data::Stream::Bulk::DoneFlag'; has filehandle => ( is => 'ro', isa => 'FileHandle', required => 1, ); sub get_more { my $self = shift; my $line = $self->filehandle->getline; return unless defined $line; return [ $line ]; } __PACKAGE__->meta->make_immutable; 1; __END__ =pod =head1 NAME Data::Stream::Bulk::FileHandle - read lines from a filehandle =head1 VERSION version 0.11 =head1 SYNOPSIS use Data::Stream::Bulk::FileHandle; use Path::Class; my $s = Data::Stream::Bulk::FileHandle->new( filehandle => file('foo.txt')->openr, ); =head1 DESCRIPTION This provides a stream API for reading lines from a file. =head1 ATTRIBUTES =over 4 =item filehandle A file handle that has been opened for reading. The stream will return lines from this file, one by one. =back =head1 METHODS =over 4 =item get_more See L. Returns the next line from the file, if it exists. =back =head1 AUTHOR Yuval Kogman =head1 COPYRIGHT AND LICENSE This software is copyright (c) 2012 by Yuval Kogman. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. =cut Path000755001750000144 011716351433 20133 5ustar00doyusers000000000000Data-Stream-Bulk-0.11/lib/Data/Stream/BulkClass.pm100644001750000144 641511716351433 21704 0ustar00doyusers000000000000Data-Stream-Bulk-0.11/lib/Data/Stream/Bulk/Pathpackage Data::Stream::Bulk::Path::Class; BEGIN { $Data::Stream::Bulk::Path::Class::AUTHORITY = 'cpan:NUFFIN'; } { $Data::Stream::Bulk::Path::Class::VERSION = '0.11'; } use Moose; # ABSTRACT: L traversal use Path::Class; use Carp qw(croak); use namespace::clean -except => 'meta'; with qw(Data::Stream::Bulk); has dir => ( isa => "Path::Class::Dir", is => "ro", required => 1, ); has depth_first => ( isa => "Bool", is => "rw", default => 1, ); has only_files => ( isa => "Bool", is => "ro", ); has chunk_size => ( isa => "Int", is => "rw", default => 250, ); has _stack => ( isa => "ArrayRef", is => "ro", default => sub { [] }, ); has _queue => ( isa => "ArrayRef", is => "ro", lazy => 1, default => sub { my $self = shift; return [ $self->dir ], }, ); sub is_done { my $self = shift; return ( @{ $self->_stack } == 0 and @{ $self->_queue } == 0 ); } sub next { my $self = shift; my $queue = $self->_queue; my $stack = $self->_stack; my $depth_first = $self->depth_first; my $only_files = $self->only_files; my $chunk_size = $self->chunk_size; my @ret; { outer: while ( @$stack ) { my $frame = $stack->[-1]; my ( $dh, $parent ) = @$frame; while ( defined(my $entry = $dh->read) ) { next if $entry eq '.' || $entry eq '..'; my $path = $parent->file($entry); if ( -d $path ) { my $dir = $parent->subdir($entry); if ( $depth_first ) { unshift @$queue, $dir; } else { push @$queue, $dir; } last outer; } else { push @ret, $path; return \@ret if @ret >= $chunk_size; } } # we're done reading this dir pop @$stack; } if ( @$queue ) { my $dir = shift @$queue; my $dh = $dir->open || croak("Can't open directory $dir: $!"); if ( $depth_first ) { push @$stack, [ $dh, $dir ]; } else { unshift @$stack, [ $dh, $dir ]; } unless ( $only_files ) { push @ret, $dir; return \@ret if @ret >= $chunk_size; } redo; } } return unless @ret; return \@ret; } __PACKAGE__->meta->make_immutable; __PACKAGE__; =pod =head1 NAME Data::Stream::Bulk::Path::Class - L traversal =head1 VERSION version 0.11 =head1 SYNOPSIS use Data::Stream::Bulk::Path::Class; my $dir = Data::Stream::Bulk::Path::Class->new( dir => Path::Class::Dir->new( ... ), ); =head1 DESCRIPTION This stream produces depth or breadth first traversal order recursion through L objects. Items are read iteratively, and a stack of open directory handles is used to keep track of state. =head1 ATTRIBUTES =over 4 =item chunk_size Defaults to 250. =item depth_first Chooses between depth first and breadth first traversal order. =item only_files If true only L items will be returned in the output streams (no directories). =back =head1 METHODS =over 4 =item is_done Returns true when no more files are left to iterate. =item next Returns the next chunk of L objects =back =head1 AUTHOR Yuval Kogman =head1 COPYRIGHT AND LICENSE This software is copyright (c) 2012 by Yuval Kogman. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. =cut __END__