URI-Fetch-0.15/000755 000765 000024 00000000000 14053745100 013360 5ustar00neilbstaff000000 000000 URI-Fetch-0.15/LICENSE000644 000765 000024 00000043663 14053745100 014401 0ustar00neilbstaff000000 000000 This software is copyright (c) 2004 by Benjamin Trott. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. Terms of the Perl programming language system itself a) the GNU General Public License as published by the Free Software Foundation; either version 1, or (at your option) any later version, or b) the "Artistic License" --- The GNU General Public License, Version 1, February 1989 --- This software is Copyright (c) 2004 by Benjamin Trott. This is free software, licensed under: The GNU General Public License, Version 1, February 1989 GNU GENERAL PUBLIC LICENSE Version 1, February 1989 Copyright (C) 1989 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The license agreements of most software companies try to keep users at the mercy of those companies. By contrast, our General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. The General Public License applies to the Free Software Foundation's software and to any other program whose authors commit to using it. You can use it for your programs, too. When we speak of free software, we are referring to freedom, not price. Specifically, the General Public License is designed to make sure that you have the freedom to give away or sell copies of free software, that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of a such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must tell them their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License Agreement applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any work containing the Program or a portion of it, either verbatim or with modifications. Each licensee is addressed as "you". 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this General Public License and to the absence of any warranty; and give any other recipients of the Program a copy of this General Public License along with the Program. You may charge a fee for the physical act of transferring a copy. 2. You may modify your copy or copies of the Program or any portion of it, and copy and distribute such modifications under the terms of Paragraph 1 above, provided that you also do the following: a) cause the modified files to carry prominent notices stating that you changed the files and the date of any change; and b) cause the whole of any work that you distribute or publish, that in whole or in part contains the Program or any part thereof, either with or without modifications, to be licensed at no charge to all third parties under the terms of this General Public License (except that you may choose to grant warranty protection to some or all third parties, at your option). c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the simplest and most usual way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this General Public License. d) You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. Mere aggregation of another independent work with the Program (or its derivative) on a volume of a storage or distribution medium does not bring the other work under the scope of these terms. 3. You may copy and distribute the Program (or a portion or derivative of it, under Paragraph 2) in object code or executable form under the terms of Paragraphs 1 and 2 above provided that you also do one of the following: a) accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Paragraphs 1 and 2 above; or, b) accompany it with a written offer, valid for at least three years, to give any third party free (except for a nominal charge for the cost of distribution) a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Paragraphs 1 and 2 above; or, c) accompany it with the information you received as to where the corresponding source code may be obtained. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form alone.) Source code for a work means the preferred form of the work for making modifications to it. For an executable file, complete source code means all the source code for all modules it contains; but, as a special exception, it need not include source code for modules which are standard libraries that accompany the operating system on which the executable file runs, or for standard header files or definitions files that accompany that operating system. 4. You may not copy, modify, sublicense, distribute or transfer the Program except as expressly provided under this General Public License. Any attempt otherwise to copy, modify, sublicense, distribute or transfer the Program is void, and will automatically terminate your rights to use the Program under this License. However, parties who have received copies, or rights to use copies, from you under this General Public License will not have their licenses terminated so long as such parties remain in full compliance. 5. By copying, distributing or modifying the Program (or any work based on the Program) you indicate your acceptance of this license to do so, and all its terms and conditions. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. 7. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of the license which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the license, you may choose any version ever published by the Free Software Foundation. 8. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 9. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 10. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS Appendix: How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to humanity, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) 19yy This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 1, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston MA 02110-1301 USA Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) 19xx name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (a program to direct compilers to make passes at assemblers) written by James Hacker. , 1 April 1989 Ty Coon, President of Vice That's all there is to it! --- The Artistic License 1.0 --- This software is Copyright (c) 2004 by Benjamin Trott. This is free software, licensed under: The Artistic License 1.0 The Artistic License Preamble The intent of this document is to state the conditions under which a Package may be copied, such that the Copyright Holder maintains some semblance of artistic control over the development of the package, while giving the users of the package the right to use and distribute the Package in a more-or-less customary fashion, plus the right to make reasonable modifications. Definitions: - "Package" refers to the collection of files distributed by the Copyright Holder, and derivatives of that collection of files created through textual modification. - "Standard Version" refers to such a Package if it has not been modified, or has been modified in accordance with the wishes of the Copyright Holder. - "Copyright Holder" is whoever is named in the copyright or copyrights for the package. - "You" is you, if you're thinking about copying or distributing this Package. - "Reasonable copying fee" is whatever you can justify on the basis of media cost, duplication charges, time of people involved, and so on. (You will not be required to justify it to the Copyright Holder, but only to the computing community at large as a market that must bear the fee.) - "Freely Available" means that no fee is charged for the item itself, though there may be fees involved in handling the item. It also means that recipients of the item may redistribute it under the same conditions they received it. 1. You may make and give away verbatim copies of the source form of the Standard Version of this Package without restriction, provided that you duplicate all of the original copyright notices and associated disclaimers. 2. You may apply bug fixes, portability fixes and other modifications derived from the Public Domain or from the Copyright Holder. A Package modified in such a way shall still be considered the Standard Version. 3. You may otherwise modify your copy of this Package in any way, provided that you insert a prominent notice in each changed file stating how and when you changed that file, and provided that you do at least ONE of the following: a) place your modifications in the Public Domain or otherwise make them Freely Available, such as by posting said modifications to Usenet or an equivalent medium, or placing the modifications on a major archive site such as ftp.uu.net, or by allowing the Copyright Holder to include your modifications in the Standard Version of the Package. b) use the modified Package only within your corporation or organization. c) rename any non-standard executables so the names do not conflict with standard executables, which must also be provided, and provide a separate manual page for each non-standard executable that clearly documents how it differs from the Standard Version. d) make other distribution arrangements with the Copyright Holder. 4. You may distribute the programs of this Package in object code or executable form, provided that you do at least ONE of the following: a) distribute a Standard Version of the executables and library files, together with instructions (in the manual page or equivalent) on where to get the Standard Version. b) accompany the distribution with the machine-readable source of the Package with your modifications. c) accompany any non-standard executables with their corresponding Standard Version executables, giving the non-standard executables non-standard names, and clearly documenting the differences in manual pages (or equivalent), together with instructions on where to get the Standard Version. d) make other distribution arrangements with the Copyright Holder. 5. You may charge a reasonable copying fee for any distribution of this Package. You may charge any fee you choose for support of this Package. You may not charge a fee for this Package itself. However, you may distribute this Package in aggregate with other (possibly commercial) programs as part of a larger (possibly commercial) software distribution provided that you do not advertise this Package as a product of your own. 6. The scripts and library files supplied as input to or produced as output from the programs of this Package do not automatically fall under the copyright of this Package, but belong to whomever generated them, and may be sold commercially, and may be aggregated with this Package. 7. C or perl subroutines supplied by you and linked into this Package shall not be considered part of this Package. 8. The name of the Copyright Holder may not be used to endorse or promote products derived from this software without specific prior written permission. 9. THIS PACKAGE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTIBILITY AND FITNESS FOR A PARTICULAR PURPOSE. The End URI-Fetch-0.15/Changes000644 000765 000024 00000010244 14053745100 014654 0ustar00neilbstaff000000 000000 Revision history for Perl module URI::Fetch 0.15 2021-05-27 NEILB - Specify min perl as 5.008001 rather than 5.008_001, which was causing problems with some versions of ExtUtils::MakeMaker. RT#133491 0.14 2021-05-26 NEILB - When Benjamin created this, he had a server set up for doing the tests, but that's gone now. So to stop all the fails, I'm skipping the live tests for now. Could do with mocking, but this is a stop-gap. 0.13_01 2021-05-25 NEILB - Changed the test suite to use some online HTTP status code test servers. Let's see how reliable they are. 0.13 2016-07-02 NEILB - Switched to using Test::RequiresInternet rather than the hand-rolled online() function that was duplicated in 2 tests. Plus it was testing whether google.com was contactable, and not the site used in the tests. 0.12 2016-06-23 NEILB - Travis support added by Jason Hall. - Added list of contributors to the doc. 0.11 2015-07-04 NEILB - Added [MetaJSON] to dist.ini, so release will include META.json. RT#105632 from ETHER++ 0.10 2014-06-23 - CPAN Testers looking good after previous developer release. - Added github repo to pod 0.09_01 2014-06-13 - If you've got caching enabled, and get a 304 response (Not Modified) with content (from the cache), then is_success() returns true. Suggested in RT#75665 - Caching now done under the original url rather than the sanitised version of it. Bug report and patch from Mario Domgoergen RT#39820 - Switched to Dist::Zilla - Reformatted Changes as per CPAN::Changes::Spec 0.09 2011-01-28 - Use $ua->env_proxy to load local proxy settings. (RT 53819) - Skip tests if we don't have a network connection. (RT 28388, 59694) - Removed sign() and auto_install() from Makefile.PL. - Removed magic svn keywords. - Added author tests (xt/) and modified SYNOPSIS for all modules to make them pass the compilation test. 0.08 2006-07-24 - Don't overwrite the User-Agent field if the caller has set it and provided the UserAgent argument. Thanks to Tatsuhiko Miyagawa for the patch. 0.071 2006-06-25 - Fixed broken is_error (broken in 0.07). Thanks to Tatsuhiko for the patch. 0.07 2006-06-18 - Use $res->header('Content-Type') instead of $res->content_type, since the latter can return an array. Thanks to Tatsuhiko Miyagawa for the patch. - $res->is_success, is_error, and is_redirect previously threw an exception when called with NoNetwork. They just work as 200 succesful request now, when the response is taken back from the cache. Thanks to Tatsuhiko Miyagawa for the patch. 0.06 2006-04-09 - Fixed issue where content-type was not stored in the cache, and was thus blank on subsequent requests. Thanks to Tatsuhiko Miyagawa for the patch. - Fixed issue with caching redirected (304) URIs. Thanks to Tatsuhiko Miyagawa for the patch. - Added is_error, is_redirect, is_success convenience methods to URI::Fetch::Response. Thanks to Tatsuhiko Miyagawa for the patch. 0.05 2006-02-24 - Added a ForceResponse option, which forces URI::Fetch->fetch to return a URI::Fetch::Response object even if the HTTP request fails for an unknown reason. Thanks to Tim Appnel for the patch. 0.04 2005-10-09 - Added Thaw and Freeze options, which allow you to define the serialization and deserialization options that are used. Thanks to Tim Appnel for the patch. 0.03 2005-05-27 - Added a NoNetwork option, allowing fetch to trust the cache completely and skip the HTTP request; or the option to do this only if the version in the cache is older than N seconds. [bradfitz] - Added a CacheEntryGrep option, to allow for not caching certain responses. [bradfitz] - Documentation fixes & clarifications. [bradfitz] 0.02 2005-05-25 - Let the caller pass in their own UserAgent and ContentAlterHook. [bradfitz] - Be more strict about invalid parameters. [bradfitz] - Documentation fixes for URI::Fetch->fetch in the SYNOPSIS. Thanks to Naoya Ito for the note. 0.01 2004-12-31 - Initial distribution. URI-Fetch-0.15/MANIFEST000644 000765 000024 00000000411 14053745100 014505 0ustar00neilbstaff000000 000000 # This file was automatically generated by Dist::Zilla::Plugin::Manifest v6.017. Changes LICENSE MANIFEST META.json META.yml Makefile.PL README dist.ini lib/URI/Fetch.pm lib/URI/Fetch/Response.pm t/00-compile.t t/01-fetch.t t/02-freezethaw.t xt/pod.t xt/synopsis.t URI-Fetch-0.15/t/000755 000765 000024 00000000000 14053745100 013623 5ustar00neilbstaff000000 000000 URI-Fetch-0.15/xt/000755 000765 000024 00000000000 14053745100 014013 5ustar00neilbstaff000000 000000 URI-Fetch-0.15/README000644 000765 000024 00000000546 14053745100 014245 0ustar00neilbstaff000000 000000 This archive contains the distribution URI-Fetch, version 0.15: Smart URI fetching/caching This software is copyright (c) 2004 by Benjamin Trott. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. This README file was generated by Dist::Zilla::Plugin::Readme v6.017. URI-Fetch-0.15/META.yml000644 000765 000024 00000001576 14053745100 014642 0ustar00neilbstaff000000 000000 --- abstract: 'Smart URI fetching/caching' author: - 'Benjamin Trott ' build_requires: Data::Dumper: '0' Test::More: '0' Test::RequiresInternet: '0.05' configure_requires: ExtUtils::MakeMaker: '0' dynamic_config: 0 generated_by: 'Dist::Zilla version 6.017, CPAN::Meta::Converter version 2.150010' license: perl meta-spec: url: http://module-build.sourceforge.net/META-spec-v1.4.html version: '1.4' name: URI-Fetch requires: Carp: '0' Class::ErrorHandler: '0' LWP::UserAgent: '0' Storable: '0' URI: '0' base: '0' constant: '0' perl: '5.008001' strict: '0' warnings: '0' resources: homepage: https://github.com/neilb/URI-Fetch repository: https://github.com/neilb/URI-Fetch.git version: '0.15' x_generated_by_perl: v5.28.2 x_serialization_backend: 'YAML::Tiny version 1.73' x_spdx_expression: 'Artistic-1.0-Perl OR GPL-1.0-or-later' URI-Fetch-0.15/lib/000755 000765 000024 00000000000 14053745100 014126 5ustar00neilbstaff000000 000000 URI-Fetch-0.15/Makefile.PL000644 000765 000024 00000002711 14053745100 015333 0ustar00neilbstaff000000 000000 # This file was automatically generated by Dist::Zilla::Plugin::MakeMaker v6.017. use strict; use warnings; use 5.008001; use ExtUtils::MakeMaker; my %WriteMakefileArgs = ( "ABSTRACT" => "Smart URI fetching/caching", "AUTHOR" => "Benjamin Trott ", "CONFIGURE_REQUIRES" => { "ExtUtils::MakeMaker" => 0 }, "DISTNAME" => "URI-Fetch", "LICENSE" => "perl", "MIN_PERL_VERSION" => "5.008001", "NAME" => "URI::Fetch", "PREREQ_PM" => { "Carp" => 0, "Class::ErrorHandler" => 0, "LWP::UserAgent" => 0, "Storable" => 0, "URI" => 0, "base" => 0, "constant" => 0, "strict" => 0, "warnings" => 0 }, "TEST_REQUIRES" => { "Data::Dumper" => 0, "Test::More" => 0, "Test::RequiresInternet" => "0.05" }, "VERSION" => "0.15", "test" => { "TESTS" => "t/*.t" } ); my %FallbackPrereqs = ( "Carp" => 0, "Class::ErrorHandler" => 0, "Data::Dumper" => 0, "LWP::UserAgent" => 0, "Storable" => 0, "Test::More" => 0, "Test::RequiresInternet" => "0.05", "URI" => 0, "base" => 0, "constant" => 0, "strict" => 0, "warnings" => 0 ); unless ( eval { ExtUtils::MakeMaker->VERSION(6.63_03) } ) { delete $WriteMakefileArgs{TEST_REQUIRES}; delete $WriteMakefileArgs{BUILD_REQUIRES}; $WriteMakefileArgs{PREREQ_PM} = \%FallbackPrereqs; } delete $WriteMakefileArgs{CONFIGURE_REQUIRES} unless eval { ExtUtils::MakeMaker->VERSION(6.52) }; WriteMakefile(%WriteMakefileArgs); URI-Fetch-0.15/META.json000644 000765 000024 00000003156 14053745100 015006 0ustar00neilbstaff000000 000000 { "abstract" : "Smart URI fetching/caching", "author" : [ "Benjamin Trott " ], "dynamic_config" : 0, "generated_by" : "Dist::Zilla version 6.017, CPAN::Meta::Converter version 2.150010", "license" : [ "perl_5" ], "meta-spec" : { "url" : "http://search.cpan.org/perldoc?CPAN::Meta::Spec", "version" : 2 }, "name" : "URI-Fetch", "prereqs" : { "configure" : { "requires" : { "ExtUtils::MakeMaker" : "0" } }, "develop" : { "requires" : { "Test::More" : "0" } }, "runtime" : { "requires" : { "Carp" : "0", "Class::ErrorHandler" : "0", "LWP::UserAgent" : "0", "Storable" : "0", "URI" : "0", "base" : "0", "constant" : "0", "perl" : "5.008001", "strict" : "0", "warnings" : "0" } }, "test" : { "requires" : { "Data::Dumper" : "0", "Test::More" : "0", "Test::RequiresInternet" : "0.05" } } }, "release_status" : "stable", "resources" : { "homepage" : "https://github.com/neilb/URI-Fetch", "repository" : { "type" : "git", "url" : "https://github.com/neilb/URI-Fetch.git", "web" : "https://github.com/neilb/URI-Fetch" } }, "version" : "0.15", "x_generated_by_perl" : "v5.28.2", "x_serialization_backend" : "Cpanel::JSON::XS version 4.24", "x_spdx_expression" : "Artistic-1.0-Perl OR GPL-1.0-or-later" } URI-Fetch-0.15/dist.ini000644 000765 000024 00000000404 14053745100 015022 0ustar00neilbstaff000000 000000 name = URI-Fetch author = Benjamin Trott license = Perl_5 copyright_holder = Benjamin Trott copyright_year = 2004 version = 0.15 [@Basic] [PkgVersion] [AutoPrereqs] [MetaJSON] [GithubMeta] [Git::Tag] tag_message= [Git::Push] URI-Fetch-0.15/lib/URI/000755 000765 000024 00000000000 14053745100 014565 5ustar00neilbstaff000000 000000 URI-Fetch-0.15/lib/URI/Fetch.pm000644 000765 000024 00000027417 14053745100 016167 0ustar00neilbstaff000000 000000 package URI::Fetch; $URI::Fetch::VERSION = '0.15'; use 5.008001; use strict; use warnings; use base qw( Class::ErrorHandler ); use LWP::UserAgent; use Carp qw( croak ); use URI; use URI::Fetch::Response; our $HAS_ZLIB; BEGIN { $HAS_ZLIB = eval "use Compress::Zlib (); 1;"; } use constant URI_OK => 200; use constant URI_MOVED_PERMANENTLY => 301; use constant URI_NOT_MODIFIED => 304; use constant URI_GONE => 410; sub fetch { my $class = shift; my($uri, %param) = @_; # get user parameters my $cache = delete $param{Cache}; my $ua = delete $param{UserAgent}; my $p_etag = delete $param{ETag}; my $p_lastmod = delete $param{LastModified}; my $content_hook = delete $param{ContentAlterHook}; my $p_no_net = delete $param{NoNetwork}; my $p_cache_grep = delete $param{CacheEntryGrep}; my $freeze = delete $param{Freeze}; my $thaw = delete $param{Thaw}; my $force = delete $param{ForceResponse}; croak("Unknown parameters: " . join(", ", keys %param)) if %param; my $ref; if ($cache) { unless ($freeze && $thaw) { require Storable; $thaw = \&Storable::thaw; $freeze = \&Storable::freeze; } if (my $blob = $cache->get($uri)) { $ref = $thaw->($blob); } } # NoNetwork support (see pod docs below for logic clarification) if ($p_no_net) { croak("Invalid NoNetworkValue (negative)") if $p_no_net < 0; if ($ref && ($p_no_net == 1 || $ref->{CacheTime} > time() - $p_no_net)) { my $fetch = URI::Fetch::Response->new; $fetch->status(URI_OK); $fetch->content($ref->{Content}); $fetch->etag($ref->{ETag}); $fetch->last_modified($ref->{LastModified}); $fetch->content_type($ref->{ContentType}); return $fetch; } return undef if $p_no_net == 1; } $ua ||= do { my $ua = LWP::UserAgent->new; $ua->agent(join '/', $class, $class->VERSION); $ua->env_proxy; $ua; }; my $req = HTTP::Request->new(GET => $uri); if ($HAS_ZLIB) { $req->header('Accept-Encoding', 'gzip'); } if (my $etag = ($p_etag || $ref->{ETag})) { $req->header('If-None-Match', $etag); } if (my $ts = ($p_lastmod || $ref->{LastModified})) { $req->if_modified_since($ts); } my $res = $ua->request($req); my $fetch = URI::Fetch::Response->new; $fetch->uri($uri); $fetch->http_status($res->code); $fetch->http_response($res); $fetch->content_type($res->header('Content-Type')); if ($res->previous && $res->previous->code == HTTP::Status::RC_MOVED_PERMANENTLY()) { $fetch->status(URI_MOVED_PERMANENTLY); $fetch->uri($res->previous->header('Location')); } elsif ($res->code == HTTP::Status::RC_GONE()) { $fetch->status(URI_GONE); $fetch->uri(undef); return $fetch; } elsif ($res->code == HTTP::Status::RC_NOT_MODIFIED()) { $fetch->status(URI_NOT_MODIFIED); $fetch->content($ref->{Content}); $fetch->etag($ref->{ETag}); $fetch->last_modified($ref->{LastModified}); $fetch->content_type($ref->{ContentType}); return $fetch; } elsif (!$res->is_success) { return $force ? $fetch : $class->error($res->message); } else { $fetch->status(URI_OK); } $fetch->last_modified($res->last_modified); $fetch->etag($res->header('ETag')); my $content = $res->content; if ($res->content_encoding && $res->content_encoding eq 'gzip') { $content = Compress::Zlib::memGunzip($content); } # let caller-defined transform hook modify the result that'll be # cached. perhaps the caller only wants the section of # HTML, or wants to change the content to a parsed datastructure # already serialized with Storable. if ($content_hook) { croak("ContentAlterHook is not a subref") unless ref $content_hook eq "CODE"; $content_hook->(\$content); } $fetch->content($content); # cache by default, if there's a cache. but let callers cancel # the cache action by defining a cache grep hook if ($cache && ($p_cache_grep ? $p_cache_grep->($fetch) : 1)) { $cache->set($uri, $freeze->({ ETag => $fetch->etag, LastModified => $fetch->last_modified, Content => $fetch->content, CacheTime => time(), ContentType => $fetch->content_type, })); } $fetch; } 1; __END__ =head1 NAME URI::Fetch - Smart URI fetching/caching =head1 SYNOPSIS use URI::Fetch; ## Simple fetch. my $res = URI::Fetch->fetch('http://example.com/atom.xml') or die URI::Fetch->errstr; do_something($res->content) if $res->is_success; ## Fetch using specified ETag and Last-Modified headers. $res = URI::Fetch->fetch('http://example.com/atom.xml', ETag => '123-ABC', LastModified => time - 3600, ) or die URI::Fetch->errstr; ## Fetch using an on-disk cache that URI::Fetch manages for you. my $cache = Cache::File->new( cache_root => '/tmp/cache' ); $res = URI::Fetch->fetch('http://example.com/atom.xml', Cache => $cache ) or die URI::Fetch->errstr; =head1 DESCRIPTION I is a smart client for fetching HTTP pages, notably syndication feeds (RSS, Atom, and others), in an intelligent, bandwidth- and time-saving way. That means: =over 4 =item * GZIP support If you have I installed, I will automatically try to download a compressed version of the content, saving bandwidth (and time). =item * I and I support If you use a local cache (see the I parameter to I), I will keep track of the I and I headers from the server, allowing you to only download pages that have been modified since the last time you checked. =item * Proper understanding of HTTP error codes Certain HTTP error codes are special, particularly when fetching syndication feeds, and well-written clients should pay special attention to them. I can only do so much for you in this regard, but it gives you the tools to be a well-written client. The response from I gives you the raw HTTP response code, along with special handling of 4 codes: =over 4 =item * 200 (OK) Signals that the content of a page/feed was retrieved successfully. =item * 301 (Moved Permanently) Signals that a page/feed has moved permanently, and that your database of feeds should be updated to reflect the new URI. =item * 304 (Not Modified) Signals that a page/feed has not changed since it was last fetched. =item * 410 (Gone) Signals that a page/feed is gone and will never be coming back, so you should stop trying to fetch it. =back =back =head2 Change from 0.09 If you make a request using a cache and get back a 304 response code (Not Modified), then if the content was returned from the cache, then C will return true, and C<$response-Econtent> will contain the cached content. I think this is the right behaviour, given the philosophy of C, but please let me (NEILB) know if you disagree. =head1 USAGE =head2 URI::Fetch->fetch($uri, %param) Fetches a page identified by the URI I<$uri>. On success, returns a I object; on failure, returns C. I<%param> can contain: =over 4 =item * LastModified =item * ETag I and I can be supplied to force the server to only return the full page if it's changed since the last request. If you're writing your own feed client, this is recommended practice, because it limits both your bandwidth use and the server's. If you'd rather not have to store the I time and I yourself, see the I parameter below (and the L above). =item * Cache If you'd like I to cache responses between requests, provide the I parameter with an object supporting the L API (e.g. I, I). Specifically, an object that supports C<$cache-Eget($key)> and C<$cache-Eset($key, $value, $expires)>. If supplied, I will store the page content, ETag, and last-modified time of the response in the cache, and will pull the content from the cache on subsequent requests if the page returns a Not-Modified response. =item * UserAgent Optional. You may provide your own LWP::UserAgent instance. Look into L if you're fetching URLs given to you by possibly malicious parties. =item * NoNetwork Optional. Controls the interaction between the cache and HTTP requests with If-Modified-Since/If-None-Match headers. Possible behaviors are: =over =item false (default) If a page is in the cache, the origin HTTP server is always checked for a fresher copy with an If-Modified-Since and/or If-None-Match header. =item C<1> If set to C<1>, the origin HTTP is never contacted, regardless of the page being in cache or not. If the page is missing from cache, the fetch method will return undef. If the page is in cache, that page will be returned, no matter how old it is. Note that setting this option means the L object will never have the http_response member set. =item C, where N E 1 The origin HTTP server is not contacted B the page is in cache B the cached page was inserted in the last N seconds. If the cached copy is older than N seconds, a normal HTTP request (full or cache check) is done. =back =item * ContentAlterHook Optional. A subref that gets called with a scalar reference to your content so you can modify the content before it's returned and before it's put in cache. For instance, you may want to only cache the EheadE section of an HTML document, or you may want to take a feed URL and cache only a pre-parsed version of it. If you modify the scalarref given to your hook and change it into a hashref, scalarref, or some blessed object, that same value will be returned to you later on not-modified responses. =item * CacheEntryGrep Optional. A subref that gets called with the I object about to be cached (with the contents already possibly transformed by your C). If your subref returns true, the page goes into the cache. If false, it doesn't. =item * Freeze =item * Thaw Optional. Subrefs that get called to serialize and deserialize, respectively, the data that will be cached. The cached data should be assumed to be an arbitrary Perl data structure, containing (potentially) references to arrays, hashes, etc. Freeze should serialize the structure into a scalar; Thaw should deserialize the scalar into a data structure. By default, I will be used for freezing and thawing the cached data structure. =item * ForceResponse Optional. A boolean that indicates a I should be returned regardless of the HTTP status. By default C is returned when a response is not a "success" (200 codes) or one of the recognized HTTP status codes listed above. The HTTP status message can then be retreived using the C method on the class. =back =head1 REPOSITORY L =head1 LICENSE I is free software; you may redistribute it and/or modify it under the same terms as Perl itself. =head1 AUTHOR & COPYRIGHT Except where otherwise noted, I is Copyright 2004 Benjamin Trott, ben+cpan@stupidfool.org. All rights reserved. Currently maintained by Neil Bowers. =head1 CONTRIBUTORS =over 4 =item * Tim Appnel =item * Mario Domgoergen =item * Karen Etheridge =item * Brad Fitzpatrick =item * Jason Hall =item * Naoya Ito =item * Tatsuhiko Miyagawa =back =cut URI-Fetch-0.15/lib/URI/Fetch/000755 000765 000024 00000000000 14053745100 015616 5ustar00neilbstaff000000 000000 URI-Fetch-0.15/lib/URI/Fetch/Response.pm000644 000765 000024 00000005730 14053745100 017757 0ustar00neilbstaff000000 000000 package URI::Fetch::Response; $URI::Fetch::Response::VERSION = '0.15'; use strict; use warnings; sub new { my $class = shift; my $feed = bless { }, $class; $feed; } sub _var { my $feed = shift; my $var = shift; $feed->{$var} = shift if @_; $feed->{$var}; } sub status { shift->_var('status', @_) } sub http_status { shift->_var('http_status', @_) } sub http_response { shift->_var('http_response', @_) } sub etag { shift->_var('etag', @_) } sub last_modified { shift->_var('last_modified', @_) } sub uri { shift->_var('uri', @_) } sub content { shift->_var('content', @_) } sub content_type { shift->_var('content_type', @_) } sub is_success { my $response = shift; if ($response->http_response) { return 1 if $response->http_response->code == 304 && defined($response->content); return $response->http_response->is_success; } return 1; } sub is_redirect { my $response = shift; return $response->http_response->is_redirect if $response->http_response; return; } sub is_error { my $response = shift; return $response->http_response->is_error if $response->http_response; return; } 1; __END__ =head1 NAME URI::Fetch::Response - Feed response for URI::Fetch =head1 SYNOPSIS use URI::Fetch; my $res = URI::Fetch->fetch('http://example.com/atom.xml') or die URI::Fetch->errstr; print $res->content; =head1 DESCRIPTION I encapsulates the response from fetching a feed using I. =head1 USAGE =head2 $res->content The contents of the feed. =head2 $res->uri The URI of the feed. If the feed was moved, this reflects the new URI; otherwise, it will match the URI that you passed to I. =head2 $res->etag The ETag that was returned in the response, if any. =head2 $res->last_modified The Last-Modified date (in seconds since the epoch) that was returned in the response, if any. =head2 $res->status The status of the response, which will match one of the following enumerations: =over 4 =item * URI::Fetch::URI_OK() =item * URI::Fetch::URI_MOVED_PERMANENTLY() =item * URI::Fetch::URI_GONE() =item * URI::Fetch::URI_NOT_MODIFIED() =back =head2 $res->http_status The HTTP status code from the response. =head2 $res->http_response The I object returned from the fetch. =head2 $res->is_success =head2 $res->is_redirect =head2 $res->is_error Wrappers around the C<$res-Eresponse> methods of the same name, for convenience. B there is one difference from the behaviour of L. If you are using a cache and get a 304 response, but the data is retrieved from the cache, then C will return true, because Ccontent> is usable. =head2 $res->content_type The Content-Type header from the response. =head1 AUTHOR & COPYRIGHT Please see the I manpage for author, copyright, and license information. =cut URI-Fetch-0.15/xt/pod.t000644 000765 000024 00000000201 14053745100 014753 0ustar00neilbstaff000000 000000 use Test::More; eval "use Test::Pod 1.00"; plan skip_all => "Test::Pod 1.00 required for testing POD" if $@; all_pod_files_ok(); URI-Fetch-0.15/xt/synopsis.t000644 000765 000024 00000000160 14053745100 016064 0ustar00neilbstaff000000 000000 use Test::More; eval "use Test::Synopsis"; plan skip_all => "Test::Synopsis required" if $@; all_synopsis_ok(); URI-Fetch-0.15/t/02-freezethaw.t000644 000765 000024 00000002516 14053745100 016377 0ustar00neilbstaff000000 000000 use strict; use Test::More skip_all => "these are not the tests you're looking for"; use Test::RequiresInternet 0.05 'httpstatuses.com' => 443; use URI::Fetch; use Data::Dumper; use constant URI_OK => 'https://httpstatuses.com/200'; my($res, $xml, $etag, $mtime); ## Test a regular fetch using a cache and alternate freeze/thaw. my $cache = My::Cache->new; $res = URI::Fetch->fetch(URI_OK, Cache => $cache, Freeze=>\&freeze, Thaw=>\&thaw); ok($res); is($res->http_status, 200); # ok($etag = $res->etag); ok($mtime = $res->last_modified); ok($xml = $res->content); ## Now hit the same URI again using the same cache and see if it has the ## the correct info to get a 304 back. $res = URI::Fetch->fetch(URI_OK, Cache => $cache, Freeze=>\&freeze, Thaw=>\&thaw); ok($res); is($res->http_status, 304); is($res->status, URI::Fetch::URI_NOT_MODIFIED()); is($res->etag, $etag); is($res->last_modified, $mtime); is($res->content, $xml); done_testing(); #--- alternate freeze/thaw routine sub freeze { my $data = shift; # ref to data structure my $d = Data::Dumper->new([$data],['data']); $d->Dump; } sub thaw { my $data; eval shift; # string from previous data dump $data; } #--- simple in memory cache object package My::Cache; sub new { bless {}, shift } sub get { $_[0]->{ $_[1] } } sub set { $_[0]->{ $_[1] } = $_[2] } URI-Fetch-0.15/t/00-compile.t000644 000765 000024 00000000075 14053745100 015657 0ustar00neilbstaff000000 000000 use strict; use Test::More tests => 1; use_ok 'URI::Fetch'; URI-Fetch-0.15/t/01-fetch.t000644 000765 000024 00000012647 14053745100 015331 0ustar00neilbstaff000000 000000 use strict; use Test::More skip_all => "these are not the tests you're looking for"; use Test::RequiresInternet 0.05 'httpstatuses.com' => 443; use Test::RequiresInternet 0.05 'httpstat.us' => 443; use URI::Fetch; use constant BASE => 'https://httpstatuses.com/'; # use constant BASE => 'https://httpstat.us/'; # use constant BASE => 'http://status.savanttools.com/'; use constant URI_OK => BASE . '200'; use constant URI_MOVED => BASE . '301'; use constant URI_GONE => 'https://httpstat.us/410'; use constant URI_ERROR => BASE . 'error.xml'; my($res, $xml, $etag, $mtime); ## Test a basic fetch. $res = URI::Fetch->fetch(URI_OK); ok($res); is($res->status, URI::Fetch::URI_OK()); is($res->http_status, 200); # ok($etag = $res->etag); ok($mtime = $res->last_modified); is($res->uri, URI_OK); ok($xml = $res->content); ## Test a fetch using last-modified. $res = URI::Fetch->fetch(URI_OK, LastModified => $mtime); ok($res); is($res->http_status, 304); is($res->status, URI::Fetch::URI_NOT_MODIFIED()); is($res->content, undef); ok(!$res->is_success); ## Test a fetch using etag. # $res = URI::Fetch->fetch(URI_OK, ETag => $etag); # ok($res); # is($res->http_status, 304); # is($res->status, URI::Fetch::URI_NOT_MODIFIED()); # is($res->content, undef); # ok(!$res->is_success); ## Test a fetch using both. # $res = URI::Fetch->fetch(URI_OK, ETag => $etag, LastModified => $mtime); # ok($res); # is($res->http_status, 304); # is($res->status, URI::Fetch::URI_NOT_MODIFIED()); # is($res->content, undef); # ok(!$res->is_success); ## Test a regular fetch using a cache. my $cache = My::Cache->new; $res = URI::Fetch->fetch(URI_OK, Cache => $cache); ok($res); is($res->http_status, 200); # ok($etag = $res->etag); ok($mtime = $res->last_modified); ok($xml = $res->content); ## Now hit the same URI again using the same cache, and hope to ## get back a not-modified response with the full content from the cache. $res = URI::Fetch->fetch(URI_OK, Cache => $cache); ok($res); is($res->http_status, 304); is($res->status, URI::Fetch::URI_NOT_MODIFIED()); # is($res->etag, $etag); is($res->last_modified, $mtime); ok($res->is_success); is($res->content, $xml); ## Test fetch of "moved permanently" resouce. $res = URI::Fetch->fetch('https://httpstat.us/301'); # $res = URI::Fetch->fetch(URI_MOVED); ok($res); is($res->status, URI::Fetch::URI_MOVED_PERMANENTLY()); # is($res->http_status, 301); is($res->uri, 'https://httpstat.us'); ## Test fetch of "gone" resource. $res = URI::Fetch->fetch('https://httpstat.us/410'); # $res = URI::Fetch->fetch(URI_GONE); ok($res); is($res->status, URI::Fetch::URI_GONE()); is($res->http_status, 410); ## Test fetch of unhandled error. $res = URI::Fetch->fetch(URI_ERROR); ok(!$res); ok(URI::Fetch->errstr); ## Test ForceResponse. $res = URI::Fetch->fetch(URI_ERROR, ForceResponse => 1); isa_ok $res, 'URI::Fetch::Response'; is $res->http_status, 404; ok $res->http_response; ## Test ContentAlterHook, wiping the cache $cache = My::Cache->new; $res = URI::Fetch->fetch(URI_OK, Cache => $cache, ContentAlterHook => sub { my $cref = shift; $$cref = "ALTERED."; }); ok($res); is($res->http_status, 200); # ok($etag = $res->etag); ok($mtime = $res->last_modified); is($res->content, "ALTERED."); ## using the same cache, should still be altered $res = URI::Fetch->fetch(URI_OK, Cache => $cache); ok($res); is($res->http_status, 304); is($res->content, "ALTERED."); ## Test NoNetwork, wiping the cache $cache = My::Cache->new; ## Content is not in cache, fetch should return undef $res = URI::Fetch->fetch(URI_OK, Cache => $cache, NoNetwork => 1); is($res, undef); ## Put the content in the cache. $res = URI::Fetch->fetch(URI_OK, Cache => $cache); ok($res); is($res->http_status, 200); ok($xml = $res->content); $res = URI::Fetch->fetch(URI_OK, Cache => $cache, NoNetwork => 1); ok($res); is($res->status, URI::Fetch::URI_OK()); is($res->content, $xml); ok(!$res->http_status); ## No http_status or http_response, because ok(!$res->http_response); ## we skipped the HTTP request entirely. ok($res->is_success); ## but still is_* should work ok(!$res->is_error); ok(!$res->is_redirect); ## Now sleep for 5 seconds, and try to get the content from the cache ## without a network connection, if the cached content is younger than ## 10 seconds. This should work. sleep 5; $res = URI::Fetch->fetch(URI_OK, Cache => $cache, NoNetwork => 10); ok($res); is($res->status, URI::Fetch::URI_OK()); is($res->content, $xml); ok(!$res->http_status); ## No http_status or http_response, because ok(!$res->http_response); ## we skipped the HTTP request entirely. ## Now try to get the content from the cache, but only if it is younger ## than 2 seconds. It is not, so we should make a full HTTP response ## with Etag and Last-modified, and get back a 304. $res = URI::Fetch->fetch(URI_OK, Cache => $cache, NoNetwork => 2); ok($res); is($res->status, URI::Fetch::URI_NOT_MODIFIED()); is($res->http_status, 304); ok($res->http_response); is($res->content, $xml); ## Test CacheEntryGrep. $cache = My::Cache->new; $res = URI::Fetch->fetch(URI_OK, Cache => $cache, CacheEntryGrep => sub { my($fetch) = @_; $fetch->uri ne URI_OK; ## Do not cache this URI. }); ok($res); is($res->http_status, 200); ## Make sure the content was not cached (it would be 304 if it were). $res = URI::Fetch->fetch(URI_OK, Cache => $cache); ok($res); is($res->http_status, 200); done_testing(); package My::Cache; sub new { bless {}, shift } sub get { $_[0]->{ $_[1] } } sub set { $_[0]->{ $_[1] } = $_[2] }