././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1698168074.996989 zfec-1.5.7.4/0000755000175100001770000000000014515776413012237 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/COPYING.GPL0000644000175100001770000005732514515776411013725 0ustar00runnerdockerThis work also comes with the added permission that you may combine it with a work licensed under the OpenSSL license (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the OpenSSL license. This work also comes with the added permission that you may combine it with a work licensed under the Eclipse Public Licence (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Eclipse Public Licence. This work also comes with the added permission that you may combine it with a work licensed under the Q Public Licence (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Q Public Licence. This work also comes with the added permission that you may combine it with a work licensed under the Apache Licence (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Apache Licence. This work also comes with the added permission that you may combine it with a work licensed under the GNU Lesser General Public License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the GNU Lesser General Public License. This work also comes with the added permission that you may combine it with a work licensed under the Zope Public License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Zope Public License. This work also comes with the added permission that you may combine it with a work licensed under the Python Software Foundation License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Python Software Foundation License. This work also comes with the added permission that you may combine it with a work licensed under the Academic Free License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Academic Free License. This work also comes with the added permission that you may combine it with a work licensed under the Apple Public Source License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Apple Public Source License. This work also comes with the added permission that you may combine it with a work licensed under the BitTorrent Open Source License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the BitTorrent Open Source License. This work also comes with the added permission that you may combine it with a work licensed under the Lucent Public License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Lucent Public License. This work also comes with the added permission that you may combine it with a work licensed under the Jabber Open Source License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Jabber Open Source License. This work also comes with the added permission that you may combine it with a work licensed under the Common Development and Distribution License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Common Development and Distribution License. This work also comes with the added permission that you may combine it with a work licensed under the Microsoft Public License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Microsoft Public License. This work also comes with the added permission that you may combine it with a work licensed under the Microsoft Reciprocal License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Microsoft Reciprocal License. This work also comes with the added permission that you may combine it with a work licensed under the Sun Industry Standards Source License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Sun Industry Standards Source License. This work also comes with the added permission that you may combine it with a work licensed under the Open Software License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Open Software License. GNU GENERAL PUBLIC LICENSE Version 2, June 1991 Copyright (C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Lesser General Public License instead.) You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you". Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does. 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. 4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. 7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. 10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) year name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (which makes passes at compilers) written by James Hacker. , 1 April 1989 Ty Coon, President of Vice This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/COPYING.TGPPL.rst0000644000175100001770000004155314515776411014774 0ustar00runnerdockerThis work also comes with the added permission that you may combine it with a work licensed under the OpenSSL license (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the OpenSSL license. This work also comes with the added permission that you may combine it with a work licensed under the Eclipse Public Licence (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Eclipse Public Licence. This work also comes with the added permission that you may combine it with a work licensed under the Q Public Licence (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Q Public Licence. This work also comes with the added permission that you may combine it with a work licensed under the Apache Licence (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Apache Licence. This work also comes with the added permission that you may combine it with a work licensed under the GNU Lesser General Public License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the GNU Lesser General Public License. This work also comes with the added permission that you may combine it with a work licensed under the Zope Public License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Zope Public License. This work also comes with the added permission that you may combine it with a work licensed under the Python Software Foundation License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Python Software Foundation License. This work also comes with the added permission that you may combine it with a work licensed under the Academic Free License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Academic Free License. This work also comes with the added permission that you may combine it with a work licensed under the Apple Public Source License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Apple Public Source License. This work also comes with the added permission that you may combine it with a work licensed under the BitTorrent Open Source License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the BitTorrent Open Source License. This work also comes with the added permission that you may combine it with a work licensed under the Lucent Public License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Lucent Public License. This work also comes with the added permission that you may combine it with a work licensed under the Jabber Open Source License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Jabber Open Source License. This work also comes with the added permission that you may combine it with a work licensed under the Common Development and Distribution License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Common Development and Distribution License. This work also comes with the added permission that you may combine it with a work licensed under the Microsoft Public License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Microsoft Public License. This work also comes with the added permission that you may combine it with a work licensed under the Microsoft Reciprocal License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Microsoft Reciprocal License. This work also comes with the added permission that you may combine it with a work licensed under the Sun Industry Standards Source License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Sun Industry Standards Source License. This work also comes with the added permission that you may combine it with a work licensed under the Open Software License (any version) and distribute the resulting combined work, as long as you follow the requirements of the licences of this work in regard to all of the resulting combined work aside from the work licensed under the Open Software License. ======================================================= Transitive Grace Period Public Licence ("TGPPL") v. 1.0 ======================================================= This Transitive Grace Period Public Licence (the "License") applies to any original work of authorship (the "Original Work") whose owner (the "Licensor") has placed the following licensing notice adjacent to the copyright notice for the Original Work: *Licensed under the Transitive Grace Period Public Licence version 1.0* 1. **Grant of Copyright License.** Licensor grants You a worldwide, royalty-free, non-exclusive, sublicensable license, for the duration of the copyright, to do the following: a. to reproduce the Original Work in copies, either alone or as part of a collective work; b. to translate, adapt, alter, transform, modify, or arrange the Original Work, thereby creating derivative works ("Derivative Works") based upon the Original Work; c. to distribute or communicate copies of the Original Work and Derivative Works to the public, with the proviso that copies of Original Work or Derivative Works that You distribute or communicate shall be licensed under this Transitive Grace Period Public Licence no later than 12 months after You distributed or communicated said copies; d. to perform the Original Work publicly; and e. to display the Original Work publicly. 2. **Grant of Patent License.** Licensor grants You a worldwide, royalty-free, non-exclusive, sublicensable license, under patent claims owned or controlled by the Licensor that are embodied in the Original Work as furnished by the Licensor, for the duration of the patents, to make, use, sell, offer for sale, have made, and import the Original Work and Derivative Works. 3. **Grant of Source Code License.** The term "Source Code" means the preferred form of the Original Work for making modifications to it and all available documentation describing how to modify the Original Work. Licensor agrees to provide a machine-readable copy of the Source Code of the Original Work along with each copy of the Original Work that Licensor distributes. Licensor reserves the right to satisfy this obligation by placing a machine-readable copy of the Source Code in an information repository reasonably calculated to permit inexpensive and convenient access by You for as long as Licensor continues to distribute the Original Work. 4. **Exclusions From License Grant.** Neither the names of Licensor, nor the names of any contributors to the Original Work, nor any of their trademarks or service marks, may be used to endorse or promote products derived from this Original Work without express prior permission of the Licensor. Except as expressly stated herein, nothing in this License grants any license to Licensor's trademarks, copyrights, patents, trade secrets or any other intellectual property. No patent license is granted to make, use, sell, offer for sale, have made, or import embodiments of any patent claims other than the licensed claims defined in Section 2. No license is granted to the trademarks of Licensor even if such marks are included in the Original Work. Nothing in this License shall be interpreted to prohibit Licensor from licensing under terms different from this License any Original Work that Licensor otherwise would have a right to license. 5. **External Deployment.** The term "External Deployment" means the use, distribution, or communication of the Original Work or Derivative Works in any way such that the Original Work or Derivative Works may be used by anyone other than You, whether those works are distributed or communicated to those persons or made available as an application intended for use over a network. As an express condition for the grants of license hereunder, You must treat any External Deployment by You of the Original Work or a Derivative Work as a distribution under section 1(c). 6. **Attribution Rights.** You must retain, in the Source Code of any Derivative Works that You create, all copyright, patent, or trademark notices from the Source Code of the Original Work, as well as any notices of licensing and any descriptive text identified therein as an "Attribution Notice." You must cause the Source Code for any Derivative Works that You create to carry a prominent Attribution Notice reasonably calculated to inform recipients that You have modified the Original Work. 7. **Warranty of Provenance and Disclaimer of Warranty.** Licensor warrants that the copyright in and to the Original Work and the patent rights granted herein by Licensor are owned by the Licensor or are sublicensed to You under the terms of this License with the permission of the contributor(s) of those copyrights and patent rights. Except as expressly stated in the immediately preceding sentence, the Original Work is provided under this License on an "AS IS" BASIS and WITHOUT WARRANTY, either express or implied, including, without limitation, the warranties of non-infringement, merchantability or fitness for a particular purpose. THE ENTIRE RISK AS TO THE QUALITY OF THE ORIGINAL WORK IS WITH YOU. This DISCLAIMER OF WARRANTY constitutes an essential part of this License. No license to the Original Work is granted by this License except under this disclaimer. 8. **Limitation of Liability.** Under no circumstances and under no legal theory, whether in tort (including negligence), contract, or otherwise, shall the Licensor be liable to anyone for any indirect, special, incidental, or consequential damages of any character arising as a result of this License or the use of the Original Work including, without limitation, damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses. This limitation of liability shall not apply to the extent applicable law prohibits such limitation. 9. **Acceptance and Termination.** If, at any time, You expressly assented to this License, that assent indicates your clear and irrevocable acceptance of this License and all of its terms and conditions. If You distribute or communicate copies of the Original Work or a Derivative Work, You must make a reasonable effort under the circumstances to obtain the express assent of recipients to the terms of this License. This License conditions your rights to undertake the activities listed in Section 1, including your right to create Derivative Works based upon the Original Work, and doing so without honoring these terms and conditions is prohibited by copyright law and international treaty. Nothing in this License is intended to affect copyright exceptions and limitations (including 'fair use' or 'fair dealing'). This License shall terminate immediately and You may no longer exercise any of the rights granted to You by this License upon your failure to honor the conditions in Section 1(c). 10. **Termination for Patent Action.** This License shall terminate automatically and You may no longer exercise any of the rights granted to You by this License as of the date You commence an action, including a cross-claim or counterclaim, against Licensor or any licensee alleging that the Original Work infringes a patent. This termination provision shall not apply for an action alleging patent infringement by combinations of the Original Work with other software or hardware. 11. **Jurisdiction, Venue and Governing Law.** Any action or suit relating to this License may be brought only in the courts of a jurisdiction wherein the Licensor resides or in which Licensor conducts its primary business, and under the laws of that jurisdiction excluding its conflict-of-law provisions. The application of the United Nations Convention on Contracts for the International Sale of Goods is expressly excluded. Any use of the Original Work outside the scope of this License or after its termination shall be subject to the requirements and penalties of copyright or patent law in the appropriate jurisdiction. This section shall survive the termination of this License. 12. **Attorneys' Fees.** In any action to enforce the terms of this License or seeking damages relating thereto, the prevailing party shall be entitled to recover its costs and expenses, including, without limitation, reasonable attorneys' fees and costs incurred in connection with such action, including any appeal of such action. This section shall survive the termination of this License. 13. **Miscellaneous.** If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. 14. **Definition of "You" in This License.** "You" throughout this License, whether in upper or lower case, means an individual or a legal entity exercising rights under, and complying with all of the terms of, this License. For legal entities, "You" includes any entity that controls, is controlled by, or is under common control with you. For purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. 15. **Right to Use.** You may use the Original Work in all ways not otherwise restricted or conditioned by this License or by law, and Licensor promises not to interfere with or be responsible for such uses by You. 16. **Modification of This License.** This License is Copyright © 2007 Zooko Wilcox-O'Hearn. Permission is granted to copy, distribute, or communicate this License without modification. Nothing in this License permits You to modify this License as applied to the Original Work or to Derivative Works. However, You may modify the text of this License and copy, distribute or communicate your modified version (the "Modified License") and apply it to other original works of authorship subject to the following conditions: (i) You may not indicate in any way that your Modified License is the "Transitive Grace Period Public Licence" or "TGPPL" and you may not use those names in the name of your Modified License; and (ii) You must replace the notice specified in the first paragraph above with the notice "Licensed under " or with a notice of your own that is not confusingly similar to the notice in this License. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/MANIFEST.in0000644000175100001770000000051714515776411013776 0ustar00runnerdockerinclude versioneer.py include zfec/_version.py include COPYING.* include changelog include TODO graft zfec include changelog NEWS.txt copyright include stridetune-bench.ba.sh stridetune-bench.py stridetune-dat.bash stridetune-graph.py include bench/*.py include tox.ini include fec.cabal Setup.lhs graft haskell include stack.yaml ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/NEWS.txt0000644000175100001770000000452714515776411013562 0ustar00runnerdockerUser visible changes in zfec. -*- outline -*- * Release 1.5.7.4 (2023-10-23) ** Python 3.7 is no longer supported. ** Python 3.12 is now supported. ** Binary packages for the macOS "universal2" and "arm64" architectures are now provided. ** The benchmark tool has been ported to Python 3. ** The GIL is now released for zfec.Encoder and zfec.Decoder initialization and for encode and decode operations. This enables multithreaded use of zfec to use multiple CPU cores for increased aggregate throughput. * Release 1.5.5 (2020-11-12) ** Upload wheel packages to the Python Package Index. Wheel packages for Python 2.7/Windows turned out to be broken (issue #34). Version 1.5.5 is simply an update to address this. No changes to zfec itself. * Release 1.5.4 (2020-09-17) ** Upload wheel packages to the Python Package Index. Wheel packages are now built for Python 2.7, 3.5, 3.7, 3.8, PyPy2, and PyPy3, for macOS, GNU/Linux, and Windows platforms, using cibuildwheel on GitHub Actions. No changes to zfec itself. * Release 1.5.3 (2018-04-07) ** Fix setup.py problem that broke builds on slackware (or other systems with setuptools-22.0.5, which is too old to know that "name" might come from the setup.cfg metadata section) * Release 1.5.2 (2018-02-06) ** Add Appveyor (CI for Windows) ** Use older C syntax to appease the Windows compilers we use on Appveyor. The previous release just didn't compile there, which broke Tahoe builds. * Release 1.5.1 (2018-02-04) ** stop using PBR, it interacts badly with Versioneer, causing bad version strings like "0+unknown" after pip install (#11) * Release 1.5.0 (2018-02-02) ** Add support for python3.5/3.6/3.7 (#1, #4, #9) ** switch to PBR for packagine (#5) ** fix unclosed-file and illegal-seek errors (#3, #6) ** fix memory leak in fec.c (#7) ** remove unused stdeb.cfg Thanks to alekibango, iammer, george-hopkins, copyninja, and theqf for bug reports and patches in this release. * Release 1.4.5 (2009-06-17) ** Bug fixes *** Fix seg fault if the Python classes Encoder or Decoder are constructed with k or m less than 1, greater than 256, or with k greater m. *** Fix several compiler warnings, add unit tests, improve Python packaging, set up more buildbots to run the unit tests on more platforms. For details about older releases, see the version control history. ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1698168074.996989 zfec-1.5.7.4/PKG-INFO0000644000175100001770000003711014515776413013336 0ustar00runnerdockerMetadata-Version: 2.1 Name: zfec Version: 1.5.7.4 Summary: An efficient, portable erasure coding tool Home-page: https://github.com/tahoe-lafs/zfec Author: Zooko O\'Whielacronx Author-email: zooko@zooko.com License: GPL-2+ Classifier: Development Status :: 5 - Production/Stable Classifier: Environment :: Console Classifier: License :: OSI Approved Classifier: License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+) Classifier: License :: DFSG approved Classifier: License :: Other/Proprietary License Classifier: Intended Audience :: Developers Classifier: Intended Audience :: End Users/Desktop Classifier: Intended Audience :: System Administrators Classifier: Operating System :: Microsoft Classifier: Operating System :: Microsoft :: Windows Classifier: Operating System :: Unix Classifier: Operating System :: POSIX :: Linux Classifier: Operating System :: POSIX Classifier: Operating System :: MacOS :: MacOS X Classifier: Operating System :: Microsoft :: Windows :: Windows NT/2000 Classifier: Operating System :: OS Independent Classifier: Natural Language :: English Classifier: Programming Language :: C Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.7 Classifier: Programming Language :: Python :: 3.8 Classifier: Topic :: Utilities Classifier: Topic :: System :: Systems Administration Classifier: Topic :: System :: Filesystems Classifier: Topic :: System :: Distributed Computing Classifier: Topic :: Software Development :: Libraries Classifier: Topic :: Communications :: Usenet News Classifier: Topic :: System :: Archiving :: Backup Classifier: Topic :: System :: Archiving :: Mirroring Classifier: Topic :: System :: Archiving Description-Content-Type: text/x-rst Provides-Extra: bench Provides-Extra: test License-File: COPYING.GPL License-File: COPYING.TGPPL.rst zfec -- efficient, portable erasure coding tool =============================================== Generate redundant blocks of information such that if some of the blocks are lost then the original data can be recovered from the remaining blocks. This package includes command-line tools, C API, Python API, and Haskell API. |build| |test| |pypi| Intro and Licence ----------------- This package implements an "erasure code", or "forward error correction code". You may use this package under the GNU General Public License, version 2 or, at your option, any later version. You may use this package under the Transitive Grace Period Public Licence, version 1.0 or, at your option, any later version. (You may choose to use this package under the terms of either licence, at your option.) See the file COPYING.GPL for the terms of the GNU General Public License, version 2. See the file COPYING.TGPPL.rst for the terms of the Transitive Grace Period Public Licence, version 1.0. The most widely known example of an erasure code is the RAID-5 algorithm which makes it so that in the event of the loss of any one hard drive, the stored data can be completely recovered. The algorithm in the zfec package has a similar effect, but instead of recovering from the loss of only a single element, it can be parameterized to choose in advance the number of elements whose loss it can tolerate. This package is largely based on the old "fec" library by Luigi Rizzo et al., which is a mature and optimized implementation of erasure coding. The zfec package makes several changes from the original "fec" package, including addition of the Python API, refactoring of the C API to support zero-copy operation, a few clean-ups and optimizations of the core code itself, and the addition of a command-line tool named "zfec". Installation ------------ ``pip install zfec`` To run the self-tests, execute ``tox`` from an unpacked source tree or git checkout. To run the tests of the Haskell API: ``cabal run test:tests`` Community --------- The source is currently available via git on the web with the command: ``git clone https://github.com/tahoe-lafs/zfec`` Please post about zfec to the Tahoe-LAFS mailing list and contribute patches: If you find a bug in zfec, please open an issue on github: Overview -------- This package performs two operations, encoding and decoding. Encoding takes some input data and expands its size by producing extra "check blocks", also called "secondary blocks". Decoding takes some data -- any combination of blocks of the original data (called "primary blocks") and "secondary blocks", and produces the original data. The encoding is parameterized by two integers, k and m. m is the total number of blocks produced, and k is how many of those blocks are necessary to reconstruct the original data. m is required to be at least 1 and at most 256, and k is required to be at least 1 and at most m. (Note that when k == m then there is no point in doing erasure coding -- it degenerates to the equivalent of the Unix "split" utility which simply splits the input into successive segments. Similarly, when k == 1 it degenerates to the equivalent of the unix "cp" utility -- each block is a complete copy of the input data.) Note that each "primary block" is a segment of the original data, so its size is 1/k'th of the size of original data, and each "secondary block" is of the same size, so the total space used by all the blocks is m/k times the size of the original data (plus some padding to fill out the last primary block to be the same size as all the others). In addition to the data contained in the blocks themselves there are also a few pieces of metadata which are necessary for later reconstruction. Those pieces are: 1. the value of K, 2. the value of M, 3. the sharenum of each block, 4. the number of bytes of padding that were used. The "zfec" command-line tool compresses these pieces of data and prepends them to the beginning of each share, so each the sharefile produced by the "zfec" command-line tool is between one and four bytes larger than the share data alone. The decoding step requires as input k of the blocks which were produced by the encoding step. The decoding step produces as output the data that was earlier input to the encoding step. Command-Line Tool ----------------- The bin/ directory contains two Unix-style, command-line tools "zfec" and "zunfec". Execute ``zfec --help`` or ``zunfec --help`` for usage instructions. Performance ----------- To run the benchmarks, execute the included bench/bench_zfec.py script with optional --k= and --m= arguments. Here's the results for an i7-12700k: ``` measuring encoding of data with K=3, M=10, encoding 1000000 bytes 1000 times in a row... Average MB/s: 364 measuring decoding of primary-only data with K=3, M=10, 1000 times in a row... Average MB/s: 1894750 measuring decoding of secondary-only data with K=3, M=10, 1000 times in a row... Average MB/s: 3298 ``` Here is a paper analyzing the performance of various erasure codes and their implementations, including zfec: http://www.usenix.org/events/fast09/tech/full_papers/plank/plank.pdf Zfec shows good performance on different machines and with different values of K and M. It also has a nice small memory footprint. API --- Each block is associated with "blocknum". The blocknum of each primary block is its index (starting from zero), so the 0'th block is the first primary block, which is the first few bytes of the file, the 1'st block is the next primary block, which is the next few bytes of the file, and so on. The last primary block has blocknum k-1. The blocknum of each secondary block is an arbitrary integer between k and 255 inclusive. (When using the Python API, if you don't specify which secondary blocks you want when invoking encode(), then it will by default provide the blocks with ids from k to m-1 inclusive.) - C API fec_encode() takes as input an array of k pointers, where each pointer points to a memory buffer containing the input data (i.e., the i'th buffer contains the i'th primary block). There is also a second parameter which is an array of the blocknums of the secondary blocks which are to be produced. (Each element in that array is required to be the blocknum of a secondary block, i.e. it is required to be >= k and < m.) The output from fec_encode() is the requested set of secondary blocks which are written into output buffers provided by the caller. Note that this fec_encode() is a "low-level" API in that it requires the input data to be provided in a set of memory buffers of exactly the right sizes. If you are starting instead with a single buffer containing all of the data then please see easyfec.py's "class Encoder" as an example of how to split a single large buffer into the appropriate set of input buffers for fec_encode(). If you are starting with a file on disk, then please see filefec.py's encode_file_stringy_easyfec() for an example of how to read the data from a file and pass it to "class Encoder". The Python interface provides these higher-level operations, as does the Haskell interface. If you implement functions to do these higher-level tasks in other languages, please send a patch to tahoe-dev@tahoe-lafs.org so that your API can be included in future releases of zfec. fec_decode() takes as input an array of k pointers, where each pointer points to a buffer containing a block. There is also a separate input parameter which is an array of blocknums, indicating the blocknum of each of the blocks which is being passed in. The output from fec_decode() is the set of primary blocks which were missing from the input and had to be reconstructed. These reconstructed blocks are written into output buffers provided by the caller. - Python API encode() and decode() take as input a sequence of k buffers, where a "sequence" is any object that implements the Python sequence protocol (such as a list or tuple) and a "buffer" is any object that implements the Python buffer protocol (such as a string or array). The contents that are required to be present in these buffers are the same as for the C API. encode() also takes a list of desired blocknums. Unlike the C API, the Python API accepts blocknums of primary blocks as well as secondary blocks in its list of desired blocknums. encode() returns a list of buffer objects which contain the blocks requested. For each requested block which is a primary block, the resulting list contains a reference to the apppropriate primary block from the input list. For each requested block which is a secondary block, the list contains a newly created string object containing that block. decode() also takes a list of integers indicating the blocknums of the blocks being passed int. decode() returns a list of buffer objects which contain all of the primary blocks of the original data (in order). For each primary block which was present in the input list, then the result list simply contains a reference to the object that was passed in the input list. For each primary block which was not present in the input, the result list contains a newly created string object containing that primary block. Beware of a "gotcha" that can result from the combination of mutable data and the fact that the Python API returns references to inputs when possible. Returning references to its inputs is efficient since it avoids making an unnecessary copy of the data, but if the object which was passed as input is mutable and if that object is mutated after the call to zfec returns, then the result from zfec -- which is just a reference to that same object -- will also be mutated. This subtlety is the price you pay for avoiding data copying. If you don't want to have to worry about this then you can simply use immutable objects (e.g. Python strings) to hold the data that you pass to zfec. - Haskell API The Haskell code is fully Haddocked, to generate the documentation, run ``runhaskell Setup.lhs haddock``. Utilities --------- The filefec.py module has a utility function for efficiently reading a file and encoding it piece by piece. This module is used by the "zfec" and "zunfec" command-line tools from the bin/ directory. Dependencies ------------ A C compiler is required. To use the Python API or the command-line tools a Python interpreter is also required. We have tested it with Python v2.7, v3.5 and v3.6. For the Haskell interface, GHC >= 6.8.1 is required. Acknowledgements ---------------- Thanks to the author of the original fec lib, Luigi Rizzo, and the folks that contributed to it: Phil Karn, Robert Morelos-Zaragoza, Hari Thirumoorthy, and Dan Rubenstein. Thanks to the Mnet hackers who wrote an earlier Python wrapper, especially Myers Carpenter and Hauke Johannknecht. Thanks to Brian Warner and Amber O'Whielacronx for help with the API, documentation, debugging, compression, and unit tests. Thanks to Adam Langley for improving the C API and contributing the Haskell API. Thanks to the creators of GCC (starting with Richard M. Stallman) and Valgrind (starting with Julian Seward) for a pair of excellent tools. Thanks to my coworkers at Allmydata -- http://allmydata.com -- Fabrice Grinda, Peter Secor, Rob Kinninmont, Brian Warner, Zandr Milewski, Justin Boreta, Mark Meras for sponsoring this work and releasing it under a Free Software licence. Thanks to Jack Lloyd, Samuel Neves, and David-Sarah Hopwood. Related Works ------------- Note: a Unix-style tool like "zfec" does only one thing -- in this case erasure coding -- and leaves other tasks to other tools. Other Unix-style tools that go well with zfec include `GNU tar`_ for archiving multiple files and directories into one file, `lzip`_ for compression, and `GNU Privacy Guard`_ for encryption or `b2sum`_ for integrity. It is important to do things in order: first archive, then compress, then either encrypt or integrity-check, then erasure code. Note that if GNU Privacy Guard is used for privacy, then it will also ensure integrity, so the use of b2sum is unnecessary in that case. Note also that you also need to do integrity checking (such as with b2sum) on the blocks that result from the erasure coding in *addition* to doing it on the file contents! (There are two different subtle failure modes -- see "more than one file can match an immutable file cap" on the `Hack Tahoe-LAFS!`_ Hall of Fame.) The `Tahoe-LAFS`_ project uses zfec as part of a complete distributed filesystem with integrated encryption, integrity, remote distribution of the blocks, directory structure, backup of changed files or directories, access control, immutable files and directories, proof-of-retrievability, and repair of damaged files and directories. `fecpp`_ is an alternative to zfec. It implements a bitwise-compatible algorithm to zfec and is BSD-licensed. .. _GNU tar: http://directory.fsf.org/project/tar/ .. _lzip: http://www.nongnu.org/lzip/lzip.html .. _GNU Privacy Guard: http://gnupg.org/ .. _b2sum: https://blake2.net/ .. _Tahoe-LAFS: https://tahoe-lafs.org .. _Hack Tahoe-LAFS!: https://tahoe-lafs.org/hacktahoelafs/ .. _fecpp: http://www.randombit.net/code/fecpp/ Enjoy! Zooko Wilcox-O'Hearn 2013-05-15 Boulder, Colorado ---- .. |pypi| image:: http://img.shields.io/pypi/v/zfec.svg :alt: PyPI release status :target: https://pypi.python.org/pypi/zfec .. |build| image:: https://github.com/tahoe-lafs/zfec/actions/workflows/build.yml/badge.svg :alt: Package Build :target: https://github.com/tahoe-lafs/zfec/actions/workflows/build.yml .. |test| image:: https://github.com/tahoe-lafs/zfec/actions/workflows/test.yml/badge.svg :alt: Tests :target: https://github.com/tahoe-lafs/zfec/actions/workflows/test.yml ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/README.rst0000644000175100001770000003333514515776411013733 0ustar00runnerdocker zfec -- efficient, portable erasure coding tool =============================================== Generate redundant blocks of information such that if some of the blocks are lost then the original data can be recovered from the remaining blocks. This package includes command-line tools, C API, Python API, and Haskell API. |build| |test| |pypi| Intro and Licence ----------------- This package implements an "erasure code", or "forward error correction code". You may use this package under the GNU General Public License, version 2 or, at your option, any later version. You may use this package under the Transitive Grace Period Public Licence, version 1.0 or, at your option, any later version. (You may choose to use this package under the terms of either licence, at your option.) See the file COPYING.GPL for the terms of the GNU General Public License, version 2. See the file COPYING.TGPPL.rst for the terms of the Transitive Grace Period Public Licence, version 1.0. The most widely known example of an erasure code is the RAID-5 algorithm which makes it so that in the event of the loss of any one hard drive, the stored data can be completely recovered. The algorithm in the zfec package has a similar effect, but instead of recovering from the loss of only a single element, it can be parameterized to choose in advance the number of elements whose loss it can tolerate. This package is largely based on the old "fec" library by Luigi Rizzo et al., which is a mature and optimized implementation of erasure coding. The zfec package makes several changes from the original "fec" package, including addition of the Python API, refactoring of the C API to support zero-copy operation, a few clean-ups and optimizations of the core code itself, and the addition of a command-line tool named "zfec". Installation ------------ ``pip install zfec`` To run the self-tests, execute ``tox`` from an unpacked source tree or git checkout. To run the tests of the Haskell API: ``cabal run test:tests`` Community --------- The source is currently available via git on the web with the command: ``git clone https://github.com/tahoe-lafs/zfec`` Please post about zfec to the Tahoe-LAFS mailing list and contribute patches: If you find a bug in zfec, please open an issue on github: Overview -------- This package performs two operations, encoding and decoding. Encoding takes some input data and expands its size by producing extra "check blocks", also called "secondary blocks". Decoding takes some data -- any combination of blocks of the original data (called "primary blocks") and "secondary blocks", and produces the original data. The encoding is parameterized by two integers, k and m. m is the total number of blocks produced, and k is how many of those blocks are necessary to reconstruct the original data. m is required to be at least 1 and at most 256, and k is required to be at least 1 and at most m. (Note that when k == m then there is no point in doing erasure coding -- it degenerates to the equivalent of the Unix "split" utility which simply splits the input into successive segments. Similarly, when k == 1 it degenerates to the equivalent of the unix "cp" utility -- each block is a complete copy of the input data.) Note that each "primary block" is a segment of the original data, so its size is 1/k'th of the size of original data, and each "secondary block" is of the same size, so the total space used by all the blocks is m/k times the size of the original data (plus some padding to fill out the last primary block to be the same size as all the others). In addition to the data contained in the blocks themselves there are also a few pieces of metadata which are necessary for later reconstruction. Those pieces are: 1. the value of K, 2. the value of M, 3. the sharenum of each block, 4. the number of bytes of padding that were used. The "zfec" command-line tool compresses these pieces of data and prepends them to the beginning of each share, so each the sharefile produced by the "zfec" command-line tool is between one and four bytes larger than the share data alone. The decoding step requires as input k of the blocks which were produced by the encoding step. The decoding step produces as output the data that was earlier input to the encoding step. Command-Line Tool ----------------- The bin/ directory contains two Unix-style, command-line tools "zfec" and "zunfec". Execute ``zfec --help`` or ``zunfec --help`` for usage instructions. Performance ----------- To run the benchmarks, execute the included bench/bench_zfec.py script with optional --k= and --m= arguments. Here's the results for an i7-12700k: ``` measuring encoding of data with K=3, M=10, encoding 1000000 bytes 1000 times in a row... Average MB/s: 364 measuring decoding of primary-only data with K=3, M=10, 1000 times in a row... Average MB/s: 1894750 measuring decoding of secondary-only data with K=3, M=10, 1000 times in a row... Average MB/s: 3298 ``` Here is a paper analyzing the performance of various erasure codes and their implementations, including zfec: http://www.usenix.org/events/fast09/tech/full_papers/plank/plank.pdf Zfec shows good performance on different machines and with different values of K and M. It also has a nice small memory footprint. API --- Each block is associated with "blocknum". The blocknum of each primary block is its index (starting from zero), so the 0'th block is the first primary block, which is the first few bytes of the file, the 1'st block is the next primary block, which is the next few bytes of the file, and so on. The last primary block has blocknum k-1. The blocknum of each secondary block is an arbitrary integer between k and 255 inclusive. (When using the Python API, if you don't specify which secondary blocks you want when invoking encode(), then it will by default provide the blocks with ids from k to m-1 inclusive.) - C API fec_encode() takes as input an array of k pointers, where each pointer points to a memory buffer containing the input data (i.e., the i'th buffer contains the i'th primary block). There is also a second parameter which is an array of the blocknums of the secondary blocks which are to be produced. (Each element in that array is required to be the blocknum of a secondary block, i.e. it is required to be >= k and < m.) The output from fec_encode() is the requested set of secondary blocks which are written into output buffers provided by the caller. Note that this fec_encode() is a "low-level" API in that it requires the input data to be provided in a set of memory buffers of exactly the right sizes. If you are starting instead with a single buffer containing all of the data then please see easyfec.py's "class Encoder" as an example of how to split a single large buffer into the appropriate set of input buffers for fec_encode(). If you are starting with a file on disk, then please see filefec.py's encode_file_stringy_easyfec() for an example of how to read the data from a file and pass it to "class Encoder". The Python interface provides these higher-level operations, as does the Haskell interface. If you implement functions to do these higher-level tasks in other languages, please send a patch to tahoe-dev@tahoe-lafs.org so that your API can be included in future releases of zfec. fec_decode() takes as input an array of k pointers, where each pointer points to a buffer containing a block. There is also a separate input parameter which is an array of blocknums, indicating the blocknum of each of the blocks which is being passed in. The output from fec_decode() is the set of primary blocks which were missing from the input and had to be reconstructed. These reconstructed blocks are written into output buffers provided by the caller. - Python API encode() and decode() take as input a sequence of k buffers, where a "sequence" is any object that implements the Python sequence protocol (such as a list or tuple) and a "buffer" is any object that implements the Python buffer protocol (such as a string or array). The contents that are required to be present in these buffers are the same as for the C API. encode() also takes a list of desired blocknums. Unlike the C API, the Python API accepts blocknums of primary blocks as well as secondary blocks in its list of desired blocknums. encode() returns a list of buffer objects which contain the blocks requested. For each requested block which is a primary block, the resulting list contains a reference to the apppropriate primary block from the input list. For each requested block which is a secondary block, the list contains a newly created string object containing that block. decode() also takes a list of integers indicating the blocknums of the blocks being passed int. decode() returns a list of buffer objects which contain all of the primary blocks of the original data (in order). For each primary block which was present in the input list, then the result list simply contains a reference to the object that was passed in the input list. For each primary block which was not present in the input, the result list contains a newly created string object containing that primary block. Beware of a "gotcha" that can result from the combination of mutable data and the fact that the Python API returns references to inputs when possible. Returning references to its inputs is efficient since it avoids making an unnecessary copy of the data, but if the object which was passed as input is mutable and if that object is mutated after the call to zfec returns, then the result from zfec -- which is just a reference to that same object -- will also be mutated. This subtlety is the price you pay for avoiding data copying. If you don't want to have to worry about this then you can simply use immutable objects (e.g. Python strings) to hold the data that you pass to zfec. - Haskell API The Haskell code is fully Haddocked, to generate the documentation, run ``runhaskell Setup.lhs haddock``. Utilities --------- The filefec.py module has a utility function for efficiently reading a file and encoding it piece by piece. This module is used by the "zfec" and "zunfec" command-line tools from the bin/ directory. Dependencies ------------ A C compiler is required. To use the Python API or the command-line tools a Python interpreter is also required. We have tested it with Python v2.7, v3.5 and v3.6. For the Haskell interface, GHC >= 6.8.1 is required. Acknowledgements ---------------- Thanks to the author of the original fec lib, Luigi Rizzo, and the folks that contributed to it: Phil Karn, Robert Morelos-Zaragoza, Hari Thirumoorthy, and Dan Rubenstein. Thanks to the Mnet hackers who wrote an earlier Python wrapper, especially Myers Carpenter and Hauke Johannknecht. Thanks to Brian Warner and Amber O'Whielacronx for help with the API, documentation, debugging, compression, and unit tests. Thanks to Adam Langley for improving the C API and contributing the Haskell API. Thanks to the creators of GCC (starting with Richard M. Stallman) and Valgrind (starting with Julian Seward) for a pair of excellent tools. Thanks to my coworkers at Allmydata -- http://allmydata.com -- Fabrice Grinda, Peter Secor, Rob Kinninmont, Brian Warner, Zandr Milewski, Justin Boreta, Mark Meras for sponsoring this work and releasing it under a Free Software licence. Thanks to Jack Lloyd, Samuel Neves, and David-Sarah Hopwood. Related Works ------------- Note: a Unix-style tool like "zfec" does only one thing -- in this case erasure coding -- and leaves other tasks to other tools. Other Unix-style tools that go well with zfec include `GNU tar`_ for archiving multiple files and directories into one file, `lzip`_ for compression, and `GNU Privacy Guard`_ for encryption or `b2sum`_ for integrity. It is important to do things in order: first archive, then compress, then either encrypt or integrity-check, then erasure code. Note that if GNU Privacy Guard is used for privacy, then it will also ensure integrity, so the use of b2sum is unnecessary in that case. Note also that you also need to do integrity checking (such as with b2sum) on the blocks that result from the erasure coding in *addition* to doing it on the file contents! (There are two different subtle failure modes -- see "more than one file can match an immutable file cap" on the `Hack Tahoe-LAFS!`_ Hall of Fame.) The `Tahoe-LAFS`_ project uses zfec as part of a complete distributed filesystem with integrated encryption, integrity, remote distribution of the blocks, directory structure, backup of changed files or directories, access control, immutable files and directories, proof-of-retrievability, and repair of damaged files and directories. `fecpp`_ is an alternative to zfec. It implements a bitwise-compatible algorithm to zfec and is BSD-licensed. .. _GNU tar: http://directory.fsf.org/project/tar/ .. _lzip: http://www.nongnu.org/lzip/lzip.html .. _GNU Privacy Guard: http://gnupg.org/ .. _b2sum: https://blake2.net/ .. _Tahoe-LAFS: https://tahoe-lafs.org .. _Hack Tahoe-LAFS!: https://tahoe-lafs.org/hacktahoelafs/ .. _fecpp: http://www.randombit.net/code/fecpp/ Enjoy! Zooko Wilcox-O'Hearn 2013-05-15 Boulder, Colorado ---- .. |pypi| image:: http://img.shields.io/pypi/v/zfec.svg :alt: PyPI release status :target: https://pypi.python.org/pypi/zfec .. |build| image:: https://github.com/tahoe-lafs/zfec/actions/workflows/build.yml/badge.svg :alt: Package Build :target: https://github.com/tahoe-lafs/zfec/actions/workflows/build.yml .. |test| image:: https://github.com/tahoe-lafs/zfec/actions/workflows/test.yml/badge.svg :alt: Tests :target: https://github.com/tahoe-lafs/zfec/actions/workflows/test.yml ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/TODO0000644000175100001770000000110314515776411012720 0ustar00runnerdocker * INSTALL doc * catch EnvironmentError when writing sharefiles and clean up * try Duff's device in _addmul1()? * memory usage analysis * What Every Programmer Should Know About Memory by Ulrich Drepper * test handling of filesystem exceptional conditions (tricky) * Jerasure ** try multiplication without a lookup table? (to save cache pressure) * conditional compilation to handle a printf that doesn't understand %Zu * try malloc() instead of alloca() (more portable, possibly better for keeping stacks aligned?) * announce on lwn, p2p-hackers * streaming mode ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1698168074.9929888 zfec-1.5.7.4/bench/0000755000175100001770000000000014515776413013316 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/bench/bench_zfec.py0000644000175100001770000001023414515776411015754 0ustar00runnerdockerfrom zfec import easyfec, Encoder, filefec, Decoder from pyutil import mathutil import os, sys from time import time FNAME="benchrandom.data" def _make_new_rand_file(size): open(FNAME, "wb").write(os.urandom(size)) def donothing(results, reslenthing): pass K=3 M=10 d = b"" ds = [] easyfecenc = None fecenc = None def _make_new_rand_data(size, k, m): global d, easyfecenc, fecenc, K, M K = k M = m d = os.urandom(size) del ds[:] ds.extend([None]*k) blocksize = mathutil.div_ceil(size, k) for i in range(k): ds[i] = d[i*blocksize:(i+1)*blocksize] ds[-1] = ds[-1] + b"\x00" * (len(ds[-2]) - len(ds[-1])) easyfecenc = easyfec.Encoder(k, m) fecenc = Encoder(k, m) from hashlib import sha256 hashers = [ sha256() for i in range(M) ] def hashem(results, reslenthing): for i, result in enumerate(results): hashers[i].update(result) def _encode_file(N): filefec.encode_file(open(FNAME, "rb"), donothing, K, M) def _encode_file_stringy(N): filefec.encode_file_stringy(open(FNAME, "rb"), donothing, K, M) def _encode_file_stringy_easyfec(N): filefec.encode_file_stringy_easyfec(open(FNAME, "rb"), donothing, K, M) def _encode_file_not_really(N): filefec.encode_file_not_really(open(FNAME, "rb"), donothing, K, M) def _encode_file_not_really_and_hash(N): filefec.encode_file_not_really_and_hash(open(FNAME, "rb"), donothing, K, M) def _encode_file_and_hash(N): filefec.encode_file(open(FNAME, "rb"), hashem, K, M) def _encode_data_not_really(N): # This function is to see how long it takes to run the Python code # that does this benchmarking and accounting and so on but not # actually do any erasure-coding, in order to get an idea of how # much overhead there is in using Python. This exercises the # basic behavior of allocating buffers to hold the secondary # shares. sz = N // K for i in range(M-K): x = '\x00' * sz def _encode_data_easyfec(N): easyfecenc.encode(d) def _encode_data_fec(N): fecenc.encode(ds) def bench(k, m): SIZE = 10**6 MAXREPS = 1000 # for f in [_encode_file_stringy_easyfec, _encode_file_stringy, _encode_file, _encode_file_not_really,]: # for f in [_encode_file,]: # for f in [_encode_file_not_really, _encode_file_not_really_and_hash, _encode_file, _encode_file_and_hash,]: # for f in [_encode_data_not_really, _encode_data_easyfec, _encode_data_fec,]: print("measuring encoding of data with K=%d, M=%d, encoding %d bytes %d times in a row..." % (k, m, SIZE, MAXREPS)) # for f in [_encode_data_fec, _encode_data_not_really]: for f in [_encode_data_fec]: def _init_func(size): return _make_new_rand_data(size, k, m) for BSIZE in [SIZE]: _init_func(BSIZE) start = time() for _ in range(MAXREPS): f(BSIZE) elapsed = (time() - start) / MAXREPS print("Average MB/s:", (BSIZE / (1024 * 1024)) / elapsed) print("measuring decoding of primary-only data with K=%d, M=%d, %d times in a row..." % (k, m, MAXREPS)) blocks = fecenc.encode(ds) sharenums = list(range(len(blocks))) decer = Decoder(k, m) start = time() for _ in range(MAXREPS): decer.decode(blocks[:k], sharenums[:k]) assert b"".join(decer.decode(blocks[:k], sharenums[:k]))[:SIZE] == b"".join(ds)[:SIZE] elapsed = (time() - start) / MAXREPS print("Average MB/s:", (sum(len(b) for b in blocks) / (1024 * 1024)) / elapsed) print("measuring decoding of secondary-only data with K=%d, M=%d, %d times in a row..." % (k, m, MAXREPS)) blocks = fecenc.encode(ds) sharenums = list(range(len(blocks))) decer = Decoder(k, m) start = time() for _ in range(MAXREPS): decer.decode(blocks[k:k+k], sharenums[k:k+k]) assert b"".join(decer.decode(blocks[k:k+k], sharenums[k:k+k]))[:SIZE] == b"".join(ds)[:SIZE] elapsed = (time() - start) / MAXREPS print("Average MB/s:", (sum(len(b) for b in blocks) / (1024 * 1024)) / elapsed) k = K m = M for arg in sys.argv: if arg.startswith('--k='): k = int(arg[len('--k='):]) if arg.startswith('--m='): m = int(arg[len('--m='):]) bench(k, m) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/changelog0000644000175100001770000000434314515776411014113 0ustar00runnerdockerWed Sep 20 10:35:00 EST 2023 exarkun@twistedmatrix.com * zfec: fix incorrect results, memory corruption, and a sometimes-crash when decoding with k = n = 256. Tue Jan 25 13:45:00 EST 2022 exarkun@twistedmatrix.com * zfec: setup: remove support for python < 3.7 Thu Nov 12 07:10:00 EST 2020 sajith@hcoop.net tagged zfec-1.5.5 Thu Sep 17 10:20:40 EST 2020 sajith@hcoop.net tagged zfec-1.5.4 Thu Dec 20 13:55:55 MST 2007 zooko@zooko.com * zfec: silence a warning when compiling on Mac OS X with gcc, and refactor a complicated #define stanza into the shared header file Thu Dec 20 13:55:32 MST 2007 zooko@zooko.com * zfec: setup: include _version.py so that the zfec package has a version number again Thu Dec 20 09:33:55 MST 2007 zooko@zooko.com tagged zfec-1.3.1 Thu Dec 20 09:31:13 MST 2007 zooko@zooko.com * zfec: dual-license under GPL and TGPPL Thu Dec 20 09:26:16 MST 2007 zooko@zooko.com tagged zfec-1.3.0 Thu Dec 20 09:25:31 MST 2007 zooko@zooko.com * zfec: add "changelog" file, which contains descriptions of the darcs patches since the last release that I think are interesting to users Thu Dec 20 09:23:41 MST 2007 zooko@zooko.com * zfec: setup: require setuptools_darcs >= 1.1.0 (fixes problem with building incomplete packages) Wed Nov 14 09:44:26 MST 2007 zooko@zooko.com * zfec: set STRIDE to 8192 after extensive experimentation on my PowerPC G4 867 MHz (256 KB L2 cache) Mon Nov 12 07:58:19 MST 2007 zooko@zooko.com * zfec: reorder the inner loop to be more cache-friendly Loop over this stride of each input block before looping over all strides of this input block. In theory, this should allow the strides of the input blocks to remain in cache while we produce all of the output blocks. Sun Nov 11 10:04:44 MST 2007 zooko@zooko.com * zfec: do encoding within a fixed window of memory in order to be cache friendly Tue Nov 13 13:13:52 MST 2007 zooko@zooko.com * zfec: conditionally-compile the right magic to use alloca() with gcc -mno-cygwin Tue Nov 13 13:11:33 MST 2007 zooko@zooko.com * zfec: setup: fix the spelling of "zfec.fec" package name Sun Nov 11 08:50:54 MST 2007 zooko@zooko.com * zfec: add a TODO note Fri Nov 9 11:17:04 MST 2007 zooko@zooko.com tagged zfec-1.2.0 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/copyright0000644000175100001770000003477014515776411014203 0ustar00runnerdockerThis package was debianized by Zooko O'Whielacronx zooko@zooko.com on Mon, 22 June 2009 23:30:00 +0000. It was originally downloaded from http://allmydata.org/trac/zfec Upstream Author: Zooko O'Whielacronx Copyright: You may use this package under the GNU General Public License, version 2 or, at your option, any later version. You may use this package under the Transitive Grace Period Public Licence, version 1.0 or, at your option, any later version. (You may choose to use this package under the terms of either licence, at your option.) See the file COPYING.GPL for the terms of the GNU General Public License, version 2. See the file COPYING.TGPPL.rst or the text of the TGPPL in this file, below, for the terms of the Transitive Grace Period Public Licence, version 1.0. ------- The Debian packaging is © 2009 Zooko O'Whielacronx - it is licensed under the same terms as the zfec source code itself. On Debian systems, the complete text of the GNU General Public License can be found in `/usr/share/common-licenses/GPL'. ------- Portions derived from the "fec" software by Luigi Rizzo, et al., the copyright notice and licence terms of which are included below for reference. fec/fec.[ch] -- forward error correction based on Vandermonde matrices 980614 (C) 1997-98 Luigi Rizzo (luigi@iet.unipi.it) Portions derived from code by Phil Karn (karn@ka9q.ampr.org), Robert Morelos-Zaragoza (robert@spectra.eng.hawaii.edu) and Hari Thirumoorthy (harit@spectra.eng.hawaii.edu), Aug 1995 Modifications by Dan Rubenstein (see Modifications.txt for their description. Modifications (C) 1998 Dan Rubenstein (drubenst@cs.umass.edu) Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE AUTHORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ------- Transitive Grace Period Public Licence ("TGPPL") v. 1.0 This Transitive Grace Period Public Licence (the "License") applies to any original work of authorship (the "Original Work") whose owner (the "Licensor") has placed the following licensing notice adjacent to the copyright notice for the Original Work: Licensed under the Transitive Grace Period Public Licence version 1.0 1. Grant of Copyright License. Licensor grants You a worldwide, royalty-free, non-exclusive, sublicensable license, for the duration of the copyright, to do the following: a. to reproduce the Original Work in copies, either alone or as part of a collective work; b. to translate, adapt, alter, transform, modify, or arrange the Original Work, thereby creating derivative works ("Derivative Works") based upon the Original Work; c. to distribute or communicate copies of the Original Work and Derivative Works to the public, with the proviso that copies of Original Work or Derivative Works that You distribute or communicate shall be licensed under this Transitive Grace Period Public Licence no later than 12 months after You distributed or communicated said copies; d. to perform the Original Work publicly; and e. to display the Original Work publicly. 2. Grant of Patent License. Licensor grants You a worldwide, royalty-free, non-exclusive, sublicensable license, under patent claims owned or controlled by the Licensor that are embodied in the Original Work as furnished by the Licensor, for the duration of the patents, to make, use, sell, offer for sale, have made, and import the Original Work and Derivative Works. 3. Grant of Source Code License. The term "Source Code" means the preferred form of the Original Work for making modifications to it and all available documentation describing how to modify the Original Work. Licensor agrees to provide a machine-readable copy of the Source Code of the Original Work along with each copy of the Original Work that Licensor distributes. Licensor reserves the right to satisfy this obligation by placing a machine-readable copy of the Source Code in an information repository reasonably calculated to permit inexpensive and convenient access by You for as long as Licensor continues to distribute the Original Work. 4. Exclusions From License Grant. Neither the names of Licensor, nor the names of any contributors to the Original Work, nor any of their trademarks or service marks, may be used to endorse or promote products derived from this Original Work without express prior permission of the Licensor. Except as expressly stated herein, nothing in this License grants any license to Licensor's trademarks, copyrights, patents, trade secrets or any other intellectual property. No patent license is granted to make, use, sell, offer for sale, have made, or import embodiments of any patent claims other than the licensed claims defined in Section 2. No license is granted to the trademarks of Licensor even if such marks are included in the Original Work. Nothing in this License shall be interpreted to prohibit Licensor from licensing under terms different from this License any Original Work that Licensor otherwise would have a right to license. 5. External Deployment. The term "External Deployment" means the use, distribution, or communication of the Original Work or Derivative Works in any way such that the Original Work or Derivative Works may be used by anyone other than You, whether those works are distributed or communicated to those persons or made available as an application intended for use over a network. As an express condition for the grants of license hereunder, You must treat any External Deployment by You of the Original Work or a Derivative Work as a distribution under section 1(c). 6. Attribution Rights. You must retain, in the Source Code of any Derivative Works that You create, all copyright, patent, or trademark notices from the Source Code of the Original Work, as well as any notices of licensing and any descriptive text identified therein as an "Attribution Notice." You must cause the Source Code for any Derivative Works that You create to carry a prominent Attribution Notice reasonably calculated to inform recipients that You have modified the Original Work. 7. Warranty of Provenance and Disclaimer of Warranty. Licensor warrants that the copyright in and to the Original Work and the patent rights granted herein by Licensor are owned by the Licensor or are sublicensed to You under the terms of this License with the permission of the contributor(s) of those copyrights and patent rights. Except as expressly stated in the immediately preceding sentence, the Original Work is provided under this License on an "AS IS" BASIS and WITHOUT WARRANTY, either express or implied, including, without limitation, the warranties of non-infringement, merchantability or fitness for a particular purpose. THE ENTIRE RISK AS TO THE QUALITY OF THE ORIGINAL WORK IS WITH YOU. This DISCLAIMER OF WARRANTY constitutes an essential part of this License. No license to the Original Work is granted by this License except under this disclaimer. 8. Limitation of Liability. Under no circumstances and under no legal theory, whether in tort (including negligence), contract, or otherwise, shall the Licensor be liable to anyone for any indirect, special, incidental, or consequential damages of any character arising as a result of this License or the use of the Original Work including, without limitation, damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses. This limitation of liability shall not apply to the extent applicable law prohibits such limitation. 9. Acceptance and Termination. If, at any time, You expressly assented to this License, that assent indicates your clear and irrevocable acceptance of this License and all of its terms and conditions. If You distribute or communicate copies of the Original Work or a Derivative Work, You must make a reasonable effort under the circumstances to obtain the express assent of recipients to the terms of this License. This License conditions your rights to undertake the activities listed in Section 1, including your right to create Derivative Works based upon the Original Work, and doing so without honoring these terms and conditions is prohibited by copyright law and international treaty. Nothing in this License is intended to affect copyright exceptions and limitations (including 'fair use' or 'fair dealing'). This License shall terminate immediately and You may no longer exercise any of the rights granted to You by this License upon your failure to honor the conditions in Section 1(c). 10. Termination for Patent Action. This License shall terminate automatically and You may no longer exercise any of the rights granted to You by this License as of the date You commence an action, including a cross-claim or counterclaim, against Licensor or any licensee alleging that the Original Work infringes a patent. This termination provision shall not apply for an action alleging patent infringement by combinations of the Original Work with other software or hardware. 11. Jurisdiction, Venue and Governing Law. Any action or suit relating to this License may be brought only in the courts of a jurisdiction wherein the Licensor resides or in which Licensor conducts its primary business, and under the laws of that jurisdiction excluding its conflict-of-law provisions. The application of the United Nations Convention on Contracts for the International Sale of Goods is expressly excluded. Any use of the Original Work outside the scope of this License or after its termination shall be subject to the requirements and penalties of copyright or patent law in the appropriate jurisdiction. This section shall survive the termination of this License. 12. Attorneys' Fees. In any action to enforce the terms of this License or seeking damages relating thereto, the prevailing party shall be entitled to recover its costs and expenses, including, without limitation, reasonable attorneys' fees and costs incurred in connection with such action, including any appeal of such action. This section shall survive the termination of this License. 13. Miscellaneous. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. 14. Definition of "You" in This License. "You" throughout this License, whether in upper or lower case, means an individual or a legal entity exercising rights under, and complying with all of the terms of, this License. For legal entities, "You" includes any entity that controls, is controlled by, or is under common control with you. For purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. 15. Right to Use. You may use the Original Work in all ways not otherwise restricted or conditioned by this License or by law, and Licensor promises not to interfere with or be responsible for such uses by You. 16. Modification of This License. This License is Copyright (c) 2007 Zooko Wilcox-O'Hearn. Permission is granted to copy, distribute, or communicate this License without modification. Nothing in this License permits You to modify this License as applied to the Original Work or to Derivative Works. However, You may modify the text of this License and copy, distribute or communicate your modified version (the "Modified License") and apply it to other original works of authorship subject to the following conditions: (i) You may not indicate in any way that your Modified License is the "Transitive Grace Period Public Licence" or "TGPPL" and you may not use those names in the name of your Modified License; and (ii) You must replace the notice specified in the first paragraph above with the notice "Licensed under " or with a notice of your own that is not confusingly similar to the notice in this License. ------- Regarding the text "License: BSD3" which appears in haskell/Codec/FEC.hs: On 2008-01-20, Zooko O'Whielacronx wrote: "Will you please give me rights to use your patches however I like? If you do so, I will immediately publish your patches under the GPL and under the TGPPL, but I will also reserve the right to use your patches however I like, or more probably to delegate that right to allmydata.com." Adam Langley replied: "Certainly. I would generally say that GPL is a little strict for this sort of thing (LGPL would be better), but I put zfec on Hackage (the Haskell CPAN) with a GPL tag. You may do with the Haskell code as you wish, including relicensing etc." ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/fec.cabal0000644000175100001770000000462314515776411013763 0ustar00runnerdockercabal-version: 3.0 name: fec version: 0.2.0 license: GPL-2.0-or-later license-file: README.rst author: Adam Langley maintainer: Adam Langley description: This code, based on zfec by Zooko, based on code by Luigi Rizzo implements an erasure code, or forward error correction code. The most widely known example of an erasure code is the RAID-5 algorithm which makes it so that in the event of the loss of any one hard drive, the stored data can be completely recovered. The algorithm in the zfec package has a similar effect, but instead of recovering from the loss of only a single element, it can be parameterized to choose in advance the number of elements whose loss it can tolerate. build-type: Simple homepage: https://github.com/tahoe-lafs/zfec synopsis: Forward error correction of ByteStrings category: Codec stability: provisional tested-with: GHC ==8.10.7 extra-source-files: COPYING.GPL COPYING.TGPPL.rst zfec/fec.h extra-doc-files: ChangeLog.md library build-depends: , base >=4.9 && <5 , bytestring >=0.10 && <0.13 , deepseq >=1.4 && <1.6 , extra >=1.7 && <1.8 exposed-modules: Codec.FEC default-language: Haskell2010 default-extensions: ForeignFunctionInterface hs-source-dirs: haskell ghc-options: -Wall c-sources: zfec/fec.c cc-options: -std=c99 include-dirs: zfec executable benchmark-zfec main-is: Main.hs ghc-options: -threaded build-depends: , base >=4.9 && <5 , bytestring >=0.10 && <0.13 , criterion >=1.1 && <1.7 , fec , random >=1.1 && <1.3 hs-source-dirs: benchmark-zfec default-language: Haskell2010 test-suite tests type: exitcode-stdio-1.0 main-is: FECTest.hs other-modules: hs-source-dirs: haskell/test ghc-options: -Wall -threaded -rtsopts -with-rtsopts=-N build-depends: , base >=4.9 && <5 , bytestring >=0.10 && <0.13 , data-serializer >=0.3 && <0.4 , fec , hspec >=2.7 && <2.12 , QuickCheck >=2.14 && <2.15 , quickcheck-instances >=0.3 && <0.4 , random >=1.1 && <1.3 default-language: Haskell2010 ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1698168074.984989 zfec-1.5.7.4/haskell/0000755000175100001770000000000014515776413013662 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1698168074.9929888 zfec-1.5.7.4/haskell/Codec/0000755000175100001770000000000014515776413014677 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/haskell/Codec/FEC.hs0000644000175100001770000003446014515776411015635 0ustar00runnerdocker{-# LANGUAGE DeriveGeneric #-} {-# LANGUAGE EmptyDataDecls #-} {-# LANGUAGE FlexibleInstances #-} {-# LANGUAGE ForeignFunctionInterface #-} {-# LANGUAGE NamedFieldPuns #-} {- | Module: Codec.FEC Copyright: Adam Langley License: GPLv2+|TGPPLv1+ (see README.rst for details) Stability: experimental The module provides k of n encoding - a way to generate (n - k) secondary blocks of data from k primary blocks such that any k blocks (primary or secondary) are sufficient to regenerate all blocks. All blocks must be the same length and you need to keep track of which blocks you have in order to tell decode. By convention, the blocks are numbered 0..(n - 1) and blocks numbered < k are the primary blocks. -} module Codec.FEC ( FECParams (paramK, paramN), initialize, fec, encode, decode, -- * Utility functions secureDivide, secureCombine, enFEC, deFEC, ) where import Control.Concurrent.Extra (Lock, newLock, withLock) import Control.DeepSeq (NFData (rnf)) import Control.Exception (Exception, throwIO) import Data.Bits (xor) import qualified Data.ByteString as B import qualified Data.ByteString.Unsafe as BU import Data.List (nub, partition, sortBy, (\\)) import Data.Word (Word8) import Foreign.C.Types (CSize (..), CUInt (..)) import Foreign.ForeignPtr ( ForeignPtr, newForeignPtr, withForeignPtr, ) import Foreign.Marshal.Alloc (allocaBytes) import Foreign.Marshal.Array (advancePtr, withArray) import Foreign.Ptr (FunPtr, Ptr, castPtr, nullPtr) import Foreign.Storable (poke, sizeOf) import GHC.Generics (Generic) import System.IO (IOMode (..), withFile) import System.IO.Unsafe (unsafePerformIO) data CFEC data FECParams = FECParams { _cfec :: !(ForeignPtr CFEC) , paramK :: Int , paramN :: Int } deriving (Generic) -- Provide an NFData instance so it's possible to use a FECParams in a -- Criterion benchmark. instance NFData FECParams where rnf FECParams{_cfec, paramK, paramN} = -- ForeignPtr has no NFData instance and I don't know how to implement -- one for it so we punt on it here. We do make it strict in the -- record definition which at least shallowly evaluates the -- ForeignPtr which is ... part of the job? rnf paramK `seq` rnf paramN instance Show FECParams where show (FECParams _ k n) = "FEC (" ++ show k ++ ", " ++ show n ++ ")" foreign import ccall unsafe "fec_init" _init :: IO () foreign import ccall unsafe "fec_new" _new :: -- | k CUInt -> -- | n CUInt -> IO (Ptr CFEC) foreign import ccall unsafe "&fec_free" _free :: FunPtr (Ptr CFEC -> IO ()) foreign import ccall unsafe "fec_encode" _encode :: Ptr CFEC -> -- | primary blocks Ptr (Ptr Word8) -> -- | (output) secondary blocks Ptr (Ptr Word8) -> -- | array of secondary block ids Ptr CUInt -> -- | length of previous CSize -> -- | block length CSize -> IO () foreign import ccall unsafe "fec_decode" _decode :: Ptr CFEC -> -- | input blocks Ptr (Ptr Word8) -> -- | output blocks Ptr (Ptr Word8) -> -- | array of input indexes Ptr CUInt -> -- | block length CSize -> IO () -- | Return true if the given @k@ and @n@ values are valid isValidConfig :: Int -> Int -> Bool isValidConfig k n | k > n = False | k < 1 = False | n < 1 = False | n > 255 = False | otherwise = True {- | The underlying library signaled that it has not been properly initialized yet. Use @initialize@ to initialize it. -} data Uninitialized = Uninitialized deriving (Ord, Eq, Show) instance Exception Uninitialized -- A lock to ensure at most one thread attempts to initialize the underlying -- library at a time. Multiple initializations are harmless but concurrent -- initializations are disallowed. _initializationLock :: Lock {-# NOINLINE _initializationLock #-} _initializationLock = unsafePerformIO newLock -- | Initialize the library. This must be done before other APIs can succeed. initialize :: IO () initialize = withLock _initializationLock _init -- | Return a FEC with the given parameters. fec :: -- | the number of primary blocks Int -> -- | the total number blocks, must be < 256 Int -> FECParams fec k n = if not (isValidConfig k n) then error $ "Invalid FEC parameters: " ++ show k ++ " " ++ show n else unsafePerformIO ( do cfec' <- _new (fromIntegral k) (fromIntegral n) -- new will return null if the library hasn't been -- initialized. if cfec' == nullPtr then throwIO Uninitialized else do params <- newForeignPtr _free cfec' return $ FECParams params k n ) -- | Create a C array of unsigned from an input array uintCArray :: [Int] -> (Ptr CUInt -> IO a) -> IO a uintCArray = withArray . map fromIntegral -- | Convert a list of ByteStrings to an array of pointers to their data byteStringsToArray :: [B.ByteString] -> (Ptr (Ptr Word8) -> IO a) -> IO a byteStringsToArray inputs f = do let l = length inputs allocaBytes (l * sizeOf (undefined :: Ptr Word8)) ( \array -> do let inner _ [] = f array inner array' (bs : bss) = BU.unsafeUseAsCString bs ( \ptr -> do poke array' $ castPtr ptr inner (advancePtr array' 1) bss ) inner array inputs ) -- | Return True iff all the given ByteStrings are the same length allByteStringsSameLength :: [B.ByteString] -> Bool allByteStringsSameLength [] = True allByteStringsSameLength (bs : bss) = all ((==) (B.length bs) . B.length) bss {- | Run the given function with a pointer to an array of @n@ pointers to buffers of size @size@. Return these buffers as a list of ByteStrings -} createByteStringArray :: -- | the number of buffers requested Int -> -- | the size of each buffer Int -> (Ptr (Ptr Word8) -> IO ()) -> IO [B.ByteString] createByteStringArray n size f = do allocaBytes (n * sizeOf (undefined :: Ptr Word8)) ( \array -> do allocaBytes (n * size) ( \ptr -> do mapM_ (\i -> poke (advancePtr array i) (advancePtr ptr (size * i))) [0 .. (n - 1)] f array mapM (\i -> B.packCStringLen (castPtr $ advancePtr ptr (i * size), size)) [0 .. (n - 1)] ) ) {- | Generate the secondary blocks from a list of the primary blocks. The primary blocks must be in order and all of the same size. There must be @k@ primary blocks. -} encode :: FECParams -> -- | a list of @k@ input blocks [B.ByteString] -> -- | (n - k) output blocks [B.ByteString] encode (FECParams params k n) inblocks | length inblocks /= k = error "Wrong number of blocks to FEC encode" | not (allByteStringsSameLength inblocks) = error "Not all inputs to FEC encode are the same length" | otherwise = unsafePerformIO ( do let sz = B.length $ head inblocks withForeignPtr params ( \cfec' -> do byteStringsToArray inblocks ( \src -> do createByteStringArray (n - k) sz ( \fecs -> do uintCArray [k .. (n - 1)] ( \block_nums -> do _encode cfec' src fecs block_nums (fromIntegral (n - k)) $ fromIntegral sz ) ) ) ) ) -- | A sort function for tagged assoc lists sortTagged :: [(Int, a)] -> [(Int, a)] sortTagged = sortBy (\a b -> compare (fst a) (fst b)) {- | Reorder the given list so that elements with tag numbers < the first argument have an index equal to their tag number (if possible) -} reorderPrimaryBlocks :: Int -> [(Int, a)] -> [(Int, a)] reorderPrimaryBlocks n blocks = inner (sortTagged pBlocks) sBlocks [] where (pBlocks, sBlocks) = partition (\(tag, _) -> tag < n) blocks inner [] sBlocks' acc = acc ++ sBlocks' inner pBlocks' [] acc = acc ++ pBlocks' inner pBlocks'@((tag, a) : ps) sBlocks'@(s : ss) acc = if length acc == tag then inner ps sBlocks' (acc ++ [(tag, a)]) else inner pBlocks' ss (acc ++ [s]) {- | Recover the primary blocks from a list of @k@ blocks. Each block must be tagged with its number (see the module comments about block numbering) -} decode :: FECParams -> -- | a list of @k@ blocks and their index [(Int, B.ByteString)] -> -- | a list the @k@ primary blocks [B.ByteString] decode (FECParams params k n) inblocks | length (nub $ map fst inblocks) /= length inblocks = error "Duplicate input blocks in FEC decode" | any ((\f -> f < 0 || f >= n) . fst) inblocks = error "Invalid block numbers in FEC decode" | length inblocks /= k = error "Wrong number of blocks to FEC decode" | not (allByteStringsSameLength $ map snd inblocks) = error "Not all inputs to FEC decode are same length" | otherwise = unsafePerformIO ( do let sz = B.length $ snd $ head inblocks inblocks' = reorderPrimaryBlocks k inblocks presentBlocks = map fst inblocks' withForeignPtr params ( \cfec' -> do byteStringsToArray (map snd inblocks') ( \src -> do b <- createByteStringArray (n - k) sz ( \out -> do uintCArray presentBlocks ( \block_nums -> do _decode cfec' src out block_nums $ fromIntegral sz ) ) let blocks = [0 .. (n - 1)] \\ presentBlocks tagged = zip blocks b allBlocks = sortTagged $ tagged ++ inblocks' return $ take k $ map snd allBlocks ) ) ) {- | Break a ByteString into @n@ parts, equal in length to the original, such that all @n@ are required to reconstruct the original, but having less than @n@ parts reveals no information about the orginal. This code works in IO monad because it needs a source of random bytes, which it gets from /dev/urandom. If this file doesn't exist an exception results Not terribly fast - probably best to do it with short inputs (e.g. an encryption key) -} secureDivide :: -- | the number of parts requested Int -> -- | the data to be split B.ByteString -> IO [B.ByteString] secureDivide n input | n < 0 = error "secureDivide called with negative number of parts" | otherwise = withFile "/dev/urandom" ReadMode ( \handle -> do let inner 1 bs = return [bs] inner n' bs = do mask <- B.hGet handle (B.length bs) let masked = B.pack $ B.zipWith xor bs mask rest <- inner (n' - 1) masked return (mask : rest) inner n input ) {- | Reverse the operation of secureDivide. The order of the inputs doesn't matter, but they must all be the same length -} secureCombine :: [B.ByteString] -> B.ByteString secureCombine [] = error "Passed empty list of inputs to secureCombine" secureCombine [a] = a secureCombine [a, b] = B.pack $ B.zipWith xor a b secureCombine (a : rest) = B.pack $ B.zipWith xor a $ secureCombine rest {- | A utility function which takes an arbitary input and FEC encodes it into a number of blocks. The order the resulting blocks doesn't matter so long as you have enough to present to @deFEC@. -} enFEC :: -- | the number of blocks required to reconstruct Int -> -- | the total number of blocks Int -> -- | the data to divide B.ByteString -> -- | the resulting blocks [B.ByteString] enFEC k n input = taggedPrimaryBlocks ++ taggedSecondaryBlocks where taggedPrimaryBlocks = zipWith B.cons [0 ..] primaryBlocks taggedSecondaryBlocks = zipWith B.cons [(fromIntegral k) ..] secondaryBlocks remainder = B.length input `mod` k paddingLength = if remainder >= 1 then k - remainder else k paddingBytes = B.replicate (paddingLength - 1) 0 `B.append` B.singleton (fromIntegral paddingLength) divide a bs | B.null bs = [] | otherwise = B.take a bs : divide a (B.drop a bs) input' = input `B.append` paddingBytes blockSize = B.length input' `div` k primaryBlocks = divide blockSize input' secondaryBlocks = encode params primaryBlocks params = fec k n -- | Reverses the operation of @enFEC@. deFEC :: -- | the number of blocks required (matches call to @enFEC@) Int -> -- | the total number of blocks (matches call to @enFEC@) Int -> -- | a list of k, or more, blocks from @enFEC@ [B.ByteString] -> B.ByteString deFEC k n inputs | length inputs < k = error "Too few inputs to deFEC" | otherwise = B.take (B.length fecOutput - paddingLength) fecOutput where paddingLength = fromIntegral $ B.last fecOutput inputs' = take k inputs taggedInputs = map (\bs -> (fromIntegral $ B.head bs, B.tail bs)) inputs' fecOutput = B.concat $ decode params taggedInputs params = fec k n ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1698168074.9929888 zfec-1.5.7.4/haskell/test/0000755000175100001770000000000014515776413014641 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/haskell/test/FECTest.hs0000644000175100001770000001311614515776411016432 0ustar00runnerdocker{-# LANGUAGE DerivingStrategies #-} module Main where import Test.Hspec (describe, hspec, it, parallel) import qualified Codec.FEC as FEC import qualified Data.ByteString as B import qualified Data.ByteString.Lazy as BL import Data.List (sortOn) import Data.Word (Word16, Word8) import System.Random (Random (randoms), mkStdGen) import Test.QuickCheck ( Arbitrary (arbitrary), Property, Testable (property), choose, conjoin, once, withMaxSuccess, (===), ) import Test.QuickCheck.Monadic (assert, monadicIO, run) -- Imported for the orphan Arbitrary ByteString instance. import Control.Monad (replicateM_) import Test.QuickCheck.Instances.ByteString () -- | Valid ZFEC parameters. data Params = Params { required :: Int -- aka k , total :: Int -- aka n } deriving (Show, Ord, Eq) -- | A somewhat efficient generator for valid ZFEC parameters. instance Arbitrary Params where arbitrary = choose (1, 255) >>= \req -> Params req <$> choose (req, 255) randomTake :: Int -> Int -> [a] -> [a] randomTake seed n values = map snd $ take n sortedValues where sortedValues = sortOn fst taggedValues taggedValues = zip rnds values rnds :: [Float] rnds = randoms gen gen = mkStdGen seed {- | Any combination of the inputs blocks and the output blocks from @FEC.encode@, as long as there are at least @k@ of them, can be recombined using @FEC.decode@ to produce the original input blocks. -} testFEC :: -- | The FEC parameters to exercise. FEC.FECParams -> -- | The length of the blocks to exercise. Word16 -> -- | A random seed to use to be able to vary the choice of which blocks to -- try to decode. Int -> -- | True if the encoded input was reconstructed by decoding, False -- otherwise. Bool testFEC fec len seed = FEC.decode fec someTaggedBlocks == origBlocks where -- Construct some blocks. Each will just be the byte corresponding to the -- block number repeated to satisfy the requested length. origBlocks = B.replicate (fromIntegral len) . fromIntegral <$> [0 .. (FEC.paramK fec - 1)] -- Encode the data to produce the "secondary" blocks which (might) add -- redundancy to the original blocks. secondaryBlocks = FEC.encode fec origBlocks -- Tag each block with its block number because the decode API requires -- this information. taggedBlocks = zip [0 ..] (origBlocks ++ secondaryBlocks) -- Choose enough of the tagged blocks (some combination of original and -- secondary) to try to use for decoding. someTaggedBlocks = randomTake seed (FEC.paramK fec) taggedBlocks -- | @FEC.secureDivide@ is the inverse of @FEC.secureCombine@. prop_divide :: Word16 -> Word8 -> Word8 -> Property prop_divide size byte divisor = monadicIO $ do let input = B.replicate (fromIntegral size + 1) byte parts <- run $ FEC.secureDivide (fromIntegral divisor) input assert (FEC.secureCombine parts == input) -- | @FEC.encode@ is the inverse of @FEC.decode@. prop_decode :: Params -> Word16 -> Int -> Property prop_decode (Params req tot) len seed = property $ do testFEC fec len seed === True where fec = FEC.fec req tot -- | @FEC.enFEC@ is the inverse of @FEC.deFEC@. prop_deFEC :: Params -> B.ByteString -> Property prop_deFEC (Params req tot) testdata = FEC.deFEC req tot minimalShares === testdata where allShares = FEC.enFEC req tot testdata minimalShares = take req allShares prop_primary_copies :: Params -> BL.ByteString -> Property prop_primary_copies (Params _ tot) primary = property $ do conjoin $ (BL.toStrict primary ===) <$> secondary where fec = FEC.fec 1 tot secondary = FEC.encode fec [BL.toStrict primary] main :: IO () main = do -- Be sure to do the required zfec initialization first. FEC.initialize hspec . parallel $ do describe "encode" $ do -- This test originally caught a bug in multi-threaded -- initialization of the C library. Since it is in the -- initialization codepath, it cannot catch the bug if it runs -- after initialization has happened. So we put it first in the -- suite and hope that nothing manages to get in before this. -- -- Since the bug has to do with multi-threaded use, we also make a -- lot of copies of this property so hspec can run them in parallel -- (QuickCheck will not do anything in parallel inside a single -- property). -- -- Still, there's non-determinism and the behavior is only revealed -- by this property sometimes. replicateM_ 20 $ it "returns copies of the primary block for all 1 of N encodings" $ withMaxSuccess 5 prop_primary_copies describe "secureCombine" $ do -- secureDivide is insanely slow and memory hungry for large inputs, -- like QuickCheck will find with it as currently defined. Just pass -- some small inputs. It's not clear it's worth fixing (or even -- keeping) thesefunctions. They don't seem to be used by anything. -- Why are they here? it "is the inverse of secureDivide n" $ once $ prop_divide 1024 65 3 describe "deFEC" $ do replicateM_ 10 $ it "is the inverse of enFEC" $ property prop_deFEC describe "decode" $ replicateM_ 10 $ do it "is (nearly) the inverse of encode" $ property $ prop_decode it "works with required=255" $ property $ prop_decode (Params 255 255) ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1698168074.996989 zfec-1.5.7.4/setup.cfg0000644000175100001770000000350114515776413014057 0ustar00runnerdocker[metadata] summary = efficient, portable erasure coding tool description_file = README.rst long_description_content_type = text/x-rst author = Zooko O\'Whielacronx author_email = zooko@zooko.com home_page = https://tahoe-lafs.org/trac/zfec license = GPL-2+ python_requires = >=3.8, <3.13 classifier = Development Status :: 5 - Production/Stable Environment :: Console License :: OSI Approved License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+) License :: DFSG approved License :: Other/Proprietary License Intended Audience :: Developers Intended Audience :: End Users/Desktop Intended Audience :: System Administrators Operating System :: Microsoft Operating System :: Microsoft :: Windows Operating System :: Unix Operating System :: POSIX :: Linux Operating System :: POSIX Operating System :: MacOS :: MacOS X Operating System :: Microsoft :: Windows :: Windows NT/2000 Operating System :: OS Independent Natural Language :: English Programming Language :: C Programming Language :: Python Programming Language :: Python :: 3 Programming Language :: Python :: 3.7 Programming Language :: Python :: 3.8 Topic :: Utilities Topic :: System :: Systems Administration Topic :: System :: Filesystems Topic :: System :: Distributed Computing Topic :: Software Development :: Libraries Topic :: Communications :: Usenet News Topic :: System :: Archiving :: Backup Topic :: System :: Archiving :: Mirroring Topic :: System :: Archiving [options] packages = find: include_package_data = True [options.entry_points] console_scripts = zfec = zfec.cmdline_zfec:main zunfec = zfec.cmdline_zunfec:main [versioneer] VCS = git style = pep440-pre versionfile_source = zfec/_version.py versionfile_build = zfec/_version.py tag_prefix = zfec- parentdir_prefix = zfec- [egg_info] tag_build = tag_date = 0 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/setup.py0000644000175100001770000000331014515776411013744 0ustar00runnerdockerfrom setuptools import setup from setuptools.extension import Extension import sys import os import versioneer DEBUGMODE = False if "--debug" in sys.argv: DEBUGMODE = True sys.argv.remove("--debug") extra_link_args = [] extra_compile_args = [] define_macros = [] undef_macros = [] for arg in sys.argv: if arg.startswith("--stride="): stride = int(arg[len("--stride="):]) define_macros.append(('STRIDE', stride)) sys.argv.remove(arg) break extra_compile_args.append("-std=c99") if DEBUGMODE: extra_compile_args.append("-O0") extra_compile_args.append("-g") extra_compile_args.append("-Wall") extra_compile_args.append("-Wextra") extra_link_args.append("-g") undef_macros.append('NDEBUG') extensions = [ Extension( "zfec._fec", [ "zfec/fec.c", "zfec/_fecmodule.c" ], include_dirs=["zfec/"], extra_link_args=extra_link_args, extra_compile_args=extra_compile_args, define_macros=define_macros, undef_macros=undef_macros ) ] # Most of our metadata lives in setup.cfg [metadata]. We put "name" here # because the setuptools-22.0.5 on slackware can't find it there, which breaks # packaging. We put "version" here so that Versioneer works correctly. setup( name="zfec", version=versioneer.get_version(), description="An efficient, portable erasure coding tool", long_description=open('README.rst', 'r').read(), url="https://github.com/tahoe-lafs/zfec", extras_require={ "bench": ["pyutil >= 3.0.0"], "test": ["twisted", "pyutil >= 3.0.0", "hypothesis"], }, ext_modules=extensions, cmdclass=versioneer.get_cmdclass(), ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/stack.yaml0000644000175100001770000000175714515776411014240 0ustar00runnerdocker# For more information, see: https://github.com/commercialhaskell/stack/blob/release/doc/yaml_configuration.md # Specifies the GHC version and set of packages available (e.g., lts-3.5, nightly-2015-09-21, ghc-7.10.2) resolver: lts-5.2 # Local packages, usually specified by relative directory name packages: - '.' # Packages to be pulled from upstream that are not in the resolver (e.g., acme-missiles-0.3) extra-deps: [] # Override default flag values for local packages and extra-deps flags: {} # Extra package databases containing global packages extra-package-dbs: [] # Control whether we use the GHC we find on the path # system-ghc: true # Require a specific version of stack, using version ranges # require-stack-version: -any # Default # require-stack-version: >= 0.1.10.0 # Override the architecture used by stack, especially useful on Windows # arch: i386 # arch: x86_64 # Extra directories used by stack for building # extra-include-dirs: [/path/to/dir] # extra-lib-dirs: [/path/to/dir] ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/stridetune-bench.ba.sh0000755000175100001770000000073714515776411016427 0ustar00runnerdocker#!/bin/bash /bin/rm -rf ./benchresults mkdir benchresults STRIDE=32 while [ $(( $STRIDE < 32769 )) ] ; do /bin/rm -rf build rm zfec/_fec.so /bin/rm -rf instdir mkdir instdir PYTHONPATH=instdir ./setup.py develop --install-dir=instdir --stride=${STRIDE} >/dev/null echo $STRIDE PYTHONPATH=instdir python -OO ./bench/bench_zfec.py >> benchresults/comp_0-stride_$STRIDE tail -1 benchresults/comp_0-stride_$STRIDE STRIDE=$(( $STRIDE + 32 )) done ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/stridetune-bench.py0000755000175100001770000000221114515776411016051 0ustar00runnerdocker#!/usr/bin/env python import bisect, random, os, re from pyutil import fileutil assert not os.path.exists("benchresults") os.mkdir("benchresults") MIN=512 MAX=1024 results = {} R=re.compile("ave rate: ([1-9][0-9]*)") def measure(stride): fileutil.rm_dir("build") fileutil.rm_dir("instdir") fileutil.remove_if_possible(os.path.join("zfec", "_fec.so")) fileutil.make_dirs("instdir") fname = os.path.join("benchresults", "comp_0-stride_%d"%stride) os.system("PYTHONPATH=instdir ./setup.py develop --install-dir=instdir --stride=%d >/dev/null" % stride) os.system("PYTHONPATH=instdir python -OO ./bench/bench_zfec.py >> %s" % fname) inf = open(fname, "rU") for l in inf: m = R.search(l) if m: result = int(m.group(1)) if results.has_key(stride): print "stride: %d, results: %d (dup %d)" % (stride, result, results[stride]) else: print "stride: %d, results: %d" % (stride, result) results[stride] = result break measure(MIN) measure(MAX) while True: stride = random.randrange(MIN, MAX+1) measure(stride) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/stridetune-dat.bash0000755000175100001770000000034414515776411016034 0ustar00runnerdocker#!/bin/bash DAT=stridetune.dat rm -f $DAT for F in benchresults/comp_0-stride_*; do NUM=${F:27} for M in `grep "^N: " benchresults/comp_0-stride_$NUM | cut -d':' -f10-` ; do echo "$NUM $M" >> $DAT done done ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/stridetune-graph.py0000755000175100001770000000032414515776411016076 0ustar00runnerdocker#!/usr/bin/env python from pyx import * def g(f): g=graph.graphxy(width=16, x=graph.axis.linear(), y=graph.axis.linear()) g.plot([graph.data.file(f, x=1, y=2)]) g.writeEPSfile(f+'.eps') g('stridetune.dat') ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/tox.ini0000644000175100001770000000021414515776411013545 0ustar00runnerdocker[tox] envlist = py37,py38,pypy3 skip_missing_interpreters = True [testenv] deps = .[test] usedevelop = True commands = trial zfec ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/versioneer.py0000644000175100001770000025122514515776411014777 0ustar00runnerdocker # Version: 0.29 """The Versioneer - like a rocketeer, but for versions. The Versioneer ============== * like a rocketeer, but for versions! * https://github.com/python-versioneer/python-versioneer * Brian Warner * License: Public Domain (Unlicense) * Compatible with: Python 3.7, 3.8, 3.9, 3.10, 3.11 and pypy3 * [![Latest Version][pypi-image]][pypi-url] * [![Build Status][travis-image]][travis-url] This is a tool for managing a recorded version number in setuptools-based python projects. The goal is to remove the tedious and error-prone "update the embedded version string" step from your release process. Making a new release should be as easy as recording a new tag in your version-control system, and maybe making new tarballs. ## Quick Install Versioneer provides two installation modes. The "classic" vendored mode installs a copy of versioneer into your repository. The experimental build-time dependency mode is intended to allow you to skip this step and simplify the process of upgrading. ### Vendored mode * `pip install versioneer` to somewhere in your $PATH * A [conda-forge recipe](https://github.com/conda-forge/versioneer-feedstock) is available, so you can also use `conda install -c conda-forge versioneer` * add a `[tool.versioneer]` section to your `pyproject.toml` or a `[versioneer]` section to your `setup.cfg` (see [Install](INSTALL.md)) * Note that you will need to add `tomli; python_version < "3.11"` to your build-time dependencies if you use `pyproject.toml` * run `versioneer install --vendor` in your source tree, commit the results * verify version information with `python setup.py version` ### Build-time dependency mode * `pip install versioneer` to somewhere in your $PATH * A [conda-forge recipe](https://github.com/conda-forge/versioneer-feedstock) is available, so you can also use `conda install -c conda-forge versioneer` * add a `[tool.versioneer]` section to your `pyproject.toml` or a `[versioneer]` section to your `setup.cfg` (see [Install](INSTALL.md)) * add `versioneer` (with `[toml]` extra, if configuring in `pyproject.toml`) to the `requires` key of the `build-system` table in `pyproject.toml`: ```toml [build-system] requires = ["setuptools", "versioneer[toml]"] build-backend = "setuptools.build_meta" ``` * run `versioneer install --no-vendor` in your source tree, commit the results * verify version information with `python setup.py version` ## Version Identifiers Source trees come from a variety of places: * a version-control system checkout (mostly used by developers) * a nightly tarball, produced by build automation * a snapshot tarball, produced by a web-based VCS browser, like github's "tarball from tag" feature * a release tarball, produced by "setup.py sdist", distributed through PyPI Within each source tree, the version identifier (either a string or a number, this tool is format-agnostic) can come from a variety of places: * ask the VCS tool itself, e.g. "git describe" (for checkouts), which knows about recent "tags" and an absolute revision-id * the name of the directory into which the tarball was unpacked * an expanded VCS keyword ($Id$, etc) * a `_version.py` created by some earlier build step For released software, the version identifier is closely related to a VCS tag. Some projects use tag names that include more than just the version string (e.g. "myproject-1.2" instead of just "1.2"), in which case the tool needs to strip the tag prefix to extract the version identifier. For unreleased software (between tags), the version identifier should provide enough information to help developers recreate the same tree, while also giving them an idea of roughly how old the tree is (after version 1.2, before version 1.3). Many VCS systems can report a description that captures this, for example `git describe --tags --dirty --always` reports things like "0.7-1-g574ab98-dirty" to indicate that the checkout is one revision past the 0.7 tag, has a unique revision id of "574ab98", and is "dirty" (it has uncommitted changes). The version identifier is used for multiple purposes: * to allow the module to self-identify its version: `myproject.__version__` * to choose a name and prefix for a 'setup.py sdist' tarball ## Theory of Operation Versioneer works by adding a special `_version.py` file into your source tree, where your `__init__.py` can import it. This `_version.py` knows how to dynamically ask the VCS tool for version information at import time. `_version.py` also contains `$Revision$` markers, and the installation process marks `_version.py` to have this marker rewritten with a tag name during the `git archive` command. As a result, generated tarballs will contain enough information to get the proper version. To allow `setup.py` to compute a version too, a `versioneer.py` is added to the top level of your source tree, next to `setup.py` and the `setup.cfg` that configures it. This overrides several distutils/setuptools commands to compute the version when invoked, and changes `setup.py build` and `setup.py sdist` to replace `_version.py` with a small static file that contains just the generated version data. ## Installation See [INSTALL.md](./INSTALL.md) for detailed installation instructions. ## Version-String Flavors Code which uses Versioneer can learn about its version string at runtime by importing `_version` from your main `__init__.py` file and running the `get_versions()` function. From the "outside" (e.g. in `setup.py`), you can import the top-level `versioneer.py` and run `get_versions()`. Both functions return a dictionary with different flavors of version information: * `['version']`: A condensed version string, rendered using the selected style. This is the most commonly used value for the project's version string. The default "pep440" style yields strings like `0.11`, `0.11+2.g1076c97`, or `0.11+2.g1076c97.dirty`. See the "Styles" section below for alternative styles. * `['full-revisionid']`: detailed revision identifier. For Git, this is the full SHA1 commit id, e.g. "1076c978a8d3cfc70f408fe5974aa6c092c949ac". * `['date']`: Date and time of the latest `HEAD` commit. For Git, it is the commit date in ISO 8601 format. This will be None if the date is not available. * `['dirty']`: a boolean, True if the tree has uncommitted changes. Note that this is only accurate if run in a VCS checkout, otherwise it is likely to be False or None * `['error']`: if the version string could not be computed, this will be set to a string describing the problem, otherwise it will be None. It may be useful to throw an exception in setup.py if this is set, to avoid e.g. creating tarballs with a version string of "unknown". Some variants are more useful than others. Including `full-revisionid` in a bug report should allow developers to reconstruct the exact code being tested (or indicate the presence of local changes that should be shared with the developers). `version` is suitable for display in an "about" box or a CLI `--version` output: it can be easily compared against release notes and lists of bugs fixed in various releases. The installer adds the following text to your `__init__.py` to place a basic version in `YOURPROJECT.__version__`: from ._version import get_versions __version__ = get_versions()['version'] del get_versions ## Styles The setup.cfg `style=` configuration controls how the VCS information is rendered into a version string. The default style, "pep440", produces a PEP440-compliant string, equal to the un-prefixed tag name for actual releases, and containing an additional "local version" section with more detail for in-between builds. For Git, this is TAG[+DISTANCE.gHEX[.dirty]] , using information from `git describe --tags --dirty --always`. For example "0.11+2.g1076c97.dirty" indicates that the tree is like the "1076c97" commit but has uncommitted changes (".dirty"), and that this commit is two revisions ("+2") beyond the "0.11" tag. For released software (exactly equal to a known tag), the identifier will only contain the stripped tag, e.g. "0.11". Other styles are available. See [details.md](details.md) in the Versioneer source tree for descriptions. ## Debugging Versioneer tries to avoid fatal errors: if something goes wrong, it will tend to return a version of "0+unknown". To investigate the problem, run `setup.py version`, which will run the version-lookup code in a verbose mode, and will display the full contents of `get_versions()` (including the `error` string, which may help identify what went wrong). ## Known Limitations Some situations are known to cause problems for Versioneer. This details the most significant ones. More can be found on Github [issues page](https://github.com/python-versioneer/python-versioneer/issues). ### Subprojects Versioneer has limited support for source trees in which `setup.py` is not in the root directory (e.g. `setup.py` and `.git/` are *not* siblings). The are two common reasons why `setup.py` might not be in the root: * Source trees which contain multiple subprojects, such as [Buildbot](https://github.com/buildbot/buildbot), which contains both "master" and "slave" subprojects, each with their own `setup.py`, `setup.cfg`, and `tox.ini`. Projects like these produce multiple PyPI distributions (and upload multiple independently-installable tarballs). * Source trees whose main purpose is to contain a C library, but which also provide bindings to Python (and perhaps other languages) in subdirectories. Versioneer will look for `.git` in parent directories, and most operations should get the right version string. However `pip` and `setuptools` have bugs and implementation details which frequently cause `pip install .` from a subproject directory to fail to find a correct version string (so it usually defaults to `0+unknown`). `pip install --editable .` should work correctly. `setup.py install` might work too. Pip-8.1.1 is known to have this problem, but hopefully it will get fixed in some later version. [Bug #38](https://github.com/python-versioneer/python-versioneer/issues/38) is tracking this issue. The discussion in [PR #61](https://github.com/python-versioneer/python-versioneer/pull/61) describes the issue from the Versioneer side in more detail. [pip PR#3176](https://github.com/pypa/pip/pull/3176) and [pip PR#3615](https://github.com/pypa/pip/pull/3615) contain work to improve pip to let Versioneer work correctly. Versioneer-0.16 and earlier only looked for a `.git` directory next to the `setup.cfg`, so subprojects were completely unsupported with those releases. ### Editable installs with setuptools <= 18.5 `setup.py develop` and `pip install --editable .` allow you to install a project into a virtualenv once, then continue editing the source code (and test) without re-installing after every change. "Entry-point scripts" (`setup(entry_points={"console_scripts": ..})`) are a convenient way to specify executable scripts that should be installed along with the python package. These both work as expected when using modern setuptools. When using setuptools-18.5 or earlier, however, certain operations will cause `pkg_resources.DistributionNotFound` errors when running the entrypoint script, which must be resolved by re-installing the package. This happens when the install happens with one version, then the egg_info data is regenerated while a different version is checked out. Many setup.py commands cause egg_info to be rebuilt (including `sdist`, `wheel`, and installing into a different virtualenv), so this can be surprising. [Bug #83](https://github.com/python-versioneer/python-versioneer/issues/83) describes this one, but upgrading to a newer version of setuptools should probably resolve it. ## Updating Versioneer To upgrade your project to a new release of Versioneer, do the following: * install the new Versioneer (`pip install -U versioneer` or equivalent) * edit `setup.cfg` and `pyproject.toml`, if necessary, to include any new configuration settings indicated by the release notes. See [UPGRADING](./UPGRADING.md) for details. * re-run `versioneer install --[no-]vendor` in your source tree, to replace `SRC/_version.py` * commit any changed files ## Future Directions This tool is designed to make it easily extended to other version-control systems: all VCS-specific components are in separate directories like src/git/ . The top-level `versioneer.py` script is assembled from these components by running make-versioneer.py . In the future, make-versioneer.py will take a VCS name as an argument, and will construct a version of `versioneer.py` that is specific to the given VCS. It might also take the configuration arguments that are currently provided manually during installation by editing setup.py . Alternatively, it might go the other direction and include code from all supported VCS systems, reducing the number of intermediate scripts. ## Similar projects * [setuptools_scm](https://github.com/pypa/setuptools_scm/) - a non-vendored build-time dependency * [minver](https://github.com/jbweston/miniver) - a lightweight reimplementation of versioneer * [versioningit](https://github.com/jwodder/versioningit) - a PEP 518-based setuptools plugin ## License To make Versioneer easier to embed, all its code is dedicated to the public domain. The `_version.py` that it creates is also in the public domain. Specifically, both are released under the "Unlicense", as described in https://unlicense.org/. [pypi-image]: https://img.shields.io/pypi/v/versioneer.svg [pypi-url]: https://pypi.python.org/pypi/versioneer/ [travis-image]: https://img.shields.io/travis/com/python-versioneer/python-versioneer.svg [travis-url]: https://travis-ci.com/github/python-versioneer/python-versioneer """ # pylint:disable=invalid-name,import-outside-toplevel,missing-function-docstring # pylint:disable=missing-class-docstring,too-many-branches,too-many-statements # pylint:disable=raise-missing-from,too-many-lines,too-many-locals,import-error # pylint:disable=too-few-public-methods,redefined-outer-name,consider-using-with # pylint:disable=attribute-defined-outside-init,too-many-arguments import configparser import errno import json import os import re import subprocess import sys from pathlib import Path from typing import Any, Callable, cast, Dict, List, Optional, Tuple, Union from typing import NoReturn import functools have_tomllib = True if sys.version_info >= (3, 11): import tomllib else: try: import tomli as tomllib except ImportError: have_tomllib = False class VersioneerConfig: """Container for Versioneer configuration parameters.""" VCS: str style: str tag_prefix: str versionfile_source: str versionfile_build: Optional[str] parentdir_prefix: Optional[str] verbose: Optional[bool] def get_root() -> str: """Get the project root directory. We require that all commands are run from the project root, i.e. the directory that contains setup.py, setup.cfg, and versioneer.py . """ root = os.path.realpath(os.path.abspath(os.getcwd())) setup_py = os.path.join(root, "setup.py") pyproject_toml = os.path.join(root, "pyproject.toml") versioneer_py = os.path.join(root, "versioneer.py") if not ( os.path.exists(setup_py) or os.path.exists(pyproject_toml) or os.path.exists(versioneer_py) ): # allow 'python path/to/setup.py COMMAND' root = os.path.dirname(os.path.realpath(os.path.abspath(sys.argv[0]))) setup_py = os.path.join(root, "setup.py") pyproject_toml = os.path.join(root, "pyproject.toml") versioneer_py = os.path.join(root, "versioneer.py") if not ( os.path.exists(setup_py) or os.path.exists(pyproject_toml) or os.path.exists(versioneer_py) ): err = ("Versioneer was unable to run the project root directory. " "Versioneer requires setup.py to be executed from " "its immediate directory (like 'python setup.py COMMAND'), " "or in a way that lets it use sys.argv[0] to find the root " "(like 'python path/to/setup.py COMMAND').") raise VersioneerBadRootError(err) try: # Certain runtime workflows (setup.py install/develop in a setuptools # tree) execute all dependencies in a single python process, so # "versioneer" may be imported multiple times, and python's shared # module-import table will cache the first one. So we can't use # os.path.dirname(__file__), as that will find whichever # versioneer.py was first imported, even in later projects. my_path = os.path.realpath(os.path.abspath(__file__)) me_dir = os.path.normcase(os.path.splitext(my_path)[0]) vsr_dir = os.path.normcase(os.path.splitext(versioneer_py)[0]) if me_dir != vsr_dir and "VERSIONEER_PEP518" not in globals(): print("Warning: build in %s is using versioneer.py from %s" % (os.path.dirname(my_path), versioneer_py)) except NameError: pass return root def get_config_from_root(root: str) -> VersioneerConfig: """Read the project setup.cfg file to determine Versioneer config.""" # This might raise OSError (if setup.cfg is missing), or # configparser.NoSectionError (if it lacks a [versioneer] section), or # configparser.NoOptionError (if it lacks "VCS="). See the docstring at # the top of versioneer.py for instructions on writing your setup.cfg . root_pth = Path(root) pyproject_toml = root_pth / "pyproject.toml" setup_cfg = root_pth / "setup.cfg" section: Union[Dict[str, Any], configparser.SectionProxy, None] = None if pyproject_toml.exists() and have_tomllib: try: with open(pyproject_toml, 'rb') as fobj: pp = tomllib.load(fobj) section = pp['tool']['versioneer'] except (tomllib.TOMLDecodeError, KeyError) as e: print(f"Failed to load config from {pyproject_toml}: {e}") print("Try to load it from setup.cfg") if not section: parser = configparser.ConfigParser() with open(setup_cfg) as cfg_file: parser.read_file(cfg_file) parser.get("versioneer", "VCS") # raise error if missing section = parser["versioneer"] # `cast`` really shouldn't be used, but its simplest for the # common VersioneerConfig users at the moment. We verify against # `None` values elsewhere where it matters cfg = VersioneerConfig() cfg.VCS = section['VCS'] cfg.style = section.get("style", "") cfg.versionfile_source = cast(str, section.get("versionfile_source")) cfg.versionfile_build = section.get("versionfile_build") cfg.tag_prefix = cast(str, section.get("tag_prefix")) if cfg.tag_prefix in ("''", '""', None): cfg.tag_prefix = "" cfg.parentdir_prefix = section.get("parentdir_prefix") if isinstance(section, configparser.SectionProxy): # Make sure configparser translates to bool cfg.verbose = section.getboolean("verbose") else: cfg.verbose = section.get("verbose") return cfg class NotThisMethod(Exception): """Exception raised if a method is not valid for the current scenario.""" # these dictionaries contain VCS-specific tools LONG_VERSION_PY: Dict[str, str] = {} HANDLERS: Dict[str, Dict[str, Callable]] = {} def register_vcs_handler(vcs: str, method: str) -> Callable: # decorator """Create decorator to mark a method as the handler of a VCS.""" def decorate(f: Callable) -> Callable: """Store f in HANDLERS[vcs][method].""" HANDLERS.setdefault(vcs, {})[method] = f return f return decorate def run_command( commands: List[str], args: List[str], cwd: Optional[str] = None, verbose: bool = False, hide_stderr: bool = False, env: Optional[Dict[str, str]] = None, ) -> Tuple[Optional[str], Optional[int]]: """Call the given command(s).""" assert isinstance(commands, list) process = None popen_kwargs: Dict[str, Any] = {} if sys.platform == "win32": # This hides the console window if pythonw.exe is used startupinfo = subprocess.STARTUPINFO() startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW popen_kwargs["startupinfo"] = startupinfo for command in commands: try: dispcmd = str([command] + args) # remember shell=False, so use git.cmd on windows, not just git process = subprocess.Popen([command] + args, cwd=cwd, env=env, stdout=subprocess.PIPE, stderr=(subprocess.PIPE if hide_stderr else None), **popen_kwargs) break except OSError as e: if e.errno == errno.ENOENT: continue if verbose: print("unable to run %s" % dispcmd) print(e) return None, None else: if verbose: print("unable to find command, tried %s" % (commands,)) return None, None stdout = process.communicate()[0].strip().decode() if process.returncode != 0: if verbose: print("unable to run %s (error)" % dispcmd) print("stdout was %s" % stdout) return None, process.returncode return stdout, process.returncode LONG_VERSION_PY['git'] = r''' # This file helps to compute a version number in source trees obtained from # git-archive tarball (such as those provided by githubs download-from-tag # feature). Distribution tarballs (built by setup.py sdist) and build # directories (produced by setup.py build) will contain a much shorter file # that just contains the computed version number. # This file is released into the public domain. # Generated by versioneer-0.29 # https://github.com/python-versioneer/python-versioneer """Git implementation of _version.py.""" import errno import os import re import subprocess import sys from typing import Any, Callable, Dict, List, Optional, Tuple import functools def get_keywords() -> Dict[str, str]: """Get the keywords needed to look up the version information.""" # these strings will be replaced by git during git-archive. # setup.py/versioneer.py will grep for the variable names, so they must # each be defined on a line of their own. _version.py will just call # get_keywords(). git_refnames = "%(DOLLAR)sFormat:%%d%(DOLLAR)s" git_full = "%(DOLLAR)sFormat:%%H%(DOLLAR)s" git_date = "%(DOLLAR)sFormat:%%ci%(DOLLAR)s" keywords = {"refnames": git_refnames, "full": git_full, "date": git_date} return keywords class VersioneerConfig: """Container for Versioneer configuration parameters.""" VCS: str style: str tag_prefix: str parentdir_prefix: str versionfile_source: str verbose: bool def get_config() -> VersioneerConfig: """Create, populate and return the VersioneerConfig() object.""" # these strings are filled in when 'setup.py versioneer' creates # _version.py cfg = VersioneerConfig() cfg.VCS = "git" cfg.style = "%(STYLE)s" cfg.tag_prefix = "%(TAG_PREFIX)s" cfg.parentdir_prefix = "%(PARENTDIR_PREFIX)s" cfg.versionfile_source = "%(VERSIONFILE_SOURCE)s" cfg.verbose = False return cfg class NotThisMethod(Exception): """Exception raised if a method is not valid for the current scenario.""" LONG_VERSION_PY: Dict[str, str] = {} HANDLERS: Dict[str, Dict[str, Callable]] = {} def register_vcs_handler(vcs: str, method: str) -> Callable: # decorator """Create decorator to mark a method as the handler of a VCS.""" def decorate(f: Callable) -> Callable: """Store f in HANDLERS[vcs][method].""" if vcs not in HANDLERS: HANDLERS[vcs] = {} HANDLERS[vcs][method] = f return f return decorate def run_command( commands: List[str], args: List[str], cwd: Optional[str] = None, verbose: bool = False, hide_stderr: bool = False, env: Optional[Dict[str, str]] = None, ) -> Tuple[Optional[str], Optional[int]]: """Call the given command(s).""" assert isinstance(commands, list) process = None popen_kwargs: Dict[str, Any] = {} if sys.platform == "win32": # This hides the console window if pythonw.exe is used startupinfo = subprocess.STARTUPINFO() startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW popen_kwargs["startupinfo"] = startupinfo for command in commands: try: dispcmd = str([command] + args) # remember shell=False, so use git.cmd on windows, not just git process = subprocess.Popen([command] + args, cwd=cwd, env=env, stdout=subprocess.PIPE, stderr=(subprocess.PIPE if hide_stderr else None), **popen_kwargs) break except OSError as e: if e.errno == errno.ENOENT: continue if verbose: print("unable to run %%s" %% dispcmd) print(e) return None, None else: if verbose: print("unable to find command, tried %%s" %% (commands,)) return None, None stdout = process.communicate()[0].strip().decode() if process.returncode != 0: if verbose: print("unable to run %%s (error)" %% dispcmd) print("stdout was %%s" %% stdout) return None, process.returncode return stdout, process.returncode def versions_from_parentdir( parentdir_prefix: str, root: str, verbose: bool, ) -> Dict[str, Any]: """Try to determine the version from the parent directory name. Source tarballs conventionally unpack into a directory that includes both the project name and a version string. We will also support searching up two directory levels for an appropriately named parent directory """ rootdirs = [] for _ in range(3): dirname = os.path.basename(root) if dirname.startswith(parentdir_prefix): return {"version": dirname[len(parentdir_prefix):], "full-revisionid": None, "dirty": False, "error": None, "date": None} rootdirs.append(root) root = os.path.dirname(root) # up a level if verbose: print("Tried directories %%s but none started with prefix %%s" %% (str(rootdirs), parentdir_prefix)) raise NotThisMethod("rootdir doesn't start with parentdir_prefix") @register_vcs_handler("git", "get_keywords") def git_get_keywords(versionfile_abs: str) -> Dict[str, str]: """Extract version information from the given file.""" # the code embedded in _version.py can just fetch the value of these # keywords. When used from setup.py, we don't want to import _version.py, # so we do it with a regexp instead. This function is not used from # _version.py. keywords: Dict[str, str] = {} try: with open(versionfile_abs, "r") as fobj: for line in fobj: if line.strip().startswith("git_refnames ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["refnames"] = mo.group(1) if line.strip().startswith("git_full ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["full"] = mo.group(1) if line.strip().startswith("git_date ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["date"] = mo.group(1) except OSError: pass return keywords @register_vcs_handler("git", "keywords") def git_versions_from_keywords( keywords: Dict[str, str], tag_prefix: str, verbose: bool, ) -> Dict[str, Any]: """Get version information from git keywords.""" if "refnames" not in keywords: raise NotThisMethod("Short version file found") date = keywords.get("date") if date is not None: # Use only the last line. Previous lines may contain GPG signature # information. date = date.splitlines()[-1] # git-2.2.0 added "%%cI", which expands to an ISO-8601 -compliant # datestamp. However we prefer "%%ci" (which expands to an "ISO-8601 # -like" string, which we must then edit to make compliant), because # it's been around since git-1.5.3, and it's too difficult to # discover which version we're using, or to work around using an # older one. date = date.strip().replace(" ", "T", 1).replace(" ", "", 1) refnames = keywords["refnames"].strip() if refnames.startswith("$Format"): if verbose: print("keywords are unexpanded, not using") raise NotThisMethod("unexpanded keywords, not a git-archive tarball") refs = {r.strip() for r in refnames.strip("()").split(",")} # starting in git-1.8.3, tags are listed as "tag: foo-1.0" instead of # just "foo-1.0". If we see a "tag: " prefix, prefer those. TAG = "tag: " tags = {r[len(TAG):] for r in refs if r.startswith(TAG)} if not tags: # Either we're using git < 1.8.3, or there really are no tags. We use # a heuristic: assume all version tags have a digit. The old git %%d # expansion behaves like git log --decorate=short and strips out the # refs/heads/ and refs/tags/ prefixes that would let us distinguish # between branches and tags. By ignoring refnames without digits, we # filter out many common branch names like "release" and # "stabilization", as well as "HEAD" and "master". tags = {r for r in refs if re.search(r'\d', r)} if verbose: print("discarding '%%s', no digits" %% ",".join(refs - tags)) if verbose: print("likely tags: %%s" %% ",".join(sorted(tags))) for ref in sorted(tags): # sorting will prefer e.g. "2.0" over "2.0rc1" if ref.startswith(tag_prefix): r = ref[len(tag_prefix):] # Filter out refs that exactly match prefix or that don't start # with a number once the prefix is stripped (mostly a concern # when prefix is '') if not re.match(r'\d', r): continue if verbose: print("picking %%s" %% r) return {"version": r, "full-revisionid": keywords["full"].strip(), "dirty": False, "error": None, "date": date} # no suitable tags, so version is "0+unknown", but full hex is still there if verbose: print("no suitable tags, using unknown + full revision id") return {"version": "0+unknown", "full-revisionid": keywords["full"].strip(), "dirty": False, "error": "no suitable tags", "date": None} @register_vcs_handler("git", "pieces_from_vcs") def git_pieces_from_vcs( tag_prefix: str, root: str, verbose: bool, runner: Callable = run_command ) -> Dict[str, Any]: """Get version from 'git describe' in the root of the source tree. This only gets called if the git-archive 'subst' keywords were *not* expanded, and _version.py hasn't already been rewritten with a short version string, meaning we're inside a checked out source tree. """ GITS = ["git"] if sys.platform == "win32": GITS = ["git.cmd", "git.exe"] # GIT_DIR can interfere with correct operation of Versioneer. # It may be intended to be passed to the Versioneer-versioned project, # but that should not change where we get our version from. env = os.environ.copy() env.pop("GIT_DIR", None) runner = functools.partial(runner, env=env) _, rc = runner(GITS, ["rev-parse", "--git-dir"], cwd=root, hide_stderr=not verbose) if rc != 0: if verbose: print("Directory %%s not under git control" %% root) raise NotThisMethod("'git rev-parse --git-dir' returned error") # if there is a tag matching tag_prefix, this yields TAG-NUM-gHEX[-dirty] # if there isn't one, this yields HEX[-dirty] (no NUM) describe_out, rc = runner(GITS, [ "describe", "--tags", "--dirty", "--always", "--long", "--match", f"{tag_prefix}[[:digit:]]*" ], cwd=root) # --long was added in git-1.5.5 if describe_out is None: raise NotThisMethod("'git describe' failed") describe_out = describe_out.strip() full_out, rc = runner(GITS, ["rev-parse", "HEAD"], cwd=root) if full_out is None: raise NotThisMethod("'git rev-parse' failed") full_out = full_out.strip() pieces: Dict[str, Any] = {} pieces["long"] = full_out pieces["short"] = full_out[:7] # maybe improved later pieces["error"] = None branch_name, rc = runner(GITS, ["rev-parse", "--abbrev-ref", "HEAD"], cwd=root) # --abbrev-ref was added in git-1.6.3 if rc != 0 or branch_name is None: raise NotThisMethod("'git rev-parse --abbrev-ref' returned error") branch_name = branch_name.strip() if branch_name == "HEAD": # If we aren't exactly on a branch, pick a branch which represents # the current commit. If all else fails, we are on a branchless # commit. branches, rc = runner(GITS, ["branch", "--contains"], cwd=root) # --contains was added in git-1.5.4 if rc != 0 or branches is None: raise NotThisMethod("'git branch --contains' returned error") branches = branches.split("\n") # Remove the first line if we're running detached if "(" in branches[0]: branches.pop(0) # Strip off the leading "* " from the list of branches. branches = [branch[2:] for branch in branches] if "master" in branches: branch_name = "master" elif not branches: branch_name = None else: # Pick the first branch that is returned. Good or bad. branch_name = branches[0] pieces["branch"] = branch_name # parse describe_out. It will be like TAG-NUM-gHEX[-dirty] or HEX[-dirty] # TAG might have hyphens. git_describe = describe_out # look for -dirty suffix dirty = git_describe.endswith("-dirty") pieces["dirty"] = dirty if dirty: git_describe = git_describe[:git_describe.rindex("-dirty")] # now we have TAG-NUM-gHEX or HEX if "-" in git_describe: # TAG-NUM-gHEX mo = re.search(r'^(.+)-(\d+)-g([0-9a-f]+)$', git_describe) if not mo: # unparsable. Maybe git-describe is misbehaving? pieces["error"] = ("unable to parse git-describe output: '%%s'" %% describe_out) return pieces # tag full_tag = mo.group(1) if not full_tag.startswith(tag_prefix): if verbose: fmt = "tag '%%s' doesn't start with prefix '%%s'" print(fmt %% (full_tag, tag_prefix)) pieces["error"] = ("tag '%%s' doesn't start with prefix '%%s'" %% (full_tag, tag_prefix)) return pieces pieces["closest-tag"] = full_tag[len(tag_prefix):] # distance: number of commits since tag pieces["distance"] = int(mo.group(2)) # commit: short hex revision ID pieces["short"] = mo.group(3) else: # HEX: no tags pieces["closest-tag"] = None out, rc = runner(GITS, ["rev-list", "HEAD", "--left-right"], cwd=root) pieces["distance"] = len(out.split()) # total number of commits # commit date: see ISO-8601 comment in git_versions_from_keywords() date = runner(GITS, ["show", "-s", "--format=%%ci", "HEAD"], cwd=root)[0].strip() # Use only the last line. Previous lines may contain GPG signature # information. date = date.splitlines()[-1] pieces["date"] = date.strip().replace(" ", "T", 1).replace(" ", "", 1) return pieces def plus_or_dot(pieces: Dict[str, Any]) -> str: """Return a + if we don't already have one, else return a .""" if "+" in pieces.get("closest-tag", ""): return "." return "+" def render_pep440(pieces: Dict[str, Any]) -> str: """Build up version string, with post-release "local version identifier". Our goal: TAG[+DISTANCE.gHEX[.dirty]] . Note that if you get a tagged build and then dirty it, you'll get TAG+0.gHEX.dirty Exceptions: 1: no tags. git_describe was just HEX. 0+untagged.DISTANCE.gHEX[.dirty] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += plus_or_dot(pieces) rendered += "%%d.g%%s" %% (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" else: # exception #1 rendered = "0+untagged.%%d.g%%s" %% (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" return rendered def render_pep440_branch(pieces: Dict[str, Any]) -> str: """TAG[[.dev0]+DISTANCE.gHEX[.dirty]] . The ".dev0" means not master branch. Note that .dev0 sorts backwards (a feature branch will appear "older" than the master branch). Exceptions: 1: no tags. 0[.dev0]+untagged.DISTANCE.gHEX[.dirty] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: if pieces["branch"] != "master": rendered += ".dev0" rendered += plus_or_dot(pieces) rendered += "%%d.g%%s" %% (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" else: # exception #1 rendered = "0" if pieces["branch"] != "master": rendered += ".dev0" rendered += "+untagged.%%d.g%%s" %% (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" return rendered def pep440_split_post(ver: str) -> Tuple[str, Optional[int]]: """Split pep440 version string at the post-release segment. Returns the release segments before the post-release and the post-release version number (or -1 if no post-release segment is present). """ vc = str.split(ver, ".post") return vc[0], int(vc[1] or 0) if len(vc) == 2 else None def render_pep440_pre(pieces: Dict[str, Any]) -> str: """TAG[.postN.devDISTANCE] -- No -dirty. Exceptions: 1: no tags. 0.post0.devDISTANCE """ if pieces["closest-tag"]: if pieces["distance"]: # update the post release segment tag_version, post_version = pep440_split_post(pieces["closest-tag"]) rendered = tag_version if post_version is not None: rendered += ".post%%d.dev%%d" %% (post_version + 1, pieces["distance"]) else: rendered += ".post0.dev%%d" %% (pieces["distance"]) else: # no commits, use the tag as the version rendered = pieces["closest-tag"] else: # exception #1 rendered = "0.post0.dev%%d" %% pieces["distance"] return rendered def render_pep440_post(pieces: Dict[str, Any]) -> str: """TAG[.postDISTANCE[.dev0]+gHEX] . The ".dev0" means dirty. Note that .dev0 sorts backwards (a dirty tree will appear "older" than the corresponding clean one), but you shouldn't be releasing software with -dirty anyways. Exceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%%d" %% pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += plus_or_dot(pieces) rendered += "g%%s" %% pieces["short"] else: # exception #1 rendered = "0.post%%d" %% pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += "+g%%s" %% pieces["short"] return rendered def render_pep440_post_branch(pieces: Dict[str, Any]) -> str: """TAG[.postDISTANCE[.dev0]+gHEX[.dirty]] . The ".dev0" means not master branch. Exceptions: 1: no tags. 0.postDISTANCE[.dev0]+gHEX[.dirty] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%%d" %% pieces["distance"] if pieces["branch"] != "master": rendered += ".dev0" rendered += plus_or_dot(pieces) rendered += "g%%s" %% pieces["short"] if pieces["dirty"]: rendered += ".dirty" else: # exception #1 rendered = "0.post%%d" %% pieces["distance"] if pieces["branch"] != "master": rendered += ".dev0" rendered += "+g%%s" %% pieces["short"] if pieces["dirty"]: rendered += ".dirty" return rendered def render_pep440_old(pieces: Dict[str, Any]) -> str: """TAG[.postDISTANCE[.dev0]] . The ".dev0" means dirty. Exceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%%d" %% pieces["distance"] if pieces["dirty"]: rendered += ".dev0" else: # exception #1 rendered = "0.post%%d" %% pieces["distance"] if pieces["dirty"]: rendered += ".dev0" return rendered def render_git_describe(pieces: Dict[str, Any]) -> str: """TAG[-DISTANCE-gHEX][-dirty]. Like 'git describe --tags --dirty --always'. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"]: rendered += "-%%d-g%%s" %% (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render_git_describe_long(pieces: Dict[str, Any]) -> str: """TAG-DISTANCE-gHEX[-dirty]. Like 'git describe --tags --dirty --always -long'. The distance/hash is unconditional. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] rendered += "-%%d-g%%s" %% (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render(pieces: Dict[str, Any], style: str) -> Dict[str, Any]: """Render the given version pieces into the requested style.""" if pieces["error"]: return {"version": "unknown", "full-revisionid": pieces.get("long"), "dirty": None, "error": pieces["error"], "date": None} if not style or style == "default": style = "pep440" # the default if style == "pep440": rendered = render_pep440(pieces) elif style == "pep440-branch": rendered = render_pep440_branch(pieces) elif style == "pep440-pre": rendered = render_pep440_pre(pieces) elif style == "pep440-post": rendered = render_pep440_post(pieces) elif style == "pep440-post-branch": rendered = render_pep440_post_branch(pieces) elif style == "pep440-old": rendered = render_pep440_old(pieces) elif style == "git-describe": rendered = render_git_describe(pieces) elif style == "git-describe-long": rendered = render_git_describe_long(pieces) else: raise ValueError("unknown style '%%s'" %% style) return {"version": rendered, "full-revisionid": pieces["long"], "dirty": pieces["dirty"], "error": None, "date": pieces.get("date")} def get_versions() -> Dict[str, Any]: """Get version information or return default if unable to do so.""" # I am in _version.py, which lives at ROOT/VERSIONFILE_SOURCE. If we have # __file__, we can work backwards from there to the root. Some # py2exe/bbfreeze/non-CPython implementations don't do __file__, in which # case we can only use expanded keywords. cfg = get_config() verbose = cfg.verbose try: return git_versions_from_keywords(get_keywords(), cfg.tag_prefix, verbose) except NotThisMethod: pass try: root = os.path.realpath(__file__) # versionfile_source is the relative path from the top of the source # tree (where the .git directory might live) to this file. Invert # this to find the root from __file__. for _ in cfg.versionfile_source.split('/'): root = os.path.dirname(root) except NameError: return {"version": "0+unknown", "full-revisionid": None, "dirty": None, "error": "unable to find root of source tree", "date": None} try: pieces = git_pieces_from_vcs(cfg.tag_prefix, root, verbose) return render(pieces, cfg.style) except NotThisMethod: pass try: if cfg.parentdir_prefix: return versions_from_parentdir(cfg.parentdir_prefix, root, verbose) except NotThisMethod: pass return {"version": "0+unknown", "full-revisionid": None, "dirty": None, "error": "unable to compute version", "date": None} ''' @register_vcs_handler("git", "get_keywords") def git_get_keywords(versionfile_abs: str) -> Dict[str, str]: """Extract version information from the given file.""" # the code embedded in _version.py can just fetch the value of these # keywords. When used from setup.py, we don't want to import _version.py, # so we do it with a regexp instead. This function is not used from # _version.py. keywords: Dict[str, str] = {} try: with open(versionfile_abs, "r") as fobj: for line in fobj: if line.strip().startswith("git_refnames ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["refnames"] = mo.group(1) if line.strip().startswith("git_full ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["full"] = mo.group(1) if line.strip().startswith("git_date ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["date"] = mo.group(1) except OSError: pass return keywords @register_vcs_handler("git", "keywords") def git_versions_from_keywords( keywords: Dict[str, str], tag_prefix: str, verbose: bool, ) -> Dict[str, Any]: """Get version information from git keywords.""" if "refnames" not in keywords: raise NotThisMethod("Short version file found") date = keywords.get("date") if date is not None: # Use only the last line. Previous lines may contain GPG signature # information. date = date.splitlines()[-1] # git-2.2.0 added "%cI", which expands to an ISO-8601 -compliant # datestamp. However we prefer "%ci" (which expands to an "ISO-8601 # -like" string, which we must then edit to make compliant), because # it's been around since git-1.5.3, and it's too difficult to # discover which version we're using, or to work around using an # older one. date = date.strip().replace(" ", "T", 1).replace(" ", "", 1) refnames = keywords["refnames"].strip() if refnames.startswith("$Format"): if verbose: print("keywords are unexpanded, not using") raise NotThisMethod("unexpanded keywords, not a git-archive tarball") refs = {r.strip() for r in refnames.strip("()").split(",")} # starting in git-1.8.3, tags are listed as "tag: foo-1.0" instead of # just "foo-1.0". If we see a "tag: " prefix, prefer those. TAG = "tag: " tags = {r[len(TAG):] for r in refs if r.startswith(TAG)} if not tags: # Either we're using git < 1.8.3, or there really are no tags. We use # a heuristic: assume all version tags have a digit. The old git %d # expansion behaves like git log --decorate=short and strips out the # refs/heads/ and refs/tags/ prefixes that would let us distinguish # between branches and tags. By ignoring refnames without digits, we # filter out many common branch names like "release" and # "stabilization", as well as "HEAD" and "master". tags = {r for r in refs if re.search(r'\d', r)} if verbose: print("discarding '%s', no digits" % ",".join(refs - tags)) if verbose: print("likely tags: %s" % ",".join(sorted(tags))) for ref in sorted(tags): # sorting will prefer e.g. "2.0" over "2.0rc1" if ref.startswith(tag_prefix): r = ref[len(tag_prefix):] # Filter out refs that exactly match prefix or that don't start # with a number once the prefix is stripped (mostly a concern # when prefix is '') if not re.match(r'\d', r): continue if verbose: print("picking %s" % r) return {"version": r, "full-revisionid": keywords["full"].strip(), "dirty": False, "error": None, "date": date} # no suitable tags, so version is "0+unknown", but full hex is still there if verbose: print("no suitable tags, using unknown + full revision id") return {"version": "0+unknown", "full-revisionid": keywords["full"].strip(), "dirty": False, "error": "no suitable tags", "date": None} @register_vcs_handler("git", "pieces_from_vcs") def git_pieces_from_vcs( tag_prefix: str, root: str, verbose: bool, runner: Callable = run_command ) -> Dict[str, Any]: """Get version from 'git describe' in the root of the source tree. This only gets called if the git-archive 'subst' keywords were *not* expanded, and _version.py hasn't already been rewritten with a short version string, meaning we're inside a checked out source tree. """ GITS = ["git"] if sys.platform == "win32": GITS = ["git.cmd", "git.exe"] # GIT_DIR can interfere with correct operation of Versioneer. # It may be intended to be passed to the Versioneer-versioned project, # but that should not change where we get our version from. env = os.environ.copy() env.pop("GIT_DIR", None) runner = functools.partial(runner, env=env) _, rc = runner(GITS, ["rev-parse", "--git-dir"], cwd=root, hide_stderr=not verbose) if rc != 0: if verbose: print("Directory %s not under git control" % root) raise NotThisMethod("'git rev-parse --git-dir' returned error") # if there is a tag matching tag_prefix, this yields TAG-NUM-gHEX[-dirty] # if there isn't one, this yields HEX[-dirty] (no NUM) describe_out, rc = runner(GITS, [ "describe", "--tags", "--dirty", "--always", "--long", "--match", f"{tag_prefix}[[:digit:]]*" ], cwd=root) # --long was added in git-1.5.5 if describe_out is None: raise NotThisMethod("'git describe' failed") describe_out = describe_out.strip() full_out, rc = runner(GITS, ["rev-parse", "HEAD"], cwd=root) if full_out is None: raise NotThisMethod("'git rev-parse' failed") full_out = full_out.strip() pieces: Dict[str, Any] = {} pieces["long"] = full_out pieces["short"] = full_out[:7] # maybe improved later pieces["error"] = None branch_name, rc = runner(GITS, ["rev-parse", "--abbrev-ref", "HEAD"], cwd=root) # --abbrev-ref was added in git-1.6.3 if rc != 0 or branch_name is None: raise NotThisMethod("'git rev-parse --abbrev-ref' returned error") branch_name = branch_name.strip() if branch_name == "HEAD": # If we aren't exactly on a branch, pick a branch which represents # the current commit. If all else fails, we are on a branchless # commit. branches, rc = runner(GITS, ["branch", "--contains"], cwd=root) # --contains was added in git-1.5.4 if rc != 0 or branches is None: raise NotThisMethod("'git branch --contains' returned error") branches = branches.split("\n") # Remove the first line if we're running detached if "(" in branches[0]: branches.pop(0) # Strip off the leading "* " from the list of branches. branches = [branch[2:] for branch in branches] if "master" in branches: branch_name = "master" elif not branches: branch_name = None else: # Pick the first branch that is returned. Good or bad. branch_name = branches[0] pieces["branch"] = branch_name # parse describe_out. It will be like TAG-NUM-gHEX[-dirty] or HEX[-dirty] # TAG might have hyphens. git_describe = describe_out # look for -dirty suffix dirty = git_describe.endswith("-dirty") pieces["dirty"] = dirty if dirty: git_describe = git_describe[:git_describe.rindex("-dirty")] # now we have TAG-NUM-gHEX or HEX if "-" in git_describe: # TAG-NUM-gHEX mo = re.search(r'^(.+)-(\d+)-g([0-9a-f]+)$', git_describe) if not mo: # unparsable. Maybe git-describe is misbehaving? pieces["error"] = ("unable to parse git-describe output: '%s'" % describe_out) return pieces # tag full_tag = mo.group(1) if not full_tag.startswith(tag_prefix): if verbose: fmt = "tag '%s' doesn't start with prefix '%s'" print(fmt % (full_tag, tag_prefix)) pieces["error"] = ("tag '%s' doesn't start with prefix '%s'" % (full_tag, tag_prefix)) return pieces pieces["closest-tag"] = full_tag[len(tag_prefix):] # distance: number of commits since tag pieces["distance"] = int(mo.group(2)) # commit: short hex revision ID pieces["short"] = mo.group(3) else: # HEX: no tags pieces["closest-tag"] = None out, rc = runner(GITS, ["rev-list", "HEAD", "--left-right"], cwd=root) pieces["distance"] = len(out.split()) # total number of commits # commit date: see ISO-8601 comment in git_versions_from_keywords() date = runner(GITS, ["show", "-s", "--format=%ci", "HEAD"], cwd=root)[0].strip() # Use only the last line. Previous lines may contain GPG signature # information. date = date.splitlines()[-1] pieces["date"] = date.strip().replace(" ", "T", 1).replace(" ", "", 1) return pieces def do_vcs_install(versionfile_source: str, ipy: Optional[str]) -> None: """Git-specific installation logic for Versioneer. For Git, this means creating/changing .gitattributes to mark _version.py for export-subst keyword substitution. """ GITS = ["git"] if sys.platform == "win32": GITS = ["git.cmd", "git.exe"] files = [versionfile_source] if ipy: files.append(ipy) if "VERSIONEER_PEP518" not in globals(): try: my_path = __file__ if my_path.endswith((".pyc", ".pyo")): my_path = os.path.splitext(my_path)[0] + ".py" versioneer_file = os.path.relpath(my_path) except NameError: versioneer_file = "versioneer.py" files.append(versioneer_file) present = False try: with open(".gitattributes", "r") as fobj: for line in fobj: if line.strip().startswith(versionfile_source): if "export-subst" in line.strip().split()[1:]: present = True break except OSError: pass if not present: with open(".gitattributes", "a+") as fobj: fobj.write(f"{versionfile_source} export-subst\n") files.append(".gitattributes") run_command(GITS, ["add", "--"] + files) def versions_from_parentdir( parentdir_prefix: str, root: str, verbose: bool, ) -> Dict[str, Any]: """Try to determine the version from the parent directory name. Source tarballs conventionally unpack into a directory that includes both the project name and a version string. We will also support searching up two directory levels for an appropriately named parent directory """ rootdirs = [] for _ in range(3): dirname = os.path.basename(root) if dirname.startswith(parentdir_prefix): return {"version": dirname[len(parentdir_prefix):], "full-revisionid": None, "dirty": False, "error": None, "date": None} rootdirs.append(root) root = os.path.dirname(root) # up a level if verbose: print("Tried directories %s but none started with prefix %s" % (str(rootdirs), parentdir_prefix)) raise NotThisMethod("rootdir doesn't start with parentdir_prefix") SHORT_VERSION_PY = """ # This file was generated by 'versioneer.py' (0.29) from # revision-control system data, or from the parent directory name of an # unpacked source archive. Distribution tarballs contain a pre-generated copy # of this file. import json version_json = ''' %s ''' # END VERSION_JSON def get_versions(): return json.loads(version_json) """ def versions_from_file(filename: str) -> Dict[str, Any]: """Try to determine the version from _version.py if present.""" try: with open(filename) as f: contents = f.read() except OSError: raise NotThisMethod("unable to read _version.py") mo = re.search(r"version_json = '''\n(.*)''' # END VERSION_JSON", contents, re.M | re.S) if not mo: mo = re.search(r"version_json = '''\r\n(.*)''' # END VERSION_JSON", contents, re.M | re.S) if not mo: raise NotThisMethod("no version_json in _version.py") return json.loads(mo.group(1)) def write_to_version_file(filename: str, versions: Dict[str, Any]) -> None: """Write the given version number to the given _version.py file.""" contents = json.dumps(versions, sort_keys=True, indent=1, separators=(",", ": ")) with open(filename, "w") as f: f.write(SHORT_VERSION_PY % contents) print("set %s to '%s'" % (filename, versions["version"])) def plus_or_dot(pieces: Dict[str, Any]) -> str: """Return a + if we don't already have one, else return a .""" if "+" in pieces.get("closest-tag", ""): return "." return "+" def render_pep440(pieces: Dict[str, Any]) -> str: """Build up version string, with post-release "local version identifier". Our goal: TAG[+DISTANCE.gHEX[.dirty]] . Note that if you get a tagged build and then dirty it, you'll get TAG+0.gHEX.dirty Exceptions: 1: no tags. git_describe was just HEX. 0+untagged.DISTANCE.gHEX[.dirty] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += plus_or_dot(pieces) rendered += "%d.g%s" % (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" else: # exception #1 rendered = "0+untagged.%d.g%s" % (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" return rendered def render_pep440_branch(pieces: Dict[str, Any]) -> str: """TAG[[.dev0]+DISTANCE.gHEX[.dirty]] . The ".dev0" means not master branch. Note that .dev0 sorts backwards (a feature branch will appear "older" than the master branch). Exceptions: 1: no tags. 0[.dev0]+untagged.DISTANCE.gHEX[.dirty] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: if pieces["branch"] != "master": rendered += ".dev0" rendered += plus_or_dot(pieces) rendered += "%d.g%s" % (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" else: # exception #1 rendered = "0" if pieces["branch"] != "master": rendered += ".dev0" rendered += "+untagged.%d.g%s" % (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" return rendered def pep440_split_post(ver: str) -> Tuple[str, Optional[int]]: """Split pep440 version string at the post-release segment. Returns the release segments before the post-release and the post-release version number (or -1 if no post-release segment is present). """ vc = str.split(ver, ".post") return vc[0], int(vc[1] or 0) if len(vc) == 2 else None def render_pep440_pre(pieces: Dict[str, Any]) -> str: """TAG[.postN.devDISTANCE] -- No -dirty. Exceptions: 1: no tags. 0.post0.devDISTANCE """ if pieces["closest-tag"]: if pieces["distance"]: # update the post release segment tag_version, post_version = pep440_split_post(pieces["closest-tag"]) rendered = tag_version if post_version is not None: rendered += ".post%d.dev%d" % (post_version + 1, pieces["distance"]) else: rendered += ".post0.dev%d" % (pieces["distance"]) else: # no commits, use the tag as the version rendered = pieces["closest-tag"] else: # exception #1 rendered = "0.post0.dev%d" % pieces["distance"] return rendered def render_pep440_post(pieces: Dict[str, Any]) -> str: """TAG[.postDISTANCE[.dev0]+gHEX] . The ".dev0" means dirty. Note that .dev0 sorts backwards (a dirty tree will appear "older" than the corresponding clean one), but you shouldn't be releasing software with -dirty anyways. Exceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += plus_or_dot(pieces) rendered += "g%s" % pieces["short"] else: # exception #1 rendered = "0.post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += "+g%s" % pieces["short"] return rendered def render_pep440_post_branch(pieces: Dict[str, Any]) -> str: """TAG[.postDISTANCE[.dev0]+gHEX[.dirty]] . The ".dev0" means not master branch. Exceptions: 1: no tags. 0.postDISTANCE[.dev0]+gHEX[.dirty] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%d" % pieces["distance"] if pieces["branch"] != "master": rendered += ".dev0" rendered += plus_or_dot(pieces) rendered += "g%s" % pieces["short"] if pieces["dirty"]: rendered += ".dirty" else: # exception #1 rendered = "0.post%d" % pieces["distance"] if pieces["branch"] != "master": rendered += ".dev0" rendered += "+g%s" % pieces["short"] if pieces["dirty"]: rendered += ".dirty" return rendered def render_pep440_old(pieces: Dict[str, Any]) -> str: """TAG[.postDISTANCE[.dev0]] . The ".dev0" means dirty. Exceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" else: # exception #1 rendered = "0.post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" return rendered def render_git_describe(pieces: Dict[str, Any]) -> str: """TAG[-DISTANCE-gHEX][-dirty]. Like 'git describe --tags --dirty --always'. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"]: rendered += "-%d-g%s" % (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render_git_describe_long(pieces: Dict[str, Any]) -> str: """TAG-DISTANCE-gHEX[-dirty]. Like 'git describe --tags --dirty --always -long'. The distance/hash is unconditional. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] rendered += "-%d-g%s" % (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render(pieces: Dict[str, Any], style: str) -> Dict[str, Any]: """Render the given version pieces into the requested style.""" if pieces["error"]: return {"version": "unknown", "full-revisionid": pieces.get("long"), "dirty": None, "error": pieces["error"], "date": None} if not style or style == "default": style = "pep440" # the default if style == "pep440": rendered = render_pep440(pieces) elif style == "pep440-branch": rendered = render_pep440_branch(pieces) elif style == "pep440-pre": rendered = render_pep440_pre(pieces) elif style == "pep440-post": rendered = render_pep440_post(pieces) elif style == "pep440-post-branch": rendered = render_pep440_post_branch(pieces) elif style == "pep440-old": rendered = render_pep440_old(pieces) elif style == "git-describe": rendered = render_git_describe(pieces) elif style == "git-describe-long": rendered = render_git_describe_long(pieces) else: raise ValueError("unknown style '%s'" % style) return {"version": rendered, "full-revisionid": pieces["long"], "dirty": pieces["dirty"], "error": None, "date": pieces.get("date")} class VersioneerBadRootError(Exception): """The project root directory is unknown or missing key files.""" def get_versions(verbose: bool = False) -> Dict[str, Any]: """Get the project version from whatever source is available. Returns dict with two keys: 'version' and 'full'. """ if "versioneer" in sys.modules: # see the discussion in cmdclass.py:get_cmdclass() del sys.modules["versioneer"] root = get_root() cfg = get_config_from_root(root) assert cfg.VCS is not None, "please set [versioneer]VCS= in setup.cfg" handlers = HANDLERS.get(cfg.VCS) assert handlers, "unrecognized VCS '%s'" % cfg.VCS verbose = verbose or bool(cfg.verbose) # `bool()` used to avoid `None` assert cfg.versionfile_source is not None, \ "please set versioneer.versionfile_source" assert cfg.tag_prefix is not None, "please set versioneer.tag_prefix" versionfile_abs = os.path.join(root, cfg.versionfile_source) # extract version from first of: _version.py, VCS command (e.g. 'git # describe'), parentdir. This is meant to work for developers using a # source checkout, for users of a tarball created by 'setup.py sdist', # and for users of a tarball/zipball created by 'git archive' or github's # download-from-tag feature or the equivalent in other VCSes. get_keywords_f = handlers.get("get_keywords") from_keywords_f = handlers.get("keywords") if get_keywords_f and from_keywords_f: try: keywords = get_keywords_f(versionfile_abs) ver = from_keywords_f(keywords, cfg.tag_prefix, verbose) if verbose: print("got version from expanded keyword %s" % ver) return ver except NotThisMethod: pass try: ver = versions_from_file(versionfile_abs) if verbose: print("got version from file %s %s" % (versionfile_abs, ver)) return ver except NotThisMethod: pass from_vcs_f = handlers.get("pieces_from_vcs") if from_vcs_f: try: pieces = from_vcs_f(cfg.tag_prefix, root, verbose) ver = render(pieces, cfg.style) if verbose: print("got version from VCS %s" % ver) return ver except NotThisMethod: pass try: if cfg.parentdir_prefix: ver = versions_from_parentdir(cfg.parentdir_prefix, root, verbose) if verbose: print("got version from parentdir %s" % ver) return ver except NotThisMethod: pass if verbose: print("unable to compute version") return {"version": "0+unknown", "full-revisionid": None, "dirty": None, "error": "unable to compute version", "date": None} def get_version() -> str: """Get the short version string for this project.""" return get_versions()["version"] def get_cmdclass(cmdclass: Optional[Dict[str, Any]] = None): """Get the custom setuptools subclasses used by Versioneer. If the package uses a different cmdclass (e.g. one from numpy), it should be provide as an argument. """ if "versioneer" in sys.modules: del sys.modules["versioneer"] # this fixes the "python setup.py develop" case (also 'install' and # 'easy_install .'), in which subdependencies of the main project are # built (using setup.py bdist_egg) in the same python process. Assume # a main project A and a dependency B, which use different versions # of Versioneer. A's setup.py imports A's Versioneer, leaving it in # sys.modules by the time B's setup.py is executed, causing B to run # with the wrong versioneer. Setuptools wraps the sub-dep builds in a # sandbox that restores sys.modules to it's pre-build state, so the # parent is protected against the child's "import versioneer". By # removing ourselves from sys.modules here, before the child build # happens, we protect the child from the parent's versioneer too. # Also see https://github.com/python-versioneer/python-versioneer/issues/52 cmds = {} if cmdclass is None else cmdclass.copy() # we add "version" to setuptools from setuptools import Command class cmd_version(Command): description = "report generated version string" user_options: List[Tuple[str, str, str]] = [] boolean_options: List[str] = [] def initialize_options(self) -> None: pass def finalize_options(self) -> None: pass def run(self) -> None: vers = get_versions(verbose=True) print("Version: %s" % vers["version"]) print(" full-revisionid: %s" % vers.get("full-revisionid")) print(" dirty: %s" % vers.get("dirty")) print(" date: %s" % vers.get("date")) if vers["error"]: print(" error: %s" % vers["error"]) cmds["version"] = cmd_version # we override "build_py" in setuptools # # most invocation pathways end up running build_py: # distutils/build -> build_py # distutils/install -> distutils/build ->.. # setuptools/bdist_wheel -> distutils/install ->.. # setuptools/bdist_egg -> distutils/install_lib -> build_py # setuptools/install -> bdist_egg ->.. # setuptools/develop -> ? # pip install: # copies source tree to a tempdir before running egg_info/etc # if .git isn't copied too, 'git describe' will fail # then does setup.py bdist_wheel, or sometimes setup.py install # setup.py egg_info -> ? # pip install -e . and setuptool/editable_wheel will invoke build_py # but the build_py command is not expected to copy any files. # we override different "build_py" commands for both environments if 'build_py' in cmds: _build_py: Any = cmds['build_py'] else: from setuptools.command.build_py import build_py as _build_py class cmd_build_py(_build_py): def run(self) -> None: root = get_root() cfg = get_config_from_root(root) versions = get_versions() _build_py.run(self) if getattr(self, "editable_mode", False): # During editable installs `.py` and data files are # not copied to build_lib return # now locate _version.py in the new build/ directory and replace # it with an updated value if cfg.versionfile_build: target_versionfile = os.path.join(self.build_lib, cfg.versionfile_build) print("UPDATING %s" % target_versionfile) write_to_version_file(target_versionfile, versions) cmds["build_py"] = cmd_build_py if 'build_ext' in cmds: _build_ext: Any = cmds['build_ext'] else: from setuptools.command.build_ext import build_ext as _build_ext class cmd_build_ext(_build_ext): def run(self) -> None: root = get_root() cfg = get_config_from_root(root) versions = get_versions() _build_ext.run(self) if self.inplace: # build_ext --inplace will only build extensions in # build/lib<..> dir with no _version.py to write to. # As in place builds will already have a _version.py # in the module dir, we do not need to write one. return # now locate _version.py in the new build/ directory and replace # it with an updated value if not cfg.versionfile_build: return target_versionfile = os.path.join(self.build_lib, cfg.versionfile_build) if not os.path.exists(target_versionfile): print(f"Warning: {target_versionfile} does not exist, skipping " "version update. This can happen if you are running build_ext " "without first running build_py.") return print("UPDATING %s" % target_versionfile) write_to_version_file(target_versionfile, versions) cmds["build_ext"] = cmd_build_ext if "cx_Freeze" in sys.modules: # cx_freeze enabled? from cx_Freeze.dist import build_exe as _build_exe # type: ignore # nczeczulin reports that py2exe won't like the pep440-style string # as FILEVERSION, but it can be used for PRODUCTVERSION, e.g. # setup(console=[{ # "version": versioneer.get_version().split("+", 1)[0], # FILEVERSION # "product_version": versioneer.get_version(), # ... class cmd_build_exe(_build_exe): def run(self) -> None: root = get_root() cfg = get_config_from_root(root) versions = get_versions() target_versionfile = cfg.versionfile_source print("UPDATING %s" % target_versionfile) write_to_version_file(target_versionfile, versions) _build_exe.run(self) os.unlink(target_versionfile) with open(cfg.versionfile_source, "w") as f: LONG = LONG_VERSION_PY[cfg.VCS] f.write(LONG % {"DOLLAR": "$", "STYLE": cfg.style, "TAG_PREFIX": cfg.tag_prefix, "PARENTDIR_PREFIX": cfg.parentdir_prefix, "VERSIONFILE_SOURCE": cfg.versionfile_source, }) cmds["build_exe"] = cmd_build_exe del cmds["build_py"] if 'py2exe' in sys.modules: # py2exe enabled? try: from py2exe.setuptools_buildexe import py2exe as _py2exe # type: ignore except ImportError: from py2exe.distutils_buildexe import py2exe as _py2exe # type: ignore class cmd_py2exe(_py2exe): def run(self) -> None: root = get_root() cfg = get_config_from_root(root) versions = get_versions() target_versionfile = cfg.versionfile_source print("UPDATING %s" % target_versionfile) write_to_version_file(target_versionfile, versions) _py2exe.run(self) os.unlink(target_versionfile) with open(cfg.versionfile_source, "w") as f: LONG = LONG_VERSION_PY[cfg.VCS] f.write(LONG % {"DOLLAR": "$", "STYLE": cfg.style, "TAG_PREFIX": cfg.tag_prefix, "PARENTDIR_PREFIX": cfg.parentdir_prefix, "VERSIONFILE_SOURCE": cfg.versionfile_source, }) cmds["py2exe"] = cmd_py2exe # sdist farms its file list building out to egg_info if 'egg_info' in cmds: _egg_info: Any = cmds['egg_info'] else: from setuptools.command.egg_info import egg_info as _egg_info class cmd_egg_info(_egg_info): def find_sources(self) -> None: # egg_info.find_sources builds the manifest list and writes it # in one shot super().find_sources() # Modify the filelist and normalize it root = get_root() cfg = get_config_from_root(root) self.filelist.append('versioneer.py') if cfg.versionfile_source: # There are rare cases where versionfile_source might not be # included by default, so we must be explicit self.filelist.append(cfg.versionfile_source) self.filelist.sort() self.filelist.remove_duplicates() # The write method is hidden in the manifest_maker instance that # generated the filelist and was thrown away # We will instead replicate their final normalization (to unicode, # and POSIX-style paths) from setuptools import unicode_utils normalized = [unicode_utils.filesys_decode(f).replace(os.sep, '/') for f in self.filelist.files] manifest_filename = os.path.join(self.egg_info, 'SOURCES.txt') with open(manifest_filename, 'w') as fobj: fobj.write('\n'.join(normalized)) cmds['egg_info'] = cmd_egg_info # we override different "sdist" commands for both environments if 'sdist' in cmds: _sdist: Any = cmds['sdist'] else: from setuptools.command.sdist import sdist as _sdist class cmd_sdist(_sdist): def run(self) -> None: versions = get_versions() self._versioneer_generated_versions = versions # unless we update this, the command will keep using the old # version self.distribution.metadata.version = versions["version"] return _sdist.run(self) def make_release_tree(self, base_dir: str, files: List[str]) -> None: root = get_root() cfg = get_config_from_root(root) _sdist.make_release_tree(self, base_dir, files) # now locate _version.py in the new base_dir directory # (remembering that it may be a hardlink) and replace it with an # updated value target_versionfile = os.path.join(base_dir, cfg.versionfile_source) print("UPDATING %s" % target_versionfile) write_to_version_file(target_versionfile, self._versioneer_generated_versions) cmds["sdist"] = cmd_sdist return cmds CONFIG_ERROR = """ setup.cfg is missing the necessary Versioneer configuration. You need a section like: [versioneer] VCS = git style = pep440 versionfile_source = src/myproject/_version.py versionfile_build = myproject/_version.py tag_prefix = parentdir_prefix = myproject- You will also need to edit your setup.py to use the results: import versioneer setup(version=versioneer.get_version(), cmdclass=versioneer.get_cmdclass(), ...) Please read the docstring in ./versioneer.py for configuration instructions, edit setup.cfg, and re-run the installer or 'python versioneer.py setup'. """ SAMPLE_CONFIG = """ # See the docstring in versioneer.py for instructions. Note that you must # re-run 'versioneer.py setup' after changing this section, and commit the # resulting files. [versioneer] #VCS = git #style = pep440 #versionfile_source = #versionfile_build = #tag_prefix = #parentdir_prefix = """ OLD_SNIPPET = """ from ._version import get_versions __version__ = get_versions()['version'] del get_versions """ INIT_PY_SNIPPET = """ from . import {0} __version__ = {0}.get_versions()['version'] """ def do_setup() -> int: """Do main VCS-independent setup function for installing Versioneer.""" root = get_root() try: cfg = get_config_from_root(root) except (OSError, configparser.NoSectionError, configparser.NoOptionError) as e: if isinstance(e, (OSError, configparser.NoSectionError)): print("Adding sample versioneer config to setup.cfg", file=sys.stderr) with open(os.path.join(root, "setup.cfg"), "a") as f: f.write(SAMPLE_CONFIG) print(CONFIG_ERROR, file=sys.stderr) return 1 print(" creating %s" % cfg.versionfile_source) with open(cfg.versionfile_source, "w") as f: LONG = LONG_VERSION_PY[cfg.VCS] f.write(LONG % {"DOLLAR": "$", "STYLE": cfg.style, "TAG_PREFIX": cfg.tag_prefix, "PARENTDIR_PREFIX": cfg.parentdir_prefix, "VERSIONFILE_SOURCE": cfg.versionfile_source, }) ipy = os.path.join(os.path.dirname(cfg.versionfile_source), "__init__.py") maybe_ipy: Optional[str] = ipy if os.path.exists(ipy): try: with open(ipy, "r") as f: old = f.read() except OSError: old = "" module = os.path.splitext(os.path.basename(cfg.versionfile_source))[0] snippet = INIT_PY_SNIPPET.format(module) if OLD_SNIPPET in old: print(" replacing boilerplate in %s" % ipy) with open(ipy, "w") as f: f.write(old.replace(OLD_SNIPPET, snippet)) elif snippet not in old: print(" appending to %s" % ipy) with open(ipy, "a") as f: f.write(snippet) else: print(" %s unmodified" % ipy) else: print(" %s doesn't exist, ok" % ipy) maybe_ipy = None # Make VCS-specific changes. For git, this means creating/changing # .gitattributes to mark _version.py for export-subst keyword # substitution. do_vcs_install(cfg.versionfile_source, maybe_ipy) return 0 def scan_setup_py() -> int: """Validate the contents of setup.py against Versioneer's expectations.""" found = set() setters = False errors = 0 with open("setup.py", "r") as f: for line in f.readlines(): if "import versioneer" in line: found.add("import") if "versioneer.get_cmdclass()" in line: found.add("cmdclass") if "versioneer.get_version()" in line: found.add("get_version") if "versioneer.VCS" in line: setters = True if "versioneer.versionfile_source" in line: setters = True if len(found) != 3: print("") print("Your setup.py appears to be missing some important items") print("(but I might be wrong). Please make sure it has something") print("roughly like the following:") print("") print(" import versioneer") print(" setup( version=versioneer.get_version(),") print(" cmdclass=versioneer.get_cmdclass(), ...)") print("") errors += 1 if setters: print("You should remove lines like 'versioneer.VCS = ' and") print("'versioneer.versionfile_source = ' . This configuration") print("now lives in setup.cfg, and should be removed from setup.py") print("") errors += 1 return errors def setup_command() -> NoReturn: """Set up Versioneer and exit with appropriate error code.""" errors = do_setup() errors += scan_setup_py() sys.exit(1 if errors else 0) if __name__ == "__main__": cmd = sys.argv[1] if cmd == "setup": setup_command() ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1698168074.996989 zfec-1.5.7.4/zfec/0000755000175100001770000000000014515776413013166 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/zfec/__init__.py0000644000175100001770000000131414515776411015274 0ustar00runnerdocker""" zfec -- fast forward error correction library with Python interface maintainer web site: U{http://tahoe-lafs.org/source/zfec} zfec web site: U{http://tahoe-lafs.org/source/zfec} """ from . import _version __version__ = _version.get_versions()['version'] from ._fec import Encoder, Decoder, Error from . import easyfec, filefec, cmdline_zfec, cmdline_zunfec quiet_pyflakes=[__version__, Error, Encoder, Decoder, cmdline_zunfec, filefec, cmdline_zfec, easyfec] # zfec -- fast forward error correction library with Python interface # # Copyright (C) 2007-2010 Allmydata, Inc. # Author: Zooko Wilcox-O'Hearn # mailto:zooko@zooko.com # # This file is part of zfec. # # See README.rst for licensing information. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/zfec/_fecmodule.c0000644000175100001770000006172314515776411015443 0ustar00runnerdocker/** * zfec -- fast forward error correction library with Python interface */ #include #include #include #if (PY_VERSION_HEX < 0x02050000) typedef int Py_ssize_t; #endif #ifndef PyVarObject_HEAD_INIT #define PyVarObject_HEAD_INIT(type, size) PyObject_HEAD_INIT(type) size, #endif #ifndef PyInt_Check #define PyInt_Check PyLong_Check #define PyInt_FromLong PyLong_FromLong #define PyInt_AsLong PyLong_AsLong #define PyInt_Type PyLong_Type #endif #ifndef PyString_FromStringAndSize #define PyString_FromStringAndSize PyBytes_FromStringAndSize #define PyString_AsString PyBytes_AsString #endif #include "fec.h" #include "stdarg.h" static PyObject *py_fec_error; static char fec__doc__[] = "\ FEC - Forward Error Correction \n\ "; static char Encoder__doc__[] = "\ Hold static encoder state (an in-memory table for matrix multiplication), and k and m parameters, and provide {encode()} method.\n\n\ @param k: the number of packets required for reconstruction \n\ @param m: the number of packets generated \n\ "; typedef struct { PyObject_HEAD /* expose these */ unsigned short kk; unsigned short mm; /* internal */ fec_t* fec_matrix; } Encoder; static PyObject * Encoder_new(PyTypeObject *type, PyObject *args, PyObject *kwdict) { Encoder *self; self = (Encoder*)type->tp_alloc(type, 0); if (self != NULL) { self->kk = 0; self->mm = 0; self->fec_matrix = NULL; } return (PyObject *)self; } static int Encoder_init(Encoder *self, PyObject *args, PyObject *kwdict) { static char *kwlist[] = { "k", "m", NULL }; int ink, inm; if (!PyArg_ParseTupleAndKeywords(args, kwdict, "ii:Encoder.__init__", kwlist, &ink, &inm)) return -1; if (ink < 1) { PyErr_Format(py_fec_error, "Precondition violation: first argument is required to be greater than or equal to 1, but it was %d", ink); return -1; } if (inm < 1) { PyErr_Format(py_fec_error, "Precondition violation: second argument is required to be greater than or equal to 1, but it was %d", inm); return -1; } if (inm > 256) { PyErr_Format(py_fec_error, "Precondition violation: second argument is required to be less than or equal to 256, but it was %d", inm); return -1; } if (ink > inm) { PyErr_Format(py_fec_error, "Precondition violation: first argument is required to be less than or equal to the second argument, but they were %d and %d respectively", ink, inm); return -1; } self->kk = (unsigned short)ink; self->mm = (unsigned short)inm; Py_BEGIN_ALLOW_THREADS self->fec_matrix = fec_new(self->kk, self->mm); Py_END_ALLOW_THREADS return 0; } static char Encoder_encode__doc__[] = "\ Encode data into m packets.\n\ \n\ @param inblocks: a sequence of k buffers of data to encode -- these are the k primary blocks, i.e. the input data split into k pieces (for best performance, make it a tuple instead of a list); All blocks are required to be the same length.\n\ @param desired_blocks_nums optional sequence of blocknums indicating which blocks to produce and return; If None, all m blocks will be returned (in order). (For best performance, make it a tuple instead of a list.)\n\ @returns: a list of buffers containing the requested blocks; Note that if any of the input blocks were 'primary blocks', i.e. their blocknum was < k, then the result sequence will contain a Python reference to the same Python object as was passed in. As long as the Python object in question is immutable (i.e. a string) then you don't have to think about this detail, but if it is mutable (i.e. an array), then you have to be aware that if you subsequently mutate the contents of that object then that will also change the contents of the sequence that was returned from this call to encode().\n\ "; static PyObject * Encoder_encode(Encoder *self, PyObject *args) { PyObject* inblocks; PyObject* desired_blocks_nums = NULL; /* The blocknums of the blocks that should be returned. */ PyObject* result = NULL; gf** check_blocks_produced = (gf**)alloca((self->mm - self->kk) * sizeof(PyObject*)); /* This is an upper bound -- we will actually use only num_check_blocks_produced of these elements (see below). */ PyObject** pystrs_produced = (PyObject**)alloca((self->mm - self->kk) * sizeof(PyObject*)); /* This is an upper bound -- we will actually use only num_check_blocks_produced of these elements (see below). */ unsigned num_check_blocks_produced = 0; /* The first num_check_blocks_produced elements of the check_blocks_produced array and of the pystrs_produced array will be used. */ const gf** incblocks = (const gf**)alloca(self->kk * sizeof(const gf*)); size_t num_desired_blocks; PyObject* fast_desired_blocks_nums = NULL; PyObject** fast_desired_blocks_nums_items; size_t * c_desired_blocks_nums = (size_t*)alloca(self->mm * sizeof(size_t)); unsigned* c_desired_checkblocks_ids = (unsigned*)alloca((self->mm - self->kk) * sizeof(unsigned)); size_t i; PyObject* fastinblocks = NULL; PyObject** fastinblocksitems; Py_ssize_t sz, oldsz = 0; unsigned char check_block_index = 0; /* index into the check_blocks_produced and (parallel) pystrs_produced arrays */ if (!PyArg_ParseTuple(args, "O|O:Encoder.encode", &inblocks, &desired_blocks_nums)) return NULL; for (i = 0; i < (size_t)self->mm - (size_t)self->kk; i++) pystrs_produced[i] = NULL; if (desired_blocks_nums) { fast_desired_blocks_nums = PySequence_Fast(desired_blocks_nums, "Second argument (optional) was not a sequence."); if (!fast_desired_blocks_nums) goto err; num_desired_blocks = PySequence_Fast_GET_SIZE(fast_desired_blocks_nums); fast_desired_blocks_nums_items = PySequence_Fast_ITEMS(fast_desired_blocks_nums); for (i = 0; i < num_desired_blocks; i++) { if (!PyInt_Check(fast_desired_blocks_nums_items[i])) { PyErr_Format(py_fec_error, "Precondition violation: second argument is required to contain int."); goto err; } c_desired_blocks_nums[i] = PyInt_AsLong(fast_desired_blocks_nums_items[i]); if (c_desired_blocks_nums[i] >= self->kk) num_check_blocks_produced++; } } else { num_desired_blocks = self->mm; for (i = 0; imm - self->kk; } fastinblocks = PySequence_Fast(inblocks, "First argument was not a sequence."); if (!fastinblocks) goto err; if (PySequence_Fast_GET_SIZE(fastinblocks) != self->kk) { PyErr_Format(py_fec_error, "Precondition violation: Wrong length -- first argument (the sequence of input blocks) is required to contain exactly k blocks. len(first): %zu, k: %d", PySequence_Fast_GET_SIZE(fastinblocks), self->kk); goto err; } /* Construct a C array of gf*'s of the input data. */ fastinblocksitems = PySequence_Fast_ITEMS(fastinblocks); if (!fastinblocksitems) goto err; for (i = 0; i < self->kk; i++) { if (!PyObject_CheckReadBuffer(fastinblocksitems[i])) { PyErr_Format(py_fec_error, "Precondition violation: %zu'th item is required to offer the single-segment read character buffer protocol, but it does not.", i); goto err; } if (PyObject_AsReadBuffer(fastinblocksitems[i], (const void**)&(incblocks[i]), &sz)) goto err; if (oldsz != 0 && oldsz != sz) { PyErr_Format(py_fec_error, "Precondition violation: Input blocks are required to be all the same length. length of one block was: %zu, length of another block was: %zu", oldsz, sz); goto err; } oldsz = sz; } /* Allocate space for all of the check blocks. */ for (i = 0; i < num_desired_blocks; i++) { if (c_desired_blocks_nums[i] >= self->kk) { c_desired_checkblocks_ids[check_block_index] = c_desired_blocks_nums[i]; pystrs_produced[check_block_index] = PyString_FromStringAndSize(NULL, sz); if (pystrs_produced[check_block_index] == NULL) goto err; check_blocks_produced[check_block_index] = (gf*)PyString_AsString(pystrs_produced[check_block_index]); if (check_blocks_produced[check_block_index] == NULL) goto err; check_block_index++; } } assert (check_block_index == num_check_blocks_produced); /* Encode any check blocks that are needed. */ Py_BEGIN_ALLOW_THREADS fec_encode(self->fec_matrix, incblocks, check_blocks_produced, c_desired_checkblocks_ids, num_check_blocks_produced, sz); Py_END_ALLOW_THREADS /* Wrap all requested blocks up into a Python list of Python strings. */ result = PyList_New(num_desired_blocks); if (result == NULL) goto err; check_block_index = 0; for (i = 0; i < num_desired_blocks; i++) { if (c_desired_blocks_nums[i] < self->kk) { Py_INCREF(fastinblocksitems[c_desired_blocks_nums[i]]); if (PyList_SetItem(result, i, fastinblocksitems[c_desired_blocks_nums[i]]) == -1) { Py_DECREF(fastinblocksitems[c_desired_blocks_nums[i]]); goto err; } } else { if (PyList_SetItem(result, i, pystrs_produced[check_block_index]) == -1) goto err; pystrs_produced[check_block_index] = NULL; check_block_index++; } } goto cleanup; err: for (i = 0; i < num_check_blocks_produced; i++) Py_XDECREF(pystrs_produced[i]); Py_XDECREF(result); result = NULL; cleanup: Py_XDECREF(fastinblocks); fastinblocks=NULL; Py_XDECREF(fast_desired_blocks_nums); fast_desired_blocks_nums=NULL; return result; } static void Encoder_dealloc(Encoder * self) { if (self->fec_matrix) fec_free(self->fec_matrix); Py_TYPE(self)->tp_free((PyObject*)self); } static PyMethodDef Encoder_methods[] = { {"encode", (PyCFunction)Encoder_encode, METH_VARARGS, Encoder_encode__doc__}, {NULL}, }; static PyMemberDef Encoder_members[] = { {"k", T_SHORT, offsetof(Encoder, kk), READONLY, "k"}, {"m", T_SHORT, offsetof(Encoder, mm), READONLY, "m"}, {NULL} /* Sentinel */ }; static PyTypeObject Encoder_type = { PyVarObject_HEAD_INIT(NULL, 0) "_fec.Encoder", /*tp_name*/ sizeof(Encoder), /*tp_basicsize*/ 0, /*tp_itemsize*/ (destructor)Encoder_dealloc, /*tp_dealloc*/ 0, /*tp_print*/ 0, /*tp_getattr*/ 0, /*tp_setattr*/ 0, /*tp_compare*/ 0, /*tp_repr*/ 0, /*tp_as_number*/ 0, /*tp_as_sequence*/ 0, /*tp_as_mapping*/ 0, /*tp_hash */ 0, /*tp_call*/ 0, /*tp_str*/ 0, /*tp_getattro*/ 0, /*tp_setattro*/ 0, /*tp_as_buffer*/ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/ Encoder__doc__, /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ Encoder_methods, /* tp_methods */ Encoder_members, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ (initproc)Encoder_init, /* tp_init */ 0, /* tp_alloc */ Encoder_new, /* tp_new */ }; static char Decoder__doc__[] = "\ Hold static decoder state (an in-memory table for matrix multiplication), and k and m parameters, and provide {decode()} method.\n\n\ @param k: the number of packets required for reconstruction \n\ @param m: the number of packets generated \n\ "; typedef struct { PyObject_HEAD /* expose these */ unsigned short kk; unsigned short mm; /* internal */ fec_t* fec_matrix; } Decoder; static PyObject * Decoder_new(PyTypeObject *type, PyObject *args, PyObject *kwdict) { Decoder *self; self = (Decoder*)type->tp_alloc(type, 0); if (self != NULL) { self->kk = 0; self->mm = 0; self->fec_matrix = NULL; } return (PyObject *)self; } static int Decoder_init(Encoder *self, PyObject *args, PyObject *kwdict) { static char *kwlist[] = { "k", "m", NULL }; int ink, inm; if (!PyArg_ParseTupleAndKeywords(args, kwdict, "ii:Decoder.__init__", kwlist, &ink, &inm)) return -1; if (ink < 1) { PyErr_Format(py_fec_error, "Precondition violation: first argument is required to be greater than or equal to 1, but it was %d", ink); return -1; } if (inm < 1) { PyErr_Format(py_fec_error, "Precondition violation: second argument is required to be greater than or equal to 1, but it was %d", inm); return -1; } if (inm > 256) { PyErr_Format(py_fec_error, "Precondition violation: second argument is required to be less than or equal to 256, but it was %d", inm); return -1; } if (ink > inm) { PyErr_Format(py_fec_error, "Precondition violation: first argument is required to be less than or equal to the second argument, but they were %d and %d respectively", ink, inm); return -1; } self->kk = (unsigned short)ink; self->mm = (unsigned short)inm; Py_BEGIN_ALLOW_THREADS self->fec_matrix = fec_new(self->kk, self->mm); Py_END_ALLOW_THREADS return 0; } #define SWAP(a,b,t) {t tmp; tmp=a; a=b; b=tmp;} static char Decoder_decode__doc__[] = "\ Decode a list blocks into a list of segments.\n\ @param blocks a sequence of buffers containing block data (for best performance, make it a tuple instead of a list)\n\ @param blocknums a sequence of integers of the blocknum for each block in blocks (for best performance, make it a tuple instead of a list)\n\ \n\ @return a list of strings containing the segment data (i.e. ''.join(retval) yields a string containing the decoded data)\n\ "; static PyObject * Decoder_decode(Decoder *self, PyObject *args) { PyObject*restrict blocks; PyObject*restrict blocknums; PyObject* result = NULL; const gf**restrict cblocks = (const gf**restrict)alloca(self->kk * sizeof(const gf*)); unsigned* cblocknums = (unsigned*)alloca(self->kk * sizeof(unsigned)); gf**restrict recoveredcstrs = (gf**)alloca(self->kk * sizeof(gf*)); /* self->kk is actually an upper bound -- we probably won't need all of this space. */ PyObject**restrict recoveredpystrs = (PyObject**restrict)alloca(self->kk * sizeof(PyObject*)); /* self->kk is actually an upper bound -- we probably won't need all of this space. */ unsigned i; PyObject*restrict fastblocknums = NULL; PyObject*restrict fastblocks; unsigned needtorecover=0; PyObject** fastblocksitems; PyObject** fastblocknumsitems; Py_ssize_t sz, oldsz = 0; long tmpl; unsigned nextrecoveredix=0; if (!PyArg_ParseTuple(args, "OO:Decoder.decode", &blocks, &blocknums)) return NULL; for (i=0; ikk; i++) recoveredpystrs[i] = NULL; fastblocks = PySequence_Fast(blocks, "First argument was not a sequence."); if (!fastblocks) goto err; fastblocknums = PySequence_Fast(blocknums, "Second argument was not a sequence."); if (!fastblocknums) goto err; if (PySequence_Fast_GET_SIZE(fastblocks) != self->kk) { PyErr_Format(py_fec_error, "Precondition violation: Wrong length -- first argument is required to contain exactly k blocks. len(first): %zu, k: %d", PySequence_Fast_GET_SIZE(fastblocks), self->kk); goto err; } if (PySequence_Fast_GET_SIZE(fastblocknums) != self->kk) { PyErr_Format(py_fec_error, "Precondition violation: Wrong length -- blocknums is required to contain exactly k blocks. len(blocknums): %zu, k: %d", PySequence_Fast_GET_SIZE(fastblocknums), self->kk); goto err; } /* Construct a C array of gf*'s of the data and another of C ints of the blocknums. */ fastblocknumsitems = PySequence_Fast_ITEMS(fastblocknums); if (!fastblocknumsitems) goto err; fastblocksitems = PySequence_Fast_ITEMS(fastblocks); if (!fastblocksitems) goto err; for (i=0; ikk; i++) { if (!PyInt_Check(fastblocknumsitems[i])) { PyErr_Format(py_fec_error, "Precondition violation: second argument is required to contain int."); goto err; } tmpl = PyInt_AsLong(fastblocknumsitems[i]); if (tmpl < 0 || tmpl > 255) { PyErr_Format(py_fec_error, "Precondition violation: block nums can't be less than zero or greater than 255. %ld\n", tmpl); goto err; } cblocknums[i] = (unsigned)tmpl; if (cblocknums[i] >= self->kk) needtorecover+=1; if (!PyObject_CheckReadBuffer(fastblocksitems[i])) { PyErr_Format(py_fec_error, "Precondition violation: %u'th item is required to offer the single-segment read character buffer protocol, but it does not.\n", i); goto err; } if (PyObject_AsReadBuffer(fastblocksitems[i], (const void**)&(cblocks[i]), &sz)) goto err; if (oldsz != 0 && oldsz != sz) { PyErr_Format(py_fec_error, "Precondition violation: Input blocks are required to be all the same length. length of one block was: %zu, length of another block was: %zu\n", oldsz, sz); goto err; } oldsz = sz; } /* Move src packets into position. At the end of this loop we want the i'th element of the arrays to be the block with block number i, if that block is among our inputs. */ for (i=0; ikk;) { if (cblocknums[i] >= self->kk || cblocknums[i] == i) i++; else { /* put pkt in the right position. */ unsigned c = cblocknums[i]; SWAP (cblocknums[i], cblocknums[c], int); SWAP (cblocks[i], cblocks[c], const gf*); SWAP (fastblocksitems[i], fastblocksitems[c], PyObject*); } } /* Allocate space for all of the recovered blocks. */ for (i=0; ifec_matrix, cblocks, recoveredcstrs, cblocknums, sz); Py_END_ALLOW_THREADS /* Wrap up both original primary blocks and decoded blocks into a Python list of Python strings. */ result = PyList_New(self->kk); if (result == NULL) goto err; for (i=0; ikk; i++) { if (cblocknums[i] == i) { /* Original primary block. */ Py_INCREF(fastblocksitems[i]); if (PyList_SetItem(result, i, fastblocksitems[i]) == -1) { Py_DECREF(fastblocksitems[i]); goto err; } } else { /* Recovered block. */ if (PyList_SetItem(result, i, recoveredpystrs[nextrecoveredix]) == -1) goto err; recoveredpystrs[nextrecoveredix] = NULL; nextrecoveredix++; } } goto cleanup; err: for (i=0; ikk; i++) Py_XDECREF(recoveredpystrs[i]); Py_XDECREF(result); result = NULL; cleanup: Py_XDECREF(fastblocks); fastblocks=NULL; Py_XDECREF(fastblocknums); fastblocknums=NULL; return result; } static void Decoder_dealloc(Decoder * self) { if (self->fec_matrix) fec_free(self->fec_matrix); Py_TYPE(self)->tp_free((PyObject*)self); } static PyMethodDef Decoder_methods[] = { {"decode", (PyCFunction)Decoder_decode, METH_VARARGS, Decoder_decode__doc__}, {NULL}, }; static PyMemberDef Decoder_members[] = { {"k", T_SHORT, offsetof(Encoder, kk), READONLY, "k"}, {"m", T_SHORT, offsetof(Encoder, mm), READONLY, "m"}, {NULL} /* Sentinel */ }; static PyTypeObject Decoder_type = { PyVarObject_HEAD_INIT(NULL, 0) "_fec.Decoder", /*tp_name*/ sizeof(Decoder), /*tp_basicsize*/ 0, /*tp_itemsize*/ (destructor)Decoder_dealloc, /*tp_dealloc*/ 0, /*tp_print*/ 0, /*tp_getattr*/ 0, /*tp_setattr*/ 0, /*tp_compare*/ 0, /*tp_repr*/ 0, /*tp_as_number*/ 0, /*tp_as_sequence*/ 0, /*tp_as_mapping*/ 0, /*tp_hash */ 0, /*tp_call*/ 0, /*tp_str*/ 0, /*tp_getattro*/ 0, /*tp_setattro*/ 0, /*tp_as_buffer*/ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/ Decoder__doc__, /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ Decoder_methods, /* tp_methods */ Decoder_members, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ (initproc)Decoder_init, /* tp_init */ 0, /* tp_alloc */ Decoder_new, /* tp_new */ }; void _hexwrite(unsigned char*s, size_t l) { size_t i; for (i = 0; i < l; i++) printf("%.2x", s[i]); } PyObject* test_from_agl(PyObject* self, PyObject* args) { unsigned char b0c[8], b1c[8]; unsigned char b0[8], b1[8], b2[8], b3[8], b4[8]; const unsigned char *blocks[3] = {b0, b1, b2}; unsigned char *outblocks[2] = {b3, b4}; unsigned block_nums[] = {3, 4}; fec_t *const fec = fec_new(3, 5); const unsigned char *inpkts[] = {b3, b4, b2}; unsigned char *outpkts[] = {b0, b1}; unsigned indexes[] = {3, 4, 2}; memset(b0, 1, 8); memset(b1, 2, 8); memset(b2, 3, 8); /*printf("_from_c before encoding:\n"); printf("b0: "); _hexwrite(b0, 8); printf(", "); printf("b1: "); _hexwrite(b1, 8); printf(", "); printf("b2: "); _hexwrite(b2, 8); printf(", "); printf("\n");*/ fec_encode(fec, blocks, outblocks, block_nums, 2, 8); /*printf("after encoding:\n"); printf("b3: "); _hexwrite(b3, 8); printf(", "); printf("b4: "); _hexwrite(b4, 8); printf(", "); printf("\n");*/ memcpy(b0c, b0, 8); memcpy(b1c, b1, 8); fec_decode(fec, inpkts, outpkts, indexes, 8); /*printf("after decoding:\n"); printf("b0: "); _hexwrite(b0, 8); printf(", "); printf("b1: "); _hexwrite(b1, 8); printf("\n");*/ if ((memcmp(b0, b0c,8) == 0) && (memcmp(b1, b1c,8) == 0)) Py_RETURN_TRUE; else Py_RETURN_FALSE; } static PyMethodDef fec_functions[] = { {"test_from_agl", test_from_agl, METH_NOARGS, NULL}, {NULL} }; #if PY_MAJOR_VERSION >= 3 static struct PyModuleDef moduledef = { PyModuleDef_HEAD_INIT, "_fec", fec__doc__, -1, fec_functions, }; #endif #ifndef PyMODINIT_FUNC /* declarations for DLL import/export */ #define PyMODINIT_FUNC void #endif PyMODINIT_FUNC #if PY_MAJOR_VERSION >= 3 #define MOD_ERROR_VAL NULL PyInit__fec(void) { #else #define MOD_ERROR_VAL init_fec(void) { #endif PyObject *module; PyObject *module_dict; if (PyType_Ready(&Encoder_type) < 0) return MOD_ERROR_VAL; if (PyType_Ready(&Decoder_type) < 0) return MOD_ERROR_VAL; #if PY_MAJOR_VERSION >= 3 module = PyModule_Create(&moduledef); #else module = Py_InitModule3("_fec", fec_functions, fec__doc__); if (module == NULL) return; #endif Py_INCREF(&Encoder_type); Py_INCREF(&Decoder_type); PyModule_AddObject(module, "Encoder", (PyObject *)&Encoder_type); PyModule_AddObject(module, "Decoder", (PyObject *)&Decoder_type); module_dict = PyModule_GetDict(module); py_fec_error = PyErr_NewException("_fec.Error", NULL, NULL); PyDict_SetItemString(module_dict, "Error", py_fec_error); fec_init(); #if PY_MAJOR_VERSION >= 3 return module; #endif } /** * originally inspired by fecmodule.c by the Mnet Project, especially Myers * Carpenter and Hauke Johannknecht */ ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1698168074.996989 zfec-1.5.7.4/zfec/_version.py0000644000175100001770000000076314515776413015372 0ustar00runnerdocker # This file was generated by 'versioneer.py' (0.29) from # revision-control system data, or from the parent directory name of an # unpacked source archive. Distribution tarballs contain a pre-generated copy # of this file. import json version_json = ''' { "date": "2023-10-23T15:56:39-0400", "dirty": false, "error": null, "full-revisionid": "65ab7977a9958ace402dd2a3399497c1a919fe56", "version": "1.5.7.4" } ''' # END VERSION_JSON def get_versions(): return json.loads(version_json) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/zfec/cmdline_zfec.py0000755000175100001770000001041214515776411016161 0ustar00runnerdocker#!/usr/bin/env python # zfec -- a fast C implementation of Reed-Solomon erasure coding with # command-line, C, and Python interfaces from __future__ import print_function import sys, argparse from zfec import filefec from zfec import __version__ as libversion __version__ = libversion DEFAULT_K=3 DEFAULT_M=8 def main(): if '-V' in sys.argv or '--version' in sys.argv: print("zfec library version: ", libversion) print("zfec command-line tool version: ", __version__) sys.exit(0) parser = argparse.ArgumentParser(description="Encode a file into a set of share files, a subset of which can later be used to recover the original file.") parser.add_argument('inputfile', help='file to encode or "-" for stdin', type=argparse.FileType('rb'), metavar='INF') parser.add_argument('-d', '--output-dir', help='directory in which share file names will be created (default ".")', default='.', metavar='D') parser.add_argument('-p', '--prefix', help='prefix for share file names; If omitted, the name of the input file will be used.', metavar='P') parser.add_argument('-s', '--suffix', help='suffix for share file names (default ".fec")', default='.fec', metavar='S') parser.add_argument('-m', '--totalshares', help='the total number of share files created (default %d)' % DEFAULT_M, default=DEFAULT_M, type=int, metavar='M') parser.add_argument('-k', '--requiredshares', help='the number of share files required to reconstruct (default %d)' % DEFAULT_K, default=DEFAULT_K, type=int, metavar='K') parser.add_argument('-f', '--force', help='overwrite any file which already in place an output file (share file)', action='store_true') parser.add_argument('-v', '--verbose', help='print out messages about progress', action='store_true') parser.add_argument('-q', '--quiet', help='quiet progress indications and warnings about silly choices of K and M', action='store_true') parser.add_argument('-V', '--version', help='print out version number and exit', action='store_true') args = parser.parse_args() is_infile_stdin = False if args.prefix is None: args.prefix = args.inputfile.name if args.prefix == "": args.prefix = "" is_infile_stdin = True if args.verbose and args.quiet: print("Please choose only one of --verbose and --quiet.") sys.exit(1) if args.totalshares > 256 or args.totalshares < 1: print("Invalid parameters, totalshares is required to be <= 256 and >= 1\nPlease see the accompanying documentation.") sys.exit(1) if args.requiredshares > args.totalshares or args.requiredshares < 1: print("Invalid parameters, requiredshares is required to be <= totalshares and >= 1\nPlease see the accompanying documentation.") sys.exit(1) if not args.quiet: if args.requiredshares == 1: print("warning: silly parameters: requiredshares == 1, which means that every share will be a complete copy of the file. You could use \"cp\" for the same effect. But proceeding to do it anyway...") if args.requiredshares == args.totalshares: print("warning: silly parameters: requiredshares == totalshares, which means that all shares will be required in order to reconstruct the file. You could use \"split\" for the same effect. But proceeding to do it anyway...") in_file = args.inputfile try: args.inputfile.seek(0, 2) fsize = args.inputfile.tell() args.inputfile.seek(0, 0) except IOError: if is_infile_stdin: contents = args.inputfile.read() fsize = len(contents) else: raise Exception("zfec - needs a real (Seekable) file handle to" " measure file size upfront.") try: return filefec.encode_to_files(in_file, fsize, args.output_dir, args.prefix, args.requiredshares, args.totalshares, args.suffix, args.force, args.verbose) finally: args.inputfile.close() # zfec -- fast forward error correction library with Python interface # # Copyright (C) 2007 Allmydata, Inc. # Author: Zooko Wilcox-O'Hearn # # This file is part of zfec. # # See README.rst for licensing information. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/zfec/cmdline_zunfec.py0000755000175100001770000000475614515776411016542 0ustar00runnerdocker#!/usr/bin/env python # zfec -- a fast C implementation of Reed-Solomon erasure coding with # command-line, C, and Python interfaces from __future__ import print_function import os, sys, argparse from zfec import filefec from zfec import __version__ as libversion __version__ = libversion def main(): if '-V' in sys.argv or '--version' in sys.argv: print("zfec library version: ", libversion) print("zunfec command-line tool version: ", __version__) return 0 parser = argparse.ArgumentParser(description="Decode data from share files.") parser.add_argument('-o', '--outputfile', required=True, help='file to write the resulting data to, or "-" for stdout', type=str, metavar='OUTF') parser.add_argument('sharefiles', nargs='*', help='shares file to read the encoded data from', type=str, metavar='SHAREFILE') parser.add_argument('-v', '--verbose', help='print out messages about progress', action='store_true') parser.add_argument('-f', '--force', help='overwrite any file which already in place of the output file', action='store_true') parser.add_argument('-V', '--version', help='print out version number and exit', action='store_true') args = parser.parse_args() if len(args.sharefiles) < 2: print("At least two sharefiles are required.") return 1 if args.force: outf = open(args.outputfile, 'wb') else: try: flags = os.O_WRONLY|os.O_CREAT|os.O_EXCL | (hasattr(os, 'O_BINARY') and os.O_BINARY) outfd = os.open(args.outputfile, flags) except OSError: print("There is already a file named %r -- aborting. Use --force to overwrite." % (args.outputfile,)) return 2 outf = os.fdopen(outfd, "wb") sharefs = [] # This sort() actually matters for performance (shares with numbers < k # are much faster to use than the others), as well as being important for # reproducibility. args.sharefiles.sort() for fn in args.sharefiles: sharefs.append(open(fn, 'rb')) try: filefec.decode_from_files(outf, sharefs, args.verbose) except filefec.InsufficientShareFilesError as e: print(str(e)) return 3 finally: outf.close() for f in sharefs: f.close() return 0 # zfec -- fast forward error correction library with Python interface # # Copyright (C) 2007 Allmydata, Inc. # Author: Zooko Wilcox-O'Hearn # # This file is part of zfec. # # See README.rst for licensing information. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/zfec/easyfec.py0000644000175100001770000000405614515776411015162 0ustar00runnerdocker# zfec -- a fast C implementation of Reed-Solomon erasure coding with # command-line, C, and Python interfaces import zfec # div_ceil() was copied from the pyutil library. def div_ceil(n, d): """ The smallest integer k such that k*d >= n. """ return (n//d) + (n%d != 0) from base64 import b32encode def ab(x): # debuggery if len(x) >= 3: return "%s:%s" % (len(x), b32encode(x[-3:]),) elif len(x) == 2: return "%s:%s" % (len(x), b32encode(x[-2:]),) elif len(x) == 1: return "%s:%s" % (len(x), b32encode(x[-1:]),) elif len(x) == 0: return "%s:%s" % (len(x), "--empty--",) class Encoder(object): def __init__(self, k, m): self.fec = zfec.Encoder(k, m) def encode(self, data): """ @param data: string @return: a sequence of m blocks -- any k of which suffice to reconstruct the input data """ chunksize = div_ceil(len(data), self.fec.k) l = [ data[i*chunksize:(i+1)*chunksize] + b"\x00" * min(chunksize, (((i+1)*chunksize)-len(data))) for i in range(self.fec.k) ] assert len(l) == self.fec.k, (len(l), self.fec.k,) assert (not l) or (not [ x for x in l if len(x) != len(l[0]) ], (len(l), [ ab(x) for x in l ], chunksize, self.fec.k, len(data),)) return self.fec.encode(l) class Decoder(object): def __init__(self, k, m): self.fec = zfec.Decoder(k, m) def decode(self, blocks, sharenums, padlen): """ @param padlen: the number of bytes of padding to strip off; Note that the padlen is always equal to (blocksize times k) minus the length of data. (Therefore, padlen can be 0.) """ data = b''.join(self.fec.decode(blocks, sharenums)) if padlen: return data[:-padlen] else: return data # zfec -- fast forward error correction library with Python interface # # Copyright (C) 2007 Allmydata, Inc. # Author: Zooko Wilcox-O'Hearn # # This file is part of zfec. # # See README.rst for licensing information. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/zfec/fec.c0000644000175100001770000004650214515776411014074 0ustar00runnerdocker/** * zfec -- fast forward error correction library with Python interface */ #include "fec.h" #include #include #include #include /* * Primitive polynomials - see Lin & Costello, Appendix A, * and Lee & Messerschmitt, p. 453. */ static const char*const Pp="101110001"; /* * To speed up computations, we have tables for logarithm, exponent and * inverse of a number. We use a table for multiplication as well (it takes * 64K, no big deal even on a PDA, especially because it can be * pre-initialized an put into a ROM!), otherwhise we use a table of * logarithms. In any case the macro gf_mul(x,y) takes care of * multiplications. */ static gf gf_exp[510]; /* index->poly form conversion table */ static int gf_log[256]; /* Poly->index form conversion table */ static gf inverse[256]; /* inverse of field elem. */ /* inv[\alpha**i]=\alpha**(GF_SIZE-i-1) */ /* * modnn(x) computes x % GF_SIZE, where GF_SIZE is 2**GF_BITS - 1, * without a slow divide. */ static gf modnn(int x) { while (x >= 255) { x -= 255; x = (x >> 8) + (x & 255); } return x; } #define SWAP(a,b,t) {t tmp; tmp=a; a=b; b=tmp;} /* * gf_mul(x,y) multiplies two numbers. It is much faster to use a * multiplication table. * * USE_GF_MULC, GF_MULC0(c) and GF_ADDMULC(x) can be used when multiplying * many numbers by the same constant. In this case the first call sets the * constant, and others perform the multiplications. A value related to the * multiplication is held in a local variable declared with USE_GF_MULC . See * usage in _addmul1(). */ static gf gf_mul_table[256][256]; #define gf_mul(x,y) gf_mul_table[x][y] #define USE_GF_MULC register gf * __gf_mulc_ #define GF_MULC0(c) __gf_mulc_ = gf_mul_table[c] #define GF_ADDMULC(dst, x) dst ^= __gf_mulc_[x] /* * Generate GF(2**m) from the irreducible polynomial p(X) in p[0]..p[m] * Lookup tables: * index->polynomial form gf_exp[] contains j= \alpha^i; * polynomial form -> index form gf_log[ j = \alpha^i ] = i * \alpha=x is the primitive element of GF(2^m) * * For efficiency, gf_exp[] has size 2*GF_SIZE, so that a simple * multiplication of two numbers can be resolved without calling modnn */ static void _init_mul_table(void) { int i, j; for (i = 0; i < 256; i++) for (j = 0; j < 256; j++) gf_mul_table[i][j] = gf_exp[modnn (gf_log[i] + gf_log[j])]; for (j = 0; j < 256; j++) gf_mul_table[0][j] = gf_mul_table[j][0] = 0; } #define NEW_GF_MATRIX(rows, cols) \ (gf*)malloc(rows * cols) /* * initialize the data structures used for computations in GF. */ static void generate_gf (void) { int i; gf mask; mask = 1; /* x ** 0 = 1 */ gf_exp[8] = 0; /* will be updated at the end of the 1st loop */ /* * first, generate the (polynomial representation of) powers of \alpha, * which are stored in gf_exp[i] = \alpha ** i . * At the same time build gf_log[gf_exp[i]] = i . * The first 8 powers are simply bits shifted to the left. */ for (i = 0; i < 8; i++, mask <<= 1) { gf_exp[i] = mask; gf_log[gf_exp[i]] = i; /* * If Pp[i] == 1 then \alpha ** i occurs in poly-repr * gf_exp[8] = \alpha ** 8 */ if (Pp[i] == '1') gf_exp[8] ^= mask; } /* * now gf_exp[8] = \alpha ** 8 is complete, so can also * compute its inverse. */ gf_log[gf_exp[8]] = 8; /* * Poly-repr of \alpha ** (i+1) is given by poly-repr of * \alpha ** i shifted left one-bit and accounting for any * \alpha ** 8 term that may occur when poly-repr of * \alpha ** i is shifted. */ mask = 1 << 7; for (i = 9; i < 255; i++) { if (gf_exp[i - 1] >= mask) gf_exp[i] = gf_exp[8] ^ ((gf_exp[i - 1] ^ mask) << 1); else gf_exp[i] = gf_exp[i - 1] << 1; gf_log[gf_exp[i]] = i; } /* * log(0) is not defined, so use a special value */ gf_log[0] = 255; /* set the extended gf_exp values for fast multiply */ for (i = 0; i < 255; i++) gf_exp[i + 255] = gf_exp[i]; /* * again special cases. 0 has no inverse. This used to * be initialized to 255, but it should make no difference * since noone is supposed to read from here. */ inverse[0] = 0; inverse[1] = 1; for (i = 2; i <= 255; i++) inverse[i] = gf_exp[255 - gf_log[i]]; } /* * Various linear algebra operations that i use often. */ /* * addmul() computes dst[] = dst[] + c * src[] * This is used often, so better optimize it! Currently the loop is * unrolled 16 times, a good value for 486 and pentium-class machines. * The case c=0 is also optimized, whereas c=1 is not. These * calls are unfrequent in my typical apps so I did not bother. */ #define addmul(dst, src, c, sz) \ if (c != 0) _addmul1(dst, src, c, sz) #define UNROLL 16 /* 1, 4, 8, 16 */ static void _addmul1(register gf*restrict dst, const register gf*restrict src, gf c, size_t sz) { USE_GF_MULC; const gf* lim = &dst[sz - UNROLL + 1]; GF_MULC0 (c); #if (UNROLL > 1) /* unrolling by 8/16 is quite effective on the pentium */ for (; dst < lim; dst += UNROLL, src += UNROLL) { GF_ADDMULC (dst[0], src[0]); GF_ADDMULC (dst[1], src[1]); GF_ADDMULC (dst[2], src[2]); GF_ADDMULC (dst[3], src[3]); #if (UNROLL > 4) GF_ADDMULC (dst[4], src[4]); GF_ADDMULC (dst[5], src[5]); GF_ADDMULC (dst[6], src[6]); GF_ADDMULC (dst[7], src[7]); #endif #if (UNROLL > 8) GF_ADDMULC (dst[8], src[8]); GF_ADDMULC (dst[9], src[9]); GF_ADDMULC (dst[10], src[10]); GF_ADDMULC (dst[11], src[11]); GF_ADDMULC (dst[12], src[12]); GF_ADDMULC (dst[13], src[13]); GF_ADDMULC (dst[14], src[14]); GF_ADDMULC (dst[15], src[15]); #endif } #endif lim += UNROLL - 1; for (; dst < lim; dst++, src++) /* final components */ GF_ADDMULC (*dst, *src); } /* * computes C = AB where A is n*k, B is k*m, C is n*m */ static void _matmul(gf * a, gf * b, gf * c, unsigned n, unsigned k, unsigned m) { unsigned row, col, i; for (row = 0; row < n; row++) { for (col = 0; col < m; col++) { gf *pa = &a[row * k]; gf *pb = &b[col]; gf acc = 0; for (i = 0; i < k; i++, pa++, pb += m) acc ^= gf_mul (*pa, *pb); c[row * m + col] = acc; } } } /* * _invert_mat() takes a matrix and produces its inverse * k is the size of the matrix. * (Gauss-Jordan, adapted from Numerical Recipes in C) * Return non-zero if singular. */ static void _invert_mat(gf* src, size_t k) { gf c; size_t irow = 0; size_t icol = 0; size_t row, col, i, ix; unsigned* indxc = (unsigned*) malloc (k * sizeof(unsigned)); unsigned* indxr = (unsigned*) malloc (k * sizeof(unsigned)); unsigned* ipiv = (unsigned*) malloc (k * sizeof(unsigned)); gf *id_row = NEW_GF_MATRIX (1, k); memset (id_row, '\0', k * sizeof (gf)); /* * ipiv marks elements already used as pivots. */ for (i = 0; i < k; i++) ipiv[i] = 0; for (col = 0; col < k; col++) { gf *pivot_row; /* * Zeroing column 'col', look for a non-zero element. * First try on the diagonal, if it fails, look elsewhere. */ if (ipiv[col] != 1 && src[col * k + col] != 0) { irow = col; icol = col; goto found_piv; } for (row = 0; row < k; row++) { if (ipiv[row] != 1) { for (ix = 0; ix < k; ix++) { if (ipiv[ix] == 0) { if (src[row * k + ix] != 0) { irow = row; icol = ix; goto found_piv; } } else assert (ipiv[ix] <= 1); } } } found_piv: ++(ipiv[icol]); /* * swap rows irow and icol, so afterwards the diagonal * element will be correct. Rarely done, not worth * optimizing. */ if (irow != icol) for (ix = 0; ix < k; ix++) SWAP (src[irow * k + ix], src[icol * k + ix], gf); indxr[col] = irow; indxc[col] = icol; pivot_row = &src[icol * k]; c = pivot_row[icol]; assert (c != 0); if (c != 1) { /* otherwhise this is a NOP */ /* * this is done often , but optimizing is not so * fruitful, at least in the obvious ways (unrolling) */ c = inverse[c]; pivot_row[icol] = 1; for (ix = 0; ix < k; ix++) pivot_row[ix] = gf_mul (c, pivot_row[ix]); } /* * from all rows, remove multiples of the selected row * to zero the relevant entry (in fact, the entry is not zero * because we know it must be zero). * (Here, if we know that the pivot_row is the identity, * we can optimize the addmul). */ id_row[icol] = 1; if (memcmp (pivot_row, id_row, k * sizeof (gf)) != 0) { gf *p = src; for (ix = 0; ix < k; ix++, p += k) { if (ix != icol) { c = p[icol]; p[icol] = 0; addmul (p, pivot_row, c, k); } } } id_row[icol] = 0; } /* done all columns */ for (col = k; col > 0; col--) if (indxr[col-1] != indxc[col-1]) for (row = 0; row < k; row++) SWAP (src[row * k + indxr[col-1]], src[row * k + indxc[col-1]], gf); free(indxc); free(indxr); free(ipiv); free(id_row); } /* * fast code for inverting a vandermonde matrix. * * NOTE: It assumes that the matrix is not singular and _IS_ a vandermonde * matrix. Only uses the second column of the matrix, containing the p_i's. * * Algorithm borrowed from "Numerical recipes in C" -- sec.2.8, but largely * revised for my purposes. * p = coefficients of the matrix (p_i) * q = values of the polynomial (known) */ void _invert_vdm (gf* src, unsigned k) { unsigned i, j, row, col; gf *b, *c, *p; gf t, xx; if (k == 1) /* degenerate case, matrix must be p^0 = 1 */ return; /* * c holds the coefficient of P(x) = Prod (x - p_i), i=0..k-1 * b holds the coefficient for the matrix inversion */ c = NEW_GF_MATRIX (1, k); b = NEW_GF_MATRIX (1, k); p = NEW_GF_MATRIX (1, k); for (j = 1, i = 0; i < k; i++, j += k) { c[i] = 0; p[i] = src[j]; /* p[i] */ } /* * construct coeffs. recursively. We know c[k] = 1 (implicit) * and start P_0 = x - p_0, then at each stage multiply by * x - p_i generating P_i = x P_{i-1} - p_i P_{i-1} * After k steps we are done. */ c[k - 1] = p[0]; /* really -p(0), but x = -x in GF(2^m) */ for (i = 1; i < k; i++) { gf p_i = p[i]; /* see above comment */ for (j = k - 1 - (i - 1); j < k - 1; j++) c[j] ^= gf_mul (p_i, c[j + 1]); c[k - 1] ^= p_i; } for (row = 0; row < k; row++) { /* * synthetic division etc. */ xx = p[row]; t = 1; b[k - 1] = 1; /* this is in fact c[k] */ for (i = k - 1; i > 0; i--) { b[i-1] = c[i] ^ gf_mul (xx, b[i]); t = gf_mul (xx, t) ^ b[i-1]; } for (col = 0; col < k; col++) src[col * k + row] = gf_mul (inverse[t], b[col]); } free (c); free (b); free (p); return; } /* There are few (if any) ordering guarantees that apply to reads and writes * of this static int across threads. This is the reason for some of the * tight requirements for how `fec_init` is called. If we could use a mutex * or a C11 atomic here we might be able to provide more flexibility to * callers. It's tricky to do that while remaining compatible with all of * macOS/Linux/Windows and CPython's MSVC requirements and not switching to * C++ (or something even more different). */ static int fec_initialized = 0; void fec_init (void) { if (fec_initialized == 0) { generate_gf(); _init_mul_table(); fec_initialized = 1; } } /* * This section contains the proper FEC encoding/decoding routines. * The encoding matrix is computed starting with a Vandermonde matrix, * and then transforming it into a systematic matrix. */ #define FEC_MAGIC 0xFECC0DEC void fec_free (fec_t *p) { assert (p != NULL && p->magic == (((FEC_MAGIC ^ p->k) ^ p->n) ^ (unsigned long) (p->enc_matrix))); free (p->enc_matrix); free (p); } fec_t * fec_new(unsigned short k, unsigned short n) { unsigned row, col; gf *p, *tmp_m; fec_t *retval; assert(k >= 1); assert(n >= 1); assert(n <= 256); assert(k <= n); if (fec_initialized == 0) { return NULL; } retval = (fec_t *) malloc (sizeof (fec_t)); retval->k = k; retval->n = n; retval->enc_matrix = NEW_GF_MATRIX (n, k); retval->magic = ((FEC_MAGIC ^ k) ^ n) ^ (unsigned long) (retval->enc_matrix); tmp_m = NEW_GF_MATRIX (n, k); /* * fill the matrix with powers of field elements, starting from 0. * The first row is special, cannot be computed with exp. table. */ tmp_m[0] = 1; for (col = 1; col < k; col++) tmp_m[col] = 0; for (p = tmp_m + k, row = 0; row + 1 < n; row++, p += k) for (col = 0; col < k; col++) p[col] = gf_exp[modnn (row * col)]; /* * quick code to build systematic matrix: invert the top * k*k vandermonde matrix, multiply right the bottom n-k rows * by the inverse, and construct the identity matrix at the top. */ _invert_vdm (tmp_m, k); /* much faster than _invert_mat */ _matmul(tmp_m + k * k, tmp_m, retval->enc_matrix + k * k, n - k, k, k); /* * the upper matrix is I so do not bother with a slow multiply */ memset (retval->enc_matrix, '\0', k * k * sizeof (gf)); for (p = retval->enc_matrix, col = 0; col < k; col++, p += k + 1) *p = 1; free (tmp_m); return retval; } /* To make sure that we stay within cache in the inner loops of fec_encode(). (It would probably help to also do this for fec_decode(). */ #ifndef STRIDE #define STRIDE 8192 #endif void fec_encode(const fec_t* code, const gf*restrict const*restrict const src, gf*restrict const*restrict const fecs, const unsigned*restrict const block_nums, size_t num_block_nums, size_t sz) { unsigned char i, j; size_t k; unsigned fecnum; const gf* p; for (k = 0; k < sz; k += STRIDE) { size_t stride = ((sz-k) < STRIDE)?(sz-k):STRIDE; for (i=0; i= code->k); memset(fecs[i]+k, 0, stride); p = &(code->enc_matrix[fecnum * code->k]); for (j = 0; j < code->k; j++) addmul(fecs[i]+k, src[j]+k, p[j], stride); } } } /** * Build decode matrix into some memory space. * * @param matrix a space allocated for a k by k matrix */ void build_decode_matrix_into_space(const fec_t*restrict const code, const unsigned*const restrict index, const unsigned k, gf*restrict const matrix) { unsigned short i; gf* p; for (i=0, p=matrix; i < k; i++, p += k) { if (index[i] < k) { memset(p, 0, k); p[i] = 1; } else { memcpy(p, &(code->enc_matrix[index[i] * code->k]), k); } } _invert_mat (matrix, k); } void fec_decode(const fec_t* code, const gf*restrict const*restrict const inpkts, gf*restrict const*restrict const outpkts, const unsigned*restrict const index, size_t sz) { gf* m_dec = (gf*)alloca(code->k * code->k); /* char is large enough for outix - it counts the number of primary blocks we are decoding for return. the most primary blocks we might have to decode is for k == 128, m == 256. in this case we might be given 128 secondary blocks and have to decode 128 primary blocks. if k decreases then the number of total blocks we might have to return decreases. if k increases then the number of secondary blocks that exist decreases so we will be passed some primary blocks and the number of primary blocks we have to decode decreases. */ unsigned char outix=0; /* row and col are compared directly to k, which could be 256, so make them large enough to represent 256. */ unsigned short row=0; unsigned short col=0; build_decode_matrix_into_space(code, index, code->k, m_dec); for (row=0; rowk; row++) { assert ((index[row] >= code->k) || (index[row] == row)); /* If the block whose number is i is present, then it is required to be in the i'th element. */ if (index[row] >= code->k) { memset(outpkts[outix], 0, sz); for (col=0; col < code->k; col++) addmul(outpkts[outix], inpkts[col], m_dec[row * code->k + col], sz); outix++; } } } /** * zfec -- fast forward error correction library with Python interface * * Copyright (C) 2007-2010 Zooko Wilcox-O'Hearn * Author: Zooko Wilcox-O'Hearn * * This file is part of zfec. * * See README.rst for licensing information. */ /* * This work is derived from the "fec" software by Luigi Rizzo, et al., the * copyright notice and licence terms of which are included below for reference. * fec.c -- forward error correction based on Vandermonde matrices 980624 (C) * 1997-98 Luigi Rizzo (luigi@iet.unipi.it) * * Portions derived from code by Phil Karn (karn@ka9q.ampr.org), * Robert Morelos-Zaragoza (robert@spectra.eng.hawaii.edu) and Hari * Thirumoorthy (harit@spectra.eng.hawaii.edu), Aug 1995 * * Modifications by Dan Rubenstein (see Modifications.txt for * their description. * Modifications (C) 1998 Dan Rubenstein (drubenst@cs.umass.edu) * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above * copyright notice, this list of conditions and the following * disclaimer in the documentation and/or other materials * provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, * OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR * TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY * OF SUCH DAMAGE. */ ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/zfec/fec.h0000644000175100001770000001057714515776411014104 0ustar00runnerdocker/** * zfec -- fast forward error correction library with Python interface * * See README.rst for documentation. */ #include typedef unsigned char gf; typedef struct { unsigned long magic; unsigned short k, n; /* parameters of the code */ gf* enc_matrix; } fec_t; #if defined(_MSC_VER) // actually, some of the flavors (i.e. Enterprise) do support restrict //#define restrict __restrict #define restrict #endif /** Initialize the fec library. * * Call this: * * - at least once * - from at most one thread at a time * - before calling any other APIs from the library * - before creating any other threads that will use APIs from the library * */ void fec_init(void); /** * param k the number of blocks required to reconstruct * param m the total number of blocks created */ fec_t* fec_new(unsigned short k, unsigned short m); void fec_free(fec_t* p); /** * @param inpkts the "primary blocks" i.e. the chunks of the input data * @param fecs buffers into which the secondary blocks will be written * @param block_nums the numbers of the desired check blocks (the id >= k) which fec_encode() will produce and store into the buffers of the fecs parameter * @param num_block_nums the length of the block_nums array * @param sz size of a packet in bytes */ void fec_encode(const fec_t* code, const gf*restrict const*restrict const src, gf*restrict const*restrict const fecs, const unsigned*restrict const block_nums, size_t num_block_nums, size_t sz); /** * @param inpkts an array of packets (size k); If a primary block, i, is present then it must be at index i. Secondary blocks can appear anywhere. * @param outpkts an array of buffers into which the reconstructed output packets will be written (only packets which are not present in the inpkts input will be reconstructed and written to outpkts) * @param index an array of the blocknums of the packets in inpkts * @param sz size of a packet in bytes */ void fec_decode(const fec_t* code, const gf*restrict const*restrict const inpkts, gf*restrict const*restrict const outpkts, const unsigned*restrict const index, size_t sz); #if defined(_MSC_VER) #define alloca _alloca #else #ifdef __GNUC__ #ifndef alloca #define alloca(x) __builtin_alloca(x) #endif #else #include #endif #endif /** * zfec -- fast forward error correction library with Python interface * * Copyright (C) 2007-2008 Allmydata, Inc. * Author: Zooko Wilcox-O'Hearn * * This file is part of zfec. * * See README.rst for licensing information. */ /* * Much of this work is derived from the "fec" software by Luigi Rizzo, et * al., the copyright notice and licence terms of which are included below * for reference. * * fec.h -- forward error correction based on Vandermonde matrices * 980614 * (C) 1997-98 Luigi Rizzo (luigi@iet.unipi.it) * * Portions derived from code by Phil Karn (karn@ka9q.ampr.org), * Robert Morelos-Zaragoza (robert@spectra.eng.hawaii.edu) and Hari * Thirumoorthy (harit@spectra.eng.hawaii.edu), Aug 1995 * * Modifications by Dan Rubenstein (see Modifications.txt for * their description. * Modifications (C) 1998 Dan Rubenstein (drubenst@cs.umass.edu) * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above * copyright notice, this list of conditions and the following * disclaimer in the documentation and/or other materials * provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, * OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, * OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR * TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY * OF SUCH DAMAGE. */ ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168073.0 zfec-1.5.7.4/zfec/filefec.py0000644000175100001770000004377614515776411015154 0ustar00runnerdockerfrom __future__ import print_function import array, os, struct from base64 import b32encode import zfec from zfec import easyfec CHUNKSIZE = 4096 def pad_size(n, k): """ The smallest number that has to be added to n to equal a multiple of k. """ if n%k: return k - n%k else: return 0 def log_ceil(n, b): """ The smallest integer k such that b^k >= n. log_ceil(n, 2) is the number of bits needed to store any of n values, e.g. the number of bits needed to store any of 128 possible values is 7. """ p = 1 k = 0 while p < n: p *= b k += 1 return k def ab(x): # debuggery if len(x) >= 3: return "%s:%s" % (len(x), b32encode(x[-3:]),) elif len(x) == 2: return "%s:%s" % (len(x), b32encode(x[-2:]),) elif len(x) == 1: return "%s:%s" % (len(x), b32encode(x[-1:]),) elif len(x) == 0: return "%s:%s" % (len(x), "--empty--",) class InsufficientShareFilesError(zfec.Error): def __init__(self, k, kb, *args, **kwargs): zfec.Error.__init__(self, *args, **kwargs) self.k = k self.kb = kb def __repr__(self): return "Insufficient share files -- %d share files are required to recover this file, but only %d were given" % (self.k, self.kb,) def __str__(self): return self.__repr__() class CorruptedShareFilesError(zfec.Error): pass def _build_header(m, k, pad, sh): """ @param m: the total number of shares; 1 <= m <= 256 @param k: the number of shares required to reconstruct; 1 <= k <= m @param pad: the number of bytes of padding added to the file before encoding; 0 <= pad < k @param sh: the shnum of this share; 0 <= k < m @return: a compressed string encoding m, k, pad, and sh """ assert m >= 1 assert m <= 2**8 assert k >= 1 assert k <= m assert pad >= 0 assert pad < k assert sh >= 0 assert sh < m bitsused = 0 val = 0 val |= (m - 1) bitsused += 8 # the first 8 bits always encode m kbits = log_ceil(m, 2) # num bits needed to store all possible values of k val <<= kbits bitsused += kbits val |= (k - 1) padbits = log_ceil(k, 2) # num bits needed to store all possible values of pad val <<= padbits bitsused += padbits val |= pad shnumbits = log_ceil(m, 2) # num bits needed to store all possible values of shnum val <<= shnumbits bitsused += shnumbits val |= sh assert bitsused >= 8, bitsused assert bitsused <= 32, bitsused if bitsused <= 16: val <<= (16-bitsused) cs = struct.pack('>H', val) assert cs[:-2] == b'\x00' * (len(cs)-2) return cs[-2:] if bitsused <= 24: val <<= (24-bitsused) cs = struct.pack('>I', val) assert cs[:-3] == b'\x00' * (len(cs)-3) return cs[-3:] else: val <<= (32-bitsused) cs = struct.pack('>I', val) assert cs[:-4] == b'\x00' * (len(cs)-4) return cs[-4:] def MASK(bits): return (1<> b2_bits_left) + 1 shbits = log_ceil(m, 2) # num bits needed to store all possible values of shnum padbits = log_ceil(k, 2) # num bits needed to store all possible values of pad val = byte & (~kbitmask) needed_padbits = padbits - b2_bits_left if needed_padbits > 0: ch = inf.read(1) if not ch: raise CorruptedShareFilesError("Share files were corrupted -- share file %r didn't have a complete metadata header at the front. Perhaps the file was truncated." % (inf.name,)) byte = struct.unpack(">B", ch)[0] val <<= 8 val |= byte needed_padbits -= 8 assert needed_padbits <= 0 extrabits = -needed_padbits pad = val >> extrabits val &= MASK(extrabits) needed_shbits = shbits - extrabits if needed_shbits > 0: ch = inf.read(1) if not ch: raise CorruptedShareFilesError("Share files were corrupted -- share file %r didn't have a complete metadata header at the front. Perhaps the file was truncated." % (inf.name,)) byte = struct.unpack(">B", ch)[0] val <<= 8 val |= byte needed_shbits -= 8 assert needed_shbits <= 0 gotshbits = -needed_shbits sh = val >> gotshbits return (m, k, pad, sh,) FORMAT_FORMAT = "%%s.%%0%dd_%%0%dd%%s" RE_FORMAT = "%s.[0-9]+_[0-9]+%s" def encode_to_files(inf, fsize, dirname, prefix, k, m, suffix=".fec", overwrite=False, verbose=False): """ Encode inf, writing the shares to specially named, newly created files. @param fsize: calling read() on inf must yield fsize bytes of data and then raise an EOFError @param dirname: the name of the directory into which the sharefiles will be written """ mlen = len(str(m)) format = FORMAT_FORMAT % (mlen, mlen,) padbytes = pad_size(fsize, k) fns = [] fs = [] got_error = False try: for shnum in range(m): hdr = _build_header(m, k, padbytes, shnum) fn = os.path.join(dirname, format % (prefix, shnum, m, suffix,)) if verbose: print("Creating share file %r..." % (fn,)) if overwrite: f = open(fn, "wb") else: flags = os.O_WRONLY|os.O_CREAT|os.O_EXCL | (hasattr(os, 'O_BINARY') and os.O_BINARY) fd = os.open(fn, flags) f = os.fdopen(fd, "wb") fs.append(f) fns.append(fn) f.write(hdr) sumlen = [0] def cb(blocks, length): assert len(blocks) == len(fs) oldsumlen = sumlen[0] sumlen[0] += length if verbose: if int((float(oldsumlen) / fsize) * 10) != int((float(sumlen[0]) / fsize) * 10): print(str(int((float(sumlen[0]) / fsize) * 10) * 10) + "% ...", end=" ") if sumlen[0] > fsize: raise IOError("Wrong file size -- possibly the size of the file changed during encoding. Original size: %d, observed size at least: %s" % (fsize, sumlen[0],)) for i in range(len(blocks)): data = blocks[i] fs[i].write(data) length -= len(data) encode_file_stringy_easyfec(inf, cb, k, m, chunksize=4096) except EnvironmentError as le: print("Cannot complete because of exception: ") print(le) got_error = True finally: for f in fs: f.close() if got_error: print("Cleaning up...") # clean up for fn in fns: if verbose: print("Cleaning up: trying to remove %r..." % (fn,)) try: os.remove(fn) except EnvironmentError: pass return 1 if verbose: print() print("Done!") return 0 # Note: if you really prefer base-2 and you change this code, then please # denote 2^20 as "MiB" instead of "MB" in order to avoid ambiguity. See: # http://en.wikipedia.org/wiki/Megabyte # Thanks. MILLION_BYTES=10**6 def decode_from_files(outf, infiles, verbose=False): """ Decode from the first k files in infiles, writing the results to outf. """ assert len(infiles) >= 2 infs = [] shnums = [] m = None k = None padlen = None byteswritten = 0 for f in infiles: (nm, nk, npadlen, shnum,) = _parse_header(f) if not (m is None or m == nm): raise CorruptedShareFilesError("Share files were corrupted -- share file %r said that m was %s but another share file previously said that m was %s" % (f.name, nm, m,)) m = nm if not (k is None or k == nk): raise CorruptedShareFilesError("Share files were corrupted -- share file %r said that k was %s but another share file previously said that k was %s" % (f.name, nk, k,)) if not (k is None or k <= len(infiles)): raise InsufficientShareFilesError(k, len(infiles)) k = nk if not (padlen is None or padlen == npadlen): raise CorruptedShareFilesError("Share files were corrupted -- share file %r said that pad length was %s but another share file previously said that pad length was %s" % (f.name, npadlen, padlen,)) padlen = npadlen infs.append(f) shnums.append(shnum) if len(infs) == k: break dec = easyfec.Decoder(k, m) while True: chunks = [ inf.read(CHUNKSIZE) for inf in infs ] if [ch for ch in chunks if len(ch) != len(chunks[-1])]: raise CorruptedShareFilesError("Share files were corrupted -- all share files are required to be the same length, but they weren't.") if len(chunks[-1]) > 0: resultdata = dec.decode(chunks, shnums, padlen=0) outf.write(resultdata) byteswritten += len(resultdata) if verbose: if ((byteswritten - len(resultdata)) / (10*MILLION_BYTES)) != (byteswritten / (10*MILLION_BYTES)): print(str(byteswritten / MILLION_BYTES) + " MB ...", end=" ") else: if padlen > 0: outf.truncate(byteswritten - padlen) return # Done. if verbose: print() print("Done!") def encode_file(inf, cb, k, m, chunksize=4096): """ Read in the contents of inf, encode, and call cb with the results. First, k "input blocks" will be read from inf, each input block being of size chunksize. Then these k blocks will be encoded into m "result blocks". Then cb will be invoked, passing a list of the m result blocks as its first argument, and the length of the encoded data as its second argument. (The length of the encoded data is always equal to k*chunksize, until the last iteration, when the end of the file has been reached and less than k*chunksize bytes could be read from the file.) This procedure is iterated until the end of the file is reached, in which case the space of the input blocks that is unused is filled with zeroes before encoding. Note that the sequence passed in calls to cb() contains mutable array objects in its first k elements whose contents will be overwritten when the next segment is read from the input file. Therefore the implementation of cb() has to either be finished with those first k arrays before returning, or if it wants to keep the contents of those arrays for subsequent use after it has returned then it must make a copy of them to keep. @param inf the file object from which to read the data @param cb the callback to be invoked with the results @param k the number of shares required to reconstruct the file @param m the total number of shares created @param chunksize how much data to read from inf for each of the k input blocks """ enc = zfec.Encoder(k, m) l = tuple([ array.array('c') for i in range(k) ]) indatasize = k*chunksize # will be reset to shorter upon EOF eof = False ZEROES=array.array('c', ['\x00'])*chunksize while not eof: # This loop body executes once per segment. i = 0 while (i= 3: return "%s:%s" % (len(x), b32encode(x[-3:]),) elif len(x) == 2: return "%s:%s" % (len(x), b32encode(x[-2:]),) elif len(x) == 1: return "%s:%s" % (len(x), b32encode(x[-1:]),) elif len(x) == 0: return "%s:%s" % (len(x), "--empty--",) def randstr(n): return os.urandom(n) def _h(k, m, ss): encer = zfec.Encoder(k, m) nums_and_blocks = list(enumerate(encer.encode(ss))) assert isinstance(nums_and_blocks, list), nums_and_blocks assert len(nums_and_blocks) == m, (len(nums_and_blocks), m,) nums_and_blocks = random.sample(nums_and_blocks, k) blocks = [ x[1] for x in nums_and_blocks ] nums = [ x[0] for x in nums_and_blocks ] decer = zfec.Decoder(k, m) decoded = decer.decode(blocks, nums) assert len(decoded) == len(ss), (len(decoded), len(ss),) assert tuple([str(s) for s in decoded]) == tuple([str(s) for s in ss]), (tuple([ab(str(s)) for s in decoded]), tuple([ab(str(s)) for s in ss]),) def _help_test_random(): m = random.randrange(1, 257) k = random.randrange(1, m+1) l = random.randrange(0, 2**9) ss = [ randstr(l//k) for x in range(k) ] _h(k, m, ss) def _h_easy(k, m, s): encer = zfec.easyfec.Encoder(k, m) nums_and_blocks = list(enumerate(encer.encode(s))) assert isinstance(nums_and_blocks, list), nums_and_blocks assert len(nums_and_blocks) == m, (len(nums_and_blocks), m,) nums_and_blocks = random.sample(nums_and_blocks, k) blocks = [ x[1] for x in nums_and_blocks ] nums = [ x[0] for x in nums_and_blocks ] decer = zfec.easyfec.Decoder(k, m) decodeds = decer.decode(blocks, nums, padlen=k*len(blocks[0]) - len(s)) assert len(decodeds) == len(s), (ab(decodeds), ab(s), k, m) assert decodeds == s, (ab(decodeds), ab(s),) def _help_test_random_easy(): m = random.randrange(1, 257) k = random.randrange(1, m+1) l = random.randrange(0, 2**9) s = randstr(l) _h_easy(k, m, s) def _help_test_random_with_l_easy(l): m = random.randrange(1, 257) k = random.randrange(1, m+1) s = randstr(l) _h_easy(k, m, s) class ZFecTest(unittest.TestCase): def test_instantiate_encoder_no_args(self): try: e = zfec.Encoder() del e except TypeError: # Okay, so that's because we're required to pass constructor args. pass else: # Oops, it should have raised an exception. self.fail("Should have raised exception from incorrect arguments to constructor.") def test_instantiate_decoder_no_args(self): try: e = zfec.Decoder() del e except TypeError: # Okay, so that's because we're required to pass constructor args. pass else: # Oops, it should have raised an exception. self.fail("Should have raised exception from incorrect arguments to constructor.") def test_from_agl_c(self): self.assertTrue(zfec._fec.test_from_agl()) def test_from_agl_py(self): e = zfec.Encoder(3, 5) b0 = b'\x01'*8 ; b1 = b'\x02'*8 ; b2 = b'\x03'*8 # print "_from_py before encoding:" # print "b0: %s, b1: %s, b2: %s" % tuple(base64.b16encode(x) for x in [b0, b1, b2]) b3, b4 = e.encode([b0, b1, b2], (3, 4)) # print "after encoding:" # print "b3: %s, b4: %s" % tuple(base64.b16encode(x) for x in [b3, b4]) d = zfec.Decoder(3, 5) r0, r1, r2 = d.decode((b2, b3, b4), (1, 2, 3)) # print "after decoding:" # print "b0: %s, b1: %s" % tuple(base64.b16encode(x) for x in [b0, b1]) @given( integers(min_value=0, max_value=15).flatmap( lambda l: integers(min_value=1, max_value=256).flatmap( lambda m: integers(min_value=1, max_value=m).flatmap( lambda k: lists( binary(min_size=l//k, max_size=l//k), min_size=k, max_size=k, ).flatmap( lambda ss: just((k, m, ss)), ), ), ), ), ) def test_small(self, kmss): """ Short primary blocks (length between 0 and 15) round-trip through Encoder / Decoder for all values of k, m, such that: * 1 <= m <= 256 * 1 <= k <= m """ (k, m, ss) = kmss _h(k, m, ss) def test_random(self): for i in range(3): _help_test_random() if VERBOSE: print("%d randomized tests pass." % (i+1)) def test_bad_args_construct_decoder(self): try: zfec.Decoder(-1, -1) except zfec.Error as e: assert "argument is required to be greater than or equal to 1" in str(e), e else: self.fail("Should have gotten an exception from out-of-range arguments.") try: zfec.Decoder(1, 257) except zfec.Error as e: assert "argument is required to be less than or equal to 256" in str(e), e else: self.fail("Should have gotten an exception from out-of-range arguments.") try: zfec.Decoder(3, 2) except zfec.Error as e: assert "first argument is required to be less than or equal to the second argument" in str(e), e else: self.fail("Should have gotten an exception from out-of-range arguments.") def test_bad_args_construct_encoder(self): try: zfec.Encoder(-1, -1) except zfec.Error as e: assert "argument is required to be greater than or equal to 1" in str(e), e else: self.fail("Should have gotten an exception from out-of-range arguments.") try: zfec.Encoder(1, 257) except zfec.Error as e: assert "argument is required to be less than or equal to 256" in str(e), e else: self.fail("Should have gotten an exception from out-of-range arguments.") def test_bad_args_dec(self): decer = zfec.Decoder(2, 4) try: decer.decode(98, []) # first argument is not a sequence except TypeError as e: assert "First argument was not a sequence" in str(e), e else: self.fail("Should have gotten TypeError for wrong type of second argument.") try: decer.decode(["a", "b", ], ["c", "d",]) except zfec.Error as e: assert "Precondition violation: second argument is required to contain int" in str(e), e else: self.fail("Should have gotten zfec.Error for wrong type of second argument.") try: decer.decode(["a", "b", ], 98) # not a sequence at all except TypeError as e: assert "Second argument was not a sequence" in str(e), e else: self.fail("Should have gotten TypeError for wrong type of second argument.") class EasyFecTest(unittest.TestCase): def test_small(self): for i in range(16): _help_test_random_with_l_easy(i) if VERBOSE: print("%d randomized tests pass." % (i+1)) def test_random(self): for i in range(3): _help_test_random_easy() if VERBOSE: print("%d randomized tests pass." % (i+1)) def test_bad_args_dec(self): decer = zfec.easyfec.Decoder(2, 4) try: decer.decode(98, [0, 1], 0) # first argument is not a sequence except TypeError as e: assert "First argument was not a sequence" in str(e), e else: self.fail("Should have gotten TypeError for wrong type of second argument.") try: decer.decode("ab", ["c", "d",], 0) except zfec.Error as e: assert "Precondition violation: second argument is required to contain int" in str(e), e else: self.fail("Should have gotten zfec.Error for wrong type of second argument.") try: decer.decode("ab", 98, 0) # not a sequence at all except TypeError as e: assert "Second argument was not a sequence" in str(e), e else: self.fail("Should have gotten TypeError for wrong type of second argument.") class FileFec(unittest.TestCase): def test_filefec_header(self): for m in [1, 2, 3, 5, 7, 9, 11, 17, 19, 33, 35, 65, 66, 67, 129, 130, 131, 254, 255, 256,]: for k in [1, 2, 3, 5, 9, 17, 33, 65, 129, 255, 256,]: if k >= m: continue for pad in [0, 1, k-1,]: if pad >= k: continue for sh in [0, 1, m-1,]: if sh >= m: continue h = zfec.filefec._build_header(m, k, pad, sh) hio = BytesIO(h) (rm, rk, rpad, rsh,) = zfec.filefec._parse_header(hio) assert (rm, rk, rpad, rsh,) == (m, k, pad, sh,), h def _help_test_filefec(self, teststr, k, m, numshs=None): if numshs == None: numshs = m TESTFNAME = "testfile.txt" PREFIX = "test" SUFFIX = ".fec" fsize = len(teststr) tempdir = fileutil.NamedTemporaryDirectory(cleanup=True) try: tempf = tempdir.file(TESTFNAME, 'w+b') tempf.write(teststr) tempf.flush() tempf.seek(0) # encode the file zfec.filefec.encode_to_files(tempf, fsize, tempdir.name, PREFIX, k, m, SUFFIX, verbose=VERBOSE) # select some share files RE=re.compile(zfec.filefec.RE_FORMAT % (PREFIX, SUFFIX,)) fns = os.listdir(tempdir.name) assert len(fns) >= m, (fns, tempdir, tempdir.name,) sharefs = [ open(os.path.join(tempdir.name, fn), "rb") for fn in fns if RE.match(fn) ] for sharef in sharefs: tempdir.register_file(sharef) random.shuffle(sharefs) del sharefs[numshs:] # decode from the share files outf = tempdir.file('recovered-testfile.txt', 'w+b') zfec.filefec.decode_from_files(outf, sharefs, verbose=VERBOSE) outf.flush() outf.seek(0) recovereddata = outf.read() assert recovereddata == teststr, (ab(recovereddata), ab(teststr),) finally: tempdir.shutdown() def test_filefec_all_shares(self): return self._help_test_filefec(b"Yellow Whirled!", 3, 8) def test_filefec_all_shares_1_b(self): return self._help_test_filefec(b"Yellow Whirled!", 4, 16) def test_filefec_all_shares_2(self): return self._help_test_filefec(b"Yellow Whirled", 3, 8) def test_filefec_all_shares_2_b(self): return self._help_test_filefec(b"Yellow Whirled", 4, 16) def test_filefec_all_shares_3(self): return self._help_test_filefec(b"Yellow Whirle", 3, 8) def test_filefec_all_shares_3_b(self): return self._help_test_filefec(b"Yellow Whirle", 4, 16) def test_filefec_all_shares_with_padding(self, noisy=VERBOSE): return self._help_test_filefec(b"Yellow Whirled!A", 3, 8) def test_filefec_min_shares_with_padding(self, noisy=VERBOSE): return self._help_test_filefec(b"Yellow Whirled!A", 3, 8, numshs=3) def test_filefec_min_shares_with_crlf(self, noisy=VERBOSE): return self._help_test_filefec(b"llow Whirled!A\r\n", 3, 8, numshs=3) def test_filefec_min_shares_with_lf(self, noisy=VERBOSE): return self._help_test_filefec(b"Yellow Whirled!A\n", 3, 8, numshs=3) def test_filefec_min_shares_with_lflf(self, noisy=VERBOSE): return self._help_test_filefec(b"Yellow Whirled!A\n\n", 3, 8, numshs=3) def test_filefec_min_shares_with_crcrlflf(self, noisy=VERBOSE): return self._help_test_filefec(b"Yellow Whirled!A\r\r\n\n", 3, 8, numshs=3) def test_filefec_mul_chunk_size(self): return self._help_test_filefec(randstr(6176761), 13, 16) class Cmdline(unittest.TestCase): def setUp(self): self.tempdir = fileutil.NamedTemporaryDirectory(cleanup=True) self.fo = self.tempdir.file("test.data", "w+b") self.fo.write(b"WHEHWHJEKWAHDLJAWDHWALKDHA") self.realargv = sys.argv self.DEFAULT_M = 8 self.DEFAULT_K = 3 self.RE = re.compile(zfec.filefec.RE_FORMAT % ('test.data', ".fec",)) def tearDown(self): sys.argv = self.realargv def test_basic(self, noisy=VERBOSE): sys.argv = ["zfec", os.path.join(self.tempdir.name, "test.data"),] retcode = zfec.cmdline_zfec.main() assert retcode == 0, retcode fns = os.listdir(self.tempdir.name) assert len(fns) >= self.DEFAULT_M, (fns, self.DEFAULT_M, self.tempdir, self.tempdir.name,) sharefns = [ os.path.join(self.tempdir.name, fn) for fn in fns if self.RE.match(fn) ] random.shuffle(sharefns) del sharefns[self.DEFAULT_K:] sys.argv = ["zunfec",] sys.argv.extend(sharefns) sys.argv.extend(['-o', os.path.join(self.tempdir.name, 'test.data-recovered'),]) retcode = zfec.cmdline_zunfec.main() assert retcode == 0, retcode import filecmp assert filecmp.cmp(os.path.join(self.tempdir.name, 'test.data'), os.path.join(self.tempdir.name, 'test.data-recovered')) def test_stdin(self, noisy=VERBOSE): sys.stdin = open(os.path.join(self.tempdir.name, "test.data")) sys.argv = ["zfec", "-"] retcode = zfec.cmdline_zfec.main() fns = os.listdir(self.tempdir.name) assert len(fns) >= self.DEFAULT_M, (fns, self.DEFAULT_M, self.tempdir, self.tempdir.name,) sharefns = [ os.path.join(self.tempdir.name, fn) for fn in fns if self.RE.match(fn) ] random.shuffle(sharefns) del sharefns[self.DEFAULT_K:] sys.argv = ["zunfec",] sys.argv.extend(sharefns) sys.argv.extend(['-o', os.path.join(self.tempdir.name, 'test.data-recovered'),]) retcode = zfec.cmdline_zunfec.main() assert retcode == 0, retcode import filecmp assert filecmp.cmp(os.path.join(self.tempdir.name, 'test.data'), os.path.join(self.tempdir.name, 'test.data-recovered')) if __name__ == "__main__": unittest.main() ././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1698168074.996989 zfec-1.5.7.4/zfec.egg-info/0000755000175100001770000000000014515776413014660 5ustar00runnerdocker././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168074.0 zfec-1.5.7.4/zfec.egg-info/PKG-INFO0000644000175100001770000003711014515776412015756 0ustar00runnerdockerMetadata-Version: 2.1 Name: zfec Version: 1.5.7.4 Summary: An efficient, portable erasure coding tool Home-page: https://github.com/tahoe-lafs/zfec Author: Zooko O\'Whielacronx Author-email: zooko@zooko.com License: GPL-2+ Classifier: Development Status :: 5 - Production/Stable Classifier: Environment :: Console Classifier: License :: OSI Approved Classifier: License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+) Classifier: License :: DFSG approved Classifier: License :: Other/Proprietary License Classifier: Intended Audience :: Developers Classifier: Intended Audience :: End Users/Desktop Classifier: Intended Audience :: System Administrators Classifier: Operating System :: Microsoft Classifier: Operating System :: Microsoft :: Windows Classifier: Operating System :: Unix Classifier: Operating System :: POSIX :: Linux Classifier: Operating System :: POSIX Classifier: Operating System :: MacOS :: MacOS X Classifier: Operating System :: Microsoft :: Windows :: Windows NT/2000 Classifier: Operating System :: OS Independent Classifier: Natural Language :: English Classifier: Programming Language :: C Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.7 Classifier: Programming Language :: Python :: 3.8 Classifier: Topic :: Utilities Classifier: Topic :: System :: Systems Administration Classifier: Topic :: System :: Filesystems Classifier: Topic :: System :: Distributed Computing Classifier: Topic :: Software Development :: Libraries Classifier: Topic :: Communications :: Usenet News Classifier: Topic :: System :: Archiving :: Backup Classifier: Topic :: System :: Archiving :: Mirroring Classifier: Topic :: System :: Archiving Description-Content-Type: text/x-rst Provides-Extra: bench Provides-Extra: test License-File: COPYING.GPL License-File: COPYING.TGPPL.rst zfec -- efficient, portable erasure coding tool =============================================== Generate redundant blocks of information such that if some of the blocks are lost then the original data can be recovered from the remaining blocks. This package includes command-line tools, C API, Python API, and Haskell API. |build| |test| |pypi| Intro and Licence ----------------- This package implements an "erasure code", or "forward error correction code". You may use this package under the GNU General Public License, version 2 or, at your option, any later version. You may use this package under the Transitive Grace Period Public Licence, version 1.0 or, at your option, any later version. (You may choose to use this package under the terms of either licence, at your option.) See the file COPYING.GPL for the terms of the GNU General Public License, version 2. See the file COPYING.TGPPL.rst for the terms of the Transitive Grace Period Public Licence, version 1.0. The most widely known example of an erasure code is the RAID-5 algorithm which makes it so that in the event of the loss of any one hard drive, the stored data can be completely recovered. The algorithm in the zfec package has a similar effect, but instead of recovering from the loss of only a single element, it can be parameterized to choose in advance the number of elements whose loss it can tolerate. This package is largely based on the old "fec" library by Luigi Rizzo et al., which is a mature and optimized implementation of erasure coding. The zfec package makes several changes from the original "fec" package, including addition of the Python API, refactoring of the C API to support zero-copy operation, a few clean-ups and optimizations of the core code itself, and the addition of a command-line tool named "zfec". Installation ------------ ``pip install zfec`` To run the self-tests, execute ``tox`` from an unpacked source tree or git checkout. To run the tests of the Haskell API: ``cabal run test:tests`` Community --------- The source is currently available via git on the web with the command: ``git clone https://github.com/tahoe-lafs/zfec`` Please post about zfec to the Tahoe-LAFS mailing list and contribute patches: If you find a bug in zfec, please open an issue on github: Overview -------- This package performs two operations, encoding and decoding. Encoding takes some input data and expands its size by producing extra "check blocks", also called "secondary blocks". Decoding takes some data -- any combination of blocks of the original data (called "primary blocks") and "secondary blocks", and produces the original data. The encoding is parameterized by two integers, k and m. m is the total number of blocks produced, and k is how many of those blocks are necessary to reconstruct the original data. m is required to be at least 1 and at most 256, and k is required to be at least 1 and at most m. (Note that when k == m then there is no point in doing erasure coding -- it degenerates to the equivalent of the Unix "split" utility which simply splits the input into successive segments. Similarly, when k == 1 it degenerates to the equivalent of the unix "cp" utility -- each block is a complete copy of the input data.) Note that each "primary block" is a segment of the original data, so its size is 1/k'th of the size of original data, and each "secondary block" is of the same size, so the total space used by all the blocks is m/k times the size of the original data (plus some padding to fill out the last primary block to be the same size as all the others). In addition to the data contained in the blocks themselves there are also a few pieces of metadata which are necessary for later reconstruction. Those pieces are: 1. the value of K, 2. the value of M, 3. the sharenum of each block, 4. the number of bytes of padding that were used. The "zfec" command-line tool compresses these pieces of data and prepends them to the beginning of each share, so each the sharefile produced by the "zfec" command-line tool is between one and four bytes larger than the share data alone. The decoding step requires as input k of the blocks which were produced by the encoding step. The decoding step produces as output the data that was earlier input to the encoding step. Command-Line Tool ----------------- The bin/ directory contains two Unix-style, command-line tools "zfec" and "zunfec". Execute ``zfec --help`` or ``zunfec --help`` for usage instructions. Performance ----------- To run the benchmarks, execute the included bench/bench_zfec.py script with optional --k= and --m= arguments. Here's the results for an i7-12700k: ``` measuring encoding of data with K=3, M=10, encoding 1000000 bytes 1000 times in a row... Average MB/s: 364 measuring decoding of primary-only data with K=3, M=10, 1000 times in a row... Average MB/s: 1894750 measuring decoding of secondary-only data with K=3, M=10, 1000 times in a row... Average MB/s: 3298 ``` Here is a paper analyzing the performance of various erasure codes and their implementations, including zfec: http://www.usenix.org/events/fast09/tech/full_papers/plank/plank.pdf Zfec shows good performance on different machines and with different values of K and M. It also has a nice small memory footprint. API --- Each block is associated with "blocknum". The blocknum of each primary block is its index (starting from zero), so the 0'th block is the first primary block, which is the first few bytes of the file, the 1'st block is the next primary block, which is the next few bytes of the file, and so on. The last primary block has blocknum k-1. The blocknum of each secondary block is an arbitrary integer between k and 255 inclusive. (When using the Python API, if you don't specify which secondary blocks you want when invoking encode(), then it will by default provide the blocks with ids from k to m-1 inclusive.) - C API fec_encode() takes as input an array of k pointers, where each pointer points to a memory buffer containing the input data (i.e., the i'th buffer contains the i'th primary block). There is also a second parameter which is an array of the blocknums of the secondary blocks which are to be produced. (Each element in that array is required to be the blocknum of a secondary block, i.e. it is required to be >= k and < m.) The output from fec_encode() is the requested set of secondary blocks which are written into output buffers provided by the caller. Note that this fec_encode() is a "low-level" API in that it requires the input data to be provided in a set of memory buffers of exactly the right sizes. If you are starting instead with a single buffer containing all of the data then please see easyfec.py's "class Encoder" as an example of how to split a single large buffer into the appropriate set of input buffers for fec_encode(). If you are starting with a file on disk, then please see filefec.py's encode_file_stringy_easyfec() for an example of how to read the data from a file and pass it to "class Encoder". The Python interface provides these higher-level operations, as does the Haskell interface. If you implement functions to do these higher-level tasks in other languages, please send a patch to tahoe-dev@tahoe-lafs.org so that your API can be included in future releases of zfec. fec_decode() takes as input an array of k pointers, where each pointer points to a buffer containing a block. There is also a separate input parameter which is an array of blocknums, indicating the blocknum of each of the blocks which is being passed in. The output from fec_decode() is the set of primary blocks which were missing from the input and had to be reconstructed. These reconstructed blocks are written into output buffers provided by the caller. - Python API encode() and decode() take as input a sequence of k buffers, where a "sequence" is any object that implements the Python sequence protocol (such as a list or tuple) and a "buffer" is any object that implements the Python buffer protocol (such as a string or array). The contents that are required to be present in these buffers are the same as for the C API. encode() also takes a list of desired blocknums. Unlike the C API, the Python API accepts blocknums of primary blocks as well as secondary blocks in its list of desired blocknums. encode() returns a list of buffer objects which contain the blocks requested. For each requested block which is a primary block, the resulting list contains a reference to the apppropriate primary block from the input list. For each requested block which is a secondary block, the list contains a newly created string object containing that block. decode() also takes a list of integers indicating the blocknums of the blocks being passed int. decode() returns a list of buffer objects which contain all of the primary blocks of the original data (in order). For each primary block which was present in the input list, then the result list simply contains a reference to the object that was passed in the input list. For each primary block which was not present in the input, the result list contains a newly created string object containing that primary block. Beware of a "gotcha" that can result from the combination of mutable data and the fact that the Python API returns references to inputs when possible. Returning references to its inputs is efficient since it avoids making an unnecessary copy of the data, but if the object which was passed as input is mutable and if that object is mutated after the call to zfec returns, then the result from zfec -- which is just a reference to that same object -- will also be mutated. This subtlety is the price you pay for avoiding data copying. If you don't want to have to worry about this then you can simply use immutable objects (e.g. Python strings) to hold the data that you pass to zfec. - Haskell API The Haskell code is fully Haddocked, to generate the documentation, run ``runhaskell Setup.lhs haddock``. Utilities --------- The filefec.py module has a utility function for efficiently reading a file and encoding it piece by piece. This module is used by the "zfec" and "zunfec" command-line tools from the bin/ directory. Dependencies ------------ A C compiler is required. To use the Python API or the command-line tools a Python interpreter is also required. We have tested it with Python v2.7, v3.5 and v3.6. For the Haskell interface, GHC >= 6.8.1 is required. Acknowledgements ---------------- Thanks to the author of the original fec lib, Luigi Rizzo, and the folks that contributed to it: Phil Karn, Robert Morelos-Zaragoza, Hari Thirumoorthy, and Dan Rubenstein. Thanks to the Mnet hackers who wrote an earlier Python wrapper, especially Myers Carpenter and Hauke Johannknecht. Thanks to Brian Warner and Amber O'Whielacronx for help with the API, documentation, debugging, compression, and unit tests. Thanks to Adam Langley for improving the C API and contributing the Haskell API. Thanks to the creators of GCC (starting with Richard M. Stallman) and Valgrind (starting with Julian Seward) for a pair of excellent tools. Thanks to my coworkers at Allmydata -- http://allmydata.com -- Fabrice Grinda, Peter Secor, Rob Kinninmont, Brian Warner, Zandr Milewski, Justin Boreta, Mark Meras for sponsoring this work and releasing it under a Free Software licence. Thanks to Jack Lloyd, Samuel Neves, and David-Sarah Hopwood. Related Works ------------- Note: a Unix-style tool like "zfec" does only one thing -- in this case erasure coding -- and leaves other tasks to other tools. Other Unix-style tools that go well with zfec include `GNU tar`_ for archiving multiple files and directories into one file, `lzip`_ for compression, and `GNU Privacy Guard`_ for encryption or `b2sum`_ for integrity. It is important to do things in order: first archive, then compress, then either encrypt or integrity-check, then erasure code. Note that if GNU Privacy Guard is used for privacy, then it will also ensure integrity, so the use of b2sum is unnecessary in that case. Note also that you also need to do integrity checking (such as with b2sum) on the blocks that result from the erasure coding in *addition* to doing it on the file contents! (There are two different subtle failure modes -- see "more than one file can match an immutable file cap" on the `Hack Tahoe-LAFS!`_ Hall of Fame.) The `Tahoe-LAFS`_ project uses zfec as part of a complete distributed filesystem with integrated encryption, integrity, remote distribution of the blocks, directory structure, backup of changed files or directories, access control, immutable files and directories, proof-of-retrievability, and repair of damaged files and directories. `fecpp`_ is an alternative to zfec. It implements a bitwise-compatible algorithm to zfec and is BSD-licensed. .. _GNU tar: http://directory.fsf.org/project/tar/ .. _lzip: http://www.nongnu.org/lzip/lzip.html .. _GNU Privacy Guard: http://gnupg.org/ .. _b2sum: https://blake2.net/ .. _Tahoe-LAFS: https://tahoe-lafs.org .. _Hack Tahoe-LAFS!: https://tahoe-lafs.org/hacktahoelafs/ .. _fecpp: http://www.randombit.net/code/fecpp/ Enjoy! Zooko Wilcox-O'Hearn 2013-05-15 Boulder, Colorado ---- .. |pypi| image:: http://img.shields.io/pypi/v/zfec.svg :alt: PyPI release status :target: https://pypi.python.org/pypi/zfec .. |build| image:: https://github.com/tahoe-lafs/zfec/actions/workflows/build.yml/badge.svg :alt: Package Build :target: https://github.com/tahoe-lafs/zfec/actions/workflows/build.yml .. |test| image:: https://github.com/tahoe-lafs/zfec/actions/workflows/test.yml/badge.svg :alt: Tests :target: https://github.com/tahoe-lafs/zfec/actions/workflows/test.yml ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168074.0 zfec-1.5.7.4/zfec.egg-info/SOURCES.txt0000644000175100001770000000122514515776412016543 0ustar00runnerdockerCOPYING.GPL COPYING.TGPPL.rst MANIFEST.in NEWS.txt README.rst TODO changelog copyright fec.cabal setup.cfg setup.py stack.yaml stridetune-bench.ba.sh stridetune-bench.py stridetune-dat.bash stridetune-graph.py tox.ini versioneer.py bench/bench_zfec.py haskell/Codec/FEC.hs haskell/test/FECTest.hs zfec/__init__.py zfec/_fecmodule.c zfec/_version.py zfec/cmdline_zfec.py zfec/cmdline_zunfec.py zfec/easyfec.py zfec/fec.c zfec/fec.h zfec/filefec.py zfec.egg-info/PKG-INFO zfec.egg-info/SOURCES.txt zfec.egg-info/dependency_links.txt zfec.egg-info/entry_points.txt zfec.egg-info/requires.txt zfec.egg-info/top_level.txt zfec/test/__init__.py zfec/test/test_zfec.py././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168074.0 zfec-1.5.7.4/zfec.egg-info/dependency_links.txt0000644000175100001770000000000114515776412020725 0ustar00runnerdocker ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168074.0 zfec-1.5.7.4/zfec.egg-info/entry_points.txt0000644000175100001770000000012214515776412020150 0ustar00runnerdocker[console_scripts] zfec = zfec.cmdline_zfec:main zunfec = zfec.cmdline_zunfec:main ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168074.0 zfec-1.5.7.4/zfec.egg-info/requires.txt0000644000175100001770000000010014515776412017246 0ustar00runnerdocker [bench] pyutil>=3.0.0 [test] twisted pyutil>=3.0.0 hypothesis ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1698168074.0 zfec-1.5.7.4/zfec.egg-info/top_level.txt0000644000175100001770000000000514515776412017404 0ustar00runnerdockerzfec